High-maneuverability multi-sensor joint automatic calibration method

Abstract

The invention discloses a high-maneuverability multi-sensor joint automatic calibration method, and belongs to the field of automatic driving. A high-maneuverability trailer hardware device is built in combination with hardware design, meanwhile, software development is matched, namely, a multi-sensor joint automatic calibration method is adopted, the method is suitable for multi-sensor joint automatic calibration in a non-fixed scene, the problem that a whole trailer needs to be returned to a factory can be solved, time and labor cost in the transportation process is saved, and the transportation efficiency is improved. And cooperative mechanical arm hardware is added, so that the multi-sensor joint calibration process can be automated, and the calibration precision is effectively improved. According to the method, multi-sensor joint automatic calibration of the automatic driving vehicle in a non-fixed scene is achieved, and compared with an existing calibration method in a fixed place, the method has the advantages of being flexible and maneuvering.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, and in particular relates to a high-maneuverability multi-sensor joint automatic calibration method. Background Art

[0002] In the existing field of multi-sensor joint calibration for autonomous driving, the sensor installation positions of different vehicles cannot be completely unified due to assembly errors before leaving the factory, and there will be certain errors in the production and assembly process of sensor components. Therefore, multiple sensors must undergo unified data collection and calibration in the calibration workshop before leaving the factory.

[0003] There are generally two common joint calibration methods.

[0004] The first is the markerless calibration method, which does not require a specific calibration plate or artificially constructed scenes. It generally performs automatic internal and external parameter calibration in outdoor natural scenes, but the vehicle is required to remain in motion during the calibration process. The advantage of this method is that the scene has a large degree of freedom and the calibration timing is flexible. The disadvantage is that natural scenes sometimes have no obvious characteristic markers, and the calibration accuracy cannot be guaranteed.

[0005] The second is the calibration method of markers in specific places. In specific scenarios, such as in the calibration workshop, the vehicle is parked in a specific position in the middle or on a rotating platform. Special calibration plates are attached around it according to a certain pattern or multiple mechanical arms are hung on the beam arm. A special calibration plate is fixed at the end of each mechanical arm. The program pre-sets the trajectory of the mechanical arm, and then starts the program to collect the camera and lidar data on the vehicle. Finally, the calibration algorithm is run to obtain the internal and external parameters of the multiple sensors. The advantage of this method is that since the calibration workshop is a relatively fixed environment, higher accuracy and more consistent sensor calibration can be achieved. For the mass production of smart driving vehicles, the calibration workshop is more convenient to form a standardized operating process. However, this type of method has strict requirements on the calibration site, and once the user replaces or disassembles a single sensor during use, the entire vehicle must be returned to the factory for recalibration. For customers, the cycle is long and the time cost is high. Especially for remote areas with weak infrastructure, the calibration and maintenance of multiple sensors of the entire vehicle is very difficult.

[0006] The innovation of the present invention is that the multi-sensor calibration process can be automatically carried out in non-fixed scenarios with guaranteed calibration accuracy, which can greatly shorten the calibration cycle and reduce customer maintenance costs. Summary of the invention

[0007] (I) Purpose of the invention

[0008] The purpose of the present invention is to provide a high-mobility multi-sensor joint automatic calibration method, which combines hardware design, namely building a high-mobility trailer hardware device, and supporting software development, namely a multi-sensor joint automatic calibration method, to solve the problem of returning the entire vehicle to the factory, thereby saving time and labor costs during transportation, and adding collaborative robotic arm hardware to automate the multi-sensor joint calibration process and effectively improve the calibration accuracy.

[0009] (II) Technical solution

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A high-mobility multi-sensor joint automatic calibration method is implemented through a high-mobility trailer hardware device and is used for multi-sensor joint automatic calibration in non-fixed scenarios;

[0012] The hardware device of the high-mobility trailer includes a six-degree-of-freedom mechanical arm, a high-precision calibration plate at the end, a fully automatic vehicle limiter, and a trailer chassis;

[0013] The six-degree-of-freedom mechanical arm is used to move the terminal high-precision calibration plate to the corresponding position according to the calibration data collection requirements through the actions of each joint when the position of the vehicle to be calibrated is fixed;

[0014] The terminal high-precision calibration plate is made of specific materials and processing technology, which can make the image data collected by perception sensors such as cameras and lidar stable and have high accuracy under different working conditions, facilitating subsequent vehicle calibration work;

[0015] The fully automatic vehicle limiter is used to fully automatically fix vehicles of different sizes to be calibrated to prevent relative movement between the vehicle and the robotic arm during the calibration process;

[0016] The trailer chassis serves as a mobile carrier of the vehicle to be calibrated, and is used to provide a unified calibration condition for the vehicle to be calibrated while having high mobility to facilitate data calculation in the calibration program;

[0017] The multi-sensor is a camera and a laser radar. The camera is set at the front position of the vehicle to be calibrated, and the laser radar is set on the roof of the vehicle to be calibrated near the front position. The position of the fully automatic vehicle limiter is fixed according to the size of the vehicle to be calibrated, and the vehicle to be calibrated is moved to the trailer chassis;

[0018] The multi-sensor starts working, the six-degree-of-freedom robotic arm drives the high-precision calibration plate at the end to move according to the preset trajectory, starts the multi-sensor joint automatic calibration program, automatically collects multi-frame time-synchronized robotic arm posture and multi-sensor data, starts to automatically calibrate the internal and external parameters of the multi-sensor, and repeatedly runs the robotic arm until the accuracy reaches the standard, the calibration process is completed, and the robotic arm stops at the initial position;

[0019] The multi-sensor joint automatic calibration procedure performs the following steps:

[0020] Step 1: Get data.

[0021] While the six-degree-of-freedom manipulator moves along the preset trajectory, the onboard camera and lidar data are collected synchronously. Suppose that n frames of synchronous sensor data Φ = {D1, D2, …, D n},in Represents the sensor data set at the i-th moment, including the position and posture of the six-degree-of-freedom robot arm Represents a set of 4-order real matrices and color images N w×h×3 Represents a three-dimensional positive integer matrix set with width w, height h, and depth 3, and a lidar point cloud Represents a set of real vectors with three-dimensional space coordinates;

[0022] Step 2, obtain the internal and external parameters of the camera;

[0023] Using image processing methods, the black and white checkerboard corner points in the calibration plate are detected in each frame of the image, and each corner point is marked with a serial number. The same corner point has different image coordinates in different images, and the camera intrinsic parameter K is obtained. C , external parameter R C and t C , as the initial value of camera calibration, so as to construct the camera projection matrix, that is:

[0024]

[0025] in represents a three-dimensional rotation matrix, Represents a three-dimensional translation vector; the function of the camera projection matrix is ​​to project points in three-dimensional space onto a two-dimensional pixel plane;

[0026] Step 3: Calculate the image coordinates of the four hollow circles in the image, extract the circular area in the calibration plate through image processing, and obtain the coordinates of the center of the circle in the image, which is recorded as As the observed value of the coordinates of the circle center in the image; where h = {1, 2, 3, 4} means that h is equal to any number in the set, corresponding to the serial number of the hollow circle;

[0027] Step 4: Calculate the three-dimensional coordinates of the center of the circle in the laser radar point cloud;

[0028] For the lidar point cloud data at time i Calculate the three-dimensional coordinates of the centers of the four hollow circles on the calibration plate By segmenting the point cloud of the calibration plate area, the hollow circle area is detected, and the three-dimensional coordinates of the center of the circle are calculated;

[0029] Step 5, calculate the two-dimensional projection coordinates of the three-dimensional coordinates of the center of the circle on the image;

[0030] Use the initial value of the transformation matrix from the camera coordinate system to the lidar coordinate system The 3D coordinates of the center of the circle Transform to camera coordinate system And using the initial values ​​of the camera internal and external parameters obtained in the second step, calculate the two-dimensional projection coordinates of the center coordinates of the four hollow circles in the image as the predicted values ​​of the center coordinates in the image, recorded as:

[0031]

[0032] Where π(·):R 3 →R 2 It is the projection transformation from three-dimensional point coordinates to two-dimensional image coordinates;

[0033] Step 6: Optimize and solve the target variable;

[0034] The target variables to be obtained include: camera intrinsic parameters K C , and the external parameters of the camera coordinate system transformed to the lidar coordinate system

[0035] Define the joint optimization objective function J:

[0036]

[0037] Each sub-item is described below;

[0038] J C It represents the chessboard corner projection error loss in camera calibration and is defined as:

[0039]

[0040] M j is the 3D coordinate of the jth checkerboard corner point in the i-th frame. Note that π(·): R 3 →R 2 It is the projection transformation from three-dimensional point coordinates to two-dimensional image coordinates;

[0041] Represents the predicted value u of the center coordinate of the hollow circle of the calibration plate l{1,2,3,4} and the observed value u of the center coordinates of the circle detected in the image h{1，2，3，4} The error loss is defined as:

[0042]

[0043] J RC It represents the error loss of the relative transformation of the checkerboard corner points in camera calibration and is defined as:

[0044]

[0045] in, Represents the transformation matrix from the i-th frame to the k-th frame obtained according to the camera external parameters, with a relative rotation R ik and relative translation t ik Transformation, the external parameters of frame i and frame k obtained during camera calibration and and Calculated, represents the relative rotation and translation transformation from the kth frame to the ith frame obtained according to the robot arm posture, which can be similarly calculated from the robot arm posture at each moment;

[0046] Similarly, J RL The error loss of the relative transformation of the three-dimensional coordinates of the center of the hollow circle in the lidar point is defined as:

[0047]

[0048] in, is the three-dimensional coordinate of the center of the jth hollow circle on the calibration plate in the laser radar coordinate system, is the coordinate of the center point of the jth hollow circle projected onto image i in the laser radar coordinate system;

[0049] Other lambda in the objective function {1,2,3,4} is a coefficient that balances the weights of various error values.

[0050] (III) Effective income

[0051] 1. The present invention has high mobility and is not limited to the factory building. It can be adapted to remote areas with weak infrastructure;

[0052] 2. The present invention introduces high-precision posture information of the robotic arm, adds information constraints of relative rotation and translation transformation to the overall optimization objective function, and can improve the calibration accuracy of the relative posture of multiple sensors;

[0053] 3. After the mechanical arm module is added to the present invention, according to the spatial position relationship between the sensor, the vehicle body and the mechanical arm during the hardware platform design, it is helpful to obtain the external parameters of the laser radar, that is, the transformation of the laser radar coordinate system relative to the vehicle body coordinate system;

[0054] 4. The overall cost of the present invention is lower than that of factory repair, and the cycle is short, the operation is simple, and it is easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the hardware device of the high mobility trailer of the present invention;

[0056] 1a is a front view of the hardware device of the high mobility trailer of the present invention;

[0057] 1b is a side view of the hardware device of the high mobility trailer of the present invention;

[0058] 1c is a top view of the hardware device of the high mobility trailer of the present invention;

[0059] Figure 2 This is a schematic diagram of the working process of the hardware installation of a high mobility trailer;

[0060] Figure 3 This is a working diagram of a high mobility trailer;

[0061] 3a is a front view of the high mobility trailer of the present invention;

[0062] 3b is a working side view of the high mobility trailer of the present invention;

[0063] 3c is a working top view of the high mobility trailer of the present invention;

[0064] Figure 4 A schematic diagram of adding an end calibration plate to the robot arm of the present invention;

[0065] Figure 5 It is a schematic diagram of the combined calibration plate of the present invention;

[0066] Figure 6 The vehicle limiter of the present invention;

[0067] 6a Side view of automatic vehicle restraint;

[0068] 6b Top view of the automatic vehicle restraint. DETAILED DESCRIPTION

[0069] The purpose of the present invention is to provide a high-mobility multi-sensor joint automatic calibration method, which combines hardware design, i.e. building a high-mobility trailer hardware device, and supporting software development, i.e. a multi-sensor joint automatic calibration method, which is suitable for multi-sensor joint automatic calibration in non-fixed scenes. The multi-sensor calibration referred to in the present invention mainly refers to the most common calibration between cameras and laser radars, and the calibration objects mainly include the internal and external parameters of the camera and the external parameters of the camera and radar. The present invention can solve the problem of multi-sensor joint automatic calibration of autonomous driving vehicles in non-fixed scenes, and has the advantage of flexibility and mobility compared to existing calibration methods in fixed places. The following introduces the implementation process of the present invention from hardware design to software development.

[0070] In terms of hardware design, the present invention provides a high-mobility trailer hardware device, such as Figure 1As shown, it includes a six-degree-of-freedom mechanical arm 1, a high-precision calibration plate 2 at the end, a fully automatic vehicle limiter 3, and a trailer chassis 4;

[0071] The six-degree-of-freedom robotic arm is used to move the high-precision calibration plate at the end to the corresponding position according to the calibration data collection requirements through the action of each joint when the position of the vehicle to be calibrated is fixed.

[0072] The terminal high-precision calibration plate is made of specific materials and processing technology. It can make the image data collected by perception sensors such as cameras and lidars stable and with high accuracy under different operating conditions, facilitating subsequent vehicle calibration work.

[0073] The fully automatic vehicle limiter is used to automatically fix vehicles of different sizes to be calibrated to prevent relative movement between the vehicle and the robotic arm during the calibration process.

[0074] The trailer chassis serves as a mobile carrier for the vehicle to be calibrated. It is used to provide a unified calibration condition for the vehicle to be calibrated while having high maneuverability to facilitate data calculation in the calibration program.

[0075] The present invention uses a standard length and width trailer as the chassis for modification and processing. The bottom plate of the trailer chassis is made of high-strength steel, which reduces the overall weight of the trailer chassis while ensuring the load-bearing strength. The upper part of the bottom plate is processed by a machining center for fastening holes, and the communication harness and power supply harness are wired at the bottom of the chassis using industrial wire trough rails. The maximum working radius of the six-degree-of-freedom mechanical arm 1 is 2m, which basically meets the calibration requirements of all passenger vehicles on the market. The six-mechanical arm and the limiter 1 adopt a modular design, and the communication cables and power supply interfaces are both standard plugs. The reserved holes on the bottom plate can be used to realize redundant installation of different calibration positions of the entire bottom plate according to the size of the vehicle to be tested. In addition, this trailer chassis can be matched with a traction head vehicle with traction qualifications, and has flexible and maneuverable characteristics to meet the actual needs of calibration.

[0076] A calibration process for a high mobility trailer hardware device is as follows: Figure 2 As shown, the following steps are included:

[0077] The multi-sensor is a camera 51 and a laser radar 52. The camera 51 is set at the front position of the vehicle to be calibrated, and the laser radar 52 is set on the roof of the vehicle 5 to be calibrated near the front position. The position of the fully automatic vehicle limiter 3 is fixed according to the size of the vehicle 5 to be calibrated, and the vehicle 5 to be calibrated is moved to the trailer chassis 1;

[0078] According to the calibration requirements, one or two sets of six-degree-of-freedom robotic arms 1 are installed symmetrically. If only the front sensor of the vehicle to be calibrated needs to be calibrated together, one set of robotic arms is installed. If both the front and rear sensors need to be calibrated together, two sets of robotic arms are installed. Alternatively, the vehicle can be reversed onto the trailer after the front calibration is completed. Figure 3 shown.

[0079] The multi-sensor starts working, the six-degree-of-freedom robot arm 1 moves the high-precision calibration plate 2 at the end according to the preset trajectory, starts the multi-sensor joint automatic calibration program, automatically collects multi-frame time-synchronized robot arm posture and multi-sensor data, and starts to automatically calibrate the internal and external parameters of the multi-sensor. If the calibration accuracy is not reached, the robot arm is repeatedly operated. If the calibration accuracy is achieved, the calibration process is completed and the robot arm stops at the initial position.

[0080] 2. Multi-sensor joint automatic calibration method

[0081] For the typical joint calibration problem of camera and lidar sensors for self-driving cars, the main objectives to be solved are the camera intrinsic parameters and the extrinsic parameters from the camera coordinate system to the lidar coordinate system. The automatic calibration method mainly corresponds to Figure 2 In the process of collecting multi-frame robot arm pose and multi-sensor data, and starting automatic calibration, two modules are used. Note that the end of the robot arm is hard-connected to the calibration plate, so the pose of the calibration plate is obtained through the pose of the robot arm. Therefore, in the description of the calibration method, unless otherwise specified, the pose data of the calibration plate or the robot arm can be regarded as the same data.

[0082] The automatic calibration method steps adopted by the present invention are:

[0083] Step 1: Get data.

[0084] While the robot arm moves along the preset trajectory, the onboard camera and lidar data are collected synchronously. Suppose that n frames of synchronous sensor data Φ = {D1, D2, …, D n},in Represents the sensor data set at the i-th moment, including the position and posture of the robot arm ( represents a set of 4-order real matrices), color images (N w ×h×3 represents a three-dimensional positive integer matrix set with a width of w, a height of h, and a depth of 3), a lidar point cloud ( Represents a set of real vectors of three-dimensional space coordinates) data;

[0085] Step 2: Obtain the camera’s internal and external parameters. Using image processing methods, detect the black and white checkerboard corners in the calibration plate in each frame of the image, and mark each corner with a serial number. The same corner has different image coordinates in different images. Using methods such as direct linear transform (DLT), obtain the camera’s internal parameters K. C , external parameter R C and t C, as the initial value of camera calibration, so as to construct the camera projection matrix, that is:

[0086]

[0087] in represents a three-dimensional rotation matrix, Represents a three-dimensional translation vector. The function of the camera projection matrix is ​​to project points in three-dimensional space onto a two-dimensional pixel plane.

[0088] Step 3: Calculate the image coordinates of the four hollow circles in the image. By using image processing methods such as Hough transform, extract the circular area in the calibration plate, and calculate the coordinates of the center of the circle in the image, which is recorded as As the observed value of the circle center coordinates in the image; where h = {1, 2, 3, 4} means that h can be equal to any number in the set, that is, the serial number of the corresponding hollow circle.

[0089] Step 4: Calculate the 3D coordinates of the center of the circle in the LiDAR point cloud. For the LiDAR point cloud data at time i First, follow step 3 to calculate the 3D coordinates of the centers of the four hollow circles on the calibration plate. The main method is to segment the point cloud of the calibration plate area, detect the hollow circle area, and then calculate the three-dimensional coordinates of the center of the circle.

[0090] Step 5: Calculate the two-dimensional projection coordinates of the three-dimensional coordinates of the center of the circle on the image. Use the initial value of the transformation matrix from the camera coordinate system to the lidar coordinate system The 3D coordinates of the center of the circle Transform to camera coordinate system And using the initial values ​​of the camera internal and external parameters obtained in the second step, calculate the two-dimensional projection coordinates of the center coordinates of the four hollow circles in the image as the predicted values ​​of the center coordinates in the image, recorded as:

[0091]

[0092] Where π(·):R 3 →R 2 It is the projection transformation from 3D point coordinates to 2D image coordinates.

[0093] Step 6: Optimize and solve the target variable. The target variable to be solved by the method of the present invention includes: camera intrinsic parameter K C , and the external parameters of the camera coordinate system transformed to the lidar coordinate system Define the joint optimization objective function J:

[0094]

[0095] Each sub-item is described below.

[0096] J C It represents the chessboard corner projection error loss in camera calibration and is defined as:

[0097]

[0098] M j is the 3D coordinate of the jth checkerboard corner point in the i-th frame. Note that π(·): R 3 →R 2 It is the projection transformation from 3D point coordinates to 2D image coordinates.

[0099] Represents the predicted value u of the center coordinate of the hollow circle of the calibration plate l{1,2,3,4} and the observed value u of the center coordinates of the circle detected in the image h{1，2，3，4} The error loss is defined as:

[0100]

[0101] J RC It represents the error loss of the relative transformation of the checkerboard corner points in camera calibration and is defined as:

[0102]

[0103] in, Represents the transformation matrix from the i-th frame to the k-th frame obtained according to the camera external parameters, with a relative rotation R ik and relative translation t ik Transformation can be obtained from the external parameters of frame i and frame k obtained during camera calibration and and Calculated, It represents the relative rotation and translation transformation from the kth frame to the ith frame obtained according to the position and posture of the robot arm, which can be similarly calculated from the posture of the robot arm at each moment.

[0104] Similarly, J RL The error loss of the relative transformation of the three-dimensional coordinates of the center of the hollow circle in the lidar point is defined as:

[0105]

[0106] in, is the three-dimensional coordinate of the center of the jth hollow circle on the calibration plate in the laser radar coordinate system, is the coordinate of the center point of the jth hollow circle projected onto image i in the laser radar coordinate system.

[0107] Other lambda in the objective function {1,2,3,4} is a coefficient that balances the weights of various error values.

[0108] The improvement of the present invention lies in that after the objective function introduces the precise position and posture constraint of the manipulator, J is increased compared with other methods. RC and J RL The optimization goal of the variable to be solved requires more consideration of error balance in the optimization of the variable to be solved, thereby improving the accuracy of the variable to be solved as a whole.

[0109] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A highly maneuverable multi-sensor joint automatic calibration method, characterized in that: It is implemented through a high-mobility trailer hardware device and is used for multi-sensor joint automatic calibration in non-fixed scenarios; The hardware device of the high-mobility trailer includes a six-degree-of-freedom mechanical arm, a high-precision calibration plate at the end, a fully automatic vehicle limiter, and a trailer chassis; The six-degree-of-freedom mechanical arm is used to move the terminal high-precision calibration plate to the corresponding position according to the calibration data collection requirements through the actions of each joint when the position of the vehicle to be calibrated is fixed; The terminal high-precision calibration plate is made of specific materials and processing technology, which can make the image data collected by perception sensors such as cameras and lidar stable and have high accuracy under different working conditions, facilitating subsequent vehicle calibration work; The fully automatic vehicle limiter is used to fully automatically fix vehicles of different sizes to be calibrated to prevent relative movement between the vehicle and the robotic arm during the calibration process; The trailer chassis serves as a mobile carrier of the vehicle to be calibrated, and is used to provide a unified calibration condition for the vehicle to be calibrated while having high mobility to facilitate data calculation in the calibration program; The multi-sensor is a camera and a laser radar. The camera is set at the front position of the vehicle to be calibrated, and the laser radar is set on the roof of the vehicle to be calibrated near the front position. The position of the fully automatic vehicle limiter is fixed according to the size of the vehicle to be calibrated, and the vehicle to be calibrated is moved to the trailer chassis; The multi-sensor starts working, the six-degree-of-freedom robotic arm drives the high-precision calibration plate at the end to move according to the preset trajectory, starts the multi-sensor joint automatic calibration program, automatically collects multi-frame time-synchronized robotic arm posture and multi-sensor data, starts to automatically calibrate the internal and external parameters of the multi-sensor, and repeatedly runs the robotic arm until the accuracy reaches the standard, the calibration process is completed, and the robotic arm stops at the initial position; The multi-sensor joint automatic calibration procedure performs the following steps: Step 1: Get data. While the six-degree-of-freedom manipulator moves along the preset trajectory, the onboard camera and lidar data are collected synchronously. Suppose that n frames of synchronous sensor data Φ = {D1, D2, …, D n },in Represents the sensor data set at the i-th moment, including the position and posture of the six-degree-of-freedom robot arm Represents a set of 4-order real matrices and color images N w×h×3 Represents a three-dimensional positive integer matrix set with width w, height h, and depth 3, and a lidar point cloud Represents a set of real vectors with three-dimensional space coordinates; Step 2, obtain the internal and external parameters of the camera; Using image processing methods, the black and white checkerboard corner points in the calibration plate are detected in each frame of the image, and each corner point is marked with a serial number. The same corner point has different image coordinates in different images, and the camera intrinsic parameter K is obtained. C , external parameter R C and t C , as the initial value of camera calibration, so as to construct the camera projection matrix, that is: in represents a three-dimensional rotation matrix, Represents a three-dimensional translation vector; the function of the camera projection matrix is ​​to project points in three-dimensional space onto a two-dimensional pixel plane; Step 3: Calculate the image coordinates of the four hollow circles in the image, extract the circular area in the calibration plate through image processing, and obtain the coordinates of the center of the circle in the image, which is recorded as As the observed value of the coordinates of the circle center in the image; where h = {1, 2, 3, 4} means that h is equal to any number in the set, corresponding to the serial number of the hollow circle; Step 4: Calculate the three-dimensional coordinates of the center of the circle in the laser radar point cloud; For the lidar point cloud data at time i Calculate the three-dimensional coordinates of the centers of the four hollow circles on the calibration plate By segmenting the point cloud of the calibration plate area, the hollow circle area is detected, and the three-dimensional coordinates of the center of the circle are calculated; Step 5, calculate the two-dimensional projection coordinates of the three-dimensional coordinates of the center of the circle on the image; Use the initial value of the transformation matrix from the camera coordinate system to the lidar coordinate system The 3D coordinates of the center of the circle Transform to camera coordinate system And using the initial values ​​of the camera internal and external parameters obtained in the second step, calculate the two-dimensional projection coordinates of the center coordinates of the four hollow circles in the image as the predicted values ​​of the center coordinates in the image, recorded as: Where π(·):R 3 →R 2 It is the projection transformation from three-dimensional point coordinates to two-dimensional image coordinates; Step 6: Optimize and solve the target variable; The target variables to be obtained include: camera intrinsic parameters K C , and the external parameters of the camera coordinate system transformed to the lidar coordinate system Define the joint optimization objective function J: Each sub-item is described below; J C It represents the chessboard corner projection error loss in camera calibration and is defined as: M j is the 3D coordinate of the jth checkerboard corner point in the i-th frame. Note that π(·): R 3 →R 2 It is the projection transformation from three-dimensional point coordinates to two-dimensional image coordinates; Represents the predicted value u of the center coordinate of the hollow circle of the calibration plate l{1,2,3,4} and the observed value u of the center coordinates of the circle detected in the image h{1，2，3，4} The error loss is defined as: J RC It represents the error loss of the relative transformation of the checkerboard corner points in camera calibration and is defined as: in, Represents the transformation matrix from the i-th frame to the k-th frame obtained according to the camera external parameters, with a relative rotation R ik and relative translation t ik Transformation, the external parameters of frame i and frame k obtained during camera calibration and and Calculated, represents the relative rotation and translation transformation from the kth frame to the ith frame obtained according to the robot arm posture, which can be similarly calculated from the robot arm posture at each moment; Similarly, J RL The error loss of the relative transformation of the three-dimensional coordinates of the center of the hollow circle in the lidar point is defined as: in, is the three-dimensional coordinate of the center of the jth hollow circle on the calibration plate in the laser radar coordinate system, is the coordinate of the center point of the jth hollow circle projected onto image i in the laser radar coordinate system; Other λ in the objective function {1,2,3,4} is a coefficient that balances the weights of various error values.

2. A highly maneuverable multi-sensor joint automatic calibration method according to claim 1, characterized in that: If only the front sensor of the vehicle to be calibrated needs to be calibrated jointly, install a set of robotic arms; if both the front and rear sensors need to be calibrated jointly, install two sets of robotic arms, or wait until the front calibration is completed before inverting the vehicle onto the trailer.

3. A high-mobility multi-sensor joint automatic calibration method according to claim 1, characterized in that: The trailer chassis is modified and processed by adopting a standard length and width trailer as the chassis.

4. A highly maneuverable multi-sensor joint automatic calibration method according to claim 1, characterized in that: The bottom plate of the trailer chassis is made of high-strength steel, which reduces the overall weight of the trailer chassis while ensuring the load-bearing strength. The upper part of the bottom plate is processed by a machining center for fastening holes, and the communication harness and power supply harness are wired at the bottom of the chassis using industrial wire trough rails.

5. The high-mobility multi-sensor joint automatic calibration method according to claim 1 is characterized in that: The six-degree-of-freedom robotic arm has a working radius of 2m, which can meet the calibration requirements of all passenger vehicles.

6. A highly maneuverable multi-sensor joint automatic calibration method according to claim 1, characterized in that: The six-degree-of-freedom robotic arm and the fully automatic vehicle limiter adopt a modular design. The communication cables and power supply interfaces are all standard plugs. Combined with the reserved holes on the bottom plate, redundant installation at different calibration positions of the entire bottom plate can be achieved according to the size of the vehicle to be tested.

7. A high-mobility multi-sensor joint automatic calibration method according to claim 1, characterized in that: The trailer chassis can be matched with a tractor head vehicle with traction qualification, and has flexible and maneuverable characteristics to meet the actual needs of calibration.

8. The high-mobility multi-sensor joint automatic calibration method according to claim 1, characterized in that: The end of the six-degree-of-freedom robotic arm is hard-connected to the end high-precision calibration plate, so the posture of the end high-precision calibration plate can be obtained through the robotic arm posture, and the end high-precision calibration plate or six-degree-of-freedom robotic arm posture data can be regarded as the same data.

9. The high-mobility multi-sensor joint automatic calibration method according to claim 1, characterized in that: The camera's intrinsic and extrinsic parameters are obtained using the direct linear transformation method.

10. The high-mobility multi-sensor joint automatic calibration method according to claim 1, characterized in that: The automatic calibration method refers to the calibration between a conventional camera and a lidar. The calibration objects include the intrinsic parameters of the camera and the extrinsic parameters of the camera and the lidar.

Images