Calibration method, system, medium, product and device for multi-sensor data

Abstract

The present application belongs to the technical field of multi-sensor fusion and calibration, and provides a multi-sensor data calibration method, system, medium, product and equipment to solve the dependence of existing calibration methods on specific scenes and low calibration efficiency. The multi-sensor data calibration method comprises synchronously starting each sensor; obtaining compensation measurement information of feature points of a calibration board corresponding to each sensor through dynamic compensation; performing multi-level feature matching on the compensation measurement information of the feature points of the calibration board corresponding to each sensor and standard calibration board feature point information to synchronize the compensation measurement information of the feature points of the calibration board corresponding to each sensor; calculating the residual of each sensor; and according to the weighted sum of the sensor residual, jointly optimizing the extrinsic transformation matrix between the camera and the laser radar, the extrinsic transformation matrix between the other sensors and the calibration board, the inertial measurement unit bias and the time synchronization offset. It can meet the efficiency and reliability requirements of the multi-sensor fusion system in real application scenarios.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor fusion and calibration technology, and particularly relates to a method, system, medium, product and equipment for calibrating multi-sensor data. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Rich environmental information can be obtained through multi-sensor fusion technology (such as fusion of lidar, millimeter-wave radar, camera, etc.) to meet the perception needs in complex scenarios. In order to achieve effective fusion of data between multiple sensors, the calibration accuracy of the sensors is crucial, including the calibration of spatial position (extrinsic parameters) and time synchronization. Existing multi-sensor calibration methods usually rely on calibration boards or other known prior information in static scenarios, which have the following problems: (1) Dependence on specific scenarios and low calibration efficiency: Most calibration methods need to be carried out in specific laboratory environments, which are difficult to adapt to the complex environment of the real world, especially in dynamic or degraded scenarios, where the calibration accuracy is easily disturbed; Existing calibration methods usually rely on manual adjustment and a lot of offline calculations, which are difficult to meet the needs of dynamic calibration and real-time. (2) Lack of comprehensive consideration of sensor characteristics: Some calibration methods are designed only for a single type of sensor and do not fully consider the differences of multiple sensors (such as radar and camera) in terms of data acquisition frequency, field of view, etc., which leads to a decrease in fusion accuracy. Summary of the Invention

[0004] To address the technical problems mentioned above, this invention provides a method, system, medium, product, and device for calibrating multi-sensor data, which can meet the efficiency and reliability requirements of multi-sensor fusion systems in real-world application scenarios.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The first aspect of the present invention provides a method for calibrating multi-sensor data.

[0007] A method for calibrating multi-sensor data includes pre-setting a calibration plate in a designated area. The calibration plate is positioned opposite a chassis, which integrates at least a camera, a lidar, and an inertial measurement unit. The method further includes:

[0008] The chassis pose fed back by the inertial measurement unit is acquired and used as a trigger signal to synchronously start each sensor;

[0009] During the movement of the chassis, the dynamic compensation module is configured to compensate the feature points of the calibration board measured by each sensor by using a Kalman filtering algorithm to obtain compensation measurement information of the feature points of the calibration board corresponding to each sensor;

[0010] The multi-level feature matching module is configured to combine the spatial distribution and gradient direction information of the feature points of the calibration board, and perform multi-level feature matching between the compensation measurement information of the feature points of the calibration board corresponding to each sensor and the standard feature point information of the calibration board, so as to synchronize the compensation measurement information of the feature points of the calibration board corresponding to each sensor.

[0011] The sensor residual error calculation module is configured to calculate the residual error of each sensor according to the synchronized compensation measurement information of the feature points of the calibration board corresponding to each sensor and the standard feature point information of the calibration board.

[0012] The joint optimization module is configured to jointly optimize the extrinsic transformation matrix between the camera and the laser radar, the extrinsic transformation matrix between the other sensors and the calibration board, the bias of the inertial measurement unit, and the time synchronization offset according to the weighted sum of the sensor residual errors.

[0013] The second aspect of the present application provides a multi-sensor data calibration system.

[0014] A multi-sensor data calibration system, wherein a calibration board is pre-set in a designated site, the calibration board is arranged opposite to a chassis, and the chassis is integrated with sensors including at least a camera, a laser radar, and an inertial measurement unit; the multi-sensor data calibration system comprises:

[0015] A multi-sensor starting module configured to obtain the chassis pose fed back by the inertial measurement unit as a trigger signal to synchronously start each sensor.

[0016] A dynamic compensation module configured to compensate the feature points of the calibration board measured by each sensor by using a Kalman filtering algorithm during the movement of the chassis to obtain compensation measurement information of the feature points of the calibration board corresponding to each sensor.

[0017] A multi-level feature matching module configured to combine the spatial distribution and gradient direction information of the feature points of the calibration board, and perform multi-level feature matching between the compensation measurement information of the feature points of the calibration board corresponding to each sensor and the standard feature point information of the calibration board, so as to synchronize the compensation measurement information of the feature points of the calibration board corresponding to each sensor.

[0018] A sensor residual error calculation module configured to calculate the residual error of each sensor according to the synchronized compensation measurement information of the feature points of the calibration board corresponding to each sensor and the standard feature point information of the calibration board.

[0019] A joint optimization module configured to jointly optimize the extrinsic transformation matrix between the camera and the laser radar, the extrinsic transformation matrix between the other sensors and the calibration board, the bias of the inertial measurement unit, and the time synchronization offset according to the weighted sum of the sensor residual errors.

[0020] A third aspect of the present application provides a computer readable storage medium.

[0021] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method of calibrating multi-sensor data as described above.

[0022] A fourth aspect of the present application provides a computer program product.

[0023] A computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the method of calibrating multi-sensor data as described above.

[0024] A fifth aspect of the present application provides an electronic device.

[0025] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method of calibrating multi-sensor data as described above when executing the program.

[0026] Compared with the prior art, the present application has the following beneficial effects:

[0027] The present application uses a multi-sensor data acquisition device combining a fixed calibration board and a moving chassis and a calibration method thereof. By arranging a calibration board in a predetermined environment, sensor data is excited by various motion modes of the chassis, and high-precision sensor synchronization is realized relying on IMU data. In combination with dynamic compensation and multi-level feature matching, the extrinsic transformation matrix between the camera and the lidar, the extrinsic transformation matrix between other sensors and the calibration board, the inertial measurement unit bias, and the time synchronization offset are jointly optimized according to the weighted sum of sensor residuals. Not only does this solve the shortcomings of traditional calibration methods in dynamic scenes, but it also greatly improves the overall accuracy and robustness of the multi-sensor fusion system.

[0028] The advantages of the additional aspects of the present application will be partially given in the following description, partially become apparent from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0029] The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the present application and together with the description serve to explain the present application. Embodiments of the present application and its

[0030] Figure 1 is a flow chart of the method of calibrating multi-sensor data according to an embodiment of the present application;

[0031] Figure 2 is a schematic diagram of the structure of a calibration system for multi-sensor data according to an embodiment of the present application;

[0032] Figure 3 is a calibration result map of multi-sensor data of an embodiment of the present application. DETAILED DESCRIPTION

[0033] The present application is further described in connection with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed description is merely exemplary and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0035] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0036] Embodiment One

[0037] With the rapid development of autonomous driving, intelligent transportation, robot navigation and other fields, multi-sensor fusion technology has become one of the research hotspots and key technologies. For example, the unmanned data collection equipment is designed based on a four-wheel drive motion chassis, which has high mobility and the ability to adapt to complex terrain. A rotatable collection bracket is installed in the center of the chassis. The bracket is compact and flexible, can support the installation of multiple sensors, and can adjust the viewing angle and angle of the sensors as needed, so as to ensure comprehensive and accurate data collection.

[0038] Taking the unmanned data collection equipment as an example, the unmanned data collection equipment is equipped with multiple advanced sensors, including laser radar, 3D millimeter wave radar, 4D millimeter wave radar, binocular ZED2i camera and fisheye camera. These sensors work together to capture environmental information from low light to complex weather conditions, providing accurate spatial positioning, object recognition and obstacle avoidance capabilities. Laser radar is used to accurately measure the three-dimensional structure of the surrounding environment, while 3D millimeter wave radar and 4D millimeter wave radar effectively fill the sensing blind area at different distances and angles, especially suitable for harsh weather or low visibility scenarios. The binocular ZED2i camera provides high-resolution depth information for the device, and the fisheye camera expands the field of view, suitable for panoramic perception.

[0039] In addition, the unmanned data acquisition device also integrates an illuminance lumen sensor, which can monitor the environmental light changes in real time, ensuring the collection quality under various lighting conditions. To ensure real-time processing and storage of data, the device is equipped with a high-performance power management system, hardware storage, and signal trigger, supporting stable transmission and storage of high-frequency data, ensuring efficient and reliable data acquisition process. This system not only has high integration, but also can adapt to dynamic changes in the environment, making it an ideal platform for high-precision unmanned testing and research and development.

[0040] To realize the calibration of multi-sensor data, a plurality of sensors are first fixedly installed on the chassis, and a calibration board is pre-installed in a designated site. In a pre-selected ideal outdoor site, a calibration board is installed. The calibration board is a chessboard, and its geometric size and corner point position are known. Let the inner corner point of the chessboard be The three-dimensional coordinates in the calibration board coordinate system are: ; wherein N is the total number of corner points, and all corner points are distributed on the calibration board plane.

[0041] The sensor data acquisition device is fixedly installed on the moving chassis, and an inertial measurement unit (IMU) is integrated on the chassis. The acceleration and angular velocity data provided by the IMU and the position information provided by the GPS are used to record the motion state of the chassis in real time. Let the state of the chassis at continuous time t be represented by position and attitude:

[0042] ;

[0043] wherein ; is a three-dimensional real vector, belonging to the three-dimensional Euclidean space ; is a three-dimensional rotation matrix, used to describe the attitude (direction) of a rigid body in three-dimensional space.

[0044] Set multiple motion modes through the chassis control system, including:

[0045] Straight driving: the state satisfies ;represents the current position at time t; represents the initial position, v represents the motion speed, and t represents the time; Curved motion: set a fixed steering angle or use a predetermined path to make the device acquire calibration board image and point cloud data at multiple angles.

[0046]

[0047] In the chassis motion mode setting, the constraint conditions and the predetermined steering path of straight driving and curve driving introduce additional geometric constraints. This part of the constraint helps to optimize the uniformity of the distribution of the feature points of the calibration board. In the straight driving mode, the chassis and the sensor will maintain a constant view angle and direction, ensuring that the captured calibration board feature point distribution has linear continuity. This linear distribution helps the estimation of the initial extrinsic parameters in the subsequent calibration. In the curve motion mode, the change of the steering angle of the chassis will cause the sensor to observe the calibration board from multiple view angles, increasing the angular coverage of the feature points. This multi-angle distribution can significantly improve the accuracy of the extrinsic parameter estimation in the calibration process. At the same time, it reduces the cumulative effect of single point deviation. If the motion trajectory of the sensor is disordered, the spatial distribution of the feature points may be concentrated in a few directions, which can easily lead to an estimated result that is too sensitive to some point deviations. Through the pre-defined motion mode, the uniform distribution of the feature points can be ensured, and the influence of local errors on the overall calibration accuracy can be reduced. Finally, in the straight driving mode, the horizontal and vertical corners of the calibration board can be uniformly distributed in a specific area of the image or point cloud, and the curve motion mode provides depth variation information of the feature points, which helps the scale constraint in the extrinsic parameter optimization.

[0048] Figure 1 A flow chart of a multi-sensor data calibration method is given according to an embodiment of the present application. In combination Figure 1 , the multi-sensor data calibration method of the embodiment of the present application, a calibration board is pre-set in a set site, the calibration board is arranged opposite to a chassis, and the chassis is integrated with sensors including at least a camera, a lidar and an inertial measurement unit; the multi-sensor data calibration method comprises:

[0049] S101: acquiring a chassis pose fed back by an inertial measurement unit and taking the chassis pose as a trigger signal to synchronously start each sensor.

[0050] In the hardware synchronization mode, the pose signal fed back by the chassis in real time is taken as a trigger signal to synchronously start data acquisition of each sensor.

[0051] The pose real-time calculation method: in the acquisition process, the system calculates the relative spatial pose of each sensor in real time. By using an extended Kalman filter (EKF) or other multi-sensor fusion algorithm, the state estimation of each sensor is updated in combination with the motion data of the IMU. For the state of a certain sensor, the following update formula is used: ;

[0052] wherein, is a sampling time interval (according to the hardware setting of each sensor), F(·) is a motion model, is a process noise, represents the addition operation of the state on the Lie group. Wherein is the initial installation position of the hardware.

[0053] It's important to note here that Lie groups are a class of mathematical structures that possess both continuous geometric structures (such as rotations and translations) and group operation rules (allowing for "addition" or "combination"). They are commonly used to describe changes in the pose (position + orientation) of objects. Lie groups are used to model the state changes of rigid bodies, such as rotations and translations. In state estimation, state updates are represented by "addition" on the Lie group, rather than simple Euclidean vector addition.

[0054] S102: During the chassis movement, the calibration plate feature points measured by each sensor are dynamically compensated using the Kalman filter algorithm to obtain the calibration plate feature point compensation measurement information corresponding to each sensor.

[0055] Chassis motion trajectory recorded by IMU and feature point data collected by sensors Kalman filtering is used to dynamically compensate for the acquired data in each frame, resulting in compensated feature point data. ; ;

[0056] in, For each sensor, the corresponding number Compensation measurement information for feature points on the calibration plate; The first one measured by the sensor One calibration board feature point; K is the filter gain; For the first predicted by IMU trajectory Each calibration plate feature point.

[0057] This embodiment utilizes IMU data (accelerometer, angular velocity) and an extended Kalman filter (EKF) to dynamically compensate for the feature point data acquired by the sensor. Simultaneously, it dynamically corrects the sensor feature points: by adjusting the filter gain, the position between the predicted point and the measured point is dynamically adjusted to form compensated feature points. This real-time dynamic compensation improves the spatial consistency of feature points in dynamic calibration and multi-sensor fusion scenarios.

[0058] This embodiment expands the constraints of target optimization by introducing the error of radar reflection points, thereby improving the overall accuracy and stability of external parameter calibration, especially when the lighting conditions are poor or there is obstruction.

[0059] S103: Combining the spatial distribution and gradient direction information of the feature points on the calibration board, the compensation measurement information of the feature points on the calibration board corresponding to each sensor is matched with the feature point information of the standard calibration board at multiple levels to synchronize the compensation measurement information of the feature points on the calibration board corresponding to each sensor.

[0060] The objective function for multi-level feature matching is:

[0061] ;

[0062] in, This is the extrinsic transformation matrix from the sensor to the calibration plate coordinate system; The external parameter transformation matrix for the radar to reach the calibration plate coordinate system is preset; For the known calibration plate, the first The inner corner points, i.e., the feature points of the calibration board; For each sensor, the corresponding number Compensation measurement information for feature points on the calibration plate It is the first one measured by the preset radar. One reflection point; For the known calibration plate, the first The characteristic points of the calibration plate corresponding to each reflection point It is the preset weight of the radar error term. For the first Multi-layer feature weights for feature points of a calibration board; It is the total number of corner points; It is the preset number of reflection points detected by the radar.

[0063] ;

[0064] in, Multi-layer feature weights; and For the weighting coefficients, satisfying ; Represents point pairs Coarse matching weights between them; Represents point pairs The precise matching weights between them.

[0065] The formula for calculating multi-layer feature weights is: point pair coarse matching weights between and point-to-point Exact matching weights between The expression is: ;

[0066] ;

[0067] ;

[0068] ;

[0069] in, Geometric distance features; Features of gradient direction; Indicates the first geometric distance features of the calibration board feature points; geometric distance features of the calibration board feature points; geometric distance features of the calibration board feature points; geometric distance features of the calibration board feature points; gradient direction features of the calibration board feature points; gradient direction features of the calibration board feature points; gradient direction features of the calibration board feature points; is the position of the calibration board feature point; is the position of the center point of the calibration board; is the maximum distance of all feature points to the center point, for normalization; and are image gradients; and are smoothing parameters.

[0070] The embodiment respectively matches visual, laser radar and millimeter wave radar data, uses collected continuous motion excitation to ensure that feature points obtained in different postures have high robustness. Meanwhile, multi-level feature matching involves information such as spatial distribution and gradient direction of features. Multi-level feature weights are introduced to achieve this, considering the saliency of corner points and the confidence of matching pairs, which are calculated through descriptor similarity. The use of feature quality and confidence is enhanced, making the matching process more robust.

[0071] S104: According to the calibration board feature points corresponding to the synchronized sensors, compensate the measurement information and the standard calibration board feature point information, and calculate the sensor residual error;

[0072] S105: According to the weighted sum of the sensor residual error, jointly optimize the extrinsic transformation matrix between the camera and the laser radar, the extrinsic transformation matrix between the other sensors and the calibration board, the inertial measurement unit bias and the time synchronization offset.

[0073] It should be noted that the types, models and structures of other sensors can be selected according to actual conditions, which will not be described in detail here.

[0074] The objective function of the extrinsic transformation matrix between the camera and the laser radar and the extrinsic transformation matrix between the other sensors and the calibration board is:

[0075] ;

[0076] ;

[0077] wherein, is the camera residual error; is the laser radar residual error; is the extrinsic transformation matrix between the camera and the laser radar; This is the extrinsic parameter transformation matrix between the millimeter radar and the calibration board; For the known calibration plate, the first The inner corner points, i.e., the feature points of the calibration plate; To set the feature points of the standard calibration board; The first one measured by the camera One calibration plate feature point; The first one measured by lidar One calibration plate feature point; is the weighting factor for the corresponding residual; k is a positive integer greater than or equal to 3; This corresponds to the sensor residual; For time synchronization offset; For inertial measurement unit bias; These are the parameters to be jointly optimized; These are the parameters after joint optimization.

[0078] Camera: Using image processing algorithms, quickly extract the corner points of the chessboard grid to obtain the first... Each calibration plate feature point , which refers to the pixel coordinates in the camera image.

[0079] LiDAR: Utilizing point cloud segmentation algorithms to extract the edges or corners of the calibration board, the result is obtained from the LiDAR's measured data. Each calibration plate feature point That is, its three-dimensional measurement points.

[0080] Millimeter-wave radar: Detects reflected signals and extracts the reflection point. The millimeter-wave radar measures the first... Each calibration plate feature point That is, its corresponding three-dimensional spatial location.

[0081] The aforementioned joint optimization objective function aims to simultaneously optimize the extrinsic parameter transformation matrix between the camera and the lidar. The extrinsic parameter transformation matrix between the millimeter-wave radar and the calibration board This is achieved by minimizing the weighted geometric error between the observation points and reference points of each sensor, thus realizing joint calibration optimization among multiple sensors. The calibration results are as follows: Figure 3 (a) in Figure 3 As shown in (c) in the figure.

[0082] Example 2

[0083] This invention also provides a calibration system for multi-sensor data, wherein a calibration plate is pre-set in a designated area, the calibration plate is positioned opposite a chassis, and the chassis integrates at least these sensors, including a camera, a lidar, and an inertial measurement unit; the multi-sensor data calibration system includes:

[0084] a multi-sensor starting module 201, configured to acquire chassis pose fed back by an inertial measurement unit and as a trigger signal to start each sensor synchronously;

[0085] a dynamic compensation module 202, configured to, during chassis motion, perform dynamic compensation on the measured feature points of the calibration board by each sensor through a Kalman filtering algorithm to obtain compensation measurement information of the feature points of the calibration board corresponding to each sensor;

[0086] a multi-level feature matching module 203, configured to combine spatial distribution and gradient direction information of the feature points of the calibration board, and perform multi-level feature matching between the compensation measurement information of the feature points of the calibration board corresponding to each sensor and standard feature point information of the calibration board, to synchronize the compensation measurement information of the feature points of the calibration board corresponding to each sensor;

[0087] a sensor residual calculation module 204, configured to calculate sensor residuals according to the synchronized compensation measurement information of the feature points of the calibration board corresponding to each sensor and the standard feature point information of the calibration board;

[0088] a joint optimization module 205, configured to jointly optimize an extrinsic transformation matrix between the camera and the lidar, an extrinsic transformation matrix between other sensors and the calibration board, an inertial measurement unit bias and a time synchronization offset according to a weighted sum of the sensor residuals.

[0089] It should be noted that each module of the embodiment of the present application corresponds to each step in the above embodiment one by one, and the specific implementation process is the same, which will not be described in detail here.

[0090] Embodiment Three

[0091] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps in the multi-sensor data calibration method.

[0092] Embodiment Four

[0093] A computer program product includes a computer program / instruction, which is executed by a processor to realize the steps in the multi-sensor data calibration method.

[0094] Embodiment Five

[0095] The embodiment provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the steps in the multi-sensor data calibration method.

[0096] The electronic device according to the present embodiment includes a central processing unit (CPU) which can perform various appropriate actions and processes in accordance with a program stored in a read only memory (ROM) or a program loaded from a storage section into a random access memory (RAM). In the RAM, various programs and data required for system operation are also stored. The central processing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0097] The following components are connected to the I / O interface: an input section including a keyboard, a mouse, etc.; an output section including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section including a hard disk, etc.; and a communication section including a network interface card such as a local area network (LAN) card, a modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as necessary. A removable media such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive as necessary, so that a computer program read out from the removable media is installed into the storage section as necessary.

[0098] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with the embodiments of the present application. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section, and / or installed from a removable medium. When the computer program is executed by the central processing unit, various functions defined in the apparatus of the present application are executed.

[0099] The present application is described with reference to the flow charts and / or block diagrams of the methods, apparatus (system) and computer program products of embodiments of the present application. It is understood that each flow and / or block in the flow charts and / or block diagrams, and combinations of flows and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow charts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 an apparatus with one or more specific functions specified in one or more flows and / or blocks.

[0100] The above merely describes the preferred embodiments of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for calibrating multi-sensor data, characterized in that, In a set site, a preset calibration board is set, the calibration board is opposite to the chassis, and the chassis is integrated with the sensors including a camera, a laser radar and an inertial measurement unit; the calibration method of the multi-sensor data comprises: Obtaining the chassis pose fed back by the inertial measurement unit as a trigger signal to synchronously start the sensors; In the chassis movement process, the feature points of the calibration board measured by the sensors are dynamically compensated by a Kalman filtering algorithm to obtain the compensation measurement information of the feature points of the calibration board corresponding to the sensors; Combining the spatial distribution and gradient direction information of the feature points of the calibration board, the compensation measurement information of the feature points of the calibration board corresponding to the sensors is matched with the standard feature point information of the calibration board in multiple levels to synchronize the compensation measurement information of the feature points of the calibration board corresponding to the sensors; According to the synchronized compensation measurement information of the feature points of the calibration board corresponding to the sensors and the standard feature point information of the calibration board, the sensor residual errors are calculated; According to the weighted sum of the sensor residual errors, the extrinsic transformation matrix between the camera and the laser radar, the extrinsic transformation matrix between the other sensors and the calibration board, the inertial measurement unit bias and the time synchronization offset are jointly optimized; The process of dynamically compensating the feature points of the calibration board measured by the sensors by the Kalman filtering algorithm is as follows: ； in, For each sensor, the corresponding number Compensation measurement information for feature points on the calibration plate; The first one measured by the sensor One calibration board feature point; K is the filter gain; For the first predicted by IMU trajectory One calibration plate feature point; The objective function of the multi-level feature matching is as follows: ； wherein, is an extrinsic transformation matrix of the sensor to the calibration board coordinate system; is an extrinsic transformation matrix of the preset radar to the calibration board coordinate system; is the i-th inner corner point on the known calibration board, i.e., the calibration board feature point; is the i-th inner corner point on the known calibration board, i.e., the calibration board feature point; is the i-th calibration board feature point compensation measurement information corresponding to each sensor, is the i-th calibration board feature point compensation measurement information corresponding to each sensor, is the i-th reflection point measured by the preset radar; is the i-th reflection point measured by the preset radar; is the i-th reflection point measured by the preset radar; is the i-th reflection point measured by the preset radar; is the weight of the preset radar error term, is the multi-layer feature weight of the i-th calibration board feature point; is the multi-layer feature weight of the i-th calibration board feature point; is the total number of corner points; is the number of reflection points measured by the preset radar; The calculation formula of the multi-layer feature weight is: ; wherein, is a multi-layer feature weight; and is a weight coefficient satisfying ; denotes a coarse matching weight between point pairs ; denotes a fine matching weight between point pairs ; The formula for calculating multi-layer feature weights is: point pair coarse matching weights between and point-to-point Exact matching weights between The expression is: ; ； ； ； wherein, is a geometric distance feature; is a gradient direction feature; denotes the geometric distance feature of the -th calibration plate feature point; denotes the geometric distance feature of the -th calibration plate feature point; denotes the gradient direction feature of the -th calibration plate feature point; denotes the gradient direction feature of the -th calibration plate feature point; is the position of a calibration plate feature point; is the center point position of a calibration plate; is the maximum distance of all feature points to the center point for normalization; and is the image gradient; and are smoothing parameters; The objective function of jointly optimizing the extrinsic transformation matrix between the camera and the laser radar and the extrinsic transformation matrix between the other sensors and the calibration board is as follows: ； ； wherein, is a camera residual; is a lidar residual; is an extrinsic transformation matrix between camera and lidar; is an extrinsic transformation matrix between millimeter-wave radar and calibration board; is the kth inner corner point on the known calibration board, i.e., calibration board feature point; is the kth inner corner point on the known calibration board, i.e., calibration board feature point; is the kth inner corner point on the known calibration board, i.e., calibration board feature point; is the kth inner corner point on the known calibration board, i.e., calibration board feature point; is the kth inner corner point on the known calibration board, i.e., calibration board feature point; is the kth inner corner point on the known calibration board, i.e., calibration board feature point; is the kth inner corner point on the known calibration board, i.e., calibration board feature point; is a weighting factor of the corresponding residual; k is a positive integer greater than or equal to 3; is a corresponding sensor residual; is a time synchronization offset; is an inertial measurement unit bias; is a parameter to be jointly optimized; is a parameter after joint optimization.

2. A system for calibrating multi-sensor data, the system comprising: Based on the calibration method of the multi-sensor data in claim 1, in a set site, a preset calibration board is set, the calibration board is opposite to the chassis, and the chassis is integrated with the sensors including a camera, a laser radar and an inertial measurement unit; the calibration system of the multi-sensor data comprises: A multi-sensor starting module for obtaining the chassis pose fed back by the inertial measurement unit as a trigger signal to synchronously start the sensors; A dynamic compensation module for dynamically compensating the feature points of the calibration board measured by the sensors by a Kalman filtering algorithm in the chassis movement process to obtain the compensation measurement information of the feature points of the calibration board corresponding to the sensors; A multi-level feature matching module for combining the spatial distribution and gradient direction information of the feature points of the calibration board, matching the compensation measurement information of the feature points of the calibration board corresponding to the sensors with the standard feature point information of the calibration board in multiple levels to synchronize the compensation measurement information of the feature points of the calibration board corresponding to the sensors; A sensor residual error calculation module for calculating the sensor residual errors according to the synchronized compensation measurement information of the feature points of the calibration board corresponding to the sensors and the standard feature point information of the calibration board; A joint optimization module for jointly optimizing the extrinsic transformation matrix between the camera and the laser radar, the extrinsic transformation matrix between the other sensors and the calibration board, the inertial measurement unit bias and the time synchronization offset according to the weighted sum of the sensor residual errors.

3. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the steps in the calibration method of the multi-sensor data in claim 1.

4. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps in the calibration method of the multi-sensor data in claim 1.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the program, implements the steps in the method for calibrating multi-sensor data as claimed in claim 1.

Images