Multi-sensor joint calibration method, device, storage medium and program product

Abstract

Embodiments of the present application provide a multi-sensor joint calibration method, device, storage medium and program product. The method comprises, for each sensor, calibrating the sensor according to a calibration object in a calibration field, obtaining a first pose conversion relationship between a sensor coordinate system of the sensor and a world coordinate system of the calibration field, calculating a second pose conversion relationship between the sensor coordinate systems of each sensor according to the first pose conversion relationship corresponding to each sensor, and determining a calibration result according to a plurality of second pose conversion relationships. Embodiments of the present application calibrate each sensor respectively according to a calibration object in a calibration field, obtain a conversion relationship between a sensor coordinate of each sensor and a world coordinate system of the calibration field, and on this basis, quickly obtain a conversion relationship between each of a plurality of sensors, complete vehicle calibration, so that the calibration period of vehicle calibration can be greatly reduced and calibration efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a multi-sensor joint calibration method, device, storage medium, and program product. Background Technology

[0002] Autonomous driving technology relies on the fusion of multiple sensors. Autonomous vehicles, equipped with cameras and LiDAR at different angles, receive richer environmental perception information. Sensor fusion enables the output of more robust perception and localization results. The accuracy of multi-sensor joint calibration is a crucial prerequisite for effective sensor fusion.

[0003] In existing technologies, a pairwise calibration method is typically used, which allows two sensors to simultaneously collect information about the calibration object within the common field of view. Then, based on the collected information, extrinsic parameters are calculated to obtain the transformation relationship between the coordinate systems of the two sensors, thus achieving calibration between the two sensors.

[0004] However, in the process of realizing this application, the inventors discovered that the prior art has at least the following problems: when there are multiple sensors in a vehicle, the method of calibrating in pairs makes the calibration cycle of the whole vehicle too long and the calibration efficiency low. Summary of the Invention

[0005] This application provides a multi-sensor joint calibration method, device, storage medium, and program product to improve calibration efficiency.

[0006] In a first aspect, embodiments of this application provide a multi-sensor joint calibration method, applied to a vehicle having multiple sensors, wherein the multiple sensors are rigidly connected, including:

[0007] For each sensor, the sensor is calibrated according to the calibration object in the calibration field to obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field;

[0008] Based on the first pose transformation relationship corresponding to each sensor, calculate the second pose transformation relationship between each pair of sensor coordinate systems;

[0009] The calibration result is determined based on multiple second pose transformation relationships.

[0010] In one possible design, calibrating the sensor based on a calibration object within the calibration field to obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field includes:

[0011] Based on the two-dimensional calibration board in the calibration field, determine the original calibration data of the camera;

[0012] The camera is calibrated based on its original calibration data to obtain the first pose transformation relationship between the camera coordinate system and the world coordinate system of the calibration field.

[0013] In one possible design, determining the sensor's raw calibration data based on a two-dimensional calibration plate within the calibration field includes:

[0014] The camera captures images of multiple two-dimensional calibration boards to obtain target images;

[0015] Obtain the world coordinates of multiple first feature points in the world coordinate system and the pixel coordinates of multiple first corresponding points in the pixel coordinate system; the multiple first feature points are feature points on multiple two-dimensional calibration plates; the multiple first corresponding points are corresponding points of the multiple first feature points in the target image respectively.

[0016] The world coordinates of multiple first feature points and the pixel coordinates of the corresponding first points of the same name are determined as the original calibration data of the camera.

[0017] In one possible design, calibrating the camera based on its original calibration data to obtain the first pose transformation relationship between the camera's camera coordinate system and the world coordinate system of the calibration field includes:

[0018] Determine the initial values ​​of the camera's intrinsic parameter matrix;

[0019] Based on the initial value of the intrinsic parameter matrix and the original calibration data of the camera, the intrinsic parameter matrix is ​​optimized to obtain the first pose transformation relationship between the camera coordinate system of the camera and the world coordinate system of the calibration field.

[0020] In one possible design, determining the initial values ​​of the camera's intrinsic parameter matrix includes:

[0021] Rotate the vehicle to different angles;

[0022] For each angle, an image to be processed is acquired by the camera; the image to be processed includes at least one two-dimensional calibration plate;

[0023] Obtain the world coordinates of multiple second feature points in the calibration plate coordinate system of the calibration plate and the pixel coordinates of multiple second corresponding points in the pixel coordinate system of the image to be processed; the multiple feature points are feature points on the at least one two-dimensional calibration plate; the multiple second corresponding points are corresponding points of the multiple second feature points in the image to be processed respectively;

[0024] Based on the world coordinates of multiple second feature points and the pixel coordinates of multiple second corresponding points, the initial value of the camera's intrinsic parameter matrix is ​​determined using the Zhang Zhengyou calibration method.

[0025] In one possible design, calibrating the sensor based on a calibration object within the calibration field to obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field includes:

[0026] Based on the three-dimensional calibration objects in the calibration field, determine the original calibration data of the lidar;

[0027] The lidar is calibrated based on its original calibration data to obtain the first pose transformation relationship between the lidar coordinate system and the world coordinate system of the calibration field.

[0028] In one possible design, determining the raw calibration data of the lidar based on the three-dimensional calibration object within the calibration field includes:

[0029] Obtain the first point cloud of the stereo calibration object in the world coordinate system;

[0030] The second point cloud of the three-dimensional calibration object in the lidar coordinate system is obtained through the lidar;

[0031] Obtain the world coordinate system of multiple third corresponding points and the lidar coordinates of multiple fourth corresponding points; the multiple third corresponding points are the corresponding points of multiple third feature points in the target plane of the three-dimensional calibration object in the first point cloud; the multiple fourth corresponding points are the corresponding points of multiple third feature points in the second point cloud.

[0032] The world coordinates of the multiple third corresponding points and the lidar coordinates of the multiple fourth corresponding points are determined as the original calibration data for the lidar.

[0033] In one possible design, obtaining the world coordinate system of multiple third corresponding points and the lidar coordinates of multiple fourth corresponding points includes:

[0034] Based on the multiple vertices of the stereo calibration object, a first target point cloud corresponding to the stereo calibration object is obtained by filtering from the first point cloud, and a second target point cloud corresponding to the stereo calibration object is obtained by filtering from the second point cloud;

[0035] Based on the first target point cloud and the second target point cloud, determine the third and fourth corresponding points of the multiple third feature points in the target plane of the stereo positioning;

[0036] Obtain the world coordinates of multiple third-named points and the lidar coordinates of multiple fourth-named points.

[0037] In one possible design, calibrating the lidar based on its original calibration data to obtain the first pose transformation relationship between the lidar coordinate system and the world coordinate system of the calibration field includes:

[0038] Obtain the normal vector of the target plane of the three-dimensional calibration object;

[0039] Based on the normal vector and the original calibration data of the lidar, the lidar is calibrated to obtain the first pose transformation relationship between the lidar coordinate system and the world coordinate system of the calibration field.

[0040] In one possible design, the method further includes:

[0041] After the vehicle is parked in the calibration field, the coordinates of the rear axle center of the vehicle body in the world coordinate system of the calibration field are determined.

[0042] In one possible design, determining the coordinates of the rear axle center of the vehicle body in the world coordinate system of the calibration field includes:

[0043] After the vehicle is parked in the calibration field, a laser point cloud of an object is acquired, the object being placed on the vehicle's wheels;

[0044] The laser point cloud of the marker is registered with the point cloud of the calibration field to obtain the world coordinates of the marker in the world coordinate system of the calibration field.

[0045] Based on the world coordinates of the marker in the calibration field's world coordinate system, and the positional relationship between the marker and the rear axle center of the vehicle, the coordinates of the rear axle center of the vehicle in the calibration field's world coordinate system are determined.

[0046] In one possible design, determining the calibration result based on multiple second pose transformation relationships includes:

[0047] Based on the second pose transformation relationship corresponding to multiple sensors, the extrinsic parameter closed-loop difference of the extrinsic parameter conduction closed loop is calculated; the extrinsic parameter conduction closed loop is composed of multiple sensors;

[0048] Determine whether the extrinsic closed-loop difference is less than or equal to a preset threshold;

[0049] If so, the calibration result is determined based on multiple second pose transformation relationships.

[0050] In one possible design, calculating the extrinsic closed-loop difference of the extrinsic conduction closed loop based on the second pose transformation relationship corresponding to multiple sensors includes:

[0051] Obtain the adjacent positional relationships of each sensor in the external parameter conduction closed loop;

[0052] Obtain the second pose transformation relationship between adjacent sensors in the extrinsic parameter conduction closed loop;

[0053] Based on the adjacent positional relationships, multiple second pose transformation relationships are multiplied sequentially, and the resulting product is determined as the extrinsic closed-loop difference.

[0054] In one possible design, after determining whether the extrinsic closed-loop difference is less than or equal to a preset threshold, the method further includes:

[0055] If not, then the second pose transformation relationship corresponding to multiple sensors is jointly optimized, and the calibration result is determined based on the multiple jointly optimized second pose transformation relationships.

[0056] In one possible design, the joint optimization of the second pose transformation relationship corresponding to multiple sensors includes:

[0057] Acquire the calibration raw data of multiple sensors; the calibration raw data of each sensor is the raw data collected when determining the first pose transformation relationship of the sensor;

[0058] For each sensor, the first pose transformation relationship of the sensor is iteratively optimized based on the corresponding calibration raw data, so that the extrinsic parameter closed-loop difference is less than or equal to the preset threshold.

[0059] Secondly, embodiments of this application provide a multi-sensor joint calibration device, comprising:

[0060] The calibration module is used to calibrate each sensor according to the calibration object in the calibration field, and obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field.

[0061] The calculation module is used to calculate the second pose transformation relationship between each pair of the sensing coordinate systems of each sensor based on the first pose transformation relationship corresponding to each sensor.

[0062] The determination module is used to determine the calibration result based on multiple second pose transformation relationships.

[0063] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory;

[0064] The memory stores computer-executed instructions;

[0065] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method described in the first aspect above and various possible designs of the first aspect.

[0066] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the first aspect and various possible designs of the first aspect.

[0067] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect and various possible designs of the first aspect.

[0068] This embodiment provides a multi-sensor joint calibration method, device, storage medium, and program product. The method includes, for each sensor, calibrating it using calibration objects within a calibration field to obtain a first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field; calculating second pose transformation relationships between each sensor's sensing coordinate system based on these first pose transformation relationships; and determining the calibration result based on multiple second pose transformation relationships. The multi-sensor joint calibration method provided in this application calibrates each sensor separately using calibration objects within a calibration field, obtaining the transformation relationship between each sensor's coordinate system and the world coordinate system of the calibration field. Based on this, the transformation relationships between multiple sensors can be quickly obtained, completing the whole-vehicle calibration. This significantly reduces the calibration cycle of the whole-vehicle calibration and improves calibration efficiency. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 Application scenario diagrams provided for embodiments of this application;

[0071] Figure 2 A schematic flowchart illustrating the multi-sensor joint calibration method provided in this application embodiment;

[0072] Figure 3 This is a schematic diagram illustrating the correspondence between coordinate systems in camera calibration provided in an embodiment of this application;

[0073] Figure 4 A schematic diagram of a scenario for vehicle body center axis calibration provided in an embodiment of this application;

[0074] Figure 5 This is a schematic diagram of the structure of the multi-sensor joint calibration device provided in the embodiments of this application;

[0075] Figure 6 This is a hardware structure block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] Autonomous driving technology relies on the fusion of multiple sensors, including multiple cameras, multiple LiDARs, and inertial navigation. Cameras provide semantic information about the planar scene, LiDARs generate 3D structures around the sensors, and inertial measurement units record information such as the instantaneous velocity of the vehicle. Currently, redundancy among multiple sensors has become a mainstream approach for autonomous driving. Autonomous vehicles often carry cameras and LiDARs at different angles to provide richer environmental perception information. Information fusion between sensors can output more robust perception and localization results. All of the above technical prerequisites depend on the accurate internal and external parameters of the sensors; therefore, multi-sensor calibration is a crucial prerequisite for adopting autonomous driving technology.

[0078] In related technologies, static offline calibration between pairs of sensors can be employed. For example, when calibrating the parameters between a LiDAR and a camera, LiDAR point clouds and camera images can be simultaneously acquired within the shared field of view of both the LiDAR and camera. Calibration objects can be extracted from the LiDAR point clouds and camera images, and extrinsic parameters can be calculated. The camera's intrinsic parameters can be solved using the Zhang Zhengyou checkerboard method from multiple perspectives. When calibrating the extrinsic parameters between two LiDARs, features within the shared field of view of both LiDARs can be extracted, and extrinsic parameters can be calculated based on these extracted features. However, the above-mentioned method of calibrating sensors pairwise not only requires a significant amount of human assistance, but also results in an excessively long calibration cycle for all sensors in autonomous vehicles equipped with multiple sensors.

[0079] To address the aforementioned technical problems, the inventors of this application have discovered through research that a dedicated calibration field can be set up. Each sensor is calibrated separately using calibration objects within the calibration field, obtaining the transformation relationship between the sensor coordinates of each sensor and the world coordinate system of the calibration field. Based on this, the pairwise transformation relationships between multiple sensors can be quickly obtained, completing the vehicle calibration and significantly reducing the calibration cycle and improving calibration efficiency. Therefore, this application provides a method for multi-sensor joint calibration, which can improve the calibration efficiency of multi-sensor joint calibration.

[0080] Figure 1 This is an application scenario diagram provided for embodiments of this application. For example... Figure 1 As shown, vehicle 101 is placed in calibration field 102. Multiple two-dimensional calibration plates 103 and multiple three-dimensional calibration objects 104 are provided in calibration field 102. Based on the distribution of the sensors, the two-dimensional calibration plates 103 and the three-dimensional calibration objects 104 can be arranged in a reasonable manner, so that each sensor can obtain the relevant raw data when determining the transformation relationship between its own sensing coordinate system and the world coordinate system of calibration field 102.

[0081] In the specific implementation process, the vehicle 101 is parked in the calibration field 102. Each sensor of the vehicle 101 (such as a camera and a lidar) acquires its own relevant raw data. Based on the raw data, the first pose transformation relationship between the sensing coordinate system of its own sensor and the world coordinate system of the calibration field 102 is determined. Then, based on the first pose transformation relationship corresponding to each sensor, the second pose transformation relationship between each pair of sensing coordinate systems of each sensor is calculated. Thus, the calibration result is determined based on multiple second pose transformation relationships, and the whole vehicle calibration is completed quickly.

[0082] The multi-sensor joint calibration method provided in this application embodiment calibrates each sensor separately using calibration objects within the calibration field 102, obtaining the transformation relationship between the sensor coordinates of each sensor and the world coordinate system of the calibration field 102. Based on this, the transformation relationship between multiple sensors can be quickly obtained pairwise, completing the whole vehicle calibration, which greatly reduces the calibration cycle of the whole vehicle calibration and improves calibration efficiency.

[0083] It should be noted that, Figure 1 The schematic diagram shown is merely an example. The multi-sensor joint calibration method and scenario described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0084] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0085] Figure 2 This is a schematic flowchart illustrating the multi-sensor joint calibration method provided in an embodiment of this application. Figure 2 As shown, this method is applied to a vehicle with multiple sensors, which are rigidly connected together, including:

[0086] 201. For each sensor, calibrate the sensor according to the calibration object in the calibration field to obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field.

[0087] The execution subject in this embodiment can be a data processing device such as a computer, tablet computer, or in-vehicle infotainment system.

[0088] Specifically, after the vehicle is parked in the calibration field, each sensor installed on the vehicle performs a first pose transformation between its own sensor coordinate system and the world coordinate system of the calibration field based on the corresponding calibration object, thereby obtaining the first pose transformation relationship corresponding to each sensor.

[0089] In this embodiment, to improve calibration efficiency, the world coordinates of the feature points of each calibration object in the calibration field in the world coordinate system of the calibration field can be predetermined. For example, the world coordinates of each corner point of the two-dimensional calibration plate in the world coordinate system of the calibration field can be predetermined, and the world coordinates of the normal vectors of each face of the three-dimensional calibration plate in the world coordinate system of the calibration field can be predetermined.

[0090] In this embodiment, the sensors installed on the vehicle can be cameras, lidar, or other sensors that require static calibration.

[0091] For example, assuming the vehicle is equipped with 6 cameras and 1 LiDAR, the 6 cameras and 1 LiDAR are calibrated simultaneously to obtain their respective first pose transformation relationships.

[0092] The following example illustrates the process of determining the first pose transformation relationship between the camera and the lidar.

[0093] Determining the first pose transformation relationship of the camera:

[0094] To clearly explain the content of camera calibration, combined with Figure 3 Examples are provided to illustrate the calibration contents of the camera's intrinsic and extrinsic parameters. Figure 3 This is a schematic diagram illustrating the correspondence between coordinate systems in camera calibration provided in an embodiment of this application. For example... Figure 3As shown, during camera calibration, images of the two-dimensional calibration plate 103 within the calibration field are acquired using the camera. This process involves the pixel coordinate system of the acquired images, the camera coordinate system, and the world coordinate system of the calibration field. The intrinsic parameter calibration of the camera is the calibration of the transformation relationship between the pixel coordinate system of the images and the camera coordinate system. The extrinsic parameter calibration of the camera is the calibration of the transformation relationship between the camera coordinate system and the world coordinate system of the calibration field.

[0095] In some embodiments, calibrating the sensor based on a calibration object in the calibration field to obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field may include: determining the camera's original calibration data based on a two-dimensional calibration plate in the calibration field; calibrating the camera based on the camera's original calibration data to obtain the first pose transformation relationship between the camera's camera coordinate system and the world coordinate system of the calibration field.

[0096] In this embodiment, in order to calibrate multiple cameras simultaneously, multiple two-dimensional calibration boards can be set so that multiple two-dimensional calibration boards are set within the field of view of each camera. In order to distinguish them, the two-dimensional calibration boards can be coded, for example, ApirGrid codes can be set.

[0097] Two-dimensional calibration plates can be of the same size, or different scaling ratios can be set on different two-dimensional calibration plates to calibrate camera intrinsic parameters.

[0098] Optionally, determining the original calibration data of the sensor based on the two-dimensional calibration plates in the calibration field may include: taking pictures of multiple two-dimensional calibration plates with the camera to obtain a target image; obtaining the world coordinates of multiple first feature points in the world coordinate system and the pixel coordinates of multiple first corresponding points in the pixel coordinate system; the multiple first feature points are feature points on multiple two-dimensional calibration plates; the multiple first corresponding points are corresponding points of the multiple first feature points in the target image; and determining the world coordinates of the multiple first feature points and the pixel coordinates of the corresponding first corresponding points as the original calibration data of the camera.

[0099] In this embodiment, when calibrating the first pose transformation relationship between the camera coordinate system and the world coordinate system of the calibration field, that is, when calibrating the extrinsic parameters of the camera relative to the calibration field, the intrinsic parameter matrix can be optimized simultaneously based on its initial value. Alternatively, the initial value of the intrinsic parameter matrix can be directly determined as its final value. In other words, during iterative optimization, only the extrinsic parameters of the camera are iteratively optimized, while the intrinsic parameter matrix is ​​fixed. The specific method described above can be selected according to the actual situation, and this embodiment does not limit it.

[0100] For example, the camera's extrinsic parameters (or intrinsic and extrinsic parameters) can be solved by iterative optimization based on the following cost function.

[0101]

[0102] Among them, R w2c and t w2c These are the rotation and translation matrices between the world coordinate system of the calibration board and the camera coordinate system of the camera, respectively; they also represent the first pose transformation relationship of the camera, i.e., the camera's extrinsic parameters. The world coordinates of the feature points of the two-dimensional calibration plate in the world coordinate system of the calibration plate. is the pixel coordinates of the corresponding points in the target image for the feature points of the 2D calibration board, and m is the number of feature points in the target image.

[0103] Based on the aforementioned cost function, optimization can be achieved using the least squares method. For example, the LM (Levenberg-Marquardt) gradient descent algorithm and the 3D-2D PnP algorithm can be employed.

[0104] Optionally, calibrating the camera based on the original calibration data to obtain the first pose transformation relationship between the camera coordinate system and the world coordinate system of the calibration field may include: determining the initial value of the camera's intrinsic parameter matrix; optimizing the intrinsic parameter matrix based on the initial value of the intrinsic parameter matrix and the original calibration data of the camera to obtain the first pose transformation relationship between the camera coordinate system and the world coordinate system of the calibration field.

[0105] Specifically, the initial values ​​of the camera's intrinsic parameter matrix can be determined first using the raw data to determine the intrinsic parameter matrix, and then the initial values ​​of the camera's intrinsic parameter matrix can be determined based on the raw data using a preset algorithm. Optionally, the preset algorithm can be the Zhang Zhengyou calibration algorithm.

[0106] There are several ways to obtain the raw data used to determine the camera intrinsic parameter matrix.

[0107] In one possible approach, a robotic arm can be used to place the 2D calibration board within the camera's field of view, displaying multiple poses, such as 30 poses. The camera then captures images of each pose, thereby obtaining images of the 2D calibration board from multiple angles.

[0108] In another possible implementation, determining the initial value of the camera's intrinsic parameter matrix may include: rotating the vehicle to different angles; acquiring an image to be processed through the camera for each angle; the image to be processed includes at least one two-dimensional calibration plate; obtaining the world coordinates of multiple second feature points in the calibration plate coordinate system and the pixel coordinates of multiple second corresponding points in the pixel coordinate system of the image to be processed; the multiple feature points are feature points on the at least one two-dimensional calibration plate; the multiple second corresponding points are corresponding points of the multiple second feature points in the image to be processed; and determining the initial value of the camera's intrinsic parameter matrix based on the Zhang Zhengyou calibration method, according to the world coordinates of the multiple second feature points and the pixel coordinates of the multiple second corresponding points.

[0109] Specifically, a turntable can be set up in the calibration field, and the vehicle can be parked on the turntable. By driving the turntable to rotate, the vehicle can be rotated to multiple angles. For different angles, the camera acquires images of the two-dimensional calibration plate, thus obtaining images of the two-dimensional calibration plate set at different positions in the calibration field. Then, the coordinates of each corner point of the two-dimensional calibration plate in the calibration plate coordinate system, as well as the pixel coordinates of the corresponding points in the image, are obtained as the raw data for determining the camera intrinsic parameter matrix.

[0110] Determining the first pose transformation relationship of the lidar:

[0111] In some embodiments, calibrating the sensor based on calibration objects in the calibration field to obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field may include: determining the original calibration data of the lidar based on the three-dimensional calibration objects in the calibration field; calibrating the lidar based on the original calibration data of the lidar to obtain the first pose transformation relationship between the lidar coordinate system of the lidar and the world coordinate system of the calibration field.

[0112] Optionally, determining the original calibration data of the lidar based on the three-dimensional calibration object in the calibration field may include: acquiring a first point cloud of the three-dimensional calibration object in the world coordinate system; obtaining a second point cloud of the three-dimensional calibration object in the lidar coordinate system through the lidar; acquiring the world coordinate system coordinates of multiple third corresponding points and the lidar coordinates of multiple fourth corresponding points; the multiple third corresponding points are the corresponding points of multiple third feature points in the target plane of the three-dimensional calibration object in the first point cloud; the multiple fourth corresponding points are the corresponding points of multiple third feature points in the second point cloud; and determining the world coordinates of the multiple third corresponding points and the lidar coordinates of the multiple fourth corresponding points as the original calibration data of the lidar.

[0113] In this embodiment, there are multiple ways to obtain the first point cloud of the stereo calibration object in the world coordinate system. In one feasible method, the first point cloud can be determined by a third-party surveying and mapping institution using professional surveying and mapping equipment so that the accuracy of the first point cloud meets the requirements.

[0114] Furthermore, there are multiple ways to obtain the world coordinate system of multiple third corresponding points and the lidar coordinates of multiple fourth corresponding points. In one possible approach, the world coordinate system of the third corresponding points can be directly obtained from the first point cloud, and the lidar coordinate system of the fourth corresponding points can be directly obtained from the second point cloud. In another possible approach, to improve accuracy, based on multiple vertices of the 3D calibration object, a first target point cloud corresponding to the 3D calibration object can be obtained from the first point cloud, and a second target point cloud corresponding to the 3D calibration object can be obtained from the second point cloud. Based on the first target point cloud and the second target point cloud, the third and fourth corresponding points corresponding to the multiple third feature points in the target plane of the 3D calibration object are determined respectively. The world coordinate system of the multiple third corresponding points and the lidar coordinates of the multiple fourth corresponding points are then obtained. That is, by first deleting redundant point clouds to obtain a more concise point cloud corresponding to the 3D calibration object, and then obtaining the coordinates of each corresponding point, the accuracy can be improved.

[0115] Optionally, calibrating the lidar based on the original calibration data to obtain the first pose transformation relationship between the lidar coordinate system and the world coordinate system of the calibration field may include: obtaining the normal vector of the target plane of the three-dimensional calibration object; calibrating the lidar based on the normal vector and the original calibration data to obtain the first pose transformation relationship between the lidar coordinate system and the world coordinate system of the calibration field.

[0116] For example, the following cost function can be used to calibrate the first pose transformation relationship of the lidar.

[0117]

[0118] Among them, R l2w and t l2w P represents the rotation and translation matrices between the world coordinate system of the calibration board and the lidar coordinate system, respectively. This is also the first pose transformation relationship of the lidar, or the extrinsic parameters of the lidar. i Let n be the lidar coordinates of a feature point within the target plane of the 3D calibration object in the lidar coordinate system. T Let be the normal vector of the target plane, d be the constant term of the target plane equation, and m be the number of feature points in the target plane.

[0119] Based on the aforementioned cost function, optimization can be achieved using the least squares method. For example, the LM (Levenberg-Marquardt) gradient descent algorithm and the 3D-2D PnP algorithm can be employed.

[0120] 202. Based on the first pose transformation relationship corresponding to each sensor, calculate the second pose transformation relationship between each pair of sensor coordinate systems.

[0121] Specifically, after obtaining the first pose transformation relationship for each sensor, the second pose transformation relationship between any two sensors can be calculated. For example, after obtaining the first pose transformation relationship between camera 1 and the calibration field... First pose transformation relationship between camera 2 and calibration field First pose transformation relationship between lidar and calibration field Then, the second pose transformation relationship between any two sensors among camera 1, camera 2, and LiDAR can be obtained. For example, it can be based on... and Determine the second pose transformation relationship between camera 1 and camera 2. It can also be based on and Determine the second pose transformation relationship between camera 2 and LiDAR. wait.

[0122] 203. Determine the calibration result based on multiple second pose transformation relationships.

[0123] In this embodiment, after obtaining the second pose transformation relationship between each pair of sensors, the calibration result can be determined in multiple ways.

[0124] In one feasible approach, multiple second pose transformation relationships can be directly determined as calibration results for output.

[0125] In another possible implementation, multiple second pose transformation relationships can be verified, and the verified second pose transformation relationship can be determined as the calibration result.

[0126] Specifically, determining the calibration result based on multiple second pose transformation relationships may include: calculating the extrinsic parameter closed-loop difference of the extrinsic parameter transmission closed loop based on the second pose transformation relationships corresponding to multiple sensors; the extrinsic parameter transmission closed loop is composed of multiple sensors; determining whether the extrinsic parameter closed-loop difference is less than or equal to a preset threshold; if so, determining the calibration result based on multiple second pose transformation relationships. If not, jointly optimizing the second pose transformation relationships corresponding to multiple sensors, and determining the calibration result based on the multiple jointly optimized second pose transformation relationships.

[0127] Optionally, calculating the extrinsic closed-loop difference of the extrinsic conduction closed loop based on the second pose transformation relationship corresponding to multiple sensors may include: obtaining the adjacent position relationship of each sensor in the extrinsic conduction closed loop; obtaining the second pose transformation relationship between adjacent sensors in the extrinsic conduction closed loop; multiplying multiple second pose transformation relationships sequentially according to the adjacent position relationship, and determining the obtained product as the extrinsic closed-loop difference.

[0128] For example, the sensors are rigidly connected to the vehicle body. The external parameters are transmitted through different sensors and eventually return to themselves. Theoretically, this is a unit array. Taking multiple sensors, including camera 1, camera 2 and lidar, as an example, the external parameter transmission relationship of each sensor and the external parameter closed-loop difference can be represented by the following expression.

[0129]

[0130] in, This represents the second pose transformation relationship between the laser radar and camera 2. This represents the second pose transformation relationship from camera 1 to camera 2. Let ΔH represent the second pose transformation relationship between the lidar and camera 1, where ΔH is the extrinsic closed-loop difference between camera 1, camera 2, and lidar.

[0131] Using the master laser coordinate system as a reference, the extrinsic parameters (second pose transformation relationship) from the laser to camera 1 are calculated by expressing (3). Multiply by the external parameters of camera 1 to camera 2 Multiply by the camera 2 to the laser The extrinsic parameters are used to obtain the extrinsic closed-loop difference ΔH.

[0132] The external parameter closed-loop difference ΔH may include rotation error ΔR and / or translation error ΔT. For example, the threshold corresponding to the rotation error can be set to 0.5 degrees, and the error corresponding to the translation error can be set to 2 cm. Of course, the specific settings can be made according to actual needs. This embodiment does not specifically limit the composition of the external parameter closed-loop difference or the size of the corresponding threshold.

[0133] In some embodiments, the joint optimization of the second pose transformation relationship corresponding to multiple sensors may include: acquiring the calibration raw data of multiple sensors; the calibration raw data of each sensor is the raw data collected when determining the first pose transformation relationship of the sensor; for each sensor, iteratively optimizing the first pose transformation relationship of the sensor based on the corresponding calibration raw data, so that the extrinsic parameter closed-loop difference is less than or equal to the preset threshold.

[0134] For example, assuming multiple sensors include camera 1, camera 2, and lidar, if ΔH is greater than a preset threshold, the first pose transformation relationship of camera 1, the first pose transformation relationship of camera 2, and the first pose transformation relationship of lidar can be jointly optimized. Specifically, the original data of camera 1 (see the description of step 201, which will not be repeated here), the original data of camera 2, and the original data of lidar can be obtained. Based on the above original data, the cost functions F1 of camera 1 and camera 2 (see expression (1)) and the cost function F2 of lidar (see expression (2)) are combined. The combined cost functions are iteratively optimized based on the preset algorithm to reduce ΔH, so that the error can be evenly distributed to camera 1, camera 2, and lidar. In the joint optimization, the gradient magnitude of the calibration data corresponding to each sensor during the optimization process can be recorded. After each iteration, the larger residual terms and the corresponding calibration data are deleted. Finally, the calibration system converges to the calibration result with the minimum overall error of camera 1, camera 2, and lidar, that is, the calibration result with the minimum ΔH is obtained.

[0135] In some embodiments, if the external parameter closed-loop difference is greater than a preset threshold or the external parameter closed-loop difference is still greater than a preset threshold after joint optimization, each sensor can be recalibrated according to steps 201 and 202.

[0136] The multi-sensor joint calibration method provided in this embodiment calibrates each sensor separately using calibration objects within the calibration field, obtaining the transformation relationship between the sensor coordinates of each sensor and the world coordinate system of the calibration field. Based on this, the pairwise transformation relationships between multiple sensors can be quickly obtained, completing the whole vehicle calibration. This significantly reduces the calibration cycle of the whole vehicle calibration and improves calibration efficiency.

[0137] In various algorithms of autonomous driving technology, it is often necessary to represent the vehicle with a coordinate point, which is usually selected at the center of the rear axle of the vehicle. Therefore, in some embodiments, the algorithm may further include: after the vehicle is parked in the calibration field, determining the coordinates of the center of the rear axle of the vehicle in the world coordinate system of the calibration field.

[0138] Specifically, such as Figure 4 As shown, a limit sticker 401 can be set to park the vehicle at the position of the limit sticker, thereby determining the coordinates of the rear axle center 402 of the vehicle body in the world coordinate system of the calibration field.

[0139] In this embodiment, there are multiple ways to determine the center of the rear axle of the vehicle body.

[0140] In one feasible approach, in the calibration field, around the vehicle's parking position, for example, a steel ruler can be stretched to the left and another to the front of the parking position. These steel rulers have graduations, and the world coordinates of these graduations in the calibration field's world coordinate system are predetermined. After the vehicle is parked in the designated position (e.g., at the position of the safety stop sticker), the world coordinates of the landing point of the vehicle's rear axle center in the calibration field's world coordinate system can be read based on the graduations of the two steel rulers.

[0141] In another possible implementation, to improve efficiency and accuracy, determining the coordinates of the rear axle center of the vehicle in the world coordinate system of the calibration field may include: after the vehicle is parked in the calibration field, acquiring a laser point cloud of a marker, the marker being placed on the vehicle's wheels; registering the laser point cloud of the marker with the point cloud of the calibration field to obtain the world coordinates of the marker in the world coordinate system of the calibration field; and determining the coordinates of the rear axle center of the vehicle in the world coordinate system of the calibration field based on the world coordinates of the marker in the world coordinate system of the calibration field and the positional relationship between the marker and the rear axle center of the vehicle.

[0142] In this embodiment, the reflectivity of the marker can be set to be greater than a preset value, and the shape can be various shapes such as circle and square. The setting method can be magnetic or snap-fit. This embodiment does not limit this.

[0143] For example, such as Figure 1 As shown, dense LiDAR sensors can be installed on four pillars around the vehicle. Multiple dense LiDAR sensors, such as four, can be installed and pointed at the vehicle to acquire its laser point cloud. After the vehicle is parked, markers are placed on each of the four wheels. Laser point clouds are acquired using the dense LiDAR sensors and registered with the first point cloud in the calibration field. Then, the target point cloud corresponding to each marker is extracted from the laser point cloud, and the world coordinates of the target point clouds of the four markers in the calibration field's world coordinate system are determined. Furthermore, based on the world coordinates of the markers, the world coordinates of the rear axle center of the vehicle in the calibration field's world coordinate system can be determined.

[0144] Figure 5 This is a schematic diagram of the structure of the multi-sensor joint calibration device provided in an embodiment of this application. Figure 5 As shown, the multi-sensor joint calibration device 50 includes: a calibration module 501, a calculation module 502, and a determination module 503.

[0145] The calibration module 501 is used to calibrate each sensor according to the calibration object in the calibration field, and obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field.

[0146] The calculation module 502 is used to calculate the second pose transformation relationship between each pair of the sensing coordinate systems of each sensor based on the first pose transformation relationship corresponding to each sensor.

[0147] The determination module 503 is used to determine the calibration result based on multiple second pose transformation relationships.

[0148] The multi-sensor joint calibration device provided in this application calibrates each sensor separately using calibration objects within the calibration field, obtaining the transformation relationship between the sensor coordinates of each sensor and the world coordinate system of the calibration field. Based on this, the transformation relationship between multiple sensors can be quickly obtained pairwise, completing the whole vehicle calibration. This significantly reduces the calibration cycle of the whole vehicle calibration and improves calibration efficiency.

[0149] The multi-sensor joint calibration device provided in this application embodiment can be used to execute the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0150] Figure 6 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. The device may be a computer, a messaging device, a tablet device, a medical device, an in-vehicle device, etc.

[0151] The device 60 may include one or more of the following components: a processing component 601, a memory 602, a power supply component 603, a multimedia component 604, an audio component 605, an input / output (I / O) interface 606, a sensor component 607, and a communication component 608.

[0152] Processing component 601 typically controls the overall operation of device 60, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 601 may include one or more processors 609 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 601 may include one or more modules to facilitate interaction between processing component 601 and other components. For example, processing component 601 may include a multimedia module to facilitate interaction between multimedia component 604 and processing component 601.

[0153] Memory 602 is configured to store various types of data to support operation on device 60. Examples of such data include instructions for any application or method operating on device 60, contact data, phonebook data, messages, pictures, videos, etc. Memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0154] Power supply component 603 provides power to the various components of device 60. Power supply component 603 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 60.

[0155] Multimedia component 604 includes a screen that provides an output interface between the device 60 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 604 includes a front-facing camera and / or a rear-facing camera. When the device 60 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0156] Audio component 605 is configured to output and / or input audio signals. For example, audio component 605 includes a microphone (MIC) configured to receive external audio signals when device 60 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 602 or transmitted via communication component 608. In some embodiments, audio component 605 also includes a speaker for outputting audio signals.

[0157] I / O interface 606 provides an interface between processing component 601 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0158] Sensor assembly 607 includes one or more sensors for providing state assessments of various aspects of device 60. For example, sensor assembly 607 can detect the on / off state of device 60, the relative positioning of components such as the display and keypad of device 60, changes in the position of device 60 or a component of device 60, the presence or absence of user contact with device 60, the orientation or acceleration / deceleration of device 60, and temperature changes of device 60. Sensor assembly 607 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 607 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 607 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0159] Communication component 608 is configured to facilitate wired or wireless communication between device 60 and other devices. Device 60 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 608 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 608 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0160] In an exemplary embodiment, device 60 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0161] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, which can be executed by a processor 609 of the device 60 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0162] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0163] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0164] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0165] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the multi-sensor joint calibration method executed by the multi-sensor joint calibration device described above.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-sensor joint calibration method, characterized in that, Applied to vehicles with multiple sensors rigidly connected together, the method includes: For each sensor, the sensor is calibrated based on the calibration objects in the calibration field to obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field; wherein, the world coordinates of the feature points of each calibration object in the calibration field in the world coordinate system of the calibration field are predetermined; Based on the first pose transformation relationship corresponding to each sensor, calculate the second pose transformation relationship between each pair of sensor coordinate systems; Obtain the adjacent positional relationships of each sensor in the external parameter conduction closed loop; Obtain the second pose transformation relationship between adjacent sensors in the extrinsic parameter conduction closed loop; Based on the adjacent positional relationships, multiple second pose transformation relationships are multiplied sequentially, and the resulting product is determined as the extrinsic parameter closed-loop difference. The extrinsic parameter closed-loop difference also includes rotation error and / or translation error. The extrinsic parameter transmission closed loop is composed of multiple sensors. Determine whether the extrinsic closed-loop difference is less than or equal to a preset threshold; If so, the calibration result is determined based on multiple second pose transformation relationships; If not, then the second pose transformation relationship corresponding to multiple sensors is jointly optimized, and the calibration result is determined based on the multiple jointly optimized second pose transformation relationships.

2. The method according to claim 1, characterized in that, The step of calibrating the sensor based on calibration objects within the calibration field to obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field includes: Based on the two-dimensional calibration board in the calibration field, determine the original calibration data of the camera; The camera is calibrated based on its original calibration data to obtain the first pose transformation relationship between the camera coordinate system and the world coordinate system of the calibration field.

3. The method according to claim 2, characterized in that, The step of determining the original calibration data of the sensor based on the two-dimensional calibration plate in the calibration field includes: The camera captures images of multiple two-dimensional calibration boards to obtain target images; Obtain the world coordinates of multiple first feature points in the world coordinate system and the pixel coordinates of multiple first corresponding points in the pixel coordinate system; the multiple first feature points are feature points on multiple two-dimensional calibration plates; the multiple first corresponding points are corresponding points of the multiple first feature points in the target image respectively. The world coordinates of multiple first feature points and the pixel coordinates of the corresponding first points of the same name are determined as the original calibration data of the camera.

4. The method according to claim 3, characterized in that, The step of calibrating the camera based on its original calibration data to obtain the first pose transformation relationship between the camera coordinate system and the world coordinate system of the calibration field includes: Determine the initial values ​​of the camera's intrinsic parameter matrix; Based on the initial value of the intrinsic parameter matrix and the original calibration data of the camera, the intrinsic parameter matrix is ​​optimized to obtain the first pose transformation relationship between the camera coordinate system of the camera and the world coordinate system of the calibration field.

5. The method according to claim 4, characterized in that, Determining the initial values ​​of the camera's intrinsic parameter matrix includes: Rotate the vehicle to different angles; For each angle, an image to be processed is acquired by the camera; the image to be processed includes at least one two-dimensional calibration plate; Obtain the world coordinates of multiple second feature points in the calibration plate coordinate system and the pixel coordinates of multiple second corresponding points in the pixel coordinate system of the image to be processed; the multiple feature points are feature points on at least one two-dimensional calibration plate; the multiple second corresponding points are corresponding points of the multiple second feature points in the image to be processed respectively; Based on the world coordinates of multiple second feature points and the pixel coordinates of multiple second corresponding points, the initial value of the camera's intrinsic parameter matrix is ​​determined using the Zhang Zhengyou calibration method.

6. The method according to claim 1, characterized in that, The step of calibrating the sensor based on calibration objects within the calibration field to obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field includes: Based on the three-dimensional calibration objects in the calibration field, determine the original calibration data of the lidar; The lidar is calibrated based on its original calibration data to obtain the first pose transformation relationship between the lidar coordinate system and the world coordinate system of the calibration field.

7. The method according to claim 6, characterized in that, The process of determining the original calibration data of the lidar based on the three-dimensional calibration objects within the calibration field includes: Obtain the first point cloud of the stereo calibration object in the world coordinate system; The second point cloud of the three-dimensional calibration object in the lidar coordinate system is obtained through the lidar; Obtain the world coordinate system of multiple third corresponding points and the lidar coordinates of multiple fourth corresponding points; the multiple third corresponding points are the corresponding points of multiple third feature points in the target plane of the three-dimensional calibration object in the first point cloud; the multiple fourth corresponding points are the corresponding points of multiple third feature points in the second point cloud. The world coordinates of the multiple third corresponding points and the lidar coordinates of the multiple fourth corresponding points are determined as the original calibration data for the lidar.

8. The method according to claim 7, characterized in that, The process of obtaining the world coordinate system of multiple third corresponding points and the lidar coordinates of multiple fourth corresponding points includes: Based on the multiple vertices of the stereo calibration object, a first target point cloud corresponding to the stereo calibration object is obtained by filtering from the first point cloud, and a second target point cloud corresponding to the stereo calibration object is obtained by filtering from the second point cloud; Based on the first target point cloud and the second target point cloud, the third corresponding point and the fourth corresponding point are determined respectively for multiple third feature points in the target plane of the stereo calibration object; Obtain the world coordinates of multiple third-named points and the lidar coordinates of multiple fourth-named points.

9. The method according to claim 7, characterized in that, The step of calibrating the lidar based on its original calibration data to obtain the first pose transformation relationship between the lidar coordinate system and the world coordinate system of the calibration field includes: Obtain the normal vector of the target plane of the three-dimensional calibration object; Based on the normal vector and the original calibration data of the lidar, the lidar is calibrated to obtain the first pose transformation relationship between the lidar coordinate system and the world coordinate system of the calibration field.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: After the vehicle is parked in the calibration field, the coordinates of the rear axle center of the vehicle body in the world coordinate system of the calibration field are determined.

11. The method according to claim 10, characterized in that, Determining the coordinates of the rear axle center of the vehicle body in the world coordinate system of the calibration field includes: After the vehicle is parked in the calibration field, a laser point cloud of an object is acquired, the object being placed on the vehicle's wheels; The laser point cloud of the marker is registered with the point cloud of the calibration field to obtain the world coordinates of the marker in the world coordinate system of the calibration field. Based on the world coordinates of the marker in the calibration field's world coordinate system, and the positional relationship between the marker and the rear axle center of the vehicle, the coordinates of the rear axle center of the vehicle in the calibration field's world coordinate system are determined.

12. The method according to claim 1, characterized in that, The joint optimization of the second pose transformation relationship corresponding to multiple sensors includes: Acquire the calibration raw data of multiple sensors; the calibration raw data of each sensor is the raw data collected when determining the first pose transformation relationship of the sensor; For each sensor, the first pose transformation relationship of the sensor is iteratively optimized based on the corresponding calibration raw data, so that the extrinsic parameter closed-loop difference is less than or equal to the preset threshold.

13. A multi-sensor joint calibration device, characterized in that, include: The calibration module is used to calibrate each sensor based on the calibration objects in the calibration field, and obtain the first pose transformation relationship between the sensor's sensing coordinate system and the world coordinate system of the calibration field; wherein, the world coordinates of the feature points of each calibration object in the calibration field in the world coordinate system of the calibration field are predetermined; The calculation module is used to calculate the second pose transformation relationship between each pair of the sensing coordinate systems of each sensor based on the first pose transformation relationship corresponding to each sensor. A determination module is used to obtain the adjacent positional relationships of each sensor in the extrinsic parameter conduction closed loop; obtain the second pose transformation relationship between adjacent sensors in the extrinsic parameter conduction closed loop; multiply multiple second pose transformation relationships sequentially according to the adjacent positional relationships, and determine the obtained product as the extrinsic parameter closed loop difference, the extrinsic parameter closed loop difference also includes rotation error and / or translation error; the extrinsic parameter conduction closed loop is composed of multiple sensors; the multiple sensors are rigidly connected; determine whether the extrinsic parameter closed loop difference is less than or equal to a preset threshold; if yes, determine the calibration result according to multiple second pose transformation relationships; if no, jointly optimize the second pose transformation relationships corresponding to multiple sensors, and determine the calibration result according to multiple jointly optimized second pose transformation relationships.

14. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the multi-sensor joint calibration method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the multi-sensor joint calibration method as described in any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-sensor joint calibration method according to any one of claims 1 to 12.

Images