Vehicle sensor calibration method, system and device and storage medium

Abstract

The invention discloses a vehicle sensor calibration method, system and device and a storage medium, relates to the field of automatic driving, and solves the problem of low vehicle sensor calibration efficiency. In the scheme, first data of scanning each calibration device in the calibration system by a total station is received, and calibration prior information is determined according to the first data; receiving second data acquired by each sensor for the calibration device; and determining calibration parameters of each sensor according to the calibration prior information and the second data. Visibly, the parameter information of each sensor can be accurately solved through the point cloud data obtained through scanning of the total station and the second data collected by the sensors, so that the calibration accuracy of the sensors is improved; according to the scheme, calculation can be automatically completed through the processor, a large amount of time and manpower resources are saved, the working efficiency is improved, and the method can be suitable for mass production.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a vehicle sensor calibration method, system, device and storage medium. Background Art

[0002] The perception system is called the eyes of autonomous driving in the field of autonomous driving. Its main task is to obtain environmental information and identify environmental status. Due to the complexity of ambient lighting, weather and scenes, the use of a single sensor has certain limitations. Therefore, multi-sensor fusion is often used to improve the redundancy and complementarity of information to ensure the safety of autonomous driving. Commonly used perception system sensors include lidar, camera and millimeter wave radar. According to different perception tasks and sensor characteristics, sensors need to be effectively combined. In order to ensure the accuracy of environmental perception, it is usually necessary to accurately calibrate the sensors on the vehicle to determine the internal and external parameters between the sensors. However, there are some problems in the existing sensor calibration methods in engineering applications, such as the need to consume a lot of time and human resources, and it is difficult to achieve mass production. Summary of the invention

[0003] The purpose of this application is to provide a vehicle sensor calibration method, system, device and storage medium. The point cloud data obtained by total station scanning and the second data collected by the sensor can be used to accurately solve the parameter information of each sensor, thereby improving the calibration accuracy of the sensor; the solution in this application can automatically complete the calculation through the processor, saving a lot of time and human resources, improving work efficiency, and can be applied to mass production needs.

[0004] In a first aspect, the present application provides a vehicle sensor calibration method, comprising:

[0005] Receive first data, and determine calibration prior information according to the first data; wherein the first data is point cloud data obtained by a total station scanning each calibration device in a calibration system, and the calibration prior information is data information of the calibration device;

[0006] Receiving second data sent by each sensor disposed on the vehicle to be calibrated, wherein the second data is data collected by each sensor on the calibration device;

[0007] The calibration parameters of each of the sensors are determined according to the calibration priori information and the second data, and the calibration parameters are used to characterize the coordinate transformation relationship between the sensors or between each of the sensors and the total station.

[0008] In one embodiment, receiving first data, and determining calibration prior information according to the first data includes:

[0009] Receiving the first data, removing data that does not meet preset requirements from the first data, and obtaining first calibration data;

[0010] The calibration prior information is determined according to the first calibration data.

[0011] In one embodiment, removing data that does not meet preset requirements from the first data to obtain first calibration data includes:

[0012] The point cloud data not including the calibration device is eliminated from the first data to obtain the first calibration data.

[0013] In one embodiment, removing point cloud data not including the calibration device from the first data to obtain the first calibration data includes:

[0014] The point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system are identified to eliminate the point cloud data corresponding to the ceiling plane and / or the ground plane and / or the surrounding wall planes in the first data.

[0015] In one embodiment, the process of identifying the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system includes:

[0016] Acquire a first distance from the center position of the total station to the ceiling plane and / or the ground plane and / or the surrounding wall planes;

[0017] Randomly select at least three first point clouds from the first data, and calculate the three-dimensional space plane where the first point clouds are located; wherein the difference between the distance from the first point cloud to the center position of the total station and the first distance is not greater than a first preset distance;

[0018] Calculate the second distance from each point cloud in the first calibration data to the three-dimensional space plane, and use the data corresponding to the point cloud whose difference between the second distance and the first distance is not greater than the second preset distance as the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane.

[0019] In one embodiment, it also includes:

[0020] Repeat the step of identifying the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system;

[0021] Determining whether the difference meets the iteration termination condition according to the difference between the number of point clouds in the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration process and the number of point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the previous iteration process;

[0022] If it meets the requirements, the iteration is terminated, and the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration process are used as the final point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes.

[0023] In one embodiment, the calibration device includes a plurality of calibration components, and determining the calibration prior information according to the first calibration data includes:

[0024] Performing spatial clustering segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain first local point cloud data corresponding to each calibration component;

[0025] The first identity information and the first coordinate information of each of the calibration components are determined according to the first local point cloud data.

[0026] In one embodiment, each of the calibration components includes at least two calibration plates, and determining the first identity information and the first coordinate information of each of the calibration components according to the first local point cloud data includes:

[0027] Determine second local point cloud data corresponding to each calibration plate from the first local point cloud data;

[0028] The second identity information and the second coordinate information corresponding to each of the calibration plates are determined according to the second local point cloud data.

[0029] In one embodiment, determining second local point cloud data corresponding to each calibration plate from the first local point cloud data includes:

[0030] Plane fitting is performed on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates in the first local point cloud data.

[0031] In one embodiment, determining the second identity information and the second coordinate information corresponding to each of the calibration plates according to the second local point cloud data includes:

[0032] Converting the second local point cloud data into a grayscale image;

[0033] Identify the grayscale image to determine the second identity information of the calibration plate;

[0034] Recognize the grayscale image to determine the first image coordinates of each key point on the calibration plate on the calibration plate;

[0035] The first three-dimensional coordinates corresponding to each of the key points are determined according to the first image coordinates corresponding to each of the key points and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0036] In one embodiment, after converting the second local point cloud data into a grayscale image, the method further includes:

[0037] Perform filtering processing on the grayscale image.

[0038] In one embodiment, obtaining second data sent by each sensor disposed on the vehicle to be calibrated includes:

[0039] Obtain a first calibration image sent by the camera to be calibrated;

[0040] Calculating calibration parameters of each of the sensors according to the calibration priori information and the second data to calibrate each of the sensors includes:

[0041] Determining, according to the first calibration image, third identity information of the calibration plate and second image coordinates of key points included in the first calibration image, wherein the second image coordinates are used to represent positions of the key points on the calibration plate;

[0042] Determine the second three-dimensional coordinates corresponding to each of the key points in the first calibration image according to the third identity information, the second image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data;

[0043] The intrinsic parameters of the camera to be calibrated are determined according to the second image coordinates and the second three-dimensional coordinates corresponding to each of the key points in the first calibration image.

[0044] In one embodiment, after determining the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates corresponding to each of the key points in the first calibration image, the method further includes:

[0045] Determine the extrinsic parameters of the camera to be calibrated according to the second image coordinates, the second three-dimensional coordinates and the intrinsic parameters of the camera to be calibrated;

[0046] The external parameters of the camera to be calibrated include at least a first coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

[0047] In one embodiment, obtaining second data sent by each sensor disposed on the vehicle to be calibrated includes:

[0048] Obtaining a second calibration image sent by each surround view camera to be calibrated;

[0049] Projecting the second calibration image using the surround stitching homography matrix to obtain a projected image;

[0050] A second coordinate transformation relationship between the coordinate systems of two adjacent surround-view cameras to be calibrated is calculated according to the same key points in the projection images corresponding to the two adjacent surround-view cameras to be calibrated.

[0051] In one embodiment, after obtaining the second calibration image sent by each surround view camera to be calibrated, the method further includes:

[0052] The second calibration image is calibrated for distortion according to the internal parameters of the camera to be calibrated to obtain a calibrated second calibration image.

[0053] In one embodiment, after obtaining the second calibration image sent by each surround view camera to be calibrated, the method further includes:

[0054] Extracting third image coordinates of each of the key points in the second calibration image on the calibration plate and fourth identity information of the calibration plate;

[0055] Filtering key points located on the ground in the second calibration image according to the third image coordinates and the fourth identity information, and obtaining third three-dimensional coordinates of each of the key points located on the ground;

[0056] Determine a surround stitching image according to the third three-dimensional coordinates of each of the key points located on the ground;

[0057] Calibrate the fourth three-dimensional coordinates of the key points in the surround view stitched image using the first reference three-dimensional coordinates corresponding to the key points in the surround view stitched image in the calibration prior information to obtain a calibrated surround view stitched image;

[0058] The surround view stitching homography matrix is ​​obtained according to the calibrated surround view stitching image.

[0059] In one embodiment, obtaining second data sent by each sensor disposed on the vehicle to be calibrated includes:

[0060] Obtain the first test point cloud data sent by the main laser radar;

[0061] The first test point cloud data is matched with the first data to obtain a third coordinate transformation relationship between the main lidar coordinate system and the total station coordinate system.

[0062] In one embodiment, after obtaining the first test point cloud data sent by the main laser radar, the method further includes:

[0063] Matching the first test point cloud data with the first data to obtain key points within the field of view of the main laser radar and fifth three-dimensional coordinates corresponding to the key points;

[0064] Determine the same key points in the first calibration image as those within the field of view of the primary laser radar according to the second three-dimensional coordinates and the fifth three-dimensional coordinates, and determine the three-dimensional coordinates of the same key points;

[0065] The fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main lidar coordinate system is calculated according to the three-dimensional coordinates of the same key point.

[0066] In one embodiment, a metal block is provided on the calibration plate, and obtaining second data sent by each sensor provided on the vehicle to be calibrated includes:

[0067] Acquire the collected data of the metal block sent by the millimeter wave radar;

[0068] Determine alignment data between the first test point cloud data and the collected data according to the data timestamp;

[0069] Projecting the sixth three-dimensional coordinate of the metal block in the first test point cloud data in the alignment data onto a horizontal plane to obtain horizontal plane projection data;

[0070] The fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main lidar coordinate system is calculated according to the horizontal plane projection data, the coordinates of the metal blocks in the collected data in the alignment data, and the positions of each metal block in the calibration system.

[0071] In one embodiment, obtaining second data sent by each sensor disposed on the vehicle to be calibrated includes:

[0072] Obtain the second test point cloud data sent by the auxiliary laser radar;

[0073] The second test point cloud data is matched with the first test point cloud data to obtain a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system.

[0074] In one embodiment, the calibration device includes a plurality of verification components, and determining the calibration prior information according to the first calibration data includes:

[0075] Performing spatial clustering segmentation on the first calibration data according to the position of each of the verification components in the calibration system to obtain third local point cloud data corresponding to each of the verification components;

[0076] The fifth identity information and the fifth coordinate information corresponding to each of the verification components are determined according to the third local point cloud data.

[0077] In one embodiment, after calculating the calibration parameters of each of the sensors according to the calibration prior information and the second data to calibrate each of the sensors, the method further includes:

[0078] The calibration parameters of each of the sensors are verified according to the verification component.

[0079] In one embodiment, when the calibration parameters of the sensor include a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of each sensor are verified according to the verification component, including:

[0080] Obtaining the seventh three-dimensional coordinates of the verification component within the field of view of the main laser radar;

[0081] Performing coordinate transformation on the seventh three-dimensional coordinate according to the third coordinate transformation relationship to obtain a seventh three-dimensional coordinate to be compared;

[0082] Comparing the seventh three-dimensional coordinate to be compared with the seventh reference three-dimensional coordinate to determine whether the third coordinate transformation relationship is accurate;

[0083] The first reference three-dimensional coordinates are the three-dimensional coordinates obtained by the total station and corresponding to the verification component within the field of view of the main laser radar.

[0084] In one embodiment, when the calibration parameters of the sensor include a fourth coordinate transformation relationship between a camera coordinate system to be calibrated and a main laser radar coordinate system, the calibration parameters of each sensor are verified according to the verification component, including:

[0085] Acquire first verification point cloud data of the verification component within the field of view of the main laser radar;

[0086] Back-projecting the first verification point cloud data onto the camera image according to the fourth coordinate transformation relationship;

[0087] It is determined whether the fourth coordinate transformation relationship is accurate based on the back-projected camera image.

[0088] In one embodiment, when the calibration parameters of the sensor include a fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the verification component, including:

[0089] Obtaining the millimeter-wave radar coordinates and angles of each calibration ball within the field of view of the millimeter-wave radar;

[0090] Acquire a seventh three-dimensional coordinate within the field of view of the main laser radar;

[0091] Projecting the millimeter wave radar coordinates and angles onto a two-dimensional plane of the main laser radar coordinate system according to the fifth coordinate transformation relationship;

[0092] The Euclidean distance is calculated according to the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates to determine whether the fifth coordinate transformation relationship is accurate.

[0093] In one embodiment, when the calibration parameters of the sensor include a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the verification component, including:

[0094] Acquire first verification point cloud data of the verification component within the field of view of the main laser radar;

[0095] Acquire second verification point cloud data of the verification component within the field of view of the auxiliary laser radar;

[0096] The second verification point cloud data is transformed into the main laser radar coordinate system according to the sixth coordinate transformation relationship to obtain the second verification point cloud data to be verified;

[0097] Determine whether the sixth coordinate transformation relationship is accurate based on the first verification point cloud data and the second verification point cloud data to be verified.

[0098] In a second aspect, the present application provides a vehicle sensor calibration system, comprising:

[0099] A first receiving unit is used to receive first data and determine calibration prior information according to the first data; wherein the first data is point cloud data obtained by scanning each calibration device in the calibration system by the total station, and the calibration prior information is data information of the calibration device;

[0100] A second receiving unit, used for acquiring second data sent by each sensor disposed on the vehicle to be calibrated, wherein the second data is data collected by each sensor on the calibration device;

[0101] A sensor calibration unit is used to determine calibration parameters of each of the sensors according to the calibration prior information and the second data, wherein the calibration parameters are used to characterize the coordinate transformation relationship between the sensors or between each of the sensors and the total station.

[0102] In a second aspect, the present application provides a vehicle sensor calibration device, comprising:

[0103] Memory for storing computer programs;

[0104] The processor is used to implement the steps of the vehicle sensor calibration method as described above when storing a computer program.

[0105] In a second aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the vehicle sensor calibration method as described above are implemented.

[0106] The present application provides a vehicle sensor calibration method, system, device and storage medium, which relates to the field of autonomous driving and solves the problem of low efficiency in vehicle sensor calibration. In this solution, the first data of the total station scanning each calibration device in the calibration system is received, and the calibration prior information is determined according to the first data; the second data collected by each sensor on the calibration device is received; the calibration parameters of each sensor are calculated according to the calibration prior information and the second data to calibrate each sensor. It can be seen that the point cloud data obtained by scanning the total station and the second data collected by the sensor in this application can accurately solve the parameter information of each sensor, thereby improving the calibration accuracy of the sensor; the solution in this application can automatically complete the calculation through the processor, saving a lot of time and human resources, improving work efficiency, and can be applied to mass production needs.

[0107] The present application also provides a vehicle sensor calibration system, device and storage medium, which have the same beneficial effects as the vehicle sensor calibration method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the prior art and the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0109] Figure 1 A schematic diagram of a vehicle sensor calibration method provided in this application;

[0110] Figure 2 A schematic diagram of a three-dimensional view of a calibration component for forward splicing provided in this application;

[0111] Figure 3 A schematic side view of a calibration component for forward splicing provided in this application;

[0112] Figure 4 A schematic diagram of a three-dimensional view of a reverse-stitched calibration component provided in this application;

[0113] Figure 5A schematic side view of a reverse-jointed calibration component provided in this application;

[0114] Figure 6 A schematic diagram of a three-dimensional view of a hybrid spliced ​​calibration component provided in this application;

[0115] Figure 7 A schematic side view of a hybrid spliced ​​calibration component provided in this application;

[0116] Figure 8 Schematic diagram of the calibration board structure provided for this application;

[0117] Fig. 9 A schematic diagram of a partial image of the four corners of the calibration code provided in this application;

[0118] Fig.10 A schematic diagram of establishing the world coordinate system provided for this application;

[0119] Fig.11 An example diagram of an existing calibration plate provided for this application;

[0120] Fig.12 Another example diagram of an existing calibration plate provided for this application;

[0121] Fig.13 A top view of the surround stitching calibration layout provided for this application;

[0122] Fig.14 A schematic diagram of the projection transformation matrix selection provided for this application;

[0123] Fig.15 A schematic diagram showing the relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system provided for this application;

[0124] Fig.16 A schematic diagram of the layout of the calibration ball provided for this application;

[0125] Fig.17 Schematic diagram of projecting the point cloud provided by this application onto a camera image;

[0126] Fig.18 Schematic diagram of point cloud point extraction provided for this application;

[0127] Fig.19 A schematic diagram of a point cloud point fitting circle provided in this application;

[0128] Fig. 20 A schematic diagram of the vehicle parking area layout provided for this application;

[0129] Fig.21 A top view of the vehicle parking area layout provided for this application;

[0130] Fig. 22 Side view of the vehicle parking area layout provided for this application;

[0131] Fig.23 A structural block diagram of a vehicle sensor calibration system provided in this application;

[0132] Fig.24 A structural block diagram of a vehicle sensor calibration device provided in this application;

[0133] Fig.25 A structural block diagram of a computer-readable storage medium provided for this application. DETAILED DESCRIPTION

[0134] The core of this application is to provide a vehicle sensor calibration method, system, device and storage medium. The point cloud data obtained by total station scanning and the second data collected by the sensor can be used to accurately solve the parameter information of each sensor, thereby improving the calibration accuracy of the sensor. The solution in this application can automatically complete the calculation through the processor, saving a lot of time and human resources, improving work efficiency, and can be applied to mass production needs.

[0135] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0136] Autonomous driving is developing rapidly and has become the focus of competition in the automotive and technology industries. Modular autonomous driving architecture design has become the mainstream solution because of its interpretability and ease of maintenance. The perception system is a key link in the autonomous driving system, acting as the eyes of the vehicle and providing environmental status information for path planning. Due to the complex environment, existing perception systems usually use multi-sensor fusion to improve information redundancy and complementarity to ensure safety. Commonly used sensors include lidar, cameras, and millimeter-wave radars, which need to be effectively combined according to the task and sensor characteristics. In order to ensure accurate perception, the sensor needs to be accurately calibrated to determine the relationship between internal and external parameters. However, existing calibration methods have problems in engineering applications. They require manual participation and are time-consuming and labor-intensive, making them difficult to mass produce. The solution process is long, and the initial value problem may cause problems with the accuracy of the results. The calibration results are difficult to understand, especially for non-professionals.

[0137] Before describing the embodiments of the present application, first of all, in order to obtain the best calibration accuracy and improve the calibration automation rate, first of all, according to the vehicle structure design parameters, the three-dimensional coordinates of the center of the main laser radar relative to the vehicle tire landing point can be obtained, and the position of the main laser mine center on the ground projection point can be further obtained. The total station is installed in the calibration site, and the center position of the total station is required to be consistent with the center of the main laser radar. The specific steps are as follows: Step 1: coincide the plumb point at the center position of the total station with the ground projection point of the main laser mine center; Step 2: make the height of the total station center relative to the ground consistent with the height of the main laser mine center relative to the ground; Step 3: make the x-axis direction of the total station coordinate system consistent with the x-axis direction of the main laser radar; Step 4: adjust the total station to a horizontal level.

[0138] Please refer to Figure 1 , Figure 1 This is a flow chart of a vehicle sensor calibration method provided by the present application. The method uses a processor or a server. The processor or the server can be located in the cloud or on the vehicle. The method includes:

[0139] S11: receiving first data, and determining calibration prior information according to the first data; wherein the first data is point cloud data obtained by the total station scanning each calibration device in the calibration system, and the calibration prior information is data information of the calibration device;

[0140] In this step, the first data is first received. These data are point cloud data obtained by scanning each calibration device in the calibration system by the total station. The total station is an instrument used to measure and draw three-dimensional space, and point cloud data can be obtained by scanning the calibration device. The calibration system is repeatedly scanned by the total station to obtain dense point cloud data of the calibration site. These point cloud data contain specific information of the calibration device and can be used to determine the data information of the calibration device, that is, calibration prior information. Calibration prior information is reference data for sensor calibration, which can help determine the internal and external parameters of the sensor.

[0141] S12: receiving second data sent by each sensor disposed on the vehicle to be calibrated, where the second data is data collected by each sensor on the calibration device;

[0142] In this step, the second data sent by each sensor on the vehicle to be calibrated is received. These second data are the data collected by each sensor for the calibration device. The sensor may include a laser radar, a camera, a millimeter wave radar, etc. By collecting these data, the performance of the sensor in the actual scene can be understood and compared with the calibration prior information.

[0143] S13: Determine calibration parameters of each sensor according to the calibration priori information and the second data, where the calibration parameters are used to characterize the coordinate transformation relationship between the sensors or between the sensors and the total station.

[0144] In this step, the calibration parameters of each sensor are calculated based on the calibration prior information and the second data to calibrate the sensor. The calibration parameters are the internal and external parameters of the sensor. By adjusting these parameters, the measurement results of the sensor can be made more accurate and reliable.

[0145] This embodiment receives the calibration device data obtained by the total station scan and the data collected by the sensor, performs calculations and analysis to obtain the calibration parameters of the sensor, and finally realizes the accurate calibration of the sensor, which can improve the perception ability and safety of the autonomous driving system and provide effective support for the development of autonomous driving technology. In addition, the calibration method in this embodiment is automatically executed by the processor without human participation, saving a lot of time and human resources, improving work efficiency, and can be applied to mass production needs.

[0146] In one embodiment, receiving first data and determining calibration prior information according to the first data includes:

[0147] Receiving first data, removing data that does not meet preset requirements from the first data, and obtaining first calibration data;

[0148] Calibration prior information is determined according to the first calibration data.

[0149] Specifically, the first data is point cloud data obtained by scanning each calibration device in the calibration system through a total station. The calibration prior information refers to the data information of the calibration device. When implementing this method, it is first necessary to receive the first data and process it. The processing step includes eliminating data that does not meet the preset requirements from the first data, thereby obtaining the first calibration data. Through this step, data that does not meet the calibration requirements can be excluded to ensure the accuracy and reliability of the first calibration data.

[0150] Next, the calibration prior information is determined based on the first calibration data. By analyzing and processing the first calibration data, data information of the calibration device can be obtained. This information can be used as prior knowledge for subsequent calculation of calibration parameters and calibration of sensors.

[0151] In this embodiment, by processing the first data, data that does not meet the preset requirements are eliminated, thereby avoiding the negative impact of these data on the calibration results. At the same time, by determining the calibration prior information based on the first calibration data, the existing data information can be better utilized to improve the calculation accuracy of the calibration parameters, thereby achieving accurate calibration of each sensor.

[0152] In one embodiment, removing data that does not meet preset requirements from the first data to obtain first calibration data includes:

[0153] The point cloud data not including the calibration device is eliminated from the first data to obtain the first calibration data.

[0154] Since there are many useless point cloud data in the calibration scene, such as point cloud data of the ground, walls, ceiling, etc. in non-marking areas, it is necessary to filter out the useless point cloud in order to reduce the calculation load of subsequent calibration of the main lidar, auxiliary lidar, etc. and improve the calibration accuracy.

[0155] Specifically, this processing step removes the point cloud data that does not include the calibration device from the first data to obtain the first calibration data. This means that only the data related to the calibration device is retained, while the data related to other factors (such as environment, background, etc.) is filtered out. Through this processing step, we can obtain a set of first calibration data that meets the preset requirements as the basis for calculating the calibration parameters of each sensor.

[0156] The method in this embodiment can ensure that only data related to the calibration device is used during the calibration process, thereby improving the accuracy and reliability of the calibration. This method is used for the calibration of various vehicle sensors, effectively improving the performance and accuracy of the sensors.

[0157] In one embodiment, removing point cloud data not including the calibration device from the first data to obtain first calibration data includes:

[0158] The point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane in the calibration system are identified to eliminate the point cloud data corresponding to the ceiling plane and / or the ground plane and / or the surrounding wall plane in the first data.

[0159] This embodiment describes a method for processing first data to obtain first calibration data. The purpose of this embodiment is to remove point cloud data corresponding to the ceiling plane and / or the ground plane and / or the surrounding wall planes in the first data. By removing these data, more accurate first calibration data can be obtained for determining calibration prior information.

[0160] This embodiment can be implemented by identifying the point cloud data of the ceiling plane, the ground plane and / or the surrounding wall plane in the calibration system. These planes can be obtained by scanning the point cloud data of each calibration device in the calibration system with a total station. During the recognition process, image processing or point cloud processing algorithms can be used to analyze the point cloud data to determine the position and shape of the ceiling plane, the ground plane and / or the surrounding wall plane. By identifying and eliminating the point cloud data corresponding to these planes, the purpose of eliminating the point cloud data related to these planes can be achieved.

[0161] In summary, more accurate first calibration data can be obtained through the method in this embodiment, thereby improving the precision and accuracy of sensor calibration. This helps to ensure that more reliable and accurate results are obtained when calculating the calibration parameters of each sensor, thereby effectively calibrating each sensor.

[0162] In one embodiment, the process of identifying the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system includes:

[0163] Obtaining a first distance from the center position of the total station to the ceiling plane and / or the ground plane and / or the surrounding wall planes;

[0164] Randomly select at least three first point clouds from the first data, and calculate the three-dimensional space plane where the first point clouds are located; wherein the difference between the distance from the first point cloud to the center position of the total station and the first distance is not greater than the first preset distance;

[0165] Calculate the second distance from each point cloud in the first calibration data to the three-dimensional space plane, and use the data corresponding to the point cloud whose difference between the second distance and the first distance is not greater than the second preset distance as the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane.

[0166] In this embodiment, since the installation position information of the total station is known (including x, y, z, whether it is horizontal, etc.), it is easy to use the positional relationship between the surrounding walls, ground, ceiling and the center of the total station of the calibration system as prior information. Therefore, in this embodiment, it is first necessary to obtain the first distance from the center position of the total station to the ceiling plane and / or the ground plane and / or the surrounding wall plane. Then, at least three first point clouds are randomly selected from the first data, and the three-dimensional space plane where these point clouds are located is calculated. The point cloud here refers to the data points obtained by the total station scanning each calibration device in the calibration system. Next, the distance from the first point cloud to the center position of the total station is calculated and compared with the first distance. If this difference is not greater than the first preset distance, the point cloud data can be used as the data of the ceiling plane and / or the ground plane and / or the surrounding wall plane. Finally, the second distance from each point cloud in the first calibration data to the three-dimensional space plane is calculated and compared with the first distance. If this difference is not greater than the second preset distance, the corresponding point cloud data can be used as the data of the ceiling plane and / or the ground plane and / or the surrounding wall plane.

[0167] The first preset distance and the second preset distance may be the same or different, and this application does not make any special limitation thereto.

[0168] In this embodiment, by measuring the total station and calculating the plane where the point cloud is located, the point cloud data corresponding to the ceiling plane and / or the ground plane and / or the surrounding wall plane are identified and excluded, thereby obtaining the first calibration data. By processing and comparing these data, the point cloud data that meets the preset conditions can be used as the calculation calibration prior information. These calibration prior information can be used to calculate the calibration parameters of each sensor to calibrate each sensor.

[0169] In one embodiment, it also includes:

[0170] Repeat the step of identifying the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system;

[0171] Determining whether the difference meets the iteration termination condition according to the difference between the number of point clouds in the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration process and the number of point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the previous iteration process;

[0172] If it meets the requirements, the iteration is terminated, and the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration process are used as the final point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes.

[0173] This embodiment describes an iterative process for identifying point cloud data of a ceiling plane and / or a ground plane and / or a surrounding wall plane in a calibration system. This process includes multiple cycles of executing the identification step, and each iteration determines whether an iteration termination condition is met by comparing the difference between the number of point cloud data obtained in the current iteration process and the number of point cloud data obtained in the previous iteration process.

[0174] In each iteration process, firstly, according to the previously obtained point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane, a point cloud data recognition step is performed. This step uses the distances from the center position measured by the total station to these planes and at least three point clouds selected from the first data to calculate the three-dimensional space plane and the distance from each point cloud to the plane. Finally, point cloud data with a small difference from the preset distance is selected from these distances and used as the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane in the new iteration.

[0175] After each iteration, the difference between the number of point cloud data obtained in the current iteration and the number of point cloud data obtained in the previous iteration is compared, and it is determined whether the difference meets the preset iteration termination condition. For example, if the difference is not greater than the preset number, it is considered that the iteration process has converged and the iteration is terminated. Alternatively, if the ratio of the number of point cloud data obtained in the current iteration to the number of point cloud data obtained in the previous iteration is not greater than the preset ratio, it is considered that the iteration process has converged and the iteration is terminated. At the same time, the point cloud data obtained in the current iteration is used as the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes.

[0176] Through this iterative approach, the recognition of point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system can be gradually improved and optimized, thereby obtaining a more accurate and reliable calibration result.

[0177] For example, assuming that the calibration site point cluster is Φ, the specific steps are as follows:

[0178] According to the spatial distribution characteristics of point clouds, in order to improve the efficiency of plane fitting, the interference effects are sorted and eliminated. The ceiling has the least interference, followed by the ground, and finally the surrounding walls. Therefore, the order of plane extraction is: ceiling point cloud → ground → surrounding walls;

[0179] Perform point cloud recognition on the ceiling plane. Set the distance z from the center of the total station to the ceiling Top as prior information.

[0180] Step 1: Set up the point cloud Randomly select the content ( Select N (N ≥ 3) points of the plane with the first preset distance (in meters), z is the vertical coordinate of the point cloud, and the distance between the N points is required to be greater than the threshold L Top (unit is meter), calculate the three-dimensional space plane formed by these N points

[0181] It should be noted that although the total station is adjusted to a horizontal level, it is difficult to ensure that the six planes in the calibration system are completely horizontal or vertical, that is, there are certain errors. For example, if the ceiling is not horizontal and has a large slope, the effect of identifying the plane based on a single condition will be very poor. Therefore, by setting a certain threshold On the one hand, it can greatly reduce the number of subsequent iterations, and on the other hand, it has a certain adaptability to non-horizontal or non-vertical planes. It should also be noted that L Top The setting of , ensure that the spatial distribution of N points is not too close. If the randomly selected N points are too close, it is easy to fit a plane with large errors.

[0182] Step 2: Calculate the point cloud set All points to the three-dimensional space plane distance, and according to the distance threshold (i.e. the second preset distance) (in meters) to obtain the distance between the plane and the three-dimensional space Close point set

[0183] Step 3: Calculate whether the iteration termination condition is met (in, is the number of point cloud data obtained during the i+1th iteration, is the number of point cloud data obtained in the i-th iteration process). If J>T top (T top is the iteration threshold, such as 0.9), then is the ceiling plane point cloud P Top Otherwise, let (by gradually reducing A more accurate plane point cloud can be obtained) and the calculation starts again from the first step.

[0184] Among them, when the first step is executed for the first time, a point cloud set Φ is randomly selected to satisfy The N (N ≥ 3) points of the condition, that is

[0185] Further, identify the ground plane point cloud P Bottom And the plane point cloud of the surrounding walls (p Left 、p Right 、p Front 、p Back The principle of ) is similar to the principle of identifying the ceiling plane point cloud mentioned above, and this application will not repeat it here. Taking the front wall as an example, the distance x from the center of the total station to the front wall front As prior information, the corresponding point cloud can be obtained by using the ceiling point cloud computing method.

[0186] In one embodiment, the calibration device includes a plurality of calibration components, and determines the calibration prior information according to the first calibration data, including:

[0187] Performing spatial clustering and segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain first local point cloud data corresponding to each calibration component;

[0188] The first identity information and the first coordinate information of each calibration component are determined according to the first local point cloud data.

[0189] In this embodiment, when the calibration device includes multiple calibration components, these calibration components have different positions in the calibration system. First, according to the point cloud data in the first data, it can be spatially clustered and segmented. This means that the points in the point cloud data can be grouped according to their positions, so that the points in the same group are closer to each other and farther away from the points in other groups. Through this segmentation process, the first local point cloud data corresponding to each calibration component is obtained. Next, the first identity information and first coordinate information of each calibration component can be determined by analyzing the first local point cloud data. By analyzing the point cloud data of each calibration component, the characteristics or shape of each calibration component can be identified and compared with the known calibration components to confirm their identity information. At the same time, the coordinate information (such as three-dimensional coordinate information) of each calibration component in the calibration system can also be determined by analyzing the position information in the point cloud data.

[0190] It should be noted that the first identity information is used to represent the identity information of the calibration component. When the calibration component includes multiple calibration plates, and each calibration plate has its own corresponding identity information, the first identity information can be a combination of the identity information of the multiple calibration plates. The first coordinate information is used to represent the spatial position of the calibration component in the calibration system, such as the three-dimensional coordinates of the center position of the calibration component in the calibration system.

[0191] Through the above process, the first identity information and the first coordinate information of each calibration component can be determined, and these information will be used to subsequently calculate the calibration parameters of each sensor to calibrate each sensor. In this way, the vehicle sensor calibration method can effectively and accurately calibrate each sensor according to the data information of the calibration device.

[0192] In one embodiment, each calibration component includes at least two calibration plates, and determining the first identity information and the first coordinate information of each calibration component according to the first local point cloud data includes:

[0193] Determine second local point cloud data corresponding to each calibration plate from the first local point cloud data;

[0194] The second identity information and the second coordinate information corresponding to each calibration plate are determined according to the second local point cloud data.

[0195] This embodiment illustrates the composition of a calibration component in a vehicle sensor calibration method and the process of determining the corresponding identity information and coordinate information. Each calibration component in the method includes at least two calibration plates. First, the first identity information and first coordinate information of each calibration component are determined based on the first local point cloud data. This can be achieved by determining the second local point cloud data corresponding to each calibration plate from the first local point cloud data. Then, based on the second local point cloud data, the second identity information and second coordinate information corresponding to each calibration plate can be determined.

[0196] It should be noted that the second identity information is used to characterize the identity information of the calibration plate. If the calibration plate includes multiple calibration codes, the identity information of the calibration plate can be a combination of the identity information of the calibration codes. In this embodiment, the combination of the second identity information of the calibration plate included on the calibration component is the first identity information of the calibration component. The second coordinate information is used to characterize the position of the calibration plate in the calibration system. If the three-dimensional coordinates of the calibration component and the relative position of the calibration plate and the calibration component are known, the position of each calibration plate can be known.

[0197] In this way, the calibration parameters of the sensor can be determined based on the calibration components on the vehicle and the data collected by the sensor. This ensures that each sensor can accurately perceive the surrounding environment and provide accurate data.

[0198] Here, the calibration member involved in the embodiment is described. Specifically, the calibration member may include at least two calibration plates and a connecting member; the connecting member is fixed between two adjacent calibration plates and is used to connect the two calibration plates. Figure 2 As shown, multiple calibration plates, such as the lower edge of calibration plate 1 and the upper edge of calibration plate 2, are connected by a connector to obtain a calibration component. The calibration component in the present application includes multiple calibration plates at the same location, such as the upper, lower, left, and right, so as to increase the number of calibrations of the horizontal and / or vertical field of view (FOV) of sensors on the vehicle, such as cameras, laser radars, and millimeter-wave radars, at the same calibration position, thereby improving the calibration accuracy.

[0199] Furthermore, in the calibration component provided by the present application, the connecting member is also used to control the angle between any one of the two calibration plates and the vertical plane where the two calibration plates are located. Specifically, the connecting member in the calibration component provided by the present application can also adjust the angle between multiple connected calibration plates. Figure 3 As shown, the angles between the calibration plates 1 and 2 and the plumb plane where the calibration component is located can be adjusted according to user needs. For example, based on the consideration that the marking codes on the calibration plates 1 and 2 are easy for sensors to identify, by adjusting the connecting parts, the marking plate 2 remains parallel to the plumb plane where the marking component is located, and the marking plate 1 is adjusted to have an angle with the plumb plane, for example, 30 degrees.

[0200] Furthermore, since the calibration component provided by the present application can adjust the angle between multiple connected calibration plates according to user needs, it is convenient for various sensors on the vehicle to effectively identify the calibration plates. In addition, the multiple calibration plates on the calibration component have angles, which increases non-coplanar points and effectively improves the calibration accuracy.

[0201] Furthermore, in the calibration component provided by the present application, the front faces of the two calibration plates are located on the same side of the calibration component. Figure 3 As shown, the calibration component provided by the present application can forward splice multiple calibration plates. At this time, the front of the calibration plate 1 and the calibration plate 2 are located on the same side of the calibration component, and the angle between the calibration plate 1 and the plumb plane is adjusted according to the sensor calibration requirements of the vehicle, such as 30 degrees. When the sensor of the vehicle needs to be calibrated by the sensor, the QR code information on the same side of the calibration plate 1 and the calibration plate 2 can be obtained for calibration between different sensors on the vehicle. Because the calibration component provided by the present application increases the non-coplanar points recognized by the calibration plate, the calibration accuracy of the sensor is improved.

[0202] Furthermore, in the calibration component provided by the present application, the front surfaces of the two calibration plates are located on both sides of the calibration component. Figure 4-5 As shown, the calibration component provided by the present application can also reversely splice multiple calibration plates. Figure 4 and Figure 5 They are respectively a stereoscopic view and a side view of the calibration component when reversely spliced. At this time, the front of calibration plate 1 and calibration plate 2 are located on both sides of the calibration component, and the angle between calibration plate 1 and the plumb plane is adjusted according to the sensor calibration requirements of the vehicle, such as 30 degrees. When the vehicle's sensor needs to be calibrated by the sensor, the QR code information on the front of calibration plate 1 and the QR code information on the back of calibration plate 2 can be obtained for calibration between different sensors on the vehicle.

[0203] In addition, if Figure 6 and Figure 7 As shown, the calibration component provided by the present application can also be obtained by combining a plurality of calibration plates spliced ​​in the forward direction and calibration plates spliced ​​in the reverse direction. Figure 6 and Figure 7 They are respectively a stereoscopic view and a side view of the calibration component obtained by combining multiple different splicing methods. Specifically, the calibration component provided in this application can be spliced ​​with multiple calibration plates at a certain angle according to the calibration requirements of sensors such as cameras, laser radars, and millimeter-wave radar fields of view. Figure 2 and Figure 4 The two calibration plates spliced ​​together are combined to obtain Figure 6At this time, when calibrating the vehicle sensor, it is necessary to identify the angle, mark code and other information of the combination of the positive and negative calibration plates or the calibration unit, which further improves the calibration accuracy of the vehicle sensor calibration.

[0204] It should be emphasized that the present application does not limit the specific number of calibration plates in the calibration component. Users can increase or decrease the number of calibration plates according to their own needs, based on the number of sensors and field of view angles on different vehicles, so as to meet the sensor calibration requirements of different vehicles.

[0205] On the basis of the above-mentioned implementation manner, since the calibration plates in the calibration component provided in the present application face different front and back sides of the vehicle sensor, the non-coplanar differences of the reference objects used for sensor calibration are further increased, thereby improving the calibration accuracy of the vehicle sensor calibration.

[0206] In one embodiment, determining second local point cloud data corresponding to each calibration plate from the first local point cloud data includes:

[0207] Plane fitting is performed on each calibration plate to extract second local point cloud data corresponding to each calibration plate in the first local point cloud data.

[0208] This embodiment describes a step of processing the first local point cloud data to determine the second local point cloud data corresponding to each calibration plate. The processing step includes performing plane fitting on each calibration plate to extract the second local point cloud data corresponding to each calibration plate. In this embodiment, for each calibration plate, it is first found in the first local point cloud data. Then, the point cloud data on each calibration plate is processed using a plane fitting technique, and the plane closest to the surface of the calibration plate is found. Plane fitting can fit a plane by minimizing the distance between the point and the plane. In this embodiment, a plane fitting technique is used to extract the second local point cloud data corresponding to each calibration plate, that is, the point cloud data located near the surface of the calibration plate.

[0209] By fitting the plane of each calibration plate, the second local point cloud data corresponding to each calibration plate can be determined, that is, the point cloud data located at a position close to the calibration plate on the plane obtained by plane fitting.

[0210] Through this processing step, the second local point cloud data corresponding to each calibration plate can be extracted from the first local point cloud data for subsequent calibration parameter calculation and sensor calibration, which can enhance the accuracy and reliability of the calibration, thereby improving the performance and precision of the vehicle sensor.

[0211] In one embodiment, performing plane fitting on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates includes:

[0212] Obtaining a first height between the center of each calibration plate and the ground, randomly selecting at least three second point clouds from the first local point cloud, and calculating a second three-dimensional space plane where the second point cloud is located;

[0213] Calculating a third distance between each point cloud in the second local point cloud data and the second three-dimensional space plane;

[0214] The data corresponding to the point cloud whose difference between the third distance and the first height is not greater than the third preset distance is used as the second local point cloud data corresponding to the calibration plate.

[0215] This embodiment describes a method for plane fitting of each calibration plate to extract the second local point cloud data corresponding to each calibration plate. The specific steps include: obtaining the first height between the center of each calibration plate and the ground. Randomly selecting at least three point clouds from the first local point cloud data as the second point cloud. Using the selected second point cloud to calculate a three-dimensional space plane. For each point cloud in the first local point cloud data, calculate its vertical distance to the second three-dimensional space plane, that is, the third distance. Compare the difference between the third distance and the first height, and filter out the point cloud data that satisfies the difference not greater than the third preset distance. These point cloud data are regarded as the second local point cloud data corresponding to the calibration plate.

[0216] For example, only the calibration component point cloud dataset is included Clustering is performed based on the spatial distance of the calibration components (since the calibration components are spaced far apart, they are easy to segment in space), and the point cloud data set of each calibration component (i.e., the first local point cloud data) is first obtained. ( is the point cloud data set corresponding to the kth calibrated component, M2 is the total number of calibrated components);

[0217] Since each calibration component is composed of S (usually S is 3 or 4) non-coplanar calibration plates, it is possible. Using the plane fitting principle, extract the point cloud data of each calibration plate from top to bottom (or from bottom to top). The specific steps are as follows:

[0218] Step 1: Calibrate the component point cloud dataset According to the center height z of the calibration plate i (i∈S) information (i.e. the first height mentioned above), randomly select ( The plane selection error threshold is in meters) and the distance between the N points is required to be greater than the threshold l. a (Usually l a<0.7w, W is the calibration width in meters), calculate the three-dimensional space plane formed by these N points

[0219] It should be noted that the calibration component is composed of multiple calibration plates. The installation height of each calibration plate (i.e., the center height of the calibration plate) is known. Through this prior information, the plane fitting efficiency can be accelerated and the plane fitting accuracy can be improved. Directly randomly find three second point clouds to fit the plane. If the three second point clouds are distributed on different calibration plates, this kind of invalid calculation is likely to occur, and fall into more invalid cycles. Therefore, this first step can not only accurately fit the plane of the calibration plate, but also improve the efficiency of plane fitting.

[0220] Step 2: Calculation All points to the 3D plane The distance (third distance), and according to the distance threshold (Unit: meter) Get the plane Close point set

[0221] Step 3: Calculate the iteration termination condition If J>T (T is the threshold, such as 0.9), then is the point cloud of the i-th calibration plate on the k-th calibration component; otherwise, let Restart the calculation from the first step.

[0222] Through the above steps, the point cloud of the i-th calibration plate can be obtained Then from Eliminate Repeat the above steps to obtain the point cloud of each calibration plate on the kth calibration component one by one

[0223] Through the above steps, the second local point cloud data corresponding to each calibration plate that meets the requirements can be extracted from the first local point cloud data, thereby achieving the goal of plane fitting for each calibration plate. This method can accurately extract the shape and position information of each calibration plate, providing valuable data reference for subsequent sensor calibration.

[0224] In one embodiment, determining the second identity information and the second coordinate information corresponding to each calibration plate according to the second local point cloud data includes:

[0225] Convert the second local point cloud data into a grayscale image;

[0226] Recognize the grayscale image to determine the second identity information of the calibration plate;

[0227] Recognize the grayscale image to determine the first image coordinates of each key point on the calibration plate on the calibration plate;

[0228] The first three-dimensional coordinates corresponding to each key point are determined according to the first image coordinates corresponding to each key point and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0229] Specifically, the present embodiment first converts the second local point cloud data into a grayscale image. This is to simplify the process of image processing and analysis, and convert the point cloud data into image data. Then, the grayscale image is identified to determine the second identity information of the calibration plate. Through the technology of image processing and pattern recognition, the calibration plate appearing in the image can be automatically identified and its identity information can be determined. Next, the grayscale image is identified to determine the first image coordinates of each key point on the calibration plate on the calibration plate. Similarly, through image processing technology, each key point on the calibration plate can be identified and their coordinate positions on the image can be determined. Finally, according to the first image coordinates corresponding to each key point and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data, the first three-dimensional coordinates corresponding to each key point can be calculated. This can be achieved by registering and matching the point cloud data and the image data to determine the position of each key point in three-dimensional space.

[0230] The first image coordinates represent the position of each key point on the calibration plate, and the first reference three-dimensional coordinates are known. If the position of the calibration plate and the first image coordinates of the key point on the calibration plate are known, the first three-dimensional coordinates corresponding to each key point can be determined. The set of all first three-dimensional coordinates is the set of three-dimensional coordinates of all key points.

[0231] In general, this embodiment utilizes the second local point cloud data collected by the sensor, through image processing and pattern recognition technology, as well as point cloud data registration and matching methods, to determine the three-dimensional position of key points on the calibration plate, thereby calibrating the sensor.

[0232] like Figure 8 As shown, Figure 8 A schematic diagram of a calibration plate provided for this application.

[0233] In one embodiment, the calibration plate includes a calibration carrier plate and a plurality of marking codes, wherein the plurality of marking codes are fixed on the calibration carrier plate, the marking codes carry two-dimensional code information for identification by the sensor, wherein the identity information corresponding to the two-dimensional code information of all the marking codes is different;

[0234] Determining second identity information corresponding to each calibration plate according to the second local point cloud data includes:

[0235] Convert the second local point cloud data into a grayscale image;

[0236] The grayscale image is segmented according to the intervals between the marker codes to obtain a local image of each marker code;

[0237] Perform two-dimensional code recognition on each partial image to determine the first ID information of each mark code, and determine the second identity information of the calibration plate according to the first ID information of each mark code on each mark plate.

[0238] This embodiment describes a calibration board design in a vehicle sensor calibration system. The calibration board includes a calibration carrier and multiple marking codes. These marking codes are fixed on the calibration carrier, and each marking code carries a two-dimensional code information for sensor recognition. It should be noted that the identity information corresponding to the two-dimensional code information of each marking code is different.

[0239] On this basis, the specific implementation method of determining the second identity information corresponding to each calibration plate according to the second local point cloud data is as follows: converting the second local point cloud data into a grayscale image to facilitate subsequent image processing and analysis. According to the interval between the marker codes (this is a known condition), the grayscale image is segmented to obtain a local image of each marker code, and each marker code is separated for separate processing. Then, each local image is subjected to two-dimensional code recognition. By decoding and analyzing the two-dimensional code image, the first ID information of each marker code can be obtained. This first ID information can be used as a unique identity identifier of the marker code. Finally, according to the first ID information of each marker code on each marker board, the second identity information of the calibration plate is determined (for example, the identity information of the marker board is a combination of the first ID information of several marker codes on the marker board in a preset manner). This means that by integrating and comparing the first ID information of each marker code, the identity information of the entire calibration plate can be determined. Since the identity information corresponding to the two-dimensional code information of each marker code is different, such a comparison can ensure that each calibration plate can be uniquely identified.

[0240] In one embodiment, after converting the second local point cloud data into a grayscale image, the method further includes: performing filtering processing on the grayscale image.

[0241] The method in this embodiment can calibrate the sensor more accurately and improve the stability and accuracy of the calibration system.

[0242] In one embodiment, determining the second coordinate information corresponding to each calibration plate according to the second local point cloud data includes:

[0243] Extract the four sides of the circumscribed rectangle of the partial image of each logo code, and determine the intersection of two adjacent sides as the key point;

[0244] Obtaining the first image coordinates of each key point, where the first image coordinates represent the position of the key point on the calibration plate;

[0245] The first three-dimensional coordinates corresponding to each key point are determined according to the position of the marking code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0246] Correspondingly, the specific implementation method of determining the second coordinate information corresponding to each calibration plate according to the second local point cloud data is: extract the four sides of the circumscribed rectangle of the local image of each marker code; determine the intersection of two adjacent sides as the key point, and obtain the first image coordinates of each key point, that is, the position of the key point on the calibration plate, and calculate the first three-dimensional coordinates corresponding to each key point according to the position of the marker code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0247] In this embodiment, by extracting the four sides of the circumscribed rectangle of the partial image of the marker code to determine the key points, the error caused by image noise can be effectively reduced and the calibration accuracy can be improved. By using the marker code as a feature point for recognition, the robustness of the system to changes in the calibration plate posture and sensor observation angle can be improved. By using the two-dimensional code technology, the first ID information of the marker code can be quickly and accurately obtained, thereby reducing the time and cost of manual processing.

[0248] In summary, the method described in this embodiment can realize the calibration of vehicle sensors by identifying key points on the calibration plate and their corresponding first three-dimensional coordinates, and has the beneficial effects of high precision, high robustness and fast calibration.

[0249] In one embodiment, the calibration plate further comprises a metal block, the metal block is fixed at the center of the calibration carrier, a plurality of marking codes are fixed on the calibration carrier and surround the metal block, and the vertices of the plurality of marking codes close to the metal block are connected to the vertices of the metal block;

[0250] Determining second identity information corresponding to each calibration plate according to the second local point cloud data also includes:

[0251] The identity information of the metal block is determined according to the first ID information of each identification code.

[0252] If the millimeter wave radar sensor needs to be calibrated, a metal block can be set in the calibration plate. When making the calibration plate, the calibration code ID information of the four neighboring areas of the metal block is known a priori information. Therefore, the identity information of the metal block can be encoded by the four first ID information of the calibration code of the four neighboring areas in a clockwise manner, such as ID lt ID rt ID rd ID ld .

[0253] Wherein, the identity information of each metal block is different. When there is a calibration code on each calibration plate, the identity information of the metal block can be a combination of the first ID information of the calibration code in a first manner. The second identity information of the calibration plate in the above embodiment can be a combination of the first ID information in a second manner, or a combination of each first ID information and the identity information of the metal block.

[0254] In one embodiment, determining the second coordinate information corresponding to each calibration plate according to the second local point cloud data further includes:

[0255] Determine the three-dimensional coordinates of the four corner points of the metal block according to the first three-dimensional coordinates of the key points corresponding to the respective marking codes and the positional relationship between the metal block and the plurality of marking codes;

[0256] The three-dimensional coordinates of the center of the metal block are determined according to the three-dimensional coordinates of the four corner points of the metal block.

[0257] In this embodiment, after obtaining the three-dimensional coordinates of the key points of the calibration code, the three-dimensional coordinates of the four corner points of the metal block can be obtained according to the relationship between the four corners of the metal block and the vertical angles of the key points of the calibration code. Then, the three-dimensional coordinates of the center of the metal block can be calculated according to the three-dimensional coordinates of the four corner points.

[0258] In the calibration plate structure described in the above embodiment, the calibration carrier is a plate body for carrying the mark code and the metal block, and the material can be selected from non-metallic materials that are not easily deformed, such as acrylic plates. The metal block located in the center of the calibration carrier is used for calibration of millimeter-wave radars and other sensors installed on autonomous driving vehicles. The mark code is a two-dimensional code that can encode identity information, such as a binary square reference mark (Augmented Reality University of Cordoba, ArUco) for camera posture estimation, which is convenient for sensors such as laser radars installed on autonomous driving vehicles to identify each mark code on the mark board. Among them, the size and number of the mark codes can be adjusted according to user needs based on factors such as camera focal length and laser radar field of view. In order to facilitate the sensor to distinguish different mark codes, all mark codes on the calibration plate cannot be repeated, and the identity information of each mark code is unique.

[0259] It should be emphasized that the metal block in the technical solution provided in the present application is used for obtaining spatial homonymous points (the same as the same key points described in the following embodiments) when calibrating the millimeter-wave radar and the main laser radar on the vehicle, thereby achieving the calibration effect of the millimeter-wave radar and the main laser radar, and the ID information carried in the mark code and the QR code information on the mark code is used for subsequent automatic calibration actions between the camera internal parameters and the camera external parameters. The ID information carried in the mark code and the QR code information on the mark code can also be used for subsequent automatic calibration actions between the camera and the laser radar.

[0260] Among them, the points with the same name are used to unify the data of different sensors in the same three-dimensional space. Only after obtaining multiple different coordinates of the same point corresponding to different sensors can these sensors be calibrated accordingly according to the differences in these coordinates. For example, the center of the same small ball in space has coordinates (u, v) in the image information obtained by the vehicle's camera; (x, y, z) in the point cloud obtained by the vehicle's laser radar; and (x, y, θ) in the point cloud obtained by the vehicle's millimeter-wave radar. These coordinates are unified in a three-dimensional coordinate system, and the parameters of the corresponding sensors are adaptively adjusted according to the differences in different coordinates, thereby achieving the calibration effect of the vehicle sensor.

[0261] The calibration plate provided in the present application includes a metal block located in the center for sensor calibration and multiple marking codes surrounding the metal block. Since the two-dimensional code information on the multiple marking codes corresponds to different identity information, the identity ID information of the metal block can be quickly identified according to the identity information. The vehicle's sensor can realize automatic recognition on images and point cloud data according to different ID information, and the automatic matching effect of key points is good; moreover, the ID information can assist in the acquisition of the ID information and spatial position information of the metal block located in the middle of the marking code. Different metal blocks will not be confused due to excessive similarity, thereby solving the problem of automatic correspondence between spatial points of laser radar and millimeter-wave radar.

[0262] The technical solution provided by the present application can identify the identity of the metal block surrounding the marking codes through a unique two-dimensional code among multiple marking codes, without the need for manual assistance, thereby improving the calibration accuracy and the calibration work efficiency.

[0263] It should be further explained that in the calibration plate provided by the present application, in order to facilitate the vehicle sensor to distinguish the mark code and the metal block on the calibration carrier, a dark paint that has good absorption of the laser point cloud can be sprayed on the calibration carrier to obtain the mark code, such as a black mark code or a mark code with a large color difference from the calibration carrier. When the vehicle sensor is calibrated, such as a laser radar, the position of the calibration carrier can be obtained through plane fitting of the calibration plate point cloud, and the calibration plate point cloud is further filtered and segmented to identify the ID information and three-dimensional coordinates of key points of each mark code on the calibration plate. At this time, in order to facilitate distinction, the surface color of the non-metallic material selected for the calibration carrier itself needs to be different from the color of the mark code coating.

[0264] Since the dark area surfaces of the calibration carrier plate and the marking code in the calibration plate provided in this application have significant differences in color and other aspects, the laser radar point clouds at different positions in the calibration plate form differences, which can improve the marking code recognition efficiency of the laser radar point cloud.

[0265] It should be further explained that, in the calibration plate provided in the present application, the metal block is a polygonal structure, the marking code is a polygonal result, the vertices of multiple marking codes close to the metal block are connected to the vertices of the metal block, and the number of vertices of the metal block is greater than or equal to 4.

[0266] Specifically, in the calibration plate provided by the present application, the metal block and the marking code on the calibration plate are both polygonal in shape, and the multiple vertices of the metal block are directly connected to the vertex angles of the multiple marking codes surrounding the metal block. At this time, the vehicle sensor only needs to obtain the identity information and position information of the multiple adjacent marking codes of the metal block to completely locate the metal block, thereby establishing an association between the identity information and the positioning information of the metal block. In order to ensure the accuracy of positioning, the number of marking codes is at least 4, and the metal block has at least four sides.

[0267] Since the metal block in the calibration plate provided by the present application is connected to the vertices of the marking code, the position and identity information of the metal block can be obtained by identifying the position and identity information of the marking code, thereby facilitating the calibration between the millimeter-wave radar and other sensors installed on the vehicle and improving the versatility of the present application.

[0268] It should be further explained that, in the calibration plate provided in the present application, the number of marking codes is 4, and the polygonal structures of the metal block and the marking codes are both rectangular.

[0269] Specifically, in the technical solution provided by the present application, the marking code and the metal block on the calibration plate are both rectangular, and the four vertices of the metal block are respectively connected to or overlapped with a vertex of the marking code close to the metal block.

[0270] On the basis of the above implementation, since the calibration plate provided by the present application can take into account the calibration of the camera's internal and external parameters, the ArUco codes corresponding to at least 4 mark codes on the same calibration plate have more than 16 key points in total. Further use the key points of multiple sets of calibration plates within the camera's field of view and determine the three-dimensional spatial coordinates of each key point based on each key point and the point cloud data obtained by the total station to perform camera internal and external parameter calibration, on the one hand, to achieve automated calibration, and on the other hand, to improve the accuracy of camera internal parameter solution.

[0271] On the basis of the above implementation, the two-dimensional code information carried by the marker code in the calibration plate provided in the present application is a quasi-two-dimensional code that is convenient for the camera to decode, such as an ArUco code.

[0272] Based on the above embodiment, an embodiment of calibrating each sensor on the vehicle is as follows:

[0273] In one embodiment, obtaining second data sent by each sensor disposed on the vehicle to be calibrated includes:

[0274] Obtain a first calibration image sent by the camera to be calibrated;

[0275] Calculating calibration parameters of each sensor according to the calibration prior information and the second data to calibrate each sensor, including:

[0276] Determining, according to the first calibration image, third identity information of the calibration plate and second image coordinates of the key points included in the first calibration image, wherein the second image coordinates are used to represent positions of the key points on the calibration plate;

[0277] Determine the second three-dimensional coordinates corresponding to each key point in the first calibration image according to the third identity information, the second image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data;

[0278] The intrinsic parameters of the camera to be calibrated are determined according to the second image coordinates and the second three-dimensional coordinates corresponding to each key point in the first calibration image.

[0279] This embodiment provides a method for calibrating the intrinsic parameters of a camera to be calibrated on a vehicle. Specifically, first, a first calibration image sent by the camera to be calibrated is obtained, and the identity information of the calibration plate and the image coordinates of the key points included in the first calibration image are determined through the first calibration image. Then, the first reference three-dimensional coordinates of each point cloud in the second local point cloud data are used to determine the second three-dimensional coordinates corresponding to the key points in the first calibration image. Finally, the intrinsic parameters of the camera to be calibrated are determined through the second image coordinates and the second three-dimensional coordinates of the key points.

[0280] Among them, the third identity information is the identity information of the calibration plate included in the first calibration image, and the set of calibration plates included in the first calibration image is a subset of the set of calibration plates used in the calibration system, so the third identity information is a part of the second identity information of all calibration plates. Then, on the basis of the third identity information of the calibration plate, the second image coordinates of each key point on the calibration plate, and the first reference three-dimensional coordinates, the second three-dimensional coordinates corresponding to each key point can be determined. That is, the second three-dimensional coordinates are a part of the first three-dimensional coordinates of all key points.

[0281] For example, the intrinsic parameters of a camera include the focal length f x 、f y , principal point coordinates u0, v0 and lens distortion k1, k2 (only mirror distortion is considered here). Taking the vehicle forward camera as an example, assume that n calibration plates (denoted as C i , i∈n), each calibration plate image has m calibration code key points (denoted as M j , j∈m).

[0282] For different calibration boards, the internal parameters of the same camera are consistent. Based on this, the steps to obtain the initial value of the internal parameter calibration are as follows:

[0283] (1) For the camera to be calibrated, collect a frame of calibration image (the first calibration image);

[0284] (2) Obtain the ID and partial image of each calibration code through the two-dimensional code recognition method. For each partial image of the calibration code, use Hough transform to extract the four sides of the circumscribed rectangle of the calibration code, and determine the second image coordinates of the key point through the intersection of two sides. (k=0,1,2,3, respectively represent the upper left corner, upper right corner, lower right corner, lower left corner). As the center, take a 11×11 local image, such as Fig. 9 For each local image, platform kernel fitting is performed to further obtain the sub-pixel coordinates (x ij ,y ij ) i∈n,j∈m (Sub-pixel extraction is performed here to further increase the solution accuracy).

[0285] Identify the second image coordinates (x ij ,y ij ) i∈n,j∈m and ID information ij(i∈n, j∈m) (the calibration code is the jth calibration code in the i-th calibration plate). Among them, the second three-dimensional coordinates (X ij , Y ij , Z ij ) i∈n,j∈m .

[0286] For each calibration plate visible in the camera field of view, establish the world coordinate system O w X w Y w Z w (In this embodiment, the coordinate system of the total station is used as the world coordinate system), that is, the key point of the upper left corner of the calibration code of the calibration plate is used as the origin of the world coordinate system O w ; The horizontal axis of the calibration plate is X w Axis, vertical axis is Y w Axis, according to the right-hand principle to establish Z w Axis, such as Fig.10 For each calibration plate visible in the camera field of view, the second image coordinates (x ij ,y ij ) i∈n,j∈m With the second three-dimensional coordinate (X ij , Y ij , Z ij ) i∈n,j∈m , according to Zhang Zhengyou calibration method, solve the camera internal parameter f x 、f y , principal point coordinates u o , v0 and lens distortion k1, k2.

[0287] It should be noted that Zhang Zhengyou's calibration method requires collecting multiple calibration images (non-coplanar) with different viewing angles and distances to obtain the camera's intrinsic parameters. Because the camera's intrinsic parameters are consistent for different images, this method can only solve the external parameters relative to each calibration plate image, but cannot obtain the true camera's external parameters.

[0288] Traditional camera intrinsic calibration requires collecting multiple calibration plates (such as chessboards, Fig.11 and Fig.12 The calibration process requires that the calibration plate be placed at different positions and on different planes (non-coplanar). In this embodiment, only one frame of image needs to be collected, which improves the efficiency of calibrating the intrinsic parameters of the camera to be calibrated.

[0289] In summary, in this embodiment, the calibration parameters of each sensor are calculated by using the identity information and image coordinates of the calibration plate and the correspondence between the local point cloud data collected by the sensor and the reference three-dimensional coordinates. Through these calibration parameters, the sensor can be accurately calibrated to improve its measurement accuracy and positioning capability.

[0290] In one embodiment, after determining the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates corresponding to each key point in the first calibration image, the method further includes:

[0291] Determine the extrinsic parameters of the camera to be calibrated according to the second image coordinates, the second three-dimensional coordinates and the intrinsic parameters of the camera to be calibrated;

[0292] The external parameters of the camera to be calibrated include at least a first coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

[0293] In this embodiment, after determining the intrinsic parameters of the camera to be calibrated, the extrinsic parameters of the camera to be calibrated are also determined. The so-called camera intrinsic parameters are the internal parameters of the camera described in the above embodiment, including focal length, principal point coordinates and distortion parameters, etc., which are used to describe the geometric characteristics inside the camera. The camera extrinsic parameters describe the geometric relationship between the camera coordinate system and the total station coordinate system, that is, the position and posture of the camera in the total station coordinate system.

[0294] In this embodiment, the intrinsic parameters of the camera to be calibrated are determined according to the second image coordinates and the second three-dimensional coordinates corresponding to each key point in the first calibration image, and the next step is to determine the extrinsic parameters of the camera to be calibrated. Specifically, the extrinsic parameters of the camera to be calibrated are determined using the second image coordinates, the second three-dimensional coordinates and the intrinsic parameters of the camera to be calibrated.

[0295] Before determining the external parameters of the camera to be calibrated, you first need to understand the relationship between the coordinate system of the camera and the coordinate system of the total station. The total station has its own independent coordinate system, and the camera to be calibrated also has its own coordinate system. There is a certain coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station, which includes translation, rotation, and scale parameters. According to the internal parameters of the camera to be calibrated and the second image coordinates and second three-dimensional coordinates corresponding to each key point in the first calibration image, the coordinate transformation relationship between the camera and the total station coordinate system can be used to determine the external parameters of the camera to be calibrated. Specifically, the external parameters of the camera to be calibrated can be determined by solving the translation, rotation, and scale parameters between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

[0296] By determining the external parameters of the camera to be calibrated, the position and posture of the camera to be calibrated in the total station coordinate system can be accurately described, so as to be used for calibrating the camera to be calibrated in the vehicle sensor calibration method of the present application.

[0297] Specifically, the steps to calibrate the external parameters of the camera to be calibrated are:

[0298] For the 3D space points of all calibration code key points within the camera field of view (Second three-dimensional coordinate) (k∈n×m, n is the number of calibration plates, m is the number of marking codes on the calibration plates), the corresponding two-dimensional pixel on the image is (x ij ,u ij ) i∈n,j∈m (Second image coordinates). Theoretically, a 3D space point passes through the camera intrinsics The two-dimensional pixel obtained after the lens distortion k1, k2 and the external parameters R, T are transformed is (x i ' j ,y i ' j ) i∈n,j∈m .

[0299] Therefore, a function F can be constructed, whose parameters are internal parameters and external parameters. The second three-dimensional coordinate passes through the function F and outputs a two-dimensional projection image coordinate (second image coordinate) (x i ' j ,y i ' j ) i∈n,j∈m .

[0300] That is, R is the rotation matrix in the first coordinate transformation relationship, and T is the translation vector in the first coordinate transformation relationship.

[0301] In practice, the identified (x ij ,y ij ) i∈n,j∈m Compared with the theoretical (x i ' j ,y i ' j ) i∈n,j∈m There is a certain deviation. Therefore, the function to be optimized can be constructed as follows:

[0302]

[0303] Use the Levenberg-Marquardt (LM) algorithm to iterate and solve. Here, we take the first step to obtain the focal length f x 、f y , principal point coordinates u0, v0 and lens distortion k1, k2 are used as initial values ​​for iterative calculation. By giving better initial values, subsequent iterations converge faster.

[0304] In one embodiment, obtaining second data sent by each sensor disposed on the vehicle to be calibrated includes:

[0305] Obtaining a second calibration image sent by each surround view camera to be calibrated;

[0306] Projecting the second calibration image using the surround stitching homography matrix to obtain a projected image;

[0307] The second coordinate transformation relationship between the coordinate systems of the two adjacent surround-view cameras to be calibrated is calculated according to the same key points in the projection images corresponding to the two adjacent surround-view cameras to be calibrated.

[0308] In this embodiment, when the sensor includes a surround view camera to be calibrated, first, the second calibration image sent by each surround view camera to be calibrated is obtained: this step involves collecting the calibration images taken by each surround view camera on the vehicle to be calibrated. The second calibration image is projected using a surround view stitching homography matrix to obtain a projection image: in this step, a surround view stitching homography matrix technology is used to project the second calibration image to generate a corresponding projection image. The second coordinate transformation relationship between the coordinate systems of two adjacent surround view cameras to be calibrated is calculated based on the same key points in the projection images corresponding to the two adjacent surround view cameras to be calibrated: by comparing the same key points in the projection images of the two adjacent surround view cameras to be calibrated, the coordinate system transformation relationship between the two adjacent surround view cameras to be calibrated can be calculated.

[0309] In this embodiment, the image data of the surround view cameras to be calibrated are collected, and the surround view stitching homography matrix technology is used for projection transformation, and finally the coordinate system transformation relationship between two adjacent surround view cameras to be calibrated is calculated. This result can be used as part of the calculation of sensor calibration parameters to calibrate each sensor on the vehicle to be calibrated.

[0310] In one embodiment, after acquiring the second calibration images sent by each of the surround view cameras to be calibrated, the method further includes: performing distortion calibration on the second calibration images according to the internal parameters of the cameras to be calibrated to obtain a calibrated second calibration image.

[0311] In one embodiment, after obtaining the second calibration image sent by each surround view camera to be calibrated, the method further includes:

[0312] Extracting third image coordinates of each key point in the second calibration image on the calibration plate and fourth identity information of the calibration plate;

[0313] Filter key points on the ground in the second calibration image according to the third image coordinates and the fourth identity information, and obtain third three-dimensional coordinates of each key point on the ground;

[0314] Determine a surround stitching image according to the third three-dimensional coordinates of each key point located on the ground;

[0315] Calibrate the fourth three-dimensional coordinates of the key points in the surround view stitching image using the first reference three-dimensional coordinates corresponding to the key points in the surround view stitching image in the calibration prior information to obtain a calibrated surround view stitching image;

[0316] The surround view stitching homography matrix is ​​obtained according to the calibrated surround view stitching image.

[0317] Among them, the fourth identity information is the identity information of each calibration plate within the field of view corresponding to the second calibration image, and the set of each calibration plate within the field of view corresponding to the second calibration image is a subset of the set of all calibration plates in the calibration system, so correspondingly, the fourth identity information is a part of all the second identity information. Similarly, the third three-dimensional coordinates are the three-dimensional coordinates of all key points within the field of view corresponding to the second calibration image and located on the ground, and the set of all key points within the field of view corresponding to the second calibration image and located on the ground is a subset of the set of all key points in the calibration system, so the third three-dimensional coordinates are a part of all the first three-dimensional coordinates. Further, the fourth three-dimensional coordinates are the three-dimensional coordinates of the key points in the surround stitching image determined according to the third three-dimensional coordinates of each key point on the ground, and the set of key points in the surround stitching image determined according to the third three-dimensional coordinates of each key point on the ground is a subset of the set of all key points within the field of view corresponding to the second calibration image and located on the ground, so the fourth three-dimensional coordinates are a part of all the third three-dimensional coordinates.

[0318] Specifically, the surround stitching calibration task is mainly to calibrate the ground projection of the surround view cameras around the vehicle body and the transformation relationship between adjacent cameras. The specific steps are as follows:

[0319] For each surround camera c i , collect the second calibration image I i , according to the internal parameters of each camera obtained in the previous step, the second calibration image I i Perform distortion calibration to obtain the calibrated second calibration image

[0320] For the second calibration image after calibration Extract the third image coordinates and ID information (fourth identity information) of the key points within its field of view, and according to the ID information, only retain the key points on the ground, and obtain the third three-dimensional coordinate information of the key points on the ground (such as Fig.13 , Fig.13 This is a top view of the surround stitching calibration layout provided in this application). Compared with manual point selection, this embodiment is aimed at mass production and solves the automation problem.

[0321] For each camera, automatically obtain the ID information of the camera. Fig.13Select the key point corresponding to the largest sign code in the left and right sides. Select the dotted rectangle ABCD in the same way as in 14. Fig.14 A schematic diagram of the projection transformation matrix selection provided for this application.

[0322] When building the calibration site, it is difficult to ensure Fig.13 All calibration codes in the image are horizontal and coplanar, which is ignored by existing methods, resulting in insufficient surround stitching accuracy. Fig.13 The three-dimensional coordinates of the key points of the calibration code (X i , Y i , Z i ), as well as the camera internal and external parameters obtained in the first step, Fig.14 In the calibration, four points ABCD are used, that is, (X i , Y i , Z i ) is transformed to the image through the internal and external parameters (no distortion parameters are used here) After calibration, the point (like Fig.14 The location of the middle black dot);

[0323] use Solve the surround stitching homography matrix E; for the key points in the overlapping fields of view of adjacent cameras, solve the transformation scale parameters according to the key point ID information and the actual size of the cooperation mark.

[0324] In one embodiment, obtaining second data sent by each sensor disposed on the vehicle to be calibrated includes:

[0325] Obtain the first test point cloud data sent by the main laser radar;

[0326] The first test point cloud data is matched with the first data to obtain a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system.

[0327] This embodiment describes the process of calibrating the main laser radar. Specifically, firstly, the first test point cloud data sent by the main laser radar is obtained. Next, the first test point cloud data is matched with the first data to obtain the third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system.

[0328] In this embodiment, the second data collected by the sensor refers to the data collected by each sensor on the calibration device. Among them, the first test point cloud data sent by the main laser radar is one of them. These sensors and calibration devices can transmit data through wireless or wired connections.

[0329] By matching the first test point cloud data with the first data, the third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system can be determined. This transformation relationship can be used in the subsequent sensor calibration process. By obtaining the third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of the main laser radar can be calculated. These parameters can include rotation matrices and / or translation vectors, etc.

[0330] In one embodiment, after obtaining the first test point cloud data sent by the main laser radar, the method further includes:

[0331] Matching the first test point cloud data with the first data to obtain key points of the field of view of the main laser radar and fifth three-dimensional coordinates corresponding to the key points;

[0332] Determine the same key points in the first calibration image and within the field of view of the main laser radar according to the second three-dimensional coordinates and the fifth three-dimensional coordinates, and determine the three-dimensional coordinates of the same key points;

[0333] The fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main lidar coordinate system is calculated based on the three-dimensional coordinates of the same key points.

[0334] It should be noted that the fifth three-dimensional coordinate is the three-dimensional coordinate corresponding to the key point of the field of view of the main laser radar. The set of key points in the field of view of the main laser radar is a subset of the set of all key points in the calibration system. Then the fifth three-dimensional coordinate is a part of all the first three-dimensional coordinates.

[0335] Specifically, assuming that camera c i The coordinates of the key points in the coordinate system are (the second three-dimensional coordinate described above), the key point coordinates in the main laser radar l1 coordinate system (that is, the fifth three-dimensional coordinates described above) are Theoretically, the transformation relationship between the same-name points in the two coordinate systems is as follows:

[0336]

[0337] Considering various errors,

[0338] The function to be optimized is as follows:

[0339] The calibration steps are:

[0340] (1) extracting the image coordinates and ID of the key points in the field of view of the c1 camera, and obtaining the corresponding three-dimensional spatial coordinates (i.e., the second three-dimensional coordinates mentioned above) according to the ID information;

[0341] (2) extracting the three-dimensional coordinates (i.e., the fifth three-dimensional coordinates) and ID of the key points in the field of view of the main laser radar l1;

[0342] (3) Based on the ID information and the relationship between the same-name points, the Levenberg-Marquardt (LM) algorithm is used to solve the external parameters (the fourth coordinate transformation relationship).

[0343] In one embodiment, a metal block is provided on the calibration plate, and obtaining second data sent by each sensor provided on the vehicle to be calibrated includes:

[0344] Obtain the data collected on the metal block sent by the millimeter wave radar;

[0345] Determine alignment data of the first test point cloud data and the collected data according to the data timestamp;

[0346] Projecting the sixth three-dimensional coordinate of the metal block in the first test point cloud data in the alignment data onto a horizontal plane to obtain horizontal plane projection data;

[0347] The fifth coordinate transformation relationship between the millimeter-wave radar coordinate system and the main lidar coordinate system is calculated according to the horizontal plane projection data, the coordinates of the metal blocks in the collected data in the alignment data, and the positions of each metal block in the calibration system.

[0348] It should be noted that the sixth three-dimensional coordinate is the three-dimensional coordinate corresponding to the metal block within the field of view of the millimeter-wave radar. The set of metal blocks within the field of view of the millimeter-wave radar is a subset of the set of all metal blocks in the calibration system. Therefore, the sixth three-dimensional coordinate is a part of the three-dimensional coordinates of all the above metal blocks.

[0349] Specifically, the millimeter wave radar can obtain the x and y coordinate information of the target, but does not have the z coordinate information of the target. Therefore, the millimeter wave coordinate system O w X w Y w To the main lidar coordinate system O l X l Y l The transformation is regarded as the transformation of the two-dimensional XY coordinate system (such as Fig.15 As shown, Fig.15 Schematic diagram of the relationship between the millimeter-wave radar coordinate system and the main lidar coordinate system provided in this application), the parameters to be calibrated include the translation vector T and the rotation angle θ.

[0350] The conversion relationship from millimeter wave coordinates to the main lidar coordinate system is:

[0351]

[0352] We can construct a function F whose parameter is T x 、Ty , θ. The three-dimensional world coordinates pass through the function F and output a two-dimensional projection image coordinate (x i ' j ,y i ' j );

[0353] (x l ,y l )=F(T x , T y ,θ,x w ,y w );

[0354] Considering the error factor,

[0355] The function to be optimized is:

[0356]

[0357] The calibration steps are as follows:

[0358] (1) Since the vehicle is fixed on the track, the conveyor belt on the track can be used to easily control the vehicle to move in a short distance in the front and rear direction (during the calibration process, it only needs to move a very short distance (such as 1 meter));

[0359] (2) Save the main laser radar data (first test point cloud data) and millimeter wave radar data (collection data of metal blocks sent by the millimeter wave radar) during the vehicle movement;

[0360] (3) According to the data timestamp, find the time-aligned data frame of the main laser radar and the millimeter-wave radar. For the three-dimensional coordinates of the central metal block of each calibration plate within the field of view of the main laser radar l1 (the sixth three-dimensional coordinates of the metal block), project it onto the horizontal plane to obtain the horizontal plane projection data; for the millimeter radar, identify the coordinates of all metal blocks within its field of view;

[0361] (4) According to the horizontal projection data and the known spatial topological relationship of the metal block, the optimal matching relationship is obtained. According to the optimal matching relationship, the Levenberg-Marquardt (LM) algorithm is used to iteratively calculate the above optimization function to solve the fifth coordinate transformation relationship.

[0362] In one embodiment, obtaining second data sent by each sensor disposed on the vehicle to be calibrated includes:

[0363] Obtain the second test point cloud data sent by the auxiliary laser radar;

[0364] The second test point cloud data is matched with the first test point cloud data to obtain a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system.

[0365] Specifically, the calibration parameters (sixth coordinate transformation relationship) between the auxiliary laser radar and the main laser radar are the rotation matrix R and the translation vector T.

[0366] Assume that the point cloud data of the main laser radar l1 are Q = {q1, q2, .., q m}(first test point cloud data), the auxiliary laser radar l2 point cloud data is P = {p1, p2, .., p m}(Second test point cloud data). Using the calibration site to calibrate the spatial topological structure of the component, according to the spatial matching of the point cloud, the coordinate transformation relationship of the spatial points with the same name in the two point cloud data is:

[0367] Constructing the optimization function: Through the Gauss-Newton or LM algorithm, R and T corresponding to the sixth coordinate transformation relationship can be obtained.

[0368] The main / auxiliary lidar calibration steps are as follows:

[0369] (1) Distributed acquisition of primary lidar and auxiliary lidar point cloud data Q and P;

[0370] (2) For the point cloud data Q and P, use the method in the preprocessing of "obtaining prior information of the calibration site" to obtain the point cloud data after removing the ceiling, ground, and surrounding walls. and Eliminating these interferences can greatly improve the calibration accuracy;

[0371] (3) Point cloud data and Use the calibration plate plane fitting method in "Obtaining prior information of the calibration site" to obtain the calibration plate plane, calibration ball spherical surface and other information in each point cloud, including its ID information. The ID of the calibration plate can be composed of its internal calibration code ID. For example, there are 6 calibration codes on it, and the IDs are ID i (i≤6) then the calibration board ID is the calibration code ID serially arranged from small to large, such as ID1ID2ID3ID4ID5ID6;

[0372] (4) According to the calibration plate ID information and the calibration ball ID, the initial value of the calibration parameter RT corresponding to the sixth coordinate transformation relationship is solved; further, according to the calibration plate fitting plane and the calibration ball spherical surface, based on the initial value of the calibration parameter RT, further refine it through NDT or ICP to obtain the final calibration parameter RT.

[0373] Traditional methods directly use point data for matching, such as NDT and various ICP improvement methods. Due to the large differences in the hardware parameters of the main / auxiliary lidar, the point cloud resolution, projection angle, etc. may cause very few true points of the same name in space. The points in two frames of laser point cloud data cannot represent the same position in space. Therefore, using the point-to-point distance as the error equation is bound to introduce random errors. In this application, by fitting the planes of each calibration plate, the above influence can be effectively reduced, that is, the calibration accuracy can be improved.

[0374] In one embodiment, the calibration device includes a plurality of calibration components. In the following embodiments, the calibration components are taken as calibration balls as an example for explanation (eg Fig.16 As shown, Fig.16 The calibration prior information is determined according to the first calibration data, including:

[0375] Performing spatial clustering segmentation on the first calibration data according to the position of each verification component in the calibration system to obtain third local point cloud data corresponding to each verification component;

[0376] The fifth identity information and the fifth coordinate information corresponding to each verification component are determined according to the third local point cloud data.

[0377] It should be noted that the fifth identity information is the identity information of the verification component. For example, when the verification component is a verification ball, the fifth identity information may be the radius information of the verification ball. The fifth coordinate information represents the position of the verification component in the calibration system, such as the spatial position.

[0378] Specifically, let P be the first calibration data after Φ removes the point clouds of the ceiling, the ground and the surrounding walls, and only includes the point clouds of each calibration component and the point cloud of the ground verification ball.

[0379] For the calibration ball, the specific steps are as follows:

[0380] (1) Since the installation height of the calibration ball is lower than the calibration component, the point cloud P can be divided into the point cloud containing only the calibration ball according to the height z coordinate. and point clouds containing only the calibrated components

[0381] (2) Clustering based on the spatial interval of the calibration balls. Perform spatial clustering segmentation to obtain the local point cloud of each verification ball (M1 is the total number of calibration balls); for each Through point cloud spherical fitting, the three-dimensional spatial coordinates of the center of the calibration ball (x j ,y j ,z j ) and the radius of the ball Rj information.

[0382] In one embodiment, after calculating the calibration parameters of each sensor according to the calibration prior information and the second data to calibrate each sensor, the method further includes:

[0383] The calibration parameters of each sensor are verified according to the verification component.

[0384] Since a multi-level perception system requires a large number of sensors, the correctness of the calibration results will directly affect the performance of the perception system. This application proposes a method for automatic calibration parameter verification, specifically, for dual-target calibration, camera and main lidar radar calibration, main / auxiliary lidar calibration, millimeter-wave radar and main lidar calibration result verification, the basic idea is:

[0385] For a sphere with known spatial position and size in the calibration system, identify the three-dimensional coordinates of its center in different coordinate systems and compare them with the true value. The specific steps are as follows:

[0386] In one embodiment, when the calibration parameters of the sensor include a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of each sensor are verified according to the verification component, including:

[0387] Obtaining the seventh three-dimensional coordinates of the verification component within the field of view of the main laser radar;

[0388] performing coordinate transformation on the seventh three-dimensional coordinate according to the third coordinate transformation relationship to obtain the seventh three-dimensional coordinate to be compared;

[0389] comparing the seventh three-dimensional coordinate to be compared with the seventh reference three-dimensional coordinate to determine whether the third coordinate transformation relationship is accurate;

[0390] The first reference three-dimensional coordinates are the three-dimensional coordinates obtained by the total station and corresponding to the verification component within the field of view of the main laser radar.

[0391] This example aims to verify the calibration accuracy of the main laser radar and the total station coordinate system. For the laser radar, the three-dimensional coordinates of the center of the sphere are obtained based on the reflectivity difference and spatial clustering through the known sphere ROI area. (i.e., the seventh three-dimensional coordinate); according to the external parameter R / t relationship between the main laser radar and the total station (i.e., the third coordinate transformation relationship), Perform coordinate transformation to obtain (that is, the seventh three-dimensional coordinate to be compared); Compare with the true value (the coordinates obtained by the total station, that is, the seventh reference three-dimensional coordinates) to determine the R / T error.

[0392] It should be noted that the seventh three-dimensional coordinate is the three-dimensional coordinate of the verification component within the field of view of the main laser radar. The set of verification components within the field of view of the main laser radar is a subset of the set of all verification components. Correspondingly, the seventh three-dimensional coordinate is a part of all the fifth coordinate information.

[0393] In one embodiment, when the calibration parameters of the sensor include a fourth coordinate transformation relationship between a camera coordinate system to be calibrated and a main laser radar coordinate system, the calibration parameters of each sensor are verified according to the verification component, including:

[0394] Acquire first verification point cloud data of the verification component within the field of view of the main laser radar;

[0395] Back-projecting the first verification point cloud data onto the camera image according to the fourth coordinate transformation relationship;

[0396] Whether the fourth coordinate transformation relationship is accurate is determined based on the back-projected camera image.

[0397] This embodiment is intended to verify the calibration parameter error between the camera and the main laser radar. In the embodiment, first obtain the first verification point cloud data of the verification component within the field of view of the main laser radar. Next, according to the fourth coordinate transformation relationship, the first verification point cloud data is back-projected onto the camera image. Through the back-projection operation, the three-dimensional coordinates of the verification point cloud are mapped to the two-dimensional image of the camera. Finally, based on the back-projected camera image, determine whether the fourth coordinate transformation relationship is accurate. By comparing the position of the verification point on the camera image and the position of the actual verification component, the accuracy of the fourth coordinate transformation relationship can be evaluated.

[0398] When the verification component is a verification ball, the first verification point cloud data of the verification component within the field of view of the main laser radar is transformed according to the external reference R corresponding to the fourth coordinate transformation relationship. c-l ,t c-l Back-project it onto each camera image; back-project the point cloud of the spherical area in each camera as follows Fig.17 As shown, the sphere is a gray prototype on the image, and the dotted line represents the back projection of the laser radar point cloud. Through grayscale and binarization processing, all circular edges in the image are extracted, and the first circular fitting equation O is obtained. i (i∈n, n is the number of circular regions in the image); according to the spatial distance difference, the point cloud data inside the sphere is obtained by clustering, and the endpoints of each horizontal line are obtained, such as Fig.18 As shown, the horizontal line is broken and fits the circle Q i (like Fig.19 Calculate O i With Q i The distance between the camera and the main lidar is further determined by the camera internal parameters and the external parameters R c-l ,t c-l(the accuracy of the fourth coordinate transformation relationship).

[0399] The purpose of this calibration method is to ensure that the coordinate transformation relationship between the camera to be calibrated and the main lidar is accurate. Through calibration, the accuracy of the calibration parameters can be verified, and the reliability and accuracy of sensor calibration can be improved.

[0400] In one embodiment, when the calibration parameters of the sensor include a fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the verification component, including:

[0401] Obtain the millimeter-wave radar coordinates and angles of each calibration ball within the field of view of the millimeter-wave radar;

[0402] Obtain the seventh three-dimensional coordinate within the field of view of the main laser radar;

[0403] Projecting the millimeter wave radar coordinates and angles onto the two-dimensional plane of the main laser radar coordinate system according to the fifth coordinate transformation relationship;

[0404] The Euclidean distance is calculated based on the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates to determine whether the fifth coordinate transformation relationship is accurate.

[0405] The purpose of this embodiment is to verify the fifth coordinate transformation relationship between the auxiliary laser radar l1 coordinate system and the main laser radar l2 coordinate system. The verification process includes the following steps: obtain the millimeter wave radar coordinates and angles of each calibration ball within the field of view of the millimeter wave radar. Obtain the seventh three-dimensional coordinates within the field of view of the main laser radar. Project the millimeter wave radar coordinates and angles to the two-dimensional plane of the main laser radar coordinate system according to the fifth coordinate transformation relationship. Calculate the Euclidean distance based on the projected millimeter wave radar coordinates and angles and the seventh three-dimensional coordinates. Determine whether the fifth coordinate transformation relationship is accurate based on the calculated Euclidean distance.

[0406] Specifically, the millimeter-wave radar coordinates and yaw angle, and the three-dimensional coordinates of the center of the sphere in the main lidar coordinate system are extracted respectively; through the fifth coordinate transformation relationship, the coordinates of the center of the sphere in the millimeter-wave data are projected onto the XY plane of the main lidar coordinate system; the Euclidean distance is calculated to determine the calibration accuracy.

[0407] Through the above verification process, the accuracy of the fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system can be determined. This can ensure the accuracy and reliability of the sensor's calibration parameters, thereby improving the application performance and accuracy of the sensor in the vehicle.

[0408] In one embodiment, when the calibration parameters of the sensor include a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the verification component, including:

[0409] Acquire first verification point cloud data of the verification component within the field of view of the main laser radar;

[0410] Acquire second verification point cloud data of the verification component within the field of view of the auxiliary laser radar;

[0411] The second verification point cloud data is transformed into the main laser radar coordinate system according to the sixth coordinate transformation relationship to obtain the second verification point cloud data to be verified;

[0412] Determine whether the sixth coordinate transformation relationship is accurate based on the first verification point cloud data and the second verification point cloud data to be verified.

[0413] The present embodiment aims to verify the sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system. Specifically, first, obtain the first verification point cloud data of the verification component within the field of view of the main laser radar. Next, obtain the second verification point cloud data of the verification component within the field of view of the auxiliary laser radar. Then, according to the sixth coordinate transformation relationship, the obtained second verification point cloud data is coordinate-converted and transformed into the main laser radar coordinate system. This transformation process usually involves rotation, translation, and possible scale changes. Finally, by comparing the first verification point cloud data with the second verification point cloud data to be verified, the accuracy of the sixth coordinate transformation relationship can be evaluated. If the difference between the two sets of point cloud data is small, it means that the sixth coordinate transformation relationship is accurate; if the difference is large, the sixth coordinate transformation relationship needs to be adjusted or recalculated.

[0414] Specifically, for the two laser radar point clouds, the spherical point clouds within their respective fields of view are extracted according to the differences in spatial distance and emissivity; (Fifth coordinate transformation relationship), transform the spherical point cloud in l2 to the l1 coordinate system; calculate the Euclidean distance and determine the accuracy of the fifth coordinate transformation relationship.

[0415] In summary, this embodiment is used to verify the accuracy of the sixth coordinate transformation relationship between the auxiliary laser radar and the main laser radar in the sensor calibration parameters. This method can help ensure that the sensor calibration parameters can accurately reflect the spatial relative position relationship between sensors in the actual scene.

[0416] In addition, in order to improve the calibration efficiency and accuracy, it is necessary to realize the rapid positioning of the vehicle to be calibrated, that is, to quickly drive into the designated position in the calibration system. This application uses the four contact points of the vehicle for positioning. Positioning includes front and rear and left and right. Figure 20-22 As shown in the figure, two tracks are installed in the calibration system, and the track width is the same as the width of the vehicle tires; the vehicle tires are parallel to the track and run on the track; there is a limiter at the front end of the track, and when the tires touch the limiter, the car can stop. After stopping, the rear wheel limiter needs to be fixed. Fixing the rear wheel is for safety considerations on the one hand. On the other hand, it is to calibrate the coordinate system of the millimeter wave radar and the laser radar.

[0417] This operation ensures that the initial positions of all vehicles to be calibrated are close, and the deviation between the main lidar coordinate system and the total station coordinate system is as small as possible. During the point cloud matching process, the deviation between the two point cloud coordinate systems is as small as possible, the iterative algorithm converges faster, and the accuracy is easier to ensure.

[0418] In addition, in order to assist the calibration of the millimeter-wave radar and the main lidar, a transmission belt device is installed on the track to achieve short-distance uniform linear motion of the vehicle.

[0419] In a second aspect, the present application provides a vehicle sensor calibration system, such as Fig.23 As shown, Fig.23 A structural block diagram of a vehicle sensor calibration system provided in this application, the system includes:

[0420] A first receiving unit 91 is used to receive first data and determine calibration prior information according to the first data; wherein the first data is point cloud data obtained by the total station scanning each calibration device in the calibration system, and the calibration prior information is data information of the calibration device;

[0421] A second receiving unit 92 is used to obtain second data sent by each sensor disposed on the vehicle to be calibrated, where the second data is data collected by each sensor on the calibration device;

[0422] The sensor calibration unit 93 is used to determine the calibration parameters of each sensor according to the calibration prior information and the second data, and the calibration parameters are used to characterize the coordinate transformation relationship between each sensor or between each sensor and the total station. For the introduction of the vehicle sensor calibration system, please refer to the above embodiment, and this application will not repeat it here.

[0423] In a second aspect, the present application provides a vehicle sensor calibration device, such as Fig.24 As shown, Fig.24 This is a structural block diagram of a vehicle sensor calibration device provided in this application, the device comprising:

[0424] Memory 201, used for storing computer programs;

[0425] The processor 202 is used to implement the steps of the vehicle sensor calibration method as described above when storing the computer program. Please refer to the above embodiment for the introduction of the vehicle sensor calibration device, and this application will not repeat it here.

[0426] In a second aspect, the present application provides a computer-readable storage medium 210, such as Fig.25 As shown, Fig.25 This is a structural block diagram of a computer-readable storage medium provided by the present application. The computer-readable storage medium 210 stores a computer program 211. When the computer program 211 is executed by the processor 202, the steps of the vehicle sensor calibration method described above are implemented. For the introduction of the computer-readable storage medium 210, please refer to the above embodiment, and this application will not repeat it here.

[0427] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0428] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle sensor calibration method, characterized in that: include: Receive first data, and determine calibration prior information according to the first data; wherein the first data is point cloud data obtained by a total station scanning each calibration device in a calibration system, and the calibration prior information is data information of the calibration device; Receiving second data sent by each sensor disposed on the vehicle to be calibrated, wherein the second data is data collected by each sensor on the calibration device; The calibration parameters of each of the sensors are determined according to the calibration priori information and the second data, and the calibration parameters are used to characterize the coordinate transformation relationship between the sensors or between each of the sensors and the total station.

2. The vehicle sensor calibration method according to claim 1, characterized in that: Receiving first data, and determining calibration prior information according to the first data, including: Receiving the first data, removing data that does not meet preset requirements from the first data, and obtaining first calibration data; The calibration priori information is determined according to the first calibration data.

3. The vehicle sensor calibration method according to claim 2, characterized in that: Eliminating data that does not meet preset requirements from the first data to obtain first calibration data includes: The point cloud data not including the calibration device is eliminated from the first data to obtain the first calibration data.

4. The vehicle sensor calibration method according to claim 3, characterized in that: Eliminating the point cloud data not including the calibration device from the first data to obtain the first calibration data includes: The point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system are identified to eliminate the point cloud data corresponding to the ceiling plane and / or the ground plane and / or the surrounding wall planes in the first data.

5. The vehicle sensor calibration method according to claim 4, characterized in that: The process of identifying the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system includes: Acquire a first distance from the center position of the total station to the ceiling plane and / or the ground plane and / or the surrounding wall planes; Randomly select at least three first point clouds from the first data, and calculate the three-dimensional space plane where the first point clouds are located; wherein the difference between the distance from the first point cloud to the center position of the total station and the first distance is not greater than a first preset distance; Calculate the second distance from each point cloud in the first calibration data to the three-dimensional space plane, and use the data corresponding to the point cloud whose difference between the second distance and the first distance is not greater than the second preset distance as the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane.

6. The vehicle sensor calibration method according to claim 5, characterized in that: Also includes: Repeat the step of identifying the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system; Determining whether the difference meets the iteration termination condition according to the difference between the number of point clouds in the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration process and the number of point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the previous iteration process; If it meets the requirements, the iteration is terminated, and the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration process are used as the final point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes.

7. The vehicle sensor calibration method according to claim 2, characterized in that: The calibration device includes a plurality of calibration components, and determining the calibration prior information according to the first calibration data includes: Performing spatial clustering segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain first local point cloud data corresponding to each calibration component; The first identity information and the first coordinate information of each of the calibration components are determined according to the first local point cloud data.

8. The vehicle sensor calibration method according to claim 7, characterized in that: Each of the calibration components includes at least two calibration plates, and determining first identity information and first coordinate information of each of the calibration components according to the first local point cloud data includes: Determine second local point cloud data corresponding to each calibration plate from the first local point cloud data; The second identity information and the second coordinate information corresponding to each of the calibration plates are determined according to the second local point cloud data.

9. The vehicle sensor calibration method according to claim 8, characterized in that: Determining second local point cloud data corresponding to each calibration plate from the first local point cloud data includes: Plane fitting is performed on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates in the first local point cloud data.

10. The vehicle sensor calibration method according to claim 8, characterized in that: Determining the second identity information and the second coordinate information corresponding to each of the calibration plates according to the second local point cloud data includes: Converting the second local point cloud data into a grayscale image; Identify the grayscale image to determine the second identity information of the calibration plate; Recognize the grayscale image to determine the first image coordinates of each key point on the calibration plate on the calibration plate; The first three-dimensional coordinates corresponding to each of the key points are determined according to the first image coordinates corresponding to each of the key points and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

11. The vehicle sensor calibration method according to claim 10, characterized in that: After converting the second local point cloud data into a grayscale image, the method further includes: Perform filtering processing on the grayscale image.

12. The vehicle sensor calibration method according to claim 10, characterized in that: Acquiring second data sent by each sensor disposed on the vehicle to be calibrated, including: Obtain a first calibration image sent by the camera to be calibrated; Calculating calibration parameters of each of the sensors according to the calibration priori information and the second data to calibrate each of the sensors includes: Determining, according to the first calibration image, third identity information of the calibration plate and second image coordinates of key points included in the first calibration image, wherein the second image coordinates are used to represent positions of the key points on the calibration plate; Determine the second three-dimensional coordinates corresponding to each of the key points in the first calibration image according to the third identity information, the second image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data; The intrinsic parameters of the camera to be calibrated are determined according to the second image coordinates and the second three-dimensional coordinates corresponding to each of the key points in the first calibration image.

13. The vehicle sensor calibration method according to claim 12, characterized in that: After determining the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates corresponding to each of the key points in the first calibration image, the method further includes: Determine the extrinsic parameters of the camera to be calibrated according to the second image coordinates, the second three-dimensional coordinates and the intrinsic parameters of the camera to be calibrated; The external parameters of the camera to be calibrated include at least a first coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

14. The vehicle sensor calibration method according to claim 13, characterized in that: Acquiring second data sent by each sensor disposed on the vehicle to be calibrated, including: Obtaining a second calibration image sent by each surround view camera to be calibrated; Projecting the second calibration image using the surround stitching homography matrix to obtain a projected image; A second coordinate transformation relationship between the coordinate systems of two adjacent surround-view cameras to be calibrated is calculated according to the same key points in the projection images corresponding to the two adjacent surround-view cameras to be calibrated.

15. The vehicle sensor calibration method according to claim 14, characterized in that: After obtaining the second calibration image sent by each surround view camera to be calibrated, the method further includes: The second calibration image is calibrated for distortion according to the internal parameters of the camera to be calibrated to obtain a calibrated second calibration image.

16. The vehicle sensor calibration method according to claim 14, characterized in that: After obtaining the second calibration image sent by each surround view camera to be calibrated, the method further includes: Extracting third image coordinates of each of the key points in the second calibration image on the calibration plate and fourth identity information of the calibration plate; Filtering key points located on the ground in the second calibration image according to the third image coordinates and the fourth identity information, and obtaining third three-dimensional coordinates of each of the key points located on the ground; Determine a surround stitching image according to the third three-dimensional coordinates of each of the key points located on the ground; Calibrate the fourth three-dimensional coordinates of the key points in the surround view stitched image using the first reference three-dimensional coordinates corresponding to the key points in the surround view stitched image in the calibration prior information to obtain a calibrated surround view stitched image; The surround view stitching homography matrix is ​​obtained according to the calibrated surround view stitching image.

17. The vehicle sensor calibration method according to claim 13, characterized in that: Acquiring second data sent by each sensor disposed on the vehicle to be calibrated, including: Obtain the first test point cloud data sent by the main laser radar; The first test point cloud data is matched with the first data to obtain a third coordinate transformation relationship between the main lidar coordinate system and the total station coordinate system.

18. The vehicle sensor calibration method according to claim 17, characterized in that: After obtaining the first test point cloud data sent by the main laser radar, it also includes: Matching the first test point cloud data with the first data to obtain key points within the field of view of the main laser radar and fifth three-dimensional coordinates corresponding to the key points; Determine the same key points in the first calibration image as those within the field of view of the primary laser radar according to the second three-dimensional coordinates and the fifth three-dimensional coordinates, and determine the three-dimensional coordinates of the same key points; The fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main lidar coordinate system is calculated according to the three-dimensional coordinates of the same key point.

19. The vehicle sensor calibration method according to claim 18, characterized in that: The calibration plate is provided with a metal block, and the second data sent by each sensor provided on the vehicle to be calibrated is obtained, including: Acquire the collected data of the metal block sent by the millimeter wave radar; Determine alignment data between the first test point cloud data and the collected data according to the data timestamp; Projecting the sixth three-dimensional coordinate of the metal block in the first test point cloud data in the alignment data onto a horizontal plane to obtain horizontal plane projection data; The fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main lidar coordinate system is calculated according to the horizontal plane projection data, the coordinates of the metal blocks in the collected data in the alignment data, and the positions of each of the metal blocks in the calibration system.

20. The vehicle sensor calibration method according to claim 18, characterized in that: Acquiring second data sent by each sensor disposed on the vehicle to be calibrated, including: Obtain the second test point cloud data sent by the auxiliary laser radar; The second test point cloud data is matched with the first test point cloud data to obtain a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system.

21. The vehicle sensor calibration method according to any one of claims 2 to 20, characterized in that: The calibration device includes a plurality of verification components, and determining the calibration prior information according to the first calibration data includes: Performing spatial clustering segmentation on the first calibration data according to the position of each of the verification components in the calibration system to obtain third local point cloud data corresponding to each of the verification components; The fifth identity information and the fifth coordinate information corresponding to each of the verification components are determined according to the third local point cloud data.

22. The vehicle sensor calibration method according to claim 21, characterized in that: After calculating the calibration parameters of each of the sensors according to the calibration priori information and the second data to calibrate each of the sensors, the method further includes: The calibration parameters of each of the sensors are verified according to the verification component.

23. The vehicle sensor calibration method according to claim 22, characterized in that: When the calibration parameters of the sensors include a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of each sensor are verified according to the verification component, including: Obtaining the seventh three-dimensional coordinates of the verification component within the field of view of the main laser radar; Performing coordinate transformation on the seventh three-dimensional coordinate according to the third coordinate transformation relationship to obtain a seventh three-dimensional coordinate to be compared; Comparing the seventh three-dimensional coordinate to be compared with the seventh reference three-dimensional coordinate to determine whether the third coordinate transformation relationship is accurate; The first reference three-dimensional coordinates are the three-dimensional coordinates obtained by the total station and corresponding to the verification component within the field of view of the main laser radar.

24. The vehicle sensor calibration method according to claim 22, characterized in that: When the calibration parameters of the sensor include a fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the verification component, including: Acquire first verification point cloud data of the verification component within the field of view of the main laser radar; Back-projecting the first verification point cloud data onto the camera image according to the fourth coordinate transformation relationship; It is determined whether the fourth coordinate transformation relationship is accurate based on the back-projected camera image.

25. The vehicle sensor calibration method according to claim 22, characterized in that: When the calibration parameters of the sensor include a fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the verification component, including: Obtaining the millimeter-wave radar coordinates and angles of each calibration ball within the field of view of the millimeter-wave radar; Acquire a seventh three-dimensional coordinate within the field of view of the main laser radar; Projecting the millimeter wave radar coordinates and angles onto a two-dimensional plane of the main laser radar coordinate system according to the fifth coordinate transformation relationship; The Euclidean distance is calculated according to the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates to determine whether the fifth coordinate transformation relationship is accurate.

26. The vehicle sensor calibration method according to claim 22, characterized in that: When the calibration parameters of the sensor include a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the verification component, including: Acquire first verification point cloud data of the verification component within the field of view of the main laser radar; Acquire second verification point cloud data of the verification component within the field of view of the auxiliary laser radar; The second verification point cloud data is transformed into the main laser radar coordinate system according to the sixth coordinate transformation relationship to obtain the second verification point cloud data to be verified; Determine whether the sixth coordinate transformation relationship is accurate based on the first verification point cloud data and the second verification point cloud data to be verified.

27. A vehicle sensor calibration system, characterized in that: include: A first receiving unit is used to receive first data and determine calibration prior information according to the first data; wherein the first data is point cloud data obtained by scanning each calibration device in the calibration system by the total station, and the calibration prior information is data information of the calibration device; A second receiving unit, used for acquiring second data sent by each sensor disposed on the vehicle to be calibrated, wherein the second data is data collected by each sensor on the calibration device; A sensor calibration unit is used to determine calibration parameters of each of the sensors according to the calibration prior information and the second data, wherein the calibration parameters are used to characterize the coordinate transformation relationship between the sensors or between each of the sensors and the total station.

28. A vehicle sensor calibration device, characterized in that: include: Memory for storing computer programs; A processor, for implementing the steps of the vehicle sensor calibration method as described in any one of claims 1 to 26 when storing a computer program.

29. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the vehicle sensor calibration method according to any one of claims 1 to 26 are implemented.

Images