Vehicle sensor calibration method, system, device and medium based on calibration system

Abstract

The present application discloses a vehicle sensor calibration method, system, device and medium based on a calibration system, which relates to the field of autonomous driving and solves the problem of low efficiency in vehicle sensor calibration. In this solution, when the vehicle to be calibrated meets the first calibration condition, the total station is controlled to scan the calibration device to obtain the first data; when the second calibration condition is met, the various sensors on the vehicle to be calibrated are controlled to collect the calibration device to obtain the second data; calibration prior information is obtained based on the first data, and the various sensors on the vehicle to be calibrated are calibrated based on the calibration prior information and the second data. It can be seen that the use of a total station and multiple calibration devices in this application can greatly reduce the personnel requirements and time consumption of the calibration process. This can reduce the calibration cost, facilitate mass production, and improve production efficiency.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a vehicle sensor calibration method, system, device and medium based on a calibration system. Background Art

[0002] In the field of autonomous driving, the perception system is known as the eyes of autonomous driving. Its primary task is to acquire environmental information and identify its state. Due to the complexity of ambient lighting, weather conditions, and scenarios, the use of a single sensor has certain limitations. Therefore, multi-sensor fusion is often used to improve information redundancy and complementarity to ensure the safety of autonomous driving. Common perception system sensors include lidar, cameras, and millimeter-wave radar. Different perception tasks and sensor characteristics require an effective combination of sensors. To ensure accurate environmental perception, precise calibration of the vehicle's sensors is often required to determine the internal and external parameters between sensors. However, existing sensor calibration methods have some problems in engineering applications, such as requiring a large amount of time and human resources, and being difficult to achieve in mass production. Summary of the Invention

[0003] The purpose of this application is to provide a vehicle sensor calibration method, system, device, and medium based on a calibration system. Using a total station and multiple calibration devices can significantly reduce the personnel required and time consumed during the calibration process. This can reduce calibration costs, facilitate mass production, and improve production efficiency.

[0004] In a first aspect, the present application provides a vehicle sensor calibration method based on a calibration system, wherein the calibration system includes a total station and multiple calibration devices, including:

[0005] When the vehicle to be calibrated meets a first calibration condition, controlling the total station to scan the calibration device to obtain first data;

[0006] When the second calibration condition is met, receiving second data acquired by the calibration device through each of the sensors on the vehicle to be calibrated;

[0007] Calibration prior information is obtained based on the first data, and each sensor on the vehicle to be calibrated is calibrated based on the calibration prior information and the second data to obtain calibration parameters of each sensor; the calibration prior information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two sensors or between the sensor and the total station.

[0008] In one embodiment, the process of determining whether the vehicle to be calibrated meets the first calibration condition includes:

[0009] Determining whether the vehicle to be calibrated is parked in a preset area;

[0010] If the vehicle stops in the preset area, it is determined that the vehicle to be calibrated meets the first calibration condition; otherwise, it is determined that the first calibration condition is not met.

[0011] In one embodiment, the process of determining whether the vehicle to be calibrated meets the first calibration condition includes:

[0012] Determining whether the vehicle to be calibrated is being calibrated for the first time based on the vehicle information of the vehicle to be calibrated;

[0013] If it is the first calibration, it is determined that the vehicle to be calibrated meets the first calibration condition; otherwise, it is determined that the first calibration condition is not met.

[0014] In one embodiment, when determining that the vehicle is not calibrated for the first time, the method further includes:

[0015] The first data corresponding to the vehicle to be calibrated stored in the database is called.

[0016] In one embodiment, the process of determining whether the second calibration condition is met includes:

[0017] Determine whether a calibration start instruction is received;

[0018] If received, it is determined that the second calibration condition is met; otherwise, it is determined that the second calibration condition is not met.

[0019] In one embodiment, obtaining calibration prior information according to the first data includes:

[0020] Eliminating data that does not meet preset requirements from the first data to obtain first calibration data;

[0021] The calibration prior information is determined according to the first calibration data.

[0022] In one embodiment, removing data that does not meet preset requirements from the first data to obtain first calibration data includes:

[0023] The point cloud data not including the calibration device is eliminated from the first data to obtain the first calibration data.

[0024] In one embodiment, the calibration device includes a plurality of calibration components, and determining the calibration prior information according to the first calibration data includes:

[0025] Performing spatial clustering and segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain first local point cloud data corresponding to each calibration component;

[0026] The first identity information and the first coordinate information of each of the calibration components are determined according to the first local point cloud data.

[0027] In one embodiment, each of the calibration components includes at least two calibration plates, and determining the first identity information and first coordinate information of each of the calibration components based on the obtained first local point cloud data includes:

[0028] Determine second local point cloud data corresponding to each calibration plate from the first local point cloud data;

[0029] The second identity information and the second coordinate information corresponding to each of the calibration plates are determined according to the second local point cloud data.

[0030] In one embodiment, each calibration member includes at least two non-coplanar calibration plates, and determining the second local point cloud data corresponding to each calibration plate according to the position of each calibration plate in each calibration member and the first local point cloud data includes:

[0031] Plane fitting is performed on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates.

[0032] In one embodiment, performing plane fitting on each calibration plate to extract second local point cloud data corresponding to each calibration plate includes:

[0033] Obtaining a first height between the center of each calibration plate and the ground, randomly selecting at least three second point clouds from the first local point cloud, and calculating a second three-dimensional space plane where the second point clouds are located;

[0034] Calculating a third distance between each point cloud in the second local point cloud data and the second three-dimensional space plane;

[0035] The data corresponding to the point cloud in which the difference between the third distance and the first height is not greater than the third preset distance is used as the second local point cloud data corresponding to the calibration plate.

[0036] In one embodiment, the calibration plate includes a calibration carrier plate and a plurality of marking codes, wherein the plurality of marking codes are fixed on the calibration carrier plate, the marking codes carrying QR code information for identification by the sensor, wherein the identity information corresponding to the QR code information of all the marking codes is different;

[0037] Determining second identity information corresponding to each of the calibration plates according to the second local point cloud data includes:

[0038] Converting the second local point cloud data into a grayscale image;

[0039] Segmenting the grayscale image according to the intervals between the marker codes to obtain a partial image of each marker code;

[0040] Perform two-dimensional code recognition on each of the partial images to determine the first ID information of each of the marking codes, and determine the second identity information of the calibration plate according to the first ID information of each of the marking codes on each of the marking plates.

[0041] In one embodiment, determining the second coordinate information corresponding to each of the calibration plates according to the second local point cloud data includes:

[0042] Extracting four sides of the circumscribed rectangle of each partial image of the logo code, and determining the intersection of two adjacent sides as a key point;

[0043] Acquire a first image coordinate of each of the key points, where the first image coordinate represents a position of the key point on the calibration plate;

[0044] The first three-dimensional coordinates corresponding to each key point are determined according to the position of the marker code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0045] In one embodiment, when there is no three-dimensional coordinate corresponding to the first image coordinate of the key point in the first reference three-dimensional coordinates, the method further includes:

[0046] Obtaining three-dimensional coordinates corresponding to several reference points adjacent to the key point;

[0047] The first three-dimensional coordinate corresponding to the key point is determined according to the three-dimensional coordinates of the plurality of reference points.

[0048] In one embodiment, a metal block is further provided on the calibration plate, and determining the second identity information corresponding to each calibration plate according to the second local point cloud data further includes:

[0049] The third identity information of the metal block is determined according to the first ID information of each identification code.

[0050] In one embodiment, determining the second coordinate information corresponding to each of the calibration plates according to the second local point cloud data further includes:

[0051] The three-dimensional coordinates of the metal block are determined according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to the marking codes.

[0052] In one embodiment, the metal block is fixed at the center of the calibration carrier, a plurality of the marking codes are fixed on the calibration carrier and surround the metal block, and a plurality of the marking codes are connected to the vertices of the metal block at vertices close to the direction of the metal block.

[0053] In one embodiment, determining the three-dimensional coordinates of the metal block according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to the marking codes includes:

[0054] Determine the three-dimensional coordinates of the four corner points of the metal block according to the first three-dimensional coordinates of the key points corresponding to the vertices of each of the marking codes close to the metal block;

[0055] The three-dimensional coordinates of the center of the metal block are determined according to the three-dimensional coordinates of the four corner points of the metal block.

[0056] In one embodiment, when the sensor includes a camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0057] Controlling the camera to be calibrated to collect data from the calibration device to obtain a first calibration image;

[0058] Identify the marker codes in the first calibration image and determine the second ID information and the partial image of each marker code in the first calibration image;

[0059] determining second image coordinates of each key point included in the first calibration image according to the ID information of each marker code and the partial image in the first calibration image;

[0060] Determining the second three-dimensional coordinates corresponding to each key point in the first calibration image according to the ID information of each marker code in the first calibration image, the second image coordinates of each key point, and the first reference three-dimensional coordinates;

[0061] Determine the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates.

[0062] In one embodiment, after determining the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates, the method further includes:

[0063] Determine the extrinsic parameters of the camera to be calibrated according to the second image coordinates, the second three-dimensional coordinates, and the intrinsic parameters of the camera to be calibrated;

[0064] The external parameters of the camera to be calibrated include at least a first coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

[0065] In one embodiment, when the sensor includes a surround view camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0066] Controlling the surround-view camera to be calibrated to collect data from the calibration device to obtain a second calibration image;

[0067] Projecting the second calibration image using the surround stitching homography matrix to obtain a projected image;

[0068] A second coordinate transformation relationship between the coordinate systems of two adjacent surround-view cameras to be calibrated is calculated according to the same key points in the projection images corresponding to the two adjacent surround-view cameras to be calibrated.

[0069] In one embodiment, after controlling the surround-view camera to be calibrated to collect data from the calibration device to obtain a second calibration image, the method further includes:

[0070] Extracting the ID information of each marker code and the third image coordinates of each key point in the second calibration image;

[0071] Filtering key points on the ground in the second calibration image according to the ID information of each marker code in the second calibration image and the third image coordinates of each key point, and obtaining third three-dimensional coordinates of each key point on the ground;

[0072] determining a surround stitching image according to the third three-dimensional coordinates of each of the key points located on the ground;

[0073] calibrating the fourth three-dimensional coordinates of the key points in the surround view stitched image using the first reference three-dimensional coordinates corresponding to the key points in the surround view stitched image in the calibration prior information to obtain a calibrated surround view stitched image;

[0074] The surround view stitching homography matrix is ​​obtained according to the calibrated surround view stitching image.

[0075] In one embodiment, when the sensor includes a main laser radar, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0076] Controlling the main laser radar to collect data from the calibration device to obtain first test point cloud data;

[0077] The first test point cloud data is matched with the first data to obtain a third coordinate transformation relationship between the main lidar coordinate system and the total station coordinate system.

[0078] In one embodiment, after controlling the main laser radar to collect data from the calibration device to obtain first test point cloud data, the method further includes:

[0079] Matching the first test point cloud data with the first data to obtain ID information of the marker code and the fifth three-dimensional coordinates of the key points within the field of view of the main laser radar;

[0080] Determine, based on the ID information of each marker code in the first calibration image, the second three-dimensional coordinates corresponding to each key point, the ID information of the marker code within the field of view of the main laser radar, and the fifth three-dimensional coordinates of the key point, the key point in the first calibration image that is identical to the key point within the field of view of the main laser radar, and determine the three-dimensional coordinates of the identical key point;

[0081] The fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main lidar coordinate system is calculated based on the three-dimensional coordinates of the same key point.

[0082] In one embodiment, when the sensor includes a millimeter-wave radar, a metal block is provided on the calibration plate, and the calibration system further includes two tracks and a conveyor belt device provided on the tracks, wherein the two tracks are parallel to each other and the distance between the two tracks is the same as the distance between the left and right wheels of the vehicle to be calibrated. During sensor calibration, the vehicle to be calibrated is positioned on the conveyor belt device;

[0083] Before controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain the second data, the method further includes:

[0084] Controlling the conveyor belt device to move the vehicle to be calibrated at a preset speed;

[0085] Controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes:

[0086] During the movement of the vehicle to be calibrated, triggering the millimeter-wave radar to collect the coordinates of the metal block within the field of view to obtain collected data, wherein the collected data at least includes the coordinates of the metal block;

[0087] Determining alignment data between the first test point cloud data and the collected data according to the data timestamp;

[0088] Projecting the sixth three-dimensional coordinate of the metal block in the first test point cloud data in the alignment data onto a horizontal plane to obtain horizontal plane projection data;

[0089] The fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main lidar coordinate system is calculated based on the horizontal plane projection data, the coordinates of the metal blocks in the collected data in the alignment data, and the positions of each metal block in the calibration system.

[0090] In one embodiment, when the sensor includes an auxiliary laser radar, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0091] controlling the auxiliary laser radar to collect data from the calibration device to obtain second test point cloud data;

[0092] The second test point cloud data is matched with the first test point cloud data to obtain a sixth coordinate transformation relationship between the auxiliary lidar coordinate system and the main lidar coordinate system.

[0093] In one embodiment, the calibration device includes a plurality of verification components, and determining the calibration prior information according to the first calibration data includes:

[0094] Performing spatial clustering and segmentation on the first calibration data according to the position of each verification component in the calibration system to obtain third local point cloud data corresponding to each verification component;

[0095] The fifth identity information and the fifth coordinate information corresponding to each of the verification components are determined according to the third local point cloud data.

[0096] In one embodiment, determining the fifth identity information and the fifth coordinate information corresponding to each of the verification components according to the third partial point cloud data includes:

[0097] The third local point cloud data is fitted to obtain structural information and three-dimensional coordinates of each verification component.

[0098] In one embodiment, the verification component is a verification sphere, and fitting the third local point cloud data to obtain structural information and three-dimensional coordinates of each verification component includes:

[0099] Spherical fitting is performed on the third local point cloud data to obtain radius information and three-dimensional coordinates of the calibration sphere.

[0100] In one embodiment, after obtaining calibration priori information based on the first data and calibrating each of the sensors on the vehicle to be calibrated based on the calibration priori information and the second data, the method further includes:

[0101] The calibration parameters of each of the sensors are verified according to the radius information and three-dimensional coordinates of each of the calibration spheres.

[0102] In one embodiment, when the calibration parameters of the sensor include a third coordinate transformation relationship between a main laser radar coordinate system and a total station coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including:

[0103] Obtaining the seventh three-dimensional coordinates of each calibration sphere within the field of view of the main laser radar;

[0104] performing coordinate transformation on the seventh three-dimensional coordinate according to the third coordinate transformation relationship to obtain a seventh three-dimensional coordinate to be compared;

[0105] comparing the seventh three-dimensional coordinate to be compared with the seventh reference three-dimensional coordinate to determine whether the third coordinate transformation relationship is accurate;

[0106] The first reference three-dimensional coordinates are the three-dimensional coordinates obtained by the total station and corresponding to the verification component within the field of view of the main laser radar.

[0107] In one embodiment, when the calibration parameters of the sensor include a fourth coordinate transformation relationship between a coordinate system of a camera to be calibrated and a main lidar coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including:

[0108] Acquire first calibration point cloud data of a calibration sphere within the field of view of the primary laser radar;

[0109] Back-projecting the first verification point cloud data onto the camera image according to the fourth coordinate transformation relationship;

[0110] Performing preset processing on the back-projected camera image to obtain a circular edge in the camera image, and determining a corresponding first circular fitting equation according to the circular edge;

[0111] The first circle fitting equation is compared with a reference circle fitting equation to determine whether the fourth coordinate transformation relationship is accurate.

[0112] In one embodiment, the calibration sphere is a metal sphere. When the calibration parameters of the sensor include a fifth coordinate transformation relationship between the millimeter-wave radar coordinate system and the main lidar coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including:

[0113] Obtaining the millimeter-wave radar coordinates and angles of each calibration sphere within the field of view of the millimeter-wave radar;

[0114] Obtaining a seventh three-dimensional coordinate within the field of view of the primary laser radar;

[0115] Projecting the millimeter-wave radar coordinates and angles onto a two-dimensional plane of the main lidar coordinate system according to the fifth coordinate transformation relationship;

[0116] The Euclidean distance is calculated based on the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates to determine whether the fifth coordinate transformation relationship is accurate.

[0117] In one embodiment, when the calibration parameters of the sensor include a sixth coordinate transformation relationship between the auxiliary lidar coordinate system and the main lidar coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including:

[0118] Acquire first calibration point cloud data of the calibration sphere within the field of view of the main laser radar;

[0119] Acquire the second calibration point cloud data of the calibration sphere within the auxiliary laser radar field of view;

[0120] transforming the second verification point cloud data into the main lidar coordinate system according to the sixth coordinate transformation relationship to obtain second verification point cloud data to be verified;

[0121] Determine whether the sixth coordinate transformation relationship is accurate based on the first verification point cloud data and the second verification point cloud data to be verified.

[0122] In a second aspect, the present application further provides a vehicle sensor calibration system based on a calibration system, the calibration system comprising a total station, a plurality of calibration devices, and a computing device, the computing device comprising:

[0123] a first control unit, configured to control the total station to scan the calibration device to obtain first data when the vehicle to be calibrated meets a first calibration condition;

[0124] a second control unit, configured to receive second data acquired by the calibration device from each of the sensors on the vehicle to be calibrated when a second calibration condition is met;

[0125] A calibration unit is used to obtain calibration prior information based on the first data, and calibrate each of the sensors on the vehicle to be calibrated based on the calibration prior information and the second data to obtain calibration parameters of each sensor; the calibration prior information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two sensors or between the sensor and the total station.

[0126] In a third aspect, the present application further provides a vehicle sensor calibration device based on a calibration system, comprising:

[0127] Memory for storing computer programs;

[0128] The processor is configured to implement the steps of the vehicle sensor calibration method based on the calibration system as described above when storing the computer program.

[0129] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the vehicle sensor calibration method based on the calibration system as described above.

[0130] The present application provides a vehicle sensor calibration method, system, device and medium based on a calibration system, which relates to the field of autonomous driving and solves the problem of low efficiency in vehicle sensor calibration. In this solution, when the vehicle to be calibrated meets the first calibration condition, the total station is controlled to scan the calibration device to obtain the first data; when the second calibration condition is met, the various sensors on the vehicle to be calibrated are controlled to collect the calibration device to obtain the second data; calibration prior information is obtained based on the first data, and the various sensors on the vehicle to be calibrated are calibrated based on the calibration prior information and the second data. It can be seen that the use of a total station and multiple calibration devices in this application can greatly reduce the personnel requirements and time consumption of the calibration process. This can reduce the calibration cost, facilitate mass production, and improve production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the prior art and the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0132] Figure 1 A flow chart of a vehicle sensor calibration method based on a calibration system provided in this application;

[0133] Figure 2 A schematic diagram of a three-dimensional diagram of a calibration component for forward splicing provided in this application;

[0134] Figure 3 A schematic side view of the calibration component for forward splicing provided in this application;

[0135] Figure 4 A schematic diagram of a three-dimensional view of a reverse-jointed calibration component provided in this application;

[0136] Figure 5 A schematic side view of the reverse-spliced ​​calibration component provided in this application;

[0137] Figure 6 A schematic diagram of a three-dimensional view of a hybrid spliced ​​calibration component provided in this application;

[0138] Figure 7 A schematic side view of a hybrid spliced ​​calibration component provided in this application;

[0139] Figure 8 Schematic diagram of the calibration board structure provided for this application;

[0140] Figure 9 A schematic diagram of a partial image of the four corners of the calibration code provided in this application;

[0141] Figure 10 A schematic diagram of establishing the world coordinate system provided for this application;

[0142] Figure 11 An example diagram of an existing calibration plate provided for this application;

[0143] Figure 12 Another example diagram of an existing calibration plate provided for this application;

[0144] Figure 13 A top view of the surround stitching calibration layout provided for this application;

[0145] Figure 14 Schematic diagram of the projection transformation matrix selection provided for this application;

[0146] Figure 15 Schematic diagram of the relationship between the millimeter wave radar coordinate system and the main lidar coordinate system provided for this application;

[0147] Figure 16 A schematic diagram of the layout of the calibration ball provided in this application;

[0148] Figure 17 Schematic diagram of projecting the point cloud provided in this application onto the camera image;

[0149] Figure 18 Schematic diagram of point cloud endpoint extraction provided for this application;

[0150] Figure 19 Schematic diagram of the point cloud point fitting circle provided by this application;

[0151] Figure 20 A schematic diagram of the vehicle parking area layout provided for this application;

[0152] Figure 21 A top view of the vehicle parking area layout provided for this application;

[0153] Figure 22 A side view of the vehicle parking area layout provided for this application;

[0154] Figure 23 A structural block diagram of a vehicle sensor calibration system based on a calibration system provided in this application;

[0155] Figure 24 This is a structural block diagram of a vehicle sensor calibration device based on a calibration system provided by this application;

[0156] Figure 25 This is a structural block diagram of a computer-readable storage medium provided in this application. DETAILED DESCRIPTION

[0157] The core of this application is to provide a vehicle sensor calibration method, system, device, and medium based on a calibration system. Using a total station and multiple calibration devices can significantly reduce the personnel required and time consumed during the calibration process. This can reduce calibration costs, facilitate mass production, and improve production efficiency.

[0158] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0159] Autonomous driving is developing rapidly, becoming a focal point of competition in the automotive and technology industries. Modular autonomous driving architecture design has become the mainstream approach due to its interpretability and ease of maintenance. The perception system is a key component of autonomous driving systems, acting as the vehicle's eyes and providing environmental status information for path planning. Due to the complex environment, existing perception systems typically utilize multi-sensor fusion to enhance information redundancy and complementarity to ensure safety. Commonly used sensors include lidar, cameras, and millimeter-wave radar, which require effective combinations based on the mission and sensor characteristics. To ensure accurate perception, precise sensor calibration is required to determine the relationship between internal and external parameters. However, existing calibration methods have challenges in engineering applications. They require manual intervention, are time-consuming and labor-intensive, and are difficult to mass-produce. The calculation process is lengthy, and initial value issues can lead to inaccurate results. Calibration results are also difficult to interpret, especially for non-experts.

[0160] Before describing the embodiments of the present application, first, in order to obtain the best calibration accuracy and improve the calibration automation rate, the three-dimensional coordinates of the center of the main laser radar relative to the vehicle tire landing point can be obtained based on the vehicle structural design parameters, and the position of the main laser mine center on the ground projection point can be further obtained. The total station is installed in the calibration site, and the center position of the total station is required to be consistent with the center of the main laser radar. The specific steps are as follows: Step 1: Align the plumb bob point at the center of the total station with the ground projection point of the main laser mine center; Step 2: Align the height of the total station center relative to the ground with the height of the main laser mine center relative to the ground; Step 3: Align the x-axis direction of the total station coordinate system with the x-axis direction of the main laser radar; Step 4: Adjust the total station to a horizontal level.

[0161] Please refer to Figure 1 , Figure 1 This is a flow chart of a vehicle sensor calibration method based on a calibration system provided in this application. The calibration system includes a total station and multiple calibration devices. The method includes:

[0162] S11: When the vehicle to be calibrated meets the first calibration condition, controlling the total station to scan the calibration device to obtain first data;

[0163] Specifically, before calibrating the vehicle sensor, it is first necessary to confirm whether the vehicle to be calibrated meets the first calibration condition. Once the condition is met, the system will control the total station to scan the calibration device to obtain the first batch of data. The total station is an instrument used to measure and map three-dimensional space, and point cloud data can be obtained by scanning the calibration device. The calibration system is repeatedly scanned by the total station to obtain dense point cloud data of the calibration site. These point cloud data contain specific information about the calibration device and can be used to determine the data information of the calibration device, that is, the calibration prior information. The calibration prior information is the reference data for sensor calibration and can help determine the internal and external parameters of the sensor.

[0164] S12: When the second calibration condition is met, receiving second data collected by the calibration device from various sensors on the vehicle to be calibrated;

[0165] In this step, after confirming that the vehicle to be calibrated meets the second calibration criteria, the system receives the second data acquired by the calibration device through the various sensors on the vehicle to be calibrated. This data can reflect the performance of the sensors in actual use scenarios and is used for subsequent sensor calibration.

[0166] Among them, the specific methods of receiving the second data may include the following two methods: first, directly controlling each of the sensors to collect the calibration device to obtain the second data; second, passively receiving the second data sent by the sensor.

[0167] S13: Obtaining calibration priori information based on the first data, and calibrating each sensor on the vehicle to be calibrated based on the calibration priori information and the second data to obtain calibration parameters of each sensor; the calibration priori information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two sensors or between a sensor and a total station.

[0168] In this step, after obtaining the first batch of data, the system generates calibration prior information based on this data. Simultaneously, the system calibrates each sensor on the vehicle being calibrated, combining the second batch of data with this prior calibration information. This prior calibration information, obtained through the total station's scanning calibration device, serves as a reference for sensor calibration. This calibration process corrects sensor measurement errors and improves sensor accuracy and reliability.

[0169] This embodiment receives calibration device data obtained from total station scans and sensor data, performs calculations and analysis to obtain sensor calibration parameters, ultimately achieving accurate sensor calibration. This improves the perception and safety of autonomous driving systems, providing effective support for the development of autonomous driving technology. Furthermore, the calibration method in this embodiment is automatically executed by a processor, eliminating the need for human intervention. This saves significant time and human resources, improves work efficiency, and is suitable for mass production.

[0170] In one embodiment, the process of determining whether the vehicle to be calibrated meets the first calibration condition includes:

[0171] Determine whether the vehicle to be calibrated has stopped in the preset area;

[0172] If the vehicle stops in the preset area, it is determined that the vehicle to be calibrated meets the first calibration condition; otherwise, it is determined that the first calibration condition is not met.

[0173] This embodiment describes a method for determining whether a vehicle to be calibrated meets a first calibration condition. The method includes the following steps: determining whether the vehicle to be calibrated is parked in a preset area; if so, determining that the vehicle to be calibrated meets the first calibration condition; otherwise, determining that the first calibration condition is not met.

[0174] This embodiment ensures that the vehicle to be calibrated meets preset conditions before sensor calibration begins. This ensures calibration accuracy and reliability. By determining whether the vehicle to be calibrated is parked within a preset area, the vehicle can be ensured to be stable during calibration. This is because within the preset area, the vehicle is stationary and unaffected by external interference, thus minimizing sensor data errors.

[0175] Furthermore, this embodiment can reduce risk and uncertainty during the calibration process. If the vehicle to be calibrated does not stop in the preset area, it does not meet the first calibration condition and can be determined to be unsuitable for sensor calibration. This can prevent inaccurate or invalid calibration results due to unstable vehicle conditions.

[0176] In one embodiment, the process of determining whether the vehicle to be calibrated meets the first calibration condition includes:

[0177] Determining whether the vehicle to be calibrated is being calibrated for the first time based on the vehicle information of the vehicle to be calibrated;

[0178] If it is the first calibration, it is determined that the vehicle to be calibrated meets the first calibration condition; otherwise, it is determined that the first calibration condition is not met. In one embodiment, when it is determined that the vehicle is not calibrated for the first time, the method further includes:

[0179] The first data corresponding to the vehicle to be calibrated stored in the database is called.

[0180] This embodiment describes the process of determining whether a vehicle to be calibrated meets the first calibration condition. First, based on the vehicle information, it is determined whether the vehicle to be calibrated is being calibrated for the first time. If this is the first time, the vehicle is determined to meet the first calibration condition. If this is not the first time, the vehicle is determined to not meet the first calibration condition.

[0181] This embodiment reduces the need for multiple vehicle calibrations. By determining whether the vehicle being calibrated is undergoing its first calibration, unnecessary recalibration can be avoided. Only during the first calibration is the total station required to scan the calibration device to obtain the first data. This improves calibration efficiency and reduces waste of time and resources. Furthermore, for previously calibrated vehicles, previously acquired prior calibration information can be directly used, significantly accelerating the sensor calibration process.

[0182] In summary, this embodiment improves the efficiency and accuracy of calibration without affecting the quality of calibration, saving time and resources.

[0183] In one embodiment, the process of determining whether the second calibration condition is met includes:

[0184] Determine whether a calibration start instruction is received;

[0185] If received, it is determined that the second calibration condition is met; otherwise, it is determined that the second calibration condition is not met.

[0186] The present embodiment relates to a process for determining whether the second calibration condition is met. This determination process includes the following steps: determining whether a calibration start instruction is received: in the control unit or other related equipment on the vehicle to be calibrated, determining whether a calibration start instruction sent by the calibration system is received. If a calibration start instruction is received: If a calibration start instruction is received, then it is determined that the second calibration condition is met. This means that the various sensors on the vehicle to be calibrated can start collecting data from the calibration device and can obtain the second data. Otherwise, it is determined that the second calibration condition is not met: If the calibration start instruction is not received, then it is determined that the second calibration condition is not met. This means that the various sensors on the vehicle to be calibrated cannot collect data from the calibration device and cannot obtain the second data.

[0187] As can be seen, the determination process in this embodiment is based on whether the calibration start command has been received. Only when the calibration start command has been received can the second calibration condition be determined to be met, thereby initiating the sensor acquisition operation. If the calibration start command has not been received, the sensor acquisition operation cannot be performed, and therefore the second calibration condition is determined to be unmet.

[0188] In one embodiment, obtaining calibration prior information according to the first data includes:

[0189] Eliminating data that does not meet preset requirements from the first data to obtain first calibration data;

[0190] Calibration priori information is determined based on the first calibration data.

[0191] Specifically, the first data is point cloud data obtained by scanning each calibration device in the calibration system using a total station. Calibration prior information refers to the data information of the calibration devices. When implementing this method, the first data must first be received and processed. This processing step includes removing data that does not meet preset requirements from the first data, thereby obtaining the first calibration data. This step eliminates data that does not meet calibration requirements, ensuring the accuracy and reliability of the first calibration data.

[0192] Next, calibration prior information is determined based on the first calibration data. By analyzing and processing the first calibration data, data information of the calibration device can be obtained. This information can be used as prior knowledge for subsequent calculation of calibration parameters and sensor calibration.

[0193] In this embodiment, by processing the first data, data that does not meet preset requirements is eliminated, thereby preventing this data from negatively impacting the calibration results. Furthermore, by determining calibration prior information based on the first calibration data, existing data can be better utilized, improving the calculation accuracy of calibration parameters and achieving accurate calibration of each sensor.

[0194] In one embodiment, removing data that does not meet preset requirements from the first data to obtain first calibration data includes:

[0195] The point cloud data not including the calibration device is eliminated from the first data to obtain first calibration data.

[0196] In one embodiment, removing point cloud data that does not include the calibration device from the first data to obtain first calibration data includes:

[0197] The point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system are identified to eliminate the point cloud data corresponding to the ceiling plane and / or the ground plane and / or the surrounding wall planes in the first data.

[0198] This embodiment describes a method for processing first data to obtain first calibration data. The purpose of this embodiment is to remove point cloud data corresponding to the ceiling plane, / or the ground plane, and / or the surrounding wall planes from the first data. By removing this data, more accurate first calibration data can be obtained, which is used to determine calibration prior information.

[0199] This embodiment can be implemented by identifying point cloud data representing the ceiling plane, floor plane, and / or surrounding wall planes in the calibration system. These planes can be obtained by scanning the various calibration devices in the calibration system using a total station, generating point cloud data. During the identification process, image processing or point cloud processing algorithms can be used to analyze the point cloud data to determine the position and shape of the ceiling plane, floor plane, and / or surrounding wall planes. By identifying and eliminating the point cloud data corresponding to these planes, the point cloud data associated with these planes can be eliminated.

[0200] In summary, the method of this embodiment can obtain more accurate first calibration data, thereby improving the precision and accuracy of sensor calibration. This helps ensure more reliable and accurate results when calculating the calibration parameters of each sensor, thereby effectively calibrating each sensor.

[0201] In one embodiment, the process of identifying point cloud data of a ceiling plane and / or a ground plane and / or surrounding wall planes in a calibration system includes:

[0202] Obtaining a first distance from the center position of the total station to the ceiling plane and / or the ground plane and / or the planes of surrounding walls;

[0203] Randomly selecting at least three first point clouds from the first data, and calculating the three-dimensional space plane where the first point clouds are located; wherein the difference between the distance from the first point cloud to the center position of the total station and the first distance is not greater than a first preset distance;

[0204] Calculate the second distance from each point cloud in the first calibration data to the three-dimensional space plane, and use the data corresponding to the point cloud whose difference between the second distance and the first distance is not greater than the second preset distance as the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes.

[0205] In this embodiment, since the total station's installation position information is known (including x, y, z coordinates, and whether it is level), it is easy to utilize the positional relationship between the calibration system's surrounding walls, floor, ceiling, and the center of the total station as prior information. Therefore, in this embodiment, a first distance from the total station's center to the ceiling plane, floor plane, and / or surrounding wall plane is first obtained. Then, at least three first point clouds are randomly selected from the first data, and the three-dimensional spatial planes in which these point clouds lie are calculated. The point clouds here refer to the data points obtained by the total station scanning each calibration device in the calibration system. Next, the distance from the first point cloud to the total station's center is calculated and compared with the first distance. If the difference is no greater than a first preset distance, the point cloud data can be used as data for the ceiling plane, floor plane, and / or surrounding wall plane. Finally, a second distance from each point cloud in the first calibration data to the three-dimensional spatial plane is calculated and compared with the first distance. If the difference is no greater than the second preset distance, the corresponding point cloud data can be used as data for the ceiling plane, floor plane, and / or surrounding wall plane.

[0206] The first preset distance and the second preset distance may be the same or different, and this application does not impose any special limitation thereto.

[0207] In this embodiment, the first calibration data is obtained by measuring the total station and calculating the plane of the point cloud. Point cloud data corresponding to the ceiling plane, the ground plane, and / or the surrounding wall planes is identified and excluded. By processing and comparing this data, point cloud data that meets preset conditions can be used to calculate calibration prior information. This calibration prior information can be used to calculate calibration parameters for each sensor, thereby calibrating each sensor.

[0208] In one embodiment, it further includes:

[0209] Repeating the step of identifying the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system;

[0210] determining whether the difference satisfies an iteration termination condition based on a difference between the number of point clouds in the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration and the number of point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the previous iteration;

[0211] If it meets the requirements, the iteration is terminated, and the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration process are used as the final point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes.

[0212] This embodiment describes an iterative process for identifying point cloud data for a ceiling plane, a floor plane, and / or surrounding wall planes in a calibration system. This process involves multiple iterations of the identification step. Each iteration determines whether an iteration termination condition is met by comparing the number of point cloud data points obtained in the current iteration with the number of point cloud data points obtained in the previous iteration.

[0213] During each iteration, a point cloud data recognition step is first performed based on the previously acquired point cloud data for the ceiling plane, the floor plane, and / or the surrounding wall planes. This step uses the distances from the center position measured by the total station to these planes and at least three point clouds selected from the first data to calculate a three-dimensional plane and the distance from each point cloud to this plane. Finally, point cloud data with a distance close to a preset distance is selected from these distances and used as the point cloud data for the ceiling plane, the floor plane, and / or the surrounding wall planes in the next iteration.

[0214] After each iteration, the difference between the number of point cloud data obtained in the current iteration and the number of point cloud data obtained in the previous iteration is compared to determine whether the difference meets the preset iteration termination criteria. For example, if the difference is not greater than a preset number, the iteration process is considered to have converged and the iteration ends. Alternatively, if the ratio of the number of point cloud data obtained in the current iteration to the number of point cloud data obtained in the previous iteration is not greater than a preset ratio, the iteration process is considered to have converged and the iteration ends. Simultaneously, the point cloud data obtained in the current iteration is used as the final point cloud data for the ceiling plane, / or the ground plane, and / or the surrounding wall planes.

[0215] Through this iterative approach, the recognition of point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system can be gradually improved and optimized, thereby obtaining a more accurate and reliable calibration result.

[0216] For example, assuming that the calibration site point cluster is Φ, the specific steps are as follows:

[0217] Based on the spatial distribution characteristics of the point cloud, in order to improve the efficiency of plane fitting, we sort and eliminate interference effects. The ceiling has the least interference, followed by the ground, and finally the surrounding walls. Therefore, the plane extraction order is: ceiling point cloud → ground → surrounding walls;

[0218] Perform point cloud recognition on the ceiling plane. Set the distance z from the center of the total station to the ceilingTop as prior information.

[0219] Step 1: Set in Point Cloud Randomly select content Select N (N ≥ 3) points with the first preset distance (in meters) for the plane, z is the vertical coordinate of the point cloud, and the distance between the N points is required to be greater than the threshold L Top (unit is meter), calculate the three-dimensional space plane formed by these N points

[0220] It should be noted that although the total station is adjusted to a horizontal level, it is difficult to ensure that the six planes in the calibration system are completely horizontal or vertical, that is, there are certain errors. For example, if the ceiling is not horizontal and has a large slope, the effect of identifying the plane based on a single condition will be very poor. Therefore, by setting a certain threshold On the one hand, it can greatly reduce the number of subsequent iterations, and on the other hand, it has a certain adaptability to non-horizontal or non-vertical planes. It should also be noted that L Top The setting of , ensure that the spatial distribution of N points is not too close. If the randomly selected N points are too close, it is easy to fit a plane with large errors.

[0221] Step 2: Calculate the point cloud set All points to the three-dimensional space plane distance, and based on the distance threshold (i.e. the second preset distance) (unit is meter) to obtain the distance between the plane and the 3D space Close point set

[0222] Step 3: Calculate whether the iteration termination condition is met (in, is the number of point cloud data obtained during the i+1th iteration, is the number of point cloud data obtained during the i-th iteration). If J>T top (T top is the iteration threshold, such as 0.9), then is the ceiling plane point cloud P Top Otherwise, let (by gradually reducing A more accurate plane point cloud can be obtained) and the calculation is restarted from the first step.

[0223] Among them, when the first step is executed for the first time, a point that satisfies The N (N ≥ 3) points of the condition, that is,

[0224] Further, identify the ground plane point cloud PBottom And the plane point cloud of the surrounding walls (P Left 、P Right 、P Front 、P Back The principle of the method is similar to the principle of identifying the ceiling plane point cloud mentioned above, and will not be described in detail in this application. Taking the front wall as an example, the distance x from the center of the total station to the front wall is front As prior information, the corresponding point cloud can be obtained using the ceiling point cloud computing method.

[0225] In one embodiment, the calibration device includes a plurality of calibration components, and determines calibration prior information according to the first calibration data, including:

[0226] Performing spatial clustering and segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain first local point cloud data corresponding to each calibration component;

[0227] The first identity information and the first coordinate information of each calibration component are determined according to the first local point cloud data.

[0228] In this embodiment, when the calibration device includes a plurality of calibration components, these calibration components have different positions in the calibration system. First, based on the point cloud data in the first data, it can be spatially clustered and segmented. This means that the points in the point cloud data can be grouped according to their positions, so that the points in the same group are closer to each other and farther away from the points in other groups. Through this segmentation process, the first local point cloud data corresponding to each calibration component is obtained. Next, the first identity information and first coordinate information of each calibration component can be determined by analyzing the first local point cloud data. By analyzing the point cloud data of each calibration component, the characteristics or shape of each calibration component can be identified and compared with known calibration components to confirm their identity information. At the same time, the coordinate information (such as three-dimensional coordinate information) of each calibration component in the calibration system can also be determined by analyzing the position information in the point cloud data.

[0229] It should be noted that the first identity information represents the identity of the calibration component. If the calibration component includes multiple calibration plates, each of which has its own corresponding identity information, the first identity information can be a combination of the identity information of the multiple calibration plates. The first coordinate information represents the spatial position of the calibration component in the calibration system, such as the three-dimensional coordinates of the center position of the calibration component in the calibration system.

[0230] Through the above process, the first identity information and first coordinate information of each calibration component can be determined. This information will be used to subsequently calculate the calibration parameters of each sensor to calibrate each sensor. In this way, the vehicle sensor calibration method can effectively and accurately calibrate each sensor based on the data information of the calibration device.

[0231] In one embodiment, each calibration component includes at least two calibration plates, and determining first identity information and first coordinate information of each calibration component based on the obtained first local point cloud data includes:

[0232] Determine second local point cloud data corresponding to each calibration plate from the first local point cloud data;

[0233] The second identity information and the second coordinate information corresponding to each calibration plate are determined according to the second local point cloud data.

[0234] This embodiment describes the composition of calibration components and the process for determining their corresponding identity and coordinate information in a vehicle sensor calibration method. Each calibration component in this method includes at least two calibration plates. First, the first identity and first coordinate information of each calibration component are determined based on first partial point cloud data. This is achieved by determining the second partial point cloud data corresponding to each calibration plate from the first partial point cloud data. Then, based on the second partial point cloud data, the second identity and second coordinate information corresponding to each calibration plate are determined.

[0235] It should be noted that the second identity information represents the identity of the calibration plate. If the calibration plate includes multiple calibration codes, the identity information of the calibration plate can be a combination of the identity information of the calibration codes. In this embodiment, the combination of the second identity information of the calibration plate included on the calibration component is the first identity information of the calibration component. The second coordinate information represents the position of the calibration plate in the calibration system. If the three-dimensional coordinates of the calibration component and the relative position of the calibration plate and the calibration component are known, the position of each calibration plate can be determined.

[0236] This method can determine the sensor calibration parameters based on the calibration components on the vehicle and the data collected by the sensors, thus ensuring that each sensor can accurately perceive the surrounding environment and provide accurate data.

[0237] In one embodiment, each calibration component includes at least two non-coplanar calibration plates, and determining second local point cloud data corresponding to each calibration plate based on the position of each calibration plate in each calibration component and the first local point cloud data includes:

[0238] Plane fitting is performed on each calibration plate to extract the second local point cloud data corresponding to each calibration plate.

[0239] This embodiment describes a step of processing the first local point cloud data to determine the second local point cloud data corresponding to each calibration plate. The processing step includes performing plane fitting on each calibration plate to extract the second local point cloud data corresponding to each calibration plate. In this embodiment, for each calibration plate, it is first found in the first local point cloud data. Then, the point cloud data on each calibration plate is processed using a plane fitting technique, and the plane closest to the surface of the calibration plate is found. Plane fitting can fit a plane by minimizing the distance between the point and the plane. In this embodiment, the plane fitting technique is used to extract the second local point cloud data corresponding to each calibration plate, that is, the point cloud data located near the surface of the calibration plate.

[0240] By fitting the plane of each calibration plate, the second local point cloud data corresponding to each calibration plate can be determined, that is, the point cloud data located at a position close to the calibration plate on the plane obtained by plane fitting.

[0241] This processing step allows the extraction of second local point cloud data corresponding to each calibration plate from the first local point cloud data for subsequent calibration parameter calculation and sensor calibration. This enhances the accuracy and reliability of calibration, thereby improving the performance and precision of vehicle sensors.

[0242] In one embodiment, performing plane fitting on each calibration plate to extract second local point cloud data corresponding to each calibration plate includes:

[0243] Obtaining a first height between the center of each calibration plate and the ground, randomly selecting at least three second point clouds from the first local point cloud, and calculating a second three-dimensional space plane where the second point clouds are located;

[0244] Calculating a third distance between each point cloud in the second local point cloud data and the second three-dimensional space plane;

[0245] The data corresponding to the point cloud in which the difference between the third distance and the first height is not greater than the third preset distance is used as the second local point cloud data corresponding to the calibration plate.

[0246] This embodiment describes a method for plane fitting of each calibration plate to extract the second local point cloud data corresponding to each calibration plate. The specific steps include: obtaining the first height between the center of each calibration plate and the ground. Randomly selecting at least three point clouds from the first local point cloud data as the second point cloud. Using the selected second point cloud to calculate a three-dimensional space plane. For each point cloud in the first local point cloud data, calculate its vertical distance to the second three-dimensional space plane, that is, the third distance. Compare the difference between the third distance and the first height, and filter out the point cloud data that meets the difference of not more than the third preset distance. These point cloud data are regarded as the second local point cloud data corresponding to the calibration plate.

[0247] For example, only the calibration component point cloud dataset is included Clustering is performed based on the spatial distance of the calibration components (since the calibration components are spaced far apart, they are easy to segment in space). First, the point cloud dataset of each calibration component (also known as the first local point cloud data) is obtained. ( is the point cloud dataset corresponding to the kth calibration component, M2 is the total number of calibration components);

[0248] Since each calibration component is composed of S (usually S is 3 or 4) non-coplanar calibration plates, it is possible to Using the plane fitting principle, extract the point cloud data of each calibration plate from top to bottom (or from bottom to top). The specific steps are as follows:

[0249] Step 1: For the calibration component point cloud dataset According to the center height z of the calibration plate i (i∈S) information (i.e. the first height mentioned above), randomly select ( The plane selection error threshold is in meters) and the distance between the N points is required to be greater than the threshold L. a (Usually L a <0.7W, W is the calibration width in meters), calculate the three-dimensional space plane formed by these N points

[0250] It should be noted that the calibration component is composed of multiple calibration plates. The installation height of each calibration plate (i.e., the center height of the calibration plate) is known. This prior information can also accelerate the efficiency and accuracy of plane fitting. Directly randomly finding three second point clouds to fit the plane is prone to invalid calculations if the three second point clouds are distributed on different calibration plates, and fall into more invalid cycles. Therefore, this first step can not only accurately fit the plane of the calibration plate, but also improve the efficiency of plane fitting.

[0251] Step 2: Calculation All points to the 3D plane The distance (third distance), and according to the distance threshold (Unit is meter) Get the plane Close point set

[0252] Step 3: Calculate the iteration termination condition If J>T (T is the threshold, such as 0.9), then is the point cloud of the i-th calibration plate on the k-th calibration component; otherwise, let Restart the calculation from the first step.

[0253] Through the above steps, the point cloud of the i-th calibration plate can be obtained Then from Eliminate Repeat the above steps to obtain the point cloud of each calibration plate on the kth calibration component one by one

[0254] Through the above steps, the second local point cloud data corresponding to each calibration plate that meets the requirements can be extracted from the first local point cloud data, thereby achieving the goal of plane fitting for each calibration plate. This method can accurately extract the shape and position information of each calibration plate, providing valuable data reference for subsequent sensor calibration.

[0255] In one embodiment, the calibration plate includes a calibration carrier plate and a plurality of marking codes, wherein the plurality of marking codes are fixed on the calibration carrier plate, the marking codes carrying QR code information for sensor recognition, wherein the identity information corresponding to the QR code information of all the marking codes is different;

[0256] Determining second identity information corresponding to each calibration plate according to the second local point cloud data includes:

[0257] Convert the second local point cloud data into a grayscale image;

[0258] The grayscale image is segmented according to the intervals between the marker codes to obtain a local image of each marker code;

[0259] Perform QR code recognition on each partial image to determine the first ID information of each marking code, and determine the second identity information of the calibration plate according to the first ID information of each marking code on each marking plate.

[0260] This embodiment describes a calibration plate design for a vehicle sensor calibration system. The calibration plate includes a calibration carrier and multiple identification codes. These identification codes are fixed to the carrier, and each identification code carries a QR code for sensor recognition. It should be noted that the identity information corresponding to the QR code information of each identification code is different.

[0261] On this basis, the specific implementation method for determining the second identity information corresponding to each calibration plate based on the second partial point cloud data is as follows: the second partial point cloud data is converted into a grayscale image to facilitate subsequent image processing and analysis. Based on the interval between the marker codes (this is a known condition), the grayscale image is segmented to obtain a partial image of each marker code. Each marker code is separated to facilitate individual processing. Next, QR code recognition is performed on each partial image. By decoding and analyzing the QR code image, the first ID information of each marker code can be obtained. This first ID information can serve as the unique identity of the marker code. Finally, based on the first ID information of each marker code on each marker plate, the second identity information of the calibration plate is determined (for example, the identity information of the marker plate is a combination of the first ID information of several marker codes on the marker plate in a predetermined manner). This means that by integrating and comparing the first ID information of each marker code, the identity information of the entire calibration plate can be determined. Since the identity information corresponding to the QR code information of each marker code is different, this comparison ensures that each calibration plate can be uniquely identified.

[0262] In one embodiment, after converting the second local point cloud data into a grayscale image, the method further includes: performing filtering processing on the grayscale image.

[0263] The method in this embodiment can calibrate the sensor more accurately and improve the stability and accuracy of the calibration system.

[0264] Here, the calibration member involved in the embodiment is described. Specifically, the calibration member may include at least two calibration plates and a connecting member; the connecting member is fixed between two adjacent calibration plates and is used to connect the two calibration plates. Figure 2 As shown, multiple calibration plates, such as the lower edge of calibration plate 1 and the upper edge of calibration plate 2, are connected by a connector to form a calibration component. The calibration component in this application, because it includes multiple calibration plates located at the same location, can increase the number of horizontal and / or vertical field of view (FOV) calibrations of sensors on the vehicle, such as cameras, lidars, and millimeter-wave radars, at the same calibration location, thereby improving calibration accuracy.

[0265] Furthermore, in the calibration component provided by the present application, the connector is also used to control the angle between any one of the two calibration plates and the vertical plane where the two calibration plates are located. Specifically, the connector in the calibration component provided by the present application can also adjust the angle between multiple connected calibration plates. Figure 3As shown, the angles between the calibration plates 1 and 2 and the plumb plane where the calibration component is located can be adjusted according to user needs. For example, based on the consideration that the marking codes on the calibration plates 1 and 2 are easy for the sensor to recognize, by adjusting the connecting parts, the marking plate 2 remains parallel to the plumb plane where the marking component is located, and the marking plate 1 is adjusted to have an angle with the plumb plane, for example, 30 degrees.

[0266] Furthermore, the calibration component provided by this application can adjust the angles between multiple connected calibration plates based on user needs, facilitating effective recognition of the calibration plates by a variety of different sensors on the vehicle. Furthermore, the angles between the multiple calibration plates on the calibration component increase the number of non-coplanar points, effectively improving calibration accuracy.

[0267] Furthermore, in the calibration component provided by this application, the front surfaces of the two calibration plates are located on the same side of the calibration component. Figure 3 As shown, the calibration component provided by the present application can forward splice multiple calibration plates. At this time, the front of the calibration plate 1 and the calibration plate 2 are located on the same side of the calibration component, and the angle between the calibration plate 1 and the plumb plane is adjusted according to the sensor calibration requirements of the vehicle, such as 30 degrees. When the sensor of the vehicle needs to be calibrated by the sensor, the QR code information on the same side of the calibration plate 1 and the calibration plate 2 can be obtained for calibration between different sensors on the vehicle. Since the calibration component provided by the present application increases the non-coplanar points recognized by the calibration plate, the calibration accuracy of the sensor is improved.

[0268] Furthermore, in the calibration component provided by this application, the front surfaces of the two calibration plates are located on both sides of the calibration component. Figure 4-5 As shown, the calibration component provided by this application can also reversely splice multiple calibration plates. Figure 4 and Figure 5 The three-dimensional and side views of the calibration component are shown, respectively, when reverse-assembled. At this point, the front faces of calibration plates 1 and 2 are positioned on either side of the calibration component, and the angle between calibration plate 1 and the vertical plane is adjusted to, for example, 30 degrees, based on the vehicle's sensor calibration requirements. When the vehicle's sensors require sensor calibration, the QR code information on the front face of calibration plate 1 and the back face of calibration plate 2 can be obtained for calibration between different sensors on the vehicle.

[0269] In addition, if Figure 6 and Figure 7 As shown, the calibration component provided by the present application can also be obtained by combining a plurality of calibration plates spliced ​​in the forward direction and calibration plates spliced ​​in the reverse direction. Figure 6 and Figure 7They are three-dimensional and side views of the calibration component obtained by combining multiple different splicing methods. Specifically, the calibration component provided by this application can be used to splice multiple calibration plates at a certain angle according to the calibration requirements of sensors such as cameras, lidars, and millimeter-wave radar fields of view. For example, Figure 2 and Figure 4 The two calibration plates spliced ​​together are combined to obtain Figure 6 At this time, when calibrating the vehicle sensor, it is necessary to identify the angle, mark code and other information of the combination of the positive and negative calibration plates or the calibration unit, which further improves the calibration accuracy of the vehicle sensor.

[0270] It should be emphasized that this application does not limit the specific number of calibration plates in the calibration component. Users can increase or decrease the number of calibration plates according to their own needs, based on the number of sensors and field of view angles on different vehicles, so as to meet the sensor calibration needs of different vehicles.

[0271] Based on the above embodiment, since the multiple calibration plates in the calibration component provided in this application face different front and back sides of the vehicle sensor, the non-coplanar differences of the reference objects used for sensor calibration are further increased, thereby improving the calibration accuracy of the vehicle sensor calibration.

[0272] In one embodiment, determining the second coordinate information corresponding to each calibration plate according to the second local point cloud data includes:

[0273] Extract the four sides of the circumscribed rectangle of the partial image of each logo code, and determine the intersection of two adjacent sides as the key point;

[0274] Obtaining the first image coordinates of each key point, where the first image coordinates represent the position of the key point on the calibration plate;

[0275] The first three-dimensional coordinates corresponding to each key point are determined according to the position of the marker code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0276] Correspondingly, the specific implementation method of determining the second coordinate information corresponding to each calibration plate based on the second local point cloud data is: extract the four sides of the circumscribed rectangle of the local image of each marker code; determine the intersection of two adjacent sides as the key point, and obtain the first image coordinates of each key point, that is, the position of the key point on the calibration plate, and calculate the first three-dimensional coordinates corresponding to each key point based on the position of the marker code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0277] In this embodiment, key points are determined by extracting the four sides of the circumscribed rectangle of a partial image of the marker code, effectively reducing errors caused by image noise and improving calibration accuracy. Using the marker code as a feature point for recognition improves the system's robustness to changes in the calibration plate's posture and sensor observation angle. Using QR code technology allows for quick and accurate acquisition of the marker code's first ID information, reducing the time and cost of manual processing.

[0278] The first image coordinates represent the position of each key point on the calibration plate. If the first reference 3D coordinates are known, then the first 3D coordinates corresponding to each key point can be determined if the position of the calibration plate and the first image coordinates of the key point on the calibration plate are known. The set of all first 3D coordinates is the set of 3D coordinates of all key points.

[0279] In summary, the method described in this embodiment can achieve calibration of vehicle sensors by identifying key points on the calibration plate and their corresponding first three-dimensional coordinates, with the beneficial effects of high precision, high robustness and fast calibration.

[0280] In one embodiment, when there are no three-dimensional coordinates corresponding to the first image coordinates of the key point in the first reference three-dimensional coordinates, the method further includes:

[0281] Obtain the three-dimensional coordinates corresponding to several reference points adjacent to the key point;

[0282] The first three-dimensional coordinates corresponding to the key point are determined according to the three-dimensional coordinates of several reference points.

[0283] When there is no directly corresponding point cloud for a key point, the pixel points in the four directions of up, down, left, and right that are closest to the key point and have a spatially corresponding point cloud can be used to find the key point. The angle information of the key point is used to perform bilinear interpolation and plane angle offset calibration to obtain the three-dimensional coordinates of the key point.

[0284] In one embodiment, a metal block is further provided on the calibration plate, and determining the second identity information corresponding to each calibration plate according to the second local point cloud data further includes:

[0285] The third identity information of the metal block is determined according to the first ID information of each identification code.

[0286] In the automated calibration system for autonomous driving sensors, when the sensor includes a millimeter-wave radar, a metal block is placed on a calibration plate. This is because the metal block simulates real-world targets on the road, such as vehicles, pedestrians, or static obstacles, and provides radar signal reflection. During the calibration process, the millimeter-wave radar emits radar waves, which the metal block reflects and provides parameters similar to those of the actual target object. By placing the metal block on the calibration plate, the millimeter-wave radar can identify the metal block at different angles and positions, helping to determine the sensor's precise position and angle.

[0287] In one embodiment, determining the second coordinate information corresponding to each of the calibration plates according to the second local point cloud data further includes:

[0288] The three-dimensional coordinates of the metal block are determined according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to the marking codes.

[0289] In this embodiment, the first step is to obtain the second local point cloud data. The second local point cloud data refers to the part related to the calibration plate extracted from the entire point cloud map. Next, the second coordinate information of the metal block on each calibration plate is determined based on the second local point cloud data. This means that by analyzing the second local point cloud data, the position of each metal block on the calibration plate can be determined. Finally, the three-dimensional coordinates of the metal block are determined based on the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to each marker code. This means that by matching the position of the metal block with the key points of the marker code, the position of the metal block in three-dimensional space can be determined.

[0290] In one embodiment, the metal block is fixed at the center of the calibration carrier, a plurality of marking codes are fixed on the calibration carrier and surround the metal block, and the vertices of the plurality of marking codes close to the metal block are connected to the vertices of the metal block.

[0291] In one embodiment, the three-dimensional coordinates of the metal block are determined based on the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to each of the marking codes, including: determining the third identity information of the metal block based on the first ID information of each marking code.

[0292] If the millimeter wave radar sensor needs to be calibrated, a metal block can be set in the calibration plate. When making the calibration plate, the calibration code ID information of the four neighboring areas of the metal block is known a priori information. Therefore, the third identity information of the metal block can be encoded by the four first ID information of the four neighboring areas of the calibration code in clockwise order, such as ID lt ID rt ID rd ID ld .

[0293] The third identity information of each metal block is different. When each calibration plate has a calibration code, the third identity information of the metal block can be a combination of the first ID information of the calibration code in the first manner. The second identity information of the calibration plate in the above embodiment can be a combination of the first ID information in the second manner, or a combination of each first ID information and the third identity information of the metal block, etc., and this application does not limit this.

[0294] In one embodiment, determining the second coordinate information corresponding to each calibration plate according to the second local point cloud data further includes:

[0295] Determine the three-dimensional coordinates of the four corner points of the metal block according to the first three-dimensional coordinates of the key points corresponding to the respective marking codes and the positional relationship between the metal block and the plurality of marking codes;

[0296] The three-dimensional coordinates of the center of the metal block are determined according to the three-dimensional coordinates of the four corner points of the metal block.

[0297] In this embodiment, after obtaining the 3D coordinates of the key points of the calibration code, the 3D coordinates of the four corner points of the metal block can be obtained based on the vertical angle relationship between the four corners of the metal block and the key points of the calibration code. The 3D coordinates of the center of the metal block can then be calculated based on the 3D coordinates of the four corner points.

[0298] In the calibration plate structure described in the above embodiment, the calibration carrier plate is a plate body for carrying the mark code and the metal block, and the material can be a non-metallic material that is not easily deformed, such as an acrylic plate. The metal block located in the center of the calibration carrier plate is used for calibration of millimeter-wave radars and other sensors installed on autonomous driving vehicles. The mark code is a two-dimensional code that can encode identity information, such as a binary square reference mark (Augmented Reality University of Cordoba, ArUco) for camera pose estimation, which facilitates the laser radar and other sensors installed on the autonomous driving vehicle to identify each mark code on the mark plate. Among them, the size and number of the mark codes can be adjusted according to user needs based on factors such as the focal length of the camera and the size of the laser radar field of view. In order to facilitate the sensor to distinguish different mark codes, all mark codes on the calibration plate cannot be repeated, and the identity information of each mark code is unique.

[0299] It should be emphasized that the metal block in the technical solution provided in this application is used for obtaining spatial homonymous points (the same as the same key points described in the following embodiments) when calibrating the millimeter-wave radar and the main lidar on the vehicle, thereby achieving the calibration effect of the millimeter-wave radar and the main lidar, and the ID information carried in the marker code and the QR code information on the marker code is used for subsequent automatic calibration actions between the camera internal parameters and the camera external parameters. The ID information carried in the marker code and the QR code information on the marker code can also be used for subsequent automatic calibration actions between the camera and the lidar.

[0300] Points of the same name are used to unify data from different sensors in the same three-dimensional space. Only after obtaining multiple different coordinates corresponding to the same point from different sensors can these sensors be calibrated accordingly based on the differences in these coordinates. For example, the center of a small sphere in space has coordinates (u, v) in the image information obtained by the vehicle's camera; (x, y, z) in the point cloud obtained by the vehicle's lidar; and (x, y, θ) in the point cloud obtained by the vehicle's millimeter-wave radar. By unifying these coordinates in a three-dimensional coordinate system and adaptively adjusting the corresponding sensor parameters based on the differences in the different coordinates, the vehicle's sensors can be calibrated.

[0301] The calibration plate provided by the present application includes a metal block located in the center for sensor calibration and multiple marking codes surrounding the metal block. Since the QR code information on the multiple marking codes corresponds to different identity information, the identity ID information of the metal block can be quickly identified based on the identity information. The vehicle's sensor can automatically identify images and point cloud data based on different ID information, and the automatic matching of key points is effective. Moreover, the ID information can assist in obtaining the ID information and spatial position information of the metal block located in the middle of the marking code. Different metal blocks will not be confused due to excessive similarity, thereby solving the problem of automatic correspondence between spatial points of lidar and millimeter-wave radar.

[0302] The technical solution provided by the present application can identify the identity of the metal block surrounding the marking codes through a unique QR code among multiple marking codes, without the need for manual assistance, thereby improving the calibration accuracy and the calibration work efficiency.

[0303] It should be further explained that in the calibration plate provided by this application, in order to facilitate the vehicle sensor to distinguish between the marking code and the metal block on the calibration carrier, a dark paint that has good absorption of the laser point cloud can be sprayed on the calibration carrier to obtain the marking code, such as a black marking code or a marking code with a large color difference from the calibration carrier. When the vehicle sensor is calibrated, such as a lidar, the position of the calibration carrier can be obtained by plane fitting of the calibration plate point cloud. The calibration plate point cloud is further filtered and segmented to identify the ID information and three-dimensional coordinates of each marking code on the calibration plate. At this time, in order to facilitate differentiation, the surface color of the non-metallic material selected for the calibration carrier itself needs to be different from the color of the marking code coating.

[0304] Since the dark area surfaces of the calibration carrier plate and the marking code in the calibration plate provided in this application have large differences in color and other aspects, the laser radar point clouds at different positions in the calibration plate form differences, which can improve the marking code recognition efficiency of the laser radar point cloud.

[0305] It should be further explained that in the calibration plate provided in this application, the metal block is a polygonal structure, the marking code is a polygonal result, and the vertices of multiple marking codes close to the metal block are connected to the vertices of the metal block, and the number of vertices of the metal block is greater than or equal to 4.

[0306] Specifically, in the calibration plate provided in the present application, the metal block and the marking code on the calibration plate are both polygonal in shape, and the multiple vertices of the metal block are directly connected to the vertex corners of the multiple marking codes surrounding the metal block. At this time, the vehicle sensor only needs to obtain the identity information and position information of the multiple adjacent marking codes of the metal block to completely locate the metal block, thereby establishing an association between the identity information and positioning information of the metal block. In order to ensure the accuracy of positioning, the number of marking codes is at least 4, and the metal block has at least four sides.

[0307] Since the metal block in the calibration plate provided by this application is connected to the vertex of the marking code, the position and identity information of the metal block can be obtained by identifying the position and identity information of the marking code, thereby facilitating the calibration between the millimeter-wave radar and other sensors installed on the vehicle and improving the versatility of this application.

[0308] It should be further explained that, in the calibration plate provided in the present application, the number of marking codes is 4, and the polygonal structures of the metal block and the marking codes are both rectangular.

[0309] Specifically, in the technical solution provided in the present application, the marking code and the metal block on the calibration plate are both rectangular, and the four vertices of the metal block are respectively connected to or overlapped with a vertex of the marking code close to the metal block.

[0310] Based on the above implementation, since the calibration plate provided by this application can take into account both the intrinsic and extrinsic calibration of the camera, the ArUco codes corresponding to at least four marker codes on the same calibration plate have a total of more than 16 key points. Furthermore, by utilizing the key points of multiple calibration plates within the camera's field of view and determining the three-dimensional spatial coordinates of each key point based on the key points and point cloud data acquired by the total station, the camera's intrinsic and extrinsic parameters can be calibrated. This not only achieves automated calibration, but also improves the accuracy of the camera's intrinsic parameter solution.

[0311] Based on the above implementation, the QR code information carried by the marker code in the calibration plate provided in this application is a QR code-like code that is easy for the camera to decode, such as an ArUco code.

[0312] In one embodiment, when the sensor includes a camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0313] Controlling the camera to be calibrated to collect data from the calibration device to obtain a first calibration image;

[0314] Identify the marker code in the first calibration image and determine the second ID information and the partial image of each marker code in the first calibration image;

[0315] Determining the second image coordinates of each key point included in the first calibration image according to the ID information of each marker code in the first calibration image and the partial image;

[0316] Determine the second three-dimensional coordinates corresponding to each key point in the first calibration image according to the ID information of each marker code in the first calibration image, the second image coordinates of each key point, and the first reference three-dimensional coordinates;

[0317] The intrinsic parameters of the camera to be calibrated are determined according to the second image coordinates and the second three-dimensional coordinates.

[0318] This embodiment provides a method for calibrating the intrinsic parameters of a camera to be calibrated on a vehicle. Specifically, a first calibration image is first acquired from the camera to be calibrated. The identity information of the calibration plate and the image coordinates of key points included in the first calibration image are determined from the first calibration image. The second three-dimensional coordinates corresponding to the key points in the first calibration image are then determined using the first reference three-dimensional coordinates of each point cloud in the second partial point cloud data. Finally, the intrinsic parameters of the camera to be calibrated are determined using the second image coordinates and the second three-dimensional coordinates of the key points.

[0319] The second ID information is the ID information of the calibration code included in the first calibration image. The set of calibration codes included in the first calibration image is a subset of the set of calibration codes used in the calibration system. Therefore, the second ID information is a portion of the first ID information of all calibration codes. Given the second ID information of the calibration code, the second image coordinates of each key point on the calibration plate, and the first reference 3D coordinates, the second 3D coordinates corresponding to each key point can be determined. In other words, the second 3D coordinates are a portion of the first 3D coordinates of all key points.

[0320] For example, the intrinsic parameters of a camera include the focal length f x 、f y , principal point coordinates u0, v0 and lens distortion k1, k2 (only mirror distortion is considered here). Taking the vehicle forward camera as an example, it is assumed that n calibration plates (denoted as C) on multiple calibration components are captured within the camera's field of view. i , i∈n), each calibration plate image has m calibration code key points (denoted as M j , j∈m).

[0321] For different calibration boards, the internal parameters of the same camera are consistent. Based on this, the steps to obtain the initial value of the internal parameter calibration are as follows:

[0322] (1) For the camera to be calibrated, collect a frame of calibration image (the first calibration image);

[0323] (2) Obtain the ID and local image of each calibration code through the QR code recognition method. For each local image of the code, use Hough transform to extract the four sides of the circumscribed rectangle of the code, and determine the second image coordinates of the key point through the intersection of two sides. (k=0, 1, 2, 3, representing the upper left corner, upper right corner, lower right corner, and lower left corner respectively). As the center, take an 11×11 local image, such as Figure 9 For each local image, perform platform kernel fitting to further obtain the sub-pixel coordinates of all key points of the landmark code (x ij ,y ij ) i∈n，j∈m (Sub-pixel extraction is performed here to further increase the solution accuracy).

[0324] Identify the second image coordinates (x ij ,y ij ) i∈n,j∈m and ID information ij(i∈n, j∈m) (the calibration code is the jth calibration code in the i-th calibration plate). According to the calibration plate ID prior information (the ID information of all calibration codes on the same calibration plate, the ownership relationship is known in the prior information acquisition process, that is, which identification codes each calibration plate contains), the second three-dimensional coordinate (X ij , Y ij , Z ij ) i∈n,j∈m .

[0325] For each calibration plate visible in the camera field of view, establish the world coordinate system O w X w Y w Z w (In this embodiment, the coordinate system of the total station is used as the world coordinate system), that is, the key point of the upper left corner of the calibration code of the calibration plate is used as the origin of the world coordinate system O w ; The horizontal axis of the calibration plate is X w Axis, vertical axis is Y w Axis, establish Z according to the right-hand principle w Axis, such as Figure 10 For each calibration plate visible in the camera field of view, the second image coordinates (x ij ,y ij ) i∈n,j∈m With the second three-dimensional coordinate (X ij , Y ij , Z ij ) i∈n,j∈m , according to Zhang Zhengyou calibration method, solve the camera internal parameter f x 、f y , principal point coordinates u0, v0 and lens distortion k1, k2.

[0326] It should be noted that Zhang Zhengyou's calibration method requires collecting multiple calibration images from different perspectives and distances (non-coplanar) to obtain the camera's intrinsic parameters. Because the camera's intrinsic parameters are consistent across different images, this method can only calculate the extrinsic parameters relative to each calibration plate image and cannot obtain the true camera extrinsic parameters.

[0327] Traditional camera intrinsic calibration requires collecting multiple calibration plates (such as chessboard, Figure 11 and Figure 12 The calibration process requires placing the calibration plate at different positions and on different planes (non-coplanar). However, in this embodiment, only one frame of image is required, which improves the efficiency of calibrating the intrinsic parameters of the camera to be calibrated.

[0328] In summary, this embodiment uses the calibration plate's identity information and image coordinates, along with the correspondence between the local point cloud data collected by the sensor and the reference 3D coordinates, to calculate the calibration parameters for each sensor. These calibration parameters allow accurate sensor calibration, improving both measurement accuracy and positioning capabilities.

[0329] In one embodiment, after determining the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates, the method further includes:

[0330] Determine the extrinsic parameters of the camera to be calibrated according to the second image coordinates, the second three-dimensional coordinates and the intrinsic parameters of the camera to be calibrated;

[0331] The external parameters of the camera to be calibrated include at least a first coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

[0332] In this embodiment, after determining the intrinsic parameters of the camera to be calibrated, the extrinsic parameters of the camera to be calibrated are also determined. The so-called camera intrinsic parameters are the internal parameters of the camera described in the above embodiments, including focal length, principal point coordinates, and distortion parameters, which are used to describe the internal geometric characteristics of the camera. The camera extrinsic parameters describe the geometric relationship between the camera coordinate system and the total station coordinate system, that is, the position and posture of the camera in the total station coordinate system.

[0333] In this embodiment, the intrinsic parameters of the camera to be calibrated are determined based on the second image coordinates and second 3D coordinates corresponding to each key point in the first calibration image. The next step is to determine the extrinsic parameters of the camera to be calibrated. Specifically, the extrinsic parameters of the camera to be calibrated are determined using the second image coordinates, second 3D coordinates, and the intrinsic parameters of the camera to be calibrated.

[0334] Before determining the external parameters of the camera to be calibrated, you first need to understand the relationship between the camera's coordinate system and the total station's coordinate system. The total station has its own independent coordinate system, and the camera to be calibrated also has its own coordinate system. There is a certain coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station. This relationship includes translation, rotation, and scale parameters. According to the internal parameters of the camera to be calibrated and the second image coordinates and second three-dimensional coordinates corresponding to each key point in the first calibration image, the coordinate transformation relationship between the camera and the total station coordinate system can be used to determine the external parameters of the camera to be calibrated. Specifically, the external parameters of the camera to be calibrated can be determined by solving the translation, rotation, and scale parameters between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

[0335] By determining the external parameters of the camera to be calibrated, the position and posture of the camera to be calibrated in the total station coordinate system can be accurately described, thereby being used to calibrate the camera to be calibrated in the vehicle sensor calibration method of the present application.

[0336] Specifically, the steps to calibrate the external parameters of the camera to be calibrated are:

[0337] For the 3D space points of all calibration code key points within the camera field of view (Second three-dimensional coordinate) (k∈n×m, n is the number of calibration plates, m is the number of marking codes on the calibration plates), the corresponding two-dimensional pixel on the image is (x ij ,y ij ) i∈n，j∈m (Second image coordinates). In theory, a 3D space point passes through the camera intrinsic parameters The two-dimensional pixel obtained after the lens distortion k1, k2 and the external parameters R, T are transformed is (x′ ij , y′ ij ) i∈n，j∈m .

[0338] Therefore, a function F can be constructed, whose parameters are internal parameters and external parameters. The second three-dimensional coordinate passes through the function F and outputs a two-dimensional projection image coordinate (second image coordinate) (x′ ij , y′ ij ) i∈n，j∈m .

[0339] That is, R is the rotation matrix in the first coordinate transformation relationship, and T is the translation vector in the first coordinate transformation relationship.

[0340] In practice, the identified (x ij ,y ij ) i∈n，j∈m Compared with the theoretical (x′ ij , y′ ij ) i∈n，j∈m There is a certain deviation. Therefore, the function to be optimized can be constructed as follows:

[0341]

[0342] Use the Levenberg-Marquardt (LM) algorithm to iteratively solve. Here, we first obtain the focal length f x 、f y , principal point coordinates u0, v0 and lens distortion k1, k2 as initial values ​​for iterative calculation. By giving a good initial value, subsequent iterations converge faster.

[0343] In one embodiment, when the sensor includes a surround view camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0344] Controlling the surround-view camera to be calibrated to collect data from the calibration device to obtain a second calibration image;

[0345] Project the second calibration image using the surround stitching homography matrix to obtain a projected image;

[0346] The second coordinate transformation relationship between the coordinate systems of the two adjacent surround-view cameras to be calibrated is calculated according to the same key points in the projection images corresponding to the two adjacent surround-view cameras to be calibrated.

[0347] In this embodiment, when the sensor includes surround-view cameras to be calibrated, first, a second calibration image transmitted by each surround-view camera to be calibrated is obtained. This step involves collecting calibration images captured by each surround-view camera on the vehicle to be calibrated. The second calibration image is projected using a surround-view stitching homography matrix to obtain a projected image. In this step, a surround-view stitching homography matrix technique is employed to projectively transform the second calibration image to generate a corresponding projected image. The second coordinate transformation relationship between the coordinate systems of two adjacent surround-view cameras to be calibrated is calculated based on identical key points in the corresponding projected images of the two adjacent surround-view cameras to be calibrated. By comparing identical key points in the projected images of the two adjacent surround-view cameras to be calibrated, the coordinate transformation relationship between the two adjacent surround-view cameras to be calibrated can be calculated.

[0348] In this example, image data from the surround-view cameras to be calibrated is collected and projected using surround-view stitching homography to calculate the coordinate transformation between two adjacent surround-view cameras. This result is used as part of the sensor calibration parameter calculation to calibrate the various sensors on the vehicle to be calibrated.

[0349] In one embodiment, after obtaining the second calibration image sent by each surround-view camera to be calibrated, the method further includes: performing distortion calibration on the second calibration image according to the intrinsic parameters of the camera to be calibrated to obtain a calibrated second calibration image.

[0350] In one embodiment, after controlling the surround view camera to be calibrated to collect data from the calibration device and obtaining the second calibration image, the method further includes:

[0351] Extracting the ID information of each marker code and the third image coordinates of each key point in the second calibration image;

[0352] Filtering the key points on the ground in the second calibration image according to the ID information of each marker code in the second calibration image and the third image coordinates of each key point, and obtaining the third three-dimensional coordinates of each key point on the ground;

[0353] Determine a surround stitching image based on the third three-dimensional coordinates of each key point on the ground;

[0354] calibrating the fourth three-dimensional coordinates of the key points in the surround view stitched image using the first reference three-dimensional coordinates corresponding to the key points in the surround view stitched image in the calibration prior information to obtain a calibrated surround view stitched image;

[0355] The surround view stitching homography matrix is ​​obtained according to the calibrated surround view stitching image.

[0356] It should be noted that the third three-dimensional coordinates are the three-dimensional coordinates of all key points within the field of view corresponding to the second calibration image and located on the ground. The set of all key points within the field of view corresponding to the second calibration image and located on the ground is a subset of the set of all key points in the calibration system, so the third three-dimensional coordinates are part of all first three-dimensional coordinates. Furthermore, the fourth three-dimensional coordinates are the three-dimensional coordinates of the key points in the surround view stitching image determined based on the third three-dimensional coordinates of each key point on the ground. The set of key points in the surround view stitching image determined based on the third three-dimensional coordinates of each key point on the ground is a subset of the set of all key points within the field of view corresponding to the second calibration image and located on the ground. So the fourth three-dimensional coordinates are part of all third three-dimensional coordinates. Specifically, the surround view stitching calibration task is mainly to calibrate the ground projection of the surround view cameras around the vehicle body and the transformation relationship between adjacent cameras. The specific steps are as follows:

[0357] For each surround camera c i , collect the second calibration image I i , according to the internal parameters of each camera obtained in the previous step, the second calibration image I i Perform distortion calibration to obtain the calibrated second calibration image

[0358] For the second calibration image after calibration Extract the third image coordinates and ID information (fourth identity information) of the key points within its field of view, and according to the ID information, only retain the key points on the ground, and obtain the third three-dimensional coordinate information of the key points on the ground (such as Figure 13 , Figure 13 This is a top view of the surround stitching calibration layout provided by this application). Compared with manual point selection, this embodiment is aimed at mass production and solves the automation problem.

[0359] For each camera, automatically obtain the camera's ID information. Figure 13 The key point corresponding to the largest sign code in the. Select the two largest cooperation sign key points on the left and right, and select the dotted rectangle ABCD according to the method in 14. Figure 14 This is a schematic diagram of the projection transformation matrix selection provided in this application.

[0360] When building the calibration site, it is difficult to ensure Figure 13All the calibration codes in are horizontal and coplanar, which is ignored by existing methods, resulting in insufficient surround stitching accuracy. Figure 13 The three-dimensional coordinates of the key points of the calibration code (X i , Y i , Z i ), and the camera internal and external parameters obtained in the first step, Figure 14 Use ABCD four points to calibrate, that is, (X i , Y i , Z i ) is transformed into an image by phase internal and external parameters (no distortion parameters are used here) , obtain the calibration point (like Figure 14 the location of the middle black dot);

[0361] use Calculate the surround stitching homography matrix E; for key points in the overlapping fields of view of adjacent cameras, calculate the transformation scale parameters based on the key point ID information and the actual size of the cooperation mark.

[0362] In one embodiment, when the sensor includes a main laser radar, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0363] Controlling the main laser radar to collect data from the calibration device to obtain first test point cloud data;

[0364] The first test point cloud data is matched with the first data to obtain a third coordinate transformation relationship between the main lidar coordinate system and the total station coordinate system.

[0365] This example describes the process of calibrating the primary LiDAR. Specifically, first, the first test point cloud data sent by the primary LiDAR is acquired. Next, the first test point cloud data is matched with the first data to obtain the third coordinate transformation relationship between the primary LiDAR coordinate system and the total station coordinate system.

[0366] In this embodiment, the second data collected by the sensors refers to the data collected by each sensor on the calibration device. The first test point cloud data sent by the main lidar is one type of data. These sensors and the calibration device can transmit data via wireless or wired connections.

[0367] By matching the first test point cloud data with the first data, the third coordinate transformation relationship between the main lidar coordinate system and the total station coordinate system can be determined. This transformation relationship can be used in the subsequent sensor calibration process. By obtaining the third coordinate transformation relationship between the main lidar coordinate system and the total station coordinate system, the calibration parameters of the main lidar can be calculated. These parameters can include rotation matrices and / or translation vectors.

[0368] In one embodiment, after controlling the main laser radar to collect data from the calibration device and obtaining the first test point cloud data, the method further includes:

[0369] Matching the first test point cloud data with the first data to obtain the ID information of the marker code and the fifth three-dimensional coordinates of the key points within the field of view of the main laser radar;

[0370] Determine the key points in the first calibration image that are identical to those in the field of view of the main laser radar based on the ID information of each marker code in the first calibration image, the second three-dimensional coordinates corresponding to each key point, the ID information of the marker code within the field of view of the main laser radar, and the fifth three-dimensional coordinates of the key points, and determine the three-dimensional coordinates of the identical key points;

[0371] The fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main lidar coordinate system is calculated based on the three-dimensional coordinates of the same key points.

[0372] It should be noted that the fifth three-dimensional coordinate is the three-dimensional coordinate corresponding to the key point of the field of view of the main laser radar. The set of key points of the field of view of the main laser radar is a subset of the set of all key points in the calibration system. Then the fifth three-dimensional coordinate is a part of all the first three-dimensional coordinates.

[0373] Specifically, assuming that the coordinates of the key points in the camera ci coordinate system are (the second three-dimensional coordinate described above), the key point coordinates in the main laser radar l1 coordinate system (that is, the fifth three-dimensional coordinates described above) are In theory, the transformation relationship between the same-name points in the two coordinate systems is as follows:

[0374]

[0375] Considering various errors,

[0376] The function to be optimized is as follows:

[0377] The calibration steps are:

[0378] (1) Extract the image coordinates and ID of the key points in the field of view of camera c1, and obtain the corresponding three-dimensional spatial coordinates (i.e., the second three-dimensional coordinates mentioned above) based on the ID information;

[0379] (2) extracting the three-dimensional coordinates (i.e., the fifth three-dimensional coordinates) and IDs of key points within the field of view of the main lidar l1;

[0380] (3) Based on the ID information, the relationship between the same-name points is used to solve the external parameters (the fourth coordinate transformation relationship) using the Levenberg-Marquardt (LM) algorithm.

[0381] In one embodiment, when the sensor includes a millimeter-wave radar, a metal block is provided on the calibration plate, and the calibration system further includes two tracks and a conveyor belt device provided on the tracks, wherein the two tracks are parallel to each other and the distance between the two tracks is the same as the distance between the left and right wheels of the vehicle to be calibrated. During the sensor calibration, the vehicle to be calibrated is positioned on the conveyor belt device;

[0382] Before controlling each sensor on the vehicle to be calibrated to collect data from the calibration device and obtain the second data, the method further includes:

[0383] The conveyor belt device is controlled to move the vehicle to be calibrated at a preset speed;

[0384] Controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain second data, including:

[0385] During the movement of the vehicle to be calibrated, the millimeter-wave radar is triggered to collect the coordinates of the metal block within the field of view to obtain collected data, which at least includes the coordinates of the metal block;

[0386] Determining alignment data between the first test point cloud data and the collected data according to the data timestamp;

[0387] Projecting the sixth three-dimensional coordinate of the metal block in the first test point cloud data in the alignment data onto a horizontal plane to obtain horizontal plane projection data;

[0388] The fifth coordinate transformation relationship between the millimeter-wave radar coordinate system and the main lidar coordinate system is calculated based on the horizontal plane projection data, the coordinates of the metal blocks in the collected data in the alignment data, and the positions of each metal block in the calibration system.

[0389] It should be noted that the sixth three-dimensional coordinate is the three-dimensional coordinate corresponding to the metal block within the field of view of the millimeter-wave radar. The set of metal blocks within the field of view of the millimeter-wave radar is a subset of the set of all metal blocks in the calibration system. Therefore, the sixth three-dimensional coordinate is part of the three-dimensional coordinates of all the above metal blocks.

[0390] Specifically, the millimeter wave radar can obtain the x and y coordinate information of the target, but does not have the z coordinate information of the target. Therefore, the millimeter wave coordinate system can be w X w Y w To the main lidar coordinate system O l X l Y l The transformation is regarded as the transformation of the two-dimensional XY coordinate system (such as Figure 15 As shown, Figure 15 Schematic diagram of the relationship between the millimeter-wave radar coordinate system and the main lidar coordinate system provided in this application), the parameters to be calibrated include the translation vector T and the rotation angle θ.

[0391] The conversion relationship from millimeter wave coordinates to the main lidar coordinate system is:

[0392]

[0393] We can construct a function F whose parameter is T x 、T y , θ. The three-dimensional world coordinate passes through the function F and outputs a two-dimensional projection image coordinate (x′ ij , y′ ij );

[0394] (x l ,y l )=F(T x , T y ,θ,x w ,y w );

[0395] Considering the error factor,

[0396] The function to be optimized is:

[0397]

[0398] The calibration steps are as follows:

[0399] (1) Since the vehicle is fixed on the track, the conveyor belt on the track can be used to easily control the vehicle to move in a short distance in the forward and backward direction (during the calibration process, only a very short distance (such as 1 meter) is required);

[0400] (2) Save the main lidar data (first test point cloud data) and millimeter wave radar data (the data collected on the metal block sent by the millimeter wave radar) during the vehicle's motion;

[0401] (3) According to the data timestamp, find the time-aligned data frame of the main lidar and millimeter-wave radar. For the three-dimensional coordinates of the metal block in the center of each calibration plate within the field of view of the main lidar l1 (the sixth three-dimensional coordinate of the metal block), project it onto the horizontal plane to obtain the horizontal plane projection data; for the millimeter-wave radar, identify the coordinates of all metal blocks within its field of view;

[0402] (4) Based on the horizontal projection data and the known spatial topological relationship of the metal blocks, the optimal matching relationship is obtained. Based on the optimal matching relationship, the Levenberg-Marquardt (LM) algorithm is used to iteratively calculate the above optimization function to solve the fifth coordinate transformation relationship.

[0403] In one embodiment, when the sensor includes an auxiliary laser radar, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0404] Controlling the auxiliary laser radar to collect data from the calibration device to obtain second test point cloud data;

[0405] The second test point cloud data is matched with the first test point cloud data to obtain a sixth coordinate transformation relationship between the auxiliary lidar coordinate system and the main lidar coordinate system.

[0406] Specifically, the calibration parameters (sixth coordinate transformation relationship) between the auxiliary lidar and the main lidar are the rotation matrix R and the translation vector T.

[0407] Assume that the main lidar l1 point cloud data are Q = {q1, q2, .., q m}(first test point cloud data), auxiliary laser radar l2 point cloud data is P = {p1, p2, .., p m ) (Second test point cloud data) Using the spatial topological structure of the calibration component at the calibration site, according to the spatial matching of the point clouds, the coordinate transformation relationship of the spatial points with the same name in the two point cloud data is:

[0408] Constructing the optimization function: The R and T corresponding to the sixth coordinate transformation relationship can be obtained through the Gauss-Newton or LM algorithm.

[0409] The steps for calibrating the main / auxiliary lidar are as follows:

[0410] (1) Distributed acquisition of primary lidar and auxiliary lidar point cloud data Q and P;

[0411] (2) For the point cloud data Q and P, respectively, the method of "obtaining prior information of the calibration site" is used to obtain the point cloud data after removing the ceiling, ground, and surrounding walls. and Eliminating these interferences can greatly improve the calibration accuracy;

[0412] (3) Point cloud data and Use the calibration plate plane fitting method in "Obtaining prior information of the calibration site" to obtain the calibration plate plane, calibration ball spherical surface and other information in each point cloud, including its ID information. The ID of the calibration plate can be composed of its internal calibration code ID. For example, there are 6 calibration codes on it, and the IDs are ID i (i≤6) then the calibration board ID is the calibration code ID serially arranged from small to large, such as ID1ID2ID3ID4ID5ID6;

[0413] (4) According to the calibration plate ID information and the calibration ball ID, the initial value of the calibration parameter RT corresponding to the sixth coordinate transformation relationship is solved; further, according to the calibration plate fitting plane and the calibration ball spherical surface, based on the initial value of the calibration parameter RT, further refine it through NDT or ICP to obtain the final calibration parameter RT.

[0414] Traditional methods, such as NDT and various ICP improvements, directly use point data for matching. However, due to the large differences in the hardware parameters of the primary and secondary lidars, the point cloud resolution, projection angle, and other factors may result in very few true points of the same name in space. The points in two frames of laser point cloud data cannot represent the same position in space. Therefore, using point-to-point distance as the error equation is bound to introduce random errors. In this application, by fitting the planes of each calibration plate, this effect can be effectively reduced, thereby improving calibration accuracy.

[0415] In one embodiment, the calibration device includes a plurality of calibration components. In the following embodiments, the calibration components are described as calibration balls (eg, Figure 16 As shown, Figure 16 Schematic diagram of the layout of the calibration ball provided in this application), determining the calibration prior information according to the first calibration data, including:

[0416] Performing spatial clustering and segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain third local point cloud data corresponding to each calibration component;

[0417] The fifth identity information and the fifth coordinate information corresponding to each verification component are determined according to the third local point cloud data.

[0418] First, the first calibration data must be spatially clustered and segmented based on the position of each verification component in the calibration system to obtain third-part point cloud data corresponding to each verification component. Spatial clustering and segmentation involves grouping adjacent or close point clouds in the first calibration data into clusters. This is done to separate the point clouds corresponding to the verification components to be calibrated in the first calibration data for subsequent processing. Clustering and segmentation can be performed based on factors such as spatial distance and density. After clustering and segmentation, each point cloud cluster can be considered as local point cloud data for a verification component. By processing each point cloud cluster, the third-part point cloud data for that verification component can be obtained. This local point cloud data can be used for subsequent calibration parameter calculation, error analysis, and other purposes. Next, based on the third-part point cloud data, the fifth identity information and fifth coordinate information corresponding to each verification component can be determined. The fifth identity information typically refers to a unique identifier for the verification component, used to distinguish different verification components. In the calibration system, the verification component's identity information can be determined by extracting and matching features such as its shape, size, and texture. The fifth coordinate information typically refers to the location and posture of the verification component within the calibration system. By registering and matching the point cloud data, the position and posture of the verification component relative to the calibration system can be determined. In this way, the corresponding fifth coordinate information of the verification component can be obtained.

[0419] In summary, by performing spatial clustering and segmentation on the first calibration data, we can obtain the third local point cloud data corresponding to each verification component. Then, by processing and analyzing this third local point cloud data, we can determine the fifth identity information and fifth coordinate information corresponding to each verification component. This can be used to calibrate sensor parameters and perform related functions of the autonomous driving system.

[0420] In one embodiment, determining the fifth identity information and the fifth coordinate information corresponding to each of the verification components according to the third partial point cloud data includes:

[0421] The third local point cloud data is fitted to obtain structural information and three-dimensional coordinates of each verification component.

[0422] This embodiment describes determining the fifth identity information and fifth coordinate information corresponding to each verification component based on the third partial point cloud data. To obtain the verification component's identity and coordinate information, the third partial point cloud data must be processed. First, the third partial point cloud data must be fitted, i.e., a mathematical model must be used to appropriately fit the point cloud data to obtain the structural information of the verification component. This fitting can be performed using a surface fitting algorithm, such as the least squares method, to fit the shape of a sphere. The structural information obtained from the fitting can be used to understand the shape, size, and other characteristics of the verification component. Secondly, the three-dimensional coordinates of the verification component can be obtained from the fitting results. Since the fitting process is primarily based on the point cloud data, the coordinates of the verification component's center point, i.e., the fifth coordinate information, can be obtained. By obtaining the coordinates of the verification component's center point, the position of the verification component can be determined. Finally, based on the verification component's structural information and three-dimensional coordinates, the verification component's identity information, i.e., the fifth identity information, can be obtained. The verification component's identity information, including its type and serial number, can be used to uniquely identify each verification component.

[0423] In summary, by fitting the third local point cloud data, we can obtain the structural information and 3D coordinates of the verification component, and then determine its identity and coordinates. This allows for precise positioning and identification of the verification component in the automated calibration system for autonomous driving sensors, improving the accuracy and stability of the calibration parameters.

[0424] In one embodiment, the verification member is a verification ball, and determining the fifth identity information and fifth coordinate information corresponding to each verification member according to the third local point cloud data includes:

[0425] Perform spherical fitting on the third local point cloud data to obtain the radius information and three-dimensional coordinates of the calibration sphere.

[0426] In this embodiment, the verification component is a verification ball. The step of determining the fifth identity information and fifth coordinate information corresponding to each verification component (i.e., the verification ball) based on the third local point cloud data includes the following two parts: performing spherical fitting on the third local point cloud data: point cloud data of the surface of the verification ball can be obtained based on the third local point cloud data. By performing spherical fitting on these point cloud data, the information of the fitted sphere can be obtained, mainly including the radius information of the verification ball and the three-dimensional coordinates of the center of the sphere. Spherical fitting is to process the point cloud data through mathematical methods so that the fitting result can best meet the spherical model. Determine the identity information and coordinate information of the verification ball: the identity information and coordinate information of the verification ball can be determined by the radius information of the verification ball and the three-dimensional coordinates of the center of the sphere obtained after spherical fitting. The identity information of the verification ball is usually used to distinguish different verification balls, while the coordinate information is used to determine the position of the verification ball in three-dimensional space.

[0427] By performing spherical fitting on the third local point cloud data, the radius and position of each calibration sphere can be obtained, thereby determining the fifth identity information and fifth coordinate information corresponding to each calibration component (i.e., calibration sphere). This information is used to verify the calibration parameters of each sensor in the autonomous driving sensor automatic calibration system.

[0428] It should be noted that the fifth identity information is the identity information of the verification component. For example, when the verification component is a verification sphere, the fifth identity information may be the radius information of the verification sphere. The fifth coordinate information represents the position of the verification component in the calibration system, such as the spatial position.

[0429] It's also important to note that calibration spheres are used to verify the calibration parameters of each sensor. This is because the calibration sphere has a relatively uniform spherical shape, appearing perfectly round regardless of the angle from which it is viewed. This spherical characteristic makes the data generated by the calibration sphere relatively easy to process and analyze. In other words, using a calibration sphere for sensor calibration provides accurate and reliable results. The uniform spherical shape of the calibration sphere ensures that the sensor acquires consistent data at different angles and orientations, reducing the possibility of calibration errors. Furthermore, the spherical nature of the calibration sphere ensures that the sensor's laser, radar, or camera signals are reflected or scattered in the same manner, ensuring data consistency and stability. Data processing using the calibration sphere is relatively simple. Because the calibration sphere is perfectly round, the image or point cloud data generated by the sensor can be easily processed and analyzed. By processing the calibration sphere data, sensor calibration parameters can be obtained and optimized and adjusted, thereby improving the sensor's precision and accuracy.

[0430] In addition to the calibration ball, the calibration member may also be implemented in other ways, such as a calibration plate, a square or cross-shaped calibration member, etc.

[0431] In summary, using a calibration sphere to verify the calibration parameters of autonomous driving sensors is crucial. The uniform spherical shape and data processing advantages of the calibration sphere ensure the accuracy and reliability of sensor calibration, effectively supporting the performance of autonomous driving systems.

[0432] Specifically, let P be the first calibration data after Φ removes the point clouds of the ceiling, ground and surrounding walls, and only contains the point clouds of each calibration component and the point cloud of the ground verification ball.

[0433] For the calibration ball, the specific steps are as follows:

[0434] (1) Since the installation height of the calibration ball is lower than the calibration component, the point cloud P can be divided into the point cloud containing only the calibration ball according to the height z coordinate. and point clouds containing only calibration components

[0435] (2) Clustering is performed based on the spatial interval of the calibration balls. For point clouds Perform spatial clustering and segmentation to obtain the local point cloud of each verification ball (M1 is the total number of calibration balls); for each By fitting the point cloud spherical surface, the three-dimensional coordinates of the center of the calibration ball (x j ,y j , z j ) and the radius of the ball R j information.

[0436] In one embodiment, after obtaining calibration priori information based on the first data and calibrating each sensor on the vehicle to be calibrated based on the calibration priori information and the second data, the method further includes:

[0437] The calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere.

[0438] Since a multi-level perception system requires a large number of sensors, the accuracy of the calibration results will directly affect the performance of the perception system. This application proposes a method for automatic calibration parameter verification. Specifically, for dual-target calibration, camera and main lidar calibration, main / auxiliary lidar calibration, and millimeter-wave radar and main lidar calibration result verification, the basic idea is:

[0439] For a sphere with known spatial position and size in the calibration system, identify the three-dimensional coordinates of its center in different coordinate systems and compare them with the true value. The specific steps are as follows:

[0440] In one embodiment, when the calibration parameters of the sensor include a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including:

[0441] Obtain the seventh three-dimensional coordinates of each calibration sphere within the field of view of the main laser radar;

[0442] performing coordinate transformation on the seventh three-dimensional coordinate according to the third coordinate transformation relationship to obtain the seventh three-dimensional coordinate to be compared;

[0443] comparing the seventh three-dimensional coordinate to be compared with the seventh reference three-dimensional coordinate to determine whether the third coordinate transformation relationship is accurate;

[0444] The first reference three-dimensional coordinates are the three-dimensional coordinates obtained by the total station and corresponding to the calibration component within the field of view of the main laser radar.

[0445] This example aims to verify the calibration accuracy of the main laser radar and the total station coordinate system. For the laser radar, the three-dimensional coordinates of the sphere center are obtained based on the reflectivity difference and spatial clustering through the pre-known sphere ROI area. (i.e. the seventh three-dimensional coordinate); according to the external parameter R / t relationship between the main laser radar and the total station (i.e. the third coordinate transformation relationship), Perform coordinate transformation to obtain (i.e., the seventh three-dimensional coordinate to be compared); Compare with the true value (the coordinates obtained by the total station, that is, the seventh reference three-dimensional coordinates) to determine the R / T error.

[0446] It should be noted that the seventh three-dimensional coordinate is the three-dimensional coordinate of the verification component within the field of view of the main laser radar. The set of verification components within the field of view of the main laser radar is a subset of the set of all verification components. Correspondingly, the seventh three-dimensional coordinate is part of all the fifth coordinate information.

[0447] In one embodiment, when the calibration parameters of the sensor include a fourth coordinate transformation relationship between the coordinate system of the camera to be calibrated and the main lidar coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including:

[0448] Acquire first calibration point cloud data of a calibration sphere within the field of view of the primary laser radar;

[0449] Back-projecting the first verification point cloud data onto the camera image according to the fourth coordinate transformation relationship;

[0450] Performing preset processing on the back-projected camera image to obtain a circular edge in the camera image, and determining a corresponding first circular fitting equation according to the circular edge;

[0451] The first circle fitting equation is compared with the reference circle fitting equation to determine whether the fourth coordinate transformation relationship is accurate.

[0452] This embodiment aims to verify the calibration parameter errors between the camera and the primary LiDAR. In this embodiment, first, a first verification point cloud data of the verification component within the field of view of the primary LiDAR is acquired. Next, based on the fourth coordinate transformation, the first verification point cloud data is back-projected onto the camera image. Through the back-projection operation, the three-dimensional coordinates of the verification point cloud are mapped onto the two-dimensional camera image. Finally, based on the back-projected camera image, the accuracy of the fourth coordinate transformation is determined. By comparing the locations of the verification points on the camera image with the actual locations of the verification component, the accuracy of the fourth coordinate transformation can be evaluated.

[0453] When the calibration component is a calibration ball, the first calibration point cloud data of the calibration component within the main laser radar field of view is transformed according to the external reference R corresponding to the fourth coordinate transformation relationship. c-l , t c-l Back-projection onto each camera image; back-projection of the spherical area point cloud in each camera is as follows Figure 17 As shown, the sphere is a gray prototype on the image, and the dotted line represents the back projection of the lidar point cloud. Through grayscale and binarization processing, all circular edges in the image are extracted, and the first circular fitting equation O is obtained. i (i∈n, n is the number of circular areas in the image); according to the spatial distance difference, the point cloud data inside the sphere is obtained by clustering, and the endpoints of each horizontal line are obtained, such as Figure 18 As shown, the horizontal line is broken and fits the circle Q i (like Figure 19 Calculate O i With Q i The distance between the camera and the main lidar is further determined by the camera internal parameters and the external parameters R c-l , t c-l (the accuracy of the fourth coordinate transformation relationship).

[0454] This calibration method ensures that the coordinate transformation between the camera to be calibrated and the primary LiDAR is accurate. This verification verifies the accuracy of the calibration parameters, improving the reliability and accuracy of sensor calibration.

[0455] In one embodiment, the calibration sphere is a metal sphere. When the calibration parameters of the sensor include a fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main lidar coordinate system, the calibration parameters of each sensor are verified based on the radius information and three-dimensional coordinates of each calibration sphere, including:

[0456] Obtaining the millimeter-wave radar coordinates and angles of each calibration sphere within the field of view of the millimeter-wave radar;

[0457] Obtain the seventh three-dimensional coordinate within the field of view of the main laser radar;

[0458] Projecting the millimeter-wave radar coordinates and angles onto the two-dimensional plane of the main lidar coordinate system according to the fifth coordinate transformation relationship;

[0459] The Euclidean distance is calculated based on the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates to determine whether the fifth coordinate transformation relationship is accurate.

[0460] This embodiment aims to verify the fifth coordinate transformation relationship between the auxiliary lidar L1 coordinate system and the primary lidar L2 coordinate system. This verification process includes the following steps: obtaining the millimeter-wave radar coordinates and angles of each calibration sphere within the millimeter-wave radar's field of view. Obtaining the seventh three-dimensional coordinates within the primary lidar's field of view. Projecting the millimeter-wave radar coordinates and angles onto a two-dimensional plane in the primary lidar coordinate system based on the fifth coordinate transformation relationship. Calculating the Euclidean distance based on the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates. Based on the calculated Euclidean distance, determining whether the fifth coordinate transformation relationship is accurate.

[0461] Specifically, the millimeter-wave radar coordinates and yaw angle, and the three-dimensional coordinates of the sphere center in the main lidar coordinate system are extracted respectively; through the fifth coordinate transformation relationship, the sphere center coordinates in the millimeter-wave data are projected onto the XY plane of the main lidar coordinate system; the Euclidean distance is calculated to determine the calibration accuracy.

[0462] Through the above verification process, the accuracy of the fifth coordinate transformation relationship between the millimeter-wave radar coordinate system and the main lidar coordinate system can be determined. This ensures the accuracy and reliability of the sensor's calibration parameters, thereby improving the sensor's application performance and accuracy in the vehicle.

[0463] In one embodiment, when the calibration parameters of the sensor include a sixth coordinate transformation relationship between the auxiliary lidar coordinate system and the main lidar coordinate system, the calibration parameters of each sensor are verified based on the radius information and three-dimensional coordinates of each calibration sphere, including:

[0464] Acquire first calibration point cloud data of the calibration sphere within the field of view of the main laser radar;

[0465] Acquire the second calibration point cloud data of the calibration sphere within the auxiliary laser radar field of view;

[0466] The second verification point cloud data is transformed into the main laser radar coordinate system according to the sixth coordinate transformation relationship to obtain the second verification point cloud data to be verified;

[0467] Whether the sixth coordinate transformation relationship is accurate is determined based on the first verification point cloud data and the second verification point cloud data to be verified.

[0468] This embodiment aims to verify the sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system. Specifically, first, obtain the first verification point cloud data of the verification component within the field of view of the main laser radar. Next, obtain the second verification point cloud data of the verification component within the field of view of the auxiliary laser radar. Then, according to the sixth coordinate transformation relationship, the obtained second verification point cloud data is coordinate-converted and transformed into the main laser radar coordinate system. This transformation process usually involves rotation, translation, and possible scale changes. Finally, by comparing the first verification point cloud data and the second verification point cloud data to be verified, the accuracy of the sixth coordinate transformation relationship can be evaluated. If the difference between the two sets of point cloud data is small, it means that the sixth coordinate transformation relationship is accurate; if the difference is large, the sixth coordinate transformation relationship needs to be adjusted or recalculated.

[0469] Specifically, for the two laser radar point clouds, the spherical point clouds within their respective fields of view are extracted according to the differences in spatial distance and emissivity; (Fifth coordinate transformation relationship), transform the spherical point cloud in L2 into the L1 coordinate system; calculate the Euclidean distance and determine the accuracy of the fifth coordinate transformation relationship.

[0470] In summary, this embodiment verifies the accuracy of the sixth coordinate transformation relationship between the auxiliary lidar and the primary lidar in the sensor calibration parameters. This method can help ensure that the sensor calibration parameters accurately reflect the spatial relative position relationship between the sensors in the actual scene.

[0471] In addition, in order to improve the calibration efficiency and accuracy, it is necessary to realize the rapid positioning of the vehicle to be calibrated, that is, to quickly drive it into the designated position in the calibration system. This application uses the four contact points of the vehicle for positioning. The positioning includes front and rear and left and right. Figure 20-22 As shown in the figure, the calibration system consists of two tracks, each as wide as the vehicle's tires. The vehicle runs parallel to the tracks, and a stopper is located at the front of the track. When the tires contact the stopper, the vehicle stops. After stopping, the rear wheels must be fixed. This is done for safety reasons and to calibrate the millimeter-wave radar and lidar coordinate systems.

[0472] This ensures that the initial positions of all vehicles to be calibrated are close, and that the deviation between the main lidar coordinate system and the total station coordinate system is as small as possible. During the point cloud matching process, minimizing the deviation between the two point cloud coordinate systems allows the iterative algorithm to converge faster and more accurately guarantee accuracy.

[0473] In addition, in order to assist the calibration of the millimeter-wave radar and the main lidar, a transmission belt device is installed on the track, which can realize the vehicle's short-distance uniform linear motion.

[0474] In a second aspect, the present application also provides a vehicle sensor calibration system based on a calibration system, such as Figure 23 As shown, Figure 23 This is a structural block diagram of a vehicle sensor calibration system based on a calibration system provided by the present application. The calibration system includes a total station, multiple calibration devices, and a computing device. The computing device includes:

[0475] A first control unit 91 is configured to control the total station to scan the calibration device to obtain first data when the vehicle to be calibrated meets a first calibration condition;

[0476] A second control unit 92 is configured to control the various sensors on the vehicle to be calibrated to collect data from the calibration device to obtain second data when the second calibration condition is met;

[0477] The calibration unit 93 is used to obtain calibration prior information based on the first data, and calibrate each sensor on the vehicle to be calibrated based on the calibration prior information and the second data to obtain calibration parameters of each sensor; the calibration prior information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two sensors or between a sensor and a total station.

[0478] For an introduction to the vehicle sensor calibration system based on the calibration system, please refer to the above embodiment, and this application will not go into details here.

[0479] In a third aspect, the present application also provides a vehicle sensor calibration device based on a calibration system, such as Figure 24 As shown, Figure 24 This is a structural block diagram of a vehicle sensor calibration device based on a calibration system provided by this application, which includes:

[0480] Memory 201, used for storing computer programs;

[0481] The processor 202 is configured to implement the steps of the vehicle sensor calibration method based on the calibration system as described above when storing the computer program.

[0482] For an introduction to the vehicle sensor calibration device based on the calibration system, please refer to the above embodiment, and this application will not go into details here.

[0483] In a fourth aspect, the present application further provides a computer-readable storage medium 210, such as Figure 25 As shown, Figure 25This is a block diagram of the structure of a computer-readable storage medium provided in this application. This computer-readable storage medium 210 stores a computer program 211. When executed by processor 202, computer program 211 implements the steps of the vehicle sensor calibration method described above. For an introduction to computer-readable storage medium 210, please refer to the above embodiment and this application will not elaborate on it here.

[0484] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0485] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle sensor calibration method based on a calibration system, characterized in that: The calibration system includes a total station and multiple calibration devices, including: When the vehicle to be calibrated meets the first calibration condition, the total station is controlled to scan the calibration device to obtain first data; the process of determining whether the vehicle to be calibrated meets the first calibration condition includes: determining whether the vehicle to be calibrated is parked in a preset area; if so, determining that the vehicle to be calibrated meets the first calibration condition, otherwise, determining that the first calibration condition is not met; or, determining whether the vehicle to be calibrated is being calibrated for the first time based on the vehicle information of the vehicle to be calibrated; if so, determining that the vehicle to be calibrated meets the first calibration condition, otherwise, determining that the first calibration condition is not met; when it is determined that the vehicle to be calibrated is not being calibrated for the first time, the process also includes: calling the first data corresponding to the vehicle to be calibrated stored in a database; When the second calibration condition is met, receiving data collected by the calibration device from each sensor on the vehicle to be calibrated to obtain second data; the process of determining whether the second calibration condition is met includes: determining whether a calibration start instruction is received; if so, determining that the second calibration condition is met; otherwise, determining that the second calibration condition is not met; Obtaining calibration priori information based on the first data, and calibrating each of the sensors on the vehicle to be calibrated based on the calibration priori information and the second data to obtain calibration parameters of each of the sensors; the calibration priori information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two of the sensors or between the sensor and the total station; Obtaining calibration prior information according to the first data includes: Eliminating data that does not meet preset requirements from the first data to obtain first calibration data; The calibration prior information is determined according to the first calibration data.

2. The vehicle sensor calibration method based on the calibration system according to claim 1, characterized in that: Eliminating data that does not meet preset requirements from the first data to obtain first calibration data includes: The point cloud data not including the calibration device is eliminated from the first data to obtain the first calibration data.

3. The vehicle sensor calibration method based on the calibration system according to claim 2, characterized in that: The calibration device includes a plurality of calibration components, and determines the calibration prior information according to the first calibration data, including: Performing spatial clustering and segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain first local point cloud data corresponding to each calibration component; The first identity information and the first coordinate information of each of the calibration components are determined according to the first local point cloud data.

4. The vehicle sensor calibration method based on the calibration system according to claim 3, characterized in that: Each of the calibration components includes at least two calibration plates, and determining first identity information and first coordinate information of each of the calibration components based on the obtained first local point cloud data includes: Determine second local point cloud data corresponding to each calibration plate from the first local point cloud data; The second identity information and the second coordinate information corresponding to each of the calibration plates are determined according to the second local point cloud data.

5. The vehicle sensor calibration method based on the calibration system according to claim 4, characterized in that: Each calibration member includes at least two non-coplanar calibration plates, and determining second local point cloud data corresponding to each calibration plate according to the position of each calibration plate in each calibration member and the first local point cloud data includes: Plane fitting is performed on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates.

6. The vehicle sensor calibration method based on the calibration system according to claim 5, characterized in that: Performing plane fitting on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates includes: Obtaining a first height between the center of each calibration plate and the ground, randomly selecting at least three second point clouds from the first local point cloud, and calculating a second three-dimensional space plane where the second point clouds are located; Calculating a third distance between each point cloud in the second local point cloud data and the second three-dimensional space plane; The data corresponding to the point cloud in which the difference between the third distance and the first height is not greater than the third preset distance is used as the second local point cloud data corresponding to the calibration plate.

7. The vehicle sensor calibration method based on the calibration system according to claim 4, characterized in that: The calibration plate includes a calibration carrier plate and a plurality of marking codes, wherein the plurality of marking codes are fixed on the calibration carrier plate, the marking codes carrying two-dimensional code information for identification by the sensor, wherein the identity information corresponding to the two-dimensional code information of all the marking codes is different; Determining second identity information corresponding to each of the calibration plates according to the second local point cloud data includes: Converting the second local point cloud data into a grayscale image; Segmenting the grayscale image according to the intervals between the marker codes to obtain a partial image of each marker code; Perform two-dimensional code recognition on each of the partial images to determine the first ID information of each of the marking codes, and determine the second identity information of the calibration plate according to the first ID information of each of the marking codes on each of the marking plates.

8. The vehicle sensor calibration method based on the calibration system according to claim 7, characterized in that: Determining second coordinate information corresponding to each of the calibration plates according to the second local point cloud data includes: Extracting four sides of the circumscribed rectangle of each partial image of the logo code, and determining the intersection of two adjacent sides as a key point; Acquire a first image coordinate of each of the key points, where the first image coordinate represents a position of the key point on the calibration plate; The first three-dimensional coordinates corresponding to each key point are determined according to the position of the marker code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

9. The vehicle sensor calibration method based on the calibration system according to claim 8, characterized in that: When there is no three-dimensional coordinate corresponding to the first image coordinate of the key point in the first reference three-dimensional coordinates, the method further includes: Obtaining three-dimensional coordinates corresponding to several reference points adjacent to the key point; The first three-dimensional coordinate corresponding to the key point is determined according to the three-dimensional coordinates of the plurality of reference points.

10. The vehicle sensor calibration method based on the calibration system according to claim 8, characterized in that: A metal block is further provided on the calibration plate, and second identity information corresponding to each calibration plate is determined according to the second local point cloud data, further comprising: The third identity information of the metal block is determined according to the first ID information of each identification code.

11. The vehicle sensor calibration method based on the calibration system according to claim 10, characterized in that: Determining second coordinate information corresponding to each of the calibration plates according to the second local point cloud data further includes: The three-dimensional coordinates of the metal block are determined according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to the marking codes.

12. The vehicle sensor calibration method based on the calibration system according to claim 11, characterized in that: The metal block is fixed at the center of the calibration carrier, a plurality of marking codes are fixed on the calibration carrier and surround the metal block, and a plurality of marking codes are connected to the vertices of the metal block at vertices close to the metal block.

13. The vehicle sensor calibration method based on the calibration system according to claim 12, characterized in that: Determining the three-dimensional coordinates of the metal block according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to the marking codes includes: Determine the three-dimensional coordinates of the four corner points of the metal block according to the first three-dimensional coordinates of the key points corresponding to the vertices of each of the marking codes close to the metal block; The three-dimensional coordinates of the center of the metal block are determined according to the three-dimensional coordinates of the four corner points of the metal block.

14. The vehicle sensor calibration method based on the calibration system according to claim 11, characterized in that: When the sensor includes a camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes: Controlling the camera to be calibrated to collect data from the calibration device to obtain a first calibration image; Identify the marker codes in the first calibration image and determine the second ID information and the partial image of each marker code in the first calibration image; determining second image coordinates of each key point included in the first calibration image according to the ID information of each marker code and the partial image in the first calibration image; Determining the second three-dimensional coordinates corresponding to each key point in the first calibration image according to the ID information of each marker code in the first calibration image, the second image coordinates of each key point, and the first reference three-dimensional coordinates; Determine the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates.

15. The vehicle sensor calibration method based on the calibration system according to claim 14, characterized in that: After determining the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates, the method further includes: Determine the extrinsic parameters of the camera to be calibrated according to the second image coordinates, the second three-dimensional coordinates, and the intrinsic parameters of the camera to be calibrated; The external parameters of the camera to be calibrated include at least a first coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

16. The vehicle sensor calibration method based on the calibration system according to claim 15, characterized in that: When the sensor includes a surround view camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes: Controlling the surround-view camera to be calibrated to collect data from the calibration device to obtain a second calibration image; Projecting the second calibration image using the surround stitching homography matrix to obtain a projected image; A second coordinate transformation relationship between the coordinate systems of two adjacent surround-view cameras to be calibrated is calculated according to the same key points in the projection images corresponding to the two adjacent surround-view cameras to be calibrated.

17. The vehicle sensor calibration method based on the calibration system according to claim 16, characterized in that: After controlling the surround-view camera to be calibrated to collect data from the calibration device and obtaining a second calibration image, the method further includes: Extracting the ID information of each marker code and the third image coordinates of each key point in the second calibration image; Filtering key points on the ground in the second calibration image according to the ID information of each marker code in the second calibration image and the third image coordinates of each key point, and obtaining third three-dimensional coordinates of each key point on the ground; determining a surround stitching image according to the third three-dimensional coordinates of each of the key points located on the ground; calibrating the fourth three-dimensional coordinates of the key points in the surround view stitched image using the first reference three-dimensional coordinates corresponding to the key points in the surround view stitched image in the calibration prior information to obtain a calibrated surround view stitched image; The surround view stitching homography matrix is ​​obtained according to the calibrated surround view stitching image.

18. The vehicle sensor calibration method based on the calibration system according to claim 15, characterized in that: When the sensor includes a main laser radar, controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes: Controlling the main laser radar to collect data from the calibration device to obtain first test point cloud data; The first test point cloud data is matched with the first data to obtain a third coordinate transformation relationship between the main lidar coordinate system and the total station coordinate system.

19. The vehicle sensor calibration method based on the calibration system according to claim 18, characterized in that: After controlling the main laser radar to collect data from the calibration device to obtain first test point cloud data, the method further includes: Matching the first test point cloud data with the first data to obtain ID information of the marker code and the fifth three-dimensional coordinates of the key points within the field of view of the main laser radar; Determine, based on the ID information of each marker code in the first calibration image, the second three-dimensional coordinates corresponding to each key point, the ID information of the marker code within the field of view of the main laser radar, and the fifth three-dimensional coordinates of the key point, the key point in the first calibration image that is identical to the key point within the field of view of the main laser radar, and determine the three-dimensional coordinates of the identical key point; The fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main lidar coordinate system is calculated based on the three-dimensional coordinates of the same key point.

20. The vehicle sensor calibration method based on the calibration system according to claim 19, characterized in that: When the sensor includes a millimeter-wave radar, a metal block is provided on the calibration plate, and the calibration system further includes two tracks and a conveyor belt device provided on the tracks, wherein the two tracks are parallel to each other and the distance between the two tracks is the same as the distance between the left and right wheels of the vehicle to be calibrated. When performing sensor calibration, the vehicle to be calibrated is located on the conveyor belt device; Before controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain the second data, the method further includes: Controlling the conveyor belt device to move the vehicle to be calibrated at a preset speed; Controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes: During the movement of the vehicle to be calibrated, triggering the millimeter-wave radar to collect the coordinates of the metal block within the field of view to obtain collected data, wherein the collected data at least includes the coordinates of the metal block; Determining alignment data between the first test point cloud data and the collected data according to the data timestamp; Projecting the sixth three-dimensional coordinate of the metal block in the first test point cloud data in the alignment data onto a horizontal plane to obtain horizontal plane projection data; The fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main lidar coordinate system is calculated based on the horizontal plane projection data, the coordinates of the metal blocks in the collected data in the alignment data, and the positions of each metal block in the calibration system.

21. The vehicle sensor calibration method based on the calibration system according to claim 20, characterized in that: When the sensor includes an auxiliary laser radar, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes: controlling the auxiliary laser radar to collect data from the calibration device to obtain second test point cloud data; The second test point cloud data is matched with the first test point cloud data to obtain a sixth coordinate transformation relationship between the auxiliary lidar coordinate system and the main lidar coordinate system.

22. The vehicle sensor calibration method based on the calibration system according to any one of claims 1 to 21, characterized in that: The calibration device includes a plurality of verification components, and determines the calibration prior information according to the first calibration data, including: Performing spatial clustering and segmentation on the first calibration data according to the position of each verification component in the calibration system to obtain third local point cloud data corresponding to each verification component; The fifth identity information and the fifth coordinate information corresponding to each of the verification components are determined according to the third local point cloud data.

23. The vehicle sensor calibration method based on the calibration system according to claim 22, characterized in that: Determining fifth identity information and fifth coordinate information corresponding to each of the verification components according to the third local point cloud data includes: The third local point cloud data is fitted to obtain structural information and three-dimensional coordinates of each verification component.

24. The vehicle sensor calibration method based on the calibration system according to claim 23, characterized in that: The verification component is a verification sphere, and fitting is performed on the third local point cloud data to obtain structural information and three-dimensional coordinates of each verification component, including: Spherical fitting is performed on the third local point cloud data to obtain radius information and three-dimensional coordinates of the calibration sphere.

25. The vehicle sensor calibration method based on the calibration system according to claim 24, characterized in that: After obtaining calibration priori information according to the first data and calibrating each of the sensors on the vehicle to be calibrated according to the calibration priori information and the second data, the method further includes: The calibration parameters of each of the sensors are verified according to the radius information and three-dimensional coordinates of each of the calibration spheres.

26. The vehicle sensor calibration method based on the calibration system according to claim 25, characterized in that: When the calibration parameters of the sensor include a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including: Obtaining the seventh three-dimensional coordinates of each calibration sphere within the field of view of the main laser radar; performing coordinate transformation on the seventh three-dimensional coordinate according to the third coordinate transformation relationship to obtain a seventh three-dimensional coordinate to be compared; comparing the seventh three-dimensional coordinate to be compared with the seventh reference three-dimensional coordinate to determine whether the third coordinate transformation relationship is accurate; The first reference three-dimensional coordinates are the three-dimensional coordinates obtained by the total station and corresponding to the verification component within the field of view of the main laser radar.

27. The vehicle sensor calibration method based on the calibration system according to claim 25, characterized in that: When the calibration parameters of the sensor include a fourth coordinate transformation relationship between the coordinate system of the camera to be calibrated and the main lidar coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including: Acquire first calibration point cloud data of a calibration sphere within the field of view of the primary laser radar; Back-projecting the first verification point cloud data onto the camera image according to the fourth coordinate transformation relationship; Performing preset processing on the back-projected camera image to obtain a circular edge in the camera image, and determining a corresponding first circular fitting equation according to the circular edge; The first circle fitting equation is compared with a reference circle fitting equation to determine whether the fourth coordinate transformation relationship is accurate.

28. The vehicle sensor calibration method based on the calibration system according to claim 25, characterized in that: The calibration sphere is a metal sphere. When the calibration parameters of the sensor include the fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main lidar coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including: Obtaining the millimeter-wave radar coordinates and angles of each calibration sphere within the field of view of the millimeter-wave radar; Obtaining a seventh three-dimensional coordinate within the field of view of the primary laser radar; Projecting the millimeter-wave radar coordinates and angles onto a two-dimensional plane of the main lidar coordinate system according to the fifth coordinate transformation relationship; The Euclidean distance is calculated based on the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates to determine whether the fifth coordinate transformation relationship is accurate.

29. The vehicle sensor calibration method based on the calibration system according to claim 25, characterized in that: When the calibration parameters of the sensor include a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration sphere, including: Acquire first calibration point cloud data of the calibration sphere within the field of view of the main laser radar; Acquire the second calibration point cloud data of the calibration sphere within the auxiliary laser radar field of view; transforming the second verification point cloud data into the main lidar coordinate system according to the sixth coordinate transformation relationship to obtain second verification point cloud data to be verified; Determine whether the sixth coordinate transformation relationship is accurate based on the first verification point cloud data and the second verification point cloud data to be verified.

30. A vehicle sensor calibration system based on a calibration system, characterized in that: The calibration system includes a total station, a plurality of calibration devices, and a computing device, wherein the computing device includes: A first control unit is configured to control the total station to scan the calibration device to obtain first data when the vehicle to be calibrated meets a first calibration condition; the process of determining whether the vehicle to be calibrated meets the first calibration condition includes: determining whether the vehicle to be calibrated is parked in a preset area; if so, determining that the vehicle to be calibrated meets the first calibration condition; otherwise, determining that the first calibration condition is not met; or determining whether the vehicle to be calibrated is being calibrated for the first time based on the vehicle information of the vehicle to be calibrated; if so, determining that the vehicle to be calibrated meets the first calibration condition; otherwise, determining that the first calibration condition is not met; when it is determined that the vehicle to be calibrated is not being calibrated for the first time, further comprising: calling first data corresponding to the vehicle to be calibrated stored in a database; a second control unit configured to receive, when a second calibration condition is met, second data acquired by the calibration device from each of the sensors on the vehicle to be calibrated; wherein the process of determining whether the second calibration condition is met comprises: determining whether a calibration start instruction is received; if so, determining that the second calibration condition is met; otherwise, determining that the second calibration condition is not met; A calibration unit is configured to obtain calibration priori information based on the first data, and to calibrate each of the sensors on the vehicle to be calibrated based on the calibration priori information and the second data to obtain calibration parameters of each of the sensors; the calibration priori information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two of the sensors or between the sensor and the total station; obtaining the calibration priori information based on the first data comprises: eliminating data that does not meet preset requirements from the first data to obtain first calibration data; and determining the calibration priori information based on the first calibration data.

31. A vehicle sensor calibration device based on a calibration system, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the vehicle sensor calibration method based on the calibration system as described in any one of claims 1 to 29 when storing a computer program.

32. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the vehicle sensor calibration method based on the calibration system according to any one of claims 1 to 29 are implemented.

Images