Vehicle sensor calibration method, system and device based on calibration system and medium

Abstract

The invention discloses a vehicle sensor calibration method, system and device based on a calibration system, and a medium, relates to the field of automatic driving, and solves the problem of low vehicle sensor calibration efficiency. In the scheme, when a to-be-calibrated vehicle meets a first calibration condition, a total station is controlled to scan a calibration device to obtain first data; when a second calibration condition is met, each sensor on the to-be-calibrated vehicle is controlled to collect the calibration device to obtain second data; and obtaining calibration prior information according to the first data, and calibrating each sensor on the to-be-calibrated vehicle according to the calibration prior information and the second data. Therefore, by using the total station and the plurality of calibration devices, the personnel demand and the time consumption in the calibration process can be greatly reduced. Therefore, the calibration cost can be reduced, mass production is facilitated, and the production efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a vehicle sensor calibration method, system, device and medium based on a calibration system. Background Art

[0002] The perception system is called the eyes of autonomous driving in the field of autonomous driving. Its main task is to obtain environmental information and identify environmental status. Due to the complexity of ambient lighting, weather and scenes, the use of a single sensor has certain limitations. Therefore, multi-sensor fusion is often used to improve the redundancy and complementarity of information to ensure the safety of autonomous driving. Commonly used perception system sensors include lidar, camera and millimeter wave radar. According to different perception tasks and sensor characteristics, sensors need to be effectively combined. In order to ensure the accuracy of environmental perception, it is usually necessary to accurately calibrate the sensors on the vehicle to determine the internal and external parameters between the sensors. However, there are some problems in the existing sensor calibration methods in engineering applications, such as the need to consume a lot of time and human resources, and it is difficult to achieve mass production. Summary of the invention

[0003] The purpose of this application is to provide a vehicle sensor calibration method, system, device and medium based on a calibration system, which can greatly reduce the personnel requirements and time consumption of the calibration process by using a total station and multiple calibration devices. This can reduce the calibration cost, facilitate mass production and improve production efficiency.

[0004] In a first aspect, the present application provides a vehicle sensor calibration method based on a calibration system, wherein the calibration system includes a total station and multiple calibration devices, including:

[0005] When the vehicle to be calibrated meets the first calibration condition, controlling the total station to scan the calibration device to obtain first data;

[0006] When the second calibration condition is met, receiving second data acquired by the calibration device through the sensors on the vehicle to be calibrated;

[0007] Calibration prior information is obtained based on the first data, and each of the sensors on the vehicle to be calibrated is calibrated based on the calibration prior information and the second data to obtain calibration parameters of each sensor; the calibration prior information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two of the sensors or between the sensor and the total station.

[0008] In one embodiment, the process of determining whether the vehicle to be calibrated meets the first calibration condition includes:

[0009] Determining whether the vehicle to be calibrated is parked in a preset area;

[0010] If the vehicle stops in the preset area, it is determined that the vehicle to be calibrated meets the first calibration condition; otherwise, it is determined that the first calibration condition is not met.

[0011] In one embodiment, the process of determining whether the vehicle to be calibrated meets the first calibration condition includes:

[0012] Determining whether the vehicle to be calibrated is being calibrated for the first time according to the vehicle information of the vehicle to be calibrated;

[0013] If it is the first calibration, it is determined that the vehicle to be calibrated meets the first calibration condition; otherwise, it is determined that the first calibration condition is not met.

[0014] In one embodiment, when determining that the vehicle is not calibrated for the first time, the method further includes:

[0015] The first data corresponding to the vehicle to be calibrated and stored in the database is called.

[0016] In one embodiment, the process of determining whether the second calibration condition is met includes:

[0017] Determine whether a calibration start instruction is received;

[0018] If received, it is determined that the second calibration condition is met; otherwise, it is determined that the second calibration condition is not met.

[0019] In one embodiment, obtaining calibration prior information according to the first data includes:

[0020] Eliminate data that does not meet preset requirements from the first data to obtain first calibration data;

[0021] The calibration prior information is determined according to the first calibration data.

[0022] In one embodiment, removing data that does not meet preset requirements from the first data to obtain first calibration data includes:

[0023] The point cloud data not including the calibration device is eliminated from the first data to obtain the first calibration data.

[0024] In one embodiment, the calibration device includes a plurality of calibration components, and determining the calibration prior information according to the first calibration data includes:

[0025] Performing spatial clustering segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain first local point cloud data corresponding to each calibration component;

[0026] The first identity information and the first coordinate information of each of the calibration components are determined according to the first local point cloud data.

[0027] In one embodiment, each of the calibration components includes at least two calibration plates, and determining the first identity information and the first coordinate information of each of the calibration components according to the first local point cloud data includes:

[0028] Determine second local point cloud data corresponding to each calibration plate from the first local point cloud data;

[0029] The second identity information and the second coordinate information corresponding to each of the calibration plates are determined according to the second local point cloud data.

[0030] In one embodiment, each of the calibration components includes at least two non-coplanar calibration plates, and determining the second local point cloud data corresponding to each calibration plate according to the position of each calibration plate in each of the calibration components and the first local point cloud data includes:

[0031] Plane fitting is performed on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates.

[0032] In one embodiment, plane fitting is performed on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates, including:

[0033] Obtaining a first height between the center of each calibration plate and the ground, randomly selecting at least three second point clouds from the first local point cloud, and calculating a second three-dimensional space plane where the second point cloud is located;

[0034] Calculating a third distance between each point cloud in the second local point cloud data and the second three-dimensional space plane;

[0035] The data corresponding to the point cloud whose difference between the third distance and the first height is not greater than the third preset distance is used as the second local point cloud data corresponding to the calibration plate.

[0036] In one embodiment, the calibration plate includes a calibration carrier plate and a plurality of marking codes, wherein the plurality of marking codes are fixed on the calibration carrier plate, the marking codes carrying two-dimensional code information for identification by the sensor, wherein the identity information corresponding to the two-dimensional code information of all the marking codes is different;

[0037] Determining second identity information corresponding to each of the calibration plates according to the second local point cloud data includes:

[0038] Converting the second local point cloud data into a grayscale image;

[0039] Segmenting the grayscale image according to the intervals between the marker codes to obtain a local image of each marker code;

[0040] Perform two-dimensional code recognition on each of the partial images to determine the first ID information of each of the marking codes, and determine the second identity information of the calibration plate according to the first ID information of each of the marking codes on each of the marking plates.

[0041] In one embodiment, determining the second coordinate information corresponding to each of the calibration plates according to the second local point cloud data includes:

[0042] Extracting four sides of the circumscribed rectangle of each partial image of the logo code, and determining the intersection of two adjacent sides as a key point;

[0043] Acquire the first image coordinates of each of the key points, where the first image coordinates represent the position of the key point on the calibration plate;

[0044] The first three-dimensional coordinates corresponding to each of the key points are determined according to the position of the marking code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0045] In one embodiment, when there is no three-dimensional coordinate corresponding to the first image coordinate of the key point in the first reference three-dimensional coordinate, the method further includes:

[0046] Obtaining three-dimensional coordinates corresponding to a plurality of reference points adjacent to the key point;

[0047] The first three-dimensional coordinate corresponding to the key point is determined according to the three-dimensional coordinates of the plurality of reference points.

[0048] In one embodiment, a metal block is further provided on the calibration plate, and determining the second identity information corresponding to each calibration plate according to the second local point cloud data further includes:

[0049] The third identity information of the metal block is determined according to the first ID information of each of the marking codes.

[0050] In one embodiment, determining the second coordinate information corresponding to each of the calibration plates according to the second local point cloud data further includes:

[0051] The three-dimensional coordinates of the metal block are determined according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to each of the marking codes.

[0052] In one embodiment, the metal block is fixed at the center of the calibration carrier, a plurality of marking codes are fixed on the calibration carrier and surround the metal block, and a plurality of marking codes are connected to the vertices of the metal block at vertices close to the direction of the metal block.

[0053] In one embodiment, determining the three-dimensional coordinates of the metal block according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to each of the marking codes includes:

[0054] Determine the three-dimensional coordinates of the four corner points of the metal block according to the first three-dimensional coordinates of the key points corresponding to the vertices of each of the marking codes close to the direction of the metal block;

[0055] The three-dimensional coordinates of the center of the metal block are determined according to the three-dimensional coordinates of the four corner points of the metal block.

[0056] In one embodiment, when the sensor includes a camera to be calibrated, controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0057] Controlling the camera to be calibrated to collect data from the calibration device to obtain a first calibration image;

[0058] Identify the marker code in the first calibration image, and determine the second ID information and the partial image of each marker code in the first calibration image;

[0059] Determining the second image coordinates of each key point included in the first calibration image according to the ID information of each marker code in the first calibration image and the partial image;

[0060] Determine the second three-dimensional coordinates corresponding to each of the key points in the first calibration image according to the ID information of each of the marking codes in the first calibration image, the second image coordinates of each of the key points, and the first reference three-dimensional coordinates;

[0061] Determine the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates.

[0062] In one embodiment, after determining the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates, the method further includes:

[0063] Determine the extrinsic parameters of the camera to be calibrated according to the second image coordinates, the second three-dimensional coordinates and the intrinsic parameters of the camera to be calibrated;

[0064] The external parameters of the camera to be calibrated include at least a first coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

[0065] In one embodiment, when the sensor includes a surround view camera to be calibrated, controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0066] Controlling the surround-view camera to be calibrated to collect data from the calibration device to obtain a second calibration image;

[0067] Projecting the second calibration image using the surround stitching homography matrix to obtain a projected image;

[0068] A second coordinate transformation relationship between the coordinate systems of two adjacent surround-view cameras to be calibrated is calculated according to the same key points in the projection images corresponding to the two adjacent surround-view cameras to be calibrated.

[0069] In one embodiment, after controlling the surround view camera to be calibrated to collect data from the calibration device and obtaining the second calibration image, the method further includes:

[0070] Extracting ID information of each marker code and third image coordinates of each key point in the second calibration image;

[0071] Filtering the key points on the ground in the second calibration image according to the ID information of each marker code in the second calibration image and the third image coordinates of each key point, and obtaining the third three-dimensional coordinates of each key point on the ground;

[0072] Determine a surround stitching image according to the third three-dimensional coordinates of each of the key points located on the ground;

[0073] Calibrate the fourth three-dimensional coordinates of the key points in the surround view stitched image using the first reference three-dimensional coordinates corresponding to the key points in the surround view stitched image in the calibration prior information to obtain a calibrated surround view stitched image;

[0074] The surround view stitching homography matrix is ​​obtained according to the calibrated surround view stitching image.

[0075] In one embodiment, when the sensor includes a main laser radar, controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0076] Controlling the main laser radar to collect data from the calibration device to obtain first test point cloud data;

[0077] The first test point cloud data is matched with the first data to obtain a third coordinate transformation relationship between the main lidar coordinate system and the total station coordinate system.

[0078] In one embodiment, after controlling the main laser radar to collect data from the calibration device and obtaining the first test point cloud data, the method further includes:

[0079] Matching the first test point cloud data with the first data to obtain ID information of the marker code and the fifth three-dimensional coordinates of the key points within the field of view of the main laser radar;

[0080] Determine the same key points in the first calibration image as those in the field of view of the main laser radar according to the ID information of each of the marker codes in the first calibration image, the second three-dimensional coordinates corresponding to each of the key points, the ID information of the marker codes within the field of view of the main laser radar, and the fifth three-dimensional coordinates of the key points, and determine the three-dimensional coordinates of the same key points;

[0081] The fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main lidar coordinate system is calculated according to the three-dimensional coordinates of the same key point.

[0082] In one embodiment, when the sensor includes a millimeter wave radar, a metal block is provided on the calibration plate, and the calibration system further includes two tracks and a conveyor belt device provided on the tracks, the two tracks are parallel to each other and the distance between the two tracks is the same as the distance between the left and right wheels of the vehicle to be calibrated, and the vehicle to be calibrated is located on the conveyor belt device when the sensor calibration is performed;

[0083] Before controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain the second data, the method further includes:

[0084] By controlling the conveyor belt device to move the vehicle to be calibrated at a preset speed;

[0085] Controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain second data, including:

[0086] During the movement of the vehicle to be calibrated, the millimeter wave radar is triggered to collect the coordinates of the metal block within the field of view to obtain collected data, wherein the collected data at least includes the coordinates of the metal block;

[0087] Determine alignment data between the first test point cloud data and the collected data according to the data timestamp;

[0088] Projecting the sixth three-dimensional coordinate of the metal block in the first test point cloud data in the alignment data onto a horizontal plane to obtain horizontal plane projection data;

[0089] The fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main lidar coordinate system is calculated according to the horizontal plane projection data, the coordinates of the metal blocks in the collected data in the alignment data, and the positions of each of the metal blocks in the calibration system.

[0090] In one embodiment, when the sensor includes an auxiliary laser radar, controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0091] Controlling the auxiliary laser radar to collect data from the calibration device to obtain second test point cloud data;

[0092] The second test point cloud data is matched with the first test point cloud data to obtain a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system.

[0093] In one embodiment, the calibration device includes a plurality of verification components, and determining the calibration prior information according to the first calibration data includes:

[0094] Performing spatial clustering segmentation on the first calibration data according to the position of each of the verification components in the calibration system to obtain third local point cloud data corresponding to each of the verification components;

[0095] The fifth identity information and the fifth coordinate information corresponding to each of the verification components are determined according to the third local point cloud data.

[0096] In one embodiment, determining the fifth identity information and the fifth coordinate information corresponding to each of the verification components according to the third local point cloud data includes:

[0097] The third local point cloud data is fitted to obtain structural information and three-dimensional coordinates of each verification component.

[0098] In one embodiment, the verification component is a verification sphere, and fitting the third local point cloud data to obtain structural information and three-dimensional coordinates of each of the verification components includes:

[0099] Spherical fitting is performed on the third local point cloud data to obtain radius information and three-dimensional coordinates of the calibration sphere.

[0100] In one embodiment, after obtaining calibration prior information according to the first data, and calibrating each of the sensors on the vehicle to be calibrated according to the calibration prior information and the second data, the method further includes:

[0101] The calibration parameters of each of the sensors are verified according to the radius information and three-dimensional coordinates of each of the verification balls.

[0102] In one embodiment, when the calibration parameters of the sensor include a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of each of the sensors are verified according to the radius information and the three-dimensional coordinates of each of the calibration spheres, including:

[0103] Obtaining the seventh three-dimensional coordinates of each calibration ball within the field of view of the main laser radar;

[0104] Performing coordinate transformation on the seventh three-dimensional coordinate according to the third coordinate transformation relationship to obtain a seventh three-dimensional coordinate to be compared;

[0105] Comparing the seventh three-dimensional coordinate to be compared with the seventh reference three-dimensional coordinate to determine whether the third coordinate transformation relationship is accurate;

[0106] The first reference three-dimensional coordinates are the three-dimensional coordinates obtained by the total station and corresponding to the verification component within the field of view of the main laser radar.

[0107] In one embodiment, when the calibration parameters of the sensor include a fourth coordinate transformation relationship between a camera coordinate system to be calibrated and a main laser radar coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration sphere, including:

[0108] Acquire first calibration point cloud data of a calibration sphere within the field of view of the main laser radar;

[0109] Back-projecting the first verification point cloud data onto the camera image according to the fourth coordinate transformation relationship;

[0110] Performing preset processing on the back-projected camera image to obtain a circular edge in the camera image, and determining a corresponding first circular fitting equation according to the circular edge;

[0111] The first circle fitting equation is compared with a reference circle fitting equation to determine whether the fourth coordinate transformation relationship is accurate.

[0112] In one embodiment, the calibration ball is a metal sphere, and when the calibration parameters of the sensor include the fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration ball, including:

[0113] Obtaining the millimeter-wave radar coordinates and angles of each calibration ball within the field of view of the millimeter-wave radar;

[0114] Acquire a seventh three-dimensional coordinate within the field of view of the main laser radar;

[0115] Projecting the millimeter wave radar coordinates and angles onto a two-dimensional plane of the main laser radar coordinate system according to the fifth coordinate transformation relationship;

[0116] The Euclidean distance is calculated according to the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates to determine whether the fifth coordinate transformation relationship is accurate.

[0117] In one embodiment, when the calibration parameters of the sensor include the sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration sphere, including:

[0118] Acquire the first calibration point cloud data of the calibration ball within the field of view of the main laser radar;

[0119] Acquire the second calibration point cloud data of the calibration ball within the field of view of the auxiliary laser radar;

[0120] The second verification point cloud data is transformed into the main laser radar coordinate system according to the sixth coordinate transformation relationship to obtain the second verification point cloud data to be verified;

[0121] Determine whether the sixth coordinate transformation relationship is accurate based on the first verification point cloud data and the second verification point cloud data to be verified.

[0122] In a second aspect, the present application further provides a vehicle sensor calibration system based on a calibration system, the calibration system comprising a total station, a plurality of calibration devices, and a computing device, the computing device comprising:

[0123] A first control unit, configured to control the total station to scan the calibration device to obtain first data when the vehicle to be calibrated meets a first calibration condition;

[0124] A second control unit, configured to receive second data acquired by the calibration device from each of the sensors on the vehicle to be calibrated when a second calibration condition is met;

[0125] A calibration unit is used to obtain calibration prior information based on the first data, and calibrate each of the sensors on the vehicle to be calibrated based on the calibration prior information and the second data to obtain calibration parameters of each sensor; the calibration prior information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two of the sensors or between the sensor and the total station.

[0126] In a third aspect, the present application further provides a vehicle sensor calibration device based on a calibration system, comprising:

[0127] Memory for storing computer programs;

[0128] The processor is used to implement the steps of the vehicle sensor calibration method based on the calibration system as described above when storing the computer program.

[0129] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the vehicle sensor calibration method based on the calibration system as described above are implemented.

[0130] The present application provides a vehicle sensor calibration method, system, device and medium based on a calibration system, which relates to the field of autonomous driving and solves the problem of low efficiency in vehicle sensor calibration. In this scheme, when the vehicle to be calibrated meets the first calibration condition, the total station is controlled to scan the calibration device to obtain the first data; when the second calibration condition is met, each sensor on the vehicle to be calibrated is controlled to collect the calibration device to obtain the second data; calibration prior information is obtained according to the first data, and each sensor on the vehicle to be calibrated is calibrated according to the calibration prior information and the second data. It can be seen that the use of a total station and multiple calibration devices in this application can greatly reduce the personnel requirements and time consumption of the calibration process. This can reduce the calibration cost, facilitate mass production, and improve production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the prior art and the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0132] Figure 1 A flow chart of a vehicle sensor calibration method based on a calibration system provided in this application;

[0133] Figure 2 A schematic diagram of a three-dimensional view of a calibration component for forward splicing provided in this application;

[0134] Figure 3 A schematic side view of a calibration component for forward splicing provided in this application;

[0135] Figure 4 A schematic diagram of a three-dimensional view of a reverse-stitched calibration component provided in this application;

[0136] Figure 5 A schematic side view of a reverse-jointed calibration component provided in this application;

[0137] Figure 6 A schematic diagram of a three-dimensional view of a hybrid spliced ​​calibration component provided in this application;

[0138] Figure 7 A schematic side view of a hybrid spliced ​​calibration component provided in this application;

[0139] Figure 8 Schematic diagram of the calibration board structure provided for this application;

[0140] Fig. 9 A schematic diagram of a partial image of the four corners of the calibration code provided in this application;

[0141] Fig.10 A schematic diagram of establishing the world coordinate system provided for this application;

[0142] Fig.11 An example diagram of an existing calibration plate provided for this application;

[0143] Fig.12 Another example diagram of an existing calibration plate provided for this application;

[0144] Fig.13 A top view of the surround stitching calibration layout provided for this application;

[0145] Fig.14 A schematic diagram of the projection transformation matrix selection provided for this application;

[0146] Fig.15 A schematic diagram showing the relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system provided for this application;

[0147] Fig.16 A schematic diagram of the layout of the calibration ball provided for this application;

[0148] Fig.17 Schematic diagram of projecting the point cloud provided by this application onto a camera image;

[0149] Fig.18 Schematic diagram of point cloud point extraction provided for this application;

[0150] Fig.19 A schematic diagram of a point cloud point fitting circle provided in this application;

[0151] Fig. 20 A schematic diagram of the vehicle parking area layout provided for this application;

[0152] Fig.21 A top view of the vehicle parking area layout provided for this application;

[0153] Fig. 22 Side view of the vehicle parking area layout provided for this application;

[0154] Fig.23 A structural block diagram of a vehicle sensor calibration system based on a calibration system provided in this application;

[0155] Fig.24 A structural block diagram of a vehicle sensor calibration device based on a calibration system provided in this application;

[0156] Fig.25 A structural block diagram of a computer-readable storage medium provided for this application. DETAILED DESCRIPTION

[0157] The core of this application is to provide a vehicle sensor calibration method, system, device and medium based on a calibration system. The use of a total station and multiple calibration devices can greatly reduce the personnel requirements and time consumption of the calibration process. This can reduce the calibration cost, facilitate mass production, and improve production efficiency.

[0158] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0159] Autonomous driving is developing rapidly and has become the focus of competition in the automotive and technology industries. Modular autonomous driving architecture design has become the mainstream solution because of its interpretability and ease of maintenance. The perception system is a key link in the autonomous driving system, acting as the eyes of the vehicle and providing environmental status information for path planning. Due to the complex environment, existing perception systems usually use multi-sensor fusion to improve information redundancy and complementarity to ensure safety. Commonly used sensors include lidar, cameras, and millimeter-wave radars, which need to be effectively combined according to the task and sensor characteristics. In order to ensure accurate perception, the sensor needs to be accurately calibrated to determine the relationship between internal and external parameters. However, existing calibration methods have problems in engineering applications. They require manual participation and are time-consuming and labor-intensive, making them difficult to mass produce. The solution process is long, and the initial value problem may cause problems with the accuracy of the results. The calibration results are difficult to understand, especially for non-professionals.

[0160] Before describing the embodiments of the present application, first of all, in order to obtain the best calibration accuracy and improve the calibration automation rate, first of all, according to the vehicle structure design parameters, the three-dimensional coordinates of the center of the main laser radar relative to the vehicle tire landing point can be obtained, and the position of the main laser mine center on the ground projection point can be further obtained. The total station is installed in the calibration site, and the center position of the total station is required to be consistent with the center of the main laser radar. The specific steps are as follows: Step 1: coincide the plumb point at the center position of the total station with the ground projection point of the main laser mine center; Step 2: make the height of the total station center relative to the ground consistent with the height of the main laser mine center relative to the ground; Step 3: make the x-axis direction of the total station coordinate system consistent with the x-axis direction of the main laser radar; Step 4: adjust the total station to a horizontal level.

[0161] Please refer to Figure 1 , Figure 1 A flow chart of a vehicle sensor calibration method based on a calibration system provided in this application, wherein the calibration system includes a total station and multiple calibration devices, and the method includes:

[0162] S11: When the vehicle to be calibrated meets the first calibration condition, controlling the total station to scan the calibration device to obtain first data;

[0163] Specifically, before calibrating the vehicle sensor, it is first necessary to confirm whether the vehicle to be calibrated meets the first calibration condition. Once the condition is met, the system will control the total station to scan the calibration device to obtain the first batch of data. The total station is an instrument used to measure and draw three-dimensional space, and point cloud data can be obtained by scanning the calibration device. The calibration system is repeatedly scanned by the total station to obtain dense point cloud data of the calibration site. These point cloud data contain specific information of the calibration device and can be used to determine the data information of the calibration device, that is, the calibration prior information. The calibration prior information is the reference data for sensor calibration, which can help determine the internal and external parameters of the sensor.

[0164] S12: When the second calibration condition is met, receiving second data collected by the calibration device through various sensors on the vehicle to be calibrated;

[0165] In this step, after confirming that the vehicle to be calibrated meets the second calibration condition, the system will receive the second data acquired by the calibration device through the collection operation of each sensor on the vehicle to be calibrated. These data can reflect the performance of the sensor in the actual use scenario and are used for subsequent sensor calibration.

[0166] Among them, the specific methods of receiving the second data may include the following two methods: first, directly controlling each of the sensors to collect the calibration device to obtain the second data; second, passively receiving the second data sent by the sensor.

[0167] S13: obtaining calibration prior information according to the first data, and calibrating each sensor on the vehicle to be calibrated according to the calibration prior information and the second data to obtain calibration parameters of each sensor; the calibration prior information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two sensors or between a sensor and a total station.

[0168] In this step, after obtaining the first batch of data, the system will obtain calibration prior information based on these data. At the same time, combined with the second batch of data and calibration prior information, calibration operations are performed on each sensor on the vehicle to be calibrated. Calibration prior information refers to the data information obtained by the total station scanning calibration device, which can be used as a reference for sensor calibration. Through the calibration operation, the measurement error of the sensor can be calibrated to improve the accuracy and reliability of the sensor.

[0169] This embodiment receives the calibration device data obtained by the total station scan and the data collected by the sensor, performs calculations and analysis to obtain the calibration parameters of the sensor, and finally realizes the accurate calibration of the sensor, which can improve the perception ability and safety of the autonomous driving system and provide effective support for the development of autonomous driving technology. In addition, the calibration method in this embodiment is automatically executed by the processor without human participation, saving a lot of time and human resources, improving work efficiency, and can be applied to mass production needs.

[0170] In one embodiment, the process of determining whether the vehicle to be calibrated meets the first calibration condition includes:

[0171] Determine whether the vehicle to be calibrated is parked in the preset area;

[0172] If the vehicle stops in the preset area, it is determined that the vehicle to be calibrated meets the first calibration condition; otherwise, it is determined that the first calibration condition is not met.

[0173] This embodiment describes a method for determining whether a vehicle to be calibrated meets a first calibration condition. The method includes the following steps: determining whether the vehicle to be calibrated is parked in a preset area; if so, determining that the vehicle to be calibrated meets the first calibration condition; otherwise, determining that the first calibration condition is not met.

[0174] This embodiment can ensure that the vehicle to be calibrated must meet the preset conditions before the sensor calibration is performed. This can ensure the accuracy and reliability of the calibration. By judging whether the vehicle to be calibrated is parked in the preset area, it can be ensured that the vehicle is in a stable state during the calibration. This is because in the preset area, the vehicle is stationary and not subject to external interference, which can avoid errors in sensor data.

[0175] In addition, this embodiment can also reduce the risk and uncertainty in the calibration process. If the vehicle to be calibrated does not stop in the preset area, it means that it does not meet the first calibration condition, and it can be judged that it is not suitable for sensor calibration. This can avoid inaccurate or invalid calibration results due to unstable vehicle status.

[0176] In one embodiment, the process of determining whether the vehicle to be calibrated meets the first calibration condition includes:

[0177] Determining whether the vehicle to be calibrated is being calibrated for the first time according to the vehicle information of the vehicle to be calibrated;

[0178] If it is the first calibration, it is determined that the vehicle to be calibrated meets the first calibration condition, otherwise, it is determined that the first calibration condition is not met. In one embodiment, when it is determined that the vehicle is not calibrated for the first time, it also includes:

[0179] The first data corresponding to the vehicle to be calibrated and stored in the database is called.

[0180] This embodiment describes the process of determining whether the vehicle to be calibrated meets the first calibration condition. First, it is determined whether the vehicle to be calibrated is being calibrated for the first time based on the vehicle information of the vehicle to be calibrated. If the vehicle to be calibrated is being calibrated for the first time, it is determined that the vehicle to be calibrated meets the first calibration condition. If the vehicle to be calibrated is not being calibrated for the first time, it is determined that the first calibration condition is not met.

[0181] This embodiment can reduce the need for multiple calibrations of vehicles. By determining whether the vehicle to be calibrated is being calibrated for the first time, unnecessary repeated calibration operations can be avoided. Only during the first calibration is it necessary to perform a total station scanning operation on the calibration device to obtain the first data. In this way, the calibration efficiency can be improved and the waste of time and resources can be reduced. In addition, for vehicles that have been calibrated, the previously obtained calibration prior information can be directly used, which greatly speeds up the sensor calibration process.

[0182] In summary, this embodiment improves the efficiency and accuracy of calibration without affecting the calibration quality, thereby saving time and resources.

[0183] In one embodiment, the process of determining whether the second calibration condition is met includes:

[0184] Determine whether a calibration start instruction is received;

[0185] If received, it is determined that the second calibration condition is met; otherwise, it is determined that the second calibration condition is not met.

[0186] The present embodiment relates to a process for determining whether the second calibration condition is met. This determination process includes the following steps: Determine whether a calibration start instruction is received: In the control unit or other related equipment on the vehicle to be calibrated, determine whether a calibration start instruction sent by the calibration system is received. If a calibration start instruction is received: If a calibration start instruction is received, then it is determined that the second calibration condition is met. This means that the various sensors on the vehicle to be calibrated can start collecting data from the calibration device and can obtain the second data. Otherwise, it is determined that the second calibration condition is not met: If the calibration start instruction is not received, then it is determined that the second calibration condition is not met. This means that the various sensors on the vehicle to be calibrated cannot collect data from the calibration device and cannot obtain the second data.

[0187] It can be seen that the judgment process in this embodiment is based on whether the calibration start instruction is received. Only when the calibration start instruction is received can the second calibration condition be determined to be met, thereby starting the acquisition operation of the sensor. If the calibration start instruction is not received, the acquisition operation of the sensor cannot be performed, so it is determined that the second calibration condition is not met.

[0188] In one embodiment, obtaining calibration prior information according to the first data includes:

[0189] Eliminate data that does not meet preset requirements from the first data to obtain first calibration data;

[0190] Calibration prior information is determined according to the first calibration data.

[0191] Specifically, the first data is point cloud data obtained by scanning each calibration device in the calibration system through a total station. The calibration prior information refers to the data information of the calibration device. When implementing this method, it is first necessary to receive the first data and process it. The processing step includes eliminating data that does not meet the preset requirements from the first data, thereby obtaining the first calibration data. Through this step, data that does not meet the calibration requirements can be excluded to ensure the accuracy and reliability of the first calibration data.

[0192] Next, the calibration prior information is determined based on the first calibration data. By analyzing and processing the first calibration data, data information of the calibration device can be obtained. This information can be used as prior knowledge for subsequent calculation of calibration parameters and calibration of sensors.

[0193] In this embodiment, by processing the first data, data that does not meet the preset requirements are eliminated, thereby avoiding the negative impact of these data on the calibration results. At the same time, by determining the calibration prior information based on the first calibration data, the existing data information can be better utilized to improve the calculation accuracy of the calibration parameters, thereby achieving accurate calibration of each sensor.

[0194] In one embodiment, removing data that does not meet preset requirements from the first data to obtain first calibration data includes:

[0195] The point cloud data not including the calibration device is eliminated from the first data to obtain the first calibration data.

[0196] In one embodiment, removing point cloud data not including the calibration device from the first data to obtain first calibration data includes:

[0197] The point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane in the calibration system are identified to eliminate the point cloud data corresponding to the ceiling plane and / or the ground plane and / or the surrounding wall plane in the first data.

[0198] This embodiment describes a method for processing first data to obtain first calibration data. The purpose of this embodiment is to remove point cloud data corresponding to the ceiling plane and / or the ground plane and / or the surrounding wall planes in the first data. By removing these data, more accurate first calibration data can be obtained for determining calibration prior information.

[0199] This embodiment can be implemented by identifying the point cloud data of the ceiling plane, the ground plane and / or the surrounding wall plane in the calibration system. These planes can be obtained by scanning the point cloud data of each calibration device in the calibration system with a total station. During the recognition process, image processing or point cloud processing algorithms can be used to analyze the point cloud data to determine the position and shape of the ceiling plane, the ground plane and / or the surrounding wall plane. By identifying and eliminating the point cloud data corresponding to these planes, the purpose of eliminating the point cloud data related to these planes can be achieved.

[0200] In summary, more accurate first calibration data can be obtained through the method in this embodiment, thereby improving the precision and accuracy of sensor calibration. This helps to ensure that more reliable and accurate results are obtained when calculating the calibration parameters of each sensor, thereby effectively calibrating each sensor.

[0201] In one embodiment, the process of identifying the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system includes:

[0202] Obtaining a first distance from the center position of the total station to the ceiling plane and / or the ground plane and / or the surrounding wall planes;

[0203] Randomly select at least three first point clouds from the first data, and calculate the three-dimensional space plane where the first point clouds are located; wherein the difference between the distance from the first point cloud to the center position of the total station and the first distance is not greater than the first preset distance;

[0204] Calculate the second distance from each point cloud in the first calibration data to the three-dimensional space plane, and use the data corresponding to the point cloud whose difference between the second distance and the first distance is not greater than the second preset distance as the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane.

[0205] In this embodiment, since the installation position information of the total station is known (including x, y, z, whether it is horizontal, etc.), it is easy to use the positional relationship between the surrounding walls, ground, ceiling and the center of the total station of the calibration system as prior information. Therefore, in this embodiment, it is first necessary to obtain the first distance from the center position of the total station to the ceiling plane and / or the ground plane and / or the surrounding wall plane. Then, at least three first point clouds are randomly selected from the first data, and the three-dimensional space plane where these point clouds are located is calculated. The point cloud here refers to the data points obtained by the total station scanning each calibration device in the calibration system. Next, the distance from the first point cloud to the center position of the total station is calculated and compared with the first distance. If this difference is not greater than the first preset distance, the point cloud data can be used as the data of the ceiling plane and / or the ground plane and / or the surrounding wall plane. Finally, the second distance from each point cloud in the first calibration data to the three-dimensional space plane is calculated and compared with the first distance. If this difference is not greater than the second preset distance, the corresponding point cloud data can be used as the data of the ceiling plane and / or the ground plane and / or the surrounding wall plane.

[0206] The first preset distance and the second preset distance may be the same or different, and this application does not make any special limitation thereto.

[0207] In this embodiment, by measuring the total station and calculating the plane where the point cloud is located, the point cloud data corresponding to the ceiling plane and / or the ground plane and / or the surrounding wall plane are identified and excluded, thereby obtaining the first calibration data. By processing and comparing these data, the point cloud data that meets the preset conditions can be used as the calculation calibration prior information. These calibration prior information can be used to calculate the calibration parameters of each sensor to calibrate each sensor.

[0208] In one embodiment, it also includes:

[0209] Repeat the step of identifying the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system;

[0210] Determining whether the difference meets the iteration termination condition according to the difference between the number of point clouds in the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration process and the number of point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the previous iteration process;

[0211] If it meets the requirements, the iteration is terminated, and the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes obtained in the current iteration process are used as the final point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes.

[0212] This embodiment describes an iterative process for identifying point cloud data of a ceiling plane and / or a ground plane and / or a surrounding wall plane in a calibration system. This process includes multiple cycles of executing the identification step, and each iteration determines whether an iteration termination condition is met by comparing the difference between the number of point cloud data obtained in the current iteration process and the number of point cloud data obtained in the previous iteration process.

[0213] In each iteration process, firstly, according to the previously obtained point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane, a point cloud data recognition step is performed. This step uses the distances from the center position measured by the total station to these planes and at least three point clouds selected from the first data to calculate the three-dimensional space plane and the distance from each point cloud to the plane. Finally, point cloud data with a small difference from the preset distance is selected from these distances and used as the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall plane in the new iteration.

[0214] After each iteration, the difference between the number of point cloud data obtained in the current iteration and the number of point cloud data obtained in the previous iteration is compared, and it is determined whether the difference meets the preset iteration termination condition. For example, if the difference is not greater than the preset number, it is considered that the iteration process has converged and the iteration is terminated. Alternatively, if the ratio of the number of point cloud data obtained in the current iteration to the number of point cloud data obtained in the previous iteration is not greater than the preset ratio, it is considered that the iteration process has converged and the iteration is terminated. At the same time, the point cloud data obtained in the current iteration is used as the point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes.

[0215] Through this iterative approach, the recognition of point cloud data of the ceiling plane and / or the ground plane and / or the surrounding wall planes in the calibration system can be gradually improved and optimized, thereby obtaining a more accurate and reliable calibration result.

[0216] For example, assuming that the calibration site point cluster is Φ, the specific steps are as follows:

[0217] According to the spatial distribution characteristics of point clouds, in order to improve the efficiency of plane fitting, the interference effects are sorted and eliminated. The ceiling has the least interference, followed by the ground, and finally the surrounding walls. Therefore, the order of plane extraction is: ceiling point cloud → ground → surrounding walls;

[0218] Perform point cloud recognition on the ceiling plane. Set the distance z from the center of the total station to the ceilingTop as prior information.

[0219] Step 1: Set up the point cloud Randomly select the content Select N (N ≥ 3) points of the plane with the first preset distance (in meters), z is the vertical coordinate of the point cloud, and the distance between the N points is required to be greater than the threshold L Top (unit is meter), calculate the three-dimensional space plane formed by these N points

[0220] It should be noted that although the total station is adjusted to a horizontal level, it is difficult to ensure that the six planes in the calibration system are completely horizontal or vertical, that is, there are certain errors. For example, if the ceiling is not horizontal and has a large slope, the effect of identifying the plane based on a single condition will be very poor. Therefore, by setting a certain threshold On the one hand, it can greatly reduce the number of subsequent iterations, and on the other hand, it has a certain adaptability to non-horizontal or non-vertical planes. It should also be noted that L Top The setting of , ensures that the spatial distribution of N points is not too close. If the randomly selected N points are too close, it is easy to fit a plane with large errors.

[0221] Step 2: Calculate the point cloud set All points to the three-dimensional space plane distance, and according to the distance threshold (i.e. the second preset distance) (in meters) to obtain the distance between the plane and the three-dimensional space Close point set

[0222] Step 3: Calculate whether the iteration termination condition is met (in, is the number of point cloud data obtained during the i+1th iteration, is the number of point cloud data obtained in the i-th iteration process). If J>T top (T top is the iteration threshold, such as 0.9), then is the ceiling plane point cloud P Top Otherwise, let (by gradually reducing A more accurate plane point cloud can be obtained) and the calculation starts again from the first step.

[0223] Among them, when the first step is executed for the first time, a point cloud set Φ is randomly selected to satisfy The N (N ≥ 3) points of the condition, that is

[0224] Further, identify the ground plane point cloud PBottom And the plane point cloud of the surrounding walls (P Left , P Right , P Front , P Back The principle of ) is similar to the principle of identifying the ceiling plane point cloud mentioned above, and this application will not repeat it here. Taking the front wall as an example, the distance x from the center of the total station to the front wall front As prior information, the corresponding point cloud can be obtained by using the ceiling point cloud computing method.

[0225] In one embodiment, the calibration device includes a plurality of calibration components, and determines the calibration prior information according to the first calibration data, including:

[0226] Performing spatial clustering segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain first local point cloud data corresponding to each calibration component;

[0227] The first identity information and the first coordinate information of each calibration component are determined according to the first local point cloud data.

[0228] In this embodiment, when the calibration device includes multiple calibration components, these calibration components have different positions in the calibration system. First, according to the point cloud data in the first data, it can be spatially clustered and segmented. This means that the points in the point cloud data can be grouped according to their positions, so that the points in the same group are closer to each other and farther away from the points in other groups. Through this segmentation process, the first local point cloud data corresponding to each calibration component is obtained. Next, the first identity information and first coordinate information of each calibration component can be determined by analyzing the first local point cloud data. By analyzing the point cloud data of each calibration component, the characteristics or shape of each calibration component can be identified and compared with the known calibration components to confirm their identity information. At the same time, the coordinate information (such as three-dimensional coordinate information) of each calibration component in the calibration system can also be determined by analyzing the position information in the point cloud data.

[0229] It should be noted that the first identity information is used to represent the identity information of the calibration component. When the calibration component includes multiple calibration plates, and each calibration plate has its own corresponding identity information, the first identity information can be a combination of the identity information of the multiple calibration plates. The first coordinate information is used to represent the spatial position of the calibration component in the calibration system, such as the three-dimensional coordinates of the center position of the calibration component in the calibration system.

[0230] Through the above process, the first identity information and the first coordinate information of each calibration component can be determined, and these information will be used to subsequently calculate the calibration parameters of each sensor to calibrate each sensor. In this way, the vehicle sensor calibration method can effectively and accurately calibrate each sensor according to the data information of the calibration device.

[0231] In one embodiment, each calibration component includes at least two calibration plates, and determining the first identity information and the first coordinate information of each calibration component according to the first local point cloud data includes:

[0232] Determine second local point cloud data corresponding to each calibration plate from the first local point cloud data;

[0233] The second identity information and the second coordinate information corresponding to each calibration plate are determined according to the second local point cloud data.

[0234] This embodiment illustrates the composition of a calibration component in a vehicle sensor calibration method and the process of determining the corresponding identity information and coordinate information. Each calibration component in the method includes at least two calibration plates. First, the first identity information and first coordinate information of each calibration component are determined based on the first local point cloud data. This can be achieved by determining the second local point cloud data corresponding to each calibration plate from the first local point cloud data. Then, based on the second local point cloud data, the second identity information and second coordinate information corresponding to each calibration plate can be determined.

[0235] It should be noted that the second identity information is used to characterize the identity information of the calibration plate. If the calibration plate includes multiple calibration codes, the identity information of the calibration plate can be a combination of the identity information of the calibration codes. In this embodiment, the combination of the second identity information of the calibration plate included on the calibration component is the first identity information of the calibration component. The second coordinate information is used to characterize the position of the calibration plate in the calibration system. If the three-dimensional coordinates of the calibration component and the relative position of the calibration plate and the calibration component are known, the position of each calibration plate can be known.

[0236] In this way, the calibration parameters of the sensor can be determined based on the calibration components on the vehicle and the data collected by the sensor. This ensures that each sensor can accurately perceive the surrounding environment and provide accurate data.

[0237] In one embodiment, each calibration component includes at least two non-coplanar calibration plates, and determining the second local point cloud data corresponding to each calibration plate according to the position of each calibration plate in each calibration component and the first local point cloud data includes:

[0238] Plane fitting is performed on each calibration plate to extract the second local point cloud data corresponding to each calibration plate.

[0239] This embodiment describes a step of processing the first local point cloud data to determine the second local point cloud data corresponding to each calibration plate. The processing step includes performing plane fitting on each calibration plate to extract the second local point cloud data corresponding to each calibration plate. In this embodiment, for each calibration plate, it is first found in the first local point cloud data. Then, the point cloud data on each calibration plate is processed using a plane fitting technique, and the plane closest to the surface of the calibration plate is found. Plane fitting can fit a plane by minimizing the distance between the point and the plane. In this embodiment, a plane fitting technique is used to extract the second local point cloud data corresponding to each calibration plate, that is, the point cloud data located near the surface of the calibration plate.

[0240] By fitting the plane of each calibration plate, the second local point cloud data corresponding to each calibration plate can be determined, that is, the point cloud data located at a position close to the calibration plate on the plane obtained by plane fitting.

[0241] Through this processing step, the second local point cloud data corresponding to each calibration plate can be extracted from the first local point cloud data for subsequent calibration parameter calculation and sensor calibration, which can enhance the accuracy and reliability of the calibration, thereby improving the performance and precision of the vehicle sensor.

[0242] In one embodiment, plane fitting is performed on each calibration plate to extract second local point cloud data corresponding to each calibration plate, including:

[0243] Obtaining a first height between the center of each calibration plate and the ground, randomly selecting at least three second point clouds from the first local point cloud, and calculating a second three-dimensional space plane where the second point clouds are located;

[0244] Calculating a third distance between each point cloud in the second local point cloud data and the second three-dimensional space plane;

[0245] The data corresponding to the point cloud whose difference between the third distance and the first height is not greater than the third preset distance is used as the second local point cloud data corresponding to the calibration plate.

[0246] This embodiment describes a method for plane fitting of each calibration plate to extract the second local point cloud data corresponding to each calibration plate. The specific steps include: obtaining the first height between the center of each calibration plate and the ground. Randomly selecting at least three point clouds from the first local point cloud data as the second point cloud. Using the selected second point cloud to calculate a three-dimensional space plane. For each point cloud in the first local point cloud data, calculate its vertical distance to the second three-dimensional space plane, that is, the third distance. Compare the difference between the third distance and the first height, and filter out the point cloud data that satisfies the difference not greater than the third preset distance. These point cloud data are regarded as the second local point cloud data corresponding to the calibration plate.

[0247] For example, only the calibration component point cloud dataset is included Clustering is performed based on the spatial distance of the calibration components (since the calibration components are spaced far apart, they are easy to segment in space), and the point cloud data set of each calibration component (i.e., the first local point cloud data) is first obtained. ( is the point cloud data set corresponding to the kth calibrated component, M2 is the total number of calibrated components);

[0248] Since each calibration component is composed of S (usually S is 3 or 4) non-coplanar calibration plates, it is possible. Using the plane fitting principle, extract the point cloud data of each calibration plate from top to bottom (or from bottom to top). The specific steps are as follows:

[0249] Step 1: Calibrate the component point cloud dataset According to the center height z of the calibration plate i (i∈S) information (i.e. the first height mentioned above), randomly select ( The plane selection error threshold is in meters) and the distance between the N points is required to be greater than the threshold L. a (Usually L a <0.7W, W is the calibration width, in meters), calculate the three-dimensional space plane formed by these N points

[0250] It should be noted that the calibration component is composed of multiple calibration plates. The installation height of each calibration plate (i.e., the center height of the calibration plate) is known. Through this prior information, the plane fitting efficiency can be accelerated and the plane fitting accuracy can be improved. Directly randomly find three second point clouds to fit the plane. If the three second point clouds are distributed on different calibration plates, this kind of invalid calculation is likely to occur, and fall into more invalid cycles. Therefore, this first step can not only accurately fit the plane of the calibration plate, but also improve the efficiency of plane fitting.

[0251] Step 2: Calculation All points to the 3D plane The distance (third distance), and according to the distance threshold (Unit: meter) Get the plane Close point set

[0252] Step 3: Calculate the iteration termination condition If J>T (T is the threshold, such as 0.9), then is the point cloud of the i-th calibration plate on the k-th calibration component; otherwise, let Restart the calculation from the first step.

[0253] Through the above steps, the point cloud of the i-th calibration plate can be obtained Then from Eliminate Repeat the above steps to obtain the point cloud of each calibration plate on the kth calibration component one by one

[0254] Through the above steps, the second local point cloud data corresponding to each calibration plate that meets the requirements can be extracted from the first local point cloud data, thereby achieving the goal of plane fitting for each calibration plate. This method can accurately extract the shape and position information of each calibration plate, providing valuable data reference for subsequent sensor calibration.

[0255] In one embodiment, the calibration plate includes a calibration carrier plate and a plurality of marking codes, wherein the plurality of marking codes are fixed on the calibration carrier plate, the marking codes carry two-dimensional code information for identification by the sensor, wherein the identity information corresponding to the two-dimensional code information of all the marking codes is different;

[0256] Determining second identity information corresponding to each calibration plate according to the second local point cloud data includes:

[0257] Convert the second local point cloud data into a grayscale image;

[0258] The grayscale image is segmented according to the intervals between the marker codes to obtain a local image of each marker code;

[0259] Perform two-dimensional code recognition on each partial image to determine the first ID information of each mark code, and determine the second identity information of the calibration plate according to the first ID information of each mark code on each mark plate.

[0260] This embodiment describes a calibration board design in a vehicle sensor calibration system. The calibration board includes a calibration carrier and multiple marking codes. These marking codes are fixed on the calibration carrier, and each marking code carries a two-dimensional code information for sensor recognition. It should be noted that the identity information corresponding to the two-dimensional code information of each marking code is different.

[0261] On this basis, the specific implementation method of determining the second identity information corresponding to each calibration plate according to the second local point cloud data is as follows: converting the second local point cloud data into a grayscale image to facilitate subsequent image processing and analysis. According to the interval between the marker codes (this is a known condition), the grayscale image is segmented to obtain a local image of each marker code, and each marker code is separated for separate processing. Then, each local image is subjected to two-dimensional code recognition. By decoding and analyzing the two-dimensional code image, the first ID information of each marker code can be obtained. This first ID information can be used as a unique identity identifier of the marker code. Finally, according to the first ID information of each marker code on each marker board, the second identity information of the calibration plate is determined (for example, the identity information of the marker board is a combination of the first ID information of several marker codes on the marker board in a preset manner). This means that by integrating and comparing the first ID information of each marker code, the identity information of the entire calibration plate can be determined. Since the identity information corresponding to the two-dimensional code information of each marker code is different, such a comparison can ensure that each calibration plate can be uniquely identified.

[0262] In one embodiment, after converting the second local point cloud data into a grayscale image, the method further includes: performing filtering processing on the grayscale image.

[0263] The method in this embodiment can calibrate the sensor more accurately and improve the stability and accuracy of the calibration system.

[0264] Here, the calibration member involved in the embodiment is described. Specifically, the calibration member may include at least two calibration plates and a connecting member; the connecting member is fixed between two adjacent calibration plates and is used to connect the two calibration plates. Figure 2 As shown, multiple calibration plates, such as the lower edge of calibration plate 1 and the upper edge of calibration plate 2, are connected by a connector to obtain a calibration component. The calibration component in the present application includes multiple calibration plates at the same location, such as the upper, lower, left, and right, so as to increase the number of calibrations of the horizontal and / or vertical field of view (FOV) of sensors on the vehicle, such as cameras, laser radars, and millimeter-wave radars, at the same calibration position, thereby improving the calibration accuracy.

[0265] Furthermore, in the calibration component provided by the present application, the connecting member is also used to control the angle between any one of the two calibration plates and the vertical plane where the two calibration plates are located. Specifically, the connecting member in the calibration component provided by the present application can also adjust the angle between multiple connected calibration plates. Figure 3As shown, the angles between the calibration plates 1 and 2 and the plumb plane where the calibration component is located can be adjusted according to user needs. For example, based on the consideration that the marking codes on the calibration plates 1 and 2 are easy for sensors to identify, by adjusting the connecting parts, the marking plate 2 remains parallel to the plumb plane where the marking component is located, and the marking plate 1 is adjusted to have an angle with the plumb plane, for example, 30 degrees.

[0266] Furthermore, since the calibration component provided by the present application can adjust the angle between multiple connected calibration plates according to user needs, it is convenient for various sensors on the vehicle to effectively identify the calibration plates. In addition, the multiple calibration plates on the calibration component have angles, which increases non-coplanar points and effectively improves the calibration accuracy.

[0267] Furthermore, in the calibration component provided by the present application, the front faces of the two calibration plates are located on the same side of the calibration component. Figure 3 As shown, the calibration component provided by the present application can forward splice multiple calibration plates. At this time, the front of the calibration plate 1 and the calibration plate 2 are located on the same side of the calibration component, and the angle between the calibration plate 1 and the plumb plane is adjusted according to the sensor calibration requirements of the vehicle, such as 30 degrees. When the sensor of the vehicle needs to be calibrated by the sensor, the QR code information on the same side of the calibration plate 1 and the calibration plate 2 can be obtained for calibration between different sensors on the vehicle. Because the calibration component provided by the present application increases the non-coplanar points recognized by the calibration plate, the calibration accuracy of the sensor is improved.

[0268] Furthermore, in the calibration component provided by the present application, the front surfaces of the two calibration plates are located on both sides of the calibration component. Figure 4-5 As shown, the calibration component provided by the present application can also reversely splice multiple calibration plates. Figure 4 and Figure 5 They are respectively a stereoscopic view and a side view of the calibration component when reversely spliced. At this time, the front of calibration plate 1 and calibration plate 2 are located on both sides of the calibration component, and the angle between calibration plate 1 and the plumb plane is adjusted according to the sensor calibration requirements of the vehicle, such as 30 degrees. When the vehicle's sensor needs to be calibrated by the sensor, the QR code information on the front of calibration plate 1 and the QR code information on the back of calibration plate 2 can be obtained for calibration between different sensors on the vehicle.

[0269] In addition, if Figure 6 and Figure 7 As shown, the calibration component provided by the present application can also be obtained by combining a plurality of calibration plates spliced ​​in the forward direction and calibration plates spliced ​​in the reverse direction. Figure 6 and Figure 7They are respectively a stereoscopic view and a side view of the calibration component obtained by combining multiple different splicing methods. Specifically, the calibration component provided in this application can be spliced ​​with multiple calibration plates at a certain angle according to the calibration requirements of sensors such as cameras, laser radars, and millimeter-wave radar fields of view. Figure 2 and Figure 4 The two calibration plates spliced ​​together are combined to obtain Figure 6 At this time, when calibrating the vehicle sensor, it is necessary to identify the angle, mark code and other information of the combination of the positive and negative calibration plates or the calibration unit, which further improves the calibration accuracy of the vehicle sensor calibration.

[0270] It should be emphasized that the present application does not limit the specific number of calibration plates in the calibration component. Users can increase or decrease the number of calibration plates according to their own needs, based on the number of sensors and field of view angles on different vehicles, so as to meet the sensor calibration requirements of different vehicles.

[0271] On the basis of the above-mentioned implementation manner, since the calibration plates in the calibration component provided in the present application face different front and back sides of the vehicle sensor, the non-coplanar differences of the reference objects used for sensor calibration are further increased, thereby improving the calibration accuracy of the vehicle sensor calibration.

[0272] In one embodiment, determining the second coordinate information corresponding to each calibration plate according to the second local point cloud data includes:

[0273] Extract the four sides of the circumscribed rectangle of the partial image of each logo code, and determine the intersection of two adjacent sides as the key point;

[0274] Obtaining the first image coordinates of each key point, where the first image coordinates represent the position of the key point on the calibration plate;

[0275] The first three-dimensional coordinates corresponding to each key point are determined according to the position of the marking code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0276] Correspondingly, the specific implementation method of determining the second coordinate information corresponding to each calibration plate according to the second local point cloud data is: extract the four sides of the circumscribed rectangle of the local image of each marker code; determine the intersection of two adjacent sides as the key point, and obtain the first image coordinates of each key point, that is, the position of the key point on the calibration plate, and calculate the first three-dimensional coordinates corresponding to each key point according to the position of the marker code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

[0277] In this embodiment, by extracting the four sides of the circumscribed rectangle of the partial image of the marker code to determine the key points, the error caused by image noise can be effectively reduced and the calibration accuracy can be improved. By using the marker code as a feature point for recognition, the robustness of the system to changes in the calibration plate posture and sensor observation angle can be improved. By using the two-dimensional code technology, the first ID information of the marker code can be quickly and accurately obtained, thereby reducing the time and cost of manual processing.

[0278] The first image coordinates represent the position of each key point on the calibration plate, and the first reference three-dimensional coordinates are known. If the position of the calibration plate and the first image coordinates of the key point on the calibration plate are known, the first three-dimensional coordinates corresponding to each key point can be determined. The set of all first three-dimensional coordinates is the set of three-dimensional coordinates of all key points.

[0279] In summary, the method described in this embodiment can realize the calibration of vehicle sensors by identifying key points on the calibration plate and their corresponding first three-dimensional coordinates, and has the beneficial effects of high precision, high robustness and fast calibration.

[0280] In one embodiment, when there is no three-dimensional coordinate corresponding to the first image coordinate of the key point in the first reference three-dimensional coordinate, the method further includes:

[0281] Obtain the three-dimensional coordinates corresponding to several reference points adjacent to the key point;

[0282] The first three-dimensional coordinates corresponding to the key point are determined according to the three-dimensional coordinates of a plurality of reference points.

[0283] When there is no directly corresponding point cloud for a key point, the pixel points that are closest to the key point in the four directions of up, down, left, right, and space and have corresponding point clouds can be used to find the pixel distances in the four directions of the image and the plane fitting method. The angle information of the key point is used to perform bilinear interpolation and plane angle offset calibration to obtain the three-dimensional coordinates of the key point.

[0284] In one embodiment, a metal block is further provided on the calibration plate, and determining the second identity information corresponding to each calibration plate according to the second local point cloud data further includes:

[0285] The third identity information of the metal block is determined according to the first ID information of each identification code.

[0286] In the automatic calibration system of autonomous driving sensors, when the sensor includes a millimeter-wave radar, a metal block needs to be set on the calibration board. Because the metal block can simulate target objects on the real road, such as vehicles, pedestrians or static obstacles, and provide reflection of radar signals. During the calibration process, the millimeter-wave radar emits radar waves, and the metal block can reflect these waves and provide parameters similar to the actual target object. By setting the metal block on the calibration board, the millimeter-wave radar can identify the metal block at different angles and positions, thereby helping to determine the exact position and angle of the sensor.

[0287] In one embodiment, determining the second coordinate information corresponding to each of the calibration plates according to the second local point cloud data further includes:

[0288] The three-dimensional coordinates of the metal block are determined according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to each of the marking codes.

[0289] In this embodiment, the first step is to obtain the second local point cloud data. The second local point cloud data refers to the part related to the calibration plate extracted from the entire point cloud map. Next, the second coordinate information of the metal block on each calibration plate is determined based on the second local point cloud data. This means that by analyzing the second local point cloud data, the position of each metal block on the calibration plate can be determined. Finally, the three-dimensional coordinates of the metal block are determined based on the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to each mark code. This means that by matching the position of the metal block with the key points of the mark code, the position of the metal block in three-dimensional space can be determined.

[0290] In one embodiment, the metal block is fixed at the center of the calibration carrier, a plurality of marking codes are fixed on the calibration carrier and surround the metal block, and the vertices of the plurality of marking codes close to the metal block are connected to the vertices of the metal block.

[0291] In one embodiment, the three-dimensional coordinates of the metal block are determined based on the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to each of the marking codes, including: determining the third identity information of the metal block based on the first ID information of each marking code.

[0292] If the millimeter wave radar sensor needs to be calibrated, a metal block can be set in the calibration plate. When making the calibration plate, the calibration code ID information of the four neighboring areas of the metal block is known a priori information. Therefore, the third identity information of the metal block can be encoded clockwise by the four first ID information of the calibration code of the four neighboring areas, such as ID lt ID rt ID rd ID ld .

[0293] The third identity information of each metal block is different. When there is a calibration code on each calibration plate, the third identity information of the metal block can be a combination of the first ID information of the calibration code in the first manner. The second identity information of the calibration plate in the above embodiment can be a combination of the first ID information in the second manner, or a combination of each first ID information and the third identity information of the metal block, etc., which is not limited in this application.

[0294] In one embodiment, determining the second coordinate information corresponding to each calibration plate according to the second local point cloud data further includes:

[0295] Determine the three-dimensional coordinates of the four corner points of the metal block according to the first three-dimensional coordinates of the key points corresponding to the respective marking codes and the positional relationship between the metal block and the plurality of marking codes;

[0296] The three-dimensional coordinates of the center of the metal block are determined according to the three-dimensional coordinates of the four corner points of the metal block.

[0297] In this embodiment, after obtaining the three-dimensional coordinates of the key points of the calibration code, the three-dimensional coordinates of the four corner points of the metal block can be obtained according to the relationship between the four corners of the metal block and the vertical angles of the key points of the calibration code. Then, the three-dimensional coordinates of the center of the metal block can be calculated according to the three-dimensional coordinates of the four corner points.

[0298] In the calibration plate structure described in the above embodiment, the calibration carrier is a plate body for carrying the mark code and the metal block, and the material can be selected from non-metallic materials that are not easily deformed, such as acrylic plates. The metal block located in the center of the calibration carrier is used for calibration of millimeter-wave radars and other sensors installed on autonomous driving vehicles. The mark code is a two-dimensional code that can encode identity information, such as a binary square reference mark (Augmented Reality University of Cordoba, ArUco) for camera posture estimation, which is convenient for sensors such as laser radars installed on autonomous driving vehicles to identify each mark code on the mark board. Among them, the size and number of the mark codes can be adjusted according to user needs based on factors such as camera focal length and laser radar field of view. In order to facilitate the sensor to distinguish different mark codes, all mark codes on the calibration plate cannot be repeated, and the identity information of each mark code is unique.

[0299] It should be emphasized that the metal block in the technical solution provided in the present application is used for obtaining spatial homonymous points (the same as the same key points described in the following embodiments) when calibrating the millimeter-wave radar and the main laser radar on the vehicle, thereby achieving the calibration effect of the millimeter-wave radar and the main laser radar, and the ID information carried in the mark code and the QR code information on the mark code is used for subsequent automatic calibration actions between the camera internal parameters and the camera external parameters. The ID information carried in the mark code and the QR code information on the mark code can also be used for subsequent automatic calibration actions between the camera and the laser radar.

[0300] Among them, the points with the same name are used to unify the data of different sensors in the same three-dimensional space. Only after obtaining multiple different coordinates of the same point corresponding to different sensors can these sensors be calibrated accordingly according to the differences in these coordinates. For example, the center of the same small ball in space has coordinates (u, v) in the image information obtained by the vehicle's camera; (x, y, z) in the point cloud obtained by the vehicle's laser radar; and (x, y, θ) in the point cloud obtained by the vehicle's millimeter-wave radar. These coordinates are unified in a three-dimensional coordinate system, and the parameters of the corresponding sensors are adaptively adjusted according to the differences in different coordinates, thereby achieving the calibration effect of the vehicle sensor.

[0301] The calibration plate provided in the present application includes a metal block located in the center for sensor calibration and multiple marking codes surrounding the metal block. Since the two-dimensional code information on the multiple marking codes corresponds to different identity information, the identity ID information of the metal block can be quickly identified according to the identity information. The vehicle's sensor can realize automatic recognition on images and point cloud data according to different ID information, and the automatic matching effect of key points is good; moreover, the ID information can assist in the acquisition of the ID information and spatial position information of the metal block located in the middle of the marking code. Different metal blocks will not be confused due to excessive similarity, thereby solving the problem of automatic correspondence between spatial points of laser radar and millimeter-wave radar.

[0302] The technical solution provided by the present application can identify the identity of the metal block surrounding the marking codes through a unique two-dimensional code among multiple marking codes, without the need for manual assistance, thereby improving the calibration accuracy and the calibration work efficiency.

[0303] It should be further explained that in the calibration plate provided by the present application, in order to facilitate the vehicle sensor to distinguish the mark code and the metal block on the calibration carrier, a dark paint that has good absorption of the laser point cloud can be sprayed on the calibration carrier to obtain the mark code, such as a black mark code or a mark code with a large color difference from the calibration carrier. When the vehicle sensor is calibrated, such as a laser radar, the position of the calibration carrier can be obtained through plane fitting of the calibration plate point cloud, and the calibration plate point cloud is further filtered and segmented to identify the ID information and three-dimensional coordinates of key points of each mark code on the calibration plate. At this time, in order to facilitate distinction, the surface color of the non-metallic material selected for the calibration carrier itself needs to be different from the color of the mark code coating.

[0304] Since the dark area surfaces of the calibration carrier plate and the marking code in the calibration plate provided in this application have significant differences in color and other aspects, the laser radar point clouds at different positions in the calibration plate form differences, which can improve the marking code recognition efficiency of the laser radar point cloud.

[0305] It should be further explained that, in the calibration plate provided in the present application, the metal block is a polygonal structure, the marking code is a polygonal result, the vertices of multiple marking codes close to the metal block are connected to the vertices of the metal block, and the number of vertices of the metal block is greater than or equal to 4.

[0306] Specifically, in the calibration plate provided by the present application, the metal block and the marking code on the calibration plate are both polygonal in shape, and the multiple vertices of the metal block are directly connected to the vertex angles of the multiple marking codes surrounding the metal block. At this time, the vehicle sensor only needs to obtain the identity information and position information of the multiple adjacent marking codes of the metal block to completely locate the metal block, thereby establishing an association between the identity information and the positioning information of the metal block. In order to ensure the accuracy of positioning, the number of marking codes is at least 4, and the metal block has at least four sides.

[0307] Since the metal block in the calibration plate provided by the present application is connected to the vertices of the marking code, the position and identity information of the metal block can be obtained by identifying the position and identity information of the marking code, thereby facilitating the calibration between the millimeter-wave radar and other sensors installed on the vehicle and improving the versatility of the present application.

[0308] It should be further explained that, in the calibration plate provided in the present application, the number of marking codes is 4, and the polygonal structures of the metal block and the marking codes are both rectangular.

[0309] Specifically, in the technical solution provided by the present application, the marking code and the metal block on the calibration plate are both rectangular, and the four vertices of the metal block are respectively connected to or overlapped with a vertex of the marking code close to the metal block.

[0310] On the basis of the above implementation, since the calibration plate provided by the present application can take into account the calibration of the camera's internal and external parameters, the ArUco codes corresponding to at least 4 mark codes on the same calibration plate have more than 16 key points in total. Further use the key points of multiple sets of calibration plates within the camera's field of view and determine the three-dimensional spatial coordinates of each key point based on each key point and the point cloud data obtained by the total station to perform camera internal and external parameter calibration, on the one hand, to achieve automated calibration, and on the other hand, to improve the accuracy of camera internal parameter solution.

[0311] On the basis of the above implementation, the two-dimensional code information carried by the marker code in the calibration plate provided in the present application is a quasi-two-dimensional code that is convenient for the camera to decode, such as an ArUco code.

[0312] In one embodiment, when the sensor includes a camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0313] Controlling the camera to be calibrated to collect data from the calibration device to obtain a first calibration image;

[0314] Identify the marker code in the first calibration image, and determine the second ID information and the partial image of each marker code in the first calibration image;

[0315] Determine the second image coordinates of each key point included in the first calibration image according to the ID information of each marker code in the first calibration image and the partial image;

[0316] Determine the second three-dimensional coordinates corresponding to each key point in the first calibration image according to the ID information of each mark code in the first calibration image, the second image coordinates of each key point and the first reference three-dimensional coordinates;

[0317] The intrinsic parameters of the camera to be calibrated are determined according to the second image coordinates and the second three-dimensional coordinates.

[0318] This embodiment provides a method for calibrating the intrinsic parameters of a camera to be calibrated on a vehicle. Specifically, first, a first calibration image sent by the camera to be calibrated is obtained, and the identity information of the calibration plate and the image coordinates of the key points included in the first calibration image are determined through the first calibration image. Then, the first reference three-dimensional coordinates of each point cloud in the second local point cloud data are used to determine the second three-dimensional coordinates corresponding to the key points in the first calibration image. Finally, the intrinsic parameters of the camera to be calibrated are determined through the second image coordinates and the second three-dimensional coordinates of the key points.

[0319] Among them, the second ID information is the ID information of the calibration code included in the first calibration image, and the set of calibration codes included in the first calibration image is a subset of the set of calibration codes used in the calibration system, so the second ID information is a part of the first ID information of all calibration codes. Then, based on the second ID information of the calibration code, the second image coordinates of each key point on the calibration board, and the first reference three-dimensional coordinates, the second three-dimensional coordinates corresponding to each key point can be determined. That is, the second three-dimensional coordinates are a part of the first three-dimensional coordinates of all key points.

[0320] For example, the intrinsic parameters of a camera include the focal length f x 、f y , principal point coordinates u0, v0 and lens distortion k1, k2 (only mirror distortion is considered here). Taking the vehicle forward camera as an example, assume that n calibration plates (denoted as C i , i∈n), each calibration plate image has m calibration code key points (denoted as M j , j∈m).

[0321] For different calibration boards, the internal parameters of the same camera are consistent. Based on this, the steps to obtain the initial value of the internal parameter calibration are as follows:

[0322] (1) For the camera to be calibrated, collect a frame of calibration image (the first calibration image);

[0323] (2) Obtain the ID and partial image of each calibration code through the two-dimensional code recognition method. For each partial image of the calibration code, use Hough transform to extract the four sides of the circumscribed rectangle of the calibration code, and determine the second image coordinates of the key point through the intersection of two sides. (k = 0, 1, 2, 3, representing the upper left corner, upper right corner, lower right corner, and lower left corner, respectively). As the center, take a 11×11 local image, such as Fig. 9 For each local image, platform kernel fitting is performed to further obtain the sub-pixel coordinates (x ij ,y ij ) i∈n，j∈m (Sub-pixel extraction is performed here to further increase the solution accuracy).

[0324] Identify the second image coordinates (x ij ,y ij ) i∈n,j∈m and ID information ij(i∈n, j∈m) (the calibration code is the jth calibration code in the i-th calibration plate). Among them, the second three-dimensional coordinates (X ij , Y ij , Z ij ) i∈n,j∈m .

[0325] For each calibration plate visible in the camera field of view, establish the world coordinate system O w X w Y w Z w (In this embodiment, the coordinate system of the total station is used as the world coordinate system), that is, the key point of the upper left corner of the calibration code of the calibration plate is used as the origin of the world coordinate system O w ; The horizontal axis of the calibration plate is X w Axis, vertical axis is Y w Axis, according to the right-hand principle to establish Z w Axis, such as Fig.10 For each calibration plate visible in the camera field of view, the second image coordinates (x ij ,y ij ) i∈n,j∈m With the second three-dimensional coordinate (X ij , Y ij , Z ij ) i∈n,j∈m , according to Zhang Zhengyou calibration method, solve the camera internal parameter f x 、f y , principal point coordinates u0, v0 and lens distortion k1, k2.

[0326] It should be noted that Zhang Zhengyou's calibration method requires collecting multiple calibration images (non-coplanar) with different viewing angles and distances to obtain the camera's intrinsic parameters. Because the camera's intrinsic parameters are consistent for different images, this method can only solve the external parameters relative to each calibration plate image, but cannot obtain the true camera's external parameters.

[0327] Traditional camera intrinsic calibration requires collecting multiple calibration plates (such as chessboards, Fig.11 and Fig.12 The calibration process requires that the calibration plate be placed at different positions and on different planes (non-coplanar). In this embodiment, only one frame of image needs to be collected, which improves the efficiency of calibrating the intrinsic parameters of the camera to be calibrated.

[0328] In summary, in this embodiment, the calibration parameters of each sensor are calculated by using the identity information and image coordinates of the calibration plate and the correspondence between the local point cloud data collected by the sensor and the reference three-dimensional coordinates. Through these calibration parameters, the sensor can be accurately calibrated to improve its measurement accuracy and positioning capability.

[0329] In one embodiment, after determining the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates, the method further includes:

[0330] Determine the extrinsic parameters of the camera to be calibrated according to the second image coordinates, the second three-dimensional coordinates and the intrinsic parameters of the camera to be calibrated;

[0331] The external parameters of the camera to be calibrated include at least a first coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

[0332] In this embodiment, after determining the intrinsic parameters of the camera to be calibrated, the extrinsic parameters of the camera to be calibrated are also determined. The so-called camera intrinsic parameters are the internal parameters of the camera described in the above embodiment, including focal length, principal point coordinates and distortion parameters, etc., which are used to describe the geometric characteristics inside the camera. The camera extrinsic parameters describe the geometric relationship between the camera coordinate system and the total station coordinate system, that is, the position and posture of the camera in the total station coordinate system.

[0333] In this embodiment, the intrinsic parameters of the camera to be calibrated are determined according to the second image coordinates and the second three-dimensional coordinates corresponding to each key point in the first calibration image, and the next step is to determine the extrinsic parameters of the camera to be calibrated. Specifically, the extrinsic parameters of the camera to be calibrated are determined using the second image coordinates, the second three-dimensional coordinates and the intrinsic parameters of the camera to be calibrated.

[0334] Before determining the external parameters of the camera to be calibrated, you first need to understand the relationship between the coordinate system of the camera and the coordinate system of the total station. The total station has its own independent coordinate system, and the camera to be calibrated also has its own coordinate system. There is a certain coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station, which includes translation, rotation, and scale parameters. According to the internal parameters of the camera to be calibrated and the second image coordinates and second three-dimensional coordinates corresponding to each key point in the first calibration image, the coordinate transformation relationship between the camera and the total station coordinate system can be used to determine the external parameters of the camera to be calibrated. Specifically, the external parameters of the camera to be calibrated can be determined by solving the translation, rotation, and scale parameters between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

[0335] By determining the external parameters of the camera to be calibrated, the position and posture of the camera to be calibrated in the total station coordinate system can be accurately described, so as to be used for calibrating the camera to be calibrated in the vehicle sensor calibration method of the present application.

[0336] Specifically, the steps to calibrate the external parameters of the camera to be calibrated are:

[0337] For the 3D space points of all calibration code key points within the camera field of view (Second three-dimensional coordinate) (k∈n×m, n is the number of calibration plates, m is the number of marking codes on the calibration plates), the corresponding two-dimensional pixel on the image is (x ij ,y ij ) i∈n，j∈m (Second image coordinates). Theoretically, a 3D space point passes through the camera intrinsics The two-dimensional pixel obtained after the lens distortion k1, k2 and the external parameters R, T are transformed is (x′ ij , y′ ij ) i∈n，j∈m .

[0338] Therefore, a function F can be constructed, whose parameters are internal parameters and external parameters. The second three-dimensional coordinate passes through the function F and outputs a two-dimensional projection image coordinate (second image coordinate) (x′ ij , y′ ij ) i∈n，j∈m .

[0339] That is, R is the rotation matrix in the first coordinate transformation relationship, and T is the translation vector in the first coordinate transformation relationship.

[0340] In practice, the identified (x ij ,y ij ) i∈n，j∈m Compared with the theoretical (x′ ij , y′ ij ) i∈n，j∈m There is a certain deviation. Therefore, the function to be optimized can be constructed as follows:

[0341]

[0342] Use the Levenberg-Marquardt (LM) algorithm to iterate and solve. Here, we take the first step to obtain the focal length f x 、f y , principal point coordinates u0, v0 and lens distortion k1, k2 are used as initial values ​​for iterative calculation. By giving better initial values, subsequent iterations converge faster.

[0343] In one embodiment, when the sensor includes a surround view camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0344] Controlling the surround-view camera to be calibrated to collect data from the calibration device to obtain a second calibration image;

[0345] Projecting the second calibration image using the surround stitching homography matrix to obtain a projected image;

[0346] The second coordinate transformation relationship between the coordinate systems of the two adjacent surround view cameras to be calibrated is calculated according to the same key points in the projection images corresponding to the two adjacent surround view cameras to be calibrated.

[0347] In this embodiment, when the sensor includes a surround view camera to be calibrated, first, the second calibration image sent by each surround view camera to be calibrated is obtained: this step involves collecting the calibration images taken by each surround view camera on the vehicle to be calibrated. The second calibration image is projected using a surround view stitching homography matrix to obtain a projection image: in this step, a surround view stitching homography matrix technology is used to project the second calibration image to generate a corresponding projection image. The second coordinate transformation relationship between the coordinate systems of two adjacent surround view cameras to be calibrated is calculated based on the same key points in the projection images corresponding to the two adjacent surround view cameras to be calibrated: by comparing the same key points in the projection images of the two adjacent surround view cameras to be calibrated, the coordinate system transformation relationship between the two adjacent surround view cameras to be calibrated can be calculated.

[0348] In this embodiment, the image data of the surround view cameras to be calibrated are collected, and the surround view stitching homography matrix technology is used for projection transformation, and finally the coordinate system transformation relationship between two adjacent surround view cameras to be calibrated is calculated. This result can be used as part of the calculation of sensor calibration parameters to calibrate each sensor on the vehicle to be calibrated.

[0349] In one embodiment, after acquiring the second calibration images sent by each of the surround view cameras to be calibrated, the method further includes: performing distortion calibration on the second calibration images according to the internal parameters of the cameras to be calibrated to obtain a calibrated second calibration image.

[0350] In one embodiment, after controlling the surround view camera to be calibrated to collect data from the calibration device and obtaining the second calibration image, the method further includes:

[0351] Extracting the ID information of each marker code in the second calibration image and the third image coordinates of each key point;

[0352] Filter the key points on the ground in the second calibration image according to the ID information of each marker code in the second calibration image and the third image coordinates of each key point, and obtain the third three-dimensional coordinates of each key point on the ground;

[0353] Determine a surround stitching image according to the third three-dimensional coordinates of each key point located on the ground;

[0354] Calibrate the fourth three-dimensional coordinates of the key points in the surround view stitching image using the first reference three-dimensional coordinates corresponding to the key points in the surround view stitching image in the calibration prior information to obtain a calibrated surround view stitching image;

[0355] The surround view stitching homography matrix is ​​obtained according to the calibrated surround view stitching image.

[0356] It should be noted that the third three-dimensional coordinates are the three-dimensional coordinates of all key points within the field of view corresponding to the second calibration image and located on the ground. The set of all key points within the field of view corresponding to the second calibration image and located on the ground is a subset of the set of all key points in the calibration system, so the third three-dimensional coordinates are part of all first three-dimensional coordinates. Furthermore, the fourth three-dimensional coordinates are the three-dimensional coordinates of the key points in the surround view stitching image determined according to the third three-dimensional coordinates of each key point on the ground. The set of key points in the surround view stitching image determined according to the third three-dimensional coordinates of each key point on the ground is a subset of the set of all key points within the field of view corresponding to the second calibration image and located on the ground. Then the fourth three-dimensional coordinates are part of all third three-dimensional coordinates. Specifically, the surround view stitching calibration task is mainly to calibrate the ground projection of the surround view cameras around the vehicle body and the transformation relationship between adjacent cameras. The specific steps are as follows:

[0357] For each surround camera c i , collect the second calibration image I i , according to the internal parameters of each camera obtained in the previous step, the second calibration image I i Perform distortion calibration to obtain the calibrated second calibration image

[0358] For the second calibration image after calibration Extract the third image coordinates and ID information (fourth identity information) of the key points within its field of view, and according to the ID information, only retain the key points on the ground, and obtain the third three-dimensional coordinate information of the key points on the ground (such as Fig.13 , Fig.13 This is a top view of the surround stitching calibration layout provided in this application). Compared with manual point selection, this embodiment is aimed at mass production and solves the automation problem.

[0359] For each camera, automatically obtain the ID information of the camera. Fig.13 Select the key point corresponding to the largest sign code in the left and right sides. Select the dotted rectangle ABCD in the same way as in 14. Fig.14 A schematic diagram of the projection transformation matrix selection provided for this application.

[0360] When building the calibration site, it is difficult to ensure Fig.13All calibration codes in the image are horizontal and coplanar, which is ignored by existing methods, resulting in insufficient surround stitching accuracy. Fig.13 The three-dimensional coordinates of the key points of the calibration code (X i , Y i , Z i ), as well as the camera internal and external parameters obtained in the first step, Fig.14 In the calibration, four points ABCD are used, that is, (X i , Y i , Z i ) is transformed to the image through the internal and external parameters (no distortion parameters are used here) After calibration, the point (like Fig.14 The location of the middle black dot);

[0361] use Solve the surround stitching homography matrix E; for the key points in the overlapping fields of view of adjacent cameras, solve the transformation scale parameters according to the key point ID information and the actual size of the cooperation mark.

[0362] In one embodiment, when the sensor includes a main laser radar, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes:

[0363] Controlling the main laser radar to collect data from the calibration device to obtain first test point cloud data;

[0364] The first test point cloud data is matched with the first data to obtain a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system.

[0365] This embodiment describes the process of calibrating the main laser radar. Specifically, firstly, the first test point cloud data sent by the main laser radar is obtained. Next, the first test point cloud data is matched with the first data to obtain the third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system.

[0366] In this embodiment, the second data collected by the sensor refers to the data collected by each sensor on the calibration device. Among them, the first test point cloud data sent by the main laser radar is one of them. These sensors and calibration devices can transmit data through wireless or wired connections.

[0367] By matching the first test point cloud data with the first data, the third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system can be determined. This transformation relationship can be used in the subsequent sensor calibration process. By obtaining the third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of the main laser radar can be calculated. These parameters can include rotation matrices and / or translation vectors, etc.

[0368] In one embodiment, after controlling the main laser radar to collect data from the calibration device and obtaining the first test point cloud data, the method further includes:

[0369] Match the first test point cloud data with the first data to obtain the ID information of the marker code and the fifth three-dimensional coordinates of the key points within the field of view of the main laser radar;

[0370] Determine the same key points in the first calibration image as those in the field of view of the main laser radar according to the ID information of each marker code in the first calibration image, the second three-dimensional coordinates corresponding to each key point, the ID information of the marker code within the field of view of the main laser radar, and the fifth three-dimensional coordinates of the key points, and determine the three-dimensional coordinates of the same key points;

[0371] The fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main lidar coordinate system is calculated based on the three-dimensional coordinates of the same key points.

[0372] It should be noted that the fifth three-dimensional coordinate is the three-dimensional coordinate corresponding to the key point of the field of view of the main laser radar. The set of key points in the field of view of the main laser radar is a subset of the set of all key points in the calibration system. Then the fifth three-dimensional coordinate is a part of all the first three-dimensional coordinates.

[0373] Specifically, assume that the coordinates of the key points in the camera ci coordinate system are (the second three-dimensional coordinate described above), the key point coordinates in the main laser radar l1 coordinate system (that is, the fifth three-dimensional coordinates described above) are Theoretically, the transformation relationship between the same-name points in the two coordinate systems is as follows:

[0374]

[0375] Considering various errors,

[0376] The function to be optimized is as follows:

[0377] The calibration steps are:

[0378] (1) extracting the image coordinates and ID of the key points in the field of view of the c1 camera, and obtaining the corresponding three-dimensional spatial coordinates (i.e., the second three-dimensional coordinates mentioned above) according to the ID information;

[0379] (2) extracting the three-dimensional coordinates (i.e., the fifth three-dimensional coordinates) and ID of the key points in the field of view of the main laser radar l1;

[0380] (3) Based on the ID information and the relationship between the same-name points, the Levenberg-Marquardt (LM) algorithm is used to solve the external parameters (the fourth coordinate transformation relationship).

[0381] In one embodiment, when the sensor includes a millimeter wave radar, a metal block is provided on the calibration plate, and the calibration system further includes two tracks and a conveyor belt device provided on the tracks, the two tracks are parallel to each other and the distance between the two tracks is the same as the distance between the left and right wheels of the vehicle to be calibrated, and the vehicle to be calibrated is located on the conveyor belt device when the sensor calibration is performed;

[0382] Before controlling each sensor on the vehicle to be calibrated to collect the calibration data, and obtaining the second data, the method further includes:

[0383] By controlling the conveyor belt device to move the vehicle to be calibrated at a preset speed;

[0384] Controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain second data, including:

[0385] During the movement of the vehicle to be calibrated, the millimeter wave radar is triggered to collect the coordinates of the metal block within the field of view to obtain collected data, which at least includes the coordinates of the metal block;

[0386] Determine alignment data of the first test point cloud data and the collected data according to the data timestamp;

[0387] Projecting the sixth three-dimensional coordinate of the metal block in the first test point cloud data in the alignment data onto a horizontal plane to obtain horizontal plane projection data;

[0388] The fifth coordinate transformation relationship between the millimeter-wave radar coordinate system and the main lidar coordinate system is calculated according to the horizontal plane projection data, the coordinates of the metal blocks in the collected data in the alignment data, and the positions of each metal block in the calibration system.

[0389] It should be noted that the sixth three-dimensional coordinate is the three-dimensional coordinate corresponding to the metal block within the field of view of the millimeter-wave radar. The set of metal blocks within the field of view of the millimeter-wave radar is a subset of the set of all metal blocks in the calibration system. Therefore, the sixth three-dimensional coordinate is a part of the three-dimensional coordinates of all the above metal blocks.

[0390] Specifically, the millimeter wave radar can obtain the x and y coordinate information of the target, but does not have the z coordinate information of the target. Therefore, the millimeter wave coordinate system O w X w Y w To the main lidar coordinate system O l X l Y l The transformation is regarded as the transformation of the two-dimensional XY coordinate system (such as Fig.15 As shown, Fig.15 Schematic diagram of the relationship between the millimeter-wave radar coordinate system and the main lidar coordinate system provided in this application), the parameters to be calibrated include the translation vector T and the rotation angle θ.

[0391] The conversion relationship from millimeter wave coordinates to the main lidar coordinate system is:

[0392]

[0393] We can construct a function F whose parameter is T x , T y , θ. The three-dimensional world coordinates pass through the function F and output a two-dimensional projection image coordinate (x′ ij , y′ ij );

[0394] (x l ,y l )=F(T x , T y ,θ,x w ,y w );

[0395] Taking the error factor into consideration,

[0396] The function to be optimized is:

[0397]

[0398] The calibration steps are as follows:

[0399] (1) Since the vehicle is fixed on the track, the conveyor belt on the track can be used to easily control the vehicle to move in a short distance in the front and rear direction (during the calibration process, it only needs to move a very short distance (such as 1 meter));

[0400] (2) Save the main laser radar data (first test point cloud data) and millimeter wave radar data (collection data of metal blocks sent by the millimeter wave radar) during the vehicle movement;

[0401] (3) According to the data timestamp, find the time-aligned data frame of the main laser radar and the millimeter-wave radar. For the three-dimensional coordinates of the central metal block of each calibration plate within the field of view of the main laser radar l1 (the sixth three-dimensional coordinates of the metal block), project it onto the horizontal plane to obtain the horizontal plane projection data; for the millimeter radar, identify the coordinates of all metal blocks within its field of view;

[0402] (4) According to the horizontal projection data and the known spatial topological relationship of the metal block, the optimal matching relationship is obtained. According to the optimal matching relationship, the Levenberg-Marquardt (LM) algorithm is used to iteratively calculate the above optimization function to solve the fifth coordinate transformation relationship.

[0403] In one embodiment, when the sensor includes an auxiliary laser radar, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain the second data includes:

[0404] Controlling the auxiliary laser radar to collect data from the calibration device to obtain second test point cloud data;

[0405] The second test point cloud data is matched with the first test point cloud data to obtain a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system.

[0406] Specifically, the calibration parameters (sixth coordinate transformation relationship) between the auxiliary laser radar and the main laser radar are the rotation matrix R and the translation vector T.

[0407] Assume that the point cloud data of the main laser radar l1 are Q = {q1, q2, .., q m}(first test point cloud data), the auxiliary laser radar l2 point cloud data is P = {p1, p2, .., p m ) (Second test point cloud data). Using the spatial topological structure of the calibration component at the calibration site, according to the spatial matching of the point clouds, the coordinate transformation relationship of the spatial points with the same name in the two point cloud data is:

[0408] Constructing the optimization function: Through the Gauss-Newton or LM algorithm, R and T corresponding to the sixth coordinate transformation relationship can be obtained.

[0409] The main / auxiliary lidar calibration steps are as follows:

[0410] (1) Distributed acquisition of primary lidar and auxiliary lidar point cloud data Q and P;

[0411] (2) For the point cloud data Q and P, use the method in the preprocessing of "obtaining prior information of the calibration site" to obtain the point cloud data after removing the ceiling, ground, and surrounding walls. and Eliminating these interferences can greatly improve the calibration accuracy;

[0412] (3) Point cloud data and Use the calibration plate plane fitting method in "Obtaining prior information of the calibration site" to obtain the calibration plate plane, calibration ball spherical surface and other information in each point cloud, including its ID information. The ID of the calibration plate can be composed of its internal calibration code ID. For example, there are 6 calibration codes on it, and the IDs are ID i (i≤6) then the calibration board ID is the calibration code ID serially arranged from small to large, such as ID1ID2ID3ID4ID5ID6;

[0413] (4) According to the calibration plate ID information and the calibration ball ID, the initial value of the calibration parameter RT corresponding to the sixth coordinate transformation relationship is solved; further, according to the calibration plate fitting plane and the calibration ball spherical surface, based on the initial value of the calibration parameter RT, further refine it through NDT or ICP to obtain the final calibration parameter RT.

[0414] Traditional methods directly use point data for matching, such as NDT and various ICP improvement methods. Due to the large differences in the hardware parameters of the main / auxiliary lidar, the point cloud resolution, projection angle, etc. may cause very few true points of the same name in space. The points in two frames of laser point cloud data cannot represent the same position in space. Therefore, using the point-to-point distance as the error equation is bound to introduce random errors. In this application, by fitting the planes of each calibration plate, the above influence can be effectively reduced, that is, the calibration accuracy can be improved.

[0415] In one embodiment, the calibration device includes a plurality of calibration components. In the following embodiments, the calibration components are taken as calibration balls as an example for explanation (eg Fig.16 As shown, Fig.16 The calibration prior information is determined according to the first calibration data, including:

[0416] Performing spatial clustering segmentation on the first calibration data according to the position of each verification component in the calibration system to obtain third local point cloud data corresponding to each verification component;

[0417] The fifth identity information and the fifth coordinate information corresponding to each verification component are determined according to the third local point cloud data.

[0418] First, the first calibration data needs to be spatially clustered and segmented according to the position of each verification component in the calibration system to obtain the third local point cloud data corresponding to each verification component. The so-called spatial cluster segmentation is to group the adjacent or close point clouds in the first calibration data together to form point cloud clusters. The purpose of this is to separate the point clouds corresponding to each verification component to be calibrated in the first calibration data for subsequent processing. Cluster segmentation can be performed based on factors such as spatial distance and density. After cluster segmentation, each point cloud cluster can be regarded as local point cloud data of a verification component. By processing each point cloud cluster, the third local point cloud data of the verification component can be obtained. These local point cloud data can be used for subsequent verification parameter calculation, error analysis, etc. Next, according to the third local point cloud data, the fifth identity information and fifth coordinate information corresponding to each verification component can be determined. The fifth identity information usually refers to the unique identification of the verification component, which is used to distinguish different verification components. In the verification system, the identity information of the verification component can be determined by extracting and matching the shape, size, texture and other features of the verification component. The fifth coordinate information usually refers to the position and posture information of the verification component in the calibration system. By registering and matching the point cloud data, the position and posture of the verification component relative to the calibration system can be determined. In this way, the fifth coordinate information corresponding to the verification component can be obtained.

[0419] In summary, by performing spatial clustering segmentation on the first calibration data, the third local point cloud data corresponding to each verification component can be obtained. Then, by processing and analyzing the third local point cloud data, the fifth identity information and fifth coordinate information corresponding to each verification component can be determined. In this way, it can be used to calibrate the parameters of the sensor and perform related work of the autonomous driving system.

[0420] In one embodiment, determining the fifth identity information and the fifth coordinate information corresponding to each of the verification components according to the third local point cloud data includes:

[0421] The third local point cloud data is fitted to obtain structural information and three-dimensional coordinates of each verification component.

[0422] This embodiment mentions that the fifth identity information and the fifth coordinate information corresponding to each verification component are determined according to the third local point cloud data. In order to obtain the identity information and coordinate information of the verification component, the third local point cloud data needs to be processed. First, the third local point cloud data needs to be fitted, that is, the point cloud data is appropriately fitted using a mathematical model to obtain the structural information of the verification component. The fitting can be performed by using a surface fitting algorithm, such as the least squares method, to fit the shape of the sphere. Through the structural information obtained by fitting, the shape, size and other characteristics of the verification component can be understood. Secondly, through the results after fitting, the three-dimensional coordinates of the verification component can be obtained. Since the fitting is mainly based on the point cloud data, the center point coordinates of the verification component, that is, the fifth coordinate information, can be obtained. By obtaining the center point coordinates of the verification component, the position of the verification component can be determined. Finally, according to the structural information and three-dimensional coordinates of the verification component, the identity information of the verification component, that is, the fifth identity information, can be obtained. The identity information of the verification component may include the type and number of the verification component, which can be used to uniquely identify each verification component.

[0423] In summary, by fitting the third local point cloud data, the structural information and three-dimensional coordinates of the verification component can be obtained, and then the identity information and coordinate information of the verification component can be determined. In this way, the verification component in the automatic calibration system of the autonomous driving sensor can be accurately located and identified, and the accuracy and stability of the calibration parameters can be improved.

[0424] In one embodiment, the verification component is a verification ball, and the fifth identity information and the fifth coordinate information corresponding to each verification component are determined according to the third local point cloud data, including:

[0425] Spherical fitting is performed on the third local point cloud data to obtain the radius information and three-dimensional coordinates of the calibration sphere.

[0426] In this embodiment, the verification component is a verification ball. The step of determining the fifth identity information and fifth coordinate information corresponding to each verification component (i.e., the verification ball) according to the third local point cloud data includes the following two parts: Spherical fitting of the third local point cloud data: Point cloud data of the surface of the verification ball can be obtained according to the third local point cloud data. By performing spherical fitting on these point cloud data, the information of the fitted spherical surface can be obtained, mainly including the radius information of the verification ball and the three-dimensional coordinates of the center of the sphere. Spherical fitting is to process the point cloud data by mathematical methods so that the fitting result can best meet the spherical model. Determine the identity information and coordinate information of the verification ball: The identity information and coordinate information of the verification ball can be determined by the radius information and the three-dimensional coordinates of the center of the verification ball obtained after spherical fitting. The identity information of the verification ball is usually used to distinguish different verification balls, and the coordinate information is used to determine the position of the verification ball in three-dimensional space.

[0427] In this way, by performing spherical fitting on the third local point cloud data, the radius information and position information of each calibration sphere can be obtained, thereby determining the fifth identity information and fifth coordinate information corresponding to each calibration component (i.e., calibration sphere). This information is used to calibrate the calibration parameters of each sensor in the automatic calibration system of the autonomous driving sensor.

[0428] It should be noted that the fifth identity information is the identity information of the verification component. For example, when the verification component is a verification ball, the fifth identity information may be the radius information of the verification ball. The fifth coordinate information represents the position of the verification component in the calibration system, such as the spatial position.

[0429] It should also be noted that the calibration sphere is used to calibrate the calibration parameters of each sensor. This is because the calibration sphere has a relatively uniform spherical shape, which presents a perfect round surface no matter from which angle it is observed. This spherical feature makes the data generated by the calibration sphere relatively easy to process and analyze. In other words, the use of the calibration sphere for sensor calibration can provide accurate and reliable results. The uniform spherical shape of the calibration sphere ensures that the sensor obtains consistent data at different angles and directions, thereby reducing the possibility of calibration errors. In addition, the spherical characteristics of the calibration sphere enable it to reflect or scatter light in the same way when receiving the laser, radar or camera signal of the sensor, thereby ensuring the consistency and stability of the data. The data processing of the calibration sphere is relatively simple. Since the calibration sphere presents a perfect round surface, the image or point cloud data formed in the sensor can be easily processed and analyzed. By processing the calibration sphere data, the calibration parameters of the sensor can be obtained, optimized and adjusted, thereby improving the precision and accuracy of the sensor.

[0430] In addition to the calibration ball, the calibration component may also be implemented in other ways, such as a calibration plate, a square or cross-shaped calibration component, etc.

[0431] In short, it is of great significance to use the calibration sphere to calibrate the calibration parameters of the autonomous driving sensor. The uniform spherical characteristics and data processing advantages of the calibration sphere can ensure the accuracy and reliability of sensor calibration and provide effective support for the performance of the autonomous driving system.

[0432] Specifically, let P be the first calibration data after Φ removes the point clouds of the ceiling, the ground and the surrounding walls, and only includes the point clouds of each calibration component and the point cloud of the ground verification ball.

[0433] For the calibration ball, the specific steps are as follows:

[0434] (1) Since the installation height of the calibration ball is lower than the calibration component, the point cloud P can be divided into the point cloud containing only the calibration ball according to the height z coordinate. and point clouds containing only the calibrated components

[0435] (2) Clustering based on the spatial interval of the calibration balls. Perform spatial clustering segmentation to obtain the local point cloud of each verification ball (M1 is the total number of calibration balls); for each Through point cloud spherical fitting, the three-dimensional spatial coordinates of the center of the calibration ball (x j ,y j , z j ) and the radius of the ball R j information.

[0436] In one embodiment, after obtaining calibration prior information according to the first data and calibrating each sensor on the to-be-calibrated vehicle according to the calibration prior information and the second data, the method further includes:

[0437] The calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere.

[0438] Since a multi-level perception system requires a large number of sensors, the correctness of the calibration results will directly affect the performance of the perception system. This application proposes a method for automatic calibration parameter verification, specifically, for dual-target calibration, camera and main lidar radar calibration, main / auxiliary lidar calibration, millimeter-wave radar and main lidar calibration result verification, the basic idea is:

[0439] For a sphere with known spatial position and size in the calibration system, identify the three-dimensional coordinates of its center in different coordinate systems and compare them with the true value. The specific steps are as follows:

[0440] In one embodiment, when the calibration parameters of the sensor include a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration sphere, including:

[0441] Obtain the seventh three-dimensional coordinates of each calibration sphere within the field of view of the main laser radar;

[0442] performing coordinate transformation on the seventh three-dimensional coordinate according to the third coordinate transformation relationship to obtain the seventh three-dimensional coordinate to be compared;

[0443] comparing the seventh three-dimensional coordinate to be compared with the seventh reference three-dimensional coordinate to determine whether the third coordinate transformation relationship is accurate;

[0444] The first reference three-dimensional coordinates are the three-dimensional coordinates obtained by the total station and corresponding to the verification component within the field of view of the main laser radar.

[0445] This example aims to verify the calibration accuracy of the main laser radar and the total station coordinate system. For the laser radar, the three-dimensional coordinates of the center of the sphere are obtained based on the reflectivity difference and spatial clustering through the known sphere ROI area. (i.e., the seventh three-dimensional coordinate); according to the external parameter R / t relationship between the main laser radar and the total station (i.e., the third coordinate transformation relationship), Perform coordinate transformation to obtain (that is, the seventh three-dimensional coordinate to be compared); Compare with the true value (the coordinates obtained by the total station, that is, the seventh reference three-dimensional coordinates) to determine the R / T error.

[0446] It should be noted that the seventh three-dimensional coordinate is the three-dimensional coordinate of the verification component within the field of view of the main laser radar. The set of verification components within the field of view of the main laser radar is a subset of the set of all verification components. Correspondingly, the seventh three-dimensional coordinate is a part of all the fifth coordinate information.

[0447] In one embodiment, when the calibration parameters of the sensor include a fourth coordinate transformation relationship between a camera coordinate system to be calibrated and a main laser radar coordinate system, the calibration parameters of each sensor are verified according to the radius information and three-dimensional coordinates of each calibration sphere, including:

[0448] Acquire first calibration point cloud data of a calibration sphere within the field of view of the main laser radar;

[0449] Back-projecting the first verification point cloud data onto the camera image according to the fourth coordinate transformation relationship;

[0450] Performing preset processing on the back-projected camera image to obtain a circular edge in the camera image, and determining a corresponding first circular fitting equation according to the circular edge;

[0451] The first circle fitting equation is compared with the reference circle fitting equation to determine whether the fourth coordinate transformation relationship is accurate.

[0452] This embodiment is intended to verify the calibration parameter error between the camera and the main laser radar. In the embodiment, first obtain the first verification point cloud data of the verification component within the field of view of the main laser radar. Next, according to the fourth coordinate transformation relationship, the first verification point cloud data is back-projected onto the camera image. Through the back-projection operation, the three-dimensional coordinates of the verification point cloud are mapped to the two-dimensional image of the camera. Finally, based on the back-projected camera image, determine whether the fourth coordinate transformation relationship is accurate. By comparing the position of the verification point on the camera image and the position of the actual verification component, the accuracy of the fourth coordinate transformation relationship can be evaluated.

[0453] When the verification component is a verification ball, the first verification point cloud data of the verification component within the field of view of the main laser radar is transformed according to the external reference R corresponding to the fourth coordinate transformation relationship. c-l ,t c-l Back-project it onto each camera image; back-project the point cloud of the spherical area in each camera as follows Fig.17 As shown, the sphere is a gray prototype on the image, and the dotted line represents the back projection of the laser radar point cloud. Through grayscale and binarization processing, all circular edges in the image are extracted, and the first circular fitting equation O is obtained. i (i∈n, n is the number of circular regions in the image); according to the spatial distance difference, the point cloud data inside the sphere is obtained by clustering, and the endpoints of each horizontal line are obtained, such as Fig.18 As shown, the horizontal line is broken and fits the circle Q i (like Fig.19 Calculate O i With Q i The distance between the camera and the main lidar is further determined by the camera internal parameters and the external parameters R c-l ,t c-l (the accuracy of the fourth coordinate transformation relationship).

[0454] The purpose of this calibration method is to ensure that the coordinate transformation relationship between the camera to be calibrated and the main lidar is accurate. Through calibration, the accuracy of the calibration parameters can be verified, and the reliability and accuracy of sensor calibration can be improved.

[0455] In one embodiment, the calibration ball is a metal sphere, and when the calibration parameters of the sensor include the fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration ball, including:

[0456] Obtaining the millimeter-wave radar coordinates and angles of each calibration ball within the field of view of the millimeter-wave radar;

[0457] Obtain the seventh three-dimensional coordinate within the field of view of the main laser radar;

[0458] Projecting the millimeter wave radar coordinates and angles onto the two-dimensional plane of the main laser radar coordinate system according to the fifth coordinate transformation relationship;

[0459] The Euclidean distance is calculated based on the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates to determine whether the fifth coordinate transformation relationship is accurate.

[0460] The purpose of this embodiment is to verify the fifth coordinate transformation relationship between the auxiliary laser radar l1 coordinate system and the main laser radar l2 coordinate system. The verification process includes the following steps: obtain the millimeter wave radar coordinates and angles of each calibration ball within the field of view of the millimeter wave radar. Obtain the seventh three-dimensional coordinates within the field of view of the main laser radar. Project the millimeter wave radar coordinates and angles to the two-dimensional plane of the main laser radar coordinate system according to the fifth coordinate transformation relationship. Calculate the Euclidean distance based on the projected millimeter wave radar coordinates and angles and the seventh three-dimensional coordinates. Determine whether the fifth coordinate transformation relationship is accurate based on the calculated Euclidean distance.

[0461] Specifically, the millimeter-wave radar coordinates and yaw angle, and the three-dimensional coordinates of the center of the sphere in the main lidar coordinate system are extracted respectively; through the fifth coordinate transformation relationship, the coordinates of the center of the sphere in the millimeter-wave data are projected onto the XY plane of the main lidar coordinate system; the Euclidean distance is calculated to determine the calibration accuracy.

[0462] Through the above verification process, the accuracy of the fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system can be determined. This can ensure the accuracy and reliability of the sensor's calibration parameters, thereby improving the application performance and accuracy of the sensor in the vehicle.

[0463] In one embodiment, when the calibration parameters of the sensor include the sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration sphere, including:

[0464] Acquire the first calibration point cloud data of the calibration ball within the field of view of the main laser radar;

[0465] Acquire the second calibration point cloud data of the calibration ball within the field of view of the auxiliary laser radar;

[0466] The second verification point cloud data is transformed into the main laser radar coordinate system according to the sixth coordinate transformation relationship to obtain the second verification point cloud data to be verified;

[0467] Determine whether the sixth coordinate transformation relationship is accurate based on the first verification point cloud data and the second verification point cloud data to be verified.

[0468] The present embodiment aims to verify the sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system. Specifically, first, obtain the first verification point cloud data of the verification component within the field of view of the main laser radar. Next, obtain the second verification point cloud data of the verification component within the field of view of the auxiliary laser radar. Then, according to the sixth coordinate transformation relationship, the obtained second verification point cloud data is coordinate-converted and transformed into the main laser radar coordinate system. This transformation process usually involves rotation, translation, and possible scale changes. Finally, by comparing the first verification point cloud data with the second verification point cloud data to be verified, the accuracy of the sixth coordinate transformation relationship can be evaluated. If the difference between the two sets of point cloud data is small, it means that the sixth coordinate transformation relationship is accurate; if the difference is large, the sixth coordinate transformation relationship needs to be adjusted or recalculated.

[0469] Specifically, for the two laser radar point clouds, the spherical point clouds within their respective fields of view are extracted according to the differences in spatial distance and emissivity; (Fifth coordinate transformation relationship), transform the spherical point cloud in l2 to the l1 coordinate system; calculate the Euclidean distance and determine the accuracy of the fifth coordinate transformation relationship.

[0470] In summary, this embodiment is used to verify the accuracy of the sixth coordinate transformation relationship between the auxiliary laser radar and the main laser radar in the sensor calibration parameters. This method can help ensure that the sensor calibration parameters can accurately reflect the spatial relative position relationship between sensors in the actual scene.

[0471] In addition, in order to improve the calibration efficiency and accuracy, it is necessary to realize the rapid positioning of the vehicle to be calibrated, that is, to quickly drive into the designated position in the calibration system. This application uses the four contact points of the vehicle for positioning. Positioning includes front and rear and left and right. Figure 20-22 As shown in the figure, two tracks are installed in the calibration system, and the track width is the same as the width of the vehicle tires; the vehicle tires are parallel to the track and run on the track; there is a limiter at the front end of the track, and when the tires touch the limiter, the car can stop. After stopping, the rear wheel limiter needs to be fixed. Fixing the rear wheel is for safety considerations on the one hand. On the other hand, it is to calibrate the coordinate system of the millimeter wave radar and the laser radar.

[0472] This operation ensures that the initial positions of all vehicles to be calibrated are close, and the deviation between the main lidar coordinate system and the total station coordinate system is as small as possible. During the point cloud matching process, the deviation between the two point cloud coordinate systems is as small as possible, the iterative algorithm converges faster, and the accuracy is easier to ensure.

[0473] In addition, in order to assist the calibration of the millimeter-wave radar and the main lidar, a transmission belt device is installed on the track to achieve short-distance uniform linear motion of the vehicle.

[0474] In a second aspect, the present application also provides a vehicle sensor calibration system based on the calibration system, such as Fig.23 As shown, Fig.23 The present application provides a structural block diagram of a vehicle sensor calibration system based on a calibration system, wherein the calibration system includes a total station, multiple calibration devices, and a computing device, wherein the computing device includes:

[0475] A first control unit 91 is used to control the total station to scan the calibration device to obtain first data when the vehicle to be calibrated meets the first calibration condition;

[0476] A second control unit 92 is used to control each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain second data when the second calibration condition is met;

[0477] The calibration unit 93 is used to obtain calibration prior information based on the first data, and calibrate each sensor on the vehicle to be calibrated based on the calibration prior information and the second data to obtain calibration parameters of each sensor; the calibration prior information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two sensors or between a sensor and a total station.

[0478] For an introduction to the vehicle sensor calibration system based on the calibration system, please refer to the above-mentioned embodiment, and this application will not go into details here.

[0479] In a third aspect, the present application also provides a vehicle sensor calibration device based on a calibration system, such as Fig.24 As shown, Fig.24 This is a structural block diagram of a vehicle sensor calibration device based on a calibration system provided in this application, the device comprising:

[0480] Memory 201, used for storing computer programs;

[0481] The processor 202 is used to implement the steps of the vehicle sensor calibration method based on the calibration system as described above when storing the computer program.

[0482] For an introduction to the vehicle sensor calibration device based on the calibration system, please refer to the above embodiment, and this application will not go into details here.

[0483] In a fourth aspect, the present application also provides a computer-readable storage medium 210, such as Fig.25 As shown, Fig.25This is a structural block diagram of a computer-readable storage medium provided by the present application. The computer-readable storage medium 210 stores a computer program 211. When the computer program 211 is executed by the processor 202, the steps of the vehicle sensor calibration method described above are implemented. For the introduction of the computer-readable storage medium 210, please refer to the above embodiment, and this application will not repeat it here.

[0484] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0485] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle sensor calibration method based on a calibration system, characterized in that: The calibration system includes a total station and multiple calibration devices, including: When the vehicle to be calibrated meets the first calibration condition, controlling the total station to scan the calibration device to obtain first data; When the second calibration condition is met, receiving second data acquired by the calibration device through the sensors on the vehicle to be calibrated; Calibration prior information is obtained based on the first data, and each of the sensors on the vehicle to be calibrated is calibrated based on the calibration prior information and the second data to obtain calibration parameters of each sensor; the calibration prior information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two of the sensors or between the sensor and the total station.

2. The vehicle sensor calibration method based on the calibration system according to claim 1, characterized in that: The process of determining whether the vehicle to be calibrated meets the first calibration condition includes: Determining whether the vehicle to be calibrated is parked in a preset area; If the vehicle stops in the preset area, it is determined that the vehicle to be calibrated meets the first calibration condition; otherwise, it is determined that the first calibration condition is not met.

3. The vehicle sensor calibration method based on the calibration system according to claim 1, characterized in that: The process of determining whether the vehicle to be calibrated meets the first calibration condition includes: Determining whether the vehicle to be calibrated is being calibrated for the first time according to the vehicle information of the vehicle to be calibrated; If it is the first calibration, it is determined that the vehicle to be calibrated meets the first calibration condition; otherwise, it is determined that the first calibration condition is not met.

4. The vehicle sensor calibration method based on the calibration system as claimed in claim 3, characterized in that: When determining that the vehicle to be calibrated is not calibrated for the first time, the method further includes: The first data corresponding to the vehicle to be calibrated and stored in the database is called.

5. The vehicle sensor calibration method based on the calibration system according to any one of claims 1 to 4, characterized in that: The process of determining whether the second calibration condition is met includes: Determine whether a calibration start instruction is received; If received, it is determined that the second calibration condition is met; otherwise, it is determined that the second calibration condition is not met.

6. The vehicle sensor calibration method based on the calibration system according to claim 1, characterized in that: Obtaining calibration prior information according to the first data includes: Eliminate data that does not meet preset requirements from the first data to obtain first calibration data; The calibration prior information is determined according to the first calibration data.

7. The vehicle sensor calibration method based on the calibration system according to claim 6, characterized in that: Eliminating data that does not meet preset requirements from the first data to obtain first calibration data includes: The point cloud data not including the calibration device is eliminated from the first data to obtain the first calibration data.

8. The vehicle sensor calibration method based on the calibration system according to claim 7, characterized in that: The calibration device includes a plurality of calibration components, and determining the calibration prior information according to the first calibration data includes: Performing spatial clustering segmentation on the first calibration data according to the position of each calibration component in the calibration system to obtain first local point cloud data corresponding to each calibration component; The first identity information and the first coordinate information of each of the calibration components are determined according to the first local point cloud data.

9. The vehicle sensor calibration method based on the calibration system as claimed in claim 8, characterized in that: Each of the calibration components includes at least two calibration plates, and determining first identity information and first coordinate information of each of the calibration components according to the first local point cloud data includes: Determine second local point cloud data corresponding to each calibration plate from the first local point cloud data; The second identity information and the second coordinate information corresponding to each of the calibration plates are determined according to the second local point cloud data.

10. The vehicle sensor calibration method based on the calibration system according to claim 9, characterized in that: Each of the calibration components includes at least two non-coplanar calibration plates, and second local point cloud data corresponding to each of the calibration plates is determined according to the position of each calibration plate in each of the calibration components and the first local point cloud data, including: Plane fitting is performed on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates.

11. The vehicle sensor calibration method based on the calibration system according to claim 12, characterized in that: Performing plane fitting on each of the calibration plates to extract second local point cloud data corresponding to each of the calibration plates includes: Obtaining a first height between the center of each calibration plate and the ground, randomly selecting at least three second point clouds from the first local point cloud, and calculating a second three-dimensional space plane where the second point cloud is located; Calculating a third distance between each point cloud in the second local point cloud data and the second three-dimensional space plane; The data corresponding to the point cloud whose difference between the third distance and the first height is not greater than the third preset distance is used as the second local point cloud data corresponding to the calibration plate.

12. The vehicle sensor calibration method based on the calibration system according to claim 9, characterized in that: The calibration plate includes a calibration carrier plate and a plurality of marking codes, wherein the plurality of marking codes are fixed on the calibration carrier plate, and the marking codes carry two-dimensional code information for identification by the sensor, wherein the identity information corresponding to the two-dimensional code information of all the marking codes is different; Determining second identity information corresponding to each of the calibration plates according to the second local point cloud data includes: Converting the second local point cloud data into a grayscale image; Segmenting the grayscale image according to the intervals between the marker codes to obtain a local image of each marker code; Perform two-dimensional code recognition on each of the partial images to determine the first ID information of each of the marking codes, and determine the second identity information of the calibration plate according to the first ID information of each of the marking codes on each of the marking plates.

13. The vehicle sensor calibration method based on the calibration system according to claim 12, characterized in that: Determining second coordinate information corresponding to each of the calibration plates according to the second local point cloud data includes: Extracting four sides of the circumscribed rectangle of each partial image of the logo code, and determining the intersection of two adjacent sides as a key point; Acquire the first image coordinates of each of the key points, where the first image coordinates represent the position of the key point on the calibration plate; The first three-dimensional coordinates corresponding to each of the key points are determined according to the position of the marking code on the calibration plate, the first image coordinates and the first reference three-dimensional coordinates of each point cloud in the second local point cloud data.

14. The vehicle sensor calibration method based on the calibration system according to claim 13, characterized in that: When there is no three-dimensional coordinate corresponding to the first image coordinate of the key point in the first reference three-dimensional coordinate, the method further includes: Obtaining three-dimensional coordinates corresponding to a plurality of reference points adjacent to the key point; The first three-dimensional coordinate corresponding to the key point is determined according to the three-dimensional coordinates of the plurality of reference points.

15. The vehicle sensor calibration method based on the calibration system according to claim 13, characterized in that: A metal block is also provided on the calibration plate, and second identity information corresponding to each calibration plate is determined according to the second local point cloud data, further comprising: The third identity information of the metal block is determined according to the first ID information of each of the marking codes.

16. The vehicle sensor calibration method based on the calibration system according to claim 15, characterized in that: Determining second coordinate information corresponding to each of the calibration plates according to the second local point cloud data also includes: The three-dimensional coordinates of the metal block are determined according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to each of the marking codes.

17. The vehicle sensor calibration method based on the calibration system according to claim 16, characterized in that: The metal block is fixed at the center of the calibration carrier, a plurality of marking codes are fixed on the calibration carrier and surround the metal block, and a plurality of vertices of the marking codes close to the metal block are connected to the vertices of the metal block.

18. The vehicle sensor calibration method based on the calibration system according to claim 17, characterized in that: Determining the three-dimensional coordinates of the metal block according to the position of the metal block on the calibration plate and the first three-dimensional coordinates of the key points corresponding to the marking codes includes: Determine the three-dimensional coordinates of the four corner points of the metal block according to the first three-dimensional coordinates of the key points corresponding to the vertices of each of the marking codes close to the direction of the metal block; The three-dimensional coordinates of the center of the metal block are determined according to the three-dimensional coordinates of the four corner points of the metal block.

19. The vehicle sensor calibration method based on the calibration system according to claim 16, characterized in that: When the sensor includes a camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes: Controlling the camera to be calibrated to collect data from the calibration device to obtain a first calibration image; Identify the marker code in the first calibration image, and determine the second ID information and the partial image of each marker code in the first calibration image; Determining the second image coordinates of each key point included in the first calibration image according to the ID information of each marker code in the first calibration image and the partial image; Determine the second three-dimensional coordinates corresponding to each of the key points in the first calibration image according to the ID information of each of the marking codes in the first calibration image, the second image coordinates of each of the key points, and the first reference three-dimensional coordinates; Determine the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates.

20. The vehicle sensor calibration method based on the calibration system according to claim 19, characterized in that: After determining the intrinsic parameters of the camera to be calibrated according to the second image coordinates and the second three-dimensional coordinates, the method further includes: Determine the extrinsic parameters of the camera to be calibrated according to the second image coordinates, the second three-dimensional coordinates and the intrinsic parameters of the camera to be calibrated; The external parameters of the camera to be calibrated include at least a first coordinate transformation relationship between the coordinate system of the camera to be calibrated and the coordinate system of the total station.

21. The vehicle sensor calibration method based on the calibration system according to claim 20, characterized in that: When the sensor includes a surround view camera to be calibrated, controlling each sensor on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes: Controlling the surround-view camera to be calibrated to collect data from the calibration device to obtain a second calibration image; Projecting the second calibration image using the surround stitching homography matrix to obtain a projected image; A second coordinate transformation relationship between the coordinate systems of two adjacent surround-view cameras to be calibrated is calculated according to the same key points in the projection images corresponding to the two adjacent surround-view cameras to be calibrated.

22. The vehicle sensor calibration method based on the calibration system according to claim 21, characterized in that: After controlling the surround view camera to be calibrated to collect data from the calibration device and obtaining a second calibration image, the method further includes: Extracting ID information of each marker code and third image coordinates of each key point in the second calibration image; Filtering the key points on the ground in the second calibration image according to the ID information of each marker code in the second calibration image and the third image coordinates of each key point, and obtaining the third three-dimensional coordinates of each key point on the ground; Determine a surround stitching image according to the third three-dimensional coordinates of each of the key points located on the ground; Calibrate the fourth three-dimensional coordinates of the key points in the surround view stitched image using the first reference three-dimensional coordinates corresponding to the key points in the surround view stitched image in the calibration prior information to obtain a calibrated surround view stitched image; The surround view stitching homography matrix is ​​obtained according to the calibrated surround view stitching image.

23. The vehicle sensor calibration method based on the calibration system as claimed in claim 20, characterized in that: When the sensor includes a main laser radar, controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes: Controlling the main laser radar to collect data from the calibration device to obtain first test point cloud data; The first test point cloud data is matched with the first data to obtain a third coordinate transformation relationship between the main lidar coordinate system and the total station coordinate system.

24. The vehicle sensor calibration method based on the calibration system as claimed in claim 23, characterized in that: After controlling the main laser radar to collect data from the calibration device and obtaining the first test point cloud data, the method further includes: Matching the first test point cloud data with the first data to obtain ID information of the marker code and the fifth three-dimensional coordinates of the key points within the field of view of the main laser radar; Determine the same key points in the first calibration image as those in the field of view of the main laser radar according to the ID information of each of the marker codes in the first calibration image, the second three-dimensional coordinates corresponding to each of the key points, the ID information of the marker codes within the field of view of the main laser radar, and the fifth three-dimensional coordinates of the key points, and determine the three-dimensional coordinates of the same key points; The fourth coordinate transformation relationship between the camera coordinate system to be calibrated and the main lidar coordinate system is calculated according to the three-dimensional coordinates of the same key point.

25. The vehicle sensor calibration method based on the calibration system as claimed in claim 24, characterized in that: When the sensor includes a millimeter wave radar, a metal block is provided on the calibration plate, and the calibration system further includes two tracks and a conveyor belt device provided on the tracks, the two tracks are parallel to each other and the distance between the two tracks is the same as the distance between the left and right wheels of the vehicle to be calibrated, and the vehicle to be calibrated is located on the conveyor belt device when the sensor calibration is performed; Before controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain the second data, the method further includes: By controlling the conveyor belt device to move the vehicle to be calibrated at a preset speed; Controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain second data, including: During the movement of the vehicle to be calibrated, the millimeter wave radar is triggered to collect the coordinates of the metal block within the field of view to obtain collected data, wherein the collected data at least includes the coordinates of the metal block; Determine alignment data between the first test point cloud data and the collected data according to the data timestamp; Projecting the sixth three-dimensional coordinate of the metal block in the first test point cloud data in the alignment data onto a horizontal plane to obtain horizontal plane projection data; The fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main lidar coordinate system is calculated according to the horizontal plane projection data, the coordinates of the metal blocks in the collected data in the alignment data, and the positions of each of the metal blocks in the calibration system.

26. The vehicle sensor calibration method based on the calibration system as claimed in claim 27, characterized in that: When the sensor includes an auxiliary laser radar, controlling each of the sensors on the vehicle to be calibrated to collect data from the calibration device to obtain second data includes: Controlling the auxiliary laser radar to collect data from the calibration device to obtain second test point cloud data; The second test point cloud data is matched with the first test point cloud data to obtain a sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system.

27. The vehicle sensor calibration method based on the calibration system according to any one of claims 6 to 26, characterized in that: The calibration device includes a plurality of verification components, and determining the calibration prior information according to the first calibration data includes: Performing spatial clustering segmentation on the first calibration data according to the position of each of the verification components in the calibration system to obtain third local point cloud data corresponding to each of the verification components; The fifth identity information and the fifth coordinate information corresponding to each of the verification components are determined according to the third local point cloud data.

28. The vehicle sensor calibration method based on the calibration system as claimed in claim 27, characterized in that: Determining fifth identity information and fifth coordinate information corresponding to each of the verification components according to the third local point cloud data includes: The third local point cloud data is fitted to obtain structural information and three-dimensional coordinates of each verification component.

29. The vehicle sensor calibration method based on the calibration system as claimed in claim 28, characterized in that: The verification component is a verification ball, and the third local point cloud data is fitted to obtain the structural information and three-dimensional coordinates of each of the verification components, including: Spherical fitting is performed on the third local point cloud data to obtain radius information and three-dimensional coordinates of the calibration sphere.

30. The vehicle sensor calibration method based on the calibration system as claimed in claim 29, characterized in that: After obtaining calibration priori information according to the first data, and calibrating each of the sensors on the vehicle to be calibrated according to the calibration priori information and the second data, the method further includes: The calibration parameters of each of the sensors are verified according to the radius information and three-dimensional coordinates of each of the verification balls.

31. The vehicle sensor calibration method based on the calibration system as claimed in claim 30, characterized in that: When the calibration parameters of the sensor include a third coordinate transformation relationship between the main laser radar coordinate system and the total station coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration sphere, including: Obtaining the seventh three-dimensional coordinates of each calibration ball within the field of view of the main laser radar; Performing coordinate transformation on the seventh three-dimensional coordinate according to the third coordinate transformation relationship to obtain a seventh three-dimensional coordinate to be compared; Comparing the seventh three-dimensional coordinate to be compared with the seventh reference three-dimensional coordinate to determine whether the third coordinate transformation relationship is accurate; The first reference three-dimensional coordinates are the three-dimensional coordinates obtained by the total station and corresponding to the verification component within the field of view of the main laser radar.

32. The vehicle sensor calibration method based on the calibration system as claimed in claim 30, characterized in that: When the calibration parameters of the sensor include a fourth coordinate transformation relationship between the coordinate system of the camera to be calibrated and the main laser radar coordinate system, the calibration parameters of each of the sensors are verified according to the radius information and the three-dimensional coordinates of each of the calibration spheres, including: Acquire first calibration point cloud data of a calibration sphere within the field of view of the main laser radar; Back-projecting the first verification point cloud data onto the camera image according to the fourth coordinate transformation relationship; Performing preset processing on the back-projected camera image to obtain a circular edge in the camera image, and determining a corresponding first circular fitting equation according to the circular edge; The first circle fitting equation is compared with a reference circle fitting equation to determine whether the fourth coordinate transformation relationship is accurate.

33. The vehicle sensor calibration method based on the calibration system as claimed in claim 30, characterized in that: The calibration ball is a metal sphere. When the calibration parameters of the sensor include the fifth coordinate transformation relationship between the millimeter wave radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration ball, including: Obtaining the millimeter-wave radar coordinates and angles of each calibration ball within the field of view of the millimeter-wave radar; Acquire a seventh three-dimensional coordinate within the field of view of the main laser radar; Projecting the millimeter wave radar coordinates and angles onto a two-dimensional plane of the main laser radar coordinate system according to the fifth coordinate transformation relationship; The Euclidean distance is calculated according to the projected millimeter-wave radar coordinates and angles and the seventh three-dimensional coordinates to determine whether the fifth coordinate transformation relationship is accurate.

34. The vehicle sensor calibration method based on the calibration system as claimed in claim 30, characterized in that: When the calibration parameters of the sensor include the sixth coordinate transformation relationship between the auxiliary laser radar coordinate system and the main laser radar coordinate system, the calibration parameters of each sensor are verified according to the radius information and the three-dimensional coordinates of each calibration sphere, including: Acquire the first calibration point cloud data of the calibration ball within the field of view of the main laser radar; Acquire the second calibration point cloud data of the calibration ball within the field of view of the auxiliary laser radar; The second verification point cloud data is transformed into the main laser radar coordinate system according to the sixth coordinate transformation relationship to obtain the second verification point cloud data to be verified; Determine whether the sixth coordinate transformation relationship is accurate based on the first verification point cloud data and the second verification point cloud data to be verified.

35. A vehicle sensor calibration system based on a calibration system, characterized in that: The calibration system includes a total station, a plurality of calibration devices, and a computing device, wherein the computing device includes: A first control unit, configured to control the total station to scan the calibration device to obtain first data when the vehicle to be calibrated meets a first calibration condition; A second control unit, configured to receive second data acquired by the calibration device from each of the sensors on the vehicle to be calibrated when a second calibration condition is met; A calibration unit is used to obtain calibration prior information based on the first data, and calibrate each of the sensors on the vehicle to be calibrated based on the calibration prior information and the second data to obtain calibration parameters of each sensor; the calibration prior information is data information of the calibration device, and the calibration parameters are used to characterize the coordinate transformation relationship between any two of the sensors or between the sensor and the total station.

36. A vehicle sensor calibration device based on a calibration system, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the vehicle sensor calibration method based on the calibration system as described in any one of claims 1-34 when storing a computer program.

37. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the vehicle sensor calibration method based on the calibration system as described in any one of claims 1 to 34 are implemented.

Images