Vehicle sensor extrinsic calibration method and device and computer readable storage medium

Abstract

The application discloses a vehicle sensor extrinsic calibration method, device and computer readable storage medium, and the method comprises the following steps: acquiring first road identification data of a corresponding high-definition map according to the current position of a vehicle; acquiring and processing first sensor data collected by a first sensor of the vehicle to obtain second road identification data; acquiring and processing second sensor data collected by a second sensor of the vehicle to obtain third road identification data; constructing a nonlinear optimization model according to the first road identification data, the second road identification data and the third road identification data; and determining the extrinsic parameters of the calibrated first sensor and the extrinsic parameters of the second sensor according to the nonlinear optimization model. The application can calibrate the extrinsic parameters of the sensors in real time according to the road identification data provided by the high-definition map and the road identification data of different sensors, and the robustness and precision of the extrinsic parameters of the sensors are improved through mutual constraints between the road identification data of different sources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a vehicle sensor extrinsic calibration method and device and a computer readable storage medium. BACKGROUND

[0002] In recent years, with the development of advanced driver assistance systems (ADAS) and automated driving (AD) technologies, research based on multi-sensor perception such as cameras and lidar has been widely applied.

[0003] Among them, the calibration accuracy of the extrinsic parameter of the sensor relative to the center of the vehicle body affects the measurement of the sensor, and further affects the actual application. For example, the road marking information such as lane lines, stop lines, and arrows detected from the road image collected by the camera is matched with the high-precision data in the high-definition map to determine the position of the vehicle during driving; the lidar can perceive the three-dimensional traffic signs around to determine the accurate position of the vehicle in the road, and then plan and control according to the driving destination. Since all detection results need to be converted to the vehicle body coordinate system through their extrinsic parameters for application, the quality of the sensor extrinsic parameter will affect the application of subsequent modules.

[0004] Many existing sensor calibration methods usually use offline calibration. Due to the inevitable vibration of the vehicle during driving, the position and angle of the sensor installation are inconsistent with the initial position, resulting in errors in the initially calibrated extrinsic parameters. To this end, some real-time calibration methods for sensors are provided in the prior art. However, these calibration methods usually only use data from one sensor for calibration, resulting in low robustness and accuracy of the calibration results. SUMMARY

[0005] The embodiments of the present application provide a vehicle sensor extrinsic calibration method, device and computer readable storage medium to improve the robustness and accuracy of the vehicle sensor extrinsic parameter.

[0006] The embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide a vehicle sensor extrinsic calibration method, wherein the method comprises:

[0008] According to the current position of the vehicle, the first road marking data in the corresponding high-definition map is obtained;

[0009] The first sensor data collected by the first sensor of the vehicle is obtained, and the first sensor data is processed to obtain second road marking data;

[0010] obtain second sensor data collected by a second sensor of the vehicle, and process the second sensor data to obtain third road mark data;

[0011] construct a non-linear optimization model according to the first road mark data, the second road mark data, and the third road mark data, the non-linear optimization model being used to calibrate the extrinsic parameters of the first sensor and the extrinsic parameters of the second sensor;

[0012] determine the calibrated extrinsic parameters of the first sensor and the calibrated extrinsic parameters of the second sensor according to the non-linear optimization model.

[0013] Optionally, the constructing the non-linear optimization model according to the first road mark data, the second road mark data, and the third road mark data comprises:

[0014] converting the first road mark data, the second road mark data, and the third road mark data into a vehicle body coordinate system;

[0015] constructing a residual constraint according to the first road mark data, the second road mark data, and the third road mark data in the vehicle body coordinate system;

[0016] constructing the non-linear optimization model according to the residual constraint.

[0017] Optionally, the first road mark data comprises 3D road mark points of a high-definition map, the second road mark data comprises 2D road mark points of a road image, and the third road mark data comprises 3D road mark points of a laser radar, and the constructing the non-linear optimization model according to the first road mark data, the second road mark data, and the third road mark data comprises:

[0018] constructing a first residual constraint according to the 3D road mark points of the high-definition map and the 2D road mark points of the road image;

[0019] constructing a second residual constraint according to the 2D road mark points of the road image and the 3D road mark points of the laser radar;

[0020] constructing a third residual constraint according to the 3D road mark points of the high-definition map and the 3D road mark points of the laser radar;

[0021] constructing the non-linear optimization model according to the first residual constraint, the second residual constraint, and the third residual constraint.

[0022] Optionally, the constructing the first residual constraint according to the 3D road mark points of the high-definition map and the 2D road mark points of the road image comprises:

[0023] projecting the 3D road mark points of the high-definition map into the road image to obtain 2D road mark points of the high-definition map;

[0024] constructing the first residual constraint according to the 2D road mark points of the high-definition map and the 2D road mark points of the road image.

[0025] Optionally, the 2D road mark points of the high-definition map and the 2D road mark points of the road image are both 2D road mark points of linear road marks, and the constructing the first residual constraint according to the 2D road mark points of the high-definition map and the 2D road mark points of the road image comprises:

[0026] fitting the 2D road mark points of linear road marks in the high-definition map into a straight line;

[0027] constructing a residual constraint of point-to-line distance according to the distance between the 2D road mark points of linear road marks in the road image and the straight line.

[0028] Optionally, the constructing the second residual constraint according to the 2D road mark points of the road image and the 3D road mark points of the lidar comprises:

[0029] projecting the 3D road mark points of the lidar into the road image to obtain 2D road mark points of the lidar;

[0030] constructing the second residual constraint according to the 2D road mark points of the lidar and the 2D road mark points of the road image.

[0031] Optionally, the constructing the third residual constraint according to the 3D road mark points of the high-definition map and the 3D road mark points of the lidar comprises:

[0032] matching the 3D road mark points of the high-definition map and the 3D road mark points of the lidar;

[0033] constructing the third residual constraint according to the matching result.

[0034] Optionally, after constructing the residual constraint according to the first road mark data, the second road mark data and the third road mark data in the vehicle body coordinate system, the method further comprises:

[0035] obtaining the historical extrinsic parameters of the first sensor and the historical extrinsic parameters of the second sensor;

[0036] construct an extrinsic parameter prior constraint according to the historical extrinsic parameter of the first sensor and the historical extrinsic parameter of the second sensor;

[0037] construct the nonlinear optimization model according to the residual constraint and the extrinsic parameter prior constraint.

[0038] In a second aspect, the embodiments of the present application further provide a device for calibrating extrinsic parameters of vehicle sensors, wherein the device comprises:

[0039] a first obtaining unit configured to obtain first road identification data in a high-definition map corresponding to a current position of a vehicle;

[0040] a second obtaining unit configured to obtain first sensor data collected by a first sensor of the vehicle, and process the first sensor data to obtain second road identification data;

[0041] a third obtaining unit configured to obtain second sensor data collected by a second sensor of the vehicle, and process the second sensor data to obtain third road identification data;

[0042] a first constructing unit configured to construct a nonlinear optimization model according to the first road identification data, the second road identification data and the third road identification data, the nonlinear optimization model being used to calibrate extrinsic parameters of the first sensor and extrinsic parameters of the second sensor;

[0043] a determining unit configured to determine the calibrated extrinsic parameters of the first sensor and the calibrated extrinsic parameters of the second sensor according to the nonlinear optimization model.

[0044] In a third aspect, the embodiments of the present application further provide an electronic device, comprising:

[0045] a processor; and

[0046] a memory arranged to store computer-executable instructions that, when executed, cause the processor to perform any of the aforementioned methods.

[0047] In a fourth aspect, the embodiments of the present application further provide a computer-readable storage medium storing one or more programs, which, when executed by an electronic device comprising multiple applications, cause the electronic device to perform any of the aforementioned methods.

[0048] The at least one technical scheme adopted by the embodiment of the present application can achieve the following beneficial effects: the vehicle sensor extrinsic calibration method of the embodiment of the present application first acquires first road mark data in the corresponding high-definition map according to the current position of the vehicle; then first sensor data collected by a first sensor of the vehicle is acquired, and the first sensor data is processed to obtain second road mark data; then second sensor data collected by a second sensor of the vehicle is acquired, and the second sensor data is processed to obtain third road mark data; then a nonlinear optimization model is constructed according to the first road mark data, the second road mark data and the third road mark data, and the nonlinear optimization model is used to calibrate the extrinsic parameters of the first sensor and the extrinsic parameters of the second sensor; finally, the extrinsic parameters of the calibrated first sensor and the extrinsic parameters of the calibrated second sensor are determined according to the nonlinear optimization model. The vehicle sensor extrinsic calibration method of the embodiment of the present application can calibrate the extrinsic parameters of the sensors in real time according to the road mark data provided by the high-definition map and the road mark data of different sensors, and the robustness and precision of the extrinsic parameters of the sensors are improved through mutual constraints between road mark data of different sources. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and the explanation of the present application, and do not constitute improper limitations on the present application. In the drawings:

[0050] Figure 1 A flowchart of a vehicle sensor extrinsic calibration method in an embodiment of the present application is shown;

[0051] Figure 2 A comparison diagram before and after extrinsic calibration of a vehicle sensor in an embodiment of the present application is shown;

[0052] Figure 3 A structure diagram of a vehicle sensor extrinsic calibration device in an embodiment of the present application is shown;

[0053] Figure 4 A structure diagram of an electronic device in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] To make the purpose, technical scheme and advantages of the present application clearer, the technical scheme of the present application will be described clearly and completely in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0055] The technical solutions provided by the embodiments of the present application are described in detail below with reference to the drawings.

[0056] The present application provides a vehicle sensor extrinsic calibration method, as shown in the accompanying drawings, a flowchart of a vehicle sensor extrinsic calibration method in the embodiments of the present application is provided, and the method comprises at least the following steps S110 to S140: Figure 1

[0057] Step S110, according to the current position of the vehicle, the first road identification data in the corresponding high-definition map is obtained.

[0058] In the calibration of the extrinsic parameters of the sensor of the vehicle, the current position of the vehicle can be determined first, which can be obtained based on the RTK (Real-Time Kinematic, real-time differential) positioning device installed on the vehicle, or can also be obtained according to the positioning result output by the combined navigation system such as RTK+IMU (Inertial Measurement Unit, inertial measurement unit) on the vehicle. The current position of the vehicle can be determined according to the actual needs of the person skilled in the art, and the specific determination method is not limited herein.

[0059] After obtaining the current position of the vehicle, the first road identification data in the corresponding local high-definition map at this position can be further obtained, which is used as the first data source for subsequent extrinsic calibration. The first road identification data may, for example, include lane lines, arrow road marking data, and speed limit signs, road signs and other three-dimensional identification data.

[0060] Step S120, the first sensor data collected by the first sensor of the vehicle is obtained, and the first sensor data is processed to obtain second road identification data.

[0061] In the calibration of the extrinsic parameters of the sensor of the vehicle, the first sensor data collected by the first sensor of the vehicle is also needed, and the first sensor may, for example, be a camera installed on the vehicle, and the first sensor data may, for example, include road images collected by the camera. By using existing road surface recognition models such as Lanenet, the road images can be recognized, and the second road identification data can be obtained, which may, for example, include lane lines, stop lines, arrows and other road marking data, and is used as the second data source for subsequent extrinsic calibration.

[0062] Step S130, the second sensor data collected by the second sensor of the vehicle is obtained, and the second sensor data is processed to obtain third road identification data.

[0063] ​The embodiment of the application also needs to obtain second sensor data collected by a second sensor of the vehicle when calibrating the extrinsic parameters of the sensor of the vehicle. The second sensor may, for example, be a laser radar installed on the vehicle. The second sensor data may include three-dimensional point cloud data collected by the laser radar. The third road mark data may be obtained by processing the three-dimensional point cloud data. Specifically, the third road mark data may include lane lines, stop lines, arrow road mark data, and three-dimensional mark data such as speed limit signs and road signs, which are used as a third data source for subsequent extrinsic parameter calibration.

[0064] It should be noted that the types of the first sensor and the second sensor are only exemplary descriptions. In actual applications, other types of vehicle sensors such as millimeter wave radars may be flexibly extended, as long as the data collected by the sensors can provide road mark information. In addition, the number of sensors is not limited to two. Three or more sensors may also be applicable to the overall processing flow of the embodiment of the application.

[0065] In step S140, a nonlinear optimization model is constructed according to the first road mark data, the second road mark data, and the third road mark data. The nonlinear optimization model is used to calibrate the extrinsic parameters of the first sensor and the extrinsic parameters of the second sensor.

[0066] In step S150, the extrinsic parameters of the calibrated first sensor and the extrinsic parameters of the calibrated second sensor are determined according to the nonlinear optimization model.

[0067] The extrinsic parameters of the first sensor of the embodiment of the application may be regarded as a conversion relationship of the first sensor to the vehicle body coordinate system. The extrinsic parameters of the second sensor may be regarded as a conversion relationship of the second sensor to the vehicle body coordinate system. The conversion relationship may specifically include a pitch angle, a yaw angle, a roll angle, and a sensor height.

[0068] In the case where the initial calibrated extrinsic parameters remain unchanged, the same road mark data collected by different sensors of the vehicle for the current position of the vehicle should be consistent after conversion by the initial calibrated extrinsic parameters. However, in actual situations, as the vehicle travels, the installation positions of different sensors may shift, resulting in a certain deviation between the road mark data of the multiple sensors based on the initial calibrated extrinsic parameters. Therefore, the initial calibrated extrinsic parameters need to be calibrated.

[0069] Based on this, the embodiment of the application may construct a nonlinear optimization model according to the above-mentioned multiple road mark data of different sources. By solving the nonlinear optimization model, the deviation between the multiple road mark data of different sources is constrained, and then the calibrated extrinsic parameters of the first sensor and the calibrated extrinsic parameters of the second sensor are obtained.

[0070] The nonlinear optimization model can be implemented by using a least square algorithm, and the preset gradient descent algorithm, such as a Gauss-Newton algorithm or a Levenberg-Marquardt algorithm, can be used to solve the calibrated sensor extrinsic parameter. Of course, the specific algorithm can be selected by a person skilled in the art according to actual needs, and is not limited herein.

[0071] The vehicle sensor extrinsic parameter calibration method can calibrate the extrinsic parameter of the sensor in real time according to the road mark data provided by the high-definition map and the road mark data of different sensors, and the robustness and accuracy of the sensor extrinsic parameter are improved through mutual constraints between road mark data of different sources.

[0072] In an embodiment of the present application, the method comprises: converting the first road mark data, the second road mark data and the third road mark data into a vehicle body coordinate system; constructing a residual constraint according to the first road mark data, the second road mark data and the third road mark data in the vehicle body coordinate system; and constructing the nonlinear optimization model according to the residual constraint.

[0073] In the construction of the nonlinear optimization model, the plurality of road mark data of different sources can be converted into the vehicle body coordinate system, for example, the first road mark data provided by the high-definition map is in the WGS-84 coordinate system, and therefore the first road mark data can be converted into the vehicle body coordinate system before subsequent processing.

[0074] Similarly, the second road mark data provided by the camera is in the camera coordinate system, and therefore the second road mark data can be converted into the vehicle body coordinate system through the initial extrinsic parameter of the camera. The third road mark data provided by the laser radar is in the laser radar coordinate system, and therefore the third road mark data can be converted into the vehicle body coordinate system through the initial extrinsic parameter of the laser radar.

[0075] It should be noted that, unless otherwise specified, the road mark data mentioned in the following embodiments can be regarded as road mark data converted into the vehicle body coordinate system.

[0076] Through the above processing, the first road mark data, the second road mark data and the third road mark data in the vehicle body coordinate system can be obtained, based on which a residual constraint of a nonlinear optimization model can be further constructed. The residual constraint can be used to constrain the deviation between road mark data from multiple different sources. By constraining the residual of the nonlinear optimization model, the nonlinear optimization model gradually converges, so that the optimal solution of the nonlinear optimization model is obtained.

[0077] In an embodiment of the present application, the first road mark data includes 3D road mark points of a high-definition map, the second road mark data includes 2D road mark points of a road image, and the third road mark data includes 3D road mark points of a laser radar. The constructing of the nonlinear optimization model according to the first road mark data, the second road mark data and the third road mark data includes: constructing a first residual constraint according to the 3D road mark points of the high-definition map and the 2D road mark points of the road image; constructing a second residual constraint according to the 2D road mark points of the road image and the 3D road mark points of the laser radar; constructing a third residual constraint according to the 3D road mark points of the high-definition map and the 3D road mark points of the laser radar; and constructing the nonlinear optimization model according to the first residual constraint, the second residual constraint and the third residual constraint.

[0078] Since the high-definition map and the laser radar can both collect three-dimensional data, while the camera can only collect two-dimensional data, the first road mark data according to the embodiment of the present application can specifically include 3D road mark points in the high-definition map, the second road mark data can specifically include 2D road mark points in the road image, and the third road mark data can specifically include 3D road mark points of the laser radar.

[0079] Since the high-definition map is often collected and constructed based on high-precision positioning equipment, the 3D road mark points and other data provided by the high-definition map can be regarded as high-precision data, which can be used as the basis for calibrating the sensor extrinsic parameters. Therefore, on the one hand, the 2D road mark points of the road image can be compared with the 3D road mark points provided by the high-definition map, so that a first residual constraint can be constructed according to the deviation between the two, and then the real-time calibration of the camera extrinsic parameters can be realized by using the various road surface mark information in the high-definition map. On the other hand, the 3D road mark points of the laser radar can also be compared with the 3D road mark points provided by the high-definition map, so that a third residual constraint can be constructed according to the deviation between the two, and then the real-time calibration of the laser radar extrinsic parameters can be realized by using the various road surface mark and three-dimensional mark information in the high-definition map.

[0080] In addition to constructing a corresponding residual constraint on the multi-sensor extrinsic calibration based on the road marking data of the high-definition map, the road marking data between the multiple sensors can also be mutually constrained. Specifically, the 2D road marking points of the road image can be compared with the 3D road marking points of the lidar, so that a second residual constraint can be constructed according to the degree of deviation between the two, and then the calibration of the extrinsic parameters of different sensors is constrained by the data between the sensors, thereby further improving the robustness and precision of the sensor extrinsic calibration.

[0081] The above process can be regarded as mutual constraint among the high-definition map, the camera, and the road marking data of the sensor. Since the data of the high-definition map is relatively accurate, the essence is to constrain the extrinsic calibration process of the camera and the sensor. When extended to more sensors, the above processing logic can also be used to realize the extrinsic calibration of more sensors.

[0082] In an embodiment of the present application, constructing the first residual constraint according to the 3D road marking points of the high-definition map and the 2D road marking points of the road image comprises: projecting the 3D road marking points of the high-definition map into the road image to obtain the 2D road marking points of the high-definition map; and constructing the first residual constraint according to the 2D road marking points of the high-definition map and the 2D road marking points of the road image.

[0083] In constructing the first residual constraint, the 3D road marking points of the high-definition map can be first converted through the initial extrinsic parameters and the initial intrinsic parameters of the camera and then projected into the road image. Specifically, it is assumed that the camera imaging model is as follows:

[0084]

[0085] For a point (x c , y c , z c ) in the vehicle coordinate system, after conversion through the camera extrinsic parameters and the camera intrinsic parameters , it can be projected to (u, v) of the road image.

[0086] Then, the corresponding 2D road marking points of the high-definition map in the road image and the original 2D road marking points of the road image are compared, so that the first residual constraint can be constructed according to the degree of deviation between the two.

[0087] In an embodiment of the present application, the 2D road mark points of the high-definition map and the 2D road mark points of the road image are both 2D road mark points of linear road marks, and the constructing the first residual constraint according to the 2D road mark points of the high-definition map and the 2D road mark points of the road image comprises: fitting the 2D road mark points of linear road marks in the high-definition map into a straight line; and constructing a residual constraint of point-to-line distance according to the distance between the 2D road mark points of linear road marks in the road image and the straight line.

[0088] Since the road image can only provide two-dimensional road mark information, and the high-definition map provides three-dimensional road mark information, the correspondence between the detected road mark points in the road image and the road mark points in the high-definition map cannot be directly determined, i.e., the specific correspondence between points cannot be determined. Therefore, for linear road marks such as lane lines and stop lines, the present application can use the distance from a point to a line to measure the error, and construct a residual constraint of line distance. For example, for the 2D road mark points of the high-definition map and the road image, the 2D road mark points of the high-definition map can be fitted into a straight line, and then the distance from each 2D road mark point in the road image to the straight line can be calculated, and a residual constraint of point-to-line distance can be constructed.

[0089] To make the embodiments of the present application clear, lane lines are taken as an example for further explanation and description. In the process of straight driving, the lane line points within a certain radius around the vehicle are fitted, and it is assumed that the lane line conforms to a two-element first-order equation, i.e., y=ax+b. Through the lane line points in the vehicle coordinate system and the parameter equation, the least square method can be used to fit:

[0090]

[0091] It is assumed that the lane line points detected by the image are , and the coordinates of the lane line points in the vehicle coordinate system are P car =[x c y c z c ] T , and the lane line matched in the high-definition map is , which can be represented by a starting point P a =[x a y a z a ] T and an ending point P b =[x b y b z b ] T , and the corresponding vector representation is P car , then the distance from the point P ab to the straight line The error can be expressed as:

[0092]

[0093] wherein V ab is the unit vector of the vector (P b -P a ).

[0094] In an embodiment of the present application, the constructing the second residual constraint according to the 2D road mark points of the road image and the 3D road mark points of the lidar comprises: projecting the 3D road mark points of the lidar into the road image to obtain 2D road mark points of the lidar; and constructing the second residual constraint according to the 2D road mark points of the lidar and the 2D road mark points of the road image.

[0095] In the construction of the second residual constraint, the 3D road mark points of the lidar can also be projected into the road image through the conversion of the initial extrinsic parameters and the initial intrinsic parameters of the camera. Then, the corresponding 2D road mark points of the lidar in the road image and the original 2D road mark points of the road image are compared, so that the second residual constraint can be constructed according to the deviation between the two.

[0096] Specifically, based on the foregoing embodiments, the road mark points in the lidar and the road mark points in the image cannot be one-to-one corresponding, so the distance from the point to the line can also be used to measure the error, that is, the 2D road mark points of the road image can be fitted into a straight line, and then the distance from each 2D road mark point of the lidar to the straight line is calculated, and the point-to-line distance residual constraint is constructed. It should be noted that, specifically, which road mark points are fitted into a straight line can be adjusted flexibly by those skilled in the art according to actual needs.

[0097] In an embodiment of the present application, the constructing the third residual constraint according to the 3D road mark points of the high-definition map and the 3D road mark points of the lidar comprises: matching the 3D road mark points of the high-definition map and the 3D road mark points of the lidar; and constructing the third residual constraint according to the matching result.

[0098] Since the high-definition map and the lidar can both provide three-dimensional road mark information, the 3D road mark points in the high-definition map and the 3D road mark points obtained by the lidar can be directly matched point by point, for example, the error can be measured by the Euclidean distance between the 3D road mark points in the high-definition map and the 3D road mark points of the lidar, and the third residual constraint is constructed.

[0099] In an embodiment of the present application, after constructing the residual constraint according to the first road mark data, the second road mark data and the third road mark data in the vehicle body coordinate system, the method further comprises: obtaining the historical extrinsic parameters of the first sensor and the historical extrinsic parameters of the second sensor; constructing an extrinsic parameter prior constraint according to the historical extrinsic parameters of the first sensor and the historical extrinsic parameters of the second sensor; and constructing the nonlinear optimization model according to the residual constraint and the extrinsic parameter prior constraint.

[0100] In addition to the construction of the residual constraint in the foregoing embodiments, the embodiments of the present application can further obtain the historical extrinsic parameters corresponding to each sensor, take the historical extrinsic parameters corresponding to each sensor as a prior constraint, and construct a nonlinear optimization model together with the residual constraint. Since the extrinsic parameter calibration process of the embodiments of the present application is performed in real time online, the historical extrinsic parameters described above can be regarded as the extrinsic parameters of each sensor after calibration at the previous time, and of course, the initially calibrated extrinsic parameters can also be used as a fixed prior condition for constraint.

[0101] Although the extrinsic parameters corresponding to each sensor change during the movement of the vehicle, resulting in inaccurate extrinsic parameters, such changes are not large, and therefore introducing the historical extrinsic parameters corresponding to each sensor as a prior constraint can greatly improve the convergence speed of the nonlinear optimization model.

[0102] In order to illustrate the calibration effect of the vehicle sensor extrinsic parameter calibration method of the present application, as shown in Figure 2 , a comparison diagram before and after the calibration of the vehicle sensor extrinsic parameter calibration method in an embodiment of the present application is provided, Figure 2 The black lane line points on the left half represent the lane line points in the high-definition map, and the gray lane line points represent the lane line points detected based on the initial extrinsic parameters. Figure 2 The black lane line points on the right half also represent the lane line points in the high-definition map, and the gray lane line points represent the lane line points detected based on the calibrated extrinsic parameters. It can be seen that the calibrated extrinsic parameters based on the vehicle sensor extrinsic parameter calibration method of the present application can detect more accurate lane line marks, providing strong data support for other automatic driving modules.

[0103] The embodiments of the present application also provide a vehicle sensor extrinsic parameter calibration device 300, as shown in Figure 3 , a structure diagram of a vehicle sensor extrinsic parameter calibration device in an embodiment of the present application is provided, the device 300 comprises: a first obtaining unit 310, a second obtaining unit 320, a third obtaining unit 330, a first constructing unit 340 and a determining unit 350, wherein:

[0104] The first obtaining unit 310 is configured to obtain first road mark data in a corresponding high-definition map according to a current position of a vehicle.

[0105] The second acquisition unit 320 is configured to acquire first sensor data collected by a first sensor of the vehicle, and process the first sensor data to obtain second road mark data.

[0106] The third acquisition unit 330 is configured to acquire second sensor data collected by a second sensor of the vehicle, and process the second sensor data to obtain third road mark data.

[0107] The first construction unit 340 is configured to construct a nonlinear optimization model according to the first road mark data, the second road mark data, and the third road mark data, the nonlinear optimization model being used to calibrate extrinsic parameters of the first sensor and extrinsic parameters of the second sensor.

[0108] The determination unit 350 is configured to determine the calibrated extrinsic parameters of the first sensor and the calibrated extrinsic parameters of the second sensor according to the nonlinear optimization model.

[0109] In an embodiment of the present application, the first construction unit 340 is specifically configured to: convert the first road mark data, the second road mark data, and the third road mark data to a vehicle body coordinate system; construct a residual constraint according to the first road mark data, the second road mark data, and the third road mark data in the vehicle body coordinate system; and construct the nonlinear optimization model according to the residual constraint.

[0110] In an embodiment of the present application, the first road mark data includes 3D road mark points of a high-definition map, the second road mark data includes 2D road mark points of a road image, and the third road mark data includes 3D road mark points of a laser radar. The first construction unit 340 is specifically configured to: construct a first residual constraint according to the 3D road mark points of the high-definition map and the 2D road mark points of the road image; construct a second residual constraint according to the 2D road mark points of the road image and the 3D road mark points of the laser radar; construct a third residual constraint according to the 3D road mark points of the high-definition map and the 3D road mark points of the laser radar; and construct the nonlinear optimization model according to the first residual constraint, the second residual constraint, and the third residual constraint.

[0111] In an embodiment of the present application, the first construction unit 340 is specifically configured to: project the 3D road mark points of the high-definition map into the road image to obtain 2D road mark points of the high-definition map; and construct the first residual constraint according to the 2D road mark points of the high-definition map and the 2D road mark points of the road image.

[0112] In an embodiment of the present application, the 2D road mark points of the high-definition map and the 2D road mark points of the road image are both 2D road mark points of linear road marks, and the first constructing unit 340 is specifically configured to: fit the 2D road mark points of linear road marks in the high-definition map into a straight line; and construct a point-to-line distance residual constraint according to the distance between the 2D road mark points of linear road marks in the road image and the straight line.

[0113] In an embodiment of the present application, the first constructing unit 340 is specifically configured to: project the 3D road mark points of the lidar into the road image to obtain 2D road mark points of the lidar; and construct the second residual constraint according to the 2D road mark points of the lidar and the 2D road mark points of the road image.

[0114] In an embodiment of the present application, the first constructing unit 340 is specifically configured to: match the 3D road mark points of the high-definition map and the 3D road mark points of the lidar; and construct the third residual constraint according to the matching result.

[0115] In an embodiment of the present application, the apparatus further includes: a fourth obtaining unit configured to obtain historical extrinsic parameters of the first sensor and historical extrinsic parameters of the second sensor; a second constructing unit configured to construct an extrinsic parameter prior constraint according to the historical extrinsic parameters of the first sensor and the historical extrinsic parameters of the second sensor; and a third constructing unit configured to construct the nonlinear optimization model according to the residual constraint and the extrinsic parameter prior constraint.

[0116] It can be understood that the vehicle sensor extrinsic parameter calibration apparatus described above can implement each step of the vehicle sensor extrinsic parameter calibration method provided in the foregoing embodiments, and the related explanations about the vehicle sensor extrinsic parameter calibration method are all applicable to the vehicle sensor extrinsic parameter calibration apparatus, which will not be repeated here.

[0117] Figure 4 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 4 At the hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The memory can include a memory such as a random-access memory (RAM), and can also include a non-volatile memory such as at least one disk memory. Of course, the electronic device can also include other hardware required by the business.

[0118] The processor, the network interface and the memory can be connected with each other through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0119] The memory is used to store programs. Specifically, the program can include program code including computer operation instructions. The memory can include an internal memory and a non-volatile memory, and provide instructions and data for the processor.

[0120] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs, and forms the extrinsic parameter calibration device of the vehicle sensor at a logical level. The processor executes the program stored in the memory, and is specifically used for executing the following operations:

[0121] According to the current position of the vehicle, first road identification data in the corresponding high-precision map is obtained;

[0122] First sensor data collected by a first sensor of the vehicle is obtained, and the first sensor data is processed to obtain second road identification data;

[0123] Second sensor data collected by a second sensor of the vehicle is obtained, and the second sensor data is processed to obtain third road identification data;

[0124] According to the first road identification data, the second road identification data and the third road identification data, a nonlinear optimization model is constructed, and the nonlinear optimization model is used to calibrate the extrinsic parameters of the first sensor and the extrinsic parameters of the second sensor;

[0125] According to the nonlinear optimization model, the calibrated extrinsic parameters of the first sensor and the calibrated extrinsic parameters of the second sensor are determined.

[0126] The above as described in the present application Figure 1The method performed by the vehicle sensor extrinsic calibration device disclosed in the embodiment can be applied in a processor or implemented by the processor. The processor can be an integrated circuit chip with signal processing capability. In the implementation, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software. The processor can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; or a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiment of the present application can be implemented or executed. The general processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0127] The electronic device can also execute Figure 1 The method performed by the vehicle sensor extrinsic calibration device, and implement the functions of the vehicle sensor extrinsic calibration device in Figure 1 The embodiment disclosed in the embodiment, which will not be repeated here.

[0128] The embodiment of the present application also proposes a computer readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by an electronic device including a plurality of application programs, can enable the electronic device to execute Figure 1 The method performed by the vehicle sensor extrinsic calibration device disclosed in the embodiment, and specifically for executing

[0129] According to the current position of the vehicle, the first road identification data in the corresponding high-precision map is obtained;

[0130] The first sensor data collected by the first sensor of the vehicle is obtained, and the first sensor data is processed to obtain second road identification data.

[0131] obtain second sensor data collected by a second sensor of the vehicle, and process the second sensor data to obtain third road mark data;

[0132] construct a non-linear optimization model according to the first road mark data, the second road mark data, and the third road mark data, the non-linear optimization model being used to calibrate the extrinsic parameters of the first sensor and the extrinsic parameters of the second sensor;

[0133] determine the calibrated extrinsic parameters of the first sensor and the calibrated extrinsic parameters of the second sensor according to the non-linear optimization model.

[0134] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0135] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagrams, and combinations of flows and / or blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram.

[0136] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufacture including an instruction means, which implements the functions specified in the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram.

[0137] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0138] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0139] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), flash memory, or a combination of non-volatile memories in different types. The memory is an example of computer readable storage media.

[0140] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0142] ​​Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code thereon for use by or in connection with an instruction execution system. For the purposes of this description, a computer-usable or computer readable storage medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0143] The foregoing is merely illustrative of the principles of the application and various modifications can be made by persons skilled in the art. The present application is not intended to be limited to the embodiments shown, but is to be accorded the full scope that resides in the art thereof. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for calibrating external parameters of a vehicle sensor, wherein: The method comprises: Obtaining first road sign data in the corresponding high-precision map according to the current position of the vehicle; Acquiring first sensor data collected by a first sensor of the vehicle, and processing the first sensor data to obtain second road sign data; Acquiring second sensor data collected by a second sensor of the vehicle, and processing the second sensor data to obtain third road sign data; constructing a nonlinear optimization model based on the first road sign data, the second road sign data, and the third road sign data, wherein the nonlinear optimization model is used to calibrate the extrinsic parameters of the first sensor and the extrinsic parameters of the second sensor; Determining, according to the nonlinear optimization model, external parameters of the calibrated first sensor and external parameters of the calibrated second sensor; The first road sign data includes road sign data of a high-precision map, road sign data of a first sensor, and road sign data of a second sensor. Constructing a nonlinear optimization model based on the first road sign data, the second road sign data, and the third road sign data includes: Constructing residual constraints between the high-precision map, the first sensor, and the second sensor based on the road sign data of the high-precision map, the road sign data of the first sensor, and the road sign data of the second sensor; Constructing the nonlinear optimization model according to the residual constraints between the high-precision map, the first sensor, and the second sensor; After constructing the residual constraint based on the first road sign data, the second road sign data, and the third road sign data in the vehicle body coordinate system, the method further includes: Acquire historical external parameters of the first sensor and historical external parameters of the second sensor; Constructing an extrinsic parameter prior constraint based on the historical extrinsic parameters of the first sensor and the historical extrinsic parameters of the second sensor; The nonlinear optimization model is constructed according to the residual constraint and the extrinsic parameter prior constraint.

2. The method according to claim 1, wherein: The constructing of a nonlinear optimization model according to the first road sign data, the second road sign data, and the third road sign data includes: Converting the first road sign data, the second road sign data, and the third road sign data into a vehicle body coordinate system; constructing a residual constraint according to the first road sign data, the second road sign data, and the third road sign data in the vehicle body coordinate system; The nonlinear optimization model is constructed according to the residual constraint.

3. The method according to claim 1, wherein: The first road marking data includes 3D road marking points of a high-precision map, the second road marking data includes 2D road marking points of a road image, and the third road marking data includes 3D road marking points of a laser radar. Constructing a nonlinear optimization model based on the first road marking data, the second road marking data, and the third road marking data includes: Constructing a first residual constraint based on the 3D road marking points of the high-precision map and the 2D road marking points of the road image; Constructing a second residual constraint based on the 2D road marking points of the road image and the 3D road marking points of the laser radar; Constructing a third residual constraint based on the 3D road marking points of the high-precision map and the 3D road marking points of the laser radar; The nonlinear optimization model is constructed according to the first residual constraint, the second residual constraint, and the third residual constraint.

4. The method according to claim 3, wherein: The constructing of the first residual constraint according to the 3D road marking points of the high-precision map and the 2D road marking points of the road image includes: Projecting the 3D road marking points of the high-precision map onto the road image to obtain the 2D road marking points of the high-precision map; The first residual constraint is constructed based on the 2D road marking points of the high-precision map and the 2D road marking points of the road image.

5. The method according to claim 4, wherein: The 2D road marking points of the high-precision map and the 2D road marking points of the road image are both 2D road marking points of linear road markings, and constructing the first residual constraint based on the 2D road marking points of the high-precision map and the 2D road marking points of the road image includes: Fitting the 2D road marking points of the linear road markings in the high-precision map into a straight line; A residual constraint of the distance from the point to the straight line is constructed according to the distance between the 2D road marking point of the linear road marking in the road image and the straight line.

6. The method of claim 3, wherein: The constructing of the second residual constraint according to the 2D road marking points of the road image and the 3D road marking points of the laser radar includes: Projecting the 3D road marking points of the laser radar onto the road image to obtain the 2D road marking points of the laser radar; The second residual constraint is constructed according to the 2D road marking points of the laser radar and the 2D road marking points of the road image.

7. The method of claim 3, wherein: The constructing of the third residual constraint according to the 3D road marking points of the high-precision map and the 3D road marking points of the laser radar includes: Matching the 3D road marking points of the high-precision map with the 3D road marking points of the laser radar; The third residual constraint is constructed according to the matching result.

8. An external parameter calibration device for a vehicle sensor, wherein: The device comprises: A first acquiring unit is configured to acquire first road identification data in a corresponding high-precision map according to a current position of the vehicle; a second acquiring unit, configured to acquire first sensor data collected by a first sensor of the vehicle, and process the first sensor data to obtain second road sign data; a third acquiring unit, configured to acquire second sensor data collected by a second sensor of the vehicle, and process the second sensor data to obtain third road sign data; a first construction unit, configured to construct a nonlinear optimization model based on the first road sign data, the second road sign data, and the third road sign data, wherein the nonlinear optimization model is used to calibrate extrinsic parameters of the first sensor and the second sensor; a determining unit, configured to determine, according to the nonlinear optimization model, an extrinsic parameter of the calibrated first sensor and an extrinsic parameter of the calibrated second sensor; The first road sign data includes road sign data of a high-precision map, road sign data of a first sensor, and road sign data of a second sensor. The first construction unit is specifically configured to: Constructing residual constraints between the high-precision map, the first sensor, and the second sensor based on the road sign data of the high-precision map, the road sign data of the first sensor, and the road sign data of the second sensor; Constructing the nonlinear optimization model according to the residual constraints between the high-precision map, the first sensor, and the second sensor; The device further comprises: a fourth acquiring unit, configured to acquire historical extrinsic parameters of the first sensor and historical extrinsic parameters of the second sensor; a second constructing unit, configured to construct an extrinsic parameter prior constraint based on the historical extrinsic parameter of the first sensor and the historical extrinsic parameter of the second sensor; The third construction unit is used to construct the nonlinear optimization model according to the residual constraint and the external parameter prior constraint.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device comprising a plurality of application programs, enables the electronic device to perform the extrinsic parameter calibration method for a vehicle sensor according to any one of claims 1 to 7.

Images