Multi-sensor joint calibration method, device, equipment and medium
By obtaining the correspondence between the sensor's extrinsic parameters and image points in the global reference coordinate system, the extrinsic parameters of the sensor are optimized, solving the problem of multi-sensor calibration error in the existing technology and achieving more efficient and accurate sensor parameter calibration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-07
- Publication Date
- 2026-04-10
AI Technical Summary
Existing sensor calibration methods for intelligent driving vehicles only consider the positional relationship between the sensor and the vehicle, failing to effectively consider the positional relationship between multiple sensors, resulting in errors in the calibrated sensor parameters.
By acquiring the extrinsic parameters of the sensors on the vehicle and the global reference coordinate system, the third extrinsic parameter between the sensors is determined, and the extrinsic parameters of the sensors are optimized based on the correspondence of image points. The error parameters are then used for optimization and calibration.
It improves the accuracy of sensor parameters, enhances the consideration of positional relationships between multiple sensors, and improves calibration efficiency and accuracy.
Smart Images
Figure CN121829593A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of sensor calibration, in particular to a joint calibration method and device for multiple sensors, equipment and medium. BACKGROUND
[0002] Intelligent driving vehicles are usually equipped with multiple radars and multiple cameras. Multiple sensors such as radars and cameras can be installed at different positions of the vehicle and have different shooting fields of view, which can shoot the environment around the vehicle in all directions and help the vehicle complete intelligent driving functions. Accurate sensor calibration is the prerequisite for intelligent driving perception and positioning. After installing the sensors, the sensors need to be calibrated to ensure the accuracy of the sensor measurement data.
[0003] Current sensor calibration of intelligent driving vehicles is mostly independent calibration. For example, when calibrating the external parameters between a group of radars and cameras, the external parameters of the radars and the external parameters of the cameras can be calibrated first, and then the external parameters between the radars and the cameras can be obtained through conversion of the calibrated external parameters. However, this calibration method only considers the positional relationship between the sensors and the vehicle, and does not consider the positional relationship between multiple sensors, resulting in errors in the calibrated sensor parameters. SUMMARY
[0004] Therefore, the present application aims to provide a joint calibration method, device, equipment and medium for multiple sensors to improve the accuracy of calibrated sensor parameters.
[0005] In a first aspect, the present application provides a joint calibration method for multiple sensors, comprising: obtaining a first external parameter of a first sensor on a vehicle and a global reference coordinate system and a second external parameter of a second sensor and the global reference coordinate system, the first sensor and the second sensor having a coincident field of view range; determining a third external parameter between the first sensor and the second sensor based on the first external parameter and the second external parameter; determining a first image point photographed by the first sensor and a second image point photographed by the second sensor, the first image point and the second image point corresponding to the same position of the coincident field of view range; converting the first image point to the coordinate system of the second sensor based on the third external parameter to obtain a first projection point; determining an error parameter based on the coordinates of the first projection point and the coordinates of the second image point; optimizing the first external parameter and the second external parameter based on the error parameter to obtain an optimized first external parameter and an optimized second external parameter.
[0006] In a possible implementation, the determining the first image point photographed by the first sensor and the second image point photographed by the second sensor comprises: placing a calibration board in the overlapping field of view range; determining a feature point on the calibration board; determining the first image point photographed by the first sensor on the feature point and the second image point photographed by the second sensor on the feature point.
[0007] In a possible implementation, the converting the first image point to the coordinate system of the second sensor based on the third extrinsic parameter to obtain a first projection point comprises: when the first sensor is a radar and the second sensor is a camera, decomposing the third extrinsic parameter into a first rotation matrix and a first translation matrix; calculating a first product of the first rotation matrix and the coordinates of the first image point; calculating a sum of the first product and the first translation matrix to obtain a first camera coordinate point; converting the first camera coordinate point based on the intrinsic parameter of the second sensor to obtain the first projection point; or when the first sensor and the second sensor are both radars, decomposing the third extrinsic parameter into a second rotation matrix and a second translation matrix; calculating a second product of the second rotation matrix and the coordinates of the first image point; obtaining the first projection point based on a sum of the second product and the second translation matrix; or when the first sensor and the second sensor are both cameras, decomposing the third extrinsic parameter into a third rotation matrix and a third translation matrix; converting the first image point based on the intrinsic parameter of the first sensor to obtain a second camera coordinate point; calculating a third product of the third rotation matrix and the coordinates of the second camera coordinate point; calculating a sum of the third product and the third translation matrix to obtain a third camera coordinate point; converting the third camera coordinate point based on the intrinsic parameter of the second sensor to obtain the first projection point.
[0008] In a possible implementation, the determining the third extrinsic parameter between the first sensor and the second sensor based on the first extrinsic parameter and the second extrinsic parameter comprises: determining an inverse matrix of the second extrinsic parameter; calculating a product of the first extrinsic parameter and the inverse matrix to obtain the third extrinsic parameter.
[0009] In a possible implementation, when the vehicle comprises a plurality of groups of the first sensor and the second sensor, the optimizing the first extrinsic parameter and the second extrinsic parameter based on the error parameter comprises: obtaining an error parameter corresponding to each group of the first sensor and the second sensor; determining a target error parameter based on a sum of the plurality of error parameters; optimizing the first extrinsic parameter and the second extrinsic parameter based on the target error parameter, to obtain the optimized first extrinsic parameter and the optimized second extrinsic parameter.
[0010] In a possible implementation, the optimizing the first extrinsic parameter and the second extrinsic parameter based on the error parameter comprises: iteratively adjusting the first extrinsic parameter and the second extrinsic parameter, and recalculating the error parameter, until the error parameter is smaller than a preset value, to obtain the optimized first extrinsic parameter and the optimized second extrinsic parameter.
[0011] In a possible implementation, the global reference coordinate system comprises a vehicle coordinate system.
[0012] In a second aspect, the present application provides a joint calibration device of multiple sensors, the device comprising: an obtaining unit, configured to obtain a first extrinsic parameter of a first sensor on a vehicle and a global reference coordinate system, and a second extrinsic parameter of a second sensor and the global reference coordinate system, the first sensor and the second sensor having a coincident field of view range; an extrinsic parameter conversion unit, configured to determine a third extrinsic parameter between the first sensor and the second sensor based on the first extrinsic parameter and the second extrinsic parameter; a determination unit, configured to determine a first image point photographed by the first sensor and a second image point photographed by the second sensor, the first image point and the second image point corresponding to a same position of the coincident field of view range; an image conversion unit, configured to convert the first image point to a coordinate system of the second sensor based on the third extrinsic parameter, to obtain a first projection point; an error determination unit, configured to determine an error parameter based on a coordinate of the first projection point and a coordinate of the second image point; a calibration unit, configured to optimize the first extrinsic parameter and the second extrinsic parameter based on the error parameter, to obtain an optimized first extrinsic parameter and an optimized second extrinsic parameter.
[0013] In a third aspect, the present application provides an electronic device, the device comprising: a memory and a processor; The memory is configured to store relevant program codes. The processor is configured to invoke the program codes to execute the multi-sensor joint calibration method according to any one of the implementation manners of the first aspect.
[0014] In a fourth aspect, the present application provides a computer readable storage medium configured to store a computer program, the computer program being configured to execute the multi-sensor joint calibration method according to any one of the implementation manners of the first aspect.
[0015] In a fifth aspect, the present application provides a computer program product, the computer program product comprising computer programs / instructions, which, when executed by a processor, implement the multi-sensor joint calibration method according to any one of the implementation manners of the first aspect.
[0016] In the above implementation of the present application, first, the first external parameter of the first sensor on the vehicle and the global reference coordinate system, and the second external parameter of the second sensor and the global reference coordinate system are obtained, and the first sensor and the second sensor have a coincident field of view range, that is, the shooting range of the first sensor and the shooting range of the second sensor have an overlapping area. Based on the first external parameter and the second external parameter, the third external parameter between the first sensor and the second sensor is determined. In the images shot by the first sensor and the second sensor, the first image point shot by the first sensor and the second image point shot by the second sensor are determined, wherein the first image point and the second image point correspond to the same position of the coincident field of view range. That is, a position point of the coincident field of view range is found to correspond to the image points shot by the first sensor and the second sensor respectively. Since the third external parameter between the first sensor and the second sensor has been determined, the first image point can be converted to the coordinate system of the second sensor based on the third external parameter, thereby obtaining the first projection point. Since the first image point and the second image point correspond to the same position, the first projection point obtained by conversion should theoretically coincide with the second image point. If they do not coincide, it indicates that there is an error in the third external parameter of the first sensor and the second sensor, and the error parameter can be determined based on the coordinates of the first projection point and the coordinates of the second image point. Since the third external parameter is converted based on the first external parameter and the second external parameter, the first external parameter and the second external parameter can be optimized based on the error parameter to obtain the optimized first external parameter and the optimized second external parameter, and the calibration of the first sensor and the second sensor is completed. Through the method provided in the present application, the external parameter between any two sensors can be converted based on the external parameter of each sensor in the global coordinate system. For multiple sensors with a coincident field of view range, the corresponding image points are determined using the external parameters between the multiple sensors, and the external parameters of the sensors are optimized and calibrated based on the position error of the corresponding image points, the position relationship between the multiple sensors with a coincident field of view range is considered, and thus the accuracy of calibrating the sensor parameters is improved. BRIEF DESCRIPTION OF DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments provided in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.
[0018] Figure 1 A flowchart of a joint calibration method of multiple sensors provided in an embodiment of the present application.
[0019] Figure 2 A schematic diagram of a joint calibration device of multiple sensors provided in an embodiment of the present application.
[0020] Figure 3This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are merely exemplary implementations of this application and not all implementation methods. Those skilled in the art can obtain other embodiments in conjunction with the embodiments of this application without creative effort, and these embodiments are also within the protection scope of this application.
[0022] Intelligent driving vehicles are typically equipped with multiple radars and cameras. These sensors can be installed in different locations within the vehicle, each with a different field of view, allowing for comprehensive imaging of the vehicle's surroundings and assisting in intelligent driving functions. Accurate sensor calibration is a prerequisite for intelligent driving perception and positioning. After installation, sensors need to be calibrated to ensure the accuracy of the measurement data.
[0023] Currently, sensor calibration for intelligent driving vehicles is mostly done independently. For example, when calibrating the extrinsic parameters between radar A and camera B, the extrinsic parameters of radar A and camera B in the vehicle coordinate system are first calibrated, and then the extrinsic parameters between radar A and camera B are obtained based on the two extrinsic parameters. This calibration method only considers the positional relationship between radar A and camera B individually and the vehicle, without considering the positional relationship between radar A and camera B, which leads to errors in the calibrated sensor parameters.
[0024] Based on this, embodiments of this application provide a joint calibration method for multiple sensors to improve the accuracy of calibrating sensor parameters. Specifically, firstly, the first extrinsic parameters of the first sensor and the global reference coordinate system, and the second extrinsic parameters of the second sensor and the global reference coordinate system are obtained. The first and second sensors have overlapping fields of view, meaning their shooting ranges overlap. Based on the first and second extrinsic parameters, a third extrinsic parameter between the first and second sensors is determined. In the images captured by the first and second sensors, a first image point captured by the first sensor and a second image point captured by the second sensor are determined, where the first image point and the second image point correspond to the same position within the overlapping field of view. That is, a position point within the overlapping field of view is found that corresponds to the image points captured by the first and second sensors, respectively. Since the third extrinsic parameter between the first and second sensors has been determined, the first image point can be transformed to the coordinate system of the second sensor based on the third extrinsic parameter, thereby obtaining a first projection point. Since the first image point and the second image point correspond to the same position, the transformed first projection point should theoretically coincide with the second image point. If they do not coincide, it indicates an error in the third extrinsic parameter of the first and second sensors. Therefore, the error parameter can be determined based on the coordinates of the first projection point and the second image point. Since the third extrinsic parameter is obtained by transforming the first and second extrinsic parameters, the first and second extrinsic parameters can be optimized based on the error parameter to obtain optimized first and second extrinsic parameters, thus completing the calibration of the first and second sensors. Using the method provided in this application, the extrinsic parameters between any two sensors can be transformed based on the extrinsic parameters of each sensor in the global coordinate system. For multiple sensors with overlapping fields of view, the corresponding image points are determined using the extrinsic parameters between the multiple sensors. The extrinsic parameters of the sensors are optimized and calibrated based on the positional error of the corresponding image points. This method considers the positional relationship between multiple sensors with overlapping fields of view, thereby improving the accuracy of sensor parameter calibration.
[0025] To facilitate understanding of the technical solutions provided in the embodiments of this application, a detailed description will be given below in conjunction with the accompanying drawings.
[0026] See Figure 1 The diagram shown is a flowchart of a multi-sensor joint calibration method provided in an embodiment of this application.
[0027] Optionally, this method can be executed by a data processing device, which can be a terminal or a server. This data processing device can acquire data collected by sensors on the vehicle and process the data.
[0028] The method may include the following steps: S101: Obtain the first extrinsic parameters of the first sensor on the vehicle and the global reference coordinate system, and the second extrinsic parameters of the second sensor and the global reference coordinate system.
[0029] The first and second sensors have overlapping fields of view. Each sensor mounted on the vehicle has its own corresponding shooting angle range, and the shooting angle ranges of the first and second sensors overlap, forming their overlapping field of view. For example, if the longitudinal axis of the vehicle is represented as 0°, the shooting angle range of the first sensor is represented as [-60°, 60°], and the shooting angle range of the second sensor is represented as [0°, 120°]. Therefore, the overlapping field of view of the first and second sensors can be represented as [0°, 60°].
[0030] To account for the spatial constraints between the first and second sensors, both sensors can be placed in a global reference coordinate system. A first extrinsic parameter is obtained between the first sensor and the global reference coordinate system to describe the position and orientation of the first sensor relative to the global reference coordinate system. Similarly, a second extrinsic parameter is obtained between the second sensor and the global reference coordinate system to describe the position and orientation of the second sensor relative to the global reference coordinate system. The first and second extrinsic parameters can be pre-set parameters based on human experience or historical data. In this embodiment, the global reference coordinate system can be a vehicle coordinate system, where the direction of the vehicle's forward movement is represented by the x-axis, and the direction perpendicular to the x-axis and pointing to the left is represented by the y-axis.
[0031] S102: Based on the first and second extrinsic parameters, determine the third extrinsic parameter between the first sensor and the second sensor.
[0032] Based on the first and second extrinsic parameters, the pose relationships of the first and second sensors with respect to the global reference coordinate system can be determined. Therefore, the pose relationship between the first and second sensors can be derived using the first and second extrinsic parameters, i.e., the third extrinsic parameter between the first and second sensors. In other words, the third extrinsic parameter represents the position and orientation of the first sensor relative to the second sensor. Specifically, the inverse matrix of the second extrinsic parameter can be determined first, and then the product of the first extrinsic parameter and the inverse matrix can be calculated to obtain the third extrinsic parameter. Similarly, when it is necessary to determine the extrinsic parameter between the second and first sensors, the inverse matrix of the first extrinsic parameter can be calculated first, and then the product of the second extrinsic parameter and the inverse matrix of the first extrinsic parameter can be calculated.
[0033] S103: Determine the first image point captured by the first sensor and the second image point captured by the second sensor.
[0034] In this context, the first image point and the second image point correspond to the same location within the overlapping field of view. That is, corresponding image points can be found in the images captured by the first sensor and the second sensor. When the first sensor is radar, the first image point captured by the first sensor represents a point cloud. Specifically, this can be determined by pre-placing a calibration board within the overlapping field of view of the first and second sensors and identifying feature points on the calibration board. For example, when the calibration board includes multiple circular holes, the center of each hole can be used as a feature point. For any feature point, a first image point captured by the first sensor and a second image point captured by the second sensor can be identified, thus obtaining a set of corresponding image points. When multiple feature points can be identified, multiple sets of corresponding first and second image points can be obtained.
[0035] S104: Based on the third extrinsic parameter, the first image point is transformed into the coordinate system of the second sensor to obtain the first projection point.
[0036] After obtaining the third extrinsic parameter, the transformation relationship between the coordinate systems of the first and second sensors is essentially determined. Therefore, the third extrinsic parameter can be used to transform the first image point captured by the first sensor into the coordinate system of the second sensor, obtaining the corresponding first projection point. Since the first image point represents a point in the image coordinate system, when the sensor is a camera, the transformation between the camera coordinate system and the image coordinate system also involves the transformation of camera intrinsic parameters. Therefore, the transformation process for obtaining the first projection point will be described below, depending on the specific type of sensor.
[0037] In one possible implementation, when the first sensor is a radar and the second sensor is a camera, the third extrinsic parameter represents the transformation relationship between the radar coordinate system and the camera coordinate system. First, the third extrinsic parameter is decomposed into a first rotation matrix and a first translation matrix. The first rotation matrix indicates how to rotate the radar coordinate system to align with the camera coordinate system. The first translation matrix represents the translation relationship between the origin of the radar coordinate system and the origin of the camera coordinate system. Therefore, the first product of the first rotation matrix and the coordinates of the first image point can be calculated, and the sum of the first product and the first translation matrix can be calculated to obtain the first camera coordinate point corresponding to the first image point in the camera coordinate system. That is, the third extrinsic parameter is used to transform the first image point captured by the first sensor (radar) to the coordinate system of the second sensor (camera) to obtain the corresponding first camera coordinate point. Since the camera has intrinsic and extrinsic parameters, the intrinsic parameters are used to transform the coordinate points in the camera coordinate system to the image coordinate system. Therefore, to obtain the coordinate point corresponding to the first image point in the image coordinate system, the intrinsic parameters of the second sensor need to be used to transform the first camera coordinate point to obtain the first projection point corresponding to the image coordinate system. That is, the first projection point is obtained by multiplying the intrinsic parameters of the second sensor with the coordinates of the first camera coordinate point. The camera's intrinsic parameters are assumed to be known values.
[0038] In one possible implementation, when both the first and second sensors are radars, the third extrinsic parameter represents the transformation relationship between the radar coordinate systems. First, the third extrinsic parameter is decomposed into a second rotation matrix and a second translation matrix. Then, the second product of the second rotation matrix and the coordinates of the first image point is calculated. Based on the sum of the second product and the second translation matrix, the first projection point is obtained.
[0039] In one possible implementation, when both the first and second sensors are cameras, the third extrinsic parameter represents the transformation relationship between camera coordinate systems. This third extrinsic parameter can be decomposed into a third rotation matrix and a third translation matrix. Since the third extrinsic parameter represents the transformation relationship between camera coordinate systems, and both the first and second image points are points in the image coordinate system, the first image point can be transformed using the intrinsic parameters of the first sensor to obtain the second camera coordinate point in the first sensor coordinate system (camera coordinate system). Specifically, the product of the inverse matrix of the first sensor's intrinsic parameters and the coordinates of the first image point is calculated to obtain the second camera coordinate point. The third product of the third rotation matrix and the coordinates of the second camera coordinate point is then calculated, and the sum of this third product and the third translation matrix is calculated to obtain the third camera coordinate point corresponding to the second sensor coordinate system (camera coordinate system). Finally, the third camera coordinate point is transformed using the intrinsic parameters of the second sensor to obtain the first projection point corresponding to the image coordinate system.
[0040] In one possible implementation, when the first sensor is a camera and the second sensor is radar, the third extrinsic parameter represents the transformation relationship between the camera coordinate system and the radar coordinate system. This third extrinsic parameter can be decomposed into a fourth rotation matrix and a fourth translation matrix. Since the camera has intrinsic parameters representing the transformation relationship between the camera coordinate system and the image coordinate system, and the first image point represents a point in the image coordinate system, the intrinsic parameters of the first sensor can be used to transform the first image point to obtain a fourth camera coordinate point in the camera coordinate system. Then, the fourth product of the fourth rotation matrix and the coordinates of the fourth camera coordinate point is calculated, and the sum of the fourth product and the fourth translation matrix is calculated to obtain the first projection point in the radar coordinate system.
[0041] S105: Determine the error parameters based on the coordinates of the first projection point and the coordinates of the second image point.
[0042] Since the first image point and the second image point correspond to the same location within the overlapping field of view, theoretically, when the third extrinsic parameter between the first and second sensors is accurate, transforming the first image point to the same coordinates as the second image point using the third extrinsic parameter should result in the first image point coinciding with the second image point. When the first projection point and the second image point do not coincide, it indicates an error in the third extrinsic parameter. Therefore, an error parameter can be determined based on the coordinates of the first projection point and the second image point. For example, the square of the Euclidean distance can be calculated as the error parameter based on the coordinates of the first projection point and the second image point.
[0043] Specifically, when the first sensor is radar Li and the second sensor is camera Ci, the coordinates of the first image point are represented as follows: The coordinates of the second image point are represented as The third extrinsic parameter can be decomposed into the first rotation matrix. and the first translation matrix The intrinsic parameters of camera Ci are represented as Then the first projection point It can be represented as Based on the first projection point With the second image point The determined error parameter can be expressed as .
[0044] When the first sensor is radar Li and the second sensor is radar Lj, the coordinates of the first image point are represented as follows: The coordinates of the second image point are represented as The third extrinsic parameter can be decomposed into the second rotation matrix. Second translation matrix Then the first projection point It can be represented as The error parameter can be expressed as .
[0045] When the first sensor is camera Ci and the second sensor is camera Cj, the coordinates of the first image point are represented as follows: The coordinates of the second image point are represented as The third extrinsic parameter can be decomposed into a third rotation matrix. and the third translation matrix The intrinsic parameters of camera Ci are represented as The intrinsic parameters of camera Cj are represented as Then the first projection point It can be represented as Based on the first projection point With the second image point The determined error parameter can be expressed as .
[0046] S106: Optimize the first and second extrinsic parameters based on the error parameters to obtain the optimized first and second extrinsic parameters.
[0047] This error parameter represents the error of the extrinsic parameters of the first and second sensors. Since the third extrinsic parameter is calculated based on the first and second extrinsic parameters, the first extrinsic parameter of the first sensor and the second extrinsic parameter of the second sensor can be optimized based on the error parameter until the error parameter meets the condition, thus obtaining the optimized first and second extrinsic parameters. Specifically, the first and second extrinsic parameters can be adjusted, and the error parameter can be recalculated based on the adjusted first and second extrinsic parameters. The above process can be repeated iteratively until the error parameter is less than a preset value, at which point the iteration stops, thereby obtaining the optimized first and second extrinsic parameters.
[0048] As can be seen from the above embodiments, there may be multiple sets of corresponding first image points and second image points between the first sensor and the second sensor. Therefore, based on each set of first image points and second image points, an error parameter can be determined, and the sum of multiple sets of error parameters can be calculated as the error parameter between the first sensor and the second sensor.
[0049] Among the multiple sensors installed on a vehicle, there may be multiple sets of first and second sensors with overlapping fields of view. Therefore, after obtaining the error parameters corresponding to each set of first and second sensors, the sum of multiple error parameters corresponding to multiple sets of first and second sensors can be calculated to determine the target error parameter. Then, based on the target error parameter, the first and second extrinsic parameters are optimized to obtain the optimized first and second extrinsic parameters.
[0050] Based on the method provided in the above embodiments, the extrinsic parameters between any two sensors can be obtained from the extrinsic parameters of each sensor in the global reference coordinate system. By utilizing the corresponding image points of multiple sensors with overlapping field of view, the extrinsic parameters of the multiple sensors are optimized and calibrated, thereby improving the accuracy of sensor parameter calibration. Furthermore, the embodiments of this application can also simultaneously optimize and calibrate the extrinsic parameters of multiple sets of multiple sensors with overlapping field of view, improving calibration efficiency.
[0051] Based on the above method embodiments, this application also provides a joint calibration device for multiple sensors. See also Figure 2 The diagram shown is a schematic of a multi-sensor joint calibration device provided in an embodiment of this application.
[0052] The device 200 includes: Acquisition unit 201 is used to acquire the first extrinsic parameters of the first sensor on the vehicle and the global reference coordinate system and the second extrinsic parameters of the second sensor and the global reference coordinate system, wherein the first sensor and the second sensor have overlapping field of view. The extrinsic parameter conversion unit 202 is used to determine a third extrinsic parameter between the first sensor and the second sensor based on the first extrinsic parameter and the second extrinsic parameter; The determining unit 203 is used to determine a first image point captured by the first sensor and a second image point captured by the second sensor, wherein the first image point and the second image point correspond to the same position in the overlapping field of view; Image conversion unit 204 is used to convert the first image point to the coordinate system of the second sensor based on the third extrinsic parameter to obtain the first projection point; The error determination unit 205 is used to determine error parameters based on the coordinates of the first projection point and the coordinates of the second image point; The calibration unit 206 is used to optimize the first extrinsic parameter and the second extrinsic parameter based on the error parameter to obtain the optimized first extrinsic parameter and the optimized second extrinsic parameter.
[0053] In one possible implementation, the determining unit 203 is configured to place a calibration plate within the overlapping field of view; determine feature points on the calibration plate; and determine the first image point obtained by the first sensor capturing the feature points and the second image point obtained by the second sensor capturing the feature points.
[0054] In one possible implementation, the image conversion unit 204 is configured to, when the first sensor is a radar and the second sensor is a camera, decompose the third extrinsic parameter into a first rotation matrix and a first translation matrix; calculate a first product of the first rotation matrix and the coordinates of the first image point; calculate the sum of the first product and the first translation matrix to obtain the first camera coordinate point; and convert the first camera coordinate point based on the intrinsic parameters of the second sensor to obtain the first projection point; or, when both the first sensor and the second sensor are radars, decompose the third extrinsic parameter into a second rotation matrix and a second translation matrix; calculate the second rotation matrix... The second product of the coordinates of the first image point and the second translation matrix is used to obtain the first projection point; or, when both the first sensor and the second sensor are cameras, the third extrinsic parameter is decomposed into a third rotation matrix and a third translation matrix; the first image point is transformed based on the intrinsic parameters of the first sensor to obtain the second camera coordinate point; the third product of the third rotation matrix and the coordinates of the second camera coordinate point is calculated; the third product of the third product and the third translation matrix is calculated to obtain the third camera coordinate point; the third camera coordinate point is transformed based on the intrinsic parameters of the second sensor to obtain the first projection point.
[0055] In one possible implementation, the extrinsic parameter conversion unit 202 is used to determine the inverse matrix of the second extrinsic parameter; calculate the product of the first extrinsic parameter and the inverse matrix to obtain the third extrinsic parameter.
[0056] In one possible implementation, when the vehicle includes multiple sets of the first sensor and the second sensor, the calibration unit 206 is used to acquire error parameters corresponding to each set of the first sensor and the second sensor; determine a target error parameter based on the sum of the multiple error parameters; and optimize the first extrinsic parameter and the second extrinsic parameter based on the target error parameter to obtain the optimized first extrinsic parameter and the optimized second extrinsic parameter.
[0057] In one possible implementation, the calibration unit 206 is used to iteratively adjust the first extrinsic parameter and the second extrinsic parameter, and recalculate the error parameter until the error parameter is less than a preset value, and then stop the iteration to obtain the optimized first extrinsic parameter and the optimized second extrinsic parameter.
[0058] In one possible implementation, the global reference coordinate system includes the vehicle coordinate system.
[0059] Based on the above method and device embodiments, this application also provides an electronic device. The following description will be provided in conjunction with the accompanying drawings.
[0060] SeeFigure 3 , Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application.
[0061] The device 300 includes: a memory 301 and a processor 302; The memory 301 is used to store relevant program code; The processor 302 is used to call the program code to execute the multi-sensor joint calibration method described in the above method embodiments.
[0062] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program for executing the multi-sensor joint calibration method described in the above method embodiments.
[0063] This application also provides a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, it implements the multi-sensor joint calibration method described in the above method embodiments.
[0064] It should be noted that the computer-readable medium described above in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0065] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this application. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0066] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units or modules described as separate components may or may not be physically separate. The components shown as units or modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the units or modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions, and operations that may be implemented by methods, apparatuses, and devices according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0068] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0069] It should also be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0070] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this application can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0071] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for joint calibration of multiple sensors, characterized in that, The method includes: Acquire the first extrinsic parameters of the first sensor on the vehicle and the global reference coordinate system, and the second extrinsic parameters of the second sensor and the global reference coordinate system, wherein the first sensor and the second sensor have overlapping fields of view; Based on the first extrinsic parameter and the second extrinsic parameter, a third extrinsic parameter between the first sensor and the second sensor is determined; Determine a first image point captured by the first sensor and a second image point captured by the second sensor, wherein the first image point and the second image point correspond to the same position within the overlapping field of view; Based on the third extrinsic parameter, the first image point is transformed into the coordinate system of the second sensor to obtain the first projection point; Based on the coordinates of the first projection point and the coordinates of the second image point, the error parameters are determined; Based on the error parameters, the first extrinsic parameter and the second extrinsic parameter are optimized to obtain the optimized first extrinsic parameter and the optimized second extrinsic parameter.
2. The method according to claim 1, characterized in that, Determining the first image point captured by the first sensor and the second image point captured by the second sensor includes: Place a calibration plate within the overlapping field of view; Determine the feature points on the calibration plate; The first image point obtained by the first sensor capturing the feature point and the second image point obtained by the second sensor capturing the feature point are determined.
3. The method according to claim 1, characterized in that, The step of transforming the first image point to the coordinate system of the second sensor based on the third extrinsic parameter to obtain the first projection point includes: When the first sensor is a radar and the second sensor is a camera, the third extrinsic parameter is decomposed into a first rotation matrix and a first translation matrix; Calculate the first product of the first rotation matrix and the coordinates of the first image point; The sum of the first product and the first translation matrix is calculated to obtain the coordinates of the first camera point; The coordinates of the first camera are transformed based on the intrinsic parameters of the second sensor to obtain the first projection point; or, When both the first sensor and the second sensor are radars, the third extrinsic parameter is decomposed into a second rotation matrix and a second translation matrix; Calculate the second product of the second rotation matrix and the coordinates of the first image point; The first projection point is obtained based on the sum of the second product and the second translation matrix; or, When both the first sensor and the second sensor are cameras, the third extrinsic parameter is decomposed into a third rotation matrix and a third translation matrix; The first image points are transformed based on the intrinsic parameters of the first sensor to obtain the coordinate points of the second camera; Calculate the third product of the third rotation matrix and the coordinates of the second camera coordinate point; The sum of the third product and the third translation matrix is calculated to obtain the coordinates of the third camera. The coordinates of the third camera are transformed based on the intrinsic parameters of the second sensor to obtain the first projection point.
4. The method according to claim 1, characterized in that, The step of determining a third extrinsic parameter between the first sensor and the second sensor based on the first extrinsic parameter and the second extrinsic parameter includes: Determine the inverse matrix of the second extrinsic parameter; The third extrinsic parameter is obtained by multiplying the first extrinsic parameter by the inverse matrix.
5. The method according to claim 1, characterized in that, When the vehicle includes multiple sets of the first sensor and the second sensor, the optimization of the first extrinsic parameters and the second extrinsic parameters based on the error parameters to obtain the optimized first extrinsic parameters and the optimized second extrinsic parameters includes: Obtain the error parameters corresponding to each group of the first sensor and the second sensor; The target error parameter is determined based on the sum of the multiple error parameters. The first extrinsic parameter and the second extrinsic parameter are optimized based on the target error parameter to obtain the optimized first extrinsic parameter and the optimized second extrinsic parameter.
6. The method according to claim 1, characterized in that, The step of optimizing the first extrinsic parameter and the second extrinsic parameter based on the error parameter to obtain the optimized first extrinsic parameter and the optimized second extrinsic parameter includes: The first and second extrinsic parameters are iteratively adjusted, and the error parameter is recalculated until the error parameter is less than a preset value. The iteration is then stopped to obtain the optimized first and second extrinsic parameters.
7. The method according to any one of claims 1 to 6, characterized in that, The global reference coordinate system includes the vehicle coordinate system.
8. A multi-sensor joint calibration device, characterized in that, The device includes: The acquisition unit is used to acquire the first extrinsic parameters of the first sensor on the vehicle and the global reference coordinate system and the second extrinsic parameters of the second sensor and the global reference coordinate system, wherein the first sensor and the second sensor have overlapping field of view. An extrinsic parameter conversion unit is used to determine a third extrinsic parameter between the first sensor and the second sensor based on the first extrinsic parameter and the second extrinsic parameter; The determining unit is used to determine a first image point captured by the first sensor and a second image point captured by the second sensor, wherein the first image point and the second image point correspond to the same position in the overlapping field of view; An image conversion unit is used to convert the first image point to the coordinate system of the second sensor based on the third extrinsic parameter to obtain the first projection point; An error determination unit is used to determine error parameters based on the coordinates of the first projection point and the coordinates of the second image point; The calibration unit is used to optimize the first extrinsic parameter and the second extrinsic parameter based on the error parameter, so as to obtain the optimized first extrinsic parameter and the optimized second extrinsic parameter.
9. An electronic device, characterized in that, The device includes: a memory and a processor; The memory is used to store the relevant program code; The processor is used to call the program code to execute the multi-sensor joint calibration method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for executing the multi-sensor joint calibration method according to any one of claims 1 to 7.