Multi-sensor joint calibration method and device, storage medium and computer equipment

Abstract

The invention discloses a multi-sensor joint calibration method and device, a storage medium and computer equipment, relates to the technical field of sensor calibration, and mainly aims to improve the calibration efficiency and calibration precision of multiple sensors. The method comprises the following steps: respectively acquiring calibration data detected by a plurality of sensors aiming at a calibration device; determining an initial internal reference of the sensor; matching feature points in the calibration data corresponding to the sensors, and constructing initial external parameters among the sensors based on coordinate information of matched feature point pairs; and performing joint iterative optimization on the initial external reference and the initial internal reference to obtain a joint calibration result of each sensor. The method is suitable for a multi-sensor calibration application scene.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor calibration, and in particular, to a multi-sensor joint calibration method, device, storage medium, and computer device. Background Art

[0002] Multi-sensor information fusion is a technology that automatically analyzes and synthesizes information from different sensors. In autonomous vehicles, this information may come from various sensors such as lidar, millimeter-wave radar, and cameras. These sensors each provide different information about the vehicle's surrounding environment, and information fusion integrates this information to provide a comprehensive and accurate environmental perception for the autonomous driving system. Based on this, in order to improve the safety of autonomous driving, it is necessary to pre-calibrate multi-sensors.

[0003] Currently, it is usually necessary to manually and continuously try to adjust the parameters of each sensor to find the calibration parameters with the best detection effect. However, this manual and continuous trial-and-error method is time-consuming, laborious, and operationally redundant, resulting in low calibration efficiency of multi-sensors. At the same time, the manual trial-and-error method also leads to low calibration accuracy of multi-sensors. Summary of the Invention

[0004] The present invention provides a multi-sensor joint calibration method, device, storage medium, and computer device, mainly capable of improving the calibration efficiency and calibration accuracy of multi-sensors.

[0005] According to a first aspect of the present invention, a multi-sensor joint calibration method is provided, including:

[0006] Obtain calibration data detected by a plurality of sensors for a calibration device respectively;

[0007] Determine the initial internal parameters of the sensors;

[0008] Match feature points in the calibration data corresponding to each sensor, and construct initial external parameters between each sensor based on the coordinate information of the matched feature point pairs;

[0009] Jointly iteratively optimize the initial external parameters and initial internal parameters to obtain the joint calibration results of each sensor.

[0010] Optionally, the multi-sensors at least include a first sensor, a second sensor, and a third sensor;

[0011] The constructing of the initial external parameters between each sensor based on the coordinate information of the matched feature point pairs includes:

[0012] Construct a first initial extrinsic parameter between the first sensor and the second sensor based on the coordinate information of the second feature point pairs matched in the calibration data corresponding to the first sensor and the calibration data corresponding to the second sensor.

[0013] Construct a second initial extrinsic parameter between the first sensor and the third sensor based on the coordinate information of the third feature point pairs matched in the calibration data corresponding to the first sensor and the calibration data corresponding to the third sensor.

[0014] Construct a third initial extrinsic parameter between the second sensor and the third sensor based on the coordinate information of the fourth feature point pairs matched in the calibration data corresponding to the second sensor and the calibration data corresponding to the third sensor.

[0015] Optionally, the method includes:

[0016] The first initial extrinsic parameter includes the rotation matrix and the translation vector of the first sensor relative to the second sensor.

[0017] And / or, the second initial extrinsic parameter includes the rotation matrix and the translation vector of the first sensor relative to the third sensor;

[0018] And / or, the third initial extrinsic parameter includes the rotation matrix and the translation vector of the second sensor relative to the third sensor.

[0019] Optionally, the method includes: constructing a loss function of the initial intrinsic parameter and the initial extrinsic parameter according to the detection data, the initial intrinsic parameter, and the initial extrinsic parameter corresponding to each sensor;

[0020] Perform weighted summation on the loss function of the initial intrinsic parameter and the initial extrinsic parameter to obtain a joint constraint equation, and based on the joint constraint equation, jointly iteratively optimize the initial extrinsic parameter and the initial intrinsic parameter of each sensor to obtain the calibration parameters of each sensor.

[0021] Optionally, the method includes: the weight coefficient of the loss function of the initial intrinsic parameter is greater than the weight coefficient corresponding to the loss function of the initial extrinsic parameter.

[0022] According to the second aspect of the present invention, there is provided a multi-sensor joint calibration device, including:

[0023] An acquisition unit, configured to respectively acquire calibration data detected by a plurality of sensors for a calibration device;

[0024] A determination unit, configured to determine the initial intrinsic parameter of each sensor;

[0025] A building unit is configured to match feature points in the calibration data corresponding to each of the sensors, and construct initial extrinsic parameters of each of the sensors based on the coordinate information of the matched feature point pairs.

[0026] A joint optimization unit is configured to perform joint iterative optimization on the initial extrinsic parameters and initial intrinsic parameters to obtain the joint calibration results of each of the sensors.

[0027] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the above multi-sensor joint calibration method is implemented.

[0028] According to a fourth aspect of the present invention, there is provided a computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the above multi-sensor joint calibration method is implemented.

[0029] According to a multi-sensor joint calibration method, device, storage medium, and computer device provided by the present invention, compared with the current method of manually and continuously probing and adjusting the parameters of each sensor to find the calibration parameters with the best detection effect, the present invention respectively obtains calibration data of multiple sensors for detecting a calibration board; determines the initial intrinsic parameters of each of the sensors; then matches feature points in the calibration data corresponding to each of the sensors, and constructs initial extrinsic parameters of each of the sensors based on the coordinate information of the matched feature point pairs; and finally performs joint iterative optimization on the initial extrinsic parameters and initial intrinsic parameters to obtain the joint calibration results of each of the sensors. Thus, the sensor parameters can be accurately solved by constructing constraint equations, and based on the physical relationships and geometric constraints between the sensors, the optimal solution can be quickly found, avoiding the blindness and inefficiency of manual probing. At the same time, the method of constructing constraint equations can implement an automated calibration process through programming, reducing the need for manual intervention and manual adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present invention and form a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0031] Figure 1 Shows a flowchart of a multi-sensor joint calibration method provided by an embodiment of the present invention;

[0032] Figure 2 Shows a sample diagram of a calibration board provided by an embodiment of the present invention;

[0033] Figure 3Shows a flowchart of another multi-sensor joint calibration method provided by an embodiment of the present invention;

[0034] Figure 4 Shows a schematic structural diagram of a multi-sensor joint calibration device provided by an embodiment of the present invention;

[0035] Figure 5 Shows a schematic physical structure diagram of a computer device provided by an embodiment of the present invention. Detailed implementation manners

[0036] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0037] Currently, the method of manually and continuously probing and adjusting the parameters of each sensor to find the calibration parameters with the best detection effect is time-consuming and laborious, and the operation is redundant. At the same time, the method of manual probing and adjustment will also result in low calibration accuracy of multi-sensors.

[0038] To solve the above problems, an embodiment of the present invention provides a multi-sensor joint calibration method, as Figure 1 shown, the method includes:

[0039] 101. Obtain calibration data detected by a plurality of sensors for the calibration device respectively.

[0040] The present invention is applicable to multi-sensor calibration in scenarios such as the field end and the vehicle end.

[0041] Among them, the plurality of sensors may include various sensors such as cameras, lidars, and millimeter-wave radars; in the present invention, a Figure 2 shown calibration device can be used, which includes a checkerboard calibration board and four corner reflectors placed at the four corners of the checkerboard. The corner points of the corner reflectors face the direction of the sensors. The calibration device is placed within the common field of view of the multi-sensors.

[0042] For the embodiment of the present invention, each sensor is used to detect the calibration device respectively to obtain calibration data for sensor calibration. For example, if the plurality of sensors include a camera, a lidar, and a millimeter-wave radar, the camera is used to take a picture of the calibration device to obtain a calibration image containing a complete image of the calibration device, the lidar is used to detect the calibration device to obtain lidar point cloud data containing all feature points in the calibration device, and at the same time, the millimeter-wave radar is used to detect the calibration device to obtain millimeter-wave radar point cloud data containing all feature points in the calibration device.

[0043] In the embodiment of the present invention, a calibration device is used to calibrate each sensor. Since the calibration device provides a standardized calibration environment, the calibration process is made more simplified and unified. Using the calibration device, it is easier to control and adjust the calibration conditions, thereby simplifying the calibration process and reducing the calibration time and cost.

[0044] 102. Determine the initial internal parameters of the sensor; perform feature point matching in the calibration data corresponding to each sensor, and construct the initial external parameters between the sensors based on the coordinate information of the matched feature point pairs.

[0045] In the embodiment of the present invention, for a sensor including internal parameters, the internal parameters of the sensor need to be calibrated first. For example, a camera. Based on this, step 102 specifically includes: determining the initial internal parameters based on the physical coordinates of each feature point in the calibration board and the sensor coordinates of the points matched with the feature points in the calibration data.

[0046] Specifically, taking a camera as an example, determine the pixel coordinates corresponding to the feature points on the calibration board in the calibration image; determine the physical coordinates of the points matched with the feature points in the calibration board; and determine the initial internal parameters of the camera based on the pixel coordinates and the physical coordinates.

[0047] For the embodiment of the present invention, the camera performs feature point matching on the calibration image captured by the calibration device and in the calibration device. If it is a checkerboard calibration board, the feature points can be checkerboard corner points, and the feature point pairs matched in the calibration image and the calibration board are obtained. The feature point pairs include the feature points in the calibration image and the feature points in the calibration board. The coordinates of the feature point pairs include the pixel coordinates of the feature points in the calibration image and the physical coordinates of the feature points in the calibration board. According to the corresponding relationship between the pixel coordinates and the physical coordinates of the feature point pairs, the initial internal parameters of the camera are calibrated, that is, the internal parameter matrix K and the distortion parameter D. Specifically, the Zhang Zhengyou calibration method can be used, and the calibration principle of this application will not be elaborated here.

[0048] Secondly, it is necessary to determine the relative initial external parameters between each pair of sensors according to the relative position relationship between the multiple sensors. That is to say, assuming that the multiple sensors are a camera, a lidar, and a millimeter-wave radar respectively, when calibrating the initial external parameters, feature points such as triangular reflector corner points and checkerboard corner points are matched in the calibration image captured by the camera for the calibration device and the radar point cloud data detected by the lidar for the calibration device, and the feature point pairs matched in the calibration image and the radar point cloud data are obtained. The feature point pairs include the feature points in the calibration image and the feature points in the radar point cloud data. The coordinates of the feature point pairs include the pixel coordinates of the feature points in the calibration image and the radar point cloud coordinates of the feature points in the radar coordinate system. According to the corresponding relationship between the pixel coordinates and the radar coordinates of the feature point pairs, the initial camera-radar external parameters between the camera and the lidar are constructed. The initial external parameters include the translation matrix and the rotation matrix of the camera relative to the lidar.

[0049] In addition, feature points such as the corner points of a triangular reflector are matched in the calibration images captured by the camera calibration device and the millimeter-wave radar point cloud data detected by the millimeter-wave radar calibration device, and the feature point pairs matched in the calibration images and the millimeter-wave radar point cloud data are obtained. The feature point pair includes the feature point in the calibration image and the feature point in the millimeter-wave radar point cloud data. The coordinates of the feature point pair include the pixel coordinates of the feature point in the calibration image and the millimeter-wave radar point cloud coordinates of the feature point in the millimeter-wave radar coordinate system. According to the correspondence between the pixel coordinates and the millimeter-wave radar point cloud coordinates of the feature point pair, the initial extrinsic parameters between the camera and the millimeter-wave radar, i.e., the camera-millimeter-wave radar initial extrinsic parameters, are constructed. The initial extrinsic parameters include the translation matrix and the rotation matrix of the camera relative to the millimeter-wave radar.

[0050] In the embodiment of the present invention, the corner points on the triangular reflector are used as feature points for matching. The triangular reflector has a specific geometric shape and reflection characteristics, and can form clear reflection points or reflection regions in the radar data. These reflection points or regions can be used as accurate reference points for calculating the relative position and attitude between sensors, thereby improving the calibration accuracy.

[0051] In addition, feature points such as the corner points of a triangular reflector are matched in the lidar point cloud data detected by the lidar calibration device and the millimeter-wave radar point cloud data detected by the millimeter-wave radar calibration device, and the feature point pairs matched in the lidar point cloud data and the millimeter-wave radar point cloud data are obtained. The feature point pair includes the feature point in the lidar point cloud data and the feature point in the millimeter-wave radar point cloud data. The coordinates of the feature point pair include the lidar point cloud coordinates of the feature point in the lidar point cloud data and the millimeter-wave radar point cloud coordinates of the feature point in the millimeter-wave radar coordinate system. Based on this, the lidar-millimeter-wave radar initial extrinsic parameters between the lidar and the millimeter-wave radar are constructed. The initial extrinsic parameters include the translation matrix and the rotation matrix of the lidar relative to the millimeter-wave radar.

[0052] It should be noted that the calibration methods for the intrinsic and extrinsic parameters of the above sensors can be implemented by existing technologies, and the present invention will not elaborate on them here.

[0053] 103. Jointly iteratively optimize the initial extrinsic parameters and the initial intrinsic parameters to obtain the joint calibration results of each sensor.

[0054] In this step, the initial extrinsic parameters obtained from the rough calibration and the initial extrinsic parameters are jointly used as the starting point for joint calibration for iterative optimization, and the parameters after the optimization ends are used as the final calibration results.

[0055] Taking a multi-sensor including a camera, a lidar, and a millimeter-wave radar as an example, joint iterative optimization is performed according to the initial internal parameters of the camera, the initial external parameters between the camera and the lidar, the initial external parameters between the camera and the millimeter-wave radar, and the initial external parameters between the lidar and the millimeter-wave radar to obtain the final internal parameters of the camera, the external parameters between the camera and the lidar, the external parameters between the camera and the millimeter-wave radar, and the external parameters between the lidar and the millimeter-wave radar.

[0056] In the traditional multi-sensor calibration method, when there are sensors involving internal parameters among the multi-sensors, generally, the internal parameters of the sensors are calibrated first, then the internal parameters are fixed, and then the external parameters between multiple sensors are calibrated. For example, first calibrate the internal parameters of the camera, the external parameters between the camera and the radar, and the external parameters between the radar and the millimeter-wave radar, and then directly obtain the external parameters between the camera and the millimeter-wave radar according to the two calibrated external parameters. The disadvantages of this method are that, firstly, when the internal parameters of the camera are inaccurately calibrated, it will greatly affect the calibration accuracy of all subsequent external parameters. Secondly, the external parameters between the camera and the millimeter-wave radar are obtained through mathematical operations based on the two known external parameters, rather than being actually measured, and there may be certain errors in the middle, which affects the calibration accuracy.

[0057] In the embodiment of the present invention, after determining the initial external parameters and initial internal parameters of the sensors, it is necessary to jointly optimize the initial external parameters and initial internal parameters to obtain the calibration results of each sensor. The initial internal parameters participate in the subsequent iterative optimization process. Therefore, the present invention can first avoid the problem that the calibration effect of the external parameters is poor due to inaccurate calibration of the internal parameters. Secondly, the present invention calibrates each sensor through joint optimization, can quickly find the optimal solution based on the physical relationship and geometric constraints between the sensors, avoids the blindness and inefficiency of manual exploration, and at the same time avoids the problem of poor accuracy in the above traditional method. Finally, the rough calibration process of the present invention finds the starting point of joint optimization for the initial external parameters and initial internal parameters, helps the calibration parameters of each sensor to converge to the optimal solution faster during the optimization process, makes the optimization process more stable, reduces the search space, thereby improving the calibration efficiency of each sensor and reducing the possibility of falling into a local optimum, thereby improving the calibration accuracy of each sensor.

[0058] Further, in order to better illustrate the above process of jointly calibrating multi-sensors, as a refinement and extension of the above embodiment, the embodiment of the present invention provides another multi-sensor joint calibration method, as Figure 3 shown, the method includes:

[0059] 201. Respectively obtain the calibration data detected by multiple sensors for the calibration device.

[0060] Among them, the multiple sensors at least include a first sensor, a second sensor, and a third sensor. Specifically, Figure 2The calibration device shown is placed within the common sensing range of each sensor, and multiple sensors are used to detect the calibration device simultaneously to obtain calibration data, that is, detection images from each sensor, such as camera images and 3D point cloud images.

[0061] 202. Determine the initial internal parameters of each sensor and the initial external parameters between each sensor.

[0062] Specifically, determine the initial internal parameters of the sensor and the relative external parameters between each sensor according to the calibration data. First, for the sensor containing internal parameters, the internal parameters of the sensor need to be calibrated first. For example, a camera.

[0063] Taking the camera as an example, first determine the initial internal parameters of the camera (the initial internal parameters include the internal parameter matrix K and distortion parameters D, etc.) according to the physical coordinates corresponding to the feature points in the calibration board and the pixel coordinates of the points matching the feature points in the calibration image captured by the first sensor (camera).

[0064] Subsequently, based on the coordinate information of the second feature point pairs matched in the calibration data corresponding to the first sensor and the calibration data corresponding to the second sensor, construct the first initial external parameter between the first sensor and the second sensor; based on the coordinate information of the third feature point pairs matched in the calibration data corresponding to the first sensor and the calibration data corresponding to the third sensor, construct the second initial external parameter between the first sensor and the third sensor; based on the coordinate information of the fourth feature point pairs matched in the calibration data corresponding to the second sensor and the calibration data corresponding to the third sensor, construct the third initial external parameter between the second sensor and the third sensor.

[0065] For the specific method, refer to the above embodiments.

[0066] 203. Construct a loss function for the initial internal parameters and the initial external parameters according to the detection data corresponding to each sensor and the initial internal parameters and the initial external parameters.

[0067] After obtaining the initial internal parameters and the initial external parameters, construct the loss function of the above calibration parameters. At this time, the detection data of the sensor is required, and this detection data can use the calibration data (but different feature points from those used in the initial internal and external parameter calibration process need to be selected), or the sensor can be used to re-collect the calibration device.

[0068] Taking the first sensor as a camera as an example in the embodiment of the present invention, the detection data includes the calibration device images collected by the camera. Determine the pixel coordinates in the sensor coordinate system of the points matching each feature point in the detection data of the first sensor, and combine the physical coordinates of the above multiple feature points to construct the initial internal parameter loss function Loss0 of the camera as follows:

[0069]

[0070] Among them, n represents the number of feature points, and P c (i) represents the pixel coordinates corresponding to the i-th feature point, and P (i) (K, D) represents the coordinates calculated jointly by the initial internal parameters K and initial distortion coefficients D of the camera, and the physical coordinates of the i-th feature point.

[0071] Next, if the second and third sensors include internal parameters, the initial internal parameters of the second and third sensors also need to be solved in the above manner, and the loss function of the initial internal parameters is further determined in the same way. For the sake of simplicity, it is assumed in this embodiment that the second and third sensors do not involve internal parameters. Then, only the first initial external parameter and loss function of the first sensor relative to the second sensor, the second initial external parameter and loss function of the first sensor relative to the third sensor, and the third initial external parameter and loss function of the second sensor relative to the third sensor need to be solved.

[0072] Specifically, the loss function Loss1 of the first initial external parameter is as follows:

[0073]

[0074] Among them, n represents the number of feature points, K represents the initial internal parameter of the first sensor, R1 represents the rotation matrix of the first sensor relative to the second sensor, t1 represents the translation vector of the first sensor relative to the second sensor, and R1 and t1 constitute the first initial external parameter. is the coordinate of the i-th feature point in the coordinate system of the second sensor (such as the 3D point coordinate of the lidar point cloud corresponding to the i-th feature point). represents the coordinate obtained by converting the i-th feature point in the coordinate system of the second sensor to the coordinate system of the first sensor according to the first initial external parameter, and P c (i) represents the coordinate of the i-th feature point in the coordinate of the first sensor.

[0075] Furthermore, the loss function Loss2 of the constructed second initial external parameter is as follows:

[0076]

[0077] Among them, m represents the number of feature points, K represents the initial internal parameter of the first sensor, R2 represents the rotation matrix of the first sensor relative to the third sensor, t2 represents the translation vector of the first sensor relative to the third sensor, and R2 and t2 constitute the second initial external parameter. is the coordinate of the i-th feature point in the coordinate system of the third sensor (such as the 3D point coordinate of the millimeter-wave radar corresponding to the i-th feature point). is the coordinate obtained by transforming the i-th feature point in the third sensor coordinate system to the first sensor coordinate system according to the second initial extrinsic parameters, P c (i) represents the coordinate of the i-th feature point in the first sensor coordinate system.

[0078] Specifically, the loss function Loss3 of the constructed third initial extrinsic parameters is as follows:

[0079]

[0080] where u represents the number of feature points, R3 represents the rotation matrix of the second sensor relative to the third sensor, t3 represents the translation vector of the second sensor relative to the third sensor, R3 and t3 constitute the third initial extrinsic parameters, P L (i) is the coordinate of the i-th feature point in the coordinate system corresponding to the second sensor. (R3·P L (i) +t3) is the coordinate obtained by transforming the i-th feature point in the second sensor coordinate system to the third sensor coordinate system according to the third initial extrinsic parameters.

[0081] It should be noted that the above is only an exemplary embodiment. In practical applications, for each sensor, if it includes intrinsic parameters, the initial intrinsic parameters need to be calibrated first, and then the initial extrinsic parameters are calibrated between every two sensors of the multi-sensor system. Then, for all the initial intrinsic parameters and initial extrinsic parameters, the loss functions need to be calculated respectively.

[0082] In addition, it should be noted that when there are sensors with different dimensional imaging capabilities in the multi-sensor system, such as a camera (two-dimensional) and a radar (three-dimensional), when calculating the loss function, the three-dimensional information needs to be projected onto the two-dimensional coordinate system, that is, the high-dimensional feature points are projected onto the low-dimensional coordinate system. For example, in the above embodiment, when the first sensor is a camera, and the second and third sensors are lidar and millimeter-wave radar respectively, the 3D feature points detected by the lidar and millimeter-wave radar need to be projected onto the camera coordinate system (two-dimensional coordinate system), rather than the other way around. Because the two-dimensional feature points lack one-dimensional information and cannot be directly projected into three dimensions. And although projecting the three-dimensional feature points onto the two-dimensional coordinate system will result in the loss of one-dimensional information, since the second and third sensors both detect 3D feature points, the setting of the loss function of the third initial extrinsic parameters just makes up for the negative impact brought by this part of information loss.

[0083] 204. Perform a weighted summation of the loss functions of the initial internal parameters and the initial external parameters to obtain a joint constraint equation. Based on the joint constraint equation, jointly iteratively optimize the initial external parameters and the initial internal parameters of each sensor to obtain the calibration parameters of each sensor.

[0084] In this step, perform a weighted summation of the loss functions of all the initial internal parameters and the loss functions of the initial external parameters. We still take the above embodiment as an example. In the above embodiment, assume that the first sensor includes initial internal parameters, and the second and third sensors do not. Determine the joint constraint equation Loss according to the following formula:

[0085] Loss = σLoss0 + αLoss1 + βLoss2 + γLoss3

[0086] Where, σ represents the weight coefficient of the loss function Loss0 of the initial internal parameters corresponding to the first sensor, α represents the weight coefficient of the loss function Loss1 of the first initial external parameter between the first sensor and the second sensor, β represents the weight coefficient of the loss function Loss2 of the second initial external parameter between the first sensor and the third sensor, and γ represents the weight coefficient of the loss function Loss3 of the third initial external parameter between the second sensor and the third sensor.

[0087] Each weight coefficient can be set according to actual needs. However, since the initial internal parameters are only related to the sensor itself and are not affected by the relative position relationship of multiple sensors, the weight coefficient settings for the loss functions of all initial internal parameters should be greater than those of the loss functions of the initial external parameters. For example, when the first sensor is a camera, since the camera internal parameter K and the distortion parameter D mainly depend on the characteristics of the camera itself and have no correlation with other sensors, in order to limit large changes in the internal parameter K and the distortion parameter D in the optimization function, the weight coefficient of the loss function Loss0 of the initial internal parameters needs to be set one to two orders of magnitude higher than the other three weight coefficients.

[0088] After determining the joint constraint equation, continuously iteratively optimize the initial internal parameters and the initial external parameters of each sensor using the joint constraint equation until the number of iterations reaches the pre-set number requirement, or the convergence of the iteration meets the requirements (such as the difference between the losses obtained from two adjacent iterative optimizations is less than the preset threshold, and the value of the joint loss function is less than the preset value), etc. Under such conditions, stop the iteration, and determine the finally iteratively optimized internal parameters and external parameters as the calibration internal parameters and calibration external parameters of each sensor.

[0089] The joint calibration method proposed by the present invention, after determining the initial extrinsic parameters and initial intrinsic parameters of the sensors, needs to jointly optimize the initial extrinsic parameters and initial intrinsic parameters to obtain the calibration results of each sensor. The initial intrinsic parameters participate in the subsequent iterative optimization process. Therefore, the present invention can first avoid the problem of poor extrinsic parameter calibration caused by inaccurate intrinsic parameter calibration. Secondly, the present invention constructs a loss function and restricts the calibration process of each sensor through constraint equations. It can quickly find the optimal solution based on the physical relationship and geometric constraints between sensors, avoiding the blindness and inefficiency of manual exploration, and at the same time avoiding the problem of poor accuracy of traditional multi-sensor calibration methods. Finally, the rough calibration process of the present invention finds a starting point for joint optimization of the initial extrinsic parameters and initial intrinsic parameters, which helps the calibration parameters of each sensor converge to the optimal solution faster during the optimization process, making the optimization process more stable, reducing the search space, thereby improving the calibration efficiency of each sensor and reducing the possibility of falling into local optima, and thus improving the calibration accuracy of each sensor.

[0090] Furthermore, as Figure 1 a specific implementation of Figure 4 this, the embodiment of the present invention provides a multi-sensor joint calibration device, as

[0091] shown, the device includes: an acquisition unit 31, a determination unit 32, a construction unit 33, and a joint optimization unit 34.

[0092] The acquisition unit 31 can be used to respectively acquire calibration data detected by a plurality of sensors for the calibration device.

[0093] The determination unit 32 can be used to determine the initial intrinsic parameters of each of the sensors.

[0094] The construction unit 33 can be used to match feature points in the calibration data corresponding to each of the sensors, and construct the initial extrinsic parameters between each of the sensors based on the coordinate information of the matched feature point pairs.

[0095] In a specific application scenario, the multi-sensors at least include a first sensor, a second sensor, and a third sensor; in order to construct the initial extrinsic parameters of each sensor, the construction unit 33 can specifically be used to construct a first initial extrinsic parameter between the first sensor and the second sensor based on the coordinate information of the second feature point pairs matched in the calibration data corresponding to the first sensor and the calibration data corresponding to the second sensor;

[0096] Construct a second initial extrinsic parameter between the first sensor and the third sensor based on the coordinate information of the third feature point pairs matched in the calibration data corresponding to the first sensor and the calibration data corresponding to the third sensor;

[0097] Construct a third initial extrinsic parameter between the second sensor and the third sensor based on the coordinate information of the fourth feature point pairs matched in the calibration data corresponding to the second sensor and the calibration data corresponding to the third sensor.

[0098] In a specific application scenario, in order to jointly iteratively optimize the initial intrinsic parameters of each sensor, the joint optimization unit 34 can specifically be used to determine the weight coefficients of the loss functions of the respective initial extrinsic parameters, and based on the weight coefficients, perform weighted combination on the loss functions of the respective initial extrinsic parameters to obtain a joint constraint equation, and based on the joint constraint equation, jointly iteratively optimize the initial intrinsic parameters and the initial extrinsic parameters of each sensor to obtain the calibration parameters of each sensor.

[0099] It should be noted that for other corresponding descriptions of each functional module involved in the multi-sensor joint calibration device provided in the embodiments of the present invention, reference can be made to Figure 1 the corresponding description of the method shown, which will not be elaborated here.

[0100] Based on the above as Figure 1 shown in the method, correspondingly, the embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented: respectively obtain calibration data detected by a plurality of sensors for a calibration device;

[0101] Determine the initial intrinsic parameters of the sensors;

[0102] Perform feature point matching in the calibration data corresponding to each of the sensors, and based on the coordinate information of the matched feature point pairs, construct the initial extrinsic parameters between each of the sensors;

[0103] Jointly iteratively optimize the initial extrinsic parameters and the initial intrinsic parameters to obtain the joint calibration results of each of the sensors.

[0104] Based on the above as Figure 1 shown in the method and as Figure 4 shown in the embodiments of the device, the embodiments of the present invention also provide a physical structure diagram of a computer device, as Figure 5As shown in the figure, the computer device includes: a processor 41, a memory 42, and a computer program stored on the memory 42 and executable on the processor. Both the memory 42 and the processor 41 are provided on a bus 43. When the processor 41 executes the program, the following steps are implemented: respectively obtaining calibration data detected by a plurality of sensors for a calibration device; determining initial internal parameters of the sensors; performing feature point matching in the calibration data corresponding to each of the sensors, and constructing initial external parameters between each of the sensors based on the coordinate information of the matched feature point pairs; jointly iteratively optimizing the initial external parameters and the initial internal parameters to obtain a joint calibration result of each of the sensors.

[0105] Through the technical solution of the present invention, after determining the initial external parameters and the initial internal parameters of the sensors, it is necessary to jointly optimize the initial external parameters and the initial internal parameters to obtain the calibration results of each sensor. The initial internal parameters participate in the subsequent iterative optimization process. Therefore, the present invention can first avoid the problem that the calibration effect of the external parameters is poor due to inaccurate calibration of the internal parameters. Secondly, the present invention constructs a loss function and restricts the calibration process of each sensor through constraint equations. It can quickly find the optimal solution based on the physical relationship and geometric constraints between the sensors, avoiding the blindness and inefficiency of manual exploration, and at the same time avoiding the problem of poor accuracy of traditional multi-sensor calibration methods. Finally, the rough calibration process of the present invention finds a starting point for joint optimization of the initial external parameters and the initial internal parameters, which helps the calibration parameters of each sensor to converge to the optimal solution faster during the optimization process, making the optimization process more stable, reducing the search space, thereby improving the calibration efficiency of each sensor, reducing the possibility of falling into a local optimum, and thus improving the calibration accuracy of each sensor.

[0106] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. In this way, the present invention is not limited to any specific combination of hardware and software.

[0107] The above are only preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-sensor joint calibration method, characterized in that, Including: Obtain the calibration data detected by multiple sensors for the calibration device respectively; Determine the initial internal parameters of the sensors; Perform feature point matching in the calibration data corresponding to each sensor, and construct the initial external parameters between each sensor based on the coordinate information of the matched feature point pairs; Perform joint iterative optimization on the initial external parameters and initial internal parameters to obtain the joint calibration results of each sensor.

2. The method according to claim 1, wherein The multi-sensors at least include a first sensor, a second sensor, and a third sensor; The constructing the initial external parameters between each sensor based on the coordinate information of the matched feature point pairs includes: Construct the first initial external parameter between the first sensor and the second sensor based on the coordinate information of the second feature point pairs matched in the calibration data corresponding to the first sensor and the calibration data corresponding to the second sensor; Construct the second initial external parameter between the first sensor and the third sensor based on the coordinate information of the third feature point pairs matched in the calibration data corresponding to the first sensor and the calibration data corresponding to the third sensor; Construct the third initial external parameter between the second sensor and the third sensor based on the coordinate information of the fourth feature point pairs matched in the calibration data corresponding to the second sensor and the calibration data corresponding to the third sensor.

3. The method according to claim 2, wherein Including: The first initial external parameter includes the rotation matrix and translation vector of the first sensor relative to the second sensor; And / or, the second initial external parameter includes the rotation matrix and translation vector of the first sensor relative to the third sensor; And / or, the third initial external parameter includes the rotation matrix and translation vector of the second sensor relative to the third sensor.

4. The method according to claim 1, wherein The performing joint iterative optimization on the initial external parameters and initial internal parameters to obtain the joint calibration results of each sensor includes: Construct the loss functions of the initial internal parameters and initial external parameters according to the detection data, initial internal parameters, and initial external parameters corresponding to each sensor; Perform weighted summation on the loss functions of the initial internal parameters and initial external parameters to obtain a joint constraint equation, and perform joint iterative optimization on the initial external parameters and initial internal parameters of each sensor based on the joint constraint equation to obtain the calibration parameters of each sensor.

5. The method according to claim 4, wherein Including: The weight coefficient of the loss function of the initial internal parameters is greater than the weight coefficient corresponding to the loss function of the initial external parameters.

6. A multi-sensor joint calibration device, characterized in that, Including: An obtaining unit, configured to obtain the calibration data detected by multiple sensors for the calibration board respectively; A determining unit, configured to determine the initial internal parameters of each sensor; A constructing unit, configured to perform feature point matching in the calibration data corresponding to each sensor, and construct the initial external parameters between each sensor based on the coordinate information of the matched feature point pairs; A joint optimization unit, configured to perform joint iterative optimization on the initial internal parameters of each sensor based on the initial external parameters to obtain the joint calibration results of each sensor.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images