Sensor calibration method and device based on Leiyu fusion, and storage medium

Abstract

The invention discloses a sensor calibration method and device based on Leiyu fusion and a storage medium, relates to the technical field of sensor calibration, and mainly aims to improve the calibration accuracy of a sensor. The method comprises the following steps: acquiring a calibration image shot by a camera device in a to-be-calibrated thundersight sensor for a calibration plate and point cloud data measured by a radar for the calibration plate; determining an initial calibration internal parameter of the camera device based on the calibration plate and the calibration image, and determining an initial calibration external parameter between the radar and the camera device based on the point cloud data and the calibration image; and performing joint optimization on the initial calibration internal reference of the camera device and the initial calibration external reference between the radar and the camera device to obtain the calibration internal reference of the camera device and the calibration external reference between the radar and the camera device. The method is suitable for a scene for calibrating the thunder vision sensor.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor calibration, and in particular, to a sensor calibration method, device, and storage medium based on radar-vision fusion. Background Art

[0002] The intelligent driving system relies on data provided by multiple sensors for decision-making and control. As an important part of it, the accuracy of the data of the radar-vision sensor is directly related to the performance and safety of the intelligent driving system. Through calibration, it can be ensured that the data output by the radar-vision sensor is consistent with the actual environment, thereby improving the overall accuracy and reliability of the intelligent driving system.

[0003] Currently, usually the internal parameters of the radar-vision sensor are calibrated first, and then the external parameters of the radar-vision sensor are calibrated according to the internal parameters. However, this calibration method will affect the calibration result of the external parameters on the basis of inaccurate internal parameter calibration, resulting in a low calibration accuracy of the radar-vision sensor. Summary of the Invention

[0004] The present invention provides a sensor calibration method, device, and storage medium based on radar-vision fusion, mainly capable of improving the calibration accuracy of the sensor.

[0005] According to the first aspect of the present invention, a sensor calibration method based on radar-vision fusion is provided, including:

[0006] Obtaining a calibration image captured by a camera device in a radar-vision sensor to be calibrated for a calibration board and point cloud data measured by a radar for the calibration board;

[0007] Based on the calibration board and the calibration image, determining initial calibration internal parameters of the camera device, and based on the point cloud data and the calibration image, determining initial calibration external parameters between the radar and the camera device;

[0008] Jointly optimizing the initial calibration internal parameters of the camera device and the initial calibration external parameters between the radar and the camera device to obtain calibration internal parameters of the camera device and calibration external parameters between the radar and the camera device.

[0009] Optionally, the determining the initial calibration internal parameters of the camera device based on the calibration board and the calibration image includes:

[0010] Determining internal parameter pixel coordinates corresponding to internal parameter feature points on the calibration board in the calibration image;

[0011] Determining internal parameter physical coordinates of points matching the internal parameter feature points in the calibration board;

[0012] Based on the internal reference pixel coordinates and the internal reference physical coordinates, determine the initial calibrated internal parameters of the imaging device.

[0013] Optionally, the determining the initial calibrated external parameters between the radar and the imaging device based on the point cloud data and the calibrated image includes:

[0014] Determine the feature point radar coordinates of the external reference feature points in the calibration board in the radar coordinate system, and initialize the external parameters between the radar and the imaging device to obtain the initialized external parameters;

[0015] Based on the initial calibrated internal parameters and the initialized external parameters, project the feature point radar coordinates onto the calibrated image to obtain the external reference projection coordinates of the external reference feature points on the calibrated image;

[0016] Determine the external reference pixel coordinates of the points matching the external reference feature points on the calibrated image;

[0017] Based on the external reference projection coordinates and the external reference pixel coordinates, determine an external reference calibration loss function, and optimize the initialized external parameters between the radar and the imaging device based on the external reference calibration loss function to obtain the initial calibrated external parameters between the radar and the imaging device.

[0018] Optionally, the jointly optimizing the initial calibrated internal parameters of the imaging device and the initial calibrated external parameters between the radar and the imaging device to obtain the calibrated internal parameters of the imaging device and the calibrated external parameters between the radar and the imaging device includes:

[0019] According to the physical coordinates of the target feature points on the checkerboard, the pixel coordinates of the target feature points in the image, and the target point cloud coordinates of the target feature points in the radar coordinate system, determine an internal reference calibration loss function, an external reference calibration loss function, and a parameter calibration loss function;

[0020] Based on the internal reference calibration loss function, the external reference calibration loss function, and the parameter calibration loss function, jointly optimize the initial calibrated internal parameters and the initial calibrated external parameters to obtain the calibrated internal parameters of the imaging device and the calibrated external parameters between the radar and the imaging device.

[0021] Optionally, it further includes:

[0022] Project the physical coordinates of the target feature points into the camera coordinate system, and then project them into the pixel coordinate system through the camera coordinate system and the initial calibrated internal parameters to obtain the first pixel coordinates;

[0023] Use the coordinates of the target feature points in the calibrated image as the second pixel coordinates;

[0024] Take the difference between the first pixel coordinate and the second pixel coordinate as the internal parameter calibration loss function;

[0025] and / or,

[0026] Project the target point cloud coordinates into the camera coordinate system using the initial calibrated external parameters, and then project them into the pixel coordinate system through the camera coordinate system and the initial calibrated internal parameters to obtain the third pixel coordinate;

[0027] Take the difference between the third pixel coordinate and the second pixel coordinate as the external parameter calibration loss function; wherein, in the external parameter calibration loss function, the initial calibrated internal parameters are fixed values;

[0028] and / or,

[0029] Take the difference between the third pixel coordinate and the second pixel coordinate as the parameter calibration loss function; wherein, in the parameter calibration loss function, both the initial calibrated internal parameters and the initial calibrated external parameters vary.

[0030] Optionally, jointly optimize the initial calibrated internal parameters and the initial calibrated external parameters based on the internal parameter calibration loss function, the external parameter calibration loss function, and the parameter calibration loss function to obtain the calibrated internal parameters of the imaging device and the calibrated external parameters between the radar and the imaging device, including:

[0031] Perform weighted summation on the internal parameter calibration loss function, the external parameter calibration loss function, and the parameter calibration loss function to obtain a joint calibration loss function;

[0032] Obtain the calibrated internal parameters of the imaging device and the calibrated external parameters between the radar and the imaging device based on the joint calibration loss function.

[0033] Optionally, it further includes:

[0034] In the joint calibration loss function, the weight of the parameter calibration loss function is more than ten times the weight value of the internal parameter calibration loss function and / or the external parameter calibration loss function.

[0035] According to the second aspect of the present invention, there is provided a sensor calibration device based on radar-vision fusion, including:

[0036] An acquisition unit for acquiring a calibration image captured by an imaging device in a to-be-calibrated radar-vision sensor for a calibration board and point cloud data measured by a radar for the calibration board;

[0037] A determination unit for determining the initial calibrated internal parameters of the imaging device based on the calibration board and the calibration image, and determining the initial calibrated external parameters between the radar and the imaging device based on the point cloud data and the calibration image;

[0038] A joint optimization unit is configured to jointly optimize the initial calibration intrinsic parameters of the camera device and the initial calibration extrinsic parameters between the radar and the camera device, so as to obtain the calibration intrinsic parameters of the camera device and the calibration extrinsic parameters between the radar and the camera device.

[0039] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the above-mentioned sensor calibration method based on radar-vision fusion is implemented.

[0040] According to a fourth aspect of the present invention, there is provided a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the above-mentioned sensor calibration method based on radar-vision fusion is implemented.

[0041] According to a sensor calibration method, device and storage medium based on radar-vision fusion provided by the present invention, compared with the current method of first calibrating the intrinsic parameters of radar-vision sensors and then calibrating the extrinsic parameters of radar-vision sensors according to the intrinsic parameters, the present invention first determines the initial calibration intrinsic parameters of the camera device and the initial calibration extrinsic parameters between the radar and the camera device through the calibration images captured by the camera device for the calibration board and the point cloud data measured by the radar for the calibration board, and then jointly optimizes the initial calibration intrinsic parameters and the initial calibration extrinsic parameters, so as to obtain the calibration intrinsic parameters of the camera device and the calibration extrinsic parameters between the radar and the camera device.

[0042] By means of the joint optimization and calibration method for intrinsic parameters and extrinsic parameters in the embodiments of the present invention, the problem that the calibration effect of the extrinsic parameters is poor due to inaccurate calibration of the intrinsic parameters can be avoided. At the same time, the present invention uses the initial calibration intrinsic parameters and the initial calibration extrinsic parameters as the starting point of joint optimization, which helps the calibration parameters of the radar-vision sensors to converge to the optimal solution faster during the optimization process, reduces the search space, thereby improving the calibration efficiency of the radar-vision sensors. And through the initial calibration, we can obtain relatively accurate initial calibration parameters of the sensors. Taking the initial calibration parameters as the starting point of joint optimization can make the optimization process more stable and reduce the possibility of falling into local optima, thereby improving the calibration accuracy of the radar-vision sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0044] Figure 1 A flowchart of a sensor calibration method based on radar-vision fusion provided by an embodiment of the present invention is shown;

[0045] Figure 2Shows a flowchart of another sensor calibration method based on radar-vision fusion provided by an embodiment of the present invention;

[0046] Figure 3 Shows a schematic structural diagram of a sensor calibration device based on radar-vision fusion provided by an embodiment of the present invention;

[0047] Figure 4 Shows a schematic structural diagram of another sensor calibration device based on radar-vision fusion provided by an embodiment of the present invention;

[0048] Figure 5 Shows a schematic physical structure diagram of a computer device provided by an embodiment of the present invention. Detailed implementation manners

[0049] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0050] Currently, the method of first calibrating the internal parameters of the radar-vision sensor and then calibrating the external parameters of the radar-vision sensor based on the internal parameters will, on the basis of inaccurate calibration of the internal parameters, affect the calibration result of the external parameters, resulting in a relatively low accuracy of calibrating the radar-vision sensor.

[0051] To solve the above problems, an embodiment of the present invention provides a sensor calibration method based on radar-vision fusion, as Figure 1 shown, the method includes:

[0052] 101. Obtain a calibration image captured by a camera device in a to-be-calibrated radar-vision sensor for a calibration board and point cloud data measured by a radar for the calibration board.

[0053] Among them, the calibration board contains multiple feature points. The calibration board can be a checkerboard calibration board, a circular grid calibration board, etc. When the calibration board is a checkerboard calibration board, its feature points are checkerboard corner points. When the calibration board is a circular grid calibration board, its feature points are the center points of circular patterns; the camera device can be a visual sensor such as a camera.

[0054] Specifically, select an environment with uniform light and no shadows, start the camera device to capture the calibration board to obtain a calibration image. At the same time, start the radar to emit a laser signal to the calibration board and receive the signal reflected back. According to the reflected signal, obtain a data set containing three-dimensional coordinate information, that is, point cloud data. Among them, both the calibration image and the point cloud data contain data corresponding to the feature points.

[0055] 102. Based on the calibration board and the calibration image, determine the initial calibration internal parameters of the camera device, and based on the point cloud data and the calibration image, determine the initial calibration external parameters between the radar and the camera device.

[0056] Among them, the initial calibrated internal parameters include the focal length, principal point coordinates, distortion coefficients, etc. of the imaging device; the initial calibrated external parameters refer to the conversion relationship between the radar coordinate system and the imaging device coordinate system.

[0057] For the embodiments of the present invention, the same feature point is determined in the calibration board and the calibration image, and based on the actual coordinates of the same feature point in the calibration board and the pixel coordinates in the calibration image, the internal parameters of the imaging device are calibrated to obtain the initial calibrated internal parameters. At the same time, based on the initial calibrated internal parameters, the calibration board feature points in the point cloud data in the radar coordinate system are projected onto the calibration image to obtain the projection coordinates, and the pixel coordinates of the points matching the feature points are determined in the calibration image. Finally, based on the projection coordinates and the pixel coordinates, the initial calibrated external parameters between the radar and the imaging device are determined.

[0058] The calibration process of the above initial calibrated internal parameters and initial calibrated external parameters can be implemented with reference to the existing methods.

[0059] 103. Jointly optimize the initial calibrated internal parameters of the imaging device and the initial calibrated external parameters between the radar and the imaging device to obtain the calibrated internal parameters of the imaging device and the calibrated external parameters between the radar and the imaging device.

[0060] In this step, both the initial calibrated internal parameters and the initial calibrated external parameters participate in the joint optimization as part of the calculation of the calibration loss, so that the calibrated internal parameters and the calibrated external parameters after optimization can be obtained simultaneously.

[0061] According to the sensor calibration method based on radar-vision fusion provided by the present invention, compared with the current method of first calibrating the internal parameters of the radar-vision sensor and then calibrating the external parameters of the radar-vision sensor according to the internal parameters, the present invention first determines the initial calibrated internal parameters of the imaging device and the initial calibrated external parameters between the radar and the imaging device through the calibration image captured by the imaging device for the calibration board and the point cloud data measured by the radar for the calibration board, and then jointly optimizes the initial calibrated internal parameters and the initial calibrated external parameters to obtain the calibrated internal parameters of the imaging device and the calibrated external parameters between the radar and the imaging device. Therefore, the embodiments of the present invention can avoid the problem that the calibration effect of the external parameters is poor due to inaccurate calibration of the internal parameters by means of jointly optimizing and calibrating the internal parameters and the external parameters. At the same time, the present invention takes the initial calibrated internal parameters and the initial calibrated external parameters as the starting point of the joint optimization, which helps the calibration parameters of the radar-vision sensor to converge to the optimal solution faster during the optimization process, reduces the search space, thereby improving the calibration efficiency of the radar-vision sensor, and through the initial calibration, relatively accurate initial calibration parameters of the sensor can be obtained. Taking the initial calibration parameters as the starting point of the joint optimization can make the optimization process more stable and reduce the possibility of falling into the local optimum, thereby improving the calibration accuracy of the radar-vision sensor.

[0062] Further, to better illustrate the above process of calibrating the sensor, as a refinement and extension of the above embodiments, the embodiments of the present invention provide another sensor calibration method based on lidar-camera fusion, as Figure 2 shown, the method includes:

[0063] 201. Obtain the calibration image captured by the camera device in the lidar-camera sensor to be calibrated for the calibration board and the point cloud data measured by the radar for the calibration board.

[0064] Specifically, the calibration board is usually a flat plate with known dimensions and patterns. The size of the calibration board should be clearly visible in the fields of view of the camera device and the radar, ensuring that the patterns on the calibration board have accurate sizes and spacings. Ensure that the camera device and the radar are correctly installed and fixed. Select an environment with uniform light, no shadows and reflections, use the camera device to capture the calibration image corresponding to the calibration board, and at the same time use the radar to measure the point cloud data of the calibration board.

[0065] 202. Determine the initial calibration internal parameters of the camera device based on the calibration board and the calibration image.

[0066] For the embodiments of the present invention, to improve the efficiency and accuracy of joint calibration, the initialized internal and external parameters should be avoided as the starting point of joint calibration. Instead, the internal parameters of the camera device should be calibrated first (coarse calibration). Based on this, step 202 specifically includes:

[0067] Determine the pixel coordinates of the internal parameter feature points on the calibration board in the calibration image; determine the physical coordinates of the points matching the internal parameter feature points in the calibration board; based on the pixel coordinates and the physical coordinates, determine the initial calibration internal parameters of the camera device.

[0068] Specifically, for example, if the internal parameter feature points are the checkerboard corner points on the calibration board, then first determine the pixel coordinates of the checkerboard corner points in the calibration image, and then determine the actual coordinates (world coordinates) of the points corresponding to the checkerboard corner points in the actual calibration board. According to the pixel coordinates and physical coordinates of multiple feature points, establish a matrix equation from the world coordinate system to the pixel coordinate system, and use algorithms such as the direct linear transformation algorithm or the least squares method to solve the matrix equation to obtain the matrix H. Among them, this matrix H represents the transformation relationship between the world coordinate system and the pixel coordinate system. Then, according to the relationship between the internal parameter matrix of the camera device and the matrix H, solve the internal parameter matrix of the camera device to obtain the initial calibration internal parameters of the camera device. The formula for specifically determining the initial calibration internal parameters is as follows:

[0069]

[0070]

[0071] Among them, the internal parameter pixel coordinates of feature point i on the calibrated image are (u i , v i ), and the internal parameter physical coordinates of this feature point i on the calibration board are (X i , Y i ). If h 11 ....h 33 are the elements in the homography matrix H, the transformation relationship between the above physical coordinates and pixel coordinates is converted into a linear equation. According to the internal parameter pixel coordinates and internal parameter physical coordinates corresponding to multiple feature points, a linear equation system about H can be obtained, and algorithms such as the least squares method are used to determine the homography matrix H that minimizes the sum of the squares of the residuals of all equations. Then, the internal parameter matrix A is determined according to the following formula:

[0072] H = AR

[0073] Among them, A is the internal parameter matrix and R is the constraint condition. Thus, through the above formula, the internal parameter matrix A can be obtained, and finally the initial calibration internal parameters of the imaging device are determined in the internal parameter matrix.

[0074] Furthermore, in order to improve the accuracy of the initial calibration internal parameters, based on the initial calibration internal parameters, the feature points in the calibration board are projected onto the calibrated image to obtain the pixel coordinates of the feature points in the calibrated image. Then, the physical coordinates of the feature points are determined in the calibration board. According to the difference between the corresponding pixel coordinates and physical coordinates of the feature points, the internal parameter calibration loss function is determined. Finally, based on the internal parameter calibration loss function, the initial calibration internal parameters are iteratively optimized to obtain the initial calibration internal parameters that meet the loss requirements.

[0075] In another embodiment of the present invention, the initial calibration internal parameters of the imaging device can also be predicted by using a preset internal parameter prediction model. In order to improve the prediction accuracy of the preset internal parameter prediction model, first, a preset internal parameter prediction model needs to be constructed. Based on this, the method includes: constructing an initial model and obtaining a sample data set. Among them, this sample data set contains various sample calibrated images with annotation information. The calibrated image is an image taken by the sample imaging device for the calibration board, and the annotation information is the actual internal parameters corresponding to the sample imaging device; dividing the sample data set into training data and test data; training the initial model with the training data and testing the trained initial model with the test data. Finally, the initial model that meets the test conditions is determined as the preset internal parameter prediction model. Furthermore, inputting the calibrated image into the preset internal parameter prediction model, the initial calibration internal parameters of the imaging device can be output through the preset internal parameter prediction model. The embodiment of the present invention predicts the internal parameters of the imaging device through the model, which can improve the prediction efficiency of the internal parameters of the imaging device.

[0076] 203. Determine the feature point radar coordinates of the external reference feature points in the calibration board in the radar coordinate system, and initialize the external reference between the radar and the imaging device to obtain the initialized external reference.

[0077] For the embodiments of the present invention, if it is a checkerboard calibration board, the external reference feature points are the checkerboard corner points. In order to calibrate the external reference between the radar and the imaging device, it is first necessary to initialize the external reference between the radar and the imaging device using the randomization parameters to obtain the initialized external reference, and at the same time accurately determine the feature point radar coordinates of the external reference feature points in the calibration board in the radar coordinate system. Based on this, step 203 specifically includes:

[0078] Perform plane fitting on the external reference feature points in the point cloud data to obtain the point cloud plane of the checkerboard calibration board in the radar coordinate system; determine the axis directions and the coordinate origin of the point cloud plane; based on the axis directions, the coordinate origin, and the initial coordinates of the external reference feature points in the radar coordinate system, project the external reference feature points in the point cloud plane onto the two-dimensional plane of the calibration board to obtain the point cloud projection coordinates;

[0079] Determine the point cloud physical coordinates of the points matching the external reference feature points on the two-dimensional plane of the calibration board; based on the point cloud projection coordinates and the point cloud physical coordinates, determine the coordinate loss function, and based on the coordinate loss function, optimize the initial coordinates of the external reference feature points in the radar coordinate system to obtain the feature point radar coordinates of the external reference feature points in the radar coordinate system.

[0080] Among them, the method of performing plane fitting on the external reference feature points to obtain the point cloud plane includes: performing plane fitting on the external reference feature points in the point cloud data to obtain the initial point cloud plane in the radar coordinate system; projecting the remaining point cloud data onto the initial point cloud plane to obtain the point cloud plane containing all the point cloud data, where the remaining point cloud data is the point cloud data obtained by removing the external reference feature points from the point cloud data.

[0081] Among them, the method of determining the axis directions and the coordinate origin of the point cloud plane includes: determining the covariance matrix of the point cloud plane, and determining the eigenvalues and the eigenvectors corresponding to the eigenvalues of the covariance matrix; based on the eigenvalues and the eigenvectors, determine the axis directions of the point cloud plane; based on the coordinate mean values of the points in the point cloud plane, determine the coordinate origin of the point cloud plane.

[0082] Specifically, first, according to the three-dimensional coordinates of each point in the point cloud data, the initial coordinates corresponding to the external parameter feature points in the point cloud data are determined. At the same time, in the radar coordinate system, the external parameter feature points in the point cloud data are plane-fitted to obtain the initial point cloud plane. Then, the remaining point cloud data in the point cloud data excluding the external parameter feature points can be projected onto the initial point cloud plane using, for example, the ray method to obtain the point cloud plane. Further, according to each point in the point cloud data on the point cloud plane, all points within a preset radius (set according to actual needs) are determined as neighborhood points. For each point, the coordinate deviation between all points in its neighborhood and this point is calculated, and a coordinate deviation matrix is formed by each coordinate deviation. Then, the covariance matrix C is determined according to the following formula:

[0083]

[0084] where n is the total number of neighborhood points corresponding to a certain point, S is the coordinate deviation matrix, and S T is the transpose matrix of the coordinate deviation matrix. Further, the covariance matrix is subjected to eigenvalue decomposition to obtain eigenvalues and eigenvectors. Then, target eigenvalues greater than a preset threshold are selected after each eigenvalue, and the coordinate axis directions of the point cloud plane are determined according to the eigenvectors corresponding to the target eigenvalues. At the same time, the coordinate mean value of each feature point in the point cloud plane in the radar coordinate system is determined as the coordinate origin of the point cloud plane. Further, according to the coordinate axis directions, coordinate origin of the point cloud plane, and the initial coordinates of the external parameter feature points in the radar coordinate system, the external parameter feature points in the point cloud plane are projected onto the calibration board to obtain the point cloud projection coordinates, and the coordinates of the points matching the external parameter feature points (point cloud physical coordinates) are determined in the calibration board. According to the difference between the point cloud projection coordinates and the point cloud physical coordinates corresponding to the same feature point, a coordinate loss function is determined. Finally, the initial coordinates of the external parameter feature points in the radar coordinate system are iteratively optimized according to the coordinate loss function to obtain radar coordinates with losses meeting the requirements. In the embodiments of the present invention, the coordinate of the feature point in the radar coordinate system is optimized through the loss function, which can ensure the coordinate accuracy of the feature point in the radar coordinate system.

[0085] 204. Based on the initial calibration internal parameters and the initialized external parameters, project the radar coordinates of the feature points onto the calibration image to obtain the external parameter projection coordinates of the external parameter feature points on the calibration image.

[0086] Specifically, the radar coordinates of the feature points are the coordinate values of the external parameter feature points in the radar coordinate system. The radar coordinates of the feature points are projected into the camera coordinate system through the initialized external parameters, and then the obtained coordinates are projected into the pixel coordinate system according to the initial calibration internal parameters to obtain the external parameter projection coordinates;

[0087] The initialized extrinsic parameters are the transformation matrix between the radar coordinate system and the camera coordinate system. Therefore, a feature point in the radar coordinate system can be transformed to obtain its coordinates in the camera coordinate system through the initialized extrinsic parameters. Similarly, since the initialized intrinsic parameters are the transformation matrix between the camera coordinate system and the pixel coordinate system, a point in the camera coordinate system can be transformed to obtain its coordinates in the pixel coordinate system through the initialized intrinsic parameters.

[0088] Therefore, by multiplying the radar coordinates of the feature point by the initialized extrinsic parameters and the initialized intrinsic parameters, the extrinsic projection coordinates in the pixel coordinate system can be obtained. Among them, the two-dimensional coordinate system where the calibration image is located is the pixel coordinate system.

[0089] 205. Determine the extrinsic pixel coordinates of the points matching the extrinsic feature points on the calibration image.

[0090] An image detection algorithm can be used to detect the calibration image to obtain the extrinsic pixel coordinates in the pixel coordinate system that match the extrinsic feature points.

[0091] 206. Based on the extrinsic projection coordinates and the extrinsic pixel coordinates, determine the extrinsic calibration loss function, and optimize the initialized extrinsic parameters between the radar and the imaging device based on the extrinsic calibration loss function to obtain the initial calibrated extrinsic parameters between the radar and the imaging device.

[0092] According to the initial calibrated intrinsic parameters of the imaging device and the feature point radar coordinates of the calibration plate feature points (extrinsic feature points) in the radar coordinate system, project the feature point radar coordinates onto the calibration image to obtain the extrinsic projection coordinates of the extrinsic feature points on the calibration image. At the same time, determine the extrinsic pixel coordinates of the points matching the extrinsic feature points on the calibration image, and determine the extrinsic calibration loss function based on the difference between the corresponding extrinsic projection coordinates and the extrinsic pixel coordinates of the same feature point. Finally, use the extrinsic calibration loss function to iteratively optimize the initialized extrinsic parameters between the radar and the imaging device to obtain the initial calibrated extrinsic parameters between the radar and the imaging device with the loss meeting the requirements. In the embodiment of the present invention, the extrinsic parameters between the radar and the imaging device are iteratively optimized through the extrinsic calibration loss function, which can improve the determination accuracy of the initial calibrated extrinsic parameters.

[0093] 207. Jointly optimize the initial calibrated intrinsic parameters of the imaging device and the initial calibrated extrinsic parameters between the radar and the imaging device to obtain the calibrated intrinsic parameters of the imaging device and the calibrated extrinsic parameters between the radar and the imaging device.

[0094] For the embodiment of the present invention, after determining the initial calibrated intrinsic parameters of the imaging device and the initial calibrated extrinsic parameters between the radar and the imaging device, it is necessary to use the initial calibrated intrinsic parameters and the initial calibrated extrinsic parameters as the starting point to jointly calibrate the intrinsic parameters of the imaging device and the extrinsic parameters between the radar and the imaging device. Based on this, step 207 specifically includes:

[0095] Determine the intrinsic calibration loss function, extrinsic calibration loss function, and parameter calibration loss function based on the physical coordinates of the target feature points on the checkerboard, the pixel coordinates of the target feature points in the calibrated image, and the target point cloud coordinates of the target feature points in the radar coordinate system;

[0096] Based on the intrinsic calibration loss function, extrinsic calibration loss function, and parameter calibration loss function, jointly optimize the initial calibrated intrinsic parameters and the initial calibrated extrinsic parameters to obtain the calibrated intrinsic parameters of the imaging device and the calibrated extrinsic parameters between the radar and the imaging device.

[0097] Among them, the calibrated image is a two-dimensional image captured by the camera; the target feature points are the feature points used for joint optimization, and these feature points can be detected by both the camera and the radar. The target feature points can still be the corner points of the checkerboard.

[0098] Specifically, joint optimization requires constructing a joint calibration loss function, which is used to optimize the initial calibrated intrinsic parameters and the initial calibrated extrinsic parameters simultaneously. Based on this, in this application, the joint calibration loss function includes three parts, namely the intrinsic calibration loss function, the extrinsic calibration loss function, and the parameter calibration loss function, where:

[0099] The intrinsic calibration loss function cost_intrinsic: project the physical coordinates of the target feature points into the camera coordinate system, and then project them into the pixel coordinate system through the camera coordinate system and the initial calibrated intrinsic parameters to obtain the first pixel coordinates; use the coordinates of the target feature points in the calibrated image as the second pixel coordinates; take the difference between the first pixel coordinates and the second pixel coordinates as the intrinsic calibration loss function.

[0100] Among them, the second pixel coordinates can use an image detection algorithm to detect the calibrated image, and then identify the coordinates of the target feature points in the pixel coordinate system. Since the calibrated image is captured by the camera and the camera has distortion, that is, the second pixel coordinates are the coordinates of the target feature points after distortion.

[0101] The physical coordinates of the target feature points are undistorted and real coordinates. Since the initial calibrated intrinsic parameters represent the distortion parameters of the camera, when the physical coordinates of the target feature points are first projected into the camera coordinate system and then projected into the pixel coordinate system through the initial calibrated intrinsic parameters, the obtained first pixel coordinates are distorted coordinates. The difference between the first pixel coordinates and the second pixel coordinates represents the calibration loss in the initial intrinsic parameter calibration process, that is, the intrinsic calibration loss function.

[0102] Extrinsic calibration loss function cost_extrinsic: The target point cloud coordinates are projected into the camera coordinate system using the initial calibrated extrinsic parameters, and then projected into the pixel coordinate system through the camera coordinate system and the initial calibrated intrinsic parameters to obtain the third pixel coordinates. The difference between the third pixel coordinates and the second pixel coordinates is used as the extrinsic calibration loss function. Among them, in the extrinsic calibration loss function, the initial calibrated intrinsic parameters are fixed values.

[0103] Since the extrinsic calibration loss is only used to measure the accuracy of the relative extrinsic calibration between the camera and the radar, in the extrinsic calibration loss function, it is necessary to fix the initial calibrated intrinsic parameters of the camera and keep them unchanged.

[0104] Parameter calibration loss function cost_intrinsic_extrinsic: The difference between the third pixel coordinates and the second pixel coordinates is used as the parameter calibration loss function. Among them, in the parameter calibration loss function, both the initial calibrated intrinsic parameters and the initial calibrated extrinsic parameters change.

[0105] The expression of the parameter calibration loss function is the same as that of the extrinsic calibration loss function. The difference is that since the parameter calibration loss function is used to measure the influence of the mutual influence and restriction between the initial calibrated extrinsic parameters and the initial calibrated intrinsic parameters on the final calibration effect, in the parameter calibration loss function, both the initial calibrated intrinsic parameters and the initial calibrated extrinsic parameters change, that is, they are both optimization objects.

[0106] Furthermore, after determining the intrinsic calibration loss function cost_intrinsic during the intrinsic calibration process of the imaging device (camera), the extrinsic calibration loss function cost_extrinsic of the initial calibrated extrinsic parameters between the radar and the imaging device during the calibration process, and the parameter calibration loss function cost_intrinsic_extrinsic, the three parts of the loss functions can be directly summed as the joint calibration loss function for multiple rounds of iterative optimization. When the preset convergence condition is met (such as the loss value is less than the threshold, or the number of iterations reaches the set number, etc.), the iteration is stopped, and the finally obtained camera intrinsic parameters and the relative extrinsic parameters between the camera and the radar are used as the final calibrated intrinsic parameters and calibrated extrinsic parameters respectively.

[0107] In another embodiment of the present application, we may hope that the influence of different loss functions on the joint calibration loss function is not exactly the same. Therefore, a weighted summation method can be used. For example, the joint calibration loss function cost can be determined using the following formula:

[0108] cost = α * cost_intrinsic + β * cost_extrinsic + γ * cost_intrinsic_extrinsic

[0109] Among them, α is the weight coefficient corresponding to the internal parameter calibration loss function, β is the weight coefficient corresponding to the external parameter calibration loss function, and γ is the weight coefficient corresponding to the parameter calibration loss function.

[0110] In the above formula, α, β, and γ are hyperparameters that determine the weights of the three loss functions (internal parameter calibration loss function, external parameter calibration loss function, parameter calibration loss function) in the joint calibration loss function. The larger their values, the greater the weights. The weight values can be adjusted according to the actual iteration effect.

[0111] However, if we want to focus on joint optimization, the value of γ can be set relatively large. The other two loss functions (internal parameter calibration loss function, external parameter calibration loss function) are loss functions of auxiliary fine-tuning nature. Therefore, the corresponding weight coefficients can be set relatively small. For example, γ can be set to more than ten times that of α and / or β.

[0112] In an embodiment of the present application, γ is set to 50; α and β can be set to 1. It should be noted that the values of α, β, and γ are not limited to the above examples. The values of α, β, and γ can be flexibly adjusted according to different scenarios, requirements, and optimization effects.

[0113] The reason for setting the ratio between each weight is as follows: In the initial calibration process, it can be considered that the initial calibrated internal parameters are already close to the most ideal solution and there is no longer a large error. Therefore, only the joint calibration method is needed to fine-tune the initial calibrated internal parameters instead of making large adjustments. So the value of γ is generally set relatively large to limit the change range of the initial calibrated internal parameters. However, to prevent the situation where there is no unique solution for the relative external parameters between the radar and the camera at far and near distances due to inaccurate internal parameter calibration, by introducing the other two loss functions, they play a balancing role in the joint calibration loss function, and these two parts of the loss should not occupy a relatively important position in the overall loss. Therefore, α and β do not need to be set too large.

[0114] The embodiment of the present invention initializes through the internal and external parameters of rough calibration, so that the starting point of optimizing each parameter is already close to the ideal value. As a result, the subsequent joint optimization effect will be better than the method without initial calibration, and at the same time better than the method of first calibrating the camera internal parameters and then calibrating the external parameters without joint calibration, thereby further improving the calibration accuracy of the sensor parameters.

[0115] It should be noted that randomly setting the starting point for optimizing the internal parameters of the camera device and the starting point for optimizing the external parameters between the radar and the camera device, and then directly using the joint calibration loss function to jointly optimize the randomly set starting points of each parameter is also a way of joint optimization. However, in the above method, since the starting points for optimizing each parameter are random and there are too many parameters to be optimized, the convergence effect is not good, resulting in an unsatisfactory optimization effect. Based on this, in another embodiment of the present invention, first, a rough calibration of the internal parameters of the camera device and a rough calibration of the external parameters between the radar and the camera device are performed to obtain the initial calibrated internal parameters and the initial calibrated external parameters. Finally, the joint calibration loss function is used to jointly optimize the initial calibrated internal parameters and the initial calibrated external parameters to obtain the calibrated internal parameters of the camera device and the calibrated external parameters between the radar and the camera device.

[0116] According to another sensor calibration method based on radar-vision fusion provided by the present invention, compared with the current method of first calibrating the internal parameters of the radar-vision sensor and then calibrating the external parameters of the radar-vision sensor according to the internal parameters, the present invention first determines the initial calibrated internal parameters of the camera device and the initial calibrated external parameters between the radar and the camera device through the calibration images taken by the camera device of the calibration board and the point cloud data measured by the radar for the calibration board, and then jointly optimizes the initial calibrated internal parameters and the initial calibrated external parameters to obtain the calibrated internal parameters of the camera device and the calibrated external parameters between the radar and the camera device. Thus, through the method of jointly optimizing and calibrating the internal parameters and the external parameters, the embodiment of the present invention can avoid the problem that the calibration effect of the external parameters is poor due to inaccurate calibration of the internal parameters. At the same time, the present invention uses the initial calibrated internal parameters and the initial calibrated external parameters as the starting points for joint optimization, which helps the calibration parameters of the radar-vision sensor to converge to the optimal solution faster during the optimization process, reduces the search space, thereby improving the calibration efficiency of the radar-vision sensor. And through the initial calibration, we can obtain relatively accurate initial calibration parameters of the sensor. Using the initial calibration parameters as the starting points for joint optimization can make the optimization process more stable, reduce the possibility of falling into local optima, and thus improve the calibration accuracy of the radar-vision sensor.

[0117] Further, as Figure 1 a specific implementation of, the embodiment of the present invention provides a sensor calibration device based on radar-vision fusion, as Figure 3 shown, the device includes: an acquisition unit 31, a determination unit 32, and a joint optimization unit 33.

[0118] The acquisition unit 31 can be used to acquire the calibration images taken by the camera device in the to-be-calibrated radar-vision sensor for the calibration board and the point cloud data measured by the radar for the calibration board.

[0119] The determining unit 32 can be used to determine the initial calibration intrinsic parameters of the camera device based on the calibration board and the calibration image, and determine the initial calibration extrinsic parameters between the radar and the camera device based on the point cloud data and the calibration image.

[0120] The joint optimization unit 33 can be used to jointly optimize the initial calibration intrinsic parameters of the camera device and the initial calibration extrinsic parameters between the radar and the camera device to obtain the calibration intrinsic parameters of the camera device and the calibration extrinsic parameters between the radar and the camera device.

[0121] In a specific application scenario, in order to determine the initial calibration intrinsic parameters of the camera device, the determining unit 32 can specifically be used to determine the intrinsic pixel coordinates corresponding to the intrinsic feature points on the calibration board in the calibration image; determine the intrinsic physical coordinates of the points matching the intrinsic feature points in the calibration board; and determine the initial calibration intrinsic parameters of the camera device based on the intrinsic pixel coordinates and the intrinsic physical coordinates.

[0122] In a specific application scenario, in order to determine the initial calibration extrinsic parameters between the radar and the camera device, as Figure 4 shown, the determining unit 32 includes a first determining module 321, a first projection module 322, and an extrinsic parameter optimization module 323.

[0123] The first determining module 321 can be used to determine the feature point radar coordinates of the extrinsic feature points in the calibration board in the radar coordinate system and initialize the extrinsic parameters between the radar and the camera device to obtain the initialized extrinsic parameters.

[0124] The first projection module 322 can be used to project the feature point radar coordinates onto the calibration image based on the initial calibration intrinsic parameters and the initialized extrinsic parameters to obtain the extrinsic projection coordinates of the extrinsic feature points on the calibration image.

[0125] The first determining module 321 can also be used to determine the extrinsic pixel coordinates of the points matching the extrinsic feature points on the calibration image.

[0126] The extrinsic parameter optimization module 323 can be used to determine an extrinsic parameter calibration loss function based on the extrinsic projection coordinates and the extrinsic pixel coordinates, and optimize the initialized extrinsic parameters between the radar and the camera device based on the extrinsic parameter calibration loss function to obtain the initial calibration extrinsic parameters between the radar and the camera device.

[0127] In a specific application scenario, in order to determine the radar coordinates of the extrinsic feature points in the calibration board in the radar coordinate system, the first determining module 321 includes a plane fitting sub-module, a determining sub-module, a projection sub-module, and a parameter optimization sub-module.

[0128] The plane fitting sub-module can be used to perform plane fitting on the external parameter feature points in the point cloud data to obtain the point cloud plane in the radar coordinate system.

[0129] The determination sub-module can be used to determine the axis direction and coordinate origin of the point cloud plane.

[0130] The projection sub-module can be used to project the external parameter feature points in the point cloud plane onto the calibration board based on the axis direction, the coordinate origin, and the initial coordinates of the external parameter feature points in the radar coordinate system to obtain the point cloud projection coordinates.

[0131] The determination sub-module can also be used to determine the physical coordinates of the point cloud of the points matching the external parameter feature points on the calibration board.

[0132] The parameter optimization sub-module can be used to determine a coordinate loss function based on the point cloud projection coordinates and the physical coordinates of the point cloud, and optimize the initial coordinates of the external parameter feature points in the radar coordinate system based on the coordinate loss function to obtain the radar coordinates of the external parameter feature points in the radar coordinate system.

[0133] In a specific application scenario, in order to perform plane fitting on the external parameter feature points in the point cloud data, the plane fitting sub-module can specifically be used to perform plane fitting on the external parameter feature points in the point cloud data to obtain the initial point cloud plane in the radar coordinate system; project the remaining point cloud data onto the initial point cloud plane to obtain a point cloud plane including all the point cloud data, where the remaining point cloud data is the point cloud data obtained by removing the external parameter feature points from the point cloud data.

[0134] In a specific application scenario, in order to determine the axis direction and coordinate origin of the point cloud plane, the determination sub-module can specifically be used to determine the covariance matrix of the point cloud plane, and determine the eigenvalues of the covariance matrix and the eigenvectors corresponding to the eigenvalues; determine the axis direction of the point cloud plane based on the eigenvalues and the eigenvectors; determine the coordinate origin of the point cloud plane based on the coordinate mean of each point in the point cloud plane.

[0135] In a specific application scenario, in order to jointly optimize the initial calibration internal parameters and the initial calibration external parameters, the joint optimization unit 33 includes a second projection module 331, a second determination module 332, and a joint optimization module 333.

[0136] The second projection module 331 can be used to project the point cloud feature points in the point cloud data onto the calibration image based on the initial calibrated internal parameters and the initial calibrated external parameters, so as to obtain the predicted pixel coordinates corresponding to the point cloud feature points.

[0137] The second determination module 332 can be used to determine the actual pixel coordinates of the points matching the point cloud feature points in the calibration image, and determine the parameter calibration loss function based on the actual pixel coordinates and the predicted pixel coordinates.

[0138] The joint optimization module 333 can be used to jointly optimize the initial calibrated internal parameters of the imaging device and the initial calibrated external parameters between the radar and the imaging device based on the parameter calibration loss function, so as to obtain the calibrated internal parameters of the imaging device and the calibrated external parameters between the radar and the imaging device.

[0139] In a specific application scenario, in order to jointly optimize the initial calibrated internal parameters and the initial calibrated external parameters, the joint optimization module 333 can specifically be used to determine a joint calibration loss function based on the internal parameter calibration loss function of the initial calibrated internal parameters in the calibration process, the external parameter calibration loss function of the initial calibrated external parameters in the calibration process, and the parameter calibration loss function; and jointly optimize the initial calibrated internal parameters of the imaging device and the initial calibrated external parameters between the radar and the imaging device based on the joint calibration loss function, so as to obtain the calibrated internal parameters of the imaging device and the calibrated external parameters between the radar and the imaging device.

[0140] It should be noted that for other corresponding descriptions of each functional module involved in the sensor calibration device based on radar-vision fusion provided in the embodiments of the present invention, reference can be made to Figure 1 the corresponding description of the method shown, which will not be elaborated here.

[0141] Based on the above as Figure 1 shown in the method, correspondingly, the embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented: obtaining a calibration image captured by an imaging device in a radar-vision sensor to be calibrated and point cloud data measured by a radar for the calibration board; determining the initial calibrated internal parameters of the imaging device based on the calibration board and the calibration image, and determining the initial calibrated external parameters between the radar and the imaging device based on the point cloud data and the calibration image; jointly optimizing the initial calibrated internal parameters of the imaging device and the initial calibrated external parameters between the radar and the imaging device to obtain the calibrated internal parameters of the imaging device and the calibrated external parameters between the radar and the imaging device.

[0142] Based on the above asFigure 1 The method and apparatus embodiments shown Figure 3 shown, embodiments of the present invention also provide an entity structure diagram of a computer device, as Figure 5 shown, the computer device includes: a processor 41, a memory 42, and a computer program stored on the memory 42 and executable on the processor. Both the memory 42 and the processor 41 are disposed on a bus 43. When the processor 41 executes the program, the following steps are implemented: obtaining a calibration image captured by a camera device in a to-be-calibrated radar-vision sensor for a calibration board and point cloud data measured by a radar for the calibration board; determining an initial calibration internal parameter of the camera device based on the calibration board and the calibration image, and determining an initial calibration external parameter between the radar and the camera device based on the point cloud data and the calibration image; jointly optimizing the initial calibration internal parameter of the camera device and the initial calibration external parameter between the radar and the camera device to obtain a calibration internal parameter of the camera device and a calibration external parameter between the radar and the camera device.

[0143] Through the technical solution of the present invention, the present invention first determines an initial calibration internal parameter of a camera device and an initial calibration external parameter between a radar and the camera device through a calibration image captured by the camera device for a calibration board and point cloud data measured by the radar for the calibration board, and then jointly optimizes the initial calibration internal parameter and the initial calibration external parameter to obtain a calibration internal parameter of the camera device and a calibration external parameter between the radar and the camera device. Thus, through the method of jointly optimizing and calibrating the internal parameter and the external parameter, the embodiment of the present invention can avoid the problem that the calibration effect of the external parameter is poor due to inaccurate calibration of the internal parameter. At the same time, the present invention uses the initial calibration internal parameter and the initial calibration external parameter as the starting point of joint optimization, which helps the calibration parameters of the radar-vision sensor to converge to the optimal solution faster during the optimization process, reduces the search space, thereby improving the calibration efficiency of the radar-vision sensor. And through the initial calibration, we can obtain relatively accurate initial calibration parameters of the sensor. Using the initial calibration parameters as the starting point of joint optimization can make the optimization process more stable and reduce the possibility of falling into a local optimum, thereby improving the calibration accuracy of the radar-vision sensor.

[0144] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed over a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a sequence different from that here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. In this way, the present invention is not limited to any specific combination of hardware and software.

[0145] The foregoing is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A sensor calibration method based on radar-vision fusion, characterized in that, Including: Obtaining a calibration image captured by a camera device in a to-be-calibrated radar-vision sensor for a calibration board and point cloud data measured by a radar for the calibration board; Based on the calibration board and the calibration image, determining initial calibration internal parameters of the camera device, and based on the point cloud data and the calibration image, determining initial calibration external parameters between the radar and the camera device; Jointly optimizing the initial calibration internal parameters of the camera device and the initial calibration external parameters between the radar and the camera device to obtain calibration internal parameters of the camera device and calibration external parameters between the radar and the camera device.

2. The method according to claim 1, wherein The determining the initial calibration internal parameters of the camera device based on the calibration board and the calibration image includes: Determining internal parameter pixel coordinates corresponding to internal parameter feature points on the calibration board in the calibration image; Determining internal parameter physical coordinates of points matching the internal parameter feature points in the calibration board; Based on the internal parameter pixel coordinates and the internal parameter physical coordinates, determining the initial calibration internal parameters of the camera device.

3. The method according to claim 1, characterized in that The determining the initial calibration external parameters between the radar and the camera device based on the point cloud data and the calibration image includes: Determining feature point radar coordinates of external parameter feature points in the calibration board in the radar coordinate system, and initializing external parameters between the radar and the camera device to obtain initialized external parameters; Based on the initial calibration internal parameters and the initialized external parameters, projecting the feature point radar coordinates onto the calibration image to obtain external parameter projection coordinates of the external parameter feature points on the calibration image; Determining external parameter pixel coordinates of points matching the external parameter feature points on the calibration image; Based on the external parameter projection coordinates and the external parameter pixel coordinates, determining an external parameter calibration loss function, and optimizing the initialized external parameters between the radar and the camera device based on the external parameter calibration loss function to obtain the initial calibration external parameters between the radar and the camera device.

4. The method according to claim 1, characterized in that, The jointly optimizing the initial calibration internal parameters of the camera device and the initial calibration external parameters between the radar and the camera device to obtain calibration internal parameters of the camera device and calibration external parameters between the radar and the camera device includes: According to the physical coordinates of target feature points on a checkerboard, the pixel coordinates of the target feature points in an image, and the target point cloud coordinates of the target feature points in the radar coordinate system, determining an internal parameter calibration loss function, an external parameter calibration loss function, and a parameter calibration loss function; Based on the internal parameter calibration loss function, the external parameter calibration loss function, and the parameter calibration loss function, jointly optimizing the initial calibration internal parameters and the initial calibration external parameters to obtain calibration internal parameters of the camera device and calibration external parameters between the radar and the camera device.

5. The method according to claim 4, characterized in that, It also includes: Projecting the physical coordinates of the target feature points into the camera coordinate system, and then projecting them into the pixel coordinate system through the camera coordinate system and the initial calibration internal parameters to obtain first pixel coordinates; Taking the coordinates of the target feature points in the calibration image as second pixel coordinates; Taking the difference between the first pixel coordinates and the second pixel coordinates as the internal parameter calibration loss function; And / or Project the coordinates of the target point cloud to the camera coordinate system using the initial calibrated extrinsic parameters, and then project them to the pixel coordinate system through the camera coordinate system and the initial calibrated intrinsic parameters to obtain the third pixel coordinates; Take the difference between the third pixel coordinates and the second pixel coordinates as the extrinsic parameter calibration loss function; wherein, in the extrinsic parameter calibration loss function, the initial calibrated intrinsic parameters are fixed values; and / or, Take the difference between the third pixel coordinates and the second pixel coordinates as the parameter calibration loss function; wherein, in the parameter calibration loss function, both the initial calibrated intrinsic parameters and the initial calibrated extrinsic parameters vary.

6. The method according to claim 4, wherein Based on the intrinsic parameter calibration loss function, the extrinsic parameter calibration loss function, and the parameter calibration loss function, jointly optimize the initial calibrated intrinsic parameters and the initial calibrated extrinsic parameters to obtain the calibrated intrinsic parameters of the imaging device and the calibrated extrinsic parameters between the radar and the imaging device, including: Perform weighted summation on the intrinsic parameter calibration loss function, the extrinsic parameter calibration loss function, and the parameter calibration loss function to obtain a joint calibration loss function; Based on the joint calibration loss function, obtain the calibrated intrinsic parameters of the imaging device and the calibrated extrinsic parameters between the radar and the imaging device.

7. The method according to claim 6, characterized in that Further include: In the joint calibration loss function, the weight of the parameter calibration loss function is more than ten times the weight value of the intrinsic parameter calibration loss function and / or the extrinsic parameter calibration loss function.

8. A sensor calibration device based on radar-vision fusion, characterized in that, Include: An acquisition unit, configured to acquire a calibration image captured by an imaging device in a to-be-calibrated radar-vision sensor for a calibration board and point cloud data measured by the radar for the calibration board; A determination unit, configured to determine the initial calibrated intrinsic parameters of the imaging device based on the calibration board and the calibration image, and determine the initial calibrated extrinsic parameters between the radar and the imaging device based on the point cloud data and the calibration image; A joint optimization unit, configured to jointly optimize the initial calibrated intrinsic parameters of the imaging device and the initial calibrated extrinsic parameters between the radar and the imaging device to obtain the calibrated intrinsic parameters of the imaging device and the calibrated extrinsic parameters between the radar and the imaging device.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images