Method, apparatus and device for calibrating extrinsic parameters between radar and camera, and storage medium

Abstract

The present application relates to the technical field of sensor extrinsic parameter calibration, and discloses a method, apparatus and device for calibrating extrinsic parameters between a radar and a camera, and a computer readable storage medium. In the method, an outdoor scenario containing a building serves as a control field, and a distant view observation mode is used for calibration, so that a calibration scenario is close to an actual working scenario; and ground truth information of a calibration site is fully utilized, so that the influence on robustness caused by a mismatch is reduced.

Description

External parameter calibration method, device, equipment and storage medium between radar and camera Technical Field

[0001] The embodiments of the present application relate to the technical field of sensor extrinsic parameter calibration, and specifically to a method, apparatus, device, and computer-readable storage medium for extrinsic parameter calibration between a radar and a camera. Background Art

[0002] Radar-camera fusion technology plays a key role in multi-sensor systems. Radar and cameras, as commonly used sensing devices, can provide rich information, but they are typically located in different locations. Therefore, extrinsic calibration is required to ensure their data can be accurately fused. Traditional calibration schemes rely heavily on calibration plates. However, due to the differences between calibration plates and actual application scenarios, extrinsic calibration suffers from poor accuracy and robustness.

[0003] Summary of the Invention

[0004] In view of the above problems, the embodiments of the present application provide a method, device, equipment and computer-readable storage medium for extrinsic parameter calibration between a radar and a camera, which are used to solve the problems of poor accuracy and poor robustness of extrinsic parameter calibration in the prior art.

[0005] According to one aspect of an embodiment of the present application, a method for calibrating external parameters between a radar and a camera is provided, which is applied to calibrate the external parameters between the radar and the camera in a preset control field, wherein the control field is an outdoor scene containing buildings, and the radar and the camera are used to perform distant view calibration in the control field. The method comprises: determining the coordinate relationship between feature points in the control field and pixel points corresponding to the feature points in the image formed by the camera based on the conversion relationship between the world coordinate system of the control field and the pixel coordinate system of the image formed by the camera, wherein the feature points include points on buildings in the control field; determining the association between the pixel points in the image formed by the camera and the control points in the control field based on the camera projection model and the camera intrinsic parameters, wherein, The number of control points is greater than the number of feature points; the coordinate relationship between the feature points in the control field and the pixel points corresponding to the feature points in the image formed by the camera is substituted into the correlation formula, and the extrinsic parameters of the camera relative to the control field are determined using the direct linear transformation method; based on the conversion relationship between the world coordinate system of the control field and the point cloud coordinate system of the radar, the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar is determined; based on the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar, the extrinsic parameters of the radar relative to the control field are determined; based on the extrinsic parameters of the camera relative to the control field and the extrinsic parameters of the radar relative to the control field, the extrinsic parameters between the radar and the camera are converted.

[0006] In an optional manner, a radar and a camera are used for multi-site observation calibration in a control field to obtain external parameters of the camera relative to the control field when located at each site and external parameters of the radar relative to the control field when located at each site; based on the external parameters of the camera relative to the control field and the external parameters of the radar relative to the control field, the external parameters between the radar and the camera are converted, including: optimizing the external parameters of the camera relative to the control field when located at each site by minimizing the reprojection error between multiple sites; optimizing the external parameters of the radar with respect to the control field when located at each site by minimizing the matching error of the same-name point clouds between multiple sites; and converting the external parameters between the radar and the camera based on the optimized external parameters of the camera relative to the control field when located at each site and the optimized external parameters of the radar relative to the control field when located at each site.

[0007] In an optional approach, the extrinsic parameters of the camera relative to the control field at each station are optimized by minimizing the reprojection error between multiple stations. The loss function of the reprojection error is as follows:

[0008] Among them, i represents the station, j represents the serial number of the pixel corresponding to the feature point in the image when the camera is located at each station, K represents the camera internal parameter, P ij With P w Represent the matching pixels and feature points respectively, R i and t i represents the pose of the camera relative to the control field when it is located at site i, where R i represents the rotation of the camera relative to the control field when it is located at site i, t i represents the translation of the camera relative to the control field when it is located at site i;

[0009] By minimizing the matching error of point clouds with the same name between multiple sites, the loss function of the matching error of point clouds with the same name is optimized in the external parameters of the control field when the radar is located at each site as follows:

[0010] Where m represents the station, n represents the serial number of the three-dimensional point corresponding to the feature point in the point cloud formed when the radar is located at each station, and P mn With P w Represent the matching 3D points and feature points, R m and t m represents the attitude of the radar relative to the control field when it is located at site m, where R m represents the rotation of the radar relative to the control field when it is located at site m, t m represents the translation of the radar relative to the control field when it is located at site m.

[0011] In an optional manner, the extrinsic parameters between the radar and the camera are converted based on the optimized extrinsic parameters of the camera relative to the control field when located at each station and the optimized extrinsic parameters of the radar relative to the control field when located at each station, including: converting the extrinsic parameters between the radar and the camera when located at each station based on the optimized extrinsic parameters of the camera relative to the control field when located at each station and the optimized extrinsic parameters of the radar relative to the control field when located at each station; projecting the radar when located at each station to the camera's perspective, using the extrinsic parameters between the radar and the camera when located at each station, matching the three-dimensional points in the point cloud formed by the projected radar with the corresponding pixel points in the image formed by the camera, and optimizing the extrinsic parameters between the radar and the camera by minimizing the reprojection error.

[0012] In an optional approach, the radar at each station is projected onto the camera's view. Using the extrinsic parameters between the radar and camera at each station, the 3D points in the projected radar point cloud are matched with the corresponding pixels in the camera image. The extrinsic parameters between the radar and camera are optimized by minimizing the reprojection error. The loss function for the reprojection error is as follows:

[0013] Among them, k represents the station, l represents the number of pixels in the camera image that matches the three-dimensional point in the radar point cloud at each station, K represents the camera internal parameter, and p kl With P kl Represent the matching pixels and 3D points, R k and t k represents the external parameters of the camera relative to the radar when it is located at site k, where R k represents the rotation of the camera relative to the radar at site k, t k represents the translation of the camera relative to the radar when located at site k.

[0014] In an optional manner, based on the conversion relationship between the world coordinate system of the control field and the pixel coordinate system of the image formed by the camera, the coordinate relationship between the feature points in the control field and the pixel points corresponding to the feature points in the image formed by the camera is determined, including: based on the conversion relationship between the world coordinate system of the control field and the projection coordinate system of the camera, determining the first coordinate relationship between the feature points in the control field and the projection points corresponding to the feature points in the camera; based on the conversion relationship between the projection coordinate system of the camera and the pixel coordinate system of the image formed by the camera, determining the second coordinate relationship between the projection points in the camera and the pixel points corresponding to the projection points in the image formed by the camera; based on the first coordinate relationship and the second coordinate relationship, determining the coordinate relationship between the feature points in the control field and the pixel points corresponding to the feature points in the image formed by the camera.

[0015] In an optional embodiment, the camera is a fisheye lens camera;

[0016] The first coordinate relationship between the feature point in the control field and the projection point corresponding to the feature point in the camera is as follows: B1=R·B2+t

[0017] Where B1 represents the coordinates of the projection point, B2 represents the coordinates of the feature point, R and T represent the coordinate transformation parameters between the feature point and the projection point, where R represents rotation and t represents translation;

[0018] Determining a second coordinate relationship between a projection point in the camera and a pixel point in the image formed by the camera corresponding to the projection point based on a conversion relationship between the projection coordinate system of the camera and the pixel coordinate system of the image formed by the camera includes:

[0019] Based on the pinhole projection principle, the coordinates of the feature point B2 (x, y, z) in the pixel coordinate system correspond to the coordinates of the pixel point B3 (u, v) as follows:

[0020] Converted to polar coordinate system:

[0021] r 2 =u 2 +v 2 ,

[0022] Based on the fisheye imaging model, the coordinates B1 (x', y') of the projection point under the fisheye lens can be obtained as follows: θ d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 )

[0023] Among them, k1, k2, k3 and k4 are all internal parameters of the fisheye lens;

[0024] The coordinates B1 (x', y') of the projection point are converted to the coordinates B3 (u, v) of the pixel point, and the second coordinate relationship is as follows: u = f x (x′)+c x , v = f y (y′)+c y

[0025] Among them, f x represents the focal length of the camera in the x direction, c x Indicates the coordinates of the principal point in the x-direction of the camera, f y Indicates the focal length of the camera in the y direction, c y Indicates the coordinates of the camera's principal point in the y direction;

[0026] The relationship between the pixel points in the camera image and the control points in the control field is determined based on the camera projection model and the camera intrinsic parameters, including:

[0027] Determine the camera intrinsic parameter matrix K as follows:

[0028] Based on the camera projection model and the camera intrinsic parameter matrix K, the relationship between the pixel points in the image formed by the camera and the control points in the control field can be obtained as follows:

[0029] Among them, Z c Indicates the Z-axis depth value of the pixel in the pixel coordinate system, R i and t i represents the pose of the camera relative to the control field when it is located at site i, where R i represents the rotation of the camera relative to the control field when it is located at site i, t i represents the translation of the camera relative to the control field when it is located at site i.

[0030] According to another aspect of an embodiment of the present application, a device for calibrating external parameters between a radar and a camera is provided, which is used to calibrate external parameters between the radar and the camera in a preset control field, wherein the control field is an outdoor scene containing buildings, and the radar and the camera are used to perform distant view calibration in the control field, and the device includes: a first coordinate relationship determination module, which is used to determine the coordinate relationship between feature points in the control field and pixel points corresponding to the feature points in the image formed by the camera based on the conversion relationship between the world coordinate system of the control field and the pixel coordinate system of the image formed by the camera, wherein the feature points include points on buildings in the control field; an association formula determination module, which is used to determine the association formula between pixel points in the image formed by the camera and control points in the control field based on the camera projection model and the camera intrinsic parameters, wherein the number of control points is greater than the number of feature points. the number of; a first extrinsic parameter determination module, used to substitute the coordinate relationship between the feature points in the control field and the pixel points corresponding to the feature points in the image formed by the camera into the correlation formula, and use the direct linear transformation method to determine the extrinsic parameters of the camera relative to the control field; a second coordinate relationship determination module, used to determine the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar based on the conversion relationship between the world coordinate system of the control field and the point cloud coordinate system of the radar; the second extrinsic parameter determination module, used to determine the extrinsic parameters of the radar relative to the control field according to the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar; a third extrinsic parameter determination module, used to convert the extrinsic parameters between the radar and the camera according to the extrinsic parameters of the camera relative to the control field and the extrinsic parameters of the radar relative to the control field.

[0031] According to another aspect of an embodiment of the present application, a radar-camera extrinsic parameter calibration device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store executable instructions, and the executable instructions enable the processor to execute the operations of the radar-camera extrinsic parameter calibration method as described in any one of the above items.

[0032] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores executable instructions. When the executable instructions are executed on an extrinsic parameter calibration device between a radar and a camera, the extrinsic parameter calibration device between the radar and the camera performs the operation of the extrinsic parameter calibration method between the radar and the camera as described in any one of the above items.

[0033] In the extrinsic parameter calibration method between radar and camera provided in the embodiment of the present application, an outdoor scene containing buildings is used as the control field, and calibration is performed by distant observation, so that the calibration scene is close to the actual working scene, and the true value information of the calibration site is fully utilized, reducing the robustness impact caused by incorrect matching. On this basis, the independent extrinsic parameters of the camera and radar are first solved, and then the extrinsic parameters between the radar and camera are obtained through conversion to improve the calibration accuracy.

[0034] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to more clearly understand the technical means of the embodiments of the present application, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0036] FIG1 is a schematic diagram of a flow chart of an extrinsic parameter calibration method between a radar and a camera according to an embodiment of the present application;

[0037] FIG2 is a schematic diagram of the sub-step flow of step 160 in FIG1 ;

[0038] FIG3 is a schematic diagram of the sub-step flow of step 163 in FIG2 ;

[0039] FIG4 is a schematic diagram of the sub-step flow of step 110 in FIG1 ;

[0040] FIG5 is a schematic diagram of the modular structure of an extrinsic parameter calibration device between a radar and a camera provided in an embodiment of the present application;

[0041] FIG6 is a schematic diagram of the structure of an external parameter calibration device between a radar and a camera provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0043] Multi-sensor perception systems play a crucial role in fields such as autonomous driving, robotic navigation, and intelligent transportation. Radar and cameras are key sensing devices within these systems. To achieve reliable detection, data from radars and cameras located in different locations must be accurately fused. External parameter calibration determines the accuracy and reliability of this data fusion.

[0044] Traditional extrinsic calibration methods between radar and camera generally obtain correlation information between the radar and camera by using a calibration plate of known geometric shape. This method has high requirements for the placement of the calibration plate and is highly dependent on the calibration plate. However, due to the differences between the calibration plate and the actual application environment, the calibration accuracy is poor and it is not well suited for dynamic environments.

[0045] Based on this, this application will use outdoor scenes containing buildings as control fields. On the basis of adopting distant observation, the external parameters of the camera in the control field are solved by matching the feature points on the building with the pixel points in the camera imaging. The external parameters of the radar in the control field are solved by matching the feature points on the building with the three-dimensional points in the point cloud formed by the radar, thereby realizing independent posture positioning and solution of the camera and radar. Finally, according to the external parameters of the camera and radar in the control field, the external parameters between the radar and the camera are converted. In the whole scheme, distant observation and the actual outdoor environment are used as calibration scenes, and feature points are selected from the buildings in the actual outdoor environment for matching. This can ensure that the calibration results are more adapted to the actual application scenarios and improve the robustness. At the same time, the external parameters of the radar and camera in the control field are first independently solved, and then the external parameters between the radar and the camera are obtained through conversion to ensure the accuracy of the calibration.

[0046] According to one aspect of an embodiment of the present application, a method for extrinsic calibration between a radar and a camera is provided. For details, please refer to FIG1 , which illustrates the process of the method for extrinsic calibration between a radar and a camera provided in an embodiment of the present application. This method can be performed by a radar and camera extrinsic calibration device, such as a computer or server. The method is applied to calibrate the extrinsic parameters between the radar and the camera in a predetermined control field, which is an outdoor scene containing buildings. The radar and the camera are used for long-range observation calibration in the control field. As shown in FIG1 , the method includes the following steps:

[0047] Step 110: Based on the conversion relationship between the world coordinate system of the control field and the pixel coordinate system of the image formed by the camera, determine the coordinate relationship between the feature points in the control field and the pixel points corresponding to the feature points in the image formed by the camera, wherein the feature points include points on the buildings in the control field.

[0048] Since the embodiment of the present application adopts long-range observation calibration, the predefined control field may include not only buildings but also large calibration plates, reflectors or other detectable feature objects. Accordingly, corresponding feature points may be set on these detectable feature objects to provide a reliable calibration reference, where the calibration points may be three-dimensional markers or other high-contrast objects.

[0049] In the actual use environment, collect radar and camera data, including the projection of characteristic objects in the control field into the radar and camera fields of view. During actual calibration, ensure that the camera and radar are accurately installed to collect enough data to cover the entire scene, and that the camera and radar simultaneously collect data from the control field.

[0050] In this step, the points on the buildings in the predefined control field are used as feature points to match the pixel points in the image to obtain the correspondence between the actual coordinate values ​​of the feature points and the coordinate values ​​of the pixel points. Since the calibration scene used is closer to the actual working scene, it is beneficial to improve the accuracy of subsequent calibration.

[0051] Step 120: Determine a correlation between pixel points in the image formed by the camera and control points in the control field based on the camera projection model and the camera intrinsic parameters, wherein the number of control points is greater than the number of feature points.

[0052] It should be noted that in this step, the control point is the set of all points in the control field, and the correlation formula between the pixel points and the control points obtained based on the camera model and the camera intrinsic parameters is the coordinate conversion formula between the control points and the pixel points calculated based on the camera intrinsic parameters and its projection imaging principle.

[0053] Step 130 , substitute the coordinate relationship between the feature points in the control field and the pixels corresponding to the feature points in the image formed by the camera into the correlation equation, and use the direct linear transformation method to determine the extrinsic parameters of the camera relative to the control field.

[0054] In this step, by substituting the actual coordinate values ​​of the corresponding feature points and the coordinate values ​​of the pixel points (that is, the coordinate relationship between the two) obtained in step 110 into the coordinate conversion formula (that is, the correlation formula) between the control points and the pixel points obtained in step 120, the direct linear transformation method can be used to obtain the posture of the camera relative to the control field, that is, the external parameters of the camera relative to the control field, specifically including rotation and translation, so as to preliminarily obtain the independent external parameters of the camera.

[0055] Step 140: Based on the conversion relationship between the world coordinate system of the control field and the point cloud coordinate system of the radar, determine the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar.

[0056] Step 150: Determine the external parameters of the radar relative to the control field based on the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar.

[0057] Steps 140 and 150 are used to determine the external parameters of the radar relative to the control field, which is similar to the above steps 110 to 130. However, since the coordinates between the camera and the control field are in a two-dimensional and three-dimensional relationship, and the coordinates between the radar and the control field are in a three-dimensional and three-dimensional relationship, in steps 140 and 150, it is only necessary to calculate and determine the external parameters of the radar relative to the control field based on the correspondence between the actual coordinate values ​​of the feature points in the control field and the coordinate values ​​of the three-dimensional points in the point cloud, thereby obtaining the independent external parameters of the radar.

[0058] Step 160: According to the extrinsic parameters of the camera relative to the control field and the extrinsic parameters of the radar relative to the control field, the extrinsic parameters between the radar and the camera are converted.

[0059] In this step, after obtaining the external parameters of the camera relative to the control field and the radar relative to the control field, a simple conversion can be used to obtain the external parameters between the radar and the camera. For example, when the external parameter of the camera relative to the control field is R a , t a , R a represents rotation, t a Represents translation, and the external parameter of the radar relative to the control field is R b , t b , which has the same meaning as the camera and will not be elaborated on here. Then the external parameter of the radar relative to the camera is R b R a -1 , t b -R b R a -1 t a .

[0060] In the extrinsic parameter calibration method between radar and camera provided in the embodiment of the present application, an outdoor scene containing buildings is used as the control field, and calibration is performed by distant observation, so that the calibration scene is close to the actual working scene, and the true value information of the calibration site is fully utilized, reducing the robustness impact caused by incorrect matching. On this basis, the independent extrinsic parameters of the camera and radar are first solved, and then the extrinsic parameters between the radar and camera are obtained through conversion to improve the calibration accuracy.

[0061] In order to further improve the calibration accuracy, the present application also proposes an embodiment in which radar and cameras are used to perform multi-site observation calibration in a control field. Multi-site observation refers to the same set of radar and cameras observing and imaging the control field at different positions and / or different angles. Based on this, the external parameters of the camera relative to the control field when it is located at each site and the external parameters of the radar relative to the control field when it is located at each site can be obtained in the above steps 130 and 150. For details, please refer to Figure 2, which shows the sub-step process of step 160. As shown in the figure, step 160 includes the following steps:

[0062] Step 161: Optimize the extrinsic parameters of the camera relative to the control field at each station by minimizing the reprojection error between multiple stations.

[0063] The images taken by the camera at each station have pixel points corresponding to the feature points in the control field. The reprojection between multiple stations refers to these pixel points corresponding to the feature points in multiple images. Based on these pixel points, the extrinsic parameters of multiple cameras relative to the control field can be determined. The calibration error can be determined by comparing these extrinsic parameters with each other. By minimizing the error, the extrinsic parameters of the camera relative to the control field at each station are optimized to improve the calculation accuracy.

[0064] In this step, the Levenberg-Marquardt method (LM method), Gauss-Newton method, or least squares method can be used to optimize the extrinsic parameters of the camera relative to the control field at each station. The reprojection error can be calculated using the following loss function:

[0065] Among them, i represents the station, j represents the serial number of the pixel corresponding to the feature point in the image when the camera is located at each station, K represents the camera internal parameter, P ij With P w Represent the matching pixels and feature points respectively, R i and t i represents the pose of the camera relative to the control field when it is located at site i, where R i represents the rotation of the camera relative to the control field when it is located at site i, t i represents the translation of the camera relative to the control field when it is located at site i.

[0066] Step 162: Optimize the extrinsic parameters of the radar relative to the control field when located at each station by minimizing the matching error of the point clouds with the same name between multiple stations.

[0067] Similarly, the point cloud formed by the radar scanning at each station contains three-dimensional points corresponding to the feature points in the control field. The point clouds with the same name between multiple stations refer to the part of three-dimensional points corresponding to the feature points in multiple point clouds. Based on this part of three-dimensional points, the external parameters of multiple radars relative to the control field can be determined, and then the error minimization optimization is performed through this part of the external parameters to improve the calculation accuracy of the external parameters of the radar relative to the control field when it is located at each station.

[0068] Similarly, the LM method can be used to optimize the external parameters of the radar relative to the control field at each station. The matching error of the point cloud with the same name can be calculated using the following loss function:

[0069] Where m represents the station, n represents the serial number of the three-dimensional point corresponding to the feature point in the point cloud formed when the radar is located at each station, and P mn With P w Represent the matching 3D points and feature points, R m and t m represents the attitude of the radar relative to the control field when it is located at site m, where R m represents the rotation of the radar relative to the control field when it is located at site m, t m represents the translation of the radar relative to the control field when it is located at site m.

[0070] Step 163: According to the optimized extrinsic parameters of the camera relative to the control field when located at each station and the optimized extrinsic parameters of the radar relative to the control field when located at each station, the extrinsic parameters between the radar and the camera are converted.

[0071] In this embodiment, the external parameters of multiple cameras relative to the control field and the external parameters of multiple radars relative to the control field are obtained by multi-site observation calibration. The external parameters of the multiple cameras relative to the control field are minimized to make the external parameters of the cameras relative to the control field at each site more accurate after optimization. The external parameters of the multiple radars relative to the control field are minimized to make the errors of the radars relative to the control field at each site more accurate after optimization. Finally, the external parameters of the cameras relative to the control field at each site and the external parameters of the radars relative to the control field at each site are converted to obtain external parameters between the radar and the camera with higher accuracy.

[0072] After optimizing the external parameters of the camera relative to the control field and the external parameters of the radar relative to the control field, in order to further improve the calibration accuracy, the present application also proposes an implementation method. Please refer to FIG3 for details. The figure shows the sub-step process of step 163. As shown in the figure, the above step 163 includes the following steps:

[0073] Step 1631: Based on the optimized extrinsic parameters of the camera relative to the control field at each station and the optimized extrinsic parameters of the radar relative to the control field at each station, the extrinsic parameters between the radar and the camera at each station are converted.

[0074] Step 1632: Project the radar at each station to the camera's view. Use the extrinsic parameters between the radar and the camera at each station to match the three-dimensional points in the projected point cloud with the corresponding pixel points in the camera image. By minimizing the reprojection error, the extrinsic parameters between the radar and the camera are optimized.

[0075] In this step, the reprojection error can use the following loss function:

[0076] Among them, k represents the station, l represents the number of pixels in the camera image that matches the three-dimensional point in the radar point cloud at each station, K represents the camera internal parameter, and p kl With P kl Represent the matching pixels and 3D points, R k and t k represents the external parameters of the camera relative to the radar when it is located at site k, where R k represents the rotation of the camera relative to the radar at site k, t k represents the translation of the camera relative to the radar when located at site k.

[0077] In this embodiment, the optimized extrinsic parameters of the camera relative to the control field when located at each station and the optimized extrinsic parameters of the radar relative to the control field when located at each station can be converted to obtain the extrinsic parameters between the radar and the camera when located at each station, that is, the extrinsic parameters between multiple radars and cameras are obtained. Based on this, the extrinsic parameters between multiple radars and cameras are also optimized by minimizing the errors, so that the final determined extrinsic parameters between the radar and the camera are more accurate.

[0078] For determining the coordinate relationship between the feature points in the control field and the pixels in the image formed by the camera, the present application further proposes an implementation method, specifically referring to FIG4 , which shows the sub-steps of step 110. As shown in the figure, step 110 includes the following steps:

[0079] Step 111: Based on the conversion relationship between the world coordinate system of the control field and the projection coordinate system of the camera, determine a first coordinate relationship between the feature points in the control field and the projection points corresponding to the feature points in the camera.

[0080] In this step, the projected coordinate system refers to the two-dimensional coordinate system on the image plane formed by the camera. The projection point corresponding to the feature point is the point on the image plane where the feature point is projected. The first coordinate relationship between the feature point and its corresponding projection point is the correspondence between the three-dimensional coordinate value and the two-dimensional coordinate value.

[0081] Step 112: Based on the conversion relationship between the camera's projection coordinate system and the pixel coordinate system of the image formed by the camera, determine a second coordinate relationship between the projection point in the camera and the pixel point corresponding to the projection point in the image formed by the camera.

[0082] In this step, the pixel coordinate system refers to the coordinates in the image data output by the camera after imaging. The pixel corresponding to the projection point is the pixel corresponding to the image data formed by the projection point on the image plane. Both the image plane and the image data are two-dimensional planes, so the second coordinate relationship between the projection point and the pixel point is the correspondence between two-dimensional coordinate values.

[0083] Step 113: Determine the coordinate relationship between the feature point in the control field and the pixel points corresponding to the feature point in the image formed by the camera based on the first coordinate relationship and the second coordinate relationship.

[0084] In the above steps 111 and 112, the first coordinate relationship between the feature point and the projection point, and the second coordinate relationship between the projection point and the pixel point are obtained respectively. By combining the first coordinate relationship and the second coordinate relationship, the coordinate relationship between the feature point and the pixel point can be determined.

[0085] Furthermore, in some embodiments, the camera is a fisheye lens camera. The fisheye lens has a very large viewing angle and is very suitable for the long-range observation of the control field of the outdoor scene in this application. It is conducive to the camera to capture and image the control field in all directions, so as to better perform feature point recognition and matching. For the solution using a fisheye lens, the first coordinate relationship determined in the above step 111 is as follows: B1 = R·B2 + t (4)

[0086] Among them, B1 represents the coordinates of the projection point, B2 represents the coordinates of the feature point, R and T represent the coordinate transformation relationship parameters between the feature point and the projection point, where R represents rotation and t represents translation.

[0087] The above step 112 includes the following steps:

[0088] Step 1121: Based on the pinhole projection principle, the coordinates B2 (x, y, z) of the feature point and the corresponding pixel coordinates B3 (u, v) in the pixel coordinate system are obtained as follows:

[0089] Converted to polar coordinate system:

[0090] r2 =u 2 +v 2 ,

[0091] Based on the fisheye imaging model, the coordinates B1 (x', y') of the projection point under the fisheye lens can be obtained as follows: θ d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 ) (8)

[0092] Among them, k1, k2, k3 and k4 are all internal parameters of the fisheye lens.

[0093] The coordinates B1 (x', y') of the projection point are converted to the coordinates B3 (u, v) of the pixel point, and the second coordinate relationship is as follows: u = f x (x′)+c x , v = f y (y′)+c y (9)

[0094] Among them, f x represents the focal length of the camera in the x direction, c x Indicates the coordinates of the principal point in the x-direction of the camera, f y Indicates the focal length of the camera in the y direction, c y Indicates the coordinates of the camera's principal point in the y direction.

[0095] In step 113, by combining the above equations (4) and (9), the corresponding relationship between the coordinate values ​​of the feature points and the coordinate values ​​of the pixel points, that is, the coordinate relationship between the two, can be obtained.

[0096] Step 120 includes the following steps:

[0097] Determine the camera intrinsic parameter matrix K as follows:

[0098] Based on the camera projection model and the camera intrinsic parameter matrix K, the relationship between the pixel points in the image formed by the camera and the control points in the control field can be obtained as follows:

[0099] Among them, Z c Indicates the Z-axis depth value of the pixel in the pixel coordinate system, R i and t i represents the pose of the camera relative to the control field when it is located at site i, where R i represents the rotation of the camera relative to the control field when it is located at site i, t irepresents the translation of the camera relative to the control field when it is located at site i.

[0100] In step 130, the coordinate values ​​of the corresponding feature points and the pixel points obtained in step 113 are substituted into equation (11) in step 120 to obtain R i and t i The specific value of , that is, the posture of the camera relative to the control field when it is located at each station (external parameter).

[0101] Through the above scheme, the external parameters of the camera relative to the control field when it is located at each site can be obtained more accurately, providing accurate data support for the subsequent determination of the external parameters between the radar and the camera.

[0102] In some embodiments, the following steps may be further included after step 160:

[0103] Step 170: Verify the extrinsic calibration results between the radar and the camera using another control field or real scene.

[0104] By verifying the external parameters between the calibrated radar and camera in another scenario, the accuracy and reliability of the calibration results can be quickly determined.

[0105] According to another aspect of the embodiment of the present application, a device for calibrating external parameters between a radar and a camera is also provided. For details, please refer to FIG5 , which shows the modular structure of the device. The device is used to calibrate the external parameters between the radar and the camera in a preset control field. The control field is an outdoor scene containing buildings. The radar and the camera are used to perform distant view calibration in the control field. The device 200 for calibrating external parameters between the radar and the camera includes: a first coordinate relationship determination module 210, which is used to determine the coordinate relationship between the feature points in the control field and the pixel coordinate system of the image formed by the camera based on the conversion relationship between the world coordinate system of the control field and the pixel coordinate system of the image formed by the camera, wherein the feature points include points on the buildings in the control field; an association determination module 220, which is used to determine the association between the pixel points in the image formed by the camera and the control points in the control field based on the camera projection model and the camera intrinsic parameters, wherein the number of control points is greater than the number of feature points; a first external parameter determination module 230, which is used to determine the coordinate relationship between the feature points in the control field and the pixel coordinate system of the image formed by the camera corresponding to the feature points. The coordinate relationship between the elements is substituted into the correlation formula, and the direct linear transformation method is used to determine the external parameters of the camera relative to the control field; the second coordinate relationship determination module 240 is used to determine the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar based on the conversion relationship between the world coordinate system of the control field and the point cloud coordinate system of the radar; the second external parameter determination module 250 is used to determine the external parameters of the radar relative to the control field based on the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar; the third external parameter determination module 260 is used to convert the external parameters between the radar and the camera based on the external parameters of the camera relative to the control field and the external parameters of the radar relative to the control field.

[0106] In some embodiments, the radar and the camera are used to perform multi-site observation calibration in the control field to obtain the external parameters of the camera relative to the control field when it is located at each site and the external parameters of the radar relative to the control field when it is located at each site; according to the external parameters of the camera relative to the control field and the external parameters of the radar relative to the control field, the external parameters between the radar and the camera are converted, including: optimizing the external parameters of the camera relative to the control field when it is located at each site by minimizing the reprojection error between multiple sites; optimizing the external parameters of the radar with respect to the control field when it is located at each site by minimizing the matching error of the same-name point clouds between multiple sites; converting the external parameters between the radar and the camera according to the optimized external parameters of the camera relative to the control field when it is located at each site and the optimized external parameters of the radar relative to the control field when it is located at each site.

[0107] In some embodiments, by minimizing the reprojection error between multiple sites, the extrinsic parameters of the camera relative to the control field at each site are optimized. The loss function of the reprojection error is as follows:

[0108] Among them, i represents the station, j represents the pixel corresponding to the feature point in the image when the camera is located at each station, K represents the camera internal parameter, P ij With P w Represent the matching pixels and feature points respectively, R i and t i represents the pose of the camera relative to the control field when it is located at site i, where R i represents the rotation of the camera relative to the control field when it is located at site i, t i represents the translation of the camera relative to the control field when it is located at site i;

[0109] By minimizing the matching error of point clouds with the same name between multiple sites, the loss function of the matching error of point clouds with the same name is optimized in the external parameters of the control field when the radar is located at each site as follows:

[0110] Where m represents the station, n represents the three-dimensional point corresponding to the feature point in the point cloud formed when the radar is located at each station, and P mn With P w Represent the matching 3D points and feature points, R m and t m represents the attitude of the radar relative to the control field when it is located at site m, where R m represents the rotation of the radar relative to the control field when it is located at site m, t m represents the translation of the radar relative to the control field when it is located at site m.

[0111] In some embodiments, the extrinsic parameters between the radar and the camera are converted based on the optimized extrinsic parameters of the camera relative to the control field when located at each station and the optimized extrinsic parameters of the radar relative to the control field when located at each station, including: converting the extrinsic parameters between the radar and the camera when located at each station based on the optimized extrinsic parameters of the camera relative to the control field when located at each station and the optimized extrinsic parameters of the radar relative to the control field when located at each station; projecting the radar at each station to the camera's perspective, using the extrinsic parameters between the radar and the camera when located at each station, matching the three-dimensional points in the point cloud formed by the projected radar with the corresponding pixel points in the image formed by the camera, and optimizing the extrinsic parameters between the radar and the camera by minimizing the reprojection error.

[0112] In some embodiments, the radar at each station is projected onto the camera's field of view. The extrinsic parameters between the radar and the camera at each station are used to match the three-dimensional points in the projected radar point cloud with the corresponding pixels in the camera image. The extrinsic parameters between the radar and the camera are optimized by minimizing the reprojection error. The loss function for the reprojection error is as follows:

[0113] Among them, k represents the station, l represents the number of pixels in the camera image that matches the three-dimensional point in the radar point cloud at each station, K represents the camera internal parameter, and p kl With P kl Represent the matching pixels and 3D points, R k and t k represents the external parameters of the camera relative to the radar when it is located at site k, where R k represents the rotation of the camera relative to the radar at site k, t k represents the translation of the camera relative to the radar when located at site k.

[0114] In some embodiments, based on the conversion relationship between the world coordinate system of the control field and the pixel coordinate system of the image formed by the camera, the coordinate relationship between the feature points in the control field and the pixel points corresponding to the feature points in the image formed by the camera is determined, including: based on the conversion relationship between the world coordinate system of the control field and the projection coordinate system of the camera, determining the first coordinate relationship between the feature points in the control field and the projection points corresponding to the feature points in the camera; based on the conversion relationship between the projection coordinate system of the camera and the pixel coordinate system of the image formed by the camera, determining the second coordinate relationship between the projection points in the camera and the pixel points corresponding to the projection points in the image formed by the camera; based on the first coordinate relationship and the second coordinate relationship, determining the coordinate relationship between the feature points in the control field and the pixel points corresponding to the feature points in the image formed by the camera.

[0115] In some embodiments, the camera is a fisheye lens camera;

[0116] The first coordinate relationship between the feature point in the control field and the projection point corresponding to the feature point in the camera is as follows: B1=R·B2+t

[0117] Where B1 represents the coordinates of the projection point, B2 represents the coordinates of the feature point, R and T represent the coordinate transformation parameters between the feature point and the projection point, where R represents rotation and t represents translation;

[0118] Determining a second coordinate relationship between a projection point in the camera and a pixel point in the image formed by the camera corresponding to the projection point based on a conversion relationship between the projection coordinate system of the camera and the pixel coordinate system of the image formed by the camera includes:

[0119] Based on the pinhole projection principle, the coordinates of the feature point B2 (x, y, z) in the pixel coordinate system correspond to the coordinates of the pixel point B3 (u, v) as follows:

[0120] Converted to polar coordinate system:

[0121] r 2 =u2 +v 2 ,

[0122] Based on the fisheye imaging model, the coordinates B1 (x', y') of the projection point under the fisheye lens can be obtained as follows: θ d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 )

[0123] Among them, k1, k2, k3 and k4 are all internal parameters of the fisheye lens;

[0124] The coordinates B1 (x', y') of the projection point are converted to the coordinates B3 (u, v) of the pixel point, and the second coordinate relationship is as follows: u = f x (x′)+c x , v = f y (y′)+c y

[0125] Among them, f x represents the focal length of the camera in the x direction, c x Indicates the coordinates of the principal point in the x-direction of the camera, f y Indicates the focal length of the camera in the y direction, c y Indicates the coordinates of the camera's principal point in the y direction;

[0126] The relationship between the pixel points in the camera image and the control points in the control field is determined based on the camera projection model and the camera intrinsic parameters, including:

[0127] Determine the camera intrinsic parameter matrix K as follows:

[0128] Based on the camera projection model and the camera intrinsic parameter matrix K, the relationship between the pixel points in the image formed by the camera and the control points in the control field can be obtained as follows:

[0129] Among them, Z c Indicates the Z-axis depth value of the pixel in the pixel coordinate system, R i and t i represents the pose of the camera relative to the control field when it is located at site i, where R i represents the rotation of the camera relative to the control field when it is located at site i, t i represents the translation of the camera relative to the control field when it is located at site i.

[0130] According to another aspect of the embodiments of the present application, an external parameter calibration device between a radar and a camera is also provided. Please refer to Figure 6 for details, which shows a structural schematic diagram of the calibration device. The specific embodiments of the present application do not limit the specific implementation of the external parameter calibration device between the radar and the camera.

[0131] As shown in FIG6 , the external parameter calibration device between the radar and the camera includes a processor 302 , a communications interface 304 , a memory 306 , and a communication bus 308 .

[0132] Processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308. Communication interface 304 is used to communicate with other devices, such as clients or other server network elements. Processor 302 is used to execute program 310, which may specifically perform the steps described in the embodiment of the radar-camera extrinsic calibration method.

[0133] Specifically, the program 310 may include program code including computer-executable instructions.

[0134] Processor 302 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the radar-camera extrinsic calibration device may be of the same type, such as one or more CPUs, or may be of different types, such as one or more CPUs and one or more ASICs.

[0135] The memory 306 is used to store the program 310. The memory 306 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0136] Program 310 may be specifically invoked by processor 302 to cause the extrinsic calibration device between the radar and the camera to perform the following operations:

[0137] Determine, based on a conversion relationship between a world coordinate system of the control field and a pixel coordinate system of an image formed by the camera, a coordinate relationship between feature points in the control field and pixels corresponding to the feature points in the image formed by the camera, wherein the feature points include points on buildings in the control field;

[0138] Determine, based on the camera projection model and the camera intrinsic parameters, a correlation between pixel points in the image formed by the camera and control points in the control field, wherein the number of control points is greater than the number of feature points;

[0139] Substitute the coordinate relationship between the feature points in the control field and the pixels corresponding to the feature points in the camera image into the correlation equation, and use the direct linear transformation method to determine the extrinsic parameters of the camera relative to the control field;

[0140] Based on the conversion relationship between the world coordinate system of the control field and the point cloud coordinate system of the radar, the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar is determined;

[0141] Determine the external parameters of the radar relative to the control field based on the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar;

[0142] According to the external parameters of the camera relative to the control field and the external parameters of the radar relative to the control field, the external parameters between the radar and the camera are converted.

[0143] An embodiment of the present application further provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed on an extrinsic parameter calibration device between a radar and a camera, the extrinsic parameter calibration device between the radar and the camera executes the extrinsic parameter calibration method between the radar and the camera in any of the above method embodiments.

[0144] The executable instructions can be used to enable the external parameter calibration device between the radar and the camera to perform the following operations:

[0145] Determine, based on a conversion relationship between a world coordinate system of the control field and a pixel coordinate system of an image formed by the camera, a coordinate relationship between feature points in the control field and pixels corresponding to the feature points in the image formed by the camera, wherein the feature points include points on buildings in the control field;

[0146] Determine, based on the camera projection model and the camera intrinsic parameters, a correlation between pixel points in the image formed by the camera and control points in the control field, wherein the number of control points is greater than the number of feature points;

[0147] Substitute the coordinate relationship between the feature points in the control field and the pixels corresponding to the feature points in the camera image into the correlation equation, and use the direct linear transformation method to determine the extrinsic parameters of the camera relative to the control field;

[0148] Based on the conversion relationship between the world coordinate system of the control field and the point cloud coordinate system of the radar, the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar is determined;

[0149] Determine the external parameters of the radar relative to the control field based on the coordinate relationship between the feature points in the control field and the three-dimensional points corresponding to the feature points in the point cloud formed by the radar;

[0150] According to the external parameters of the camera relative to the control field and the external parameters of the radar relative to the control field, the external parameters between the radar and the camera are converted.

[0151] An embodiment of the present application provides a computer program that can be called by a processor to enable an extrinsic parameter calibration device between a radar and a camera to perform the extrinsic parameter calibration method between a radar and a camera in any of the above method embodiments.

[0152] The algorithm or demonstration provided here are not inherently relevant to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the present application.

[0153] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0154] Similarly, it should be understood that in order to streamline the present application and assist in understanding one or more of the various aspects of the invention, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof.

[0155] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and can be divided into multiple submodules or subunits or subassemblies. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying abstract and drawings) and all processes or units of any method or device disclosed in this manner can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying abstract and drawings) can be replaced by alternative features providing the same, equivalent or similar purpose.

[0156] The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising a number of distinct elements and by means of a suitably programmed computer. The use of the words first, second, and third, etc., does not denote any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order in which they are performed.

Claims

1. A method for calibrating external parameters between a radar and a camera, applied to calibrate external parameters between the radar and the camera in a preset control field, characterized in that: The control field is an outdoor scene including buildings, the radar and the camera are used to perform distant view calibration in the control field, and the method includes: Determine, based on a conversion relationship between a world coordinate system of the control field and a pixel coordinate system of the image formed by the camera, a coordinate relationship between a feature point in the control field and a pixel point in the image formed by the camera corresponding to the feature point, wherein the feature point includes a point on a building in the control field; Determine, based on a camera projection model and camera internal parameters, a correlation formula between pixel points in the image formed by the camera and control points in the control field, wherein the number of the control points is greater than the number of the feature points; Substituting the coordinate relationship between the feature point in the control field and the pixel point corresponding to the feature point in the image formed by the camera into the correlation formula, and using the direct linear transformation method to determine the external parameter of the camera relative to the control field; Determine, based on a conversion relationship between a world coordinate system of the control field and a point cloud coordinate system of the radar, a coordinate relationship between the feature point in the control field and a three-dimensional point corresponding to the feature point in the point cloud formed by the radar; Determine the external parameters of the radar relative to the control field according to the coordinate relationship between the feature point in the control field and the three-dimensional point corresponding to the feature point in the point cloud formed by the radar; According to the external parameters of the camera relative to the control field and the external parameters of the radar relative to the control field, the external parameters between the radar and the camera are converted.

2. The external parameter calibration method between radar and camera according to claim 1, characterized in that: The radar and the camera are used to perform multi-site observation calibration in the control field to obtain external parameters of the camera relative to the control field when located at each site and external parameters of the radar relative to the control field when located at each site; The converting the external parameters between the radar and the camera according to the external parameters of the camera relative to the control field and the external parameters of the radar relative to the control field includes: By minimizing the reprojection error between multiple sites, optimizing the external parameters of the camera relative to the control field when located at each site; By minimizing the matching error of the same-name point clouds between multiple sites, optimizing the external parameters of the radar for the control field when located at each site; According to the optimized external parameters of the camera relative to the control field when located at each station and the optimized external parameters of the radar relative to the control field when located at each station, the external parameters between the radar and the camera are converted.

3. The external parameter calibration method between radar and camera according to claim 2, characterized in that: By minimizing the reprojection error between multiple sites, the loss function of the reprojection error in optimizing the external parameters of the camera relative to the control field when located at each site is as follows: Where i represents the station, j represents the serial number of the pixel corresponding to the feature point in the image formed when the camera is located at each station, K represents the camera internal parameter, P ij With P w Respectively represent the matching pixels and feature points, R i and t i represents the attitude of the camera relative to the control field when it is located at site i, where R i represents the rotation of the camera relative to the control field when it is located at site i, t i represents the translation of the camera relative to the control field when it is located at site i; By minimizing the matching error of the same-name point clouds between multiple sites, the loss function of the same-name point cloud matching error in optimizing the external parameters of the radar for the control field when located at each site is as follows: Where m represents the station, n represents the serial number of the three-dimensional point corresponding to the feature point in the point cloud formed when the radar is located at each station, and P mn With P w Respectively represent the matching 3D points and feature points, R m and t m represents the attitude of the radar relative to the control field when it is located at site m, where R m represents the rotation of the radar relative to the control field when it is located at site m, t m represents the translation of the radar relative to the control field when it is located at site m.

4. The external parameter calibration method between radar and camera according to claim 2 or 3, characterized in that: The step of converting the optimized external parameters of the camera relative to the control field when located at each station and the optimized external parameters of the radar relative to the control field when located at each station to obtain the external parameters between the radar and the camera includes: According to the optimized external parameters of the camera relative to the control field when located at each station and the optimized external parameters of the radar relative to the control field when located at each station, convert the external parameters between the radar and the camera when located at each station; The radar at each station is projected to the camera's viewing angle. The extrinsic parameters between the radar and the camera at each station are used to match the three-dimensional points in the point cloud formed by the projected radar with the corresponding pixel points in the image formed by the camera. The extrinsic parameters between the radar and the camera are optimized by minimizing the reprojection error.

5. The external parameter calibration method between radar and camera according to claim 4, characterized in that: The radar at each station is projected to the camera's viewing angle, and the extrinsic parameters between the radar and the camera at each station are used to match the three-dimensional points in the point cloud formed by the projected radar with the corresponding pixel points in the image formed by the camera. The extrinsic parameters between the radar and the camera are optimized by minimizing the reprojection error. The loss function of the reprojection error is as follows: Where k represents the station, l represents the number of pixels in the camera image that matches the three-dimensional point in the radar point cloud at each station, K represents the camera internal parameter, and p kl With P kl Respectively represent the matching pixels and 3D points, R k and t k represents the external parameters of the camera relative to the radar when located at site k, where R k represents the rotation of the camera relative to the radar at station k, t k represents the translation of the camera relative to the radar when located at station k.

6. The external parameter calibration method between radar and camera according to claim 2, characterized in that: The determining, based on the conversion relationship between the world coordinate system of the control field and the pixel coordinate system of the image formed by the camera, the coordinate relationship between the feature point in the control field and the pixel point corresponding to the feature point in the image formed by the camera comprises: Determine a first coordinate relationship between a feature point in the control field and a projection point in the camera corresponding to the feature point based on a conversion relationship between a world coordinate system of the control field and a projection coordinate system of the camera; Determining a second coordinate relationship between a projection point in the camera and a pixel point in the image formed by the camera corresponding to the projection point based on a conversion relationship between a projection coordinate system of the camera and a pixel coordinate system of the image formed by the camera; The coordinate relationship between the feature point in the control field and the pixel point corresponding to the feature point in the image formed by the camera is determined according to the first coordinate relationship and the second coordinate relationship.

7. The external parameter calibration method between radar and camera according to claim 6, characterized in that: The camera is a fisheye lens camera; The first coordinate relationship between the feature point in the control field and the projection point corresponding to the feature point in the camera is as follows: B1=R·B2+t Where B1 represents the coordinates of the projection point, B2 represents the coordinates of the feature point, R and T represent the coordinate transformation relationship parameters between the feature point and the projection point, where R represents rotation and t represents translation; The determining, based on a conversion relationship between a projection coordinate system of the camera and a pixel coordinate system of an image formed by the camera, a second coordinate relationship between a projection point in the camera and a pixel point in the image formed by the camera corresponding to the projection point comprises: Based on the pinhole projection principle, the coordinates B2 (x, y, z) of the feature point correspond to the coordinates B3 (u, v) of the pixel point in the pixel coordinate system as follows: Converted to polar coordinate system: Based on the fisheye imaging model, the coordinates B1 (x', y') of the projection point under the fisheye lens are as follows: i d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 ) Among them, k1, k2, k3 and k4 are the internal parameters of the fisheye lens; Convert the coordinates B1 (x', y') of the projection point to the coordinates B3 (u, v) of the pixel point, and the second coordinate relationship is as follows: u=f x (x′)+c x ,v=f y (y′)+c y Among them, f x represents the focal length of the camera in the x direction, c x Indicates the coordinates of the principal point in the x direction of the camera, f y Indicates the focal length of the camera in the y direction, c y Indicates the coordinates of the principal point of the camera in the y direction; The determining of the correlation between the pixel points in the image formed by the camera and the control points in the control field based on the camera projection model and the camera internal parameters includes: Determine the camera intrinsic parameter matrix K as follows: Based on the camera projection model and the camera intrinsic parameter matrix K, the relationship between the pixel points in the image formed by the camera and the control points in the control field can be obtained as follows: Among them, Z c Represents the Z-axis depth value of the pixel in the pixel coordinate system, R i and t i represents the attitude of the camera relative to the control field when it is located at site i, where R i represents the rotation of the camera relative to the control field when it is located at site i, t i Represents the translation of the camera relative to the control field when it is located at site i.

8. A device for calibrating external parameters between a radar and a camera, used for calibrating external parameters between the radar and the camera in a preset control field, characterized in that: The control field is an outdoor scene including buildings, the radar and the camera are used to perform distant view calibration in the control field, and the device includes: A first coordinate relationship determination module, used to determine the coordinate relationship between the feature points in the control field and the pixel points in the image formed by the camera corresponding to the feature points based on the conversion relationship between the world coordinate system of the control field and the pixel coordinate system of the image formed by the camera, wherein the feature points include points on the buildings in the control field; A correlation determination module, used to determine the correlation between the pixel points in the image formed by the camera and the control points in the control field based on the camera projection model and the camera internal parameters, wherein the number of the control points is greater than the number of the feature points; A first external parameter determination module is used to substitute the coordinate relationship between the feature point in the control field and the pixel point corresponding to the feature point in the image formed by the camera into the association formula, and determine the external parameter of the camera relative to the control field by using a direct linear transformation method; The second coordinate relationship determination module is used to determine the characteristic point in the control field and the point cloud coordinate system of the radar based on the conversion relationship between the world coordinate system of the control field and the point cloud coordinate system of the radar. The coordinate relationship between the three-dimensional points corresponding to the feature points in the formed point cloud; A second external parameter determination module, used to determine the external parameters of the radar relative to the control field according to the coordinate relationship between the feature point in the control field and the three-dimensional point corresponding to the feature point in the point cloud formed by the radar; The third external parameter determination module is used to convert the external parameters between the radar and the camera according to the external parameters of the camera relative to the control field and the external parameters of the radar relative to the control field.

9. An external parameter calibration device between a radar and a camera, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store executable instructions, and the executable instructions enable the processor to perform the operation of the external parameter calibration method between the radar and the camera as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores executable instructions, and when the executable instructions are executed on the external parameter calibration device between the radar and the camera, the external parameter calibration device between the radar and the camera performs the operation of the external parameter calibration method between the radar and the camera as described in any one of claims 1 to 7.

Images