Radar camera external parameter calibration method and system based on normalized information distance

Abstract

The invention relates to a radar camera external parameter calibration method and system based on a normalized information distance, and the method comprises the steps: U1, enabling a vehicle to run on a road, obtaining the image data information of the road in real time based on a vehicle-mounted camera, obtaining the data information of the point cloud of the road in real time based on a vehicle-mounted laser radar, and carrying out the data synchronization processing, the image data information and the point cloud data information of the road after synchronization processing are obtained; and U2, based on the image data information of the road after synchronization processing, adopting a roadside feature recognition algorithm improved based on the yov5 to recognize the image of the vehicle running to the roundabout area, and obtaining the data information of the image of the roundabout area. According to the invention, external parameter calibration can be carried out under the condition of no visual field crossing or small visual field crossing, a specially-made calibration target position is not needed, the cost is reduced, and the calibration efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor extrinsic parameter calibration, and in particular to a radar camera extrinsic parameter calibration method and system based on normalized information distance. Background Art

[0002] Sensor calibration, as the core foundational technology of intelligent connected vehicle perception systems, undertakes the key task of establishing a spatial and temporal coordinate system for multi-source heterogeneous sensors. Its accuracy directly affects the subsequent sensor fusion effect and the reliability of the decision-making system. The current technical solution mainly includes a three-level system of production line calibration, after-sales calibration, and online calibration. The current industry generally faces two technical bottlenecks:

[0003] Hardware configuration constraints: Due to cost constraints, mechanical radars are expensive, and solid-state lidars are now mostly used instead. However, their FOV (field of view) is reduced to one-third that of mechanical radars, significantly reducing the spatial overlap with surround-view cameras. In a typical vehicle layout, approximately 42% of sensor combinations have no overlap in field of view or an overlap of less than 5%, increasing the risk of failure of traditional calibration methods.

[0004] Environmental adaptability defects: Existing online calibration technologies mostly rely on special calibration targets (such as checkerboards), which cannot be implemented in natural road scenes due to the lack of artificial calibration objects. Summary of the Invention

[0005] In view of the above problems, the present invention provides a radar camera extrinsic parameter calibration method and system based on normalized information distance. Not only can extrinsic parameter calibration be performed when there is no field of view intersection or a small field of view intersection, but also no special calibration target is required, thus reducing costs and improving calibration efficiency.

[0006] In order to achieve the above-mentioned and other related purposes, the present invention provides the following technical solutions:

[0007] A radar camera extrinsic parameter calibration method based on normalized information distance, the method comprising:

[0008] U1. A vehicle traveling on a road acquires real-time road image data using its onboard camera and point cloud data using its onboard lidar. These data are then synchronized and processed to produce synchronized road image data and point cloud data.

[0009] U2 based on the synchronously processed road image data information, the use of improved roadside feature recognition algorithm based on yolov5 to identify the image of the vehicle traveling to the roundabout area, to obtain the image data information of the roundabout area;

[0010] U3. Based on the image data information of the roundabout area and the point cloud data information of the synchronously processed road, a SLAM-based common view construction algorithm is used to characterize the dense point cloud within the FOV of each frame image to obtain the dense point cloud data information within the FOV of each frame image;

[0011] U4. Based on the data information of the dense point cloud within the FOV of each frame of the image, a cross-modal NID registration algorithm is used to calibrate the external parameters of the vehicle's radar and camera to obtain data information of the extrinsic parameters of the calibrated radar and camera.

[0012] Furthermore, in step U2, the use of the improved roadside feature recognition algorithm based on yolov5 to recognize the image and point cloud of the vehicle traveling to the roundabout area includes:

[0013] U21. Based on the synchronized image data information of the road, construct a road image dataset;

[0014] U22 inputs the road image dataset into the yolov5 target detection model, characterizes the target detection feature points of the roundabout area image, and obtains data information of the target detection feature points of the roundabout area image;

[0015] U23. Based on the data information of the target detection feature points in the roundabout area, establish an image recognition function Q of the roundabout area,

[0016] ,

[0017] Among them, x is the data information of the target detection feature points in the roundabout area, ɑ1, ɑ2, and ɑ3 are the image recognition factors of the roundabout area, and the image of the vehicle driving into the roundabout area is recognized to obtain the data information of the image of the roundabout area.

[0018] Furthermore, the image recognition factors ɑ1, ɑ2 and ɑ3 of the island area are:

[0019] ,

[0020] ,

[0021] ,

[0022] Among them, x is the data information of the target detection feature points in the island area.

[0023] Furthermore, in step U3, the characterization of the dense point cloud within the FOV of each frame image using the SLAM-based common view construction algorithm includes:

[0024] U31. Based on the synchronously processed road point cloud data information, the SLAM algorithm is used to estimate the vehicle's trajectory to obtain the estimated vehicle trajectory data information;

[0025] U32. Based on the data information of the estimated vehicle's driving trajectory, adaptively voxelize the point cloud and construct an optimization function W.

[0026] ,

[0027] Among them, n l is the normal vector of the plane, q l is a point in the plane, N l is the total number of points in the plane, l is the plane of point cloud adaptive voxelization, n l T is the transpose of the plane normal vector, r is the coordinate of the point cloud under all poses of the target radar projected to the first frame pose, optimize the distance from the point to the plane in each voxel, and output the optimized vehicle trajectory;

[0028] U33. Based on the data information of the optimized vehicle's driving trajectory and the image of the roundabout area, the initial external parameters of the camera are obtained, the point cloud is converted to the radar coordinate system corresponding to each frame pose, and then reprojected to the camera coordinate system, the point cloud is traversed, and the point cloud within the camera FOV is saved to obtain a dense point cloud within the FOV of each frame image.

[0029] Furthermore, the adaptive voxelization of the point cloud is to repeatedly voxelize each frame of the point cloud after conversion. The voxelization strategy is to save the voxel if all the points in the voxel are located in the same plane, otherwise the voxel is further decomposed into 8 8-voids until the minimum size is reached, and then determine whether all the points in the voxel are in the same plane. By calculating the co-defense difference matrix of all the points in the voxel, the ratio of the maximum eigenvalue to the minimum eigenvalue is determined. If it is greater than a certain value, it can be determined to be in the same plane, otherwise it is not in the same plane.

[0030] Furthermore, in step U4, the extrinsic parameters of the vehicle's radar and camera are calibrated using the cross-modal NID registration algorithm, including:

[0031] U41. Based on the dense point cloud data within the FOV of each frame image, use the deep learning models DeepLab and RangeNet++ to infer the point cloud and image, respectively, and output the corresponding 2D pixel points and 3D point semantic labels;

[0032] U42. Based on the corresponding 2D pixel points and 3D point semantic labels, establish the NID function f of the point cloud and the image,

[0033] ,

[0034] Where X is the corresponding 2D pixel point, Y is the corresponding 3D point semantic label, H(X,Y) is the joint entropy of the two sets of discrete variables, G(X;Y) is the mutual information of the two discrete variables, and the NID of the two sets of discrete random variables of the point cloud and image semantic labels is represented to obtain the data information of the NID of the two sets of discrete random variables of the point cloud and image semantic labels;

[0035] U43. Based on the NID data information of the two sets of discrete random variables of the point cloud and image semantic labels, establish the external parameter calibration function S of the radar and camera,

[0036] ,

[0037] in, is the external parameter of the radar and camera, N is N frames of data, N is a positive integer, s is the corresponding cropped point cloud under each frame pose, g cam (s) The point cloud is reprojected onto the pixel plane, corresponding to the label value of the image semantic segmentation, g lidar (s) is the label value of the semantic segmentation of the point cloud. The external parameters of the vehicle's radar and camera are calibrated to obtain the data information of the extrinsic parameters of the calibrated radar and camera.

[0038] In order to achieve the above-mentioned objectives and other related objectives, the present invention also provides a radar camera extrinsic parameter calibration system based on normalized information distance, including a computer device, which is programmed or configured to execute any one of the steps of the radar camera extrinsic parameter calibration method based on normalized information distance.

[0039] In order to achieve the above-mentioned and other related purposes, the present invention also provides a computer-readable storage medium, which stores a computer program programmed or configured to execute any one of the radar camera extrinsic parameter calibration methods based on normalized information distance.

[0040] The present invention has the following positive effects:

[0041] The present invention adopts an improved roadside feature recognition algorithm based on yolov5 to identify the images and point clouds of vehicles driving into the roundabout area, and combines it with the common view construction algorithm based on SLAM to characterize the dense point cloud within the FOV of each frame image. It further adopts the cross-modal NID registration algorithm to calibrate the extrinsic parameters of the vehicle's radar and camera. Not only can extrinsic parameter calibration be performed when there is no field of view intersection or the field of view intersection is small, but there is no need for the use of special calibration targets, which reduces costs and improves calibration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the method flow of the present invention;

[0043] Figure 2 This is a flow chart of the improved roadside feature recognition algorithm based on yolov5 of the present invention;

[0044] Figure 3 A schematic diagram of a common view area construction algorithm based on SLAM of the present invention;

[0045] Figure 4 Schematic diagram of the process of the cross-modal NID registration algorithm of the present invention;

[0046] Figure 5 Schematic diagram of the collection points of the present invention. DETAILED DESCRIPTION

[0047] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0048] Example 1: Figure 1 or Figure 5 As shown, a radar camera extrinsic parameter calibration method based on normalized information distance includes:

[0049] U1. A vehicle traveling on a road acquires real-time road image data using its onboard camera and point cloud data using its onboard lidar. These data are then synchronized and processed to produce synchronized road image data and point cloud data.

[0050] U2 based on the synchronously processed road image data information, the use of improved roadside feature recognition algorithm based on yolov5 to identify the image of the vehicle traveling to the roundabout area, to obtain the image data information of the roundabout area;

[0051] U3. Based on the image data information of the roundabout area and the point cloud data information of the synchronously processed road, a SLAM-based common view construction algorithm is used to characterize the dense point cloud within the FOV of each frame image to obtain the dense point cloud data information within the FOV of each frame image;

[0052] U4. Based on the data information of the dense point cloud within the FOV of each frame of the image, a cross-modal NID registration algorithm is used to calibrate the external parameters of the vehicle's radar and camera to obtain data information of the extrinsic parameters of the calibrated radar and camera.

[0053] In this embodiment, if Figure 2 As shown, in step U2, the use of the improved roadside feature recognition algorithm based on yolov5 to recognize the image and point cloud of the vehicle traveling to the roundabout area includes:

[0054] U21. Based on the synchronized image data information of the road, construct a road image dataset;

[0055] U22 inputs the road image dataset into the yolov5 target detection model, characterizes the target detection feature points of the roundabout area image, and obtains data information of the target detection feature points of the roundabout area image;

[0056] U23. Based on the data information of the target detection feature points in the roundabout area, establish an image recognition function Q of the roundabout area,

[0057] ,

[0058] Among them, x is the data information of the target detection feature points in the roundabout area, ɑ1, ɑ2, and ɑ3 are the image recognition factors of the roundabout area, and the image of the vehicle driving into the roundabout area is recognized to obtain the data information of the image of the roundabout area.

[0059] In this embodiment, the image recognition factors ɑ1, ɑ2 and ɑ3 of the island area are:

[0060] ,

[0061] ,

[0062] ,

[0063] Among them, x is the data information of the target detection feature points in the island area.

[0064] In this embodiment, if Figure 3 As shown, in step U3, the characterization of the dense point cloud within the FOV of each frame image using the SLAM-based common view construction algorithm includes:

[0065] U31. Based on the synchronously processed road point cloud data information, the SLAM algorithm is used to estimate the vehicle's trajectory to obtain the estimated vehicle trajectory data information;

[0066] U32. Based on the data information of the estimated vehicle's driving trajectory, adaptively voxelize the point cloud and construct an optimization function W.

[0067] ,

[0068] Among them, n lis the normal vector of the plane, q l is a point in the plane, N l is the total number of points in the plane, l is the plane of point cloud adaptive voxelization, n l T is the transpose of the plane normal vector, r is the coordinate of the point cloud under all poses of the target radar projected to the first frame pose, optimize the distance from the point to the plane in each voxel, and output the optimized vehicle trajectory;

[0069] U33. Based on the data information of the optimized vehicle's driving trajectory and the image of the roundabout area, the initial external parameters of the camera are obtained, the point cloud is converted to the radar coordinate system corresponding to each frame pose, and then reprojected to the camera coordinate system, the point cloud is traversed, and the point cloud within the camera FOV is saved to obtain a dense point cloud within the FOV of each frame image.

[0070] In this embodiment, the adaptive voxelization of the point cloud is to repeatedly voxelize each frame of the point cloud after conversion. The voxelization strategy is to save the voxel if all the points in the voxel are located in the same plane, otherwise the voxel is further decomposed into 8 8-voids until the minimum size is reached. It is judged whether all the points in the voxel are in the same plane by calculating the defense difference matrix of all the points in the voxel and judging the ratio of the maximum eigenvalue to the minimum eigenvalue. If it is greater than a certain value, it can be determined to be in the same plane, otherwise it is not in the same plane.

[0071] Example 2: Based on the radar camera extrinsic parameter calibration method based on normalized information distance in Example 1, the present invention is further illustrated and described below.

[0072] like Figure 1 or Figure 5 As shown, a radar camera extrinsic parameter calibration method based on normalized information distance includes:

[0073] U1. A vehicle traveling on a road acquires real-time road image data using its onboard camera and point cloud data using its onboard lidar. These data are then synchronized and processed to produce synchronized road image data and point cloud data.

[0074] U2 based on the synchronously processed road image data information, the use of improved roadside feature recognition algorithm based on yolov5 to identify the image of the vehicle traveling to the roundabout area, to obtain the image data information of the roundabout area;

[0075] U3. Based on the image data information of the roundabout area and the point cloud data information of the synchronously processed road, a SLAM-based common view construction algorithm is used to characterize the dense point cloud within the FOV of each frame image to obtain the dense point cloud data information within the FOV of each frame image;

[0076] U4. Based on the data information of the dense point cloud within the FOV of each frame of the image, a cross-modal NID registration algorithm is used to calibrate the external parameters of the vehicle's radar and camera to obtain data information of the extrinsic parameters of the calibrated radar and camera.

[0077] In this embodiment, if Figure 4 As shown, in step U4, the extrinsic parameters of the vehicle's radar and camera are calibrated using the cross-modal NID registration algorithm, including:

[0078] U41. Based on the dense point cloud data within the FOV of each frame image, use the deep learning models DeepLab and RangeNet++ to infer the point cloud and image, respectively, and output the corresponding 2D pixel points and 3D point semantic labels;

[0079] U42. Based on the corresponding 2D pixel points and 3D point semantic labels, establish the NID function f of the point cloud and the image,

[0080] ,

[0081] Where X is the corresponding 2D pixel point, Y is the corresponding 3D point semantic label, H(X,Y) is the joint entropy of the two sets of discrete variables, G(X;Y) is the mutual information of the two discrete variables, and the NID of the two sets of discrete random variables of the point cloud and image semantic labels is represented to obtain the data information of the NID of the two sets of discrete random variables of the point cloud and image semantic labels;

[0082] U43. Based on the NID data information of the two sets of discrete random variables of the point cloud and image semantic labels, establish the external parameter calibration function S of the radar and camera,

[0083] ,

[0084] in, is the external parameter of the radar and camera, N is N frames of data, N is a positive integer, s is the corresponding cropped point cloud under each frame pose, g cam (s) The point cloud is reprojected onto the pixel plane, corresponding to the label value of the image semantic segmentation, g lidar (s) is the label value of the semantic segmentation of the point cloud. The external parameters of the vehicle's radar and camera are calibrated to obtain the data information of the extrinsic parameters of the calibrated radar and camera.

[0085] Images can be semantically segmented with the help of deep learning models such as DeepLab, which output semantic labels for each pixel (such as "vehicle," "pedestrian," "lane line," etc.). Radar point clouds can be semantically segmented with the help of models such as RangeNet++, assigning semantic labels to each 3D point.

[0086] Define a joint semantic space and construct discrete probability distributions for the image and point cloud semantic segmentation results. Construct a NID problem to measure the similarity between the two semantic distributions.

[0087] In this embodiment, the present invention provides a radar camera extrinsic parameter calibration system based on normalized information distance, including a computer device that is programmed or configured to perform any one of the steps of the radar camera extrinsic parameter calibration method based on normalized information distance.

[0088] In this embodiment, the present invention provides a computer-readable storage medium storing a computer program programmed or configured to execute any one of the radar camera extrinsic parameter calibration methods based on normalized information distance.

[0089] Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0090] In summary, the present invention can not only perform external parameter calibration when there is no visual field intersection or a small visual field intersection, but also does not require the use of special calibration targets, thereby reducing costs and improving calibration efficiency.

[0091] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A radar camera extrinsic parameter calibration method based on normalized information distance, characterized in that: The method comprises: U1. A vehicle traveling on a road acquires real-time road image data using its onboard camera and point cloud data using its onboard lidar. These data are then synchronized and processed to produce synchronized road image data and point cloud data. U2 based on the synchronously processed road image data information, the use of improved roadside feature recognition algorithm based on yolov5 to identify the image of the vehicle traveling to the roundabout area, to obtain the image data information of the roundabout area; U3. Based on the image data information of the roundabout area and the point cloud data information of the synchronously processed road, a SLAM-based common view construction algorithm is used to characterize the dense point cloud within the FOV of each frame image to obtain the dense point cloud data information within the FOV of each frame image; U4. Based on the data information of the dense point cloud within the FOV of each frame of the image, a cross-modal NID registration algorithm is used to calibrate the external parameters of the vehicle's radar and camera to obtain data information of the extrinsic parameters of the calibrated radar and camera.

2. The radar camera extrinsic parameter calibration method based on normalized information distance according to claim 1, characterized in that: In step U2, the use of the improved roadside feature recognition algorithm based on yolov5 to recognize the image and point cloud of the vehicle traveling to the roundabout area includes: U21. Based on the synchronized image data information of the road, construct a road image dataset; U22 inputs the road image dataset into the yolov5 target detection model, characterizes the target detection feature points of the roundabout area image, and obtains data information of the target detection feature points of the roundabout area image; U23. Based on the data information of the target detection feature points in the roundabout area, establish an image recognition function Q of the roundabout area, ， Among them, x is the data information of the target detection feature points in the roundabout area, ɑ1, ɑ2, and ɑ3 are the image recognition factors of the roundabout area, and the image of the vehicle driving into the roundabout area is recognized to obtain the data information of the image of the roundabout area.

3. The radar camera extrinsic parameter calibration method based on normalized information distance according to claim 2, characterized in that: The image recognition factors ɑ1, ɑ2 and ɑ3 of the island area are, ， ， ， Among them, x is the data information of the target detection feature points in the island area.

4. The radar camera extrinsic parameter calibration method based on normalized information distance according to claim 1, characterized in that: In step U3, the characterization of the dense point cloud within the FOV of each frame image using the SLAM-based common view construction algorithm includes: U31. Based on the synchronously processed road point cloud data information, the SLAM algorithm is used to estimate the vehicle's trajectory to obtain the estimated vehicle trajectory data information; U32. Based on the data information of the estimated vehicle's driving trajectory, adaptively voxelize the point cloud and construct an optimization function W. ， Among them, n l is the normal vector of the plane, q l is a point in the plane, N l is the total number of points in the plane, l is the plane of point cloud adaptive voxelization, n l T is the transpose of the plane normal vector, r is the coordinate of the point cloud under all poses of the target radar projected to the first frame pose, optimize the distance from the point to the plane in each voxel, and output the optimized vehicle trajectory; U33. Based on the data information of the optimized vehicle's driving trajectory and the image of the roundabout area, the initial external parameters of the camera are obtained, the point cloud is converted to the radar coordinate system corresponding to each frame pose, and then reprojected to the camera coordinate system, the point cloud is traversed, and the point cloud within the camera FOV is saved to obtain a dense point cloud within the FOV of each frame image.

5. The radar camera extrinsic parameter calibration method based on normalized information distance according to claim 4, characterized in that: The adaptive voxelization of the point cloud is to repeatedly voxelize each frame of the point cloud after conversion. The voxelization strategy is to save the voxel if all the points in the voxel are located in the same plane, otherwise the voxel is further decomposed into 8 8-divisions until the minimum size is reached. It is judged whether all the points in the voxel are in the same plane by calculating the defense difference matrix of all the points in the voxel and judging the ratio of the maximum eigenvalue to the minimum eigenvalue. If it is greater than a certain value, it can be judged that they are in the same plane, otherwise they are not in the same plane.

6. The radar camera extrinsic parameter calibration method based on normalized information distance according to claim 1, characterized in that: In step U4, the extrinsic parameters of the vehicle's radar and camera are calibrated using the cross-modal NID registration algorithm, including: U41. Based on the dense point cloud data within the FOV of each frame image, use the deep learning models DeepLab and RangeNet++ to infer the point cloud and image, respectively, and output the corresponding 2D pixel points and 3D point semantic labels; U42. Based on the corresponding 2D pixel points and 3D point semantic labels, establish the NID function f of the point cloud and the image, ， Where X is the corresponding 2D pixel point, Y is the corresponding 3D point semantic label, H(X,Y) is the joint entropy of the two sets of discrete variables, G(X;Y) is the mutual information of the two discrete variables, and the NID of the two sets of discrete random variables of the point cloud and image semantic labels is represented to obtain the data information of the NID of the two sets of discrete random variables of the point cloud and image semantic labels; U43. Based on the NID data information of the two sets of discrete random variables of the point cloud and image semantic labels, establish the external parameter calibration function S of the radar and camera, ， in, is the external parameter of the radar and camera, N is N frames of data, N is a positive integer, s is the corresponding cropped point cloud under each frame pose, g cam (s) The point cloud is reprojected onto the pixel plane, corresponding to the label value of the image semantic segmentation, g lidar (s) is the label value of the semantic segmentation of the point cloud. The external parameters of the vehicle's radar and camera are calibrated to obtain the data information of the extrinsic parameters of the calibrated radar and camera.

7. A radar camera extrinsic parameter calibration system based on normalized information distance, comprising a computer device, characterized in that: The computer device is programmed or configured to execute the steps of the radar camera extrinsic parameter calibration method based on normalized information distance as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program programmed or configured to execute the radar camera extrinsic parameter calibration method based on normalized information distance according to any one of claims 1 to 6.

Images