A data fusion method and system based on visual deep learning and laser point cloud, and a medium

Abstract

The application relates to a data fusion method and system based on visual deep learning and laser point cloud, and a medium, the method comprising F1: during vehicle driving, target point cloud data is acquired based on a vehicle-mounted laser radar and is time-stamped, and target image data is acquired based on a vehicle-mounted camera and is time-stamped; F2: the target point cloud data and the target image data are time-synchronized, and synchronized target point cloud data and synchronized target image data are output; F3: all point cloud data in a view cone corresponding to an image detection frame is extracted and clustered, and point cloud clustering results in the view cone are output. The application not only solves the correlation failure problem of current perception system visual and laser post-fusion, but also solves the frequent sudden braking problem caused by the fact that, when an automatic driving vehicle drives on uneven road, a rut mark or a road protrusion not affecting driving is clustered as an obstacle, and the driving fluency of the automatic driving vehicle is improved.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor data fusion technology, and in particular to a data fusion method, system and medium based on visual deep learning and laser point clouds. Background Technology

[0002] LiDAR is a sensor used to accurately acquire the three-dimensional position of objects. It has excellent performance in measuring the position and contour of targets. However, due to its relatively low angular resolution, the points scanned on the target are sparse, resulting in weak semantic information. In contrast, visual sensors can acquire complex information about the surrounding environment at high frame rates and high resolutions, and they have a strong ability to understand the semantic information of the scene. Therefore, LiDAR and image sensors each have their own advantages and disadvantages.

[0003] In response to the frequent accidents involving autonomous vehicles, and to improve the safety of the entire autonomous vehicle system, different sensors are integrated and used together, complementing each other's strengths and combining redundancy. This integrated perception solution has become one of the key technologies for autonomous driving.

[0004] In existing technologies, the post-fusion technology of LiDAR and vision has relatively accurate 3D position attributes of LiDAR, but the accuracy of obstacle position estimation by vision is relatively low and the position jump is serious. This makes it easy for mismatch to occur during the fusion process, resulting in fusion failure and failing to effectively utilize the advantages of both. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides a data fusion method, system and medium based on visual deep learning and laser point cloud. It not only solves the problem of failure of post-fusion association between vision and laser in current perception systems, but also the fusion algorithm can solve the problem of frequent emergency braking caused by autonomous vehicles encountering obstacles such as ruts or road bumps that do not affect driving when driving on uneven roads, thereby improving the driving smoothness of autonomous vehicles.

[0006] To achieve the above and other related objectives, the technical solution provided by this invention is as follows: A data fusion method based on visual deep learning and laser point clouds, comprising the following steps:

[0007] F1: During vehicle operation, target point cloud data is acquired and timestamped based on the vehicle-mounted LiDAR, and target image data is acquired and timestamped based on the vehicle-mounted camera.

[0008] F2: Synchronizes the target point cloud data and target image data in time, and outputs synchronized target point cloud data and synchronized target image data;

[0009] F3: Project the point cloud data of the synchronized target points onto the synchronized target image data, extract all point cloud data in the view frust corresponding to the image detection box and cluster them, and output the point cloud clustering results within the view frust.

[0010] F4: Based on synchronized target image data, perform image deep learning network inference and use ByteTrack's MOT algorithm to output the tracked and filtered 2D detection box with target ID attribute;

[0011] F5: Fit and match the clustering results of the point cloud within the view frustum with the 2D detection box of the image, and output the fused target position, contour and category.

[0012] Furthermore, in step F3, the extraction and clustering of all point cloud data in the view frustum corresponding to the image detection box includes the following steps:

[0013] F31: Perform ground fitting on all point cloud data in the view frust corresponding to the image detection box, delete ground points, and output the remaining point cloud data in the view frust.

[0014] F32: Cluster the remaining point cloud data within the view frustum and output the point cloud clustering results within the view frustum.

[0015] Furthermore, in step F4, the ByteTrack MOT algorithm includes the following steps:

[0016] F41: Based on the synchronized target image data, obtain the score of the image detection box and divide the score of the image detection box into high-scoring boxes and low-scoring boxes;

[0017] F42: Match the high-scoring bounding box with the previous tracking trajectory of the target, and then match the low-scoring bounding box with the tracking trajectory that did not match the high-scoring bounding box. For the detection bounding box that did not match the tracking trajectory but had a high score, create a new tracking trajectory and output the matching tracking trajectory.

[0018] F43: Based on the matching tracking trajectory and Kalman filtering, output the image 2D detection box with the target ID attribute.

[0019] Furthermore, the score of the image detection box is the number of frames in the image. An image with more than 30 frames is considered a high-scoring box, while an image with more than 15 frames but less than 30 frames is considered a low-scoring box.

[0020] Furthermore, in step F5, the in-frust point cloud clustering result and the 2D target bounding box of the image are fitted and matched, including the following steps:

[0021] F51: Calculate the IOU between the clustering results of the point cloud within the view frustum and the 2D detection box of the image, and sort them from largest to smallest;

[0022] F52: Based on the IOU, calculate the aspect ratio lidar_whr of the point cloud clustering result within the view frustum projected onto the image, the aspect ratio img_whr of the image detection box, and then calculate the aspect ratio index whr of the cluster. The calculation function is whr = (1 - lidar_whr / img_whr). Output the aspect ratios of all clusters in ascending order.

[0023] F53: Based on the aspect ratios of all clusters sorted from smallest to largest, a preset threshold is set, and the aspect ratios of clusters greater than the preset threshold are output, thereby outputting the fused target location, outline, and category.

[0024] Furthermore, the aspect ratio of the cluster is used as the criterion for judging the geometric consistency between the point cloud clustering result within the view frustum and the 2D detection box of the image.

[0025] Furthermore, the IOU is a standard for measuring the accuracy of detecting corresponding objects in a specific dataset.

[0026] To achieve the above and other related objectives, a data fusion system based on visual deep learning and laser point clouds includes a computer device programmed or configured to perform the steps of any of the data fusion methods based on visual deep learning and laser point clouds described above.

[0027] To achieve the above and other related objectives, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program programmed or configured to perform any of the data fusion methods based on visual deep learning and laser point clouds.

[0028] The present invention has the following positive effects:

[0029] 1. This invention solves the problem of post-fusion correlation failure between vision and laser in current perception systems.

[0030] 2. This invention incorporates visual deep learning to filter out false detections of laser clustering caused by dust, water mist, or uneven ground at the algorithm level rather than at the level of a specific strategy, and has good generalization (environmental adaptability).

[0031] 3. The fusion algorithm of this invention can solve the problem of frequent emergency braking caused by autonomous vehicles encountering obstacles such as ruts or road bumps that do not affect driving when driving on uneven roads, thereby improving the driving smoothness of autonomous vehicles.

[0032] 4. This invention proposes to combine DIOU and COIU data fusion algorithms to improve the matching accuracy of multi-sensor target detection, increase the detection rate of obstacles, avoid missing low obstacles, increase safety redundancy, and ensure driving safety.

[0033] 5. Without increasing costs, this invention fully leverages the advantages of existing visual sensors and introduces visual deep learning algorithms to add one-dimensional category information to the perceived target object, providing the decision-making system with richer obstacle information, enabling the decision to control the vehicle more "intelligently," thereby ensuring the driving comfort and safety of autonomous vehicles. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0035] Figure 2 This is a schematic diagram of the MOT algorithm flow of ByteTrack in this invention. Detailed Implementation

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Example 1: As Figure 1 As shown, a data fusion method based on visual deep learning and laser point clouds includes the following steps:

[0038] F1: During vehicle operation, target point cloud data is acquired and timestamped based on the vehicle-mounted LiDAR, and target image data is acquired and timestamped based on the vehicle-mounted camera.

[0039] F2: Synchronizes the target point cloud data and target image data in time, and outputs synchronized target point cloud data and synchronized target image data;

[0040] F3: Project the point cloud data of the synchronized target points onto the synchronized target image data, extract all point cloud data in the view frust corresponding to the image detection box and cluster them, and output the point cloud clustering results within the view frust.

[0041] F4: Based on synchronized target image data, perform image deep learning network inference and use ByteTrack's MOT algorithm to output the tracked and filtered 2D detection box with target ID attribute;

[0042] F5: Fit and match the clustering results of the point cloud within the view frustum with the 2D detection box of the image, and output the fused target position, contour and category.

[0043] Furthermore, in step F3, the extraction and clustering of all point cloud data in the view frustum corresponding to the image detection box includes the following steps:

[0044] F31: Perform ground fitting on all point cloud data in the view frust corresponding to the image detection box, delete ground points, and output the remaining point cloud data in the view frust.

[0045] F32: Cluster the remaining point cloud data within the view frustum and output the point cloud clustering results within the view frustum.

[0046] Among them, such as Figure 2 As shown, in step F4, the ByteTrack MOT algorithm includes the following steps:

[0047] F41: Based on the synchronized target image data, obtain the score of the image detection box and divide the score of the image detection box into high-scoring boxes and low-scoring boxes;

[0048] F42: Match the high-scoring bounding box with the previous tracking trajectory of the target, and then match the low-scoring bounding box with the tracking trajectory that did not match the high-scoring bounding box. For the detection bounding box that did not match the tracking trajectory but had a high score, create a new tracking trajectory and output the matching tracking trajectory.

[0049] F43: Based on the matching tracking trajectory and Kalman filtering, output the image 2D detection box with the target ID attribute.

[0050] The score of the image detection box is determined by the number of frames in the image. An image with more than 30 frames is considered a high-scoring box, while an image with more than 15 frames but less than 30 frames is considered a low-scoring box.

[0051] In step F5, the point cloud clustering result within the view frustum is fitted and matched with the 2D target bounding box of the image, including the following steps:

[0052] F51: Calculate the IOU between the clustering results of the point cloud within the view frustum and the 2D detection box of the image, and sort them from largest to smallest;

[0053] F52: Based on the IOU, calculate the aspect ratio lidar_whr of the point cloud clustering result within the view frustum projected onto the image, the aspect ratio img_whr of the image detection box, and then calculate the aspect ratio index whr of the cluster. The calculation function is whr = (1 - lidar_whr / img_whr). Output the aspect ratios of all clusters in ascending order.

[0054] F53: Based on the aspect ratios of all clusters sorted from smallest to largest, a preset threshold is set, and the aspect ratios of clusters greater than the preset threshold are output, thereby outputting the fused target location, outline, and category.

[0055] The aspect ratio of the cluster is used as the criterion for judging the geometric consistency between the point cloud clustering result within the view frustum and the 2D detection box of the image.

[0056] The IOU is a standard for measuring the accuracy of detecting a corresponding object in a specific dataset.

[0057] Specifically, 1) There is only one IOU1; 1) If there is only one IOU and the IOU is greater than the low threshold of 0.1, it is considered to be associated, the index value of the IOU is returned, and the process ends; otherwise, the association fails; 2) There are multiple IOU2; If only one of these multiple IOUs is greater than the high threshold of 0.85, it is considered to be associated, the index of that IOU is returned, and the process ends; otherwise, the subsequent process is executed; 2.2) If there are multiple IOUs that are all greater than the high threshold of 0.85, then the nearest cluster result is associated, that is, the index of the nearest minY is returned, and the process ends; otherwise, the subsequent process is executed; 2.3) If all these IOUs are less than the high threshold of 0.85, then geometric consistency is used for association, that is, it is judged whether the aspect ratio is less than whr_thresh(0.3): if so, it is considered to be associated, the index of the smallest aspect ratio is returned, and the process ends; then the cluster result with the largest IOU is associated, that is, the index of the nearest largest IOU is returned, and the process ends.

[0058] To achieve the above and other related objectives, a data fusion system based on visual deep learning and laser point clouds includes a computer device programmed or configured to perform the steps of any of the data fusion methods based on visual deep learning and laser point clouds described above.

[0059] To achieve the above and other related objectives, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program programmed or configured to perform any of the data fusion methods based on visual deep learning and laser point clouds.

[0060] Example 2: Based on the data fusion method, system and medium based on visual deep learning and laser point cloud in Example 1, the present invention will be further described and explained.

[0061] like Figure 2 As shown, the ByteTrack MOT algorithm includes the following steps:

[0062] F41: Based on the synchronized target image data, obtain the score of the image detection box and divide the score of the image detection box into high-scoring boxes and low-scoring boxes;

[0063] F42: Match the high-scoring bounding box with the previous tracking trajectory of the target, and then match the low-scoring bounding box with the tracking trajectory that did not match the high-scoring bounding box. For the detection bounding box that did not match the tracking trajectory but had a high score, create a new tracking trajectory and output the matching tracking trajectory.

[0064] F43: Based on the matching tracking trajectory and Kalman filtering, output the image 2D detection box with the target ID attribute.

[0065] Specifically, based on the synchronized target image data, the scores of the image detection boxes are obtained. The image data of the high-scoring boxes are compared with the vehicle's tracking trajectory to obtain the tracking trajectory. Then, based on the tracking trajectory and Kalman filtering, the image 2D detection boxes with target ID attributes are obtained.

[0066] This application discloses a computer-readable storage medium storing a computer program that, when executed, implements a data fusion method based on visual deep learning and laser point clouds. The computer-readable storage medium may include, but is not limited to, floppy disks, optical disks, CD-ROMs (Read-Only Optical Disk Memory), magneto-optical disks, ROMs (Read-Only Memory), RAMs (Random Access Memory), EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable Programmable Read-Only Memory), magnetic cards or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions. The computer-readable storage medium may be a product not connected to a computer device or a component used in a computer device.

[0067] In summary, this invention not only solves the problem of post-fusion association failure of vision and laser in current perception systems, but also solves the problem of frequent emergency braking caused by autonomous vehicles encountering obstacles such as ruts or road bumps that do not affect driving when driving on uneven roads, thus improving the driving smoothness of autonomous vehicles.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data fusion method based on visual deep learning and laser point cloud, characterized in that, The method comprises the following steps: F1. During vehicle driving, target point cloud data is acquired based on a vehicle-mounted laser radar and time stamp calibration is performed, and target image data is acquired based on a vehicle-mounted camera and time stamp calibration is performed; F2. The target point cloud data and the target image data are time-synchronized, and synchronized target point cloud data and synchronized target image data are output; F3. The synchronized target point cloud data is projected to the synchronized target image data, all point cloud data in a view cone corresponding to an image detection frame is extracted and clustered, and a point cloud clustering result in the view cone is output; F4. Based on the synchronized target image data, an image deep learning network is inferred, a MOT algorithm of ByteTrack is used, and an image 2D detection frame with a target ID attribute after tracking and filtering is output; F5. The point cloud clustering result in the view cone and the image 2D detection frame are fitted and matched, and a fused target position, contour and category are output; In step F5, the point cloud clustering result in the view cone and the image 2D detection frame are fitted and matched, comprising the following steps: F51. The IOU of the point cloud clustering result in the view cone and the image 2D detection frame is calculated, and the IOU is sorted from large to small; F52. Based on the IOU, the width-height ratio lidar_whr of the point cloud clustering result in the view cone projected to the image, the width-height ratio img_whr of the image detection frame, and the width-height ratio index whr of the clustering are calculated, the calculation function is whr=(1-lidar_whr / img_whr), and the width-height ratios of all clusters are sorted from small to large; F53. Based on the width-height ratios of all clusters sorted from small to large, a preset threshold is set, the width-height ratios of the clusters greater than the preset threshold are output, and a fused target position, contour and category are output. 2.The data fusion method based on visual deep learning and laser point cloud according to claim 1, wherein, In step F3, the extraction of all point cloud data in the view cone corresponding to the image detection frame and the clustering comprise the following steps: F31. All point cloud data in the view cone corresponding to the image detection frame are ground fitted, and ground points are deleted, and remaining point cloud data in the view cone is output; F32. The remaining point cloud data in the view cone is clustered, and a point cloud clustering result in the view cone is output. 3.The data fusion method based on visual deep learning and laser point cloud according to claim 1, wherein, In step F4, the MOT algorithm of ByteTrack comprises the following steps: F41. Based on the synchronized target image data, the score of the image detection frame is acquired, and the score of the image detection frame is divided into high-score frames and low-score frames; F42. The high-score frames are matched with the previous tracking trajectories of the target, the low-score frames are matched with the tracking trajectories that do not match the high-score frames, a new tracking trajectory is newly built for the detection frame that does not match the tracking trajectory and has a high score, and a matched tracking trajectory is output; F43. Based on the matched tracking trajectory and Kalman filtering, an image 2D detection frame with a target ID attribute is output. 4.The data fusion method based on visual deep learning and laser point cloud according to claim 3, characterized in that: The score of the image detection frame is the frame number of the image, the high-score frame is a frame number greater than 30, and the low-score frame is a frame number greater than 15 and less than 30. 5.The data fusion method based on visual deep learning and laser point cloud according to claim 1, characterized in that: The width-height ratio of the clustering is a geometric consistency judgment standard of the point cloud clustering result in the view cone and the image 2D detection frame. 6.The data fusion method based on visual deep learning and laser point cloud according to claim 1, characterized in that: The IOU is a standard for measuring the accuracy of detecting a corresponding object in a specific data set.

7. A data fusion system based on visual deep learning and laser point cloud, comprising a computer device, characterized in that, The computer device is programmed or configured to perform the steps of the data fusion method based on visual deep learning and laser point cloud according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored thereon a computer program programmed or configured to perform the data fusion method based on visual deep learning and laser point cloud according to any one of claims 1-6.

Images