Target detection method based on binocular vision and true-false laser point cloud
By combining binocular vision and 16-line LiDAR to construct true-false LiDAR point clouds, the problem of high cost limitation was solved, and high-precision target detection effect was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-08
- Publication Date
- 2026-04-07
AI Technical Summary
In the existing technology, the cost of 64-line LiDAR is too high, which limits its application in the field of autonomous driving, and the target detection accuracy needs to be improved.
A target detection method based on binocular vision and 16-line LiDAR is adopted, which combines an IGEV-Stereo depth estimation module, a true-false LiDAR point cloud fusion module, and an AVOD 3D target detection module. True-false LiDAR point clouds are constructed using a binocular vision camera and a 16-line LiDAR, and fusion is performed using the Iterative Closest Point registration principle to improve the target detection accuracy.
It effectively reduces the hardware cost of intelligent driving vehicles while improving target detection accuracy, achieving a detection effect comparable to 64-line LiDAR.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing, specifically to a target detection method based on binocular vision and true-false laser point clouds. Background Technology
[0002] In 2022, the adoption rate of Level 2 (L2) driver assistance systems reached 34%. It is predicted that by 2030, the adoption rate of Level 3 (L3) autonomous driving systems will reach 70%. With the continuous improvement of L3 autonomous driving systems, the era of intelligent vehicles has arrived. As a crucial component of autonomous driving systems, object detection technology has received considerable attention and extensive research. It can accurately identify and track surrounding traffic participants and obstacles, providing autonomous vehicles with the necessary environmental perception capabilities. Thanks to the rapid development of computer vision and deep learning technologies, object detection has made significant progress in the field of autonomous driving. The emergence of deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), has enabled object detection to more accurately identify and locate target objects in complex scenes.
[0003] In the current technology, 64-line LiDAR is used for target detection. The detection results are relatively accurate and have good practicality. However, the cost of 64-line LiDAR is too high, which greatly limits its application in the field of autonomous driving. Summary of the Invention
[0004] This invention aims to provide a target detection method based on binocular vision and true-false laser point cloud technology. This method combines binocular vision and true-false laser point cloud technology, which can effectively reduce the hardware cost of intelligent driving vehicles while improving the target detection accuracy.
[0005] The technical solution of the present invention is as follows:
[0006] The target detection method based on binocular vision and true-false laser point clouds, using a binocular vision camera and a 16-line LiDAR, is characterized by the following steps:
[0007] A. Construct a neural network, which includes an IGEV-Stereo depth estimation module, a true-false laser point cloud fusion module, and an AVOD 3D target detection module connected in sequence; train the neural network using samples from the international open dataset KITTI;
[0008] B. Obtain the left and right views captured by the binocular vision camera, and input them into the IGEV-Stereo depth estimation module. The IGEV-Stereo depth estimation module then processes the images to obtain the final disparity image.
[0009] C. Construct a laser point cloud coordinate system using a 16-line LiDAR to detect the real laser point cloud; project each pixel in the final parallax image onto the laser point cloud coordinate system constructed by the 16-line LiDAR to generate a pseudo laser point cloud; then downsample the pseudo laser point cloud using the Voxel point cloud filtering algorithm; and finally register and fuse the pseudo laser point cloud and the real laser point cloud using the Iterative Closest Point point cloud registration principle to obtain the true-pseudo laser point cloud.
[0010] D. Input the left view and the true-false laser point cloud into the AVOD 3D target detection module, process them to obtain the target type, position and size, mark the target in the left view and the true-false laser point cloud respectively, and obtain the final target detection result.
[0011] In step B, the processing procedure in the IGEV-Stereo depth estimation module is as follows:
[0012] a. The left and right views are processed by the Feature Network module, and the results of the two Feature Network modules are simultaneously input into the group-wise correlation volume module and the All-pairsCorrelations module for further processing.
[0013] The results from the group-wise correlation volume module are processed sequentially by the 3D RegularizationNetwork module and the Encoding Volume module. The results from the Encoding Volume module are split into two paths: one path is processed by the soft argmin module to obtain the initial disparity map, and the other path is processed sequentially by the 1D average pooling module and the Geometry Encoding Volume module. The results from the Geometry Encoding Volume module and the All-pairs Correlations module are input into the CGEV module for further processing to obtain the CGEV module's results.
[0014] b. The left view is input into the context network module for processing to obtain the context network module processing result; the initial disparity map and the CGEV module processing result are subjected to linear interpolation calculation, and the calculation result and the context network module processing result are respectively input into the first Conv GRU module for processing. The first Conv GRU module processing result is added to and fused with the initial disparity map to obtain the first fusion result; at the same time, the first Conv GRU module iterates to obtain the second Conv GRU module;
[0015] The first fusion result and the CGEV module processing result are linearly interpolated. This calculation result and the contextNetwork module processing result are respectively input into the second Conv GRU module for processing. The second Conv GRU module processing result is added to the initial disparity map and fused to obtain the second fusion result. At the same time, the second Conv GRU module iterates to obtain the third Conv GRU module. This process is repeated continuously to obtain the Nth fusion result, which is the final disparity map.
[0016] In step a, the Encoding Volume module is calculated using the following formula: Geometric Encoding Volume C G :
[0017]
[0018] Here, <·,·> represents the inner product, d represents the disparity index, and N... c This represents the number of feature channels. We use the softargmin function to extract the geometric coding volume C. G Regress the initial disparity d0:
[0019]
[0020] Where d0 is 1 / 4 of the original image size, it is then used as the initial disparity map. Multi-scale context features are used to initialize the hidden layers of the 3-level GRU, and then the ConvGRUs module is used to iteratively optimize the disparity map.
[0021] In step C, the pseudo-laser point cloud data generation process is as follows:
[0022] The depth information Z(u,v) of each pixel (u,v) in the two-dimensional image is obtained by using a stereo depth estimation model. Then, the obtained depth information is projected into the laser point cloud coordinate system constructed by the three-dimensional 16-line lidar: the pixel (u,v) of the image is converted into the corresponding (x,y,z) in the three-dimensional point cloud space by the following formula.
[0023]
[0024] Where g, (c U c V ) is the camera center, f U and f V It refers to the horizontal and vertical focal lengths.
[0025] In step C, the process of registering and fusing the pseudo laser point cloud and the real laser point cloud using the Iterative Closest Point registration principle is as follows:
[0026] a) The ICP algorithm minimizes the total error between the pseudo-laser point cloud and the real laser point cloud by finding the optimal transformation matrix M. Iteratively, each point in the pseudo-laser point cloud is matched with a plane in the real laser point cloud to calculate the distance error from the point to the plane. Then, by minimizing these distance errors, the ICP algorithm adjusts the parameters of the transformation matrix M, gradually optimizing the position and orientation of the pseudo-laser point cloud to better match the real laser point cloud. The specific formula is as follows:
[0027]
[0028] Where M is a 4×4 three-dimensional rigid body transformation matrix, s i Represents pseudo-laser point cloud, d i Represents a 16-line true laser point cloud, n i It is d i The unit normal vector at that location;
[0029] b. Unify the real laser point cloud and the pseudo laser point cloud into a 3D spatial coordinate system, and calculate the Euclidean distance from the pseudo point cloud to the real laser point cloud with the real laser point cloud as the center:
[0030]
[0031] Where D represents the Euclidean distance between the real and pseudo laser point clouds, (x1, y1, z1) represents the 3D spatial position of the real laser point cloud, and (x2, y2, z2) represents the 3D spatial position of the pseudo laser point cloud. Then, a radius threshold r is defined.
[0032] D≤r (6)
[0033] The radius threshold r is used to determine the Euclidean distance between points. If the distance between a point and the center point is less than or equal to the threshold, it is retained, resulting in a pseudo laser point cloud belonging to the real laser point cloud or near the real laser point cloud. At the same time, the real laser point cloud and the pseudo laser point cloud adjacent to it are retained.
[0034] The processing procedure in the AVOD 3D target detection module is as follows:
[0035] a) 3D Anchor grid generates a 3D anchor frame grid for candidate regions;
[0036] b. The left view first undergoes feature extraction in the Feature Extractor module to obtain an image feature map. After the image feature map is processed by 1×1 convolution, the convolutional image and the 3D anchor frame mesh are input into the Crop and Resize module. In the Crop and Resize module, the convolutional image is placed into the 3D anchor frame mesh to mark the preliminary target bounding boxes of the target's location, orientation, and category. The preliminary target bounding boxes in these convolutional images are cropped to obtain small area maps of each target. These small area maps are then input into the Fusion module.
[0037] The true and false laser point clouds are first processed by the Feature Extractor module to extract features to obtain laser point cloud feature maps. After the laser point cloud feature maps are processed by 1×1 convolution, the laser point cloud convolution image and the 3D anchor frame mesh are input into the Crop and Resize module. In the Crop and Resize module, the laser point cloud convolution image is put into the 3D anchor frame mesh to mark the preliminary target boxes of the target's location, orientation, and category. The preliminary target boxes in these laser point cloud convolution images are cropped to obtain small area laser point cloud maps of each target. These small area laser point cloud maps are input into the first Fusion module.
[0038] In the first Fusion module, the small-area maps and the small-area laser point cloud maps are fused in a multimodal manner. Then, after being processed by the fully connected layer and the NMS module, the optimized target bounding box with the target's location, orientation, and category is obtained.
[0039] c. Input the optimized target bounding boxes and image feature maps into the Crop and Resize module; in the Crop and Resize module, the image feature maps are placed into the optimized target bounding boxes, and the optimized target bounding boxes in these image feature maps are cropped to obtain the optimized small region maps where each target is located. These optimized small region maps are then input into the second Fusion module.
[0040] The optimized target bounding boxes and laser point cloud feature maps are input into the Crop and Resize module. In the Crop and Resize module, the laser point cloud feature maps are placed into the optimized target bounding boxes. The optimized target bounding boxes in these laser point cloud feature maps are cropped to obtain the optimized small-area laser point cloud maps of each target. These optimized small-area laser point cloud maps are then input into the second Fusion module.
[0041] In the second Fusion module, each optimized small-area map and each optimized small-area laser point cloud map are fused in a multimodal manner. Then, after being processed by a fully connected layer and an NMS module, optimized target bounding boxes with the target's location, orientation, and category are obtained. These optimized target bounding boxes are marked in 2D and 3D in the left view, and also marked in 3D in the real and fake laser point clouds to obtain the final target detection result.
[0042] In step b, the size of the small region image cropped from the initial target bounding box in the convolutional image is 3×3, and the size of the small region laser point cloud image cropped from the initial target bounding box in the laser point cloud convolutional image is 3×3.
[0043] In step c, the size of the optimized small region map cropped from the optimized target box in the image feature map is 7×7, and the size of the optimized small laser point cloud region map cropped from the optimized target box in the laser point cloud feature map is 7×7.
[0044] The specific network structure in the IGEV-Stereo depth estimation module of this invention adopts the existing technical literature: Xu G, Wang X, Ding X, et al. Iterative Geometry Encoding Volume for Stereo Matching[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:21919-21928.
[0045] The specific network structure in the AVOD 3D object detection module adopts the existing technical literature: Ku J, Mozifian M, Lee J, et al. Joint 3d proposal generation and object detection from viewaggregation[C] / / 2018IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS).IEEE,2018:1-8.
[0046] Compared with the prior art, the present invention has the following advantages:
[0047] (1) The IGEV depth estimation network was used to obtain the final disparity image, which can obtain more accurate depth information;
[0048] (2) Using pseudo laser point cloud data instead of high-thread laser point cloud data can replace 64-line laser radar with 16-line laser radar, effectively reducing the hardware cost of intelligent driving vehicles.
[0049] (3) This invention proposes a true-false laser point cloud fusion algorithm, which further improves the target detection accuracy.
[0050] The experiment in Example 2 demonstrates that the algorithm of this invention can effectively reduce the hardware cost of intelligent driving vehicles while improving target detection accuracy. Attached Figure Description
[0051] Figure 1 This is the overall flowchart of Example 1;
[0052] Figure 2 This refers to the AVOD target detection module in Example 1;
[0053] Figure 3 This is a comparison chart of the target detection results of the algorithm in Example 1 and other algorithms on the KITTI dataset; Detailed Implementation
[0054] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0055] Example 1
[0056] The target detection method based on binocular vision and true-false laser point clouds includes the following steps:
[0057] The target detection method based on binocular vision and true-false laser point clouds, using a binocular vision camera and a 16-line LiDAR, is characterized by the following steps:
[0058] A. Construct a neural network, which consists of an IGEV-Stereo depth estimation module, a true-to-false laser point cloud fusion module, and an AVOD 3D object detection module connected in sequence. Samples from the international open dataset KITTI are used, containing a total of 14,999 images, of which 7,481 are used for neural network training and 7,518 are used for testing. The training data is divided into three parts: 3,712 images are used for training, and 3,769 images are used for validation.
[0059] B. Obtain the left and right views captured by the binocular vision camera, and input them into the IGEV-Stereo depth estimation module. The IGEV-Stereo depth estimation module then processes the images to obtain the final disparity image.
[0060] The processing procedure in the IGEV-Stereo depth estimation module is as follows:
[0061] a. The left and right views are processed by the Feature Network module, and the results of the two Feature Network modules are simultaneously input into the group-wise correlation volume module and the All-pairsCorrelations module for further processing.
[0062] The results from the group-wise correlation volume module are processed sequentially by the 3D RegularizationNetwork module and the Encoding Volume module. The results from the Encoding Volume module are split into two paths: one path is processed by the soft argmin module to obtain the initial disparity map, and the other path is processed sequentially by the 1D average pooling module and the Geometry Encoding Volume module. The results from the Geometry Encoding Volume module and the All-pairs Correlations module are input into the CGEV module for further processing to obtain the CGEV module's results.
[0063] b. The left view is input into the context network module for processing to obtain the context network module processing result; the initial disparity map and the CGEV module processing result are subjected to linear interpolation calculation, and the calculation result and the context network module processing result are respectively input into the first Conv GRU module for processing. The first Conv GRU module processing result is added to and fused with the initial disparity map to obtain the first fusion result; at the same time, the first Conv GRU module iterates to obtain the second Conv GRU module;
[0064] The first fusion result and the CGEV module processing result are linearly interpolated. This calculation result and the contextNetwork module processing result are respectively input into the second Conv GRU module for processing. The second Conv GRU module processing result is added to the initial disparity map and fused to obtain the second fusion result. At the same time, the second Conv GRU module iterates to obtain the third Conv GRU module. This process is repeated continuously to obtain the Nth fusion result, which is the final disparity map.
[0065] In step a, the Encoding Volume module is calculated using the following formula: Geometric Encoding Volume C G :
[0066]
[0067] Here, <·,·> represents the inner product, d represents the disparity index, and N... c This represents the number of feature channels. We use the softargmin function to extract the geometric coding volume C. G Regress the initial disparity d0:
[0068]
[0069] Where d0 is 1 / 4 of the original image size, and then it is used as the initial disparity map. Multi-scale context features are used to initialize the hidden layer of the 3-level GRU, and then the ConvGRUs module is used to iteratively optimize the disparity map.
[0070] C. Construct a laser point cloud coordinate system using a 16-line LiDAR to detect the real laser point cloud; project each pixel in the final parallax image onto the laser point cloud coordinate system constructed by the 16-line LiDAR to generate a pseudo laser point cloud; then downsample the pseudo laser point cloud using the Voxel point cloud filtering algorithm; and finally register and fuse the pseudo laser point cloud and the real laser point cloud using the Iterative Closest Point point cloud registration principle to obtain the true-pseudo laser point cloud.
[0071] The process of generating pseudo-laser point cloud data is as follows:
[0072] The depth information Z(u,v) of each pixel (u,v) in the two-dimensional image is obtained by using a stereo depth estimation model. Then, the obtained depth information is projected into the laser point cloud coordinate system constructed by the three-dimensional 16-line lidar: the pixel (u,v) of the image is converted into the corresponding (x,y,z) in the three-dimensional point cloud space by the following formula.
[0073]
[0074] Where g, (c U c V ) is the camera center, f U and f V It refers to the horizontal and vertical focal lengths.
[0075] The process of registering and fusing pseudo laser point clouds and real laser point clouds using the Iterative Closest Point registration principle is as follows:
[0076] a) The ICP algorithm minimizes the total error between the pseudo-laser point cloud and the real laser point cloud by finding the optimal transformation matrix M. Iteratively, each point in the pseudo-laser point cloud is matched with a plane in the real laser point cloud to calculate the distance error from the point to the plane. Then, by minimizing these distance errors, the ICP algorithm adjusts the parameters of the transformation matrix M, gradually optimizing the position and orientation of the pseudo-laser point cloud to better match the real laser point cloud. The specific formula is as follows:
[0077]
[0078] Where M is a 4×4 three-dimensional rigid body transformation matrix, s i Represents pseudo-laser point cloud, d i Represents a 16-line true laser point cloud, n i It is d i The unit normal vector at that location;
[0079] b. Unify the real laser point cloud and the pseudo laser point cloud into a 3D spatial coordinate system, and calculate the Euclidean distance from the pseudo point cloud to the real laser point cloud with the real laser point cloud as the center:
[0080]
[0081] Where D represents the Euclidean distance between the real and pseudo laser point clouds, (x1, y1, z1) represents the 3D spatial position of the real laser point cloud, and (x2, y2, z2) represents the 3D spatial position of the pseudo laser point cloud. Then, a radius threshold r is defined.
[0082] D≤r (6)
[0083] The radius threshold r is used to determine the Euclidean distance between points. If the distance between a point and the center point is less than or equal to the threshold, it is retained, resulting in a pseudo laser point cloud belonging to the real laser point cloud or near the real laser point cloud. At the same time, the real laser point cloud and the pseudo laser point cloud adjacent to it are retained.
[0084] D. Input the left view and the true-false laser point cloud into the AVOD 3D target detection module, process them to obtain the target type, position and size, mark the target in the left view and the true-false laser point cloud respectively, and obtain the final target detection result.
[0085] The processing procedure in the AVOD 3D target detection module is as follows:
[0086] a) 3D Anchor grid generates a 3D anchor frame grid for candidate regions;
[0087] b. The left view first undergoes feature extraction in the Feature Extractor module to obtain an image feature map. After the image feature map is processed by 1×1 convolution, the convolutional image and the 3D anchor frame mesh are input into the Crop and Resize module. In the Crop and Resize module, the convolutional image is placed into the 3D anchor frame mesh to mark the preliminary target bounding boxes of the target's location, orientation, and category. The preliminary target bounding boxes in these convolutional images are cropped to obtain small area maps of each target. These small area maps are then input into the Fusion module.
[0088] The true and false laser point clouds are first processed by the Feature Extractor module to extract features to obtain laser point cloud feature maps. After the laser point cloud feature maps are processed by 1×1 convolution, the laser point cloud convolution image and the 3D anchor frame mesh are input into the Crop and Resize module. In the Crop and Resize module, the laser point cloud convolution image is put into the 3D anchor frame mesh to mark the preliminary target boxes of the target's location, orientation, and category. The preliminary target boxes in these laser point cloud convolution images are cropped to obtain small area laser point cloud maps of each target. These small area laser point cloud maps are input into the first Fusion module.
[0089] In the first Fusion module, the small-area maps and the small-area laser point cloud maps are fused in a multimodal manner. Then, after being processed by the fully connected layer and the NMS module, the optimized target bounding box with the target's location, orientation, and category is obtained.
[0090] c. Input the optimized target bounding boxes and image feature maps into the Crop and Resize module; in the Crop and Resize module, the image feature maps are placed into the optimized target bounding boxes, and the optimized target bounding boxes in these image feature maps are cropped to obtain the optimized small region maps where each target is located. These optimized small region maps are then input into the second Fusion module.
[0091] The optimized target bounding boxes and laser point cloud feature maps are input into the Crop and Resize module. In the Crop and Resize module, the laser point cloud feature maps are placed into the optimized target bounding boxes. The optimized target bounding boxes in these laser point cloud feature maps are cropped to obtain the optimized small-area laser point cloud maps of each target. These optimized small-area laser point cloud maps are then input into the second Fusion module.
[0092] In the second Fusion module, each optimized small-area map and each optimized small-area laser point cloud map are fused in a multimodal manner. Then, after being processed by a fully connected layer and an NMS module, optimized target bounding boxes with the target's location, orientation, and category are obtained. These optimized target bounding boxes are marked in 2D and 3D in the left view, and also marked in 3D in the real and fake laser point clouds to obtain the final target detection result.
[0093] In step b, the size of the small region image cropped from the initial target bounding box in the convolutional image is 3×3, and the size of the small region laser point cloud image cropped from the initial target bounding box in the laser point cloud convolutional image is 3×3.
[0094] In step c, the size of the optimized small region map cropped from the optimized target box in the image feature map is 7×7, and the size of the optimized small laser point cloud region map cropped from the optimized target box in the laser point cloud feature map is 7×7.
[0095] Example 2
[0096] Object detection was performed using the publicly available dataset KITTI. The results of the object detection were compared with those of References 1 and 2 using the same measurement standard.
[0097] Document 1: Ku J, Mozifian M, Lee J, et al.Joint 3d proposal generation and object detection from view aggregation[C] / / 2018IEEE / RSJ InternationalConference on Intelligent Robots and Systems(IROS).IEEE,2018:1-8.
[0098] Document 2: Wang Y, Chao WL, Garg D, et al. Pseudo-lidar from visual depthestimation: Bridging the gap in 3d object detection for autonomous driving[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2019:8445-8453.
[0099] The parameters used in references 1-2 are the same as those in the original text, and they are guaranteed to be the optimal parameters for the model.
[0100] After comparing three methods—AVOD, PR:AVOD, and PR:F-POINTNet—at IOU=0.7, as follows: Figure 3 As shown in (b) and (c), the detection accuracy of Embodiment 1 of the present invention is improved by 10.6% and 11.8% respectively compared with PR:AVOD and PR:F-POINTNet. This is mainly attributed to the use of a true-pseudo laser point cloud method in the present invention, instead of a pure pseudo point cloud. Pure pseudo point clouds have errors in depth estimation, which leads to inaccurate reflection of the position of distant targets. Therefore, when using pure pseudo point clouds for distant target detection, there may be missed detections or false alarms. In contrast, our true-pseudo laser point cloud method significantly improves the accuracy of distant object detection. Meanwhile, Embodiment 1 of the present invention is compared with 64-line lidar detection as follows... Figure 3 As shown in Figure (a), the detection result of Embodiment 1 of the present invention is 85.5%, which is comparable to the AVOD detection result using a 64-line lidar. Data comparisons are shown in Table 1.
[0101] Table 1. Performance comparison of the detection method provided in Example 1 with the detection methods in references 1, 2, and 3.
[0102]
[0103] Table 1 shows our study on target detection using pseudo-laser point cloud data obtained from depth estimation, employing the IGEV-Stereo technique. With an IOU of 0.7, we achieved satisfactory detection accuracies of 77.3% and 79.3% based on the AVOD and F-POINTNet network models, respectively. Furthermore, we explored target detection using a real-pseudo-laser point cloud fusion algorithm. At the same IOU threshold of 0.7, we achieved even more significant improvements using the AVOD and F-POINTNet network models, reaching detection accuracies of 85.5% and 84.6%, respectively. This result is almost comparable to target detection performance based on 64-line laser point clouds, demonstrating the superiority of our fusion algorithm in target detection.
Claims
1. A target detection method based on binocular vision and true-false laser point clouds, using a binocular vision camera and a 16-line lidar, characterized in that... Includes the following steps: A. Construct a neural network, which includes an IGEV-Stereo depth estimation module, a true-false laser point cloud fusion module, and an AVOD 3D target detection module connected in sequence; B. Obtain the left and right views captured by the binocular vision camera, and input them into the IGEV-Stereo depth estimation module. The IGEV-Stereo depth estimation module then processes the images to obtain the final disparity image. C. Construct a laser point cloud coordinate system using a 16-line LiDAR to detect the real laser point cloud; project each pixel in the final parallax image onto the laser point cloud coordinate system constructed by the 16-line LiDAR to generate a pseudo laser point cloud; then downsample the pseudo laser point cloud using the Voxel point cloud filtering algorithm; and finally register and fuse the pseudo laser point cloud and the real laser point cloud using the Iterative Closest Point point cloud registration principle to obtain the true-pseudo laser point cloud. The process of generating pseudo-laser point cloud data is as follows: Each pixel in the 2D image is obtained using a stereo depth estimation model. depth information Then, the acquired depth information is projected onto the laser point cloud coordinate system constructed by the 3D 16-line LiDAR: the image pixels... Convert to 3D point cloud space using the following formula. , that is, ； Where g, It is the center of the camera. and These are the horizontal and vertical focal lengths; The process of registering and fusing pseudo laser point clouds and real laser point clouds using the Iterative Closest Point registration principle is as follows: a) The ICP algorithm minimizes the total error between the pseudo-laser point cloud and the real laser point cloud by finding the optimal transformation matrix M. Iteratively, each point in the pseudo-laser point cloud is matched with a plane in the real laser point cloud to calculate the distance error from the point to the plane. Then, by minimizing these distance errors, the ICP algorithm adjusts the parameters of the transformation matrix M, gradually optimizing the position and orientation of the pseudo-laser point cloud to better match the real laser point cloud. The specific formula is as follows: ； Where M is a 4×4 three-dimensional rigid body transformation matrix. This represents a pseudo-laser point cloud. This represents a 16-line true laser point cloud. yes The unit normal vector at that location; b. Unify the real laser point cloud and the pseudo laser point cloud into a 3D spatial coordinate system, and calculate the Euclidean distance from the pseudo point cloud to the real laser point cloud with the real laser point cloud as the center: ； Where D represents the Euclidean distance between the real laser point cloud and the pseudo laser point cloud. Represents the 3D spatial location of the actual laser point cloud. The 3D spatial location of the pseudo-laser point cloud is represented, and then a radius threshold r is defined; ； The radius threshold r is used to determine the Euclidean distance between points. If the distance between a point and the center point is less than or equal to the threshold, it is retained, resulting in a pseudo laser point cloud belonging to the real laser point cloud or near the real laser point cloud. At the same time, the real laser point cloud and the pseudo laser point cloud adjacent to it are retained. D. Input the left view and the true-false laser point cloud into the AVOD 3D target detection module, process them to obtain the target type, position and size, mark the target in the left view and the true-false laser point cloud respectively, and obtain the final target detection result.
2. The target detection method based on binocular vision and true-false laser point clouds as described in claim 1, characterized in that: In step B, the processing procedure in the IGEV-Stereo depth estimation module is as follows: a. The left and right views are processed by the Feature Network module, and the results of the two Feature Network module processing are simultaneously input into the group-wise correlation volume module and the All-pairsCorrelations module for further processing. The results from the group-wise correlation volume module are processed sequentially by the 3D RegularizationNetwork module and the Encoding Volume module. The results from the Encoding Volume module are split into two paths: one path is processed by the soft argmin module to obtain the initial disparity map, and the other path is processed sequentially by the 1D average pooling module and the Geometry Encoding Volume module. The results from the Geometry Encoding Volume module and the All-pairs Correlations module are input into the CGEV module for further processing to obtain the CGEV module's results. b. The left view is input into the context Network module for processing to obtain the context Network module processing result; the initial disparity map and the CGEV module processing result are subjected to linear interpolation calculation, and the linear interpolation calculation result and the contextNetwork module processing result are respectively input into the first Conv GRU module for processing. The first Conv GRU module processing result is added to and fused with the initial disparity map to obtain the first fusion result; at the same time, the first Conv GRU module iterates to obtain the second Conv GRU module; The first fusion result and the CGEV module processing result are linearly interpolated. This calculation result and the contextNetwork module processing result are respectively input into the second Conv GRU module for processing. The second Conv GRU module processing result is added to the initial disparity map and fused to obtain the second fusion result. At the same time, the second Conv GRU module iterates to obtain the third Conv GRU module. This process is repeated continuously to obtain the Nth fusion result, which is the final disparity map.
3. The target detection method based on binocular vision and true-false laser point clouds as described in claim 2, characterized in that: In step a, the Encoding Volume module calculates volume using the following formula: Geometric Encoding Volume : ； Here, <·,·> represents the inner product, and d represents the disparity index. To represent the number of feature channels, we use the softargmin function to extract the geometric coding volume. Regress the initial parallax : ； Where d0 is 1 / 4 of the original image size, it is then used as the initial disparity map. Multi-scale context features are used to initialize the hidden layer of the 3-level GRU, and then the ConvGRUs module is used to iteratively optimize the disparity map.
4. The target detection method based on binocular vision and true-false laser point clouds as described in claim 1, characterized in that: The processing procedure in the AVOD 3D target detection module is as follows: a) 3D Anchor grid generates a 3D anchor frame grid for candidate regions; b. The left view first undergoes feature extraction in the Feature Extractor module to obtain an image feature map. After the image feature map is processed by 1×1 convolution, a convolutional image is obtained. This convolutional image and the 3D anchor frame mesh are input into the Crop and Resize module. In the Crop and Resize module, the convolutional image is placed into the 3D anchor frame mesh to mark the preliminary target bounding boxes of the target's location, orientation, and category. These preliminary target bounding boxes in the convolutional images are cropped to obtain small area maps of each target. These small area maps are then input into the Fusion module. The true and false laser point clouds are first processed by the Feature Extractor module to extract features and obtain laser point cloud feature maps. After the laser point cloud feature maps are processed by 1×1 convolution, the laser point cloud convolution image is obtained. This laser point cloud convolution image and the 3D anchor frame mesh are input into the Crop and Resize module. In the Crop and Resize module, the laser point cloud convolution image is put into the 3D anchor frame mesh to mark the preliminary target boxes of the target's location, orientation, and category. The preliminary target boxes in these laser point cloud convolution images are cropped to obtain small area laser point cloud maps of each target. These small area laser point cloud maps are input into the first Fusion module. In the first Fusion module, the small-area maps and the small-area laser point cloud maps are fused in a multimodal manner. Then, after being processed by the fully connected layer and the NMS module, the optimized target bounding box with the target's location, orientation, and category is obtained. c. Input the optimized target bounding boxes and image feature maps into the Crop and Resize module; in the Crop and Resize module, the image feature maps are placed into the optimized target bounding boxes, and the optimized target bounding boxes in these image feature maps are cropped to obtain the optimized small region maps where each target is located. These optimized small region maps are then input into the second Fusion module. The optimized target bounding boxes and laser point cloud feature maps are input into the Crop and Resize module. In the Crop and Resize module, the laser point cloud feature maps are placed into the optimized target bounding boxes. The optimized target bounding boxes in these laser point cloud feature maps are cropped to obtain the optimized small-area laser point cloud maps of each target. These optimized small-area laser point cloud maps are then input into the second Fusion module. In the second Fusion module, each optimized small-area map and each optimized small-area laser point cloud map are fused in a multimodal manner. Then, after being processed by the fully connected layer and the NMS module, the optimized target bounding box with the target's location, orientation, and category is obtained. These optimized target bounding boxes are marked in 2D and 3D form in the left view, and simultaneously marked in 3D form in the true-false laser point cloud to obtain the final target detection result.
5. The target detection method based on binocular vision and true-false laser point clouds as described in claim 4, characterized in that: In step b, the size of the small region image cropped from the initial target bounding box in the convolutional image is 3×3, and the size of the small region laser point cloud image cropped from the initial target bounding box in the laser point cloud convolutional image is 3×3.
6. The target detection method based on binocular vision and true-false laser point clouds as described in claim 4, characterized in that: In step c, the size of the optimized small region map cropped from the optimized target box in the image feature map is 7×7, and the size of the optimized small laser point cloud region map cropped from the optimized target box in the laser point cloud feature map is 7×7.