A multi-modal fusion target detection method and system based on clustering optimization

Abstract

The application provides a multi-modal fusion target detection method and system based on clustering optimization, which comprises the following steps: obtaining a 2D image and a 3D point cloud containing a target to be detected; inputting the 2D image into a two-dimensional target detection network with a CBMA attention module to obtain a 2D detection frame of each target in the 2D image; performing coordinate mapping on the 2D detection frame of each target and the 3D point cloud to generate a conical region of interest of each target; inputting the conical region of interest of each target into a three-dimensional target detection network to obtain a 3D detection frame of each target; and based on the 2D detection frame and the 3D detection frame of each target, a clustering method of prior information points is used to obtain a final 3D detection frame of each target. The application adopts a 3D target detection network architecture based on clustering optimization, which greatly improves the average detection accuracy of small targets such as pedestrians and cyclists.

Description

Technical Field

[0001] This invention relates to the field of target detection, and more specifically, to a multimodal fusion target detection method and system based on clustering optimization. Background Technology

[0002] In the field of road target detection, the detection of small targets reveals that point clouds themselves are sparse. For example, when the point cloud of the KITTI dataset is projected onto the corresponding RGB image, only about 3% of the pixels have a corresponding point cloud. Furthermore, about half of the target point clouds in the Moderate and Hard difficulty levels of this dataset have fewer than 60 points. This results in incomplete structural and semantic information of small 3D targets, and the target is easily confused with the background. In particular, the point clouds of distant pedestrians and straight poles may exhibit almost identical geometric shapes, leading to false detections.

[0003] To achieve this goal, the main background technology used is the image point cloud detection algorithm based on F-PointNet, which still has some shortcomings in the current application environment.

[0004] The first type of problem arises from the imbalance of large and small target data in the KITTI dataset:

[0005] 1. Data Acquisition Bias: Most of the KITTI dataset was acquired using vehicle-mounted sensors (such as LiDAR and cameras), which are better at detecting large targets, such as cars and trucks. This results in a relatively abundant amount of data on large targets, while the amount of data on small targets (such as pedestrians and bicycles) is limited. This imbalance may cause deep learning models to be more inclined to learn large targets, while their detection performance on small targets is poor.

[0006] 2. Bias in Object Detection Models: Many traditional object detection models, such as Faster R-CNN and YOLO, are designed to prioritize the detection of large objects, while performing poorly on small objects. This is because these models typically have higher loss values ​​for large objects during training, thus affecting their learning of smaller objects. Therefore, model bias is a sub-problem contributing to the imbalance between large and small object detection.

[0007] The second type is the problem of sparse point clouds of small 3D targets in the distance, which are easily confused with the background:

[0008] 1. Sparse Point Cloud Problem: Small targets at a distance may result in very sparse point cloud data collected by lidar or other sensors due to their considerable distance. This means that a large number of data points are missing in the point cloud, making it difficult to accurately capture the target's shape and features. How to process this sparse point cloud data to accurately detect and identify small targets is an important issue.

[0009] 2. Background Confusion Problem: In complex urban environments, the point cloud of small targets may be confused with the surrounding background because they may share similar features with environmental elements such as buildings, roads, and trees. This leads to difficulty in distinguishing between targets and the background, increasing the risk of false detection. Solving this problem requires developing algorithms to accurately distinguish small targets from the background and utilizing contextual information to improve recognition accuracy.

[0010] 3. Distant Target Detection: Distant target detection is more challenging than near target detection because the point cloud of small targets at a distance can be affected by large-scale scattering, changes in illumination, and sensor limitations. Therefore, developing algorithms to effectively detect and identify small targets at a distance is another important issue. Summary of the Invention

[0011] This invention addresses the technical problems existing in the prior art by providing a multimodal fusion target detection method and system based on clustering optimization.

[0012] According to a first aspect of the present invention, a multimodal fusion target detection method based on clustering optimization is provided, comprising:

[0013] Acquire 2D images and 3D point clouds containing the target to be detected;

[0014] The 2D image is input into a 2D object detection network with a CBMA attention module to obtain the 2D detection box for each object in the 2D image;

[0015] Map the coordinates of the 2D detection box of each target to the 3D point cloud to generate a cone-shaped region of interest for each target;

[0016] The cone-shaped region of interest for each target is input into the 3D target detection network to obtain the 3D detection box for each target;

[0017] Based on the 2D and 3D detection boxes of each target, a clustering method that integrates prior information points is used to obtain the final 3D detection box of each target.

[0018] Based on the above technical solution, the present invention can also be improved as follows.

[0019] Optionally, the two-dimensional object detection network is a Yolov5 network enhanced with a CBAM attention mechanism, wherein training the two-dimensional object detection network includes:

[0020] Obtain the original training set, which includes multiple 2D images. Each 2D image contains large and / or small targets. Large targets refer to targets with a size exceeding a preset size, and small targets refer to targets with a size smaller than a preset size.

[0021] Large targets in some 2D images are compressed proportionally and then copied to some 2D images that do not contain targets of that category to obtain a training dataset that has been augmented with small targets.

[0022] The Yolov5 network is trained based on the expanded training dataset to obtain the trained two-dimensional object detection network;

[0023] The YOLOv5 network includes a CBAM attention module, which comprises a spatial attention submodule and a channel attention submodule, respectively enhancing the YOLOv5 network's spatial and channel attention to the target region of interest.

[0024] Optionally, the step of obtaining the final 3D detection box for each target by fusing prior information points using a clustering method based on the 2D and 3D detection boxes of each target includes:

[0025] For targets whose 2D detection bounding box height is greater than a preset number of pixels, the 3D detection bounding box of the target is directly used as the final 3D detection bounding box of the target.

[0026] For targets that have 2D detection boxes but no 3D detection boxes, when the confidence of the target's 2D detection box is greater than the first preset confidence, the final 3D detection box of each target is obtained by a clustering method that fuses prior information points based on the target's 2D detection box and the target's cone region of interest.

[0027] For a target with both 2D and 3D bounding boxes, when the confidence of the target's 3D bounding box is less than a second preset confidence level, the final 3D bounding box of each target is obtained by a clustering method that fuses prior information points based on the target's 2D bounding box and the target's cone region of interest.

[0028] Optionally, the step of obtaining the final 3D detection box of each target by fusing prior information points using a clustering method based on the 2D detection box of the target and the cone-shaped region of interest of the target includes:

[0029] The point cloud of the cone-shaped region of interest of the target is clustered based on Euclidean distance to obtain at least one clustering result;

[0030] When multiple clustering results exist, the clustering result with the closest center distance to the target's 2D detection box is selected as the target's final 3D detection box.

[0031] Optionally, when there are multiple clustering results that are closest to the center of the 2D detection box of the target, the clustering result with the largest number of point clouds is selected as the final 3D detection box of the target.

[0032] Optionally, this also includes patching the position and size of each small target:

[0033] Based on the size of each small target output by the 3D target detection network, the average size of small targets in each category is calculated, and the average size includes the average length, width and height.

[0034] Adjust the size of the final 3D detection box for each category of small targets to the average size.

[0035] Optionally, the original training dataset records the correspondence between the height of the 2D detection box of each target category and its distance relative to the LiDAR.

[0036] When filtering the clustering results, the location range of the center of the 3D detection box of the target is determined based on the height of the 2D detection box of the target and the correspondence between the height of the 2D detection box of each category of the target and its distance relative to the lidar.

[0037] If the center of the clustering result of the target is within the specified location range, the clustering result is retained; otherwise, the clustering result is discarded.

[0038] Optionally, the original training dataset records the rotation angle of the 2D detection box for each category of object;

[0039] When filtering clustering results, the rotation angle of each clustering result is determined based on the angle between the line connecting the farthest and nearest points in the horizontal direction of the point cloud in each clustering result and the x-axis. If the difference between the rotation angle of the clustering result and the rotation angle of the 2D detection box of the same category of target is less than a preset difference value, the clustering result is retained; otherwise, the clustering result is discarded.

[0040] According to a second aspect of the present invention, a multimodal fusion target detection system based on clustering optimization is provided, comprising:

[0041] The first acquisition module is used to acquire a 2D image and a 3D point cloud containing the target to be detected;

[0042] The second acquisition module is used to input the 2D image into a two-dimensional target detection network with a CBMA attention module to acquire the 2D detection box of each target in the 2D image;

[0043] The mapping module is used to map the coordinates of the 2D detection box of each target to the 3D point cloud, and generate a cone-shaped region of interest for each target;

[0044] The third acquisition module is used to input the cone-shaped region of interest of each target into the 3D target detection network to obtain the 3D detection box of each target;

[0045] The fusion module is used to obtain the final 3D detection box for each target by fusing the 2D and 3D detection boxes of prior information points through a clustering method.

[0046] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the processor is configured to implement a multimodal fusion target detection method based on clustering optimization when executing a computer management program stored in the memory.

[0047] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer management class program is stored, wherein when executed by a processor, the computer management class program implements the steps of a multimodal fusion target detection method based on clustering optimization.

[0048] This invention provides a multimodal fusion target detection method and system based on cluster optimization, which adopts a 3D target detection network architecture based on cluster optimization, greatly improving the average detection accuracy of smaller targets such as pedestrians and cyclists. Attached Figure Description

[0049] Figure 1 A flowchart of a multimodal fusion target detection method based on clustering optimization is provided by the present invention;

[0050] Figure 2 This is an example of how a large target can be scaled down proportionally and then pasted and copied into other images.

[0051] Figure 3 This is a schematic diagram of the CBAM attention module.

[0052] Figure 4 A schematic diagram of the point cloud clustering process;

[0053] Figure 5 A diagram illustrating the completion of clustering boxes;

[0054] Figure 6 This is a schematic diagram of clustering under occlusion conditions;

[0055] Figure 7-1 This is a schematic diagram showing the relationship between the height of a pedestrian's 2D bounding box and its distance from the lidar sensor.

[0056] Figure 7-2 A schematic diagram showing the relationship between the height of the 2D bounding box of a cyclist and its distance from the LiDAR sensor;

[0057] Figure 8 This is a schematic diagram of rotational misalignment;

[0058] Figure 9 This is a schematic diagram of the overall architecture of a multimodal fusion target detection method based on clustering optimization;

[0059] Figure 10 A schematic diagram of the structure of a multimodal fusion target detection system based on clustering optimization provided by the present invention;

[0060] Figure 11 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0062] Figure 1 A flowchart of a multimodal fusion target detection method based on clustering optimization provided by this invention is shown below. Figure 1 As shown, the method includes:

[0063] Step 1: Obtain a 2D image and a 3D point cloud containing the target to be detected.

[0064] Understandably, when detecting a target, a 2D image and a 3D point cloud containing the target are acquired, and then the target is detected in 3D based on the 2D image and the 3D point cloud.

[0065] Step 2: Input the 2D image into a 2D target detection network with a CBMA attention module to obtain the 2D detection box for each target in the 2D image.

[0066] Understandably, when performing object detection, a 2D image is input into a 2D object detection network to obtain a 2D detection box for each object in the 2D image, where the 2D detection box is a rectangular detection box.

[0067] The two-dimensional object detection network is a Yolov5 network enhanced with a CBAM attention mechanism, wherein training the two-dimensional object detection network includes:

[0068] Obtain the original training set, which includes multiple 2D images. Each 2D image contains large and / or small targets. Large targets refer to targets with a size exceeding a preset size, and small targets refer to targets with a size smaller than a preset size.

[0069] Large targets in some 2D images are compressed proportionally and then copied to some 2D images that do not contain targets of that category to obtain a training dataset that has been augmented with small targets.

[0070] The Yolov5 network is trained based on the expanded training dataset to obtain the trained two-dimensional object detection network;

[0071] The YOLOv5 network includes a CBAM attention module, which comprises a spatial attention submodule and a channel attention submodule, respectively enhancing the YOLOv5 network's spatial and channel attention to the target region of interest.

[0072] Understandably, the KITTI dataset suffers from a severe imbalance between large and small targets, with large targets far outnumbering small ones. To improve the detection accuracy of Yolov5 for small targets, an improved copy-paste module is first used to augment the images in the KITTI dataset, thereby enhancing the network's performance in detecting small targets. Instead of directly copying and pasting targets back into the original image, targets with a height greater than 25 pixels in the Pedestrian and Cyclist categories are proportionally reduced and randomly pasted into images that do not contain instances of those categories, thus increasing the number of small target instances. After data augmentation, compared to before augmentation, the number of Cyclist category instances doubled, and the number of Pedestrian category instances increased by 30%.

[0073] The Yolov5 network is trained using the augmented dataset to obtain a trained two-dimensional object detection network. Figure 2 This is an image showing the improved copy-paste module proposed in this invention applied to the KITTI dataset after being enlarged. Figure 2 The area within the middle box represents the augmented labeled target. During data augmentation, it is ensured that the copied and pasted target does not overlap with the original target. This method increases the contribution of small targets to the loss during training, thereby preventing the network from converging on larger targets.

[0074] The two-dimensional object detection network is a Yolov5 network improved based on the CBAM attention module. CBAM (Convolutional Block Attention Module) is an attention mechanism for convolutional neural networks, and its network structure is as follows: Figure 3 As shown, it combines the Spatial Attention Module (SAM) and the Channel Attention Module (CAM). In Spatial Attention, the network model learns the weight of each pixel to weight the feature vectors at different locations, thus focusing the model more on the important regions of the current task. The Channel Attention Module learns the importance of each channel in the feature map, adjusting the weights of these channels to enhance the overall performance of the model.

[0075] Adding the CBAM attention module to a 2D detector can effectively improve the network's ability to extract target features. Therefore, in order to improve the detection performance of YOLOv5 for small targets, this paper adds the CBAM attention module to YOLOv5 to improve its performance.

[0076] Step 3: Map the coordinates of the 2D detection box of each target to the 3D point cloud to generate a cone-shaped region of interest for each target.

[0077] Step 4: Input the cone-shaped region of interest of each target into the 3D target detection network to obtain the 3D detection box of each target.

[0078] Understandably, the coordinates of the 2D bounding box and 3D point cloud of each target in the 2D image detected by the 2D object detection network are mapped to generate a cone-shaped region of interest for each target. Then, the cone-shaped region of interest for each target is input into the 3D object detection network to obtain the 3D bounding box for each target.

[0079] Step 5: Based on the 2D and 3D detection boxes of each target, a clustering method that integrates prior information points is used to obtain the final 3D detection box of each target.

[0080] Understandably, based on the 2D and 3D detection boxes of each target, the final 3D detection box of each target is determined. In this invention, the results of each 2D and 3D detection box are fused based on the clustering optimization module.

[0081] For targets whose 2D detection bounding box height is greater than a preset number of pixels, the target's 3D detection bounding box is directly used as the final 3D detection bounding box. For example, for targets in the car category and targets whose 2D detection bounding box height is greater than 30 pixels, the clustering module is not used to optimize the detection results in the 3D target detection part. That is, if it is a large target, the clustering optimization module is not used, and the target's 3D detection bounding box is directly output.

[0082] For targets with 2D bounding boxes but no 3D bounding boxes, when the confidence of the target's 2D bounding box is greater than the first preset confidence, the final 3D bounding box of each target is obtained by a clustering method that fuses prior information points based on the target's 2D bounding box and the target's cone region of interest.

[0083] Understandably, for targets detected in 2D but not in 3D, only a 2D detection box exists. The clustering optimization module is only used to supplement the 2D detection results to obtain the target's 3D detection box if the confidence of the 2D detection box is greater than 0.2. If the confidence of the target's 2D detection box is less than 0.2, it indicates that the 2D detection box is also unreliable, and the target is determined to be undetectable or non-existent.

[0084] For a target with both 2D and 3D bounding boxes, when the confidence of the target's 3D bounding box is less than a second preset confidence level, the final 3D bounding box of each target is obtained by a clustering method that fuses prior information points based on the target's 2D bounding box and the target's cone region of interest.

[0085] Understandably, when both 2D and 3D components are detected, if the confidence level of the target's 3D detection box is less than 0.5, a clustering module is used to optimize the 2D detection results to obtain the target's 3D detection box. If the confidence level of the target's 3D detection box is greater than 0.5, the 3D detection box of the target detected by the 3D target detection network is directly output as the final result.

[0086] As an example, the step of obtaining the final 3D detection box of each target by fusing prior information points based on the 2D detection box of the target and the cone-shaped region of interest of the target includes: clustering the point cloud of the cone-shaped region of interest of the target based on Euclidean distance to obtain at least one clustering result; when there are multiple clustering results, selecting the clustering result with the closest center distance to the 2D detection box of the target as the final 3D detection box of the target.

[0087] Understandably, clustering is an unsupervised machine learning technique that groups data based on the principle of similarity. First, a feature of the data is selected and a feature threshold is set. Then, classes with the most similar features that are less than the threshold are continuously merged until all classes have features greater than the threshold, at which point clustering is complete. The clustering module proposed in this embodiment of the invention is based on Euclidean clustering of point clouds. Euclidean clustering is a clustering algorithm based on Euclidean distance. It can search for neighboring points of each point cloud through established topological relationships and calculate the Euclidean distance between them. Then, clustering is completed based on the Euclidean distance. The specific process is as follows: Figure 4 As shown.

[0088] Point cloud clustering typically generates multiple clustering results. If the number of clustering results is greater than one, the results are filtered and optimized. Since the target is often located at the center of the 2D detection box, the clustering result closest to the center of the cone-shaped region of interest generated by the 2D detection box is most likely the target. Therefore, the clustering result with the closest center distance to the target's 2D detection box is selected from the multiple clustering results as the final 3D detection box for that target. When there are multiple clustering results with the closest center distance to the target's 2D detection box, the clustering result with the largest number of points is selected as the final 3D detection box for that target.

[0089] Implementing point cloud clustering is not difficult, but ensuring the accuracy of the clustering results is a challenge. Multiple experiments have revealed that even after filtering the original clustering results, many errors still exist. Research has found that the inaccuracy of the clustering results is mainly caused by the following three issues, which will be explained below along with their solutions:

[0090] First, deep learning-based 3D object detection algorithms can acquire prior information about samples through training. When predicting 3D objects, even if the point cloud shape is incomplete, a bounding box of similar size to the actual target can still be obtained. This allows the evaluation script to classify the predicted box as a positive sample when calculating the intersection-union ratio (IU). Clustering, on the other hand, is an unsupervised algorithm. The boundaries of the predicted boxes obtained by clustering are limited by the outermost points and cannot automatically complete to the size the target should have. Figure 5 As shown, the solid-line box represents the actual target box, and the dashed-line box represents the box generated by clustering. It can be seen that although the clustering box correctly captures the target point cloud, the intersection-union ratio (IUU) between the predicted box and the actual box is less than the threshold because it is not automatically filled in to the target size. Therefore, the evaluation script will judge the predicted box as an incorrect result.

[0091] To address the aforementioned issues, the present invention further includes refining the position and size of each small target: based on the size of each small target output by the 3D target detection network, the average size of each category of small targets is calculated, the average size including average length, width, and height; the size of the final 3D detection box of each category of small targets is adjusted to the average size.

[0092] Specifically, prior information about the targets is used to optimize the clustering prediction boxes. For the Pedestrian and Cyclist categories, all samples in the KITTI dataset were analyzed to calculate the average size of these two target classes. The average dimensions of the Pedestrian category are 0.83 meters, 0.64 meters, and 1.77 meters, respectively; the average dimensions of the Cyclist category are 1.77 meters, 0.58 meters, and 1.73 meters, respectively. After clustering, the clustering prediction boxes are completed based on the location of the cluster centers and prior information about the target sizes, such as... Figure 5 As shown in the dashed box, this allows the clustering results to better match the actual target boxes.

[0093] Second, clustering can produce multiple results. Although the target with the most cluster centers in the front view of the region of interest in the point cloud can be prioritized as the final result, the correctness of the clustering result cannot be guaranteed if occlusion exists. For example... Figure 6 As shown in the 2D image, there are a total of four targets, two of which are severely occluded. In this case, using clustering methods will produce results in the wrong locations.

[0094] To address the aforementioned issues, the original training dataset records the correspondence between the height of the 2D detection box for each target category and its distance relative to the LiDAR. When filtering the clustering results, the location range of the center of the 3D detection box for the target is determined based on the height of the 2D detection box and the correspondence between the height of the 2D detection box for each target category and its distance relative to the LiDAR. If the center of the clustering result for the target is within the specified location range, the clustering result is retained; otherwise, the clustering result is discarded.

[0095] Understandably, in order to solve the second type of problem mentioned above, this embodiment of the invention statistically analyzed the correspondence between the height of the 2D detection box of the same target in the KITTI training set and its relative distance to the LiDAR. Figure 7-1 The curve showing the relationship between the height of the 2D detection bounding box of a pedestrian and its relative distance to the LiDAR is presented. Figure 7-2The graph illustrates the relationship between the height of the 2D bounding box for a cyclist (Cyclist) and its relative distance to the LiDAR sensor. The horizontal axis represents the 2D bounding box height in pixels, and the vertical axis represents the distance of the 3D target from the LiDAR sensor in meters. When filtering clustering results, the height of the 2D bounding box can be used to determine a reasonable range for the 3D center of the target, thus discarding some erroneous results.

[0096] Third, after determining the cluster centers and the length, width, and height of the 3D bounding boxes, it is also necessary to know the rotation angle of the target to better generate the predicted bounding boxes. For example... Figure 8 As shown, solid lines represent actual target boxes, and dashed lines represent clustered target boxes. For the Cyclist class, incorrect rotation angles can significantly affect the calculation of the intersection-union ratio.

[0097] To address this issue, the original training dataset records the rotation angle of the 2D detection boxes for each category of target. When filtering the clustering results, the rotation angle of each clustering result is determined based on the angle between the line connecting the farthest and nearest points in the horizontal direction of the point cloud in each clustering result and the x-axis. If the difference between the rotation angle of the clustering result and the rotation angle of the 2D detection boxes of the same category of target is less than a preset difference value, the clustering result is retained; otherwise, the clustering result is discarded.

[0098] For the third type of problem, the rotation angle of the predicted bounding box can be directly determined based on the angle between the line connecting the farthest and nearest points in the horizontal direction of the point cloud in the clustering results and the x-axis. The difference between the rotation angle of the predicted bounding box and the rotation angle of the 2D detection box is then calculated. If the difference between the rotation angle of the predicted bounding box and the rotation angle of the 2D detection box is too large, it indicates that the target prediction is inaccurate and should be abandoned.

[0099] See Figure 9 This is a flowchart illustrating the overall process of a multimodal fusion object detection method based on clustering optimization. A Yolov5 network enhanced with CBAM attention mechanism is used as the 2D object detection network. After generating a cone-shaped region of interest (ROI) from the 2D detection results, the point cloud data within the ROI is simultaneously fed into both the 3D object detection network and the clustering module. The final result is then determined by combining the outputs of both the 3D object detection network and the clustering module. If the target output by the 2D object detection network is large, no point cloud clustering is performed; the result from the 3D object detection network is used directly. If the target is small, a comprehensive judgment is made based on the height of the 2D detection box, the score output by the 3D object detection network, and the clustering result, ultimately outputting the final detection box result.

[0100] This invention employs a cluster-optimized 3D target detection network architecture, which greatly improves the average detection accuracy of smaller targets such as pedestrians and cyclists.

[0101] From a performance perspective, tests on the Kitti dataset show improvements over the classic VoxelNet across all three difficulty levels. Compared to F-PointNet, the algorithm increases the average precision (AP) for the three classes by 1.22%, 5.23%, and 5.07% at moderate difficulty, and by 2.6%, 6.76%, and 8.48% at hard difficulty. Compared to PV-RCNN, the algorithm improves the performance of the cyclist class by 3.1% at hard difficulty.

[0102] From a generalization perspective, the proposed data augmentation algorithm can be transferred to other datasets, demonstrating broad application value. The algorithm improves the model's generalization ability, helping it better adapt to different scenarios and data distributions. Because the model is exposed to various variations during training, it is more likely to handle different data conditions during testing.

[0103] From an efficiency perspective, the feature-level fusion scheme using point clouds and images requires significantly less computation compared to point-level fusion schemes. Feature-level fusion typically involves fusing high-level feature maps without performing fusion operations at every point in the original input data. This significantly reduces computational complexity, especially for large-scale input data, thereby improving the training and inference efficiency of the model.

[0104] See Figure 10 This invention provides a multimodal fusion target detection system based on clustering optimization, comprising a first acquisition module 1001, a second acquisition module 1002, a mapping module 1003, a third acquisition module 1004, and a fusion module 1005, wherein:

[0105] The first acquisition module 1001 is used to acquire a 2D image and a 3D point cloud containing the target to be detected;

[0106] The second acquisition module 1002 is used to input the 2D image into a two-dimensional target detection network with a CBMA attention module to acquire the 2D detection box of each target in the 2D image.

[0107] The mapping module 1003 is used to map the coordinates of the 2D detection box of each target to the 3D point cloud to generate a cone-shaped region of interest for each target.

[0108] The third acquisition module 1004 is used to input the cone-shaped region of interest of each target into the three-dimensional target detection network to obtain the 3D detection box of each target;

[0109] The fusion module 1005 is used to obtain the final 3D detection box of each target by fusing the 2D and 3D detection boxes of each target with the clustering method of prior information points.

[0110] It is understood that the multimodal fusion target detection system based on cluster optimization provided by the present invention corresponds to the multimodal fusion target detection method based on cluster optimization provided in the foregoing embodiments. The relevant technical features of the multimodal fusion target detection system based on cluster optimization can be referred to the relevant technical features of the slice fusion method of multi-source heterogeneous models, and will not be repeated here.

[0111] Please see Figure 11 , Figure 11 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 11 As shown, this embodiment provides a computer-readable storage medium 1100, on which a computer program 1111 is stored. When the computer program 1111 is executed by a processor, it implements the steps of a multimodal fusion target detection method based on clustering optimization.

[0112] This invention provides a multimodal fusion target detection method and system based on clustering optimization. Addressing the issue of imbalanced large and small target data, an improved copy-paste module is employed. Large targets are proportionally scaled down and pasted into other images to increase the number of small targets, thereby increasing their contribution to the training loss. A point cloud Euclidean clustering module that integrates target prior information is also proposed. This module is used to optimize and supplement the detection results output by the 3D target detection network, significantly improving the overall network accuracy for detecting small 3D targets. Furthermore, a CBAM attention module is added to the 2D target detection Yolov5 network, making the network focus more on important regions of the image to improve the detection accuracy of Yolov5 for small targets. This allows the network to generate better cone-shaped point cloud regions of interest using the 2D detection results.

[0113] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0114] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0119] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multimodal fusion target detection method based on clustering optimization, characterized in that, include: Acquire 2D images and 3D point clouds containing the target to be detected; The 2D image is input into a 2D object detection network with a CBMA attention module to obtain the 2D detection box for each object in the 2D image; Map the coordinates of the 2D detection box of each target to the 3D point cloud to generate a cone-shaped region of interest for each target; The cone-shaped region of interest for each target is input into the 3D target detection network to obtain the 3D detection box for each target; Based on the 2D and 3D detection boxes of each target, a clustering method that integrates prior information points is used to obtain the final 3D detection box of each target. The step of obtaining the final 3D detection box for each target by fusing prior information points from the 2D and 3D detection boxes of each target includes: For targets whose 2D detection bounding box height is greater than a preset number of pixels, the 3D detection bounding box of the target is directly used as the final 3D detection bounding box of the target. For targets that have 2D detection boxes but no 3D detection boxes, when the confidence of the target's 2D detection box is greater than the first preset confidence, the final 3D detection box of each target is obtained by a clustering method that fuses prior information points based on the target's 2D detection box and the target's cone region of interest. For a target with both 2D and 3D bounding boxes, when the confidence of the target's 3D bounding box is less than the second preset confidence level, the final 3D bounding box of each target is obtained by a clustering method that fuses prior information points based on the target's 2D bounding box and the target's cone region of interest. The method also includes patching the position and size of each small target: Based on the size of each small target output by the 3D target detection network, the average size of small targets in each category is calculated, and the average size includes the average length, width and height. Adjust the size of the final 3D detection box for each category of small targets to the average size.

2. The multimodal fusion target detection method according to claim 1, characterized in that, The two-dimensional object detection network is a Yolov5 network enhanced with a CBAM attention mechanism, wherein training the two-dimensional object detection network includes: Obtain the original training dataset, which includes multiple 2D images. Each 2D image contains large and / or small targets. Large targets refer to targets with a size exceeding a preset size, and small targets refer to targets with a size smaller than a preset size. Large targets in some 2D images are compressed proportionally and then copied to some 2D images that do not contain targets of that category to obtain a training dataset that has been augmented with small targets. The Yolov5 network is trained based on the expanded training dataset to obtain the trained two-dimensional object detection network; The YOLOv5 network includes a CBAM attention module, which comprises a spatial attention submodule and a channel attention submodule, respectively enhancing the YOLOv5 network's spatial and channel attention to the target region of interest.

3. The multimodal fusion target detection method according to claim 1, characterized in that, The step of obtaining the final 3D detection box for each target by fusing prior information points using a clustering method based on the target's 2D detection box and its cone-shaped region of interest includes: The point cloud of the cone-shaped region of interest of the target is clustered based on Euclidean distance to obtain at least one clustering result; When multiple clustering results exist, the clustering result with the closest center distance to the target's 2D detection box is selected as the target's final 3D detection box.

4. The multimodal fusion target detection method according to claim 3, characterized in that, When there are multiple clustering results that are closest to the center of the 2D detection box of the target, the clustering result with the largest number of point clouds is selected as the final 3D detection box of the target.

5. The multimodal fusion target detection method according to claim 3, characterized in that, The original training dataset records the relationship between the height of the 2D detection box for each type of target and its distance relative to the LiDAR. When filtering the clustering results, the location range of the center of the 3D detection box of the target is determined based on the height of the 2D detection box of the target and the correspondence between the height of the 2D detection box of each category of the target and its distance relative to the lidar. If the center of the clustering result of the target is within the specified location range, the clustering result is retained; otherwise, the clustering result is discarded.

6. The multimodal fusion target detection method according to claim 3, characterized in that, The original training dataset records the rotation angle of the 2D detection box for each category of object; When filtering clustering results, the rotation angle of each clustering result is determined based on the angle between the line connecting the farthest and nearest points in the horizontal direction of the point cloud in each clustering result and the x-axis. If the difference between the rotation angle of the clustering result and the rotation angle of the 2D detection box of the same category of target is less than a preset difference value, the clustering result is retained; otherwise, the clustering result is discarded.

7. A multimodal fusion target detection system based on clustering optimization, characterized in that, include: The first acquisition module is used to acquire a 2D image and a 3D point cloud containing the target to be detected; The second acquisition module is used to input the 2D image into a two-dimensional target detection network with a CBMA attention module to acquire the 2D detection box of each target in the 2D image; The mapping module is used to map the coordinates of the 2D detection box of each target to the 3D point cloud, and generate a cone-shaped region of interest for each target; The third acquisition module is used to input the cone-shaped region of interest of each target into the 3D target detection network to obtain the 3D detection box of each target; The fusion module is used to obtain the final 3D detection box for each target by fusing the 2D and 3D detection boxes of prior information points using a clustering method. Specifically, the fusion module is used to include: For targets whose 2D detection bounding box height is greater than a preset number of pixels, the 3D detection bounding box of the target is directly used as the final 3D detection bounding box of the target. For targets that have 2D detection boxes but no 3D detection boxes, when the confidence of the target's 2D detection box is greater than the first preset confidence, the final 3D detection box of each target is obtained by a clustering method that fuses prior information points based on the target's 2D detection box and the target's cone region of interest. For a target with both 2D and 3D bounding boxes, when the confidence of the target's 3D bounding box is less than the second preset confidence level, the final 3D bounding box of each target is obtained by a clustering method that fuses prior information points based on the target's 2D bounding box and the target's cone region of interest. The fusion module is also used to repair the position and size of each small target: Based on the size of each small target output by the 3D target detection network, the average size of small targets in each category is calculated, and the average size includes the average length, width and height. Adjust the size of the final 3D detection box for each category of small targets to the average size.

8. A computer-readable storage medium, characterized in that, It stores a computer management program, which, when executed by a processor, implements the steps of the multimodal fusion target detection method based on clustering optimization as described in any one of claims 1-6.

Images