Fusion detection method based on YOLOv8 segmentation network and DBSACN clustering

Abstract

The present invention belongs to the field of autonomous driving technology, specifically a fusion detection method based on the YOLOv8 segmentation network and DBSACN clustering. The method of the present invention includes: screening and obtaining 9 categories of data sets from a traffic image data set, training the YOLOv8 segmentation network to obtain an image detection model; lightweighting the model and performing 2D detection on the image; performing ground point filtering and detection frame filtering on the time-synchronized radar point cloud data set, clustering the filtered point cloud using a clustering algorithm, calculating the position and length, width and height of the clustered objects to obtain a 3D detection frame; back-projecting the 3D detection frame onto the 2D image to obtain a new 2D mask, and calculating the IOU with the previous 2D mask, screening the overlapping detection frame with the largest IOU, and obtaining the object category and 3D position contained in the detection frame. The present invention can improve recognition accuracy, save computing time, and is suitable for complex driving scenarios under vehicle-road collaborative conditions.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and specifically relates to a fusion detection method based on YOLOv8 segmentation network and DBSACN clustering. Background Art

[0002] In autonomous driving systems, multimodal fusion perception is an extremely important task in perceptual autonomous driving systems. Multimodal autonomous driving perception methods, including radar sensors and RGB camera sensors, solve object detection and semantic segmentation tasks. From the perspective of the fusion stage, it can be divided into data-level, feature-level, and object-level fusion. Data-level fusion (also known as early fusion) directly fuses raw sensor data of different modalities through spatial alignment. Feature-level fusion (also known as deep fusion) focuses on mixing multimodal data in the feature space. Object-level fusion is to combine the model to focus on making the final decision on the prediction results in each modality.

[0003] In traditional autonomous driving detection and recognition tasks, single-modal data perception based on a single radar or camera has inherent flaws. RGB camera data primarily captures data from the lower front view. However, in more complex autonomous driving environments, object segmentation and localization are significantly challenged by occlusion and the camera's depth limitations. Radar perception in the context of vehicle-infrastructure collaboration is affected by range and resolution, and even in extreme weather conditions such as fog and heavy rain, radar can have certain limitations in object detection and semantic segmentation. Therefore, the fusion of radar and RGB cameras can achieve higher precision and more comprehensive perception information for tasks such as object detection, semantic segmentation, depth completion, and prediction. Thanks to the rapid development of autonomous driving technology, multi-sensor collaborative perception fusion technology has also rapidly developed. Furthermore, thanks to the rise of emerging technologies such as 5G and artificial intelligence, fusion perception technologies for lidar, millimeter-wave radar, and visual detection devices are being developed. Raw data is directly fused through spatial alignment and projection, while semantic features from image branches are fused with the raw radar point cloud. This allows for more information about detected objects in vehicle-infrastructure collaboration and faster detection speeds tailored to different hardware devices.

[0004] Most existing radar and RGB camera fusion detection methods use YOLOv3 or YOLOv5 for image detection to obtain 2D detection boxes, or use a single sensor to perform perception tasks. Existing fusion detection methods suffer from the following problems: First, when using YOLOv3 or YOLOv5 for image detection, the detection information obtained is a 2D detection box of the detected object, which contains a lot of image noise, significantly affecting subsequent fusion detection tasks and unable to accurately complete the identification and positioning of the detected object. Second, the point cloud layer uses few filtering algorithms for the point cloud of the detected object, and directly performs unsupervised clustering on the raw point cloud, resulting in many invalid 3D detection boxes, which affects the 3D detection and positioning results. Third, there is usually no appropriate lightweight deployment strategy for the resulting detection model. Using the trained PT model or PTH model directly for object detection relies on the PyTorch or TensorFlow environment, resulting in low operational efficiency and complex environment deployment. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to propose a collaborative perception fusion detection method based on YOLOv8 segmentation network and DBSACN clustering with high recognition accuracy and easy deployment and installation.

[0006] The present invention proposes a fusion detection method based on the segmentation network of YOLOv8 and DBSACN clustering, which includes preprocessing the traffic data set, extracting the data set of specific categories and converting the format of the data set label; using the segmentation network based on YOLOv8 [1] The pre-processed dataset is trained and the model is lightweighted to generate the model file. Compared with YOLOv8 [3] The original network, the segmentation network of YOLOv8, better identifies the contour features of the object, and the detection accuracy and recall rate are also improved, and the training process is more stable. For the same data set, two networks are used for training to verify the superiority of the segmentation network of YOLOv8 used in this paper. The image processed by time synchronization is used to detect the image using the model to generate a 2D mask, and the detection box filter and ground point filter are used to filter the noise point cloud processed by time synchronization; the density-based DBSCAN clustering algorithm [2] is used to cluster the filtered point cloud to obtain a 3D detection box; back-projection is performed to obtain a new 2D mask, which is matched with the 2D mask obtained by image detection, so as to obtain the type of the detected object and the three-dimensional coordinates in the three-dimensional space with the vehicle body radar as the coordinate system; the specific steps are as follows:

[0007] (1) First, the label type extraction is performed on the traffic fine-labeled dataset (5000 fine-labeled images), and the labels for semantic segmentation are screened. The 33 classes in the dataset are screened and 8 classes are extracted, namely "person" (pedestrian), "rider" (cyclist), "car" (car), "truck" (truck), "bus" (bus), "train" (train), "motorcycle" (motorcycle), and "bicycle" (bicycle); the label information of each image contains the outline of the object, that is, the points contained in the shape contour parameters (polygon) in the label file (json). Suppose there are n detected object labels in an image, and each label contains m points describing the outline, where m ≥ 3. Suppose the jth outline point of the i-th object in the label Assume that the format of the converted contour point is The conversion formula is as follows:

[0008]

[0009] After obtaining the converted data set, a random allocation algorithm is used to divide the data set into training set, validation set, and test set in a ratio of 4:2:1. The number of training times is set to 500, and the number of data samples captured during one training process is 16. The YOLOv8 segmentation network is trained on GPU to obtain the segmentation model pt. Compared with the original YOLOv8 network, the YOLOv8 segmentation network first replaces the C2f with the C3 module at the head of the neural network, and the 3-6-9-3 structure in the original network is converted to the 3-6-6-3 structure in the segmentation network. The first convolution is removed in the FPN structure at the neck of the neural network; the SiLU activation function is used, which can alleviate the gradient disappearance problem compared to the ReLU activation function used in the original network. It is also a smooth function, which means that it has derivatives in the entire domain (which the ReLU activation function cannot satisfy), which is conducive to network optimization.

[0010] (2) The model obtained above is lightweighted, and the pt model is converted to an onnx model (Open Neural Network Exchange), and then converted to an openvino model. At the same time, the image input size of the image segmentation model is defined as 640x640, and the time-synchronized image dataset is detected to obtain a series of 2D masks and categories describing the contours of the detected objects. The obtained 2D mask set is represented as follows:

[0011]

[0012] Among them, suppose there are n segmentation targets detected, each target has m contour points, and the set of detected targets is mask 2d .

[0013] (3) The time-synchronized radar point cloud dataset is read frame by frame, and the CSF point cloud ground point filter and the mask set obtained above are used to perform detection frame filtering on the original point cloud according to the relative position relationship between the radar and the camera, and only the point cloud associated with the detection mask in the non-ground point cloud is retained;

[0014] (4) The processed filtered point cloud is clustered based on DBSCAN, in which the KD-Tree algorithm (a variant of the binary search tree) is used to speed up the search for nearby points, and the PCL library is used to calculate the center of mass and the boundary of the point cloud to determine the 3D detection box of the point cloud. 3d ;

[0015] (5) A series of 2D masks obtained by the YOLOv8 segmentation network 2d And a series of 3D detection boxes obtained by DBSCAN clustering 3d , and then obtain the 2D mask box through rotation transformation r2d , and calculate the IOU intersection-of-union ratio one by one, and compare it with the threshold to get the type and position of the object.

[0016] Furthermore, the specific steps of CSF point cloud ground point filtering and detection box filtering in step (3) are as follows:

[0017] (1) According to the external parameter rotation matrix, internal parameter rotation matrix and distortion matrix between the camera and radar, the radar point cloud is projected onto the time-synchronized image; according to the 2D mask set obtained by recognition 2d , filter the point cloud according to the range of each 2D mask, and retain the point cloud within the mask; the expression for calculating the rotation transformation of the point cloud projection to the image is as follows:

[0018]

[0019] Among them, cx0, cy0, u0, v0 are camera intrinsic parameters, f is the camera focal length, R represents the rotation matrix, T represents the translation vector, s represents the scale factor, x, y, z are the point clouds to be projected, rx, ry are the image coordinates corresponding to the point cloud after projection; if rx, ry exists in the 2D mask set mask 2d If it is within the range, the corresponding point cloud will be retained, otherwise it will be filtered;

[0020] (2) Perform CSF ground point filtering on the point cloud after detection box filtering. Based on the curvature information of the points in the point cloud and the local features of the point cloud data, by analyzing the curvature values of the neighborhood of each point and screening the ground points according to the curvature threshold, the steps are as follows: construct the storage structure of the point cloud data; calculate the curvature of each point in the stored point cloud and determine whether the point is a ground point according to the curvature threshold; for the points in the neighborhood of the determined ground points, perform reachability analysis to further filter out non-ground points. Repeat the above operations until all points are processed, and thus obtain the ground point cloud and non-ground point cloud.

[0021] Further, the DBSCAN clustering method described in step (4) is specifically as follows:

[0022] First, use the KD-Tree algorithm to find the core points according to the number of neighborhood points, and then use the region growing method to perform unsupervised clustering on the point cloud to obtain a series of classified point clouds. Use the compute3DCentroid() algorithm of the PCL point cloud library to calculate the center point of the clustered object, and at the same time obtain the minimum boundary value point and the maximum boundary value point from the PCL library function getMinMax3D(), and then calculate the bounding box to obtain a series of boxes 3d , assuming that n 3D bounding boxes are obtained, which are box0, box1,..., box n , and the expression form is as follows:

[0023] box 3d ={box0, box1,..., box n}(4)

[0024] Further, the specific process of step (5) is as follows:

[0025] Convert the obtained 3D bounding box into a 2D detection box box on the image through formula 3 2d , and the form expression is as follows:

[0026] box 2d ={box2d0, box2d1,..., box2d n}(5)

[0027] Assume that the i-th radar conversion detection box obtained, where 0 ≤ i < n and the j-th image detection box, 0 ≤ j < n are:

[0028]

[0029] Among them, (x li , y li ), (x ri , y ri) represent the upper left corner and lower right corner of the radar point cloud conversion detection frame, respectively. Let S1 and S2 represent the areas of the above two detection frames, and the area of ​​the intersection is set to S t , then the calculation formula for the intersection-over-union (IOU) is:

[0030]

[0031] Design a suitable intersection-over-union (IOU) threshold. If the IOU is greater than the IOU threshold, the two detection frames are considered to intersect. Then, the category of the detected object is determined based on the image detection information, and the position of the detected object is determined using the radar point cloud detection frame.

[0032] The collaborative perception fusion detection method proposed in this invention has high accuracy and simple deployment, making it particularly suitable for complex driving scenarios under vehicle-road collaborative conditions. Compared with existing technologies, this invention has the following advantages:

[0033] First, compared with the existing methods of using YOLOv3 or YOLOv5 for image detection, the present invention adopts the image segmentation method to describe the contour of the detected object, and uses a 2D mask instead of a 2D detection box. The noise points in the image are reduced, and the radar point cloud can be effectively filtered, which is conducive to the development and application of point cloud layer clustering algorithms.

[0034] Second, compared with previous algorithms that directly perform clustering on point clouds, the present invention adopts a detection box filtering algorithm and a ground point filtering algorithm, which greatly reduces the number of noisy point clouds and point clouds to be processed, which is beneficial to the accuracy of point cloud-level clustering algorithms in detecting 3D objects and reducing the time of point cloud processing.

[0035] Third, compared to previous detection methods using open-source Python machine learning libraries like Pytorch or TensorFlow, the algorithm proposed in this paper relies on OpenVino and is deployed in C++. This simplifies the deployment process and, due to the lightweight training model, enables faster recognition. This makes it particularly suitable for complex driving scenarios under vehicle-road cooperative conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the present invention.

[0037] Figure 2 Schematic diagram of testing using labels and images after converting the traffic dataset label format.

[0038] Figure 3 This figure shows a comparison of the accuracy of the original YOLOv8 network of the present invention and the original model obtained by training the Cityscape traffic dataset using the YOLOv8 segmentation network used in this article.

[0039] Figure 4 This figure shows a comparison of the recall rates of the original YOLOv8 network of the present invention and the original model obtained by training the Cityscape traffic dataset using the YOLOv8 segmentation network used in this article.

[0040] Figure 5 Schematic diagram of the YOLOv8 segmentation network test image.

[0041] Figure 6 Schematic diagram of the original point cloud after radar fusion, detection box filtering, and ground point filtering.

[0042] Figure 7 Schematic diagram of 3D detection boxes obtained by DBSCAN clustering of point clouds.

[0043] Figure 8 Schematic diagram of the present invention performing IOU calculation on an image 2D mask and a rotated 2D mask. DETAILED DESCRIPTION

[0044] The present invention will be further described below through embodiments in conjunction with the accompanying drawings.

[0045] The collaborative perception fusion detection method based on the segmentation network and DBSACN clustering of YOLOv8 provided by the present invention includes: firstly extracting the label types of a precisely annotated traffic dataset (Cityscape dataset) (5000 precisely annotated images), screening and processing the labels for semantic segmentation, screening the 33 classes in the dataset, and extracting 8 of them, including "person" (pedestrian), "rider" (cyclist), "car" (car), "truck" (truck), "bus" (bus), "train" (train), "motorcycle" (motorcycle), and "bicycle" (bicycle). The label information of each image includes the outline of the object; after obtaining the converted dataset, a random allocation algorithm is used to divide the dataset into a training set, a validation set, and a test set in a ratio of 4:2:1, and the YOLOv8 segmentation network is trained on a GPU of model 3090 to obtain a segmentation model pt. The model obtained above is lightweighted, the pt model is converted to the onnx model (Open NeuralNetwork Exchange), and then converted to the openvino model. At the same time, the image input size of the image segmentation model is defined as 640x640, and the time-synchronized image dataset is detected to obtain a series of 2D masks and categories describing the contours of the detected objects; the time-synchronized radar point cloud dataset is read frame by frame, and the CSF point cloud ground point filter and the above-mentioned mask set are used to perform detection frame filtering on the original point cloud according to the relative position relationship between the radar and the camera, and only the point cloud associated with the detection mask in the non-ground point cloud is retained; the processed filtered point cloud is subjected to a DBSCAN-based clustering algorithm, in which the KD-Tree algorithm is used to speed up the search for nearby points, and the PCL library is used to calculate the point cloud centroid and boundary to determine the 3D detection box of the point cloud. 3d ; A series of 2D masks obtained by the YOLOv8 segmentation network 2d And a series of 3D detection boxes obtained by DBSCAN clustering 3d , and then the 2D mask box obtained by rotation transformation r2d Calculate the IOU intersection-of-union ratio one by one, and compare the IOU with the threshold to obtain the type and location of the object.

[0046] A. Calculation of the formula for converting json tags to txt tags

[0047] Suppose there are n detected object labels in an image, each label contains m points describing the contour, where m ≥ 3, and let the jth contour point of the i-th object in the label be Assume that the format of the converted contour point is The conversion formula is as follows:

[0048]

[0049] After obtaining the converted dataset, a random allocation algorithm is used to divide the dataset into training, validation, and test sets in a ratio of 4:2:1. The training parameters epoch is set to 500 and batch-size is set to 16. The YOLOv8 segmentation network is trained on a 3090 graphics card to obtain the segmentation model pt.

[0050] B. Lightweight the obtained model

[0051] The model obtained above was lightweighted. The pt model was converted to an onnx model (Open NeuralNetwork Exchange), and then to an openvino model. The image input size of the image segmentation model was defined as 640x640. The time-synchronized image dataset was tested to obtain a series of 2D masks describing the contours of the detected objects and their categories. The obtained 2D mask set is represented as follows:

[0052]

[0053] Among them, suppose there are n segmentation targets detected, each target has m contour points, and the set of detected targets is mask 2d .

[0054] C. CSF point cloud ground point filtering and detection box filtering

[0055] (1) Based on the external parameter rotation matrix, internal parameter rotation matrix and distortion matrix between the camera and the radar, the radar point cloud is projected onto the time-synchronized image. 2d , filter the point cloud according to the range of each 2D mask, and retain the point cloud that exists within the mask. The expression for calculating the rotation transformation of the point cloud projected to the image is as follows:

[0056]

[0057] Among them, Cx0, Cy0, u0, v0 are camera internal parameters, f is the camera focal length, R represents the rotation matrix, T represents the translation vector, s represents the scale factor, x, y, z are the point clouds to be projected, rx, ry are the image coordinates corresponding to the point cloud after projection. If rx, ry exist in the 2D mask set mask 2d If it is within the range, the corresponding point cloud will be retained, otherwise it will be filtered.

[0058] (2) Perform CSF point cloud ground point filtering on the point cloud after detection box filtering. Based on the curvature information of the points in the point cloud and the local features of the point cloud data, analyze the curvature values of each point's neighborhood and screen the ground points according to the curvature threshold. The steps are as follows: construct the storage structure of the point cloud data; calculate the curvature of each point in the stored point cloud and determine whether the point is a ground point according to the curvature threshold; for the points in the neighborhood determined to be ground points, perform reachability analysis to further filter out non-ground points. Repeat the above operations until all points are processed, thereby obtaining the ground point cloud and the non-ground point cloud;

[0059] D. DBSCAN clustering algorithm process

[0060] Use the DBSCAN clustering algorithm based on point cloud density to cluster the non-ground point cloud obtained above. First, use the KD-Tree algorithm to find the core points according to the number of neighborhood points, and then use the region growing method to perform unsupervised clustering on the point cloud to obtain a series of classified point clouds. Use the compute3DCentroid() algorithm of the PCL point cloud library to calculate the center points of the clustered objects, and at the same time use the PCL library function getMinMax3D() to obtain the minimum and maximum boundary points, and then calculate the bounding box to obtain a series of boxes 3d , assuming that n 3D bounding boxes are obtained, which are box0, box1,..., box n , and the expression form is as follows:

[0061] box 3d = {box0, box1,..., box n} (4)

[0062] E. Calculate the intersection over union (IOU) to obtain the category and location of the object

[0063] Assume that the obtained 3D bounding box can be converted into a 2D detection box box on the image through formula 3 2d , and the form expression is as follows:

[0064] box 2d = {box2d0, box2d1,..., box2d n} (5)

[0065] Assume that the i-th radar conversion detection box obtained, where 0 ≤ i < n and the j-th image detection box, where 0 ≤ j < n are:

[0066]

[0067] Among them, (x li , y li ), (x ri , yri ) represent the upper left corner and lower right corner of the radar point cloud conversion detection frame, respectively. Let S1 and S2 represent the areas of the above two detection frames, and the area of ​​the intersection is set to S t , then the calculation formula for the intersection-over-union (IOU) is:

[0068]

[0069] Design a suitable intersection-over-union (IOU) threshold. If the IOU is greater than the IOU threshold, the two detection frames are considered to intersect. Then, the category of the detected object is determined based on the image detection information, and the position of the detected object is determined using the radar point cloud detection frame.

[0070] The collaborative perception fusion detection method based on YOLOv8 segmentation network and DBSACN clustering, such as Figure 1 As shown, the following steps are included:

[0071] The first step is to convert the json tags of the traffic dataset and verify the converted tags with the corresponding pictures. Figure 2 As shown, the left picture shows the position of the detected object label on the original picture, and the right picture is the original picture. The converted dataset is trained using the YOLOv8 segmentation network and the YOLOv8 network to obtain the original model, and the accuracy and recall rates of the two network training models are compared to obtain the following results: Figure 3 and Figure 4 The comparison curves shown in the figure show that the accuracy of the original network and the improved network are 71.34% and 51.23% respectively, and the recall rates are 42.38% and 40.71% respectively. It can be seen that the values ​​of the segmentation network are higher, and the growth rate of the values ​​is relatively slow compared with the original network, which highlights the superiority of the algorithm proposed in this invention. Figure 5 The detection results shown in the image display the category of the detected object, 2D mask, and accuracy.

[0072] In the second step, the original point cloud is filtered by using the methods of radar-visual fusion, detection frame filtering, and ground point filtering. Figure 6 As shown in the figure, the four figures are the original point cloud, the point cloud from the camera perspective after the radar fusion filter, the point cloud after the detection frame filter, and the point cloud after the ground point filter. It can be seen that the number of point clouds decreases in sequence, and the number of point clouds is 62421, 22422, 13001, and 1945 respectively. Figure 6 Then, DBSCAN clustering is performed on the non-ground point cloud after ground point filtering to finally obtain a series of 3D detection frames of different colors as shown in Figure 7 shown.

[0073] The 3D detection frame is rotated according to the relative position relationship between the camera and the radar to obtain a series of 2D back-projection detection frames with rotational transformations. The 2D mask obtained by YOLOv8 detection on the image is calculated with the 2D back-projection detection frame of the back-projection transformation. The IOU is then matched one by one to determine the type and position of the detected target. The two detection frames, the type of the detected target, and the distance of the detected object relative to the radar coordinate system are displayed on the 2D image as shown below. Figure 8 As shown in the figure, the upper part is the detection picture of the YOLOv8 segmentation network, and the lower part is the picture of the object category and object position detected by fusion of 2D mask and 2D back projection detection frame.

[0074] References

[0075] [1] Zhao, Xu, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, MingTang, and **qiao Wang. "Fast segment anything." arxiv preprint arxiv:2306.12156(2023).

[0076] [2] Ye Shiqiu. Research on DBSCAN clustering algorithm based on locality sensitive hashing[D]. Chongqing University, 2019.

[0077] [3] Yang, Guoliang, Jixiang Wang, Ziling Nie, Hao Yang, and Shuaiying Yu. "A lightweight YOLOv8 tomato detection algorithm combining feature enhancement and attention." Agronomy 13, no. 7 (2023): 1824.

Claims

1. A fusion detection method based on YOLOv8 segmentation network and DBSACN clustering, characterized in that: This includes preprocessing the traffic dataset, extracting datasets of specific categories, and converting the format of dataset labels. A segmentation network based on YOLOv8 is used to train the preprocessed dataset, and after model lightweighting, a model file is finally generated. The model is used to detect the time-synchronized image and generate a 2D mask. The noise point cloud is filtered out using detection box filtering and ground point filtering. The density-based DBSCAN clustering algorithm is used to cluster the filtered point cloud to obtain a 3D detection box. Back-projection is performed to obtain a new 2D mask, which is matched with the 2D mask obtained from image detection to determine the type of the detected object and its 3D coordinates in 3D space using the vehicle body radar as the coordinate system. The specific steps are as follows: (1) First, the label types of the traffic annotation dataset are extracted, and the labels for semantic segmentation are screened. The 33 classes in the dataset are screened and 8 classes are extracted, namely pedestrians, cyclists, cars, trucks, buses, trains, motorcycles, and bicycles. The label information of each image contains the outline of the object, that is, the points contained in the shape contour parameters in the label file. Suppose there are n labels of detected objects in an image, and each label contains m points describing the outline, where m ≥ 3. Suppose the jth outline point of the i-th object in the label Assume that the format of the converted contour point is The conversion formula is as follows: After obtaining the converted data set, a random allocation algorithm is used to divide the data set into training set, validation set, and test set in a ratio of 4:2:

1. The number of training times is set, and the number of data samples captured during one training process is 16. The YOLOv8 segmentation network is trained on a GPU to obtain the segmentation model pt. Compared with the original YOLOv8 network, the YOLOv8 segmentation network makes the following changes: the C2f module is replaced with the C3 module at the head of the neural network; the 3-6-9-3 structure in the original network is converted to the 3-6-6-3 structure in the segmentation network; the first convolution in the FPN structure at the neck of the neural network is removed; and the SiLU activation function is used. (2) The model obtained above is lightweighted, and the pt model is converted to an onnx model, and then to an openvino model. At the same time, the image input size of the image segmentation model is defined as 640×640, and the time-synchronized image dataset is detected to obtain a series of 2D masks and categories describing the contours of the detected objects. The obtained 2D mask set is expressed as follows: Among them, suppose there are n segmentation targets detected, each target has m contour points, and the set of detected targets is mask 2d ; (3) The time-synchronized radar point cloud dataset is read frame by frame, and the CSF point cloud ground point filter and the mask set obtained above are used to perform detection frame filtering on the original point cloud according to the relative position relationship between the radar and the camera, and only the point cloud associated with the detection mask in the non-ground point cloud is retained; (4) The processed filtered point cloud is clustered based on DBSCAN, in which the KD-Tree algorithm is used to speed up the search for nearby points, and the PCL library is used to calculate the center of mass and the boundary of the point cloud to determine the 3D detection box of the point cloud. 3d ; (5) A series of 2D masks obtained by the YOLOv8 segmentation network 2d And a series of 3D detection boxes obtained by DBSCAN clustering 3d , and then obtain the 2D mask box through rotation transformation r2d , and calculate the IOU intersection-of-union ratio one by one, and compare it with the threshold to get the type and position of the object.

2. The fusion detection method according to claim 1, characterized in that: The specific steps of CSF point cloud ground point filtering and detection box filtering in step (3) are as follows: (1) According to the external parameter rotation matrix, internal parameter rotation matrix and distortion matrix between the camera and radar, the radar point cloud is projected onto the time-synchronized image; according to the 2D mask set obtained by recognition 2d , filter the point cloud according to the range of each 2D mask, and retain the point cloud within the mask; the expression for calculating the rotation transformation of the point cloud projection to the image is as follows: Among them, cx0, cy0, u0, v0 are camera intrinsic parameters, f is the camera focal length, R represents the rotation matrix, T represents the translation vector, s represents the scale factor, x, y, z are the point clouds to be projected, rx, ry are the image coordinates corresponding to the point cloud after projection; if rx, ry exists in the 2D mask set mask 2d If it is within the range, the corresponding point cloud will be retained, otherwise it will be filtered; (2) CSF ground point filtering is performed on the point cloud that has been filtered by the detection frame. Based on the curvature information of the points in the point cloud and the local characteristics of the point cloud data, the curvature value of the neighborhood of each point is analyzed, and the ground points are filtered according to the curvature threshold. The steps are as follows: construct a storage structure for point cloud data; calculate the curvature of each point in the stored point cloud, and determine whether the point is a ground point according to the curvature threshold; for the points in the neighborhood determined to be ground points, perform reachability analysis to further filter out non-ground points; repeat the above operations until all points are processed, thereby obtaining ground point cloud and non-ground point cloud.

3. The fusion detection method according to claim 2, characterized in that: The DBSCAN clustering method described in step (4) is specifically: First, the KD-Tree algorithm is used to find the core point according to the number of neighborhood points. Then, the region growing method is used to perform unsupervised clustering on the point cloud to obtain a series of classified point clouds. The PCL point cloud library algorithm compute3DCentroid() is used to calculate the center point of the clustered object. At the same time, the minimum and maximum boundary points are obtained from the PCL library function getMinMax3D(), and then the bounding box is calculated to obtain a series of box 3d , suppose we get n 3D bounding boxes, box0, box1, ..., box n , the expression is as follows: box 3d ={box0,box1,...,box n } (4)。 4. The fusion detection method according to claim 3, characterized in that: The specific process of step (5) is as follows: The obtained 3D bounding box is converted into a 2D detection box on the image through formula 3 2d , the formal expression is as follows: box 2d ={box2d0,box2d1,...,box2d n } (5) Suppose the obtained $i$-th radar conversion detection box, where $0\leq i < n$, and the $j$-th image detection box, $0\leq j < n$, are as follows: Among them, (x li ,y li ),(x ri ,y ri ) represent the upper left corner and lower right corner of the radar point cloud conversion detection frame, let S1 and S2 represent the area of ​​the above two detection frames, and the area of ​​the intersection is set to S t , then the calculation formula for the intersection-over-union (IOU) is: Design an appropriate Intersection over Union (IoU) threshold. If the IoU is greater than the IoU threshold, it is considered that the two detection boxes intersect. Then, determine the category of the detected object based on the image detection information and determine the position of the detected object using the radar point cloud detection box.

Images