Multi-modal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion and electronic equipment thereof

Through the Gumbel distributed sampling and uncertainty fusion module, combined with point cloud and image features, the accuracy problem of traditional three-dimensional object detection technology under noise and environmental changes is solved, and more efficient and robust detection results are achieved.

CN118942082APending Publication Date: 2024-11-12SHANGHAI UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411041513.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Under the influence of noise, environmental changes and object occlusion, traditional three-dimensional object detection technology has problems such as sampling results rely on the initial point, lack of substitution invariance, sensitivity to outliers, and limited field of view, resulting in insufficient detection accuracy and reliability. .

Method used

Using a multimodal three-dimensional object detection method based on Gumbel distributed sampling, the point cloud data is distributedly sampled through the Gumbel-Softmax function, combined with the uncertainty fusion module, point cloud and image features are used for feature extraction and fusion, and the detection is improved. Awesome and accuracy.

Benefits of technology

This method effectively reduces the computational burden of subsequent processing, improves processing efficiency, significantly improves the robustness of the system and the accuracy of detection results, and is suitable for autonomous driving and a variety of point cloud processing tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942082A_ABST
    Figure CN118942082A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computer deep learning and target detection, in particular to a multi-modal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion, and the method comprises the steps: S1, obtaining an original point cloud data set and an original image of a scene containing a to-be-detected target; s2, performing distributed sampling on the original point cloud data set by using a Gumbel-Softmax function, and selecting representative point clouds from the original point cloud data set to obtain representative point cloud subsets; s3, performing feature extraction on the representative point cloud subset through a point cloud feature extraction model to obtain point cloud features; s4, performing feature extraction on the original image by using the image feature extraction model to obtain image features; s5, fusing the point cloud feature information and the image feature information by using an uncertainty fusion module to obtain fused features; s6, inputting the fusion features into a detection head, and carrying out the detection and recognition of a target. The target detection method is high in detection precision and good in robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer deep learning and target detection, and in particular to a multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion and an electronic device thereof. Summary of the invention

[0002] In the field of deep learning and computer vision, 3D object detection is a key technology, especially in the field of autonomous driving. Traditional 3D object detection technology mainly relies on LiDAR and cameras as the main sources of sensor data. These systems can directly provide the 3D coordinates of objects and rich semantic information. However, these sensor data are often affected by many factors, including the noise of the sensor itself, changes in environmental conditions, and occlusion of objects. These factors may have a significant impact on the accuracy of 3D object detection.

[0003] First, the farthest point sampling (FPS) method, which is widely used in traditional lidar point cloud sampling methods, has the following problems: 1. The sampling result depends on the initial point, that is, it does not have permutation invariance; 2. This method is task-dependent and designed for low-dimensional Euclidean space; 3. This method is extremely sensitive to outliers.

[0004] Secondly, in complex road conditions, the forward-facing LiDAR only provides a limited field of view, which may lead to incomplete object detection. For example, the reflectivity of some objects (such as black cars) may be lower than the detection threshold of the LiDAR, so the LiDAR point reflections of these objects may be insufficient. In addition, the camera module is often easily blocked due to weather conditions such as dust, which requires us to effectively manage and optimize the confidence of the sensor data to ensure the accuracy of detection and the reliability of the system.

[0006] In view of the problems and shortcomings in the prior art, the purpose of the present invention is to provide a multimodal three-dimensional target detection method and electronic device based on Gumbel distribution sampling and uncertainty fusion.

[0007] To achieve the purpose of the invention, the technical solution adopted by the present invention is as follows:

[0008] The first aspect of the present invention provides a multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion, and the specific steps are as follows:

[0009] S1: Get the original point cloud dataset of the scene containing the target to be detected and the original point cloud dataset X i The corresponding original image;

[0010] S2: Use the Gumbel-Softmax function to perform distribution sampling on the original point cloud data set, select representative point clouds from the original point cloud data set, and obtain a representative point cloud subset

[0011] S3: Extract features from the representative point cloud subset using a point cloud feature extraction model to obtain point cloud features F l ;

[0012] S4: Extract features from the original image using an image feature extraction model to obtain image features F c ;

[0013] S5: Using the uncertainty fusion module to transform the point cloud feature information F l With the image feature information F c Perform fusion to obtain fusion feature F;

[0014] S6: Input the fusion feature F into the detection head to detect and identify the target.

[0015] According to the above-mentioned multimodal 3D target detection method based on Gumbel distribution sampling and uncertainty fusion, preferably, in step S5, an uncertainty fusion module is used to transform the point cloud feature information F l With the image feature information F c The specific operations for fusion are:

[0016] S501: Based on the Euclidean distance function, the point cloud features F are calculated using the uncertainty score calculation formula. l , image features F c The uncertainty score The uncertainty score is calculated as follows:

[0017]

[0018] In the formula, f dist is a multi-layer perceptron that predicts the distance between the RoI and the ground-truth bounding box for each modality s; F s is the ROI feature of mode s, mode s is the point cloud feature F l Or image feature F c ; exp is the exponential function; is the uncertainty fraction of mode s.

[0019] S502: Cat feature connection operation and FFN feedforward neural network are used to construct a feature fusion equation, and the point cloud feature F is converted into l With the image feature F cThe fusion is performed to obtain the fusion feature F; wherein the formula of the feature fusion equation is as follows:

[0020]

[0021] In the formula, is the point cloud feature F l The uncertain fraction of is the image feature F c The uncertainty score.

[0022] According to the above-mentioned multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion, preferably, step S2 is specifically:

[0023] S201: Calculate the original point cloud dataset X using the weight matrix i The attention score a for each point cloud in i , the weight matrix contains learnable parameters for selecting representative point clouds, the attention score a i The calculation formula is as follows:

[0024] a i =W T X i

[0025] Where W T Represents the weight matrix

[0026] S202: The attention score a of each point cloud obtained in step S201 is i Input the Gumbel-Softmax function to get the probability of each point cloud being sampled, and then compare the probability of each point cloud being sampled with the original point cloud dataset X i Multiply them together to get the representative point cloud (i.e. the sampled point cloud); the calculation formula corresponding to this process is as follows:

[0027] y gumbel =Gumbel-Softmax(a i )·X i #

[0028] In the formula, y gumbel represents the representative point cloud obtained after sampling, Gumbel-Softmax represents the Gumbel-Softmax function, a i Represents the attention score.

[0029] According to the above-mentioned multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion, preferably, the Gumbel-Softmax function is a Softmax function with Gumbel distribution noise added, and the calculation formula of the Gumbel-Softmax function is as follows:

[0030]

[0031] Where y represents the probability distribution vector; τ is the annealing temperature parameter; g is the independent and identically distributed Gumbel noise drawn from the Gumbel (0,1) distribution; S represents the category distribution, s = (s1, s2, …, s M ), where M is the number of categories, s i is the probability of the ith category.

[0032] According to the above-mentioned multimodal 3D target detection method based on Gumbel distribution sampling and uncertainty fusion, preferably, step S4 is specifically: using the BEVFormer network model to extract features from the original image to obtain 2D image features, converting the 2D image features to a bird's-eye view to obtain image features F c .

[0033] According to the above-mentioned multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion, preferably, the point cloud feature extraction model in step S3 is a VoxelNet network model.

[0034] According to the above-mentioned multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion, preferably, the detection head in step S6 extracts information through two 3×3 convolutions and one 1×1 convolution, and finally regresses the position information and category information respectively.

[0035] According to the above-mentioned multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion, preferably, before using the Gumbel-Softmax function to distribute the original point cloud data set in step S2, it is necessary to train the Gumbel-Softmax function, and the initial value of τ in the calculation formula of the Gumbel-Softmax function during training is set to be greater than or equal to 5.0.

[0036] According to the above-mentioned multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion, preferably, the specific steps of Gumbel-Softmax function training are:

[0037] (1) Obtaining point cloud and image datasets

[0038] Acquire a sample set, wherein the sample set includes a plurality of mutually corresponding point cloud samples and image samples, wherein the point cloud samples contain location information and type information of the target, and the image samples contain location information and type information of the target, and randomly divide the sample set into a training set, a validation set, and a test set according to a proportion;

[0039] (2) Input the point cloud sample data in the training set into the Gumbel-Softmax function with an initial value of τ set to be greater than or equal to 5.0. At this time, the sampling probability of the point cloud is about 0.5. Select the point cloud with a sampling probability greater than 0.5 as the sampling result and input it into the pre-built VoxelNet network model to extract the feature of the point cloud and obtain the point cloud feature. Input the image sample data in the training set into the pre-built image feature extraction model (preferably the BEVFormer network model) to extract the image feature and obtain the image feature; input the point cloud feature and the image feature into the pre-built uncertainty fusion module for fusion to obtain the fusion feature; use the pre-built detection head to regress the position and type information of the target on the fusion feature: construct the loss function according to the obtained target position information and type information, and update the parameters in the Gumbel-Softmax function through the gradient feedback according to the value calculated by the loss function. At the same time, as the training progresses, the value of the Gumbel-Softmax function τ will gradually decrease, and the probability of point cloud adoption will be distributed in the range close to 0 or close to 1. Repeat the above steps. The training is completed when the value of the loss function no longer decreases and tends to be stable. The formula of the loss function is:

[0040]

[0041] Among them, α and β are weight coefficients used to balance the influence of position loss and category loss; the position loss uses smooth L1 loss to calculate the difference between the predicted bounding box and the true bounding box; p i represents the predicted bounding box coordinates, g i Represents the true bounding box coordinates, N represents the number of bounding boxes; the category loss uses the cross entropy loss (CrossEntropyloss) to calculate the difference between the predicted category probability and the true category, where y i represents the true category label, q i Represents the predicted class probability.

[0042] A second aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion as described in the first aspect.

[0043] A second aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion described in the first aspect is implemented.

[0044] Compared with the prior art, the present invention has the following effects:

[0045] (1) The present invention uses the Gumbel-Softmax function to perform distributed sampling on the original point cloud set. The Gumbel-Softmax function can generate a "soft" continuous subset in the training phase and generate a "hard" discrete subset through an annealing process in the test phase. This sampling method not only maintains the arrangement invariance of the data, but also can effectively select representative subsets from high-dimensional representations and select highly representative subsets from complex high-dimensional point cloud data. This strategy not only reduces the computational burden of subsequent processing steps, but also improves processing efficiency, enabling the system to respond to and process large-scale point cloud data more quickly.

[0046] (2) Improvement of the robustness of the system of the present invention: The uncertainty fusion module of the present invention uses the Euclidean distance function to accurately quantify the uncertainty of the modalities (point cloud features, image features) from different sensors (radar, camera), and performs weighted fusion of the features according to the uncertainty of each modality, which significantly improves the stability and reliability of the system in the face of sensor noise or data loss, ensures the accuracy of the detection results, and greatly improves the robustness of the target detection method of the present invention.

[0047] (3) The target detection method of the present invention does not depend on specific tasks or application scenarios. Its versatility makes it suitable for a variety of point cloud processing tasks such as autonomous driving, robot navigation, and environmental monitoring. At the same time, its modular design enables the components to be easily integrated into existing systems or used in combination with other technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the process flow of the multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion of the present invention;

[0049] Figure 2 Schematic diagram of the process of distributed sampling of the original point cloud data set using the Gumbel-Softmax function in the method of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the present invention will be further described in detail below in conjunction with specific embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0051] Embodiment 1:

[0052] The first aspect of the present invention provides a multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion, the specific steps (such as Figure 1 As shown) as follows:

[0053] S1: Use LiDAR or other 3D sensor systems to obtain the original point cloud dataset of the scene containing the target to be detected The camera device is used to obtain an original image of the scene containing the target to be detected, and the original image is compared with the original point cloud dataset X i Corresponding.

[0054] S2: Use the Gumbel-Softmax function to perform distribution sampling on the original point cloud data set, select representative point clouds from the original point cloud data set, and obtain a representative point cloud subset

[0055] S3: Extract features from the representative point cloud subset using a point cloud feature extraction model to obtain point cloud features F l ; Preferably, the point cloud feature extraction model can adopt the VoxelNet network model.

[0056] S4: extract features from the original image using an image feature extraction model to obtain two-dimensional image features, and convert the two-dimensional image features to a bird's-eye view to obtain image features F c ; Preferably, the image feature extraction model is a BEVFormer network model.

[0057] S5: Using the uncertainty fusion module to transform the point cloud feature information F l With the image feature information F c Perform fusion to obtain fusion feature F;

[0058] S6: Input the fusion feature F into the detection head to detect and identify the target and / or locate the target, and further obtain the confidence kernel target frame of the target.

[0059] Among them, the specific operation of step S2 (such as Figure 2 shown) is:

[0060] S201: Calculate the original point cloud dataset X using the weight matrix i The attention score a for each point cloud in i , the weight matrix contains learnable parameters for selecting representative point clouds, the attention score a i The calculation formula is as follows:

[0061] a i =W T X i

[0062] Where W T represents the weight matrix;

[0063] S202: The attention score a of each point cloud obtained in step S201 is i Input the Gumbel-Softmax function to get the probability of each point cloud being sampled, and then compare the probability of each point cloud being sampled with the original point cloud dataset X i Multiply them together to get the representative point cloud (i.e. the sampled point cloud); the calculation formula corresponding to this process is as follows:

[0064] y gumbel =Gumbel-Softmax(a i )·X i #

[0065] In the formula, y gumbel represents the representative point cloud obtained after sampling, a i represents the attention score, and Gumbel-Softmax represents the Gumbel-Softmax function. The Gumbel-Softmax function is a Softmax function with Gumbel distribution noise added. The calculation formula of the Gumbel-Softmax function is as follows:

[0066]

[0067] Where y represents the probability distribution vector; τ is the annealing temperature parameter; g is the independent and identically distributed Gumbel noise drawn from the Gumbel (0,1) distribution; S represents the category distribution, s = (s1, s2, ..., s M ), where M is the number of categories, s i is the probability of the ith category.

[0068] Among them, the specific steps of Gumbel-Softmax function training are:

[0069] (1) Obtaining point cloud and image datasets

[0070] Acquire a sample set, wherein the sample set includes a plurality of mutually corresponding point cloud samples and image samples, wherein the point cloud samples contain location information and type information of the target, and the image samples contain location information and type information of the target, and randomly divide the sample set into a training set, a validation set, and a test set according to a proportion;

[0071] (2) Input the point cloud sample data in the training set into the Gumbel-Softmax function with an initial value of τ set to be greater than or equal to 5.0. At this time, the sampling probability of the point cloud is about 0.5. Select the point cloud with a sampling probability greater than 0.5 as the sampling result and input it into the pre-built VoxelNet network model to extract the feature of the point cloud and obtain the point cloud feature. Input the image sample data in the training set into the pre-built image feature extraction model (preferably the BEVFormer network model) to extract the image feature and obtain the image feature; input the point cloud feature and the image feature into the pre-built uncertainty fusion module for fusion to obtain the fusion feature; use the pre-built detection head to regress the position and type information of the target on the fusion feature: construct the loss function according to the obtained target position information and type information, and update the parameters in the Gumbel-Softmax function through the gradient feedback according to the value calculated by the loss function. At the same time, as the training progresses, the value of the Gumbel-Softmax function τ will gradually decrease, and the probability of point cloud adoption will be distributed in the range close to 0 or close to 1. Repeat the above steps. The training is completed when the value of the loss function no longer decreases and tends to be stable. The formula of the loss function is:

[0072]

[0073] Among them, α and β are weight coefficients used to balance the influence of position loss and category loss; the position loss uses smooth L1 loss to calculate the difference between the predicted bounding box and the true bounding box; p i represents the predicted bounding box coordinates, g i Represents the true bounding box coordinates, N represents the number of bounding boxes; the category loss uses the cross entropy loss (CrossEntropyloss) to calculate the difference between the predicted category probability and the true category, where y i represents the true category label, q i Represents the predicted class probability.

[0074] In step S5, the point cloud feature information F is fused using an uncertainty fusion module. l With the image feature information F c The specific operations for fusion are:

[0075] S501: Based on the Euclidean distance function, the point cloud features F are calculated using the uncertainty score calculation formula. l , image features F c The uncertainty score The uncertainty score is calculated as follows:

[0076]

[0077] In the formula, f dist is a multi-layer perceptron that predicts the distance between the RoI and the ground-truth bounding box for each modality s; F s is the ROI feature of mode s, mode s is the point cloud feature F l Or image feature F c ; exp is the exponential function; is the uncertainty fraction of mode s.

[0078] The derivation process of the uncertainty score calculation formula in the present invention is as follows: The uncertainty score is based on the Euclidean distance D between the predicted bounding box and the true bounding box. xy is defined based on each modal feature F s (Image feature F c Or point cloud feature F l ), the uncertainty score can be expressed as In the formula, f reg is the function used to regress the bounding box, B is the true bounding box, and F s is the RoI (Region of Interest) feature from modality s, modality s is the point cloud feature F l Or image feature F c , exp is an exponential function. Since the position of the real bounding box is unknown to the model, the present invention further uses a multi-layer perceptron to predict the distance between the RoI of each modality and the real bounding box, thereby obtaining the uncertainty score calculation of the present application:

[0079] S502: Cat feature connection operation and FFN feedforward neural network are used to construct a feature fusion equation, and the point cloud feature F is converted into l With the image feature F c The fusion is performed to obtain the fusion feature F; wherein the formula of the feature fusion equation is as follows:

[0080]

[0081] In the formula, is the point cloud feature F l The uncertain fraction of is the image feature Fc The uncertainty score.

[0082] Embodiment 2:

[0083] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion as described in the above-mentioned embodiment 1 is implemented.

[0084] Embodiment 3:

[0085] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion as described in the above-mentioned embodiment 1.

[0086] In order to verify the detection performance of the target detection method of the present invention,

[0087] The multimodal 3D target detection method of the present invention is used to perform simple, medium and difficult detection on the KITTI official test set. The average precision (AP) of the target (the target is a car) detection is used as the evaluation index. The Nvidia GeForce 3090*8 graphics card is used to perform experiments in the operating system Ubuntu 18.04, Python 3.8, and Pytorch 1.8.0 environment. At the same time, in order to compare with the target detection method of the present invention, the present invention also uses the existing known model to detect the KITTI official test set. The experimental results are shown in Table 1.

[0088] Table 1 is a performance comparison between the target detection method of the present invention and the existing detection model

[0089]

[0090] As can be seen from Table 1, for automobile target detection in three-dimensional detection, the detection accuracy of the target detection method of the present invention at the simple, medium and difficult levels reached 94.80%, 91.76% and 90.49% respectively, which is significantly higher than the existing three-dimensional target detection model. This excellent performance is attributed to the unique feature extraction and fusion mechanism of the target detection method of the present invention, which can more effectively handle target detection problems in complex scenes.

[0091] In order to further verify the effectiveness of the distributed sampling using the Gumbel-Softmax function in the target detection method of the present invention, an ablation experiment was performed on the nuScenes dataset and compared with the existing FPS sampling. The results are shown in Table 2 below.

[0092] Table 2 shows the ablation experiment results of the nuScenes dataset

[0093] method mAP NDS Inference time FPS 55.3 54.2 25ms Gumbel-Softmax function of the present invention 57.3 55.3 20ms

[0094] As can be seen from Table 2, we can see that the use of Gumbel sampling distribution not only improves mAP by 2%, but also improves NDS and Inference time by 0.9% and 5ms respectively, proving that this method can not only improve model performance, but also improve inference speed and processing efficiency.

[0095] Finally, it should be noted that the above embodiments are only preferred embodiments of the present invention, and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the above technical content as inspiration to make changes or modifications. This is an equivalent embodiment of equivalent changes. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical concept of the present invention still fall within the scope of protection of the claims of the present invention.

Claims

1. A multimodal 3D object detection method based on Gumbel distribution sampling and uncertainty fusion, It is characterized in that The following steps are involved: S1: Get the original point cloud dataset X of the scene containing the target to be detected i and the original image; S2: Use the Gumbel-Softmax function to perform distribution sampling on the original point cloud data set, select representative point clouds from the original point cloud data set, and obtain a representative point cloud subset X i+1 ; S3: Extract features from the representative point cloud subset using a point cloud feature extraction model to obtain point cloud features F l ; S4: Extract features from the original image using an image feature extraction model to obtain image features F c ; S5: Using the uncertainty fusion module to transform the point cloud feature information F l With the image feature information F c Perform fusion to obtain fusion feature F; S6: Input the fusion feature F into the detection head to detect and identify the target.

2. The multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion according to claim 1, It is characterized in that In step S5, the uncertainty fusion module is used to transform the point cloud feature information F l With the image feature information F c The fusion is as follows: S501: Based on the Euclidean distance function, the point cloud features F are calculated using the uncertainty score calculation formula. l , image features F c The uncertainty score The uncertainty score is calculated as follows: In the formula, f dist is a multi-layer perceptron that predicts the distance between the RoI and the ground-truth bounding box for each modality s; F s is the ROI feature of mode s, mode s is the point cloud feature F l Or image feature F c ; exp is the exponential function; is the uncertainty fraction of mode s; S502: Cat feature connection operation and FFN feedforward neural network are used to construct a feature fusion equation, and the point cloud feature F is converted into l With the image feature F c Fusion is performed; wherein the formula of the feature fusion equation is as follows: In the formula, is the point cloud feature F l The uncertain fraction of is the image feature F c The uncertainty score.

3. The multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion according to claim 2, It is characterized in that Step S2 is specifically as follows: S201: Calculate the original point cloud dataset X using the weight matrix i The attention score a for each point cloud in i , the attention score a i The calculation formula is as follows: a i =W T X i Where W T represents the weight matrix; S202: The attention score a of each point cloud obtained in step S201 is i Input the Gumbel-Softmax function to get the probability of each point cloud being sampled, and then compare the probability of each point cloud being sampled with the original point cloud dataset X i Multiply them together to get the representative point cloud; the calculation formula corresponding to this process is as follows: y gumbel =Gumbel-Softmax(a i )·X i # In the formula, y gumbel represents the representative point cloud obtained after sampling, Gumbel-Softmax represents the Gumbel-Softmax function, a i Represents the attention score.

4. The multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion according to claim 3, It is characterized in that The Gumbel-Softmax function is a Softmax function with Gumbel distribution noise added. The calculation formula of the Gumbel-Softmax function is as follows: Where y represents the probability distribution vector; τ is the annealing temperature parameter; g is the independent and identically distributed Gumbel noise drawn from the Gumbel (0,1) distribution; S represents the category distribution, s = (s 1 ,s 2 ,…,s M ), where M is the number of categories, s i is the probability of the ith category.

5. The multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion according to any one of claims 1 to 4, It is characterized in that Step S4 specifically includes: using the BEVFormer network model to extract features from the original image to obtain two-dimensional image features, converting the two-dimensional image features to a bird's-eye view to obtain image features F c .

6. The multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion according to claim 5, It is characterized in that The point cloud feature extraction model in step S3 is a VoxelNet network model; the detection head in step S6 extracts information through two 3×3 convolutions and one 1×1 convolution, and finally regresses the position information and category information respectively.

7. An electronic device comprising a memory and a processor, It is characterized in that The memory stores a computer program, and when the processor executes the computer program, the multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the multimodal three-dimensional target detection method based on Gumbel distribution sampling and uncertainty fusion as described in any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Game model plagiarism detection method based on geometric features of three-dimensional model

    CN120411379A

  • A game model plagiarism detection method based on three-dimensional model geometric features

    CN120411379B

  • Target detection method and device based on uncertainty and automatic driving vehicle

    CN120526405A