A multimodal object detection ensemble method based on joint probability and entropy normalization

Abstract

The present invention discloses a multimodal target detection integration method based on joint probability and entropy normalization, the method comprising the following steps: 1. initializing a detector output set and an integrated detection set; 2. inter-modality projection; 3. entropy normalization; 4. confidence integration; 5. detection frame integration; 6. looping. By integrating the outputs of trained detectors, the present invention provides more accurate detection results than traditional single-modality detectors without requiring additional training costs, and properly handles the situation where traditional integration methods may fail when a modality is missing. In addition, when the extrinsic parameter calibration of the multimodal detector is not ideal, it provides higher detection accuracy than traditional integration methods, plays a positive role in improving the reliability and safety of target detection tasks, reduces the risk of accidents caused by false detection and missed detection, and is of great significance for improving the reliability and accuracy of target detection in cost-constrained engineering tasks.

Description

Technical Field

[0001] The present invention belongs to the field of target detection in computer vision and relates to a target detection method, in particular to a multimodal target detection integration method based on joint probability and entropy normalization. Background Art

[0002] With the rapid advancement of computer technology, object detection technology is increasingly being applied to various environmental perception tasks, enabling safer and more efficient implementation. Deep learning-based object detection methods have reached a relatively mature stage, achieving excellent accuracy across a wide range of datasets. However, training and optimizing complex models on increasingly large datasets demands increasing time and hardware resources, making it increasingly difficult to train and deploy deep learning models in low-cost, domain-specific engineering projects. Reducing model training costs and improving model detection accuracy are currently key challenges in object detection engineering applications. Ensemble methods combine the strengths of multiple models to achieve superior generalization performance, reduce model training bias, and improve robustness.

[0003] Traditional ensemble methods typically rely on a pre-set Intersection over Union (IoU) threshold to determine whether overlapping objects are the same instance. However, in multimodal object detection, bumps and vibrations can degrade the accuracy of extrinsic calibration between sensors over time. This loss of accuracy can further lead to false detections when processing multimodal data. This can lead to falsely classifying detection boxes provided by different models as different instances due to the small IoU between them, resulting in false positives. This frequent problem in engineering currently has no effective solution. Summary of the Invention

[0004] To reduce the learning and training costs of target detection models and address the reduced detection accuracy of traditional ensemble methods in multimodal target detection tasks due to external parameter calibration errors or fixed IoU thresholds, the present invention provides a multimodal target detection ensemble method based on joint probability and entropy normalization. This method integrates the confidence scores and detection boxes output by different target detection models using a joint probability method, uses probabilistic marginalization to address possible modality loss, and introduces entropy normalization to reduce false detections due to external parameter calibration errors. This improves the overall target detection accuracy and robustness, and is of great significance for improving the reliability and accuracy of target detection in cost-constrained engineering tasks.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A multimodal target detection ensemble method based on joint probability and entropy normalization includes the following steps:

[0007] Step 1: Initialize the detector output set D and the integrated detection set F, d i ={s i ,c i}∈D,d i ′={s i ′,c i ′}∈F, Among them, s i ,c i Represents the confidence and detection box coordinates output by the detector, s i ′,c i ′ represents the integrated confidence and detection frame coordinates, and the elements in D are divided by s i Arrange them from largest to smallest;

[0008] Step 2: Intermodal Projection: Based on the extrinsic matrix, the detection frame coordinates output by the lidar detector are converted from the 3D lidar coordinate system to the 2D camera coordinate system. The lidar points are then projected into pixels based on the camera's focal length in the x- and y-axes and the origin of the camera coordinate system. The specific projection method is as follows:

[0009] If you want to convert a lidar point Projected to the camera plane, first transform the lidar coordinate system to the camera coordinate system:

[0010]

[0011] in, Indicates point L i 3D coordinates in the laser radar coordinate system, Indicates point L i The 3D coordinates in the camera coordinate system, T is the external parameter matrix, and then the 3D coordinates are converted to 2D coordinates:

[0012]

[0013] Among them, (u i ,v i ) is the pixel coordinate of the lidar point on the camera plane, (f x ,f y ) is the focal length of the camera in the x-axis and y-axis directions in pixel units, (c x ,c y ) is the origin of the camera coordinate system. So far, the projection of the lidar point to the camera pixel point is completed;

[0014] Step 3: Entropy normalization:

[0015] Find s from D i Maximum detection d max, and all detections d that intersect with its detection box i Add to set I and normalize entropy according to the confidence ratio of elements in I:

[0016]

[0017] Among them, H γ is the expected entropy value, H1 is the entropy value when γ = 1, γ is the value to be solved, and p i is the probability distribution of the output result of the i-th detector, and E represents the expectation;

[0018] Step 4: Confidence Integration:

[0019] For all the values ​​in I that are equal to d max The detections whose IoU exceeds the threshold are compared with their confidence scores d max integrated:

[0020] c i ∝(p(f|x1)p(f|x2)) γ

[0021] Among them, p(f|x i ) represents the conditional probability distribution of the i-th detector;

[0022] Step 5: Detection frame integration:

[0023] Integrate the coordinates of the detection boxes that are greater than the IoU threshold:

[0024]

[0025] Where f represents a continuous random variable consisting of the centroid, width and height of the detection box, μ i Represents the detection box coordinates, is the variance of the distribution;

[0026] Step 6: Loop:

[0027] d max Remove from D and add d i ′={s i ′,c i '} to F, and repeat steps 1 to 5 until D is empty.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] In response to the increasing training costs of deep learning methods and the unsatisfactory performance of traditional integration methods on multimodal data, the present invention introduces the joint probability of confidence and detection box during independent distribution detection, and combines entropy normalization to approximate the probability distribution between different modalities, thereby proposing a multimodal target detection integration method based on joint probability and entropy normalization. By integrating the outputs of trained detectors, the present invention provides more accurate detection results than traditional single-modal detectors without requiring additional training costs, and properly handles the situation where traditional integration methods may fail when the modality is missing. In addition, when the external parameter calibration of the multimodal detector is not ideal, it provides higher detection accuracy than traditional integration methods, which plays a positive role in improving the reliability and safety of target detection tasks and reducing the risk of accidents caused by false detection and missed detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The overall flow chart of the multimodal target detection integration method based on joint probability and entropy normalization. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0032] The present invention provides a multimodal target detection integration method based on joint probability and entropy normalization, comprising the following steps:

[0033] Step 1: Projection between different modalities:

[0034] Step 1. Calculate the external parameter matrix between modes:

[0035] The extrinsic matrix describes the transformation from the world coordinate system to the camera coordinate system, defining the position and orientation of the sensor in the world coordinate system. The form of the extrinsic matrix T is generally similar to:

[0036]

[0037] Among them, (U, V, W) is the world coordinate system, (R, T N ) are rotation and translation transformations;

[0038] Step 1 and 2: Projection of points between different modes:

[0039] If you want to convert a lidar point Projected to the camera plane, first transform the lidar coordinate system to the camera coordinate system:

[0040]

[0041] in, Indicates point L i 3D coordinates in the laser radar coordinate system, Indicates point L i The 3D coordinates in the camera coordinate system, T is the external parameter matrix, and then the 3D coordinates are converted to 2D coordinates:

[0042]

[0043] Among them, (u i ,v i ) is the pixel coordinate of the lidar point on the camera plane, (f x ,f y ) is the focal length of the camera in the x-axis and y-axis directions in pixel units, (c x ,c y ) is the origin of the camera coordinate system. So far, the projection of the lidar point to the camera pixel point is completed;

[0044] Step 2: Entropy normalization to process external parameter errors:

[0045] Step 21. Definition of entropy:

[0046] For the distribution of n detector results p1,p2,…,p n , their entropy H is defined as:

[0047] H=-Σp i logp i =E[-logp i ]

[0048] Among them, p i is the probability distribution of the output result of the i-th detector, and E represents the expectation;

[0049] P i Use a power transformation:

[0050]

[0051] in, is the power transformation p i The probability distribution of , γ is the value to be solved, which can make the entropy a specified value;

[0052] The power transformation ensures the monotonicity of the distribution while preserving the main information of the distribution, and the entropy decreases monotonically with respect to γ;

[0053] Step 22: Iteration of entropy:

[0054] Define the expected entropy value H γ :

[0055]

[0056] Expand it at γ=1:

[0057]

[0058] Right now:

[0059]

[0060] Where H1 is the entropy value when γ = 1;

[0061] Starting from γ=1, we can iterate the above formula to get the desired entropy value γ, then the updated joint probability distribution can be expressed as:

[0062]

[0063] in, represents the joint probability distribution of n detector results, p(f|x i ) represents the conditional probability distribution of the i-th detector, represents the conditional probability distribution of the i-th detector after power transformation;

[0064] Entropy normalization approximates the distribution between different detectors by softening overconfident estimates, thereby alleviating the degradation of target detection accuracy when the inter-modality extrinsic parameter calibration is not ideal;

[0065] Step 3: Confidence Integration

[0066] Step 31: Integration object screening:

[0067] Assume that for the same label y, there are n detectors that output x1, x2, ..., x n , select the detection x with the highest confidence score max , according to a pre-set threshold (usually 0.5), and x max Detection results whose IoU is less than the threshold do not participate in the integration process;

[0068] Step 3.2: Give the conditional independence formulas for different detector outputs:

[0069] Since each detector is trained independently, their conditions are independent, so for the two detector outputs x1, x2:

[0070]

[0071] Where p(y) is the prior distribution for label y;

[0072] Extend it to n detectors participating in the ensemble:

[0073]

[0074] This method multiplies the conditional probabilities of each individual detector and divides them by the class prior distribution. The final result is normalized to produce a final score confidence. In this process, the method handles the missing modalities through probabilistic marginalization, effectively solving the problem that traditional ensemble methods may fail when faced with missing modalities.

[0075] Step 3. Calculate the final score based on the logit score:

[0076] Use logit score to represent the softmax posterior distribution:

[0077]

[0078] Among them, s i [k] represents the logit score of the k-th class label under the i-th detector; the above formula gives the calculation method of confidence integration, which is equivalent to adding the logit scores of each detector. When more detectors are consistent, this method can make the detection more convincing and use more stable values ​​for comparison between detectors;

[0079] Step 4: Detection frame integration:

[0080] Step 4.1 Parameterization of detection box:

[0081] For the detection results involved in the confidence integration, f is used to represent the continuous random variable consisting of the centroid, width and height of the detection box, and it is assumed that the detector provides a posterior normal distribution where μ i Represents the detection box coordinates, is the variance of the distribution;

[0082] Step 4.2: Calculation of detection box integration:

[0083] The calculation formula for detection box integration is as follows:

[0084]

[0085] This method integrates the detection frame coordinates by calculating the weighted average, where the weight is the inverse covariance, generally using The denominator is the conditional probability distribution of the result of the i-th detector when the label is y, which means that when the detection box is integrated, the detection with higher confidence has a higher weight.

[0086] like Figure 1 As shown, the specific steps of the method are as follows:

[0087] Step 1: Initialize the detector output set D and the integrated detection set F, d i ={s i ,c i}∈D,d i ′={s i ′,c i ′}∈F, Among them, s i ,c i Represents the confidence and detection box coordinates output by the detector, s i ′,c i ′ represents the confidence and detection frame coordinates after integration of this method. Each element in the set contains two attributes: confidence and detection frame coordinate distribution. At this time, F is an empty set. The elements in D are divided into s i Arrange them from largest to smallest.

[0088] Step 2: Intermodal Projection: Based on the extrinsic matrix, the detection frame coordinates output by the lidar detector are converted from the 3D lidar coordinate system to the 2D camera coordinate system. The lidar points are then projected into pixels based on the camera's focal length in the x- and y-axis directions and the origin of the camera coordinate system.

[0089] Step 3: Entropy normalization:

[0090] Find s from D i Maximum detection d max , and all detections d that intersect with its detection box i Add to set I and normalize entropy according to the confidence ratio of elements in I:

[0091]

[0092] Step 4: Confidence Integration:

[0093] For all the values ​​in I that are equal to d max The detections whose IoU exceeds the threshold (usually set to 0.5) are compared with their confidence scores d max integrated:

[0094] c i ∝(p(f|x1)p(f|x2)) γ

[0095] Step 5: Detection frame integration:

[0096] Integrate the coordinates of the detection boxes that are greater than the IoU threshold:

[0097]

[0098] Step 6: Loop:

[0099] d max Remove from D and add d i ′={s i ′,c i '} to F, and repeat steps 1 to 5 until D is empty.

[0100] Example 1:

[0101] This example validates the method presented in this paper on the single-modal public VOC dataset. We selected 10,870 images containing five common road labels: bicycle, bus, car, motorbike, and person. The method uses Mask R-CNN and Faster R-CNN as the ensemble baseline, and mAP as the evaluation metric. The results are shown in Table 1.

[0102] Table 1 Implementation results on the VOC dataset

[0103]

[0104] Results show that, in single-modal datasets, this method significantly improves detection accuracy compared to traditional ensemble methods. Furthermore, the method can achieve an inference speed of 20 fps on an Nvidia AGX Xavier, meeting the typical 10 fps real-time requirement for object detection.

[0105] Example 2:

[0106] This example validates the method of the present invention on the multimodal public KITTI dataset. Three common road labels, Car, Pedestrian, and Cyclist, were selected for the experiment, totaling 7,518 frames. The method uses PointPillars and Faster R-CNN as the ensemble baseline, and mAP as the evaluation metric. The results are shown in Table 2:

[0107] Table 2 Implementation results on the KITTI dataset

[0108]

[0109]

[0110] Results show that, in multimodal datasets, this method significantly improves detection accuracy compared to traditional ensemble methods. Furthermore, the method can achieve an inference speed of 14 fps on an Nvidia AGX Xavier, meeting the typical real-time requirement of 10 fps for object detection.

[0111] It can be seen from the above embodiments that the method of the present invention improves the problems of false detection and missed detection in the field of target detection compared with traditional integrated methods, has the ability to resist modality loss, and has higher accuracy and stronger robustness.

Claims

1. A multimodal target detection ensemble method based on joint probability and entropy normalization, characterized by The method comprises the following steps: Step 1: Initialize the detector output set D and the integrated detection set F, d i ={s i ,c i }∈D,d i ′={s i ′,c i ′}∈F, Among them, s i ,c i Represents the confidence and detection box coordinates output by the detector, s i ′,c i ′ represents the integrated confidence and detection frame coordinates, and the elements in D are divided by s i Arrange them from largest to smallest; Step 2: Intermodal Projection: Based on the extrinsic matrix, the detection frame coordinates output by the lidar detector are converted from the 3D lidar coordinate system to the 2D camera coordinate system. The lidar points are then projected into pixels based on the camera's focal lengths in the x- and y-axes and the origin of the camera coordinate system. Step 3: Entropy normalization: Find s from D i Maximum detection d max , and all detections d that intersect with its detection box i Add to set I and normalize entropy according to the confidence ratio of elements in I: Among them, H γ is the expected entropy value, H1 is the entropy value when γ = 1, γ is the value to be solved, and p i is the probability distribution of the output result of the i-th detector, and E represents the expectation; Step 4: Confidence Integration For all the values ​​in I that are equal to d max The detections whose IoU exceeds the threshold are compared with their confidence scores d max integrated: c i ∝(p(f|x1)p(f|x2)) γ Among them, p(f|x i ) represents the conditional probability distribution of the i-th detector; Step 5: Detection frame integration: Integrate the coordinates of the detection boxes that are greater than the IoU threshold: Where f represents a continuous random variable consisting of the centroid, width and height of the detection box, μ i Represents the detection box coordinates, is the variance of the distribution; Step 6: Loop: d max Remove from D and add d i ′={s i ′,c i '} to F, and repeat steps 1 to 5 until D is empty.

2. The multimodal target detection ensemble method based on joint probability and entropy normalization according to claim 1 is characterized in that In step 2, the extrinsic parameter matrix describes the transformation from the world coordinate system to the camera coordinate system, and defines the position and orientation of the sensor in the world coordinate system. The form of the extrinsic parameter matrix T is as follows: Among them, (U, V, W) is the world coordinate system, (R, T N ) are the rotation and translation transformations.

3. The multimodal target detection ensemble method based on joint probability and entropy normalization according to claim 1 is characterized in that In the step 2, the specific projection method is as follows: If you want to convert a lidar point Projected to the camera plane, first transform the lidar coordinate system to the camera coordinate system: in, Indicates point L i 3D coordinates in the laser radar coordinate system, Indicates point L i The 3D coordinates in the camera coordinate system, T is the external parameter matrix, and then the 3D coordinates are converted to 2D coordinates: Among them, (u i ,v i ) is the pixel coordinate of the lidar point on the camera plane, (f x ,f y ) is the focal length of the camera in the x-axis and y-axis directions in pixel units, (c x ,c y ) is the origin of the camera coordinate system. At this point, the projection of the lidar point to the camera pixel point is completed.

4. The multimodal target detection integration method based on joint probability and entropy normalization according to claim 1 is characterized in that In the step five, The denominator is the conditional probability distribution of the i-th detector result when the label is y.

Images