Robust multi-modal object detection method and device based on uncertainty

Abstract

The application discloses a kind of robust multimodal target detection method and device based on uncertainty, method includes: pre-training different modal target detector;RGB image and infrared image are input to each target detector, and the prediction detection frame and classification result of each mode are obtained;The uncertainty estimation result of the prediction detection frame of each mode is determined;According to the IoU information of each prediction detection frame, the uncertainty of prediction detection frame is given to the corresponding true value frame, and the uncertainty perception true value is obtained;Finally, according to the uncertainty perception true value, the target detector is optimized training, to complete multimodal target detection according to the optimized target detector.The present application improves the prediction accuracy, can reduce the interference of low-quality samples in training samples to training process, and can be widely applied in computer technology field.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a robust multimodal target detection method and apparatus based on uncertainty. Background Technology

[0002] Object detection is one of the core advanced tasks in computer vision. Its main task is to find all objects of interest in an image and determine their location and category, solving the "where and what" problem. Object detection plays an indispensable role in many applications, including but not limited to medical diagnosis, autonomous driving, facial recognition, traffic monitoring, and industrial inspection. Rotated object detection, compared to traditional object detection, requires an additional angle information output. This rotated bounding box detector can achieve more accurate prediction results. Based on a rectangular box rotated by a certain angle, it obtains not only the position, size, and shape information of the object of interest, but also the object's orientation information.

[0003] The cost of acquiring infrared images has decreased significantly with the development of hardware systems. Using infrared imaging equipment to obtain environmental information has become a common solution. Due to its advantages such as low price, convenient deployment, and strong correlation with visible light, infrared cameras have been widely used, especially in fields such as autonomous driving and security monitoring.

[0004] Infrared and visible light offer complementary object characteristics, providing different perspectives for object recognition. In good lighting conditions, visible light exhibits sharp edge information and clear texture details, greatly aiding in object identification and location. However, in low-light conditions, visible light images typically suffer from low contrast and sharpness, resulting in a drastic decrease in visual quality. The wavelength limitation of visible light also restricts its observation range and depth; in some situations, it fails to reveal scene features, impacting object detection and tracking. Furthermore, visible light images cannot detect targets hidden in complex backgrounds, limiting their application in various complex environments.

[0005] Infrared imaging is more adaptable to different environments than visible light imaging. More specifically, it performs better at night and in adverse weather conditions. Infrared radiation can also penetrate clouds and fog, exhibiting strong anti-interference capabilities, thus making it widely used in remote sensing. It can identify camouflaged targets that are invisible to visible light. However, infrared images generally have some drawbacks, such as unclear details, blurred edges, and indistinct object outlines. When the temperature difference between objects is small, the contrast between the target and background in infrared images is usually low. Typically, the spatial resolution of infrared images is lower than that of visible light, resulting in blurred key targets and backgrounds in infrared images. The grayscale difference within the same area is usually small, therefore, it cannot fully reflect the texture and edge features of a scene, leading to more low-frequency components in the image. Infrared images also cannot fully represent the brightness information of a scene, thus lacking good spectral information. Generally, dark objects emit strong radiation, while bright or light objects emit weak radiation. Therefore, dark objects with the same temperature have higher grayscale values ​​in infrared images, while light-colored objects have lower grayscale values, which affects the detection efficiency of targets. Infrared imaging equipment is also susceptible to thermal noise and calibration noise, resulting in generally low signal-to-noise ratios (SNR) in such images.

[0006] Therefore, combining infrared and visible light images and fusing their complementary feature information can theoretically guarantee clear texture details, high contrast, and visibility, enabling clear object identification in various environments. Furthermore, benefiting from the decreasing barriers to obtaining multimodal data, deep learning-based multimodal object detection has become an increasingly active research area. The FLIR data solution and LLVIP dataset provide important benchmarks for infrared and visible light object detection tasks. Since the goal of multimodal object detection, like single-modal detection, is to locate and classify objects of interest in a scene, many methods are based on common visible light detection models such as RetinaNet and YOLO. Through various fusion mechanisms, the accuracy of object detection in various environments is improved, enhancing the robustness and security of deep learning algorithms.

[0007] In the field of deep learning-based rotated bounding box object detection, the main problems are as follows:

[0008] 1. When using deep learning, data is crucial for model fitting. If low-quality data is provided, the model's output will also be of low quality (garbage in, garbage out). Therefore, data quality is very important.

[0009] 2. Current methods cannot judge the quality and correctness of data, so they are easily affected by erroneous samples, resulting in poor model fit. This is especially true when applied in real-world scenarios, where various complex environments can lead to a large number of label errors and ambiguities.

[0010] 3. Deep learning has made great strides in object detection, and a large number of excellent large-scale datasets have emerged. However, investigations have revealed that these datasets contain mislabeled and ambiguous labels, which can affect model fitting.

[0011] 4. Manually cleaning the dataset would require a lot of manpower and resources. The current dataset has reached the level of millions of images, and the bounding box annotations are several times that number. Summary of the Invention

[0012] In view of this, embodiments of the present invention provide a robust multimodal target detection method and apparatus based on uncertainty with high prediction accuracy, so as to reduce the interference of low-quality samples in the training samples on the training process.

[0013] One aspect of this invention provides a robust multimodal target detection method based on uncertainty, comprising:

[0014] Pre-training yields target detectors with different modalities; each target detector is used to detect the object's position information, rotation angle, and category information, respectively.

[0015] RGB and infrared images are input into each of the target detectors, and the modal features of each modality are fused to obtain the predicted detection box and classification result of each modality.

[0016] Multimodal uncertainty estimation is performed based on the predicted detection boxes and classification results for each modality, and the uncertainty estimation results for the predicted detection boxes are determined.

[0017] Based on the IoU information of each predicted detection box, the uncertainty of the predicted detection box is assigned to the corresponding truth box to obtain the uncertainty-aware truth value.

[0018] Based on the uncertainty perception ground truth, the target detector is optimized and trained to complete multimodal target detection.

[0019] Optionally, the step of inputting RGB and infrared images into each of the target detectors and fusing the modal features of each modality to obtain the predicted detection bounding box and classification result for each modality includes:

[0020] Input RGB and infrared images, and extract the corresponding modal features through the ResNet-50 encoder network to obtain RGB modal features and infrared modal features;

[0021] The RGB modal features and the infrared modal features are input into a multimodal feature fusion module for fusion processing to obtain fused features;

[0022] The RGB modal features, the infrared modal features, and the fused features are input into the base detector to obtain the predicted detection boxes and classification results for each modality.

[0023] Optionally, in the step of inputting the RGB modal features and the infrared modal features into the multimodal feature fusion module for fusion processing to obtain the fused features, the calculation formula for the fusion processing is as follows:

[0024]

[0025] in, Represents the fused features; i represents the features of the i-th layer; Represents RGB modal characteristics; Represents infrared modal characteristics.

[0026] Optionally, the step of performing multimodal uncertainty estimation based on the predicted detection boxes and classification results of each modality, and determining the uncertainty estimation result of the predicted detection boxes, includes:

[0027] The predicted detection boxes of different modalities are merged to obtain the total detection boxes and the total classification result;

[0028] Multiple cluster centers of the total detection box are determined using the NMS method, and the classification result corresponding to each cluster center is determined. The predicted detection boxes of non-cluster centers with an IoU greater than 0.7 near the cluster center are assigned to the current cluster center to form a cluster.

[0029] Within each cluster, the uncertainty estimate for each cluster is calculated based on the classification result of each predicted detection box.

[0030] Optionally, the formula for calculating the uncertainty estimation result is:

[0031]

[0032] Where u represents the uncertainty estimation result; This represents the classification score of the i-th predicted detection box in the cluster.

[0033] Optionally, the step of assigning the uncertainty of each predicted detection box to the corresponding ground truth box based on the IoU information of each predicted detection box to obtain the uncertainty-aware ground truth includes:

[0034] Calculate the IoU value between each ground truth box and each predicted detection box as the association score between them, and obtain an association matrix;

[0035] Iterate through each truth box, find the predicted detection box with the highest associated score, and assign the uncertainty of the predicted detection box to the truth box.

[0036] During training, truth boxes with uncertainty exceeding a preset threshold are removed.

[0037] The uncertainty of the truth box is converted into weight information.

[0038] Optionally, the expression for the step of converting the uncertainty of the truth box into weight information is:

[0039] w=base+1-sigmoid(squeeze*(u-shift))

[0040] Where w represents the transformed weight information; base represents the first custom reference coefficient; sigmoid represents the activation function; squeeze represents the second custom reference coefficient; u represents the uncertainty estimation result; and shift represents the third custom reference coefficient.

[0041] Another aspect of this invention provides a robust multimodal target detection device based on uncertainty, comprising:

[0042] The first module is used to pre-train target detectors of different modalities; each target detector is used to detect the position information, rotation angle and category information of an object.

[0043] The second module is used to input RGB images and infrared images into each of the target detectors, and to perform fusion processing on the modal features of each modality to obtain the predicted detection box and classification results of each modality.

[0044] The third module is used to perform multimodal uncertainty estimation based on the predicted detection boxes and classification results of each modality, and to determine the uncertainty estimation results of the predicted detection boxes.

[0045] The fourth module is used to assign the uncertainty of the predicted detection box to the corresponding truth box based on the IoU information of each predicted detection box, so as to obtain the uncertainty-aware truth value.

[0046] The fifth module is used to optimize and train the target detector based on the uncertainty perception truth value, so as to complete multimodal target detection based on the optimized target detector.

[0047] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0048] The memory is used to store programs;

[0049] The processor executes the program to implement the method described above.

[0050] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0051] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0052] In embodiments of the present invention, firstly, target detectors of different modalities are pre-trained; each target detector is used to detect the position information, rotation angle, and category information of an object, respectively; then, RGB images and infrared images are input into each target detector, and the modal features of each modality are fused to obtain the predicted detection boxes and classification results for each modality; then, multimodal uncertainty estimation is performed based on the predicted detection boxes and classification results of each modality to determine the uncertainty estimation result of the predicted detection boxes; based on the IoU information of each predicted detection box, the uncertainty of the predicted detection box is assigned to the corresponding ground truth box to obtain the uncertainty-aware ground truth; finally, based on the uncertainty-aware ground truth, the target detectors are optimized and trained to complete multimodal target detection. The present invention improves prediction accuracy and can reduce the interference of low-quality samples in the training samples on the training process. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the implementation environment provided for an embodiment of the present invention;

[0055] Figure 2 A flowchart illustrating the overall steps of an embodiment of the present invention;

[0056] Figure 3This is a network structure diagram of ResNet50 provided in an embodiment of the present invention;

[0057] Figure 4 A schematic diagram illustrating clustering and uncertainty calculation provided in an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram illustrating the calculation of IoU values ​​provided in an embodiment of the present invention;

[0059] Figure 6 A schematic diagram of the sigmoid function provided in an embodiment of the present invention;

[0060] Figure 7 An example diagram of the DroneVehicle dataset provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] The following is a description of the relevant technical terms used in the embodiments of this invention:

[0063] Deep learning: Deep learning is a machine learning method that mimics the way the human brain works, processing and understanding data through multi-layered neural networks. The "depth" in this term refers to the multiple layers or depth of the neural network, typically including an input layer, multiple hidden layers, and an output layer. The depth and complexity of these neural networks enable them to extract features, patterns, and correlations from large-scale and complex data, thereby achieving efficient data analysis and decision-making.

[0064] Multimodal: Multimodal refers to data and information processing involving multiple information sources. These sensory modalities can include multiple senses such as vision, hearing, touch, smell, and taste. In this paper, multimodal refers to images formed from two spectra: visible light images and infrared images. Multimodal methods can integrate information from different perspectives of the same object to obtain a more comprehensive understanding.

[0065] Uncertainty in deep learning can be divided into two types: Aleatoric uncertainty, which arises from random factors. For example, in image classification, a photo of a dog might vary depending on its pose, lighting, and background. Epistemic uncertainty, on the other hand, arises from the model's uncertainty regarding the distribution of unknown data, typically due to insufficient training data or an imperfect model structure.

[0066] To address the problems of existing technologies, this invention utilizes the prediction results from a multimodal detector, combined with uncertainty learning, to enable the model to identify low-quality samples in the dataset during training. This invention can process low-quality samples, adjust network parameters, and reduce the interference of these samples during training, thus making the target detector's predictions increasingly accurate.

[0067] Specifically, one aspect of the present invention provides a robust multimodal target detection method based on uncertainty, including:

[0068] Pre-training yields target detectors with different modalities; each target detector is used to detect the object's position information, rotation angle, and category information, respectively.

[0069] RGB and infrared images are input into each of the target detectors, and the modal features of each modality are fused to obtain the predicted detection box and classification result of each modality.

[0070] Multimodal uncertainty estimation is performed based on the predicted detection boxes and classification results for each modality, and the uncertainty estimation results for the predicted detection boxes are determined.

[0071] Based on the IoU information of each predicted detection box, the uncertainty of the predicted detection box is assigned to the corresponding truth box to obtain the uncertainty-aware truth value.

[0072] Based on the uncertainty perception ground truth, the target detector is optimized and trained to complete multimodal target detection.

[0073] Optionally, the step of inputting RGB and infrared images into each of the target detectors and fusing the modal features of each modality to obtain the predicted detection bounding box and classification result for each modality includes:

[0074] Input RGB and infrared images, and extract the corresponding modal features through the ResNet-50 encoder network to obtain RGB modal features and infrared modal features;

[0075] The RGB modal features and the infrared modal features are input into a multimodal feature fusion module for fusion processing to obtain fused features;

[0076] The RGB modal features, the infrared modal features, and the fused features are input into the base detector to obtain the predicted detection boxes and classification results for each modality.

[0077] Optionally, in the step of inputting the RGB modal features and the infrared modal features into the multimodal feature fusion module for fusion processing to obtain the fused features, the calculation formula for the fusion processing is as follows:

[0078]

[0079] in, Represents the fused features; i represents the features of the i-th layer; Represents RGB modal characteristics; Represents infrared modal characteristics.

[0080] Optionally, the step of performing multimodal uncertainty estimation based on the predicted detection boxes and classification results of each modality, and determining the uncertainty estimation result of the predicted detection boxes, includes:

[0081] The predicted detection boxes of different modalities are merged to obtain the total detection boxes and the total classification result;

[0082] Multiple cluster centers of the total detection box are determined using the NMS method, and the classification result corresponding to each cluster center is determined. The predicted detection boxes of non-cluster centers with an IoU greater than 0.7 near the cluster center are assigned to the current cluster center to form a cluster.

[0083] Within each cluster, the uncertainty estimate for each cluster is calculated based on the classification result of each predicted detection box.

[0084] Optionally, the formula for calculating the uncertainty estimation result is:

[0085]

[0086] Where u represents the uncertainty estimation result; This represents the classification score of the i-th predicted detection box in the cluster.

[0087] Optionally, the step of assigning the uncertainty of each predicted detection box to the corresponding ground truth box based on the IoU information of each predicted detection box to obtain the uncertainty-aware ground truth includes:

[0088] Calculate the IoU value between each ground truth box and each predicted detection box as the association score between them, and obtain an association matrix;

[0089] Iterate through each truth box, find the predicted detection box with the highest associated score, and assign the uncertainty of the predicted detection box to the truth box.

[0090] During training, truth boxes with uncertainty exceeding a preset threshold are removed.

[0091] The uncertainty of the truth box is converted into weight information.

[0092] Optionally, the expression for the step of converting the uncertainty of the truth box into weight information is:

[0093] w=base+1-sigmoid(squeeze*(u-shift))

[0094] Where w represents the transformed weight information; base represents the first custom reference coefficient; sigmoid represents the activation function; squeeze represents the second custom reference coefficient; u represents the uncertainty estimation result; and shift represents the third custom reference coefficient.

[0095] Another aspect of this invention provides a robust multimodal target detection device based on uncertainty, comprising:

[0096] The first module is used to pre-train target detectors of different modalities; each target detector is used to detect the position information, rotation angle and category information of an object.

[0097] The second module is used to input RGB images and infrared images into each of the target detectors, and to perform fusion processing on the modal features of each modality to obtain the predicted detection box and classification results of each modality.

[0098] The third module is used to perform multimodal uncertainty estimation based on the predicted detection boxes and classification results of each modality, and to determine the uncertainty estimation results of the predicted detection boxes.

[0099] The fourth module is used to assign the uncertainty of the predicted detection box to the corresponding truth box based on the IoU information of each predicted detection box, so as to obtain the uncertainty-aware truth value.

[0100] The fifth module is used to optimize and train the target detector based on the uncertainty perception truth value, so as to complete multimodal target detection based on the optimized target detector.

[0101] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0102] The memory is used to store programs;

[0103] The processor executes the program to implement the method described above.

[0104] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0105] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0106] The specific implementation process of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings:

[0107] like Figure 1 As shown, Figure 1 This is a schematic diagram of the implementation environment of the present invention, where Loss is the loss of each mode and UAGT is the true value of each mode after uncertainty weighting. These are the features extracted by the backbone for each modality.

[0108] refer to Figure 2 In a specific implementation environment, the method of the present invention includes the following steps:

[0109] Step 1: Pre-train each modality detector

[0110] For the basic detector, we chose Roi Transformer as the rotating target detector, which can obtain the position information, rotation angle and category information of the object in three modes.

[0111] Step 2: Output the detection frames for each modality

[0112] Step 1: Encoder

[0113] Input RGB and infrared images, and extract corresponding modal features through the ResNet-50 encoder network. The network structure diagram is shown below. Figure 3 ,get (RGB modal characteristics) and (Infrared modal characteristics).

[0114] Step 2: Multimodal Feature Fusion

[0115] Will and The data is fed into the multimodal feature fusion module to obtain the fused features. The specific formula is as follows:

[0116]

[0117] Where the subscript i represents the i-th layer in the feature, and concat represents the concatenation operation of the channel dimension.

[0118] Step 3: The basic detector outputs the detection results.

[0119] Based on RGB modal features Infrared modal characteristics and fusion features Input the base detector to obtain the predicted detection boxes for each modality. and and classification results and

[0120] and It is an (N×5) matrix, where N is the number of detection boxes and 5 is the number of parameters of the detection boxes, representing the x-coordinate, y-coordinate, width, height and angle (x,y,w,h,θ) of the box.

[0121] and It is an (N×C) matrix, where N is the number of detection boxes and C is the number of categories.

[0122] Step 3: Multi-modal Uncertainty Estimation Module (MUEM)

[0123] For uncertainty estimation, we used a statistical data estimation method with multimodal redundant boxes to obtain the uncertainty estimate, as illustrated in the diagram below. Figure 4 .

[0124] Step 1: Redundant detection box clustering

[0125] Merge the detection boxes from the three modalities to obtain the complete set of detection boxes. and classification results Find cluster centers using NMS and their classification results The non-cluster center detection boxes with an IoU greater than 0.7 near the cluster center are then assigned to that cluster center to form a cluster.

[0126] Step 2: Data Uncertainty Estimation

[0127] In each cluster, the uncertainty u is calculated based on the classification score of each box. The uncertainty u of a cluster is calculated as follows:

[0128]

[0129] in, This is the classification score of the i-th detection box in the cluster. The larger the uncertainty u, the higher the uncertainty of the predicted box. Step 4: Uncertainty Assign Module (UAM)

[0130] Step 1: Distribute uncertainty based on the maximum IoU

[0131] Calculate the IoU between each ground truth bounding box and each predicted bounding box as the affinity score, such as... Figure 5 This yields an N×M affinity matrix, where N is the number of ground truth boxes and M is the number of predicted boxes. The IoU calculation formula is as follows:

[0132]

[0133] Iterate through each truth box, find the prediction box with the highest associated score, and assign the uncertainty of the prediction box to the truth box.

[0134] Step 2: Cleaning up faulty samples

[0135] Record the ground truth boxes where the uncertainty is higher than the threshold ignore_thr, and discard these samples directly during subsequent training.

[0136] Step 3: Weight Calculation

[0137] To convert uncertainty into weights, we devised a calculation method based on the sigmoid normalization function, such as... Figure 6 It introduces three coefficients: base, squeeze, and shift. By adjusting these three parameters, the weight generation method can be flexibly adjusted. The calculation formula is as follows:

[0138] w=base+1-sigmoid(squeeze*(u-shift))

[0139] Step 5: Retrain using dynamic weights from Uncertainty-Aware Truth-Ground (UAGT)

[0140] The Uncertainty-Perceived Truth Value (UAGT) is obtained in step 4, as follows: Figure 2 As shown, the retraining step uses the cleaned dataset from step 3 and its sample weights for weighted training:

[0141] L total =L cls +L reg

[0142] Where L cls L represents classification loss. reg The regression loss is calculated as follows:

[0143]

[0144]

[0145] Among them, V i B is the UAGT classification vector. i This indicates the location of the UAGT detection box.

[0146] In summary, this invention introduces uncertainty as a quantitative indicator of sample quality, and then adaptively processes these samples, enabling the network to fit the correct samples in the dataset. Compared with existing technologies, this invention has the following characteristics:

[0147] 1. Robustness: It can maintain good detection performance when errors occur in the dataset.

[0148] 2. High versatility: The basic frame of the detector has strong versatility. The basic detector can be replaced with any type of detector, whether it is a rectangular frame detector or a rotating frame detector.

[0149] 3. Adaptive Sample Selection: It has the ability to learn from training samples and can adaptively process training samples based on the model's output.

[0150] Furthermore, comparative verification experiments were conducted on the DroneVehicle dataset in this embodiment of the invention. The results are shown in Table 1. The dataset contains a total of 5 categories: car, truck, bus, van, and freight car. The lighting conditions are complex and challenging, and it contains a large number of annotation errors. Example images are shown below. Figure 7 As shown.

[0151] Table 1

[0152]

[0153]

[0154] In detection tasks, mAP (mean of average precision) and classAP (class average precision) are commonly used to evaluate a detector; the higher the two metrics, the better the performance.

[0155] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0156] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0157] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0159] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0160] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0161] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0162] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0163] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A robust multimodal target detection method based on uncertainty, characterized in that, include: Pre-training yields target detectors with different modalities; each target detector is used to detect the object's position information, rotation angle, and category information, respectively. RGB and infrared images are input into each of the target detectors, and the modal features of each modality are fused to obtain the predicted detection box and classification result of each modality. Multimodal uncertainty estimation is performed based on the predicted detection boxes and classification results for each modality, and the uncertainty estimation results for the predicted detection boxes are determined. Based on the IoU information of each predicted detection box, the uncertainty of the predicted detection box is assigned to the corresponding truth box to obtain the uncertainty-aware truth value. Based on the uncertainty perception truth value, the target detector is optimized and trained to complete multimodal target detection. The step of assigning the uncertainty of each predicted detection box to the corresponding ground truth box based on the IoU information of each predicted detection box to obtain the uncertainty-aware ground truth includes: Calculate the IoU value between each ground truth box and each predicted detection box as the association score between them, and obtain an association matrix; Iterate through each truth box, find the predicted detection box with the highest associated score, and assign the uncertainty of the predicted detection box to the truth box. During training, truth boxes with uncertainty exceeding a preset threshold are removed. The uncertainty of the truth box is converted into weight information; The expression for the step of converting the uncertainty of the truth box into weight information is: in, This represents the weight information obtained from the transformation; This represents the first custom reference coefficient; Represents the activation function; This represents the second custom reference coefficient; This represents the result of uncertainty estimation; This represents the third custom reference coefficient.

2. The robust multimodal target detection method based on uncertainty according to claim 1, characterized in that, The process involves inputting RGB and infrared images into each of the target detectors, fusing the modal features of each modality, and obtaining the predicted detection bounding boxes and classification results for each modality, including: Input RGB and infrared images, and extract the corresponding modal features through the ResNet-50 encoder network to obtain RGB modal features and infrared modal features; The RGB modal features and the infrared modal features are input into a multimodal feature fusion module for fusion processing to obtain fused features; The RGB modal features, the infrared modal features, and the fused features are input into the base detector to obtain the predicted detection boxes and classification results for each modality.

3. The robust multimodal target detection method based on uncertainty according to claim 2, characterized in that, In the step of inputting the RGB modal features and the infrared modal features into the multimodal feature fusion module for fusion processing to obtain the fused features, the calculation formula for the fusion processing is as follows: in, Represents the characteristics of integration; Representing the Layer features; Represents RGB modal characteristics; Represents infrared modal characteristics.

4. The robust multimodal target detection method based on uncertainty according to claim 1, characterized in that, The step of performing multimodal uncertainty estimation based on the predicted detection boxes and classification results for each modality, and determining the uncertainty estimation result of the predicted detection boxes, includes: The predicted detection boxes of different modalities are merged to obtain the total detection boxes and the total classification result; Multiple cluster centers of the total detection box are determined using the NMS method, and the classification result corresponding to each cluster center is determined. The predicted detection boxes of non-cluster centers with an IoU greater than 0.7 near the cluster center are assigned to the current cluster center to form a cluster. Within each cluster, the uncertainty estimate for each cluster is calculated based on the classification result of each predicted detection box.

5. The robust multimodal target detection method based on uncertainty according to claim 4, characterized in that, The formula for calculating the uncertainty estimation result is as follows: in, This represents the result of uncertainty estimation; Represents the first in the cluster The classification score of each predicted detection box.

6. An apparatus for implementing the uncertainty-based robust multimodal target detection method as described in any one of claims 1-5, characterized in that, include: The first module is used to pre-train target detectors of different modalities; each target detector is used to detect the position information, rotation angle and category information of an object. The second module is used to input RGB images and infrared images into each of the target detectors, and to perform fusion processing on the modal features of each modality to obtain the predicted detection box and classification results of each modality. The third module is used to perform multimodal uncertainty estimation based on the predicted detection boxes and classification results of each modality, and to determine the uncertainty estimation results of the predicted detection boxes. The fourth module is used to assign the uncertainty of the predicted detection box to the corresponding truth box based on the IoU information of each predicted detection box, so as to obtain the uncertainty-aware truth value. The fifth module is used to optimize and train the target detector based on the uncertainty perception truth value, so as to complete multimodal target detection based on the optimized target detector.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images