Multimodal interaction target detection method and system based on soft fusion supplement, and readable storage medium
By introducing a soft fusion supplement to multimodal target detection, and utilizing a cross-modal soft fusion module and a multi-scale exchange gating module, the problems of noise propagation and reduced interaction capability in traditional methods are solved, thereby improving the accuracy and robustness of aerial image target detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- KUNMING UNIV OF SCI & TECH
- Filing Date
- 2025-11-19
- Publication Date
- 2026-06-23
AI Technical Summary
Traditional multimodal target detection methods are prone to problems such as noise propagation at different levels and reduced interactivity in aerial images, resulting in significant differences in multi-branch detection.
A multimodal interactive target detection method based on soft fusion is adopted. Through a four-layer bi-branch feature extraction layer, a cross-modal soft fusion module, a multi-scale exchange gating module, and a fast spatial pyramid pooling module, feature interaction and context dependence are enhanced, while noise propagation is suppressed.
It improves the accuracy and robustness of multimodal target detection, reduces the difference between bi-branch detection, and enhances the anti-interference ability in complex scenarios.
Smart Images
Figure CN121392676B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multimodal target detection technology, and in particular to a multimodal interactive target detection method, system and readable storage medium based on soft fusion supplementation. Background Technology
[0002] Aerial image target detection is a key technology in Earth observation and remote sensing, with its core task being the identification of the location and category of target objects in aerial images. In recent years, with the widespread adoption of drones and high-resolution remote sensing platforms, aerial image target detection has not only received extensive attention in scientific research but has also gradually become a core supporting technology in smart cities and land resource management. Multimodal target detection plays a crucial role in the all-weather, multi-scenario application of aerial imagery. Multimodal models possess greater potential than single-modal models in terms of accuracy and multi-scenario application.
[0003] In the process of conceiving and implementing this application, the inventors discovered that when traditional multimodal target detection methods are applied to aerial image target detection, the traditional multimodal target detection methods that use fusion features or cross-modal interaction to correct or enhance the weights of the two branches are prone to noise propagation in the hierarchical process and a decrease in interaction capability, which in turn leads to large differences in detection between the multiple branches.
[0004] In view of this, this application proposes a new multimodal interactive target detection method, which aims to solve the problems that traditional multimodal feature extraction cannot suppress noise propagation at different levels and the interaction capability is reduced due to feature loss. Summary of the Invention
[0005] The main purpose of this application is to provide a multimodal interactive target detection method based on soft fusion supplementation, which aims to solve the problems of how to suppress noise propagation in layers and the decline in interactive capability due to feature loss.
[0006] To achieve the above objectives, this application provides a multimodal interactive target detection method based on soft fusion supplementation, applied to a multimodal interactive target detection model. The multimodal interactive target detection model includes four bi-branch feature extraction layers. Each bi-branch feature extraction layer is followed by a cross-modal soft fusion module and two multi-scale exchange gating modules. A fast spatial pyramid pooling module is set after the last bi-branch feature extraction layer. The method includes the following steps:
[0007] When the input multimodal aerial image dataset to be tested is obtained, the extracted features obtained from the multimodal aerial image dataset to be tested by each of the dual-branch feature extraction layers are input to the cross-modal soft fusion module for feature supplementation.
[0008] The two multi-scale exchange gating modules are used to enhance the fused features after feature supplementation, and the enhanced fused features are input to the next layer of the dual-branch feature extraction layer.
[0009] After the features of the multimodal aerial image dataset pass through four dual-branch feature extraction layers, the features output by the last dual-branch feature extraction layer are input into the fast spatial pyramid pooling module for context dependency capture, and then output to the neck.
[0010] The features of the neck are input into the detection head to obtain the aerial target detection result.
[0011] Optionally, the feature supplementation step of the cross-modal soft fusion module includes:
[0012] A single-channel saliency map is generated using a lightweight convolutional network, which is the first extracted feature output by the dual-branch feature extraction layer. With the second extracted features Generate a saliency mask and :
[0013]
[0014]
[0015] Based on the key region features emphasized by the saliency mask and the complementary region features of the counterpart branch saliency mask, soft fusion features are generated. and :
[0016]
[0017]
[0018] The acquired soft fusion features and By performing dimensionality reduction and spatial attention enhancement through splicing, fused features are obtained. :
[0019]
[0020] Based on the fusion features soft fusion features and Enhancement is performed to obtain the fused features after feature supplementation. and :
[0021]
[0022]
[0023] In the formula, For activation functions; and Here, H represents the corresponding convolution operation, W represents the feature height, and C represents the feature width. This indicates splicing in the channel dimension; This indicates that the concatenated features are processed through grouped convolution. Dimensionality reduction is performed, and the obtained preliminary fusion features are dynamically weighted according to spatial location using a spatial attention module. Then, 1×1 convolution is used for channel adjustment, and finally, normalization is performed.
[0024] Optionally, the steps of the multi-scale exchange gating module in enhancing the fused features after feature supplementation include:
[0025] Fusion features after feature supplementation Enhancement:
[0026]
[0027]
[0028]
[0029]
[0030]
[0031]
[0032] Fusion features after feature supplementation Enhancement:
[0033]
[0034]
[0035]
[0036]
[0037]
[0038]
[0039] In the formula, Indicates splicing from dimensions as well as , This represents the interaction features obtained after downsampling. This represents the interactive features that have been recovered to their original scale. Indicates splicing from dimensions and , Indicates the gating weight, This indicates the enhanced fusion features; This represents the visible light characteristics after the missing supplementary features are added. Indicates splicing from dimensions as well as , This represents the interaction features obtained after downsampling. This represents the interactive features that have been recovered to their original scale. Indicates splicing from dimensions and , Indicates the gating weight, Indicates the enhanced fusion features , It is a downsampling feature extraction function. This is the upsampled feature recovery function.
[0040] Optionally, during the context dependency capture process, the fast spatial pyramid pooling module satisfies the following mathematical expression:
[0041]
[0042]
[0043]
[0044]
[0045]
[0046]
[0047]
[0048]
[0049] In the formula, It consists of a convolutional module, batch normalization, and the SiLU activation function. It consists of multiple Bottleneck blocks, each containing two convolutional layers; This indicates that cross-modal soft fusion operations are performed on the extracted visible and infrared light; This indicates the performance of a multi-scale exchange gating operation; SPPF represents the fusion feature obtained by element-level addition of the enhanced bi-branch features; SPPF represents the fast spatial pyramid pooling module.
[0050] Optionally, the loss function of the multimodal interactive object detection model includes bounding box regression loss, binary cross-entropy loss, and distribution focus loss, and the expression of the loss function is:
[0051]
[0052] In the formula, For bounding box regression loss, For binary cross-entropy loss, For distribution focus loss;
[0053] The mathematical expression for the bounding box regression loss includes:
[0054]
[0055] In the formula, IoU represents the ratio of the overlapping area to the non-overlapping area; c represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c represents the length of the diagonal of the outer bounding box. Indicates the aspect ratio consistency item; Indicates the balance coefficient;
[0056] The mathematical expression for the binary cross-entropy loss includes:
[0057]
[0058] In the formula, Indicates the confidence level of the predicted category; Indicates the adjustment parameter; Indicates the category balance factor;
[0059] The mathematical expression for the distribution focus loss includes:
[0060]
[0061] In the formula, Represents the true distribution; This represents the predicted distribution.
[0062] Furthermore, to achieve the above objectives, this application also provides a multimodal interactive target detection model, characterized in that the multimodal interactive target detection model includes:
[0063] A four-layer dual-branch feature extraction layer is used to extract the features obtained from the input multimodal aerial image dataset.
[0064] A cross-modal soft fusion module and two multi-scale exchange gating modules are set after each of the dual-branch feature extraction layers. The cross-modal soft fusion module is used to perform feature supplementation, and the multi-scale exchange gating modules are used to enhance the fused features after feature supplementation and input the enhanced fused features into the next dual-branch feature extraction layer.
[0065] The fast spatial pyramid pooling module, located after the last double-branch feature extraction layer, is used to input the features output by the last double-branch feature extraction layer into the fast spatial pyramid pooling module for context dependency capture and then output them to the neck.
[0066] A detection head is used to obtain aerial target detection results based on the features output from the neck.
[0067] In addition, to achieve the above objectives, this application also provides a computer system, the computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the multimodal interactive target detection method based on soft fusion supplementation as described in any of the preceding claims.
[0068] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multimodal interactive target detection method based on soft fusion supplementation as described in any of the preceding claims.
[0069] This application has at least the following beneficial effects:
[0070] 1. Utilize cross-modal feature soft fusion modules to form fused features, supplementing the two branches with modal missing features and increasing the coupling of cross-modal features;
[0071] 2. The salient regions of the bi-branch features are adaptively enhanced and supplemented through a multi-scale interactive gating module;
[0072] 3. By using a fast spatial pyramid pooling module, the fusion timing of the last layer of feature extraction is improved, further enhancing the global alignment across modalities in deep semantics. The method in this application can effectively handle various complex scenarios, avoiding problems such as large differences between two branches and weak anti-interference ability when one branch fails, exhibiting excellent accuracy and robustness. Attached Figure Description
[0073] Figure 1 This is a schematic diagram of the architecture of the multimodal interactive target detection model involved in the embodiments of this application;
[0074] Figure 2This is a flowchart illustrating the multimodal interactive target detection method based on soft fusion supplementation involved in the embodiments of this application;
[0075] Figure 3 This is a schematic diagram of the soft fusion module architecture involved in the embodiments of this application;
[0076] Figure 4 This is a schematic diagram of the multi-scale interactive gating module architecture involved in an embodiment of this application;
[0077] Figure 5 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.
[0078] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0079] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.
[0080] First Embodiment
[0081] This embodiment provides a multimodal interactive target detection method based on soft fusion supplementation. This method is applied to the multimodal interactive target detection model proposed in this embodiment, with reference to... Figure 1 The model mainly includes the following architecture:
[0082] The system consists of four bi-branch feature extraction layers (each red and blue block of the same scale in the figure constitutes a bi-branch feature extraction layer). Each bi-branch feature extraction layer is followed by a Cross-modal Feature Soft Fusion Module (CFSF) and two Multi-Scale Interactive Gating Modules (MSIG). The Cross-modal Feature Soft Fusion Module (CFSF) and the two Multi-Scale Interactive Gating Modules (MSIG) constitute the Soft Fusion Supplementary Multimodal Interaction Module (SFSMIM) proposed in this embodiment.
[0083] A fast spatial pyramid pooling (SPPF) module is set up after the last double-branch feature extraction layer.
[0084] Reference Figure 2 The various components of this architecture, when performing object detection, include the following steps:
[0085] S10, when the input multimodal aerial image dataset to be tested is obtained, the extracted features obtained from the multimodal aerial image dataset to be tested by each of the dual-branch feature extraction layers are input to the cross-modal soft fusion module for feature supplementation;
[0086] In this embodiment, after the feature extraction is completed, the output features are not directly input to the next layer. Instead, the features are supplemented and interacted with by the soft fusion supplementation module and the multi-scale interactive gating module before being output to the next layer.
[0087] The cross-modal feature soft fusion module aims to supplement the missing modalities in the two branches, increase the coupling of cross-modal features, and better achieve the tasks of saliency alignment and global context interaction of cross-modal information. In multimodal feature fusion, hard fusion usually refers to directly combining features from different modalities using fixed rules such as concatenation, addition, or averaging. Its advantages are simplicity and low computational cost, but it is difficult to adapt to the differences in quality and semantics between different modalities. Soft fusion, on the other hand, dynamically assigns weights to different modalities through a learnable mechanism, thereby achieving more flexible feature complementarity and saliency alignment. This embodiment adopts saliency-guided modal soft fusion and, through a subsequent convolution-spatial attention mechanism, integrates the complementary advantages of visible light and infrared in different regions to obtain the optimal modal fusion features, thus compensating for the missing features in the two branches.
[0088] Further and optionally, refer to Figure 3 The schematic diagram of the soft fusion module architecture shown assumes that the multimodal features input to the complementary fusion module are as follows: and First, a single-channel saliency map is generated using a lightweight convolutional network, which then generates fine-grained saliency masks for the VIS and IR features respectively. and This is used to characterize the attention given to different modalities at different spatial locations. Then, based on the key region features emphasized by the saliency mask and the complementary region features of the counterpart branch saliency mask, soft-fused features are generated. and While adjusting the weights of features, complementary features from the other modality are added, effectively preserving and complementing the salient features of both modalities. The soft fusion approach avoids selecting only VIS or IR at a certain position, achieving better consistency in cross-modal features through a smooth transition. The dynamic soft allocation mechanism allows the network to adaptively select the optimal modal feature based on saliency, thereby significantly improving the robustness of detection. In short:
[0089] S11, a single-channel saliency map is generated through a lightweight convolutional network, which is the first extracted feature output by the dual-branch feature extraction layer. With the second extracted features Generate a saliency mask and :
[0090]
[0091]
[0092] S12, based on the key region features emphasized by the saliency mask and the complementary region features of the counterpart branch saliency mask, generates soft fusion features. and :
[0093]
[0094]
[0095] To enhance the coupling of cross-modal bi-branch features, after obtaining the soft-fused features... and The two were then spliced together to perform dimensionality reduction and spatial attention enhancement to obtain fusion features. The fused features are used to update the main branches to alleviate the misalignment of different modalities at object boundaries, textures, and local details, providing global supplementation for subsequent branch enhancements. In other words:
[0096] S13, the acquired soft fusion features and By performing dimensionality reduction and spatial attention enhancement through splicing, fused features are obtained. :
[0097]
[0098] S14, based on the fusion features soft fusion features and Enhancement is performed to obtain the fused features after feature supplementation. and :
[0099]
[0100]
[0101] In the formula, For activation functions; and Here, H represents the corresponding convolution operation, W represents the feature height, and C represents the feature width. This indicates splicing in the channel dimension; This indicates that the concatenated features are processed through grouped convolution. Dimensionality reduction is performed, and the obtained preliminary fusion features are dynamically weighted according to spatial location using a spatial attention module. Then, 1×1 convolution is used for channel adjustment, and finally, normalization is performed.
[0102] S20, the fused features after feature supplementation are enhanced by the two multi-scale exchange gating modules respectively, and the enhanced fused features are input to the next layer of the dual-branch feature extraction layer respectively;
[0103] In this embodiment, the supplemented bi-branch features are adaptively enhanced in the salient region through a multi-scale interactive gating module.
[0104] Further and optionally, refer to Figure 4 The diagram shown illustrates the architecture of the multi-scale interactive gating module. The steps of the multi-scale exchange gating module in enhancing the fused features after feature supplementation are as follows:
[0105] S21, fused features after feature supplementation Enhancement:
[0106]
[0107]
[0108]
[0109]
[0110]
[0111]
[0112] S22, fused features after feature supplementation Enhancement:
[0113]
[0114]
[0115]
[0116]
[0117]
[0118]
[0119] In the formula, Indicates splicing from dimensions as well as , This represents the interaction features obtained after downsampling. This represents the interactive features that have been recovered to their original scale. Indicates splicing from dimensions and , Indicates the gating weight, This indicates the enhanced fusion features; This represents the visible light characteristics after the missing supplementary features are added. Indicates splicing from dimensions as well as , This represents the interaction features obtained after downsampling. This represents the interactive features that have been recovered to their original scale. Indicates splicing from dimensions and , Indicates the gating weight, Indicates the enhanced fusion features , It is a downsampling feature extraction function. This is the upsampled feature recovery function.
[0120] As some alternative implementation methods It is a downsampling feature extraction function, which consists of a 3×3 convolutional layer with a stride of 2, batch normalization, and GELU activation function in sequence. It is used to reduce the spatial resolution of the input feature map to half of its original value, while extracting local interaction features to obtain a larger receptive field. The upsampling feature recovery function consists of a 3×3 convolutional layer, batch normalization, GELU activation function, and bilinear interpolation upsampling (scalefactor = 2) in sequence. It is used to restore the feature map to the original spatial resolution after extracting multi-scale context information, thereby achieving smooth propagation of local to global information.
[0121] It should be noted that the above steps are described from the perspective of a single-layer bi-branch feature extraction layer and a subsequent cross-modal soft fusion module and two multi-scale exchange gating modules. In this embodiment, the bi-branch feature extraction layer has four layers, and all four layers perform the corresponding actions as described in S10 and S20, until the last bi-branch feature extraction layer is passed, proceeding to step S30.
[0122] S30, after the features of the multimodal aerial image dataset pass through four dual-branch feature extraction layers, the features output by the last dual-branch feature extraction layer are input into the fast spatial pyramid pooling module for context-dependent capture, and then output to the neck.
[0123] In this embodiment, after the features of the multimodal aerial image dataset are processed through four dual-branch feature extraction layers, the SPPF module, which is shared by all four feature extraction layers, is used to capture the contextual dependencies of cross-modal scenes in the fused features, further improving the global alignment of cross-modal features under deep semantics, and then inputting it into the neck.
[0124] Furthermore, and optionally, introducing a shared SPPF into the fused high semantic space can more comprehensively capture contextual dependencies across modal scenes, enhancing the model's adaptability to complex targets and multi-scale variations. Moreover, unified multi-scale context extraction within the shared deep semantic space can improve the semantic coupling between RGB and IR information. This process can be represented as:
[0125]
[0126]
[0127]
[0128]
[0129]
[0130]
[0131]
[0132]
[0133] In the formula, It consists of a convolutional module, batch normalization, and the SiLU activation function. It consists of multiple Bottleneck blocks, each containing two convolutional layers. This indicates that the above-mentioned cross-modal soft fusion operation is performed on the extracted visible light and infrared light. This indicates the above multi-scale exchange gating operation. This represents the fused feature obtained by element-wise addition to the enhanced bi-branch features. SPPF stands for Fast Spatial Pyramid Pooling Module.
[0134] S40, the features of the neck are input into the detection head to obtain the aerial target detection result.
[0135] In this embodiment, after the above processing, the features in the neck are input into the detection head to obtain the aerial target detection result.
[0136] In the technical solution provided in this embodiment, a dual-stream backbone network is used to extract multimodal features. After each layer of feature extraction, a soft fusion supplementary multi-scale interaction module is added, which includes a cross-modal soft fusion module and two multi-scale exchange gating modules. The cross-modal feature soft fusion module is used to form fused features, which supplement the two branches to compensate for modal missing features and increase the coupling of cross-modal features. The supplemented dual-branch features are adaptively enhanced in saliency regions through multi-scale interaction gating modules. The fusion timing of the last layer of feature extraction is improved to further enhance the global alignment of cross-modal features under deep semantics, thereby avoiding the problems of large differences between the two branches and weak anti-interference ability when one branch fails.
[0137] Second Embodiment
[0138] Based on the first embodiment, in this embodiment, the detection head of the multimodal interactive object detection model is YOLOv8. This is to address three key issues: localization accuracy, bounding box refinement, and classification robustness, thereby achieving higher accuracy and stability in the detection task. The loss function of the multimodal interactive object detection model can be expressed as:
[0139]
[0140] In the formula, For bounding box regression loss, For binary cross-entropy loss, For distribution focus loss;
[0141] The mathematical expression for the bounding box regression loss includes:
[0142]
[0143] In the formula, IoU represents the ratio of the overlapping area to the non-overlapping area; c represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c represents the length of the diagonal of the outer bounding box. Indicates the aspect ratio consistency item; Indicates the balance coefficient;
[0144] The mathematical expression for the binary cross-entropy loss includes:
[0145]
[0146] In the formula, Indicates the confidence level of the predicted category; Indicates the adjustment parameter; Indicates the category balance factor;
[0147] The mathematical expression for the distribution focus loss includes:
[0148]
[0149] In the formula, Represents the true distribution; This represents the predicted distribution.
[0150] Third Embodiment
[0151] As a verification embodiment, based on the content of any of the above embodiments, in order to verify the effectiveness of the multimodal interactive target detection method based on soft fusion supplementation proposed in this embodiment, feature extraction is performed using two backbones with different parameter sizes, YOLOv8n and YOLOv8s, and the multimodal interactive target detection model and the designed method proposed above are compared with the current mainstream models and methods.
[0152] 1) Introduction to the dataset
[0153] The DroneVehicle dataset contains 28,439 pairs of day and night RGB-IR images covering urban roads, residential areas, and parking lots, containing 953,087 vehicle instances. It includes five vehicle types: sedans, trucks, vans, buses, and minivans. The original image resolution was 640×512; researchers added a 100-pixel blank area around the images to facilitate object annotation at boundaries, resulting in a final resolution of 840×712. The dataset was divided into 17,990 training pairs, 1,469 validation pairs, and 8,980 test pairs.
[0154] The VEDAI dataset consists of 1246 small-sized images covering diverse backgrounds such as grasslands, roads, mountains, and cities. All images have two different sizes: 1024×1024 pixels and 512×512 pixels. All images were taken during the day from a 90-degree vertical perspective, and each image within the same scene includes both RGB and infrared modalities. It contains 3640 vehicle instances across 11 categories, including boats, cars, campervans, and airplanes. The dataset is divided into 10 cross-validation batches: each batch contains 1089 training images and 121 test images.
[0155] The ODinMJ dataset contains 23,075 annotated RGB-T image pairs with 45,617 entity bounding boxes, covering various poses, scales, backgrounds, and occlusion conditions. All images are 640×512 pixels. The dataset exhibits diversity in lighting, slope, camouflage, and occlusion, effectively reflecting the complexity of real-world mountain and jungle scenes. We divided all image pairs into three sets in an 8:1:1 ratio: 18,457 training pairs, 2,309 validation pairs, and 2,309 test pairs. Further partitioning was performed according to different dimensions during testing.
[0156] (2) Evaluation indicators
[0157] We use two mean precision (mAP) metrics as the primary evaluation metrics: mAP50 and mAP50-95. In addition, we use two commonly used metrics—model parameters (Params) and billion floating-point operations per second (GFLOPs)—to evaluate the model's performance and efficiency. mAP is calculated based on the precision-recall (PR) curve, comprehensively reflecting the model's overall accuracy across multiple classes. An improvement in mAP indicates enhanced detection performance across all categories. A higher P-value indicates higher prediction accuracy for positive samples; a higher R-value means the model can more accurately identify positive samples. mAP50 and mAP50-95 are variations of mAP, the core difference being the definition range of the Intersection over Union (IoU) threshold. mAP50 represents the average precision for all classes when IoU ≥ 0.5, while mAP50-95 represents the average precision across a series of IoU thresholds within the range of 0.5 to 0.95, with a step size of 0.05. The calculation formulas are as follows:
[0158]
[0159]
[0160]
[0161]
[0162]
[0163] True positive (TP) is the number of positive samples that are correctly identified, true negative (TN) is the number of negative samples that are correctly identified, false positive (FP) is the number of negative samples that are falsely reported, and false negative (FN) is the number of positive samples that are missed.
[0164] (3) Implementation details
[0165] The proposed method is implemented using the PyTorch framework and trained and tested on an NVIDIA RTX 4060 Ti GPU with 16GB of memory. The model's code environment is based on CUDA 11.6 and PyTorch 1.13.1, employing a stochastic gradient descent optimizer. During training, a two-stream feature extraction architecture is used with parameters set to momentum 0.937, weight decay 0.0005, and an initial learning rate of 0.01. For the DroneVehicle and ODinMJ datasets, the input image size is set to 512, the batch size to 16, and the training epochs to 200. For the VEDAI dataset, the image size is set to 1024, the batch size to 8, and the training epochs to 300. Random scaling is used to enhance the images during training. Specifically, for the DroneVehicle dataset, infrared modality labels are used as ground truth labels, while for the ODinMJ dataset, a fusion label combining the two modalities is used. When training our model using the VEDAI dataset, we removed instances with a total number less than 50, such as airplanes, motorcycles, and buses.
[0166] The comparison results of the public datasets DroneVehicle, VEDAI, and OdinMJ are shown in Tables 1, 2, and 3. The best results in the tables are shown in bold.
[0167] Table 1. Quantitative comparison of different methods on the DroneVehicle dataset
[0168]
[0169] Table 2. Quantitative comparison of different methods on the DroneVehicle dataset
[0170]
[0171] Table 3. Quantitative comparison of different methods on the OdinMJ dataset
[0172]
[0173] Experimental results show that the method proposed in this invention outperforms multiple methods on multiple datasets, verifying its effectiveness and superiority.
[0174] In addition, as an implementation scheme, refer to Figure 1 This application also provides a multimodal interactive target detection model, the multimodal interactive target detection model comprising:
[0175] A four-layer dual-branch feature extraction layer is used to extract the features obtained from the input multimodal aerial image dataset.
[0176] A cross-modal soft fusion module and two multi-scale exchange gating modules are set after each of the dual-branch feature extraction layers. The cross-modal soft fusion module is used to perform feature supplementation, and the multi-scale exchange gating modules are used to enhance the fused features after feature supplementation and input the enhanced fused features into the next dual-branch feature extraction layer.
[0177] The fast spatial pyramid pooling module, located after the last double-branch feature extraction layer, is used to input the features output by the last double-branch feature extraction layer into the fast spatial pyramid pooling module for context dependency capture and then output them to the neck.
[0178] A detection head is used to obtain aerial target detection results based on the features output from the neck.
[0179] As one implementation scheme, Figure 5 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.
[0180] like Figure 5 As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0181] Those skilled in the art will understand that Figure 5 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0182] like Figure 5 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.
[0183] exist Figure 5 In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used to communicate data with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.
[0184] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:
[0185] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:
[0186] When the input multimodal aerial image dataset to be tested is obtained, the extracted features obtained from the multimodal aerial image dataset to be tested by each of the dual-branch feature extraction layers are input to the cross-modal soft fusion module for feature supplementation.
[0187] The two multi-scale exchange gating modules are used to enhance the fused features after feature supplementation, and the enhanced fused features are input to the next layer of the dual-branch feature extraction layer.
[0188] After the features of the multimodal aerial image dataset pass through four dual-branch feature extraction layers, the features output by the last dual-branch feature extraction layer are input into the fast spatial pyramid pooling module for context dependency capture, and then output to the neck.
[0189] The features of the neck are input into the detection head to obtain the aerial target detection result.
[0190] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.
[0191] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the multimodal interactive target detection method based on soft fusion supplementation as described in the above embodiments.
[0192] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.
[0193] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.
[0194] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0195] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0196] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0197] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0198] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
[0199] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0200] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A multimodal interactive target detection method based on soft fusion supplementation, characterized in that, An application is made to a multimodal interactive target detection model, which includes four bi-branch feature extraction layers. Each bi-branch feature extraction layer is followed by a cross-modal soft fusion module and two multi-scale exchange gating modules. A fast spatial pyramid pooling module is set after the last bi-branch feature extraction layer. The method includes the following steps: When the input multimodal aerial image dataset to be tested is obtained, the extracted features obtained from the multimodal aerial image dataset to be tested by each of the dual-branch feature extraction layers are input to the cross-modal soft fusion module for feature supplementation. The two multi-scale exchange gating modules are used to enhance the fused features after feature supplementation, and the enhanced fused features are input to the next layer of the dual-branch feature extraction layer. After the features of the multimodal aerial image dataset pass through four dual-branch feature extraction layers, the features output by the last dual-branch feature extraction layer are input into the fast spatial pyramid pooling module for context dependency capture, and then output to the neck. The features of the neck are input into the detection head to obtain the aerial target detection result; The feature supplementation steps performed by the cross-modal soft fusion module include: A single-channel saliency map is generated using a lightweight convolutional network, which is the first extracted feature output by the dual-branch feature extraction layer. With the second extracted features Generate a saliency mask and : ； ； Based on the key region features emphasized by the saliency mask and the complementary region features of the counterpart branch saliency mask, soft fusion features are generated. and : ； ； The acquired soft fusion features and By performing dimensionality reduction and spatial attention enhancement through splicing, fused features are obtained. : ； Based on the fusion features soft fusion features and Enhancement is performed to obtain the fused features after feature supplementation. and : ； ； In the formula, For activation functions; and Here, H represents the corresponding convolution operation, W represents the feature height, and C represents the feature width. This indicates splicing in the channel dimension; This indicates that the concatenated features are processed through grouped convolution. Dimensionality reduction is performed by using a spatial attention module to dynamically weight the feature map according to its spatial location on the obtained preliminary fusion features, then channel adjustment is performed using a 1×1 convolution, and finally normalization is performed. The steps of the multi-scale exchange gating module in enhancing the fused features after feature supplementation include: Fusion features after feature supplementation Enhancement: ； ； ； ； ； ； Fusion features after feature supplementation Enhancement: ； ； ； ； ； ； In the formula, Indicates splicing from dimensions as well as , This represents the interaction features obtained after downsampling. This represents the interactive features that have been recovered to their original scale. Indicates splicing from dimensions and , Indicates the gating weight, This indicates the enhanced fusion features; This represents the visible light characteristics after the missing supplementary features are added. Indicates splicing from dimensions as well as , This represents the interaction features obtained after downsampling. This represents the interactive features that have been recovered to their original scale. Indicates splicing from dimensions and , Indicates the gating weight, Indicates the enhanced fusion features , It is a downsampling feature extraction function. This is the upsampled feature recovery function.
2. The method as described in claim 1, characterized in that, During the context dependency capture process of the fast spatial pyramid pooling module, the following mathematical expression is satisfied: ； ； ； ； ； ； ； ； In the formula, It consists of a convolutional module, batch normalization, and the SiLU activation function. It consists of multiple Bottleneck blocks, each containing two convolutional layers; This indicates that cross-modal soft fusion operations are performed on the extracted visible and infrared light; This indicates the performance of a multi-scale exchange gating operation; This represents the fused feature obtained by element-level addition to the enhanced bi-branch features; SPPF stands for Fast Space Pyramid Pooling Module.
3. The method as described in claim 1, characterized in that, The loss function of the multimodal interactive object detection model includes bounding box regression loss, binary cross-entropy loss, and distribution focus loss. The expression of the loss function is as follows: ； In the formula, For bounding box regression loss, For binary cross-entropy loss, For distribution focus loss; The mathematical expression for the bounding box regression loss includes: ； In the formula, IoU represents the ratio of the overlapping area to the non-overlapping area; c represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c represents the length of the diagonal of the outer bounding box. Indicates the aspect ratio consistency item; Indicates the balance coefficient; The mathematical expression for the binary cross-entropy loss includes: ； In the formula, Indicates the confidence level of the predicted category; Indicates the adjustment parameter; Indicates the category balance factor; The mathematical expression for the distribution focus loss includes: ； In the formula, Represents the true distribution; This represents the predicted distribution.
4. A system for implementing the multimodal interactive target detection method based on soft fusion supplementation as described in claim 1, characterized in that, The system includes: A four-layer dual-branch feature extraction layer is used to extract the features obtained from the input multimodal aerial image dataset. A cross-modal soft fusion module and two multi-scale exchange gating modules are set after each of the dual-branch feature extraction layers. The cross-modal soft fusion module is used to perform feature supplementation, and the multi-scale exchange gating modules are used to enhance the fused features after feature supplementation and input the enhanced fused features into the next dual-branch feature extraction layer. The fast spatial pyramid pooling module, located after the last double-branch feature extraction layer, is used to input the features output by the last double-branch feature extraction layer into the fast spatial pyramid pooling module for context dependency capture and then output them to the neck. A detection head is used to obtain aerial target detection results based on the features output from the neck.
5. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the multimodal interactive target detection method based on soft fusion supplementation as described in any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multimodal interactive target detection method based on soft fusion supplementation as described in any one of claims 1 to 3.