A multi-spectral target detection method based on multi-modal interaction and fusion

Abstract

The application discloses a multispectral target detection method based on multimodal interaction and fusion, and relates to the technical field of automatic driving multimodal information processing. The method can optimize the fusion effect in real time according to target characteristics and environmental changes, significantly improve the detection accuracy of the algorithm, meet the application requirements of real-time processing and resource limitation, supplement the missing information of each mode through interaction between the modes before fusion, and significantly improve the quality of cross-modal feature fusion. Fine-grained feature integration is adopted, which not only provides more accurate and rich local features, but also gradually establishes and enhances the long-range dependency relationship between modes, improves the global consistency and representation ability of multimodal features. Finally, the light and efficient interaction and fusion module designed by the method has stronger practicability, is suitable for current target detection tasks, can be popularized to multispectral target detection in other fields such as remote sensing, and has wide applicability and expansibility.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information processing technology for autonomous driving, and particularly relates to a multispectral target detection method based on multimodal interaction and fusion. Background Technology

[0002] The rise of multispectral target detection technology is rooted in the rapid advancement of sensing technology and its deep interdisciplinary integration, particularly in the fields of autonomous driving and intelligent surveillance, where it is leading a technological revolution. Companies like FluxData have provided customized solutions that have demonstrated remarkable effectiveness in various fields, including military reconnaissance, industrial monitoring, and scientific research. This innovative practice validates the broad application potential of multispectral imaging technology. The core of multispectral target detection technology lies in the ingenious combination of the advantages of visible light and infrared imaging, overcoming the limitations of single-modal imaging to achieve accurate target identification and tracking under all weather and terrain conditions.

[0003] In well-lit environments, visible light sensors, with their high resolution and rich color and texture information, provide a solid foundation for initial object identification. However, facing the challenges of low light or nighttime conditions, infrared imaging technology, with its strong penetrating power and ability to capture thermal radiation sources, fills the gap in visible light technology, ensuring the continuity and reliability of target detection under extreme lighting conditions. Although infrared imaging performs excellently in low-light vision, its lower spatial resolution limits the accurate capture of target details.

[0004] Therefore, the introduction of multispectral imaging technology aims to overcome the limitations of their individual applications by integrating complementary information from visible light and infrared images, achieving a comprehensive improvement from basic visual processing to advanced target recognition. This fusion not only enhances target detection capabilities in complex lighting and adverse weather conditions but also improves the recognition rate of occluded or poorly visible objects by analyzing multi-band spectral information, laying a technological foundation for the safe navigation of autonomous vehicles and the efficient operation of intelligent security systems. Multispectral target detection algorithms, acting as a bridge between theory and practice, combine the advantages of visible light and infrared images, significantly improving performance under conditions of drastic lighting changes and adverse weather. Visible light images provide rich color and texture information, while infrared images are more advantageous at night or in low-visibility conditions. This integration greatly enhances the accuracy and robustness of target detection, laying a solid technical foundation for the development of real-time monitoring and intelligent navigation technologies.

[0005] Therefore, developing a target detection algorithm that can efficiently fuse multispectral information is of significant research and application value for improving detection accuracy and system efficiency. Although current pixel-level, feature-level, and decision-level fusion methods each have their advantages, feature-level fusion has become the mainstream research approach due to its balance between detection accuracy and processing speed. However, existing technologies still face challenges when fusing infrared and visible light images. The main problem is that the two modal features are often generated independently, each with its own missing information, and there is a lack of an efficient mechanism to construct their long-range dependencies. This limits the quality of the fused features, and consequently, restricts the improvement of model accuracy.

[0006] "CN113361466B: A Multispectral Target Detection Method Based on Multimodal Cross-Guided Learning" extracts features from visible light and infrared images using a YOLOv5 network and designs weighted perception for precise weighting and feedback to promote long-term dependencies between modalities. However, its model generalization ability is limited to the KAIST multispectral driving dataset, which only contains pedestrian targets. "CN117830878A: A Small Target Detection Method for UAVs Based on Multispectral Interactive Attention Fusion" targets a self-built remote sensing multispectral dataset. It utilizes an improved YOLOv5 model and an adaptive fusion module based on Transformer to achieve cross-modal information fusion in the feature extraction and fusion stages. While it achieves high-precision information fusion, its computational cost is high, making it unsuitable for resource-constrained environments such as edge computing devices. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a multispectral target detection method based on multimodal interaction and fusion, which solves the problem that current methods for multimodal fusion suffer from missing information in both infrared and visible light features, and the lack of long-range dependencies, thus affecting the quality of fused features.

[0008] The technical solution of this invention is as follows:

[0009] A multispectral target detection method based on multimodal interaction and fusion includes the following steps:

[0010] Step 1: Obtain a multispectral target detection dataset; the multispectral target detection dataset includes several samples, each sample includes a visible light image, the corresponding infrared image, and annotations of the target label category and bounding box position in the visible light image and the infrared image; the information at the corresponding pixel positions in the visible light image and the corresponding infrared image are consistent;

[0011] Step 2: Denoise the visible light and infrared images in the spectral target detection dataset to obtain the denoised spectral target detection dataset;

[0012] Step 3: Divide the denoised spectral target detection dataset into training and testing sets according to the set ratio;

[0013] Step 4: Construct an improved dual-stream YOLOv5 target detection model;

[0014] The improved dual-stream YOLOv5 target detection model includes a first backbone network, a second backbone network, a cross-modal feature interaction and fusion module group, a feature fusion network, and a detection module;

[0015] The first backbone network is used to extract features from the input light image at different levels, and sends the features of the last three levels as the first shallow feature, the first middle feature, and the first deep feature to the cross-modal feature interaction and fusion module group;

[0016] The second backbone network is used to extract features from the input infrared image at different levels. Similar to the first backbone network, it only uses the features of the last three levels as the second shallow features, the second middle features, and the second deep features and sends them to the cross-modal feature interaction and fusion module group.

[0017] Both the first and second backbone networks adopt the backbone network structure of the YOLOv5 model.

[0018] The cross-modal feature interaction and fusion module group is used to perform cross-modal feature interaction and fusion on the first shallow feature and the second shallow feature, the first middle feature and the second middle feature, the first deep feature and the second deep feature, respectively, to obtain shallow multimodal fusion features, middle multimodal fusion features and deep multimodal fusion features, and send them to the feature fusion network;

[0019] The feature fusion network is used to supplement the scale information of the input shallow multimodal fusion features, mid-level multimodal fusion features, and deep multimodal fusion features, and to perform multi-level feature fusion to obtain the first multi-scale fusion feature, the second multi-scale fusion feature, and the third multi-scale fusion feature, which are then passed to the detection module; the feature fusion network adopts the neck network of the YOLOv5 model;

[0020] The detection module includes three detection heads, each of which is used to predict the first multi-scale fusion feature, the second multi-scale fusion feature and the third multi-scale fusion feature, and generate prediction results, including the target classification, confidence score and bounding box.

[0021] Furthermore, the cross-modal feature interaction and fusion module group includes a first cross-modal feature interaction and fusion module, a second cross-modal feature interaction and fusion module, and a third cross-modal feature interaction and fusion module;

[0022] The first cross-modal feature interaction and fusion module takes a first shallow feature and a second shallow feature as input and outputs a shallow multimodal fusion feature. The second cross-modal feature interaction and fusion module takes a first mid-level feature and a second mid-level feature as input and outputs a mid-level multimodal fusion feature. The third cross-modal feature interaction and fusion module takes a first deep feature and a second deep feature as input and outputs a deep multimodal fusion feature.

[0023] All three cross-modal feature interaction and fusion modules include a cross-modal feature interaction module based on adaptive feature switching and a cross-modal feature fusion module based on global context aggregation. The cross-modal feature interaction and fusion modules supplement the missing information between modalities by performing cross-modal feature interaction on the input features through the cross-modal feature interaction module based on adaptive feature switching, and perform cross-modal global and contextual information fusion between modalities through the cross-modal feature fusion module based on global context aggregation to obtain multimodal fusion features at different levels and send them to the feature fusion network.

[0024] The cross-modal feature interaction module based on adaptive feature switching performs a process of supplementing missing information between modalities through cross-modal feature interaction on the input features, specifically as follows:

[0025] First, the features of the two modalities extracted from the two backbone networks, namely the features X of the visible light image, are then analyzed. V and the characteristics of infrared images X I ∈R C×H×W Both features are divided into G subgroups along the channel dimension, denoted as follows: and Let the features of the two modal subgroups be respectively k = 1, 2, ..., G, where For the subgroup features of the visible light image modality, For the subgroup features of the infrared image modality, k is the subgroup number, C is the number of channels of the input feature, H is the height of the input feature, W is the width of the input feature, and G is the number of subgroup features;

[0026] Furthermore, the subgroup features of each visible light image mode are... and its corresponding infrared image mode subgroup features Divide the data equally into two branches, and for each visible light image mode, divide the subgroup features into two branches. and its corresponding infrared image mode subgroup features Sub-features of the first branch and Global average pooling is used to encode global information along the horizontal direction, thereby obtaining a two-modal global feature vector:

[0027]

[0028] Among them, s V s is the global feature vector of the visible light mode. I This represents the global feature vector of the infrared mode. and These represent the global average pooling operations for the visible light mode and the infrared mode, respectively. and Let (i,j) represent the sub-features of the first branch of the visible light mode and the infrared mode, respectively. (i,j) is the pixel point in the spatial dimension, where i is the coordinate of the pixel point in the height direction and j is the coordinate of the pixel point in the width direction, and (i,j)∈{(i,j)|i∈N,j∈N,0≤i≤H,0≤j≤W};

[0029] Then, a combination of fully connected layers and the Sigmoid activation function is used to process the global feature vectors s of the visible light modes. V and the global eigenvectors s of the infrared modes I ∈R C / 2G×1×1 Normalization is performed to obtain the evaluation weights of the channel dimension that reflect the importance of each modal channel. and

[0030]

[0031] Here, f1 and f2 are denoted as the fully connected layers for the first branch of the visible light mode and the infrared mode, respectively. The weights learned by the fully connected layer f1 of the first branch of the visible light mode. These are the weights learned by the fully connected layer f2 of the first branch of the infrared modality. and σ represents the biases learned by the fully connected layers f1 and f2 for the visible light mode and infrared mode, respectively, and σ is the sigmoid activation function. Indicates element-wise multiplication. and These represent the evaluation weights for the visible light modal features and the infrared modal features, respectively.

[0032] In another branch, group normalization is combined with the sigmoid function to obtain the evaluation weights for the two modal spatial dimensions. and

[0033]

[0034] in, and This represents the feature of the second branch in the visible light mode and infrared mode subgroup features, with GN corresponding to the group normalization operation. and These are the weights learned by the normalized layer for the visible light mode and infrared mode groups. and This is the bias learned by the normalized layer for the visible light and infrared modes, where σ is the sigmoid activation function. Indicates element-wise multiplication. and These represent the evaluation weights for the spatial dimensions of the visible light modal features and the infrared modal features, respectively.

[0035] Next, the sub-features of the first branch (i.e., the feature map in the channel dimension feature) and the features of the second branch (i.e., the pixel vector in the spatial dimension feature) that are not significant in the final prediction information are replaced with the corresponding feature map or pixel vector from another modality; the criterion for determining that the prediction information is not significant is: the evaluation weight is lower than a predefined threshold.

[0036] Specifically:

[0037] In the first branch performing channel-dimensional operations, sub-features of the first branch are used for both the visible light mode and the infrared mode. and Corresponding channel weights and The process of switching between the two modalities after evaluating each feature map in the sub-features of the first branch:

[0038]

[0039] Where c = 1, 2, ..., C / 2G represents the c-th channel, denoted as... and They represent and The c-th feature map in the channel dimension, and These represent the c-th elements in the evaluation weights for the visible light and infrared mode channels, respectively, where α is the threshold. and To evaluate the c-th feature map after the switch; when the weight of the c-th channel in the first branch sub-feature of the visible light mode or infrared mode is lower than the threshold α, the feature map of the c-th channel is replaced with the feature map of the corresponding channel of the other mode; the output is... and They represent and The c-th feature map in the output, therefore the first branch sub-feature of the two modes is: and

[0040] Meanwhile, the feature switching process of the second branch performing spatial dimension operations is as follows:

[0041]

[0042] in, and They represent and A pixel vector in the mid-space dimension. and represents the weight of pixel (i,j) in the evaluation weight of the spatial dimension of the visible light mode and the infrared mode, respectively; β is a threshold. When the weight of the pixel vector in the second branch sub-feature of the visible light mode or the infrared mode is lower than the threshold β, the pixel vector is replaced with the corresponding pixel vector of the other mode. and Let represent the pixel vectors after feature switching, and represent the second branch sub-features output. and

[0043] Finally, the visible light mode subgroup features and the infrared mode subgroup features are obtained by aggregation through channel connection operations. and

[0044]

[0045] in, and These are sub-features of the first and second branches of the visible light mode. and These are sub-features of the first and second branches of the infrared mode; concat is denoted as the channel connection operation. and Features of the k-th visible light mode subgroup and infrared mode subgroup;

[0046] And a single-channel shuffling operation is applied to obtain visible light mode features and infrared mode features:

[0047]

[0048] Here, shuffle refers to the channel shuffling operation. and Visible light modal features and infrared modal features output by the cross-modal feature interaction module based on adaptive feature switching;

[0049] The cross-modal feature fusion module based on global context aggregation includes two complementary branches, each branch generating an adjustment unit for aggregating context features, and the two adjustment units adaptively adjust the query of each branch.

[0050] The process of fusing global and contextual information between modalities by the cross-modal feature fusion module based on global context aggregation is as follows:

[0051] Visible light modal features and infrared two-mode features output by the cross-modal feature interaction module based on adaptive feature switching and Layer normalization is performed to obtain the normalized visible light mode features F. V and infrared modal characteristics F I As input features, their tensor shapes are reshaped and connected along the channel dimension to obtain the fused feature F. fuse :

[0052] F fuse =concat(F V ,F I )∈R H×W×2C (10)

[0053] Among them, F fuse Features of fusion;

[0054] Through linear layer f z The obtained fusion features F fuse Projected into a new feature space:

[0055] Z 0 =f z (F fuse )∈R H×W×C (11)

[0056] Among them, f z For linear layers, Z 0 These are the context sub-features of the initial layer;

[0057] Hierarchical sub-representations Z of two modal contexts are obtained by using stacked and cascaded L-layer depthwise convolutions. (l) :

[0058]

[0059] Where l represents the number of iterations, and l = 0, 1, ..., L, L is a hyperparameter; Z 0 Z corresponds to the context sub-features of the initial layer (l) Z is the context sub-feature of the l-th layer in the two-modality architecture. (l-1) The context sub-features corresponding to the two modalities of the previous layer and l = 1, ..., L; For contextualization, DWConv is a depthwise separable convolution, and GELU is the activation function;

[0060] At layer L+1, a combination of global average pooling and GELU activation function is applied to the context sub-features of the Lth layer of the two modalities:

[0061]

[0062] Among them, Z L+1 Z represents the context sub-features of the (L+1)th layer of the two modalities obtained after global average pooling. L For the context sub-features of the Lth layer in the two-modality architecture, gap is the global average pooling operation;

[0063] At this point, all L+1 layer two-modal context sub-feature sequences have been obtained.

[0064] Two gating weight sequences are obtained by using a linear layer combined with the Softmax activation function. and The specific process is as follows:

[0065]

[0066] Where l represents the number of iterations, l∈[1,…,L+1], For the gate weight sequence The gating weights of the l-th layer, For the gate weight sequence The gating weights of the l-th layer, g V and g I This represents the linear projection operation of a linear layer;

[0067] The two obtained gate weight sequences and After performing element-wise multiplication with the context sub-features of the l-th layer of the two modalities obtained in the previous step, weighted summation is performed to aggregate the contexts of the two modalities into... and

[0068]

[0069] in, and These are the global context aggregation features for the visible light mode and the infrared mode, respectively;

[0070] Subsequently, a shared linear layer h is applied to perform affine transformations on the two global context aggregation features obtained in the previous step, resulting in two adjustment units M. V and M I :

[0071]

[0072] Here, the shared linear layer is denoted as h, and the modulation units for the visible and infrared modes are denoted as M, respectively. V and M I .

[0073] Finally, the features from the two branches are aggregated to form a cross-modal fusion feature Y based on global context aggregation:

[0074]

[0075] Where q1 and q2 are linear layers, and Y is a cross-modal fusion feature;

[0076] Cross-modal fusion features based on global context aggregation are passed to the feedforward layer (FFN), and residual connections are applied to the input features F based on the visible light mode and infrared mode. V and F I The individual information in the cross-modal fusion features based on global context aggregation after the feedforward layer is supplemented as the final output of the fusion module.

[0077] Step 5: Train the improved dual-stream YOLOv5 object detection model using the training set to obtain the trained improved dual-stream YOLOv5 object detection model;

[0078] Step 6: Input the trained improved dual-stream YOLOv5 target detection model with visible light and infrared images from the test set to obtain the target category, confidence score, and bounding box in the image.

[0079] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0080] First, the multispectral target detection method based on multimodal interaction and fusion provided by this invention can optimize the fusion effect in real time according to target characteristics and environmental changes, significantly improving the detection accuracy of the algorithm and meeting the application requirements of real-time processing and resource-constrained applications. By performing intermodal interaction before fusion to supplement the missing information of each modality, the quality of cross-modal feature fusion is significantly improved. The use of fine-grained feature integration not only provides more accurate and richer local features, but also gradually establishes and enhances long-range dependencies between modalities, improving the global consistency and representational ability of multimodal features. Finally, the lightweight and efficient interaction and fusion module designed in this method has stronger practicality, applicable not only to current target detection tasks but also extendable to multispectral target detection in other fields such as remote sensing, demonstrating broad applicability and scalability.

[0081] To achieve these effects, this patent designs two key modules: a "cross-modal feature interaction module based on adaptive feature switching" and a "cross-modal feature fusion module based on global context aggregation." The former uses a Shuffle Attention (SA) mechanism that simultaneously encodes channel and spatial attention to evaluate feature importance, switching unimportant features to features from another modality. This effectively supplements missing information from a single modality, thereby enhancing the expressive power of the features. The latter utilizes a layer-by-layer decoupled deep separable convolutional structure to progressively expand the receptive field to capture global and contextual information, supplementing missing fine-grained information, thus achieving higher-quality feature integration during fusion.

[0082] By adding a lightweight and efficient adaptive feature switching mechanism before fusion, this method can effectively supplement missing information between modalities, improve the information quality of each modality, and facilitate further mining of effective information by the network. Fine-grained fusion not only gradually establishes and strengthens long-range dependencies between modalities but also more effectively utilizes multimodal features, improving overall feature representation and task performance. In summary, this invention, through optimizing the fusion mechanism and fine-grained feature integration, not only improves detection accuracy and real-time processing capabilities but also enhances the global consistency and expressive power of features. It effectively solves the problems of missing information and lack of long-range dependencies in current infrared and visible light features used for multimodal fusion, demonstrating its innovation and practicality in the field of multimodal feature fusion. Attached Figure Description

[0083] Figure 1 Structure diagram of the improved dual-stream YOLOv5 target detection model in this embodiment of the invention;

[0084] Figure 2 A schematic diagram of the cross-modal feature interaction process based on adaptive feature switching in an embodiment of the present invention;

[0085] Figure 3 A schematic diagram of the cross-modal feature fusion process based on global context aggregation in an embodiment of the present invention. Detailed Implementation

[0086] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0087] A multispectral target detection method based on multimodal interaction and fusion includes the following steps:

[0088] Step 1: Obtain the Multispectral Object Detection (MOD) dataset; the MOD dataset includes several samples, each sample includes a visible light image, the corresponding infrared image, and annotations of the target label category and bounding box position in the visible light image and the infrared image; the information at the corresponding pixel positions in the visible light image and the corresponding infrared image are consistent;

[0089] In this embodiment, the spectral target detection datasets are specifically the M3FD and VEDAI datasets;

[0090] Step 2: Denoise the visible light and infrared images in the spectral target detection dataset to obtain the denoised spectral target detection dataset;

[0091] Step 3: Divide the denoised spectral target detection dataset into training and testing sets according to the set ratio;

[0092] Step 4: Construct as follows Figure 1 The improved dual-stream YOLOv5 target detection model shown;

[0093] The improved dual-stream YOLOv5 target detection model, such as Figure 1 The diagram shows a first backbone network (Backbone_1), a second backbone network (Backbone_2), a cross-modal feature interaction and fusion module group, a feature fusion network (Neck), and a detection module (Head).

[0094] The first backbone network is used to extract features at different levels from the input light image. The first backbone network can extract features at 5 levels, but only the features at the last three levels are used as the first shallow feature, the first middle feature, and the first deep feature and sent to the cross-modal feature interaction and fusion module group.

[0095] The second backbone network is used to extract features from the input infrared image at different levels. Similar to the first backbone network, it only uses the features of the last three levels as the second shallow features, the second middle features, and the second deep features and sends them to the cross-modal feature interaction and fusion module group.

[0096] Both the first and second backbone networks adopt the backbone network structure of the YOLOv5 model.

[0097] The cross-modal feature interaction and fusion module group is used to perform cross-modal feature interaction and fusion on the first shallow feature and the second shallow feature, the first middle feature and the second middle feature, and the first deep feature and the second deep feature, respectively, and obtain shallow multimodal fusion features, middle multimodal fusion features and deep multimodal fusion features, respectively, and send them to the feature fusion network; through the embedded cross-modal interaction and fusion module group, the complementary advantages between modalities are fully explored and utilized to generate cross-modal fusion features, thereby performing accurate target detection tasks.

[0098] The feature fusion network is used to supplement the scale information of the input shallow multimodal fusion features, mid-level multimodal fusion features, and deep multimodal fusion features, and to perform multi-level feature fusion to obtain the first multi-scale fusion feature, the second multi-scale fusion feature, and the third multi-scale fusion feature, which are then passed to the detection module; the feature fusion network adopts the neck network of the YOLOv5 model;

[0099] In this implementation, the feature fusion network (Neck) employs a Path Aggregation Network (PANet) and a Feature Pyramid Network (FPN) to achieve a bidirectional feature fusion mechanism, moving both top-down and bottom-up. This design enhances the model's robustness and flexibility when handling targets at different scales. FPN ensures that high-level semantic information propagates downwards to feature maps at various scales, while PANet enhances the semantic richness of low-level features when propagating upwards. This bidirectional information flow mechanism significantly improves the expressive power of features and enhances the model's ability to detect multi-scale targets in complex scenes. Through fusion operations such as convolution, upsampling, and downsampling, the Neck effectively integrates feature maps from different levels, enabling the final detection head to perform more accurate target localization and classification based on the fused features.

[0100] The detection module includes three detection heads, each of which is used to predict the first multi-scale fusion feature, the second multi-scale fusion feature and the third multi-scale fusion feature, and generate prediction results, including the target classification, confidence score and bounding box.

[0101] The detection head generates prediction results containing target category, location, and confidence level through convolutional layers and a multi-scale prediction mechanism. Multi-scale prediction ensures detection performance for targets of different sizes. Anchor box mechanisms are applied to provide initial bounding box references to further optimize the regression accuracy of the predicted boxes. After generating the initial prediction results, YOLOv5 uses Non-Maximum Suppression (NMS) for post-processing. NMS filters out redundant detection boxes with high overlap by calculating the intersection-union ratio between predicted boxes, retaining only the detection results with the highest confidence and optimal location. The post-processing step is mainly used to filter the classification confidence and localization boxes of the three input targets, ensuring that each target corresponds to only one high-confidence detection result.

[0102] Furthermore, the cross-modal feature interaction and fusion module group includes a first cross-modal feature interaction and fusion module, a second cross-modal feature interaction and fusion module, and a third cross-modal feature interaction and fusion module;

[0103] The first cross-modal feature interaction and fusion module takes a first shallow feature and a second shallow feature as input and outputs a shallow multimodal fusion feature. The second cross-modal feature interaction and fusion module takes a first mid-level feature and a second mid-level feature as input and outputs a mid-level multimodal fusion feature. The third cross-modal feature interaction and fusion module takes a first deep feature and a second deep feature as input and outputs a deep multimodal fusion feature.

[0104] The cross-modal feature interaction and fusion modules each include a cross-modal feature interaction module based on adaptive feature switching and a cross-modal feature fusion module based on global context aggregation. The cross-modal feature interaction and fusion modules are used to supplement missing information between modalities by using the cross-modal feature interaction module based on adaptive feature switching to process the input features through cross-modal feature interaction, and to fuse global and contextual information between modalities by using the cross-modal feature fusion module based on global context aggregation to obtain multimodal fusion features at different levels and send them to the feature fusion network.

[0105] The cross-modal feature interaction module based on adaptive feature switching completes the process of supplementing missing information between modalities through cross-modal feature interaction on the input features, such as... Figure 2 As shown, specifically:

[0106] The design idea of ​​the cross-modal feature interaction module based on adaptive feature switching is to replace the feature map of one modality with the corresponding feature of another modality, so that each modality retains its unique features while effectively fusing features from other modalities.

[0107] First, the features of the two modalities extracted from the backbone network, namely the features X of the visible light image, are... V and the characteristics of infrared images XI ∈R C×H×W Both features are divided into G subgroups along the channel dimension, denoted as follows: and To simultaneously evaluate the weights of each channel and the spatial pixel vector in the input feature subgroup, let the two modal subgroup features be denoted as follows: k = 1, 2, ..., G, where For the subgroup features of the visible light image modality, For the subgroup features of the infrared image modality, k is the subgroup number, C is the number of channels of the input feature, H is the height of the input feature, W is the width of the input feature, and G is the number of subgroup features. For the three scales of input features, namely the first shallow feature and the second shallow feature, the first middle feature and the second middle feature, and the first deep feature and the second deep feature, G is set to 2, 4, and 8 respectively.

[0108] Furthermore, the subgroup features of each visible light image mode are... and its corresponding infrared image mode subgroup features Divide the data equally into two branches, and for each visible light image mode, divide the subgroup features into two branches. and its corresponding infrared image mode subgroup features Sub-features of the first branch and Global average pooling (gap) is used to encode global information along the horizontal direction, thereby obtaining a two-modal global feature vector:

[0109]

[0110] Among them, s V s is the global feature vector of the visible light mode. I This represents the global feature vector of the infrared mode. and These represent the global average pooling operations for the visible light mode and the infrared mode, respectively. and Let (i,j) represent the sub-features of the first branch of the visible light mode and the infrared mode, respectively. (i,j) is the pixel point in the spatial dimension, where i is the coordinate of the pixel point in the height direction and j is the coordinate of the pixel point in the width direction, and (i,j)∈{(i,j)|i∈N,j∈N,0≤i≤H,0≤j≤W};

[0111] Then, the combination of a fully connected layer and a Sigmoid activation function (denoted as σ) is used to process the global feature vector s of the visible light modes. V and the global eigenvectors s of the infrared modes I ∈R C / 2G×1×1Normalization is performed to obtain the evaluation weights of the channel dimension that reflect the importance of each modal channel. and

[0112]

[0113] Here, f1 and f2 are denoted as the fully connected layers for the first branch of the visible light mode and the infrared mode, respectively. The weights learned by the fully connected layer f1 of the first branch of the visible light mode. These are the weights learned by the fully connected layer f2 of the first branch of the infrared modality. and σ represents the biases learned by the fully connected layers f1 and f2 for the visible light mode and infrared mode, respectively, and σ is the sigmoid activation function. Indicates element-wise multiplication. and These represent the evaluation weights for the visible light modal features and the infrared modal features, respectively.

[0114] In another branch, the group normalization operation (GN) is combined with the sigmoid function (denoted as σ) to obtain the evaluation weights for the two modal spatial dimensions. and

[0115]

[0116] in, and This represents the feature of the second branch in the visible light mode and infrared mode subgroup features, with GN corresponding to the group normalization operation. and These are the weights learned by the normalized layer for the visible light mode and infrared mode groups. and This is the bias learned by the normalized layer for the visible light and infrared modes, where σ is the sigmoid activation function. Indicates element-wise multiplication. and These represent the evaluation weights for the spatial dimensions of the visible light modal features and the infrared modal features, respectively.

[0117] Next, the sub-features of the first branch (i.e., the feature map in the channel dimension feature) and the features of the second branch (i.e., the pixel vector in the spatial dimension feature) that are not significant in the final prediction information are replaced with the corresponding feature map or pixel vector from another modality; the criterion for determining that the prediction information is not significant is: the evaluation weight is lower than a predefined threshold.

[0118] Specifically:

[0119] In the first branch performing channel-dimensional operations, sub-features of the first branch are used for both the visible light mode and the infrared mode. and Corresponding channel weights and The process of switching between the two modalities after evaluating each feature map in the sub-features of the first branch:

[0120]

[0121] Where c = 1, 2, ..., C / 2G represents the c-th channel, then let and They represent and The c-th feature map in the channel dimension, and These represent the c-th elements in the evaluation weights for the visible light and infrared mode channels, respectively, where α is the threshold. and To evaluate the c-th feature map after the switch; when the weight of the c-th channel in the first branch sub-feature of the visible light mode or infrared mode is lower than the threshold α, the feature map of the c-th channel is replaced with the feature map of the corresponding channel of the other mode, and α is set to 1e-4; the output and They represent and The c-th feature map in the output, therefore the first branch sub-feature of the two modes is: and

[0122] Meanwhile, in the second branch performing spatial dimension operations, the feature switching process is as follows:

[0123]

[0124] in, and They represent and A pixel vector in the mid-space dimension. and These represent the weights at pixel (i,j) in the evaluation weights of the visible light mode and infrared mode spatial dimensions, respectively. β is a threshold; when the weight of a pixel vector in the second branch sub-feature of the visible light mode or infrared mode is lower than the threshold β, the pixel vector is replaced with the corresponding pixel vector of the other mode. β is set to 1e-4. and Let represent the pixel vectors after feature switching, and represent the second branch sub-features output. and

[0125] Parallel branches of spatial and channel attention enable modules to adaptively select features based on the importance of both dimensions simultaneously, resulting in more accurate and effective representations.

[0126] Finally, the visible light mode subgroup features and the infrared mode subgroup features are obtained by aggregation through channel connection operations. and

[0127]

[0128] in, and These are sub-features of the first and second branches of the visible light mode. and These are sub-features of the first and second branches of the infrared mode; concat is denoted as the channel connection operation. and Features of the k-th visible light mode subgroup and infrared mode subgroup;

[0129] A channel shuffle operation is applied to enhance cross-channel interactions within each mode:

[0130]

[0131] Here, shuffle refers to the channel shuffling operation. and Visible light modal features and infrared modal features are output by the cross-modal feature interaction module based on adaptive feature switching.

[0132] By adaptively switching between features of another modality based on the importance of each modality's channel and spatial dimension features, information missing from a single modality can be effectively supplemented, improving semantic consistency of features. This interactive approach enables each modality to obtain a more powerful and information-rich feature representation, thereby achieving more accurate fusion.

[0133] The cross-modal feature fusion module based on global context aggregation, such as Figure 3 As shown, the aggregation and individual query processes are decoupled to alleviate the problem of the visual Transformer attention module requiring a large number of complex interactions and aggregations, thereby effectively reducing the number of parameters and more efficiently modeling long-range dependencies between modalities.

[0134] Specifically, the cross-modal feature fusion module based on global context aggregation includes two complementary branches, each generating an adjustment unit for aggregating context features. Therefore, the two adjustment units can adaptively adjust the query of each branch. The core of this module lies in obtaining context aggregation information between modalities, including two steps: (1) obtaining context features at different granularities, from local to global scope; (2) aggregating and compressing the context features at each granularity into the adjustment unit through a gating mechanism.

[0135] The first step involves processing the visible light modal features and infrared two-modal features output by the cross-modal feature interaction module based on adaptive feature switching. and Layer normalization is performed to obtain the normalized visible light modal features F. V and infrared modal characteristics F I As input features, their tensor shapes are reshaped and connected along the channel dimension to obtain the fused feature F. fuse :

[0136] F fuse =concat(F V ,F I )∈R H×W×2C (10)

[0137] Among them, F fuse Features of fusion;

[0138] Through linear layer f z The obtained fusion features F fuse Projected into a new feature space:

[0139] Z 0 =f z (F fuse )∈R H×W×C (11)

[0140] Among them, f z For linear layers, Z 0 These are the context sub-features of the initial layer;

[0141] Hierarchical sub-representations Z of two modal contexts are obtained by using stacked and cascaded L-layer depthwise convolutions. (l) :

[0142]

[0143] Where l represents the number of iterations, and l = 0, 1, ..., L, L is a hyperparameter, which is set to 3 in the experiment; Z 0 Z corresponds to the context sub-features of the initial layer (l=0). (l)Z is the context sub-feature of the l-th layer in the two-modality architecture. (l-1) (l=1,…,L) corresponds to the context sub-features of the previous layer in both modalities. This is a contextualization function, specifically using a convolution kernel with size p. l The depthwise separable convolution DWConv is used in conjunction with the GELU activation function. Let r be the receptive field of the l-th layer. l , p l This demonstrates that using this decoupled structure, the receptive field of depthwise convolution is significantly expanded, facilitating further extraction of effective information from both modalities. Furthermore, compared to conventional convolution, depthwise convolution requires less computation, and compared to pooling operations, depthwise separable convolution exhibits greater learnability and structure awareness within the network.

[0144] To obtain the overall semantic and structural features of the feature space, a combination of global average pooling and GELU activation function is applied to the contextual sub-features of the Lth layer of the two modalities at the (L+1)th layer to provide generalized global information:

[0145]

[0146] Among them, Z L+1 Z represents the context sub-features of the (L+1)th layer of the two modalities obtained after global average pooling. L For the context sub-features of the Lth layer in the two-modality architecture, gap is the global average pooling operation;

[0147] At this point, all L+1 layer two-modal context sub-feature sequences have been obtained. Together they capture short-term and long-term contexts at different granular levels.

[0148] Step 2: In order to map and compress the context sub-features of the L+1 layer obtained by hierarchical contextualization into adjustment units, two gating sequences are used to control how much information each query aggregates from different levels.

[0149] Specifically, a linear layer combined with the Softmax activation function is first used to obtain two sequences of gating weights. and The specific process is as follows:

[0150]

[0151] Where l represents the number of iterations, l∈[1,…,L+1], For the gate weight sequence The gating weights of the l-th layer, For the gate weight sequence The gating weights of the l-th layer, g V and g IThis represents the linear projection operation of a linear layer;

[0152] The two obtained gate weight sequences and After performing element-wise multiplication with the context sub-features of the l-th layer of the two modalities obtained in the previous step, weighted summation is performed to aggregate the contexts of the two modalities into... and

[0153]

[0154] in, and These are the global context aggregation features for the visible light mode and the infrared mode, respectively;

[0155] Subsequently, a shared linear layer h is applied to perform affine transformations on the two global context aggregation features obtained in the previous step, resulting in two adjustment units M. V and M I :

[0156]

[0157] Here, the shared linear layer is denoted as h, and the modulation units for the visible and infrared modes are denoted as M, respectively. V and M I .

[0158] Finally, the features from the two branches are aggregated to form a cross-modal fusion feature Y based on global context aggregation after interaction between the global and local contexts of the two modalities:

[0159]

[0160] Among them, the input features F for visible light mode and infrared mode V and F I Two linear layers, denoted as q1 and q2, are applied to implement query projection mapping. It is element-wise addition, and Y is a cross-modal fusion feature based on global context aggregation.

[0161] Cross-modal fusion features based on global context aggregation are passed to the feedforward layer (FFN). The two modules are cascaded to form a fusion module group, and residual connections are applied to input features F based on visible light and infrared modes. V and F I The individual information in the cross-modal fusion features based on global context aggregation after the feedforward layer is supplemented as the final output of the fusion module.

[0162] Step 5: Train the improved dual-stream YOLOv5 object detection model using the training set to obtain the trained improved dual-stream YOLOv5 object detection model;

[0163] This implementation uses the loss function of a conventional object detection model to guide model training, which includes three parts: classification loss, object loss, and regression loss. The classification and object losses use binary cross-entropy loss, the regression loss uses CIoU loss, and the total loss is the weighted sum of the three losses: classification loss, object loss, and regression loss. The calculation formula is as follows:

[0164] Loss=λ1L cls +λ2L obj +λ3L loc (18)

[0165] Loss represents the model loss function, L cls L represents the classification loss. obj L represents the target loss. loc Let λ1, λ2, and λ3 represent the confidence loss, and λ3 be hyperparameters.

[0166] Step 6: Input the trained improved dual-stream YOLOv5 target detection model with visible light and infrared images from the test set to obtain the target category, confidence score, and bounding box in the image.

[0167] This patent selects the M3FD multispectral dataset for broader driving scenarios from Liu J, Fan X, Huang Z, et al. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection [C]. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2022:5802-5811. Its imaging system includes a binocular optical camera and a synchronized binocular infrared sensor. The baselines (distance between the focal centers of the binocular lenses) of the visible and infrared binocular cameras are 12 cm and 20 cm, respectively. The optical center distance between the visible and infrared sensors is 4 cm. The high resolution of the visible image is 1024×768, while the standard resolution of the infrared image is 640×512, with a wavelength range of 8-14 μm. This dataset includes four scenarios: daytime, cloudy, nighttime, and a challenge scenario, totaling 4200 pairs of visible light and infrared images and 34407 target labels, covering six common driving target classes: People, Car, Bus, Motorcycle, Lamp, and Truck. The experiments employed a random partitioning method, selecting 3200 image pairs for training and 1000 pairs for validation. Input images were uniformly resized to 640×640 to improve model efficiency.

[0168] The ablation experimental results of this patented model on the M3FD dataset, categorized by type, are as follows:

[0169] Table 1. Ablation experiments of the M3FD multispectral dataset by category (AP) 0.5 )

[0170]

[0171] The model implementing only "cross-modal feature fusion based on global context aggregation" is denoted as CAFF. The model implementing both "cross-modal feature interaction based on adaptive feature switching" and "cross-modal feature fusion based on global context aggregation" is denoted as CIAFF. It can be seen that CAFF achieves a 2.9% improvement over the higher of the two multimodal baseline models based on feature addition and channel concatenation. CIAFF further improves upon the CAFF baseline model by 1.2%. Meanwhile, the model achieves good results in most categories, with CIAFF showing a significant improvement over CAFF on the difficult sample classes Lamp and People, reaching 2.1% and 2% respectively.

[0172] Table 2 shows the comparison results of this patented model with different algorithms on the M3FD dataset in terms of relevant evaluation metrics.

[0173] Table 2 Comparison of relevant evaluation metrics of this application and different algorithms on the M3FD multispectral dataset.

[0174]

[0175] The CFT proposed by Qingyun F, Dapeng H, Zhaokui W. Cross-modality fusion transformer for multispectral object detection[J].arXiv:2111.00273,2021. is the first cross-modal fusion method based on the Transformer self-attention mechanism, achieving effective intra-modal and inter-modal fusion. The ADCNet proposed by He, Mingzhou, et al. Misaligned RGB-Infrared Object Detection via Adaptive Dual-Discrepancy Calibration[J].Remote Sensing,2023,15(19):4887 / 1-23. is also based on the dual-stream YOLOv5 framework, embedding a spatial difference calibration module and a domain difference calibration module. The former achieves spatial alignment of features through adaptive affine transformation, while the latter aligns object and background features of different modalities to improve the discriminative power of the fused features. The mAP of this patented model... 0.5 and mAP 0.5:0.95 The figures were 87.1% and 55.6% respectively, in mAP 0.5 It outperforms the CFT model by 2.7% and the ADCNet model by 1.2%. Furthermore, it achieves state-of-the-art performance across the vast majority of target classes.

[0176] To verify the model's generalization ability, the VEDAI remote sensing scene multispectral dataset, derived from Razakarivony S, Julie F. Vehicle detection inaerial imagery: A small target detection benchmark[J]. Journal of Visual Communication and Image Representation, 2016, 34: 187-203, was selected. This dataset contains 1246 pairs of RGB-IR images and over 3700 labeled targets, covering various scenes such as grasslands, rural roads, mountainous areas, and urban landscapes, all captured at a constant altitude. The dataset provides two resolutions; the experiment used a 1024×1024 resolution. The ablation experiment results of this patented model on the VEDAI dataset, categorized by type, are as follows:

[0177] Table 3 Ablation experiments by category in the VEDAI multispectral dataset (AP) 0.5 )

[0178]

[0179] As can be seen, CAFF and CIAFF achieved good results in most classes, with CIAFF showing a significant improvement over CAFF in the difficult sample classes boat and van.

[0180] Table 4 shows the comparison results of this patented model with different algorithms on the VEDAI dataset in terms of relevant evaluation metrics.

[0181] Table 4 Comparison of relevant evaluation metrics of this application and different algorithms on the VEDAI multispectral dataset.

[0182]

[0183] SuperYOLO, from Zhang J, Lei J, Xie W, et al. SuperYOLO: Super resolution assisted object detection in multimodal remote sensing imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-15., introduces assisted super-resolution learning to fuse multimodal data and constructs a lightweight detection framework, significantly improving the detection accuracy and computational efficiency for small targets at multiple scales. ICAFSion, from Shen J, Chen Y, Liu Y, et al. ICAFSion: Iterative cross-attention guided feature fusion for multispectral object detection[J]. Pattern Recognition, 2024, 145: 109913 / 1-12., utilizes two independent Transformer-based cross-attention modules to capture cross-modal feature interactions and leverages an iterative mechanism to reduce model complexity and computational cost. The mAP of the model in this embodiment... 0.5 and mAP 0.5:0.95 The figures were 76.6% and 46.2% respectively, in mAP 0.5 Exceeding the SuperYOLO model by 3%. In mAP 0.5:0.95 This exceeds the ICA Fusion model by 1.3%.

Claims

1. A multispectral target detection method based on multimodal interaction and fusion, characterized in that, Includes the following steps: Step 1: Obtain the multispectral target detection dataset; Step 2: Denoise the visible light and infrared images in the spectral target detection dataset to obtain the denoised spectral target detection dataset; Step 3: Divide the denoised spectral target detection dataset into training and testing sets according to the set ratio; Step 4: Construct an improved dual-stream YOLOv5 target detection model; Step 5: Train the improved dual-stream YOLOv5 object detection model using the training set to obtain the trained improved dual-stream YOLOv5 object detection model; The improved dual-stream YOLOv5 target detection model includes a first backbone network, a second backbone network, a cross-modal feature interaction and fusion module group, a feature fusion network, and a detection module; The first backbone network and the second backbone network perform feature extraction at different levels on the input visible light image and infrared image, respectively. The three levels of features obtained from the feature extraction of the visible light image are used as the first shallow feature, the first middle feature, and the first deep feature and sent to the cross-modal feature interaction and fusion module group; the last three levels of features obtained from the feature extraction of the infrared image are used as the second shallow feature, the second middle feature, and the second deep feature and sent to the cross-modal feature interaction and fusion module group. Both the first and second backbone networks adopt the backbone network structure of the YOLOv5 model. The cross-modal feature interaction and fusion module group is used to perform cross-modal feature interaction and fusion on the first shallow feature and the second shallow feature, the first middle feature and the second middle feature, and the first deep feature and the second deep feature, respectively, to obtain shallow, middle and deep multimodal fused features and send them to the feature fusion network; The feature fusion network is used to supplement the scale information of the input shallow, medium and deep multimodal fusion features, and to perform multi-level feature fusion to obtain the first, second and third multi-scale fusion features and pass them to the detection module; the feature fusion network adopts the neck network of the YOLOv5 model; The detection module includes three detection heads, each of which is used to predict the first, second and third multi-scale fusion features and generate prediction results, including the target classification, confidence score and bounding box. The cross-modal feature interaction and fusion module group includes three modal feature interaction and fusion modules. Each of the three cross-modal feature interaction and fusion modules includes a cross-modal feature interaction module based on adaptive feature switching and a cross-modal feature fusion module based on global context aggregation. The cross-modal feature interaction and fusion modules supplement the missing information between modalities by using the cross-modal feature interaction module based on adaptive feature switching to supplement the missing information between modalities. The cross-modal feature fusion module based on global context aggregation performs global and contextual information fusion between modalities to obtain multimodal fusion features at different levels and sends them to the feature fusion network. Step 6: Input the trained improved dual-stream YOLOv5 target detection model with visible light and infrared images from the test set to obtain the target category, confidence score, and bounding box in the image.

2. The multispectral target detection method based on multimodal interaction and fusion according to claim 1, characterized in that, The multispectral target detection dataset mentioned in step 1 includes several samples. Each sample includes a visible light image, an infrared image corresponding to the visible light image, and annotations on the target label category and bounding box position in the visible light image and the infrared image. The information at the corresponding pixel positions in the visible light image and the infrared image corresponding to the visible light image is consistent.

3. The multispectral target detection method based on multimodal interaction and fusion according to claim 2, characterized in that, The cross-modal feature interaction and fusion module group includes a first cross-modal feature interaction and fusion module, a second cross-modal feature interaction and fusion module, and a third cross-modal feature interaction and fusion module; The input to the first cross-modal feature interaction and fusion module is the first shallow feature and the second shallow feature, and the output is the shallow multimodal fusion feature; The input to the second cross-modal feature interaction and fusion module is the first mid-level feature and the second mid-level feature, and the output is the mid-level multimodal fusion feature; the input to the third cross-modal feature interaction and fusion module is the first deep feature and the second deep feature, and the output is the deep multimodal fusion feature.

4. The multispectral target detection method based on multimodal interaction and fusion according to claim 3, characterized in that, The cross-modal feature interaction module based on adaptive feature switching performs a process of supplementing missing information between modalities through cross-modal feature interaction on the input features, specifically as follows: First, the features of the two modalities extracted from the two backbone networks are used to extract the features of the visible light image. Features of infrared images Both features are divided into G subgroups along the channel dimension, denoted as follows: and Let the features of the two modal subgroups be respectively ,in For the subgroup features of the visible light image modality, Subgroup features of infrared image modes, Where C is the subgroup number, H is the input feature height, W is the input feature width, and G is the number of subgroup features. Furthermore, the subgroup features of each visible light image mode are... and its corresponding infrared image mode subgroup features Divide the data equally into two branches, and for each visible light image mode, divide the subgroup features into two branches. and its corresponding infrared image mode subgroup features Sub-features of the first branch and Global average pooling is used to encode global information along the horizontal direction, thereby obtaining a two-modal global feature vector: （1） in, This represents the global feature vector of the visible light mode. This represents the global feature vector of the infrared mode. and These represent the global average pooling operations for the visible light mode and the infrared mode, respectively. and These represent the sub-features of the first branch of the visible light mode and the infrared mode, respectively. For pixels in spatial dimensions, Let be the coordinates of the pixel in the height direction. Let be the coordinates of the pixel in the width direction, and ; Then, using fully connected layers and Combinations of activation functions to separately apply to the global feature vectors of visible light modes Global eigenvectors of infrared modes Normalization is performed to obtain the evaluation weights of the channel dimension that reflect the importance of each modal channel. and : （2） in, and These are denoted as the fully connected layers for the first branch applications of the visible light mode and the infrared mode, respectively. Fully connected layer for the first branch of visible light modes The weight of learning It is a fully connected layer of the first branch of infrared modes. The weight of learning and Fully connected layers for visible light and infrared modes, respectively. and Learning bias for Activation function Indicates element-wise multiplication. and These represent the evaluation weights for the visible light modal features and the infrared modal features, respectively. In another branch, the group normalization operation is applied and... Function combinations are used to obtain the evaluation weights for the two modal spatial dimensions. and : （3） in, and This represents the feature of the second branch in the visible light mode and infrared mode subgroup features, with GN corresponding to the group normalization operation. and These are the weights learned by the normalized layer for the visible light mode and infrared mode groups. and It is the bias learned by the normalization layer for visible light mode and infrared mode groups. for Activation function Indicates element-wise multiplication. and These represent the evaluation weights for the spatial dimensions of visible light modal features and infrared modal features, respectively. Next, the sub-features of the first branch (i.e., the feature map in the channel dimension feature) and the features of the second branch (i.e., the pixel vector in the spatial dimension feature) that are not significant in the final prediction information are replaced with the corresponding feature map or pixel vector from another modality; the criterion for determining that the prediction information is not significant is: the evaluation weight is lower than a predefined threshold. Specifically: In the first branch performing channel-dimensional operations, sub-features of the first branch are used for both the visible light mode and the infrared mode. and Corresponding channel weights and The process of evaluating and switching between each feature map in the sub-features of the first branch of each of the two modalities: （4） （5） in, Indicates the first One channel, record and They represent and Channel dimension Each feature map and The weights of the visible light mode and infrared mode channel dimensions are respectively represented by the first value in the evaluation weights. One element, For the threshold, and To conduct an evaluation of the first after the switch The first feature map; when the first branch sub-feature of the visible light mode or infrared mode is... The weight of each channel is below the threshold. At that time, the first The feature map of one channel is replaced with the feature map of the corresponding channel of another modality; the output and They represent and The Middle Therefore, the output of the first branch sub-feature of the two modalities is: and ; Meanwhile, the feature switching process of the second branch performing spatial dimension operations is as follows: （6） （7） in, and They represent and A pixel vector in the mid-space dimension. and The pixel values ​​in the evaluation weights represent the spatial dimensions of the visible light mode and the infrared mode, respectively. Weight at each location; The threshold is set when the weight of the pixel vector in the second branch sub-feature of the visible light mode or infrared mode is lower than the threshold. When this happens, the pixel vector is replaced with the corresponding pixel vector of another modality; and Let represent the pixel vectors after feature switching, and represent the second branch sub-features output. and ; Finally, the visible light mode subgroup features and the infrared mode subgroup features are obtained by aggregation through channel connection operations. and : （8） in, and These are sub-features of the first and second branches of the visible light mode. and These are sub-features of the first and second branches of the infrared mode. This is recorded as a channel connection operation. and For the first Features of visible light mode subgroups and infrared mode subgroups; And a single-channel shuffling operation is applied to obtain visible light mode features and infrared mode features: （9） in, This is recorded as a channel shuffling operation. and Visible light modal features and infrared modal features are output by the cross-modal feature interaction module based on adaptive feature switching.

5. The multispectral target detection method based on multimodal interaction and fusion according to claim 3, characterized in that, The cross-modal feature fusion module based on global context aggregation includes two complementary branches, each branch generating an adjustment unit for aggregating context features, and the two adjustment units adaptively adjust the query of each branch. The process of fusing global and contextual information between modalities by the cross-modal feature fusion module based on global context aggregation is as follows: Visible light modal features and infrared two-mode features output by the cross-modal feature interaction module based on adaptive feature switching and Layer normalization is performed to obtain the normalized visible light mode features. and infrared modal characteristics As input features, their tensor shapes are reshaped and connected along the channel dimension to obtain fused features. : （10） in, Features of fusion; Through linear layer The obtained fusion features Projected into a new feature space: （11） in, For linear layers, These are the context sub-features of the initial layer; Using stacked series Layered depthwise convolutions obtain hierarchical sub-representations of two modal contexts. : （12） in, Indicates the number of iteration levels, and , For hyperparameters; Corresponding to the context sub-features of the initial layer, It is a two-modal first Contextual sub-features of the layer Corresponding to the context sub-features of the upper layer of the two modalities and ; For contextualized functions, For depthwise separable convolution, For activation functions; In the Layer to two modes The context sub-features of the layer are applied using global average pooling and Activation function combination operations: （13） in, The two modes obtained after global average pooling are... Contextual sub-features of the layer For two modes, the first Contextual sub-features of the layer This is a global average pooling operation; At this point, all has been obtained. Layer two-modal context sub-feature sequence ; Combined with a linear layer The activation function obtains two sequences of gating weights. and The specific process is as follows: （14） in, Indicates the number of iteration levels. , For the gate weight sequence The Middle Layer gating weights, For the gate weight sequence The Middle Layer gating weights, and This represents the linear projection operation of a linear layer; The two obtained gate weight sequences and Compared with the two modes obtained in the previous step, the first... After element-wise multiplication of the context sub-features of each layer, a weighted summation is performed to aggregate the two modal contexts into... and : （15） in, and These are the global context aggregation features for the visible light mode and the infrared mode, respectively; Subsequently, a shared linear layer is applied. Affine transformations are performed on the two global context aggregation features obtained in the previous step to obtain two adjustment units. and : （16） Wherein, the shared linear layer is denoted as The modulation units for visible light and infrared modes are denoted as follows: and ; Finally, the features from the two branches are aggregated to form a cross-modal fusion feature based on global context aggregation. : ， （17） in, and For linear layers, For cross-modal fusion features; Cross-modal fusion features based on global context aggregation are passed to the feedforward layer (FFN), and residual connections are applied to input features based on visible light and infrared modes. and The individual information in the cross-modal fusion features based on global context aggregation after the feedforward layer is supplemented as the final output of the fusion module.

Images