Target detection system and method based on visible-infrared remote sensing image fusion

Abstract

The present application relates to the technical field of image processing, and more particularly to a target detection system and method based on visible light-infrared remote sensing image fusion, wherein an infrared feature extraction branch and a visible light feature extraction branch extract multi-scale features from input infrared images and visible light images respectively; a feature fusion branch performs difference perception and frequency domain interaction on two features of the same scale to obtain complementary items of the two features; state space sequences are then obtained from the two features and the two complementary items, and spatial direction fusion is performed according to the state space sequences to obtain fusion features of corresponding scales; and a target detection branch uses a detection head to obtain infrared detection results, visible light detection results and fusion detection results, fuses the three detection results and outputs a final detection result. The present application can realize accurate detection of targets under complex remote sensing imaging conditions such as day-night changes, weak light and occlusion.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a target detection system and method based on visible light-infrared remote sensing image fusion. Background Technology

[0002] In recent years, the application of remote sensing platforms (especially drones) in scenarios such as security patrol, intelligent transportation, emergency rescue, and military reconnaissance has been continuously expanding, driving the development of remote sensing image target detection technology. Many existing methods primarily rely on visible light (RGB) images, which offer advantages such as clear texture and rich semantics under daytime conditions. However, under complex imaging conditions such as low illumination at night, rain and fog, backlighting, and strong reflections, target contrast decreases and details degrade, easily leading to missed detections and false detections. Infrared imaging can utilize target thermal radiation information, and can still stably present the target's position and outline under low light or poor visibility conditions, but it usually suffers from insufficient texture detail and low spatial resolution. Therefore, utilizing the complementarity of visible light and infrared modalities for fusion detection is an effective way to improve the reliability of all-weather, all-time remote sensing target detection. However, existing visible light-infrared fusion detection methods still face the following key problems: (1) There is a lack of reliable selection of fusion region and fusion scale. Simple feature splicing, element-by-element addition or global weighting can easily bring noise, pseudo-texture and conflicting semantics into the fusion space, resulting in noise amplification and semantic confusion; (2) Insufficient cross-modal collaborative modeling. Most methods only use strong representation modules as post-processing operators in the "fusion first, model later" pipeline, making it difficult to explicitly characterize the collaborative evolution of the two modes and their interaction terms at the model structure level; (3) It is difficult to consistently integrate the multi-branch detection results during the inference stage. Independent branch predictions often have duplicate boxes, drift boxes and inconsistent confidence levels for the same physical target, and lack an interpretable adaptive weight allocation mechanism in modal imbalance scenarios such as nighttime.

[0003] Therefore, there is an urgent need for a visible light-infrared remote sensing target detection method that can achieve "differential perception and controllable complementarity" at the feature level, "explicit dual-input co-evolution" at the state space level, and "target-level consistent fusion" at the reasoning stage. Summary of the Invention

[0004] In view of this, the present invention aims to provide a target detection system and method based on visible light-infrared remote sensing image fusion. Through a dual-path SimAM-wavelet attention mechanism, modal differences are jointly modeled in the spatial and frequency domains. Adaptive saliency measurement and wavelet domain energy decomposition highlight target discrimination details and suppress background interference, improving the separability of cross-modal complementary features. The binary fusion Mamba module utilizes state-space sequence modeling to achieve long-range dependency representation of visible light and infrared features and efficient cross-modal interactive fusion. Dynamic allocation of modal weights through gating coefficients enhances robustness in complex scenes. Furthermore, a three-branch perception-decision fusion detection head is proposed, which performs coordinate projection, matching, and confidence reweighting on multi-branch detection results to achieve adaptive consistency fusion and reduce false negatives and false negatives.

[0005] To achieve the above objectives, the technical solution created by this invention is implemented as follows: A target detection system based on visible light-infrared remote sensing image fusion includes: an infrared feature extraction branch, which extracts multi-scale infrared features from the input infrared image; a visible light feature extraction branch, which extracts multi-scale visible light features from the input visible light image; a feature fusion branch, which uses a dual-path SimAM-wavelet attention mechanism to perform difference perception and frequency domain interaction on infrared and visible light features of the same scale to obtain complementary terms of infrared and visible light features; then, a binary fusion Mamba module is used to obtain a state space sequence of infrared features, visible light features, and the two complementary terms, and spatial direction fusion is performed according to the state space sequence to obtain fused features of the corresponding scale; and a target detection branch, which uses a detection head to obtain infrared detection results, visible light detection results, and fused detection results based on the multi-scale features output by the infrared feature extraction branch, visible light feature extraction branch, and feature fusion branch, and fuses the three detection results to output the final detection result.

[0006] Furthermore, in the infrared feature extraction branch, multiple cascaded VSS modules are used to extract features from the infrared image step by step. The resulting multi-scale features are then aggregated with context information through spatial pyramid pooling, and then multi-scale information is integrated from top to bottom and from bottom to top to obtain multi-scale infrared features.

[0007] Furthermore, in the visible light feature extraction branch, CSPDarknet53 is used to extract features from the visible light image step by step. The obtained multi-scale features are then aggregated with context information through spatial pyramid pooling, and then multi-scale information is integrated from top to bottom and from bottom to top to obtain multi-scale visible light features.

[0008] Furthermore, the dual-path SimAM-wavelet attention mechanism includes: determining the energy of infrared and visible light features at corresponding scales respectively; performing single-mode attention enhancement on the infrared and visible light features using the two energies respectively to obtain the corresponding attention features; calculating the difference between the two energies and dividing the difference into wavelet complementary regions and attention enhancement regions according to a set difference threshold to form a mask; performing wavelet transform on the infrared and visible light features respectively to obtain the low-frequency subbands and high-frequency subbands of the two modal features; after swapping the high-frequency subbands of the two features, performing inverse wavelet transform on the frequency subbands of the swapped two modal features respectively to obtain the complementary features of the two modal features; and, based on the mask, adding the two attention features and the two complementary features according to modality weights to obtain the complementary terms of the two modal features.

[0009] Furthermore, the energy of the two modal characteristics is calculated using the following formula: ; Where e represents energy, f represents infrared or visible light features, μ represents the element mean calculated for the features per channel, and var represents variance; The attention weights for the two modalities are obtained using the following formula: a = σ(e + 0.5); Where a represents the attention weight, and σ represents the Sigmoid function; The two modality features are multiplied by their respective attention weights to obtain the corresponding attention features.

[0010] Furthermore, the difference between the two energies is calculated using the following formula: b=σ|e r -e v |; Where b represents the difference, σ represents the Sigmoid function, and e r and e v These represent infrared and visible light characteristics, respectively. Differences greater than the difference threshold are classified as wavelet complement regions, and differences less than or equal to the difference threshold are classified as attention enhancement regions; elements in the mask corresponding to wavelet complement regions are defined as 1, and elements in the mask corresponding to attention enhancement regions are defined as 0. The complementary terms for the two modal features are obtained using the following formula: ; ; Where, d v and d r These represent the complementary terms corresponding to visible light features and infrared features, respectively, and B represents the mask. and These represent the visible light and infrared characteristics after the exchange, respectively. and These represent the attention features corresponding to visible light features and infrared features, respectively. This indicates element-wise multiplication.

[0011] Furthermore, in the binary fusion Mamba module: infrared and visible light features are added to their respective complementary terms, and then depthwise convolutions are performed on the two added features to obtain infrared and visible light mixed features; the infrared and visible light mixed features are input into the spatial scanning module to perform spatial scanning, extract the spatial dependency between the two features, obtain the state space sequence, and obtain the associated features based on the state space sequence; the associated features are fused with the two added features respectively, and the two obtained features are then element-wise added to the associated features and the two added features to obtain the fused features.

[0012] Furthermore, in the spatial scanning module: the infrared mixed features and the visible light mixed features are fused by feature splicing and convolution; the infrared mixed features, the visible light mixed features, and the features after their fusion are unfolded into multiple directional sequences through two-dimensional cross-scanning; For each direction sequence, the state is updated using the following formula: ; Among them, h k and h k-1 Let A represent the hidden states of the k-th and (k-1)-th sequence indices, respectively, and let B represent the learnable parameters. x Obtained by processing visible light mixing features through a linear layer, parameter B y Obtained by processing infrared mixed features through a linear layer, parameter B xy The feature is obtained by fusing infrared and visible light mixed features and then processing it with a linear layer. The sampling interval during cross-scanning is obtained by the following formula: ; Where, α x α y and α xy Indicates the gating coefficient. , and The base timescale is represented by linear mappings to the infrared mixed features, visible light mixed features, and the features resulting from their fusion. The output sequence for each direction sequence is obtained using the following formula: z k =Ch k +Dxk +Ey k ; Among them, z k The element at the k-th sequence index in the output sequence is represented by the parameter C, which represents the output obtained by linearly mapping the added features after adding the elements of the infrared mixed features, the visible light mixed features, and the features after their fusion by the two. D and E represent the direct transfer matrices of the infrared features and the visible light features, respectively. The output sequence is backfilled and fused using a reverse scanning method to obtain the associated features.

[0013] Furthermore, in the target detection branch: the detection boxes from the infrared detection results, visible light detection results, and fused detection results are projected onto the same image coordinate system; using any one detection result as a reference, the IoU between the current reference detection box and the detection boxes in the other two detection results that have overlapping parts is calculated. If the IoU is greater than a set threshold, it is determined that the three detection boxes correspond to the same physical target, and then the three detection results corresponding to the same physical target are aggregated into an association group; in each association group, according to the gating coefficient α... x α y and α xy We construct visible light basic prior weights, infrared basic prior weights, and fusion basic prior weights. We then perform weighted fusion of the three detection results based on the three basic prior weights to obtain the final detection result.

[0014] Furthermore, in the process of determining the same physical target: if there are multiple overlapping detection boxes between the current reference detection box and the other two detection results, resulting in multiple identical physical targets, the physical target corresponding to the largest IoU is selected.

[0015] Furthermore, the process of obtaining the final detection result includes: gating the gating coefficients α at all scales. x Summing these values ​​yields the fundamental prior weights for visible light, and the gating coefficients α for all scales are then calculated. ySumming is performed to obtain the infrared basic prior weights. The three gating coefficients at all scales are summed to obtain the fusion basic prior weights. The confidence levels of the three detection results are weighted using these three basic prior weights to obtain the corresponding weighted confidence levels. The three weighted confidence levels are summed to obtain the fusion confidence level. If the fusion confidence level is within a preset range, visible light detection results are retained in strong light scenarios, and visible light detection results are retained in weak light scenarios. Fusion detection results from any scenario are also retained. If the target categories in the three detection results are consistent, then this category is the target category in the final detection result. If the target categories in the three detection results are inconsistent, then the category with the highest weighted confidence level is directly selected as the target category in the final detection result. The detection box coordinates of the three detection results are weighted and fused using the three basic prior weights to obtain the detection box coordinates in the final detection result.

[0016] Furthermore, the process of determining the weighted confidence and fusion confidence also includes: if the two input images are an infrared image and a visible light image in a strong light scene, then an additional weighting term is added to the basic prior weight of the visible light image; if the two input images are an infrared image and a visible light image in a weak light scene, then an additional weighting term is added to the basic prior weight of the infrared image; then the three basic prior weights with the additional weighting term are used to weight the confidence of the three detection results respectively to obtain the corresponding weighted confidence; the three weighted confidences with the additional weighting term are summed to obtain the fusion confidence.

[0017] A target detection method based on visible light-infrared remote sensing image fusion includes: S1: Obtain the visible light-infrared remote sensing image dataset, preprocess the dataset to obtain the training set; S2: Construct a target detection system based on visible light-infrared remote sensing image fusion as provided in this invention; S3: Use the training set obtained in step S1 to train the target detection system constructed in step S2 to obtain the target detection model; S4: Input the visible light-infrared remote sensing image to be detected into the target detection model obtained in step S3 to obtain the predicted target recognition result.

[0018] Compared with the prior art, the present invention can achieve the following beneficial effects: This invention creates a target detection system and method based on visible light-infrared remote sensing image fusion. It employs a dual-path SimAM-wavelet attention mechanism to evaluate cross-modal differences at the pixel level, performing wavelet frequency domain complementation only in high-complementarity regions and robust single-modal enhancement in low-complementarity regions, thus suppressing noise amplification and pseudo-texture migration. By explicitly introducing visible light-driven, infrared-driven, and interaction-driven mechanisms at the state-space equation level through a binary fusion Mamba module, it achieves cross-modal collaborative evolution and linear complexity global dependency modeling, balancing accuracy and efficiency. Finally, it performs IoU association and branch / illuminance prior weighted fusion on the three detection results, significantly reducing duplicate and drifting bounding boxes and improving stability in complex scenes such as nighttime and occlusion. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the overall target detection system based on visible light-infrared remote sensing image fusion as described in an embodiment of the present invention; Figure 2 A schematic diagram of the feature processing of the dual-path SimAM-wavelet attention mechanism described in the embodiments of the present invention; Figure 3 A schematic diagram illustrating the feature processing of the binary fusion Mamba module described in an embodiment of the present invention; Figure 4 A schematic diagram illustrating the feature processing of the spatial scanning module described in an embodiment of the present invention; Figure 5 A schematic diagram illustrating the feature calculation of the spatial scanning module described in an embodiment of the present invention; Figure 6 A schematic diagram of the feature processing of the target detection branch described in the embodiments of the present invention; Figure 7 A schematic flowchart of the target detection method based on visible light-infrared remote sensing image fusion as described in the embodiments of the present invention; Figure 8 Comparison of detection results using the method provided in this invention and the UA-CMDet method in nighttime / occlusion scenarios; Figure 9 The ablation experiment results of the binary fusion Mamba module described in the embodiment of the present invention are shown in the figure. Figure 10 Ablation experiment results of the dual-path SimAM-wavelet attention mechanism described in the embodiments of the present invention; Figure 10(a) Mask images under different difference thresholds as described in the embodiments of the present invention; Figure 10 (b) mAP curves under different difference thresholds as described in the embodiments of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0021] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] like Figure 1 As shown in the embodiment of the present invention, the target detection system based on visible light-infrared remote sensing image fusion includes an infrared feature extraction branch, a visible light feature extraction branch, a feature fusion branch, and a target detection branch.

[0023] The infrared feature extraction branch extracts multi-scale infrared features from the input infrared image; the visible light feature extraction branch extracts multi-scale visible light features from the input visible light image.

[0024] In some embodiments, the feature processing of the infrared feature extraction branch includes: using multiple cascaded VSS modules (Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., & Wang, X. (2024). VisionMamba: Efficient Visual Representation Learning with Bidirectional StateSpace Model. ArXiv, abs / 2401.09417.) to extract features from the infrared image step by step. The obtained multi-scale features are then aggregated with contextual information through spatial pyramid pooling, and then multi-scale information is integrated from top to bottom and from bottom to top to obtain multi-scale infrared features. The feature processing in the visible light feature extraction branch includes: using CSPDarknet53 (Bochkovskiy, A., Wang, C., & Liao, HM (2020). YOLOv4: Optimal Speed ​​and Accuracy of Object Detection. ArXiv, abs / 2004.10934.) to extract features from the visible light image step by step. Specifically, the CBM module in CSPDarknet53 is used for step-by-step feature extraction. The obtained multi-scale features are then aggregated with contextual information through spatial pyramid pooling, and then multi-scale information is integrated from top to bottom and bottom to top to obtain multi-scale visible light features.

[0025] The feature fusion branch utilizes a dual-path SimAM-wavelet attention mechanism to perform difference sensing and frequency domain interaction on infrared and visible light features of the same scale, obtaining complementary terms for the infrared and visible light features. Then, a binary fusion Mamba module is used to obtain a state space sequence from the infrared features, visible light features, and the two complementary terms, and spatial direction fusion is performed based on the state space sequence to obtain fused features at the corresponding scale. In this embodiment of the invention, the obtained multi-scale fused features in the feature fusion branch are aggregated with contextual information through spatial pyramid pooling, and then multi-scale information integration is performed from top to bottom and from bottom to top.

[0026] In some embodiments, the feature processing procedure of the dual-path SimAM-wavelet attention mechanism is as follows: Figure 2 As shown, it includes: The energies of the infrared and visible light features at the corresponding scales were determined respectively; Single-mode attention enhancement was performed on infrared and visible light features using two different energies to obtain the corresponding attention features. The difference between the two energies is calculated, and the difference is divided into wavelet complementary region and attention enhancement region according to the set difference threshold to form a mask; Wavelet transforms are performed on infrared and visible light features respectively to obtain low-frequency and high-frequency subbands of each mode feature. After swapping the high-frequency subbands of the two features, inverse wavelet transforms are performed on the frequency subbands of the swapped two mode features to obtain complementary features of each mode feature. Based on the mask, the two attention features and the two complementary features are added together according to modality to obtain the complementary terms of the two modality features.

[0027] In this embodiment of the invention: the difference threshold is set according to the applicability of the actual situation, and the test shows that the effect is best when the threshold is 0.85; wavelet transform is performed on the infrared feature and the visible light feature respectively to obtain the low frequency subband LL, horizontal high frequency subband LH, vertical high frequency subband HL, and diagonal high frequency subband HH of the two modal features respectively; In response to the complementarity of infrared and visible light, this embodiment of the invention uses the respective low-frequency subbands LL as the main structural reference, and focuses on cross-modal interaction of the horizontal high-frequency subbands LH and vertical high-frequency subbands HL of the two modes to improve the contour and contrast. After subband-level complementarity and modulation, inverse wavelet transform is performed on the two modes respectively to obtain the complementary features of the two modes.

[0028] In some embodiments, the energy of the two modal characteristics is calculated using the following formula to characterize the response intensity of the channel at the current spatial location: ; Where e represents energy, f represents infrared or visible light features, μ represents the elemental mean calculated for the features per channel, and var represents variance.

[0029] The attention weights for the two modalities are obtained using energy through the following formula: a = σ(e + 0.5); Where a represents the attention weight and σ represents the Sigmoid function.

[0030] Multiplying the two modal features by their respective attention weights yields the corresponding pixel-level attention features: ; ; in, and These represent the attention features corresponding to visible light features and infrared features, respectively. f represents element-wise multiplication. v and f rThese represent visible light and infrared features, respectively. This invention utilizes energy terms to construct pixel-level attention, highlighting the dominant information regions of each modality while suppressing unreliable regions.

[0031] In some embodiments, to measure the statistical difference between the two modes at each spatial location, the present invention constructs a cross-modal uncertainty gating based on the absolute value of the energy term difference, i.e., the difference between the two energies is calculated using the following formula: b=σ|e r -e v |; Where b represents the difference, σ represents the Sigmoid function, and e r and e v Let e ​​represent the infrared feature and the visible light feature, respectively. The energy e is essentially the standardized deviation intensity of the feature on the statistical scale of the channel. Then the difference b measures the relative intensity difference between the two modes at the same location from their respective means.

[0032] This invention controls the proportion of frequency domain complementation by setting a difference threshold. Differences greater than the difference threshold are suitable for frequency domain complementation. Therefore, differences greater than the difference threshold τ are divided into wavelet complementation regions, and differences less than or equal to the difference threshold τ are divided into attention enhancement regions. The elements in the mask corresponding to the wavelet complementary region are defined as 1, and the elements in the mask corresponding to the attention enhancement region are defined as 0.

[0033] The complementary terms for the two modal features are obtained using the following formula: ; ; Where, d v and d r These represent the complementary terms corresponding to visible light features and infrared features, respectively, and B represents the mask. and These represent the visible light and infrared characteristics after the exchange, respectively.

[0034] This invention utilizes a difference threshold to obtain a mask with wavelet complementarity and attention enhancement regions, enabling explicit quantization of cross-modal differences at the pixel level. It releases cross-modal gain in high complementarity regions and maintains robust single-modal representation in low complementarity regions, thereby avoiding noise amplification and pseudo-texture migration caused by full-image uniform fusion. The frequency complementarity provided by this invention can also effectively reduce the fusion burden and improve its modeling efficiency and upper limit for cross-modal relationships.

[0035] In some embodiments, the feature processing procedure of the binary fusion Mamba module is as follows: Figure 3 As shown, it includes: After adding the infrared and visible light features to their respective complementary terms, depth convolution is performed on the two added features to obtain the infrared mixed features and the visible light mixed features. Infrared and visible light mixed features are input into the spatial scanning module to perform spatial scanning on the infrared and visible light mixed features, extract the spatial dependency between the two features, obtain the state space sequence, and obtain the associated features based on the state space sequence. After fusing the associated features with the two summed features respectively, the resulting two features are then element-wise added to the associated features and the two summed features to obtain the fused features.

[0036] In this embodiment of the invention, after adding the infrared features and visible light features to their respective complementary terms, the two added features are sequentially normalized and linearly mapped, and then sequentially subjected to depthwise convolution and SiLU activation to obtain the infrared mixed features and the visible light mixed features, as shown in the following formula: ; ; Among them, g v and g r These represent visible light mixed features and infrared mixed features, respectively. Norm represents layer normalization, DWC represents depthwise convolution, and Linear represents linear mapping. Infrared and visible light mixed features are input into the spatial scanning module to obtain associated features; after layer normalization, the associated features are then fused with the two summed features using the following formula: ; ; Among them, G f G represents the associated features. v and G r This represents the visible light fusion characteristics and infrared fusion characteristics after the fusion operation; The obtained visible light fusion feature G v Infrared fusion feature G r With associated feature G f Add the elements of the two combined features to the following formula to obtain the fused feature: f m =Linear(G f +G v +G r )+(f v +d v )+(f r +d r ); Among them, f mThis indicates the fusion feature.

[0037] In some embodiments, the feature processing procedure of the spatial scanning module is as follows: Figure 4 and Figure 5 As shown, it includes: The infrared mixed features and visible light mixed features are fused by feature concatenation and convolution. In this embodiment of the invention, the fusion operation is performed as follows: g f =Conv(Concat(g v ,g r )); Among them, g f The two values ​​represent the features after fusion. Conv represents a 3×3 convolution operation, and Concat represents a channel concatenation operation.

[0038] Infrared mixed features, visible light mixed features, and the features resulting from their fusion are expanded into multiple directions using a two-dimensional cross-scan, including at least horizontal, vertical, and two diagonal directions, to reduce direction sensitivity and cover two-dimensional long-range dependencies. Then, for each direction, the hidden state is updated using the following dual-input state-space equation: ; ; Where h(t) represents the hidden state, x(t) and y(t) represent the infrared and visible light features, and f(x(t),y(t)) represents the fused features. Let z(t) represent the updated hidden state, z(t) represent the output feature of the spatial scanning module, D and E represent the direct transfer matrices of the infrared and visible light features, respectively, and A represent the learnable parameters. In practice, the learnable parameters are in logarithmic form, so A is obtained by taking the negative exponent of the learnable parameters. Parameter B... x It is obtained by processing visible light mixing features through a linear layer, and in this embodiment of the invention, it is obtained by the following formula: B x =Linear(g v ); Parameter B y The infrared mixing features are obtained through linear layer processing, and in this embodiment of the invention, they are obtained by the following formula: B y =Linear(g r ); Parameter B xy The feature obtained by fusing infrared and visible light mixed features and then processing with a linear layer is obtained in this embodiment of the invention by the following formula: B xy =Linear(g f ); Parameter C represents the output obtained by linearly mapping the added features after element-wise addition of infrared mixed features, visible light mixed features, and the features resulting from their fusion, as shown in the following equation: C=Linear(g r +g v +g f ).

[0039] Since the features after cross-scanning are a sequence, this invention uses zero-order hold (ZOH) to discretize the continuous two-input state-space equations, with a sampling interval of... We obtain the following formula: ; Where, α x α y and α xy The gating coefficients are obtained during the system's training and learning process. , and The basic time scale is represented by linear mappings of infrared mixed features, visible light mixed features, and the features resulting from their fusion. In this embodiment of the invention, it is obtained through the following formula: ; ; .

[0040] For each direction sequence, the state is updated using the following formula: ; Among them, h k and h k-1 Let x represent the hidden states of the k-th and (k-1)-th sequence indices, respectively. k y k and f k Let x(t), y(t), and f(x(t), y(t)) represent the k-th sequence index, respectively. The output sequence for each direction sequence is obtained using the following formula: z k =Ch k +Dx k +Ey k ; Among them, z k This represents the element at the k-th sequence index in the output sequence; The output sequence is backfilled and fused using a reverse scanning method to obtain the associated features.

[0041] In the feature fusion branch designed in this invention, the binary fusion Mamba module completes scale-wise and position-wise reliability reweighting through nonlinear interactive projection transformation on the input side; simultaneously, on the state space side, it performs direction-independent global modeling of cross-modal dependencies through a spatial scanning module. Therefore, the binary fusion Mamba module is no longer a post-processing module for pre-fused features in a single-input state space, but rather couples the dual-modal driving force and interaction terms to the state equation itself, allowing cross-modal relationships to be continuously modeled during the latent state evolution. Compared to the traditional simple fusion followed by single-input SSM scheme, the binary fusion Mamba module is structurally more consistent with multimodal physical and statistical characteristics, and can explicitly utilize the differences and complementary information between modes while maintaining linear complexity. Furthermore, the dual-path SimAM-wavelet attention mechanism and the binary fusion Mamba module form a complementary synergistic mechanism in both the spatial and frequency domains. The dual-path SimAM-wavelet attention mechanism explicitly extracts and reconstructs high-frequency details in the local spatial neighborhood through gated wavelet transform, highlighting discriminative information such as contours and edges in advance on the feature map. However, wavelet transform is essentially a local operator, and its influence is limited to a fixed neighborhood and scale. Building on this, the binary fusion Mamba module utilizes a dual-input state space model and two-dimensional selective scanning to embed these frequency-enhanced local details into the cross-location, cross-modal latent state evolution, allowing local high-frequency information to propagate long distances with the latent state and interact with the global context. In other words, the dual-path SimAM-wavelet attention mechanism provides clean and structured local details, while the binary fusion Mamba module compensates for the locality of wavelet transform through global context modeling, effectively amplifying and integrating complementary local information globally.

[0042] The target detection branch utilizes the multi-scale features output by the infrared feature extraction branch, visible light feature extraction branch, and feature fusion branch to obtain infrared detection results, visible light detection results, and fused detection results. These three detection results are then fused to output the final detection result. The detection result includes detection box parameters (including the center coordinates, width, and height of the detection box, and its rotation angle), target confidence, and target category probability. In this embodiment, the detection head disclosed in (Bochkovskiy, A., Wang, C., & Liao, HM (2020). YOLOv4: Optimal Speed ​​and Accuracy of Object Detection. ArXiv, abs / 2004.10934.) is specifically used.

[0043] In some embodiments, the feature processing of the target detection branch is as follows: Figure 6 As shown, it includes: Project the detection boxes from the infrared detection results, visible light detection results, and fused detection results onto the same image coordinate system; Using any one of the detection results as a reference, calculate the IoU between the current reference detection box and the detection boxes in the other two detection results that have overlapping parts. If the IoU is greater than the set threshold, it is determined that the three detection boxes correspond to the same physical target. Then, the three detection results corresponding to the same physical target are aggregated into an association group. In each association group, according to the gating coefficient α x α y and α xy We construct visible light basic prior weights, infrared basic prior weights, and fusion basic prior weights. We then perform weighted fusion of the three detection results based on the three basic prior weights to obtain the final detection result.

[0044] If there are multiple overlapping detection boxes between the current reference detection box and the other two detection results, resulting in multiple instances of the same physical target, the physical target corresponding to the largest IoU will be selected.

[0045] In some embodiments, the gating coefficients α for all scales are... x Summing these values ​​yields the fundamental prior weights for visible light, and the gating coefficients α for all scales are then calculated. y Summing these values ​​yields the infrared fundamental prior weights. Summing the three gating coefficients across all scales yields the fusion fundamental prior weights, as shown in the following equation: ; ; ; Where, α v α r and α f These represent the visible light basic prior weight, the infrared basic prior weight, and the fusion basic prior weight, respectively. s represents the scale index, and S represents the number of scales. In this embodiment of the invention, S=4. The three basic prior weights reflect the relative contributions of the three features to the overall representation at the state space level. The confidence scores of the three detection results are weighted using three basic prior weights to obtain the corresponding weighted confidence scores; the three weighted confidence scores are then summed to obtain the fusion confidence score. If the fusion confidence level is within the preset confidence level range, the visible light detection results are retained in strong light scenes and in weak light scenes, and the fusion detection results in any scene are retained. If the target category is consistent in the three detection results, then this category is the target category in the final detection result. If the target category is inconsistent in the three detection results, then the category with the highest weighted confidence is directly selected as the target category in the final detection result. Using three basic prior weights, the bounding box coordinates of the three detection results are weighted and fused to obtain the bounding box coordinates in the final detection result.

[0046] In this embodiment of the invention, the preset range is set to brightness = 127, that is, an image with an average brightness greater than 127 is considered daytime, and one with a brightness less than 127 is considered nighttime.

[0047] In some embodiments, the process of determining the weighted confidence level and the fusion confidence level further includes: If the two input images are an infrared image and a visible light image in a strong light scene, then an additional weighting term is added to the basic prior weight of the visible light image to highlight its advantages in information such as texture and color. If the two input images are an infrared image and a visible light image in a low-light scene, then an additional weighting term is added to the basic infrared prior weights to enhance its stability under low-light conditions. The confidence scores of the three detection results are then weighted by the three basic prior weights with the additional weighting terms to obtain the corresponding weighted confidence scores. The fusion confidence is obtained by summing the three weighted confidences with the additional weighting term.

[0048] In this embodiment of the invention, the three basic prior weights with introduced additional weighting terms are normalized. Then, the normalized three basic prior weights are used to weight the confidence levels of the three detection results to obtain corresponding weighted confidence levels. These three weighted confidence levels are then summed to obtain the fused confidence level. Furthermore, in this embodiment, unassociated detection boxes are designated as false alarms. If the confidence level of a false alarm exceeds a preset range, the detection box is retained as an independent detection target; low-confidence boxes that do not meet the above conditions are discarded. This design avoids simply discarding all unassociated boxes, thus preserving potential real targets while maintaining accuracy. Through the above association, weighting, and filtering processes, this invention finally outputs the fused detection result, further compensating for residual mismatches in complex scenarios based on feature-level fusion, significantly improving the overall stability and reliability of the detection.

[0049] This invention also provides a target detection method based on visible light-infrared remote sensing image fusion, combined with Figure 1 and Figure 7 ,include: S1: Obtain the visible light-infrared remote sensing image dataset, preprocess the dataset, and obtain the training set.

[0050] S2: Construct a target detection system based on visible light-infrared remote sensing image fusion as provided in this invention.

[0051] S3: Use the training set obtained in step S1 to train the object detection system constructed in step S2 to obtain the object detection model.

[0052] In some embodiments, the following loss is used for training: L total =L fu +L v +L r ; Among them, L total L represents the loss function for training. fu L vis and L ir These represent the loss functions for the fused detection results, visible light detection results, and infrared detection results, respectively.

[0053] The three loss functions are consistent, specifically: L=5.0L box +1.0L cls +1.0L obj ; Among them, L box L represents the loss during bounding box regression. cls L represents the classification loss. obj This indicates a targeted loss.

[0054] Furthermore, the hyperparameters of the training process in this embodiment of the invention include: Batch size = 4, Epoch = 500, learning rate = 1e-5, weight decay = 5e-4, and optimizer = AdamW.

[0055] S4: Input the visible light-infrared remote sensing image to be detected into the target detection model obtained in step S3 to obtain the predicted target recognition result.

[0056] To demonstrate the effectiveness of the method provided by this invention, the embodiments of this invention first performed detection tasks in scenarios such as nighttime / occlusion, and the task results are as follows: Figure 8 As shown, an ablation experiment was conducted on the binary fusion Mamba module provided by this invention, and the experimental results are as follows. Figure 9 As shown, an ablation experiment was conducted on the dual-path SimAM-wavelet attention mechanism provided by this invention, and the experimental results are as follows. Figure 10 As shown.

[0057] like Figure 8As shown, a comparative experiment was conducted between the embodiments of the present invention and the existing UA-CMDet method. Due to the influence of nighttime and occlusion scenarios, coupled with the small target size, the detection performance of a single modality is poor. Furthermore, during the fusion detection of two modalities, a large amount of redundant and misleading information is introduced, making it difficult to effectively predict the target's location and category, especially in scenes such as under trees or at night. The target detection method for infrared and visible light remote sensing image fusion provided by the present invention can still effectively recall these targets and accurately identify their category and location, fully demonstrating the fusion detection capability of the present invention for infrared and visible light images.

[0058] like Figure 9 As shown, Figure 9 The top left image shows data from the DroneVehicle dataset, and the three images on the right visualize the feature maps of different fusion schemes. Compared to the traditional method of concatenating channels followed by convolution (…),… Figure 9 The top right corner "Concat+Conv") and the single input spatial state mode after channel splicing ( Figure 9 The method of "Concat+Mamna" in the lower left corner is used to merge Mamba modules in binary form. Figure 9 (Lower right) It can accurately extract more effective features, and the features are more prominent than the background information, which fully demonstrates the capabilities of the BFMamba module in the field of multi-source feature fusion.

[0059] Figure 10 This is a diagram of the mask and its mAP (mAP) plot of the dual-path SimAM-wavelet attention mechanism under different difference thresholds according to an embodiment of the present invention. Figure 10 The left side shows the data from the DroneVehicle dataset. Figure 10 (a) shows the mask images under different difference thresholds. The white area is the wavelet complementation region, and the black area is the single-mode enhancement region. Figure 10 (b) shows the mAP curves at different difference thresholds. Figure 9It can be concluded that the mask visualization shows that the wavelet complement region mainly focuses on the target and its edges. This is because visible light mainly encodes texture and reflection structure, while infrared mainly encodes thermal radiation and thermal contrast. These two modes often exhibit complementary or significantly different response patterns in the target area, resulting in a large relative intensity difference between the two modes. Large background areas, on the other hand, are usually closer to their respective means in both modes, and their normalization deviations are more consistent. Furthermore, the performance of forcibly using wavelet complement across the entire image is even slightly lower than that of the attention-only method. This is because in flat regions where the information of the two modes is highly consistent, frequency domain decomposition and reconstruction may introduce additional numerical perturbations and pseudo-textures, disrupting the original structural priors. Simultaneously, the lack of a difference threshold will disperse the complement operation to locations with lower detection contributions. In contrast, the dual-path SimAM-wavelet attention mechanism activates wavelet complement only in highly complementary regions through a difference threshold, while retaining attention enhancement in other regions, thus effectively suppressing the negative impact of indiscriminate frequency domain operations. The above explanation fully demonstrates that the dual-path SimAM-wavelet attention mechanism can adaptively focus frequency domain complement on regions with high discriminative contribution, thereby suppressing non-selective enhancement of background noise.

[0060] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0061] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A target detection system based on visible light-infrared remote sensing image fusion, characterized in that, include: The infrared feature extraction branch extracts multi-scale infrared features from the input infrared image; The visible light feature extraction branch extracts multi-scale visible light features from the input visible light image; The feature fusion branch utilizes a dual-path SimAM-wavelet attention mechanism to perform difference perception and frequency domain interaction on infrared and visible light features of the same scale, obtaining complementary terms of infrared and visible light features. Then, the binary fusion Mamba module is used to obtain the state space sequence of infrared features, visible light features and the two complementary terms, and the spatial direction is fused according to the state space sequence to obtain the fused features of the corresponding scale. The target detection branch uses the detection head to obtain infrared detection results, visible light detection results, and fused detection results based on the multi-scale features output by the infrared feature extraction branch, visible light feature extraction branch, and feature fusion branch. The three detection results are then fused to output the final detection result.

2. The target detection system based on visible light-infrared remote sensing image fusion according to claim 1, characterized in that, In the infrared feature extraction branch, multiple cascaded VSS modules are used to extract features from the infrared image step by step. The resulting multi-scale features are then aggregated with context information through spatial pyramid pooling, and then multi-scale information is integrated from top to bottom and from bottom to top to obtain multi-scale infrared features.

3. The target detection system based on visible light-infrared remote sensing image fusion according to claim 1, characterized in that, In the visible light feature extraction branch, CSPDarknet53 is used to extract features from the visible light image step by step. The obtained multi-scale features are then aggregated with context information through spatial pyramid pooling, and then multi-scale information is integrated from top to bottom and bottom to top to obtain multi-scale visible light features.

4. The target detection system based on visible light-infrared remote sensing image fusion according to claim 1, characterized in that, The dual-path SimAM-wavelet attention mechanism includes: The energy of the infrared and visible light features at the corresponding scales is calculated using the following formula: ； Where e represents energy, f represents infrared or visible light features, μ represents the element mean calculated for the features per channel, and var represents variance; The attention weights for the two modalities are obtained using the following formula: a = σ(e + 0.5); Where a represents the attention weight, and σ represents the Sigmoid function; The two modal features are multiplied by their respective attention weights to obtain the corresponding attention features; The difference between the two energies is calculated, and the difference is divided into wavelet complementary region and attention enhancement region according to the set difference threshold to form a mask; Wavelet transforms are performed on infrared and visible light features respectively to obtain low-frequency and high-frequency subbands of each mode feature. After swapping the high-frequency subbands of the two features, inverse wavelet transforms are performed on the frequency subbands of the swapped two mode features to obtain complementary features of each mode feature. The difference between the two energies is calculated using the following formula: b=σ|e r -e v |; Where b represents the difference, σ represents the Sigmoid function, and e r and e v These represent infrared and visible light characteristics, respectively. Differences greater than the difference threshold are classified as wavelet complement regions, and differences less than or equal to the difference threshold are classified as attention enhancement regions. The elements in the mask corresponding to the wavelet complementary region are defined as 1, and the elements in the mask corresponding to the attention enhancement region are defined as 0. The complementary terms for the two modal features are obtained using the following formula: ； ； Where, d v and d r These represent the complementary terms corresponding to visible light features and infrared features, respectively, and B represents the mask. and These represent the visible light and infrared characteristics after the exchange, respectively. and These represent the attention features corresponding to visible light features and infrared features, respectively. This indicates element-wise multiplication.

5. The target detection system based on visible light-infrared remote sensing image fusion according to claim 1, characterized in that, In the binary fusion Mamba module: After adding the infrared and visible light features to their respective complementary terms, depth convolution is performed on the two added features to obtain the infrared mixed features and the visible light mixed features. Infrared and visible light mixed features are input into the spatial scanning module to perform spatial scanning on the infrared and visible light mixed features, extract the spatial dependency between the two features, obtain the state space sequence, and obtain the associated features based on the state space sequence. After fusing the associated features with the two summed features respectively, the resulting two features are then element-wise added to the associated features and the two summed features to obtain the fused features; In the spatial scanning module: The infrared mixed features and the visible light mixed features are fused by feature splicing and convolution. Infrared mixed features, visible light mixed features, and the features resulting from their fusion are expanded into multiple directional sequences through two-dimensional cross-scanning; For each direction sequence, the state is updated using the following formula: ； Among them, h k and h k-1 Let A represent the hidden states of the k-th and (k-1)-th sequence indices, respectively, and let B represent the learnable parameters. x Obtained by processing visible light mixing features through a linear layer, parameter B y Obtained by processing infrared mixed features through a linear layer, parameter B xy The feature is obtained by fusing infrared and visible light mixed features and then processing it with a linear layer. The sampling interval during cross-scanning is obtained by the following formula: ； Where, α x α y and α xy Indicates the gating coefficient. , and The basic time scale is represented by linear mappings to the infrared mixed features, the visible light mixed features, and the features resulting from their fusion. The output sequence for each direction sequence is obtained using the following formula: z k =Ch k +Dx k +Ey k ; Among them, z k The element at the k-th sequence index in the output sequence is represented by the parameter C, which represents the output obtained by linearly mapping the added features after adding the elements of the infrared mixed features, the visible light mixed features, and the features after their fusion by the two. D and E represent the direct transfer matrices of the infrared features and the visible light features, respectively. The output sequence is backfilled and fused using a reverse scanning method to obtain the associated features.

6. The target detection system based on visible light-infrared remote sensing image fusion according to claim 5, characterized in that, In the object detection branch: Project the detection boxes from the infrared detection results, visible light detection results, and fused detection results onto the same image coordinate system; Using any one of the detection results as a reference, calculate the IoU between the current reference detection box and the detection boxes in the other two detection results that have overlapping parts. If the IoU is greater than the set threshold, it is determined that the three detection boxes correspond to the same physical target. Then, the three detection results corresponding to the same physical target are aggregated into an association group. In each association group, according to the gating coefficient α x α y and α xy We construct visible light basic prior weights, infrared basic prior weights, and fusion basic prior weights. We then perform weighted fusion of the three detection results based on the three basic prior weights to obtain the final detection result.

7. The target detection system based on visible light-infrared remote sensing image fusion according to claim 6, characterized in that, The process of obtaining the final test results includes: Gating coefficients α for all scales x Summing these values ​​yields the fundamental prior weights for visible light, and the gating coefficients α for all scales are then calculated. y Summing yields the infrared basic prior weights; summing the three gating coefficients across all scales yields the fusion basic prior weights. The confidence scores of the three detection results are weighted using three basic prior weights to obtain the corresponding weighted confidence scores; the three weighted confidence scores are then summed to obtain the fusion confidence score. If the fusion confidence level is within the preset confidence level range, the visible light detection results are retained in strong light scenes and in weak light scenes, and the fusion detection results in any scene are retained. If the target category is consistent in the three detection results, then this category is the target category in the final detection result. If the target category is inconsistent in the three detection results, then the category with the highest weighted confidence is directly selected as the target category in the final detection result. Using three basic prior weights, the bounding box coordinates of the three detection results are weighted and fused to obtain the bounding box coordinates in the final detection result.

8. The target detection system based on visible light-infrared remote sensing image fusion according to claim 7, characterized in that, The process of determining the weighted confidence level and the fusion confidence level also includes: If the two input images are an infrared image and a visible light image in a strong light scene, then an additional weighting term is added to the basic prior weight of the visible light image. If the two input images are an infrared image and a visible light image in a low-light scene, then an additional weighting term is added to the basic infrared prior weights. The confidence scores of the three detection results are then weighted by the three basic prior weights with the additional weighting terms to obtain the corresponding weighted confidence scores. The fusion confidence is obtained by summing the three weighted confidences with the additional weighting term.

9. A target detection method based on visible light-infrared remote sensing image fusion, characterized in that, include: S1: Obtain the visible light-infrared remote sensing image dataset, preprocess the dataset to obtain the training set; S2: Construct a target detection system based on visible light-infrared remote sensing image fusion as described in any one of claims 1 to 8; S3: Use the training set obtained in step S1 to train the target detection system constructed in step S2 to obtain the target detection model; S4: Input the visible light-infrared remote sensing image to be detected into the target detection model obtained in step S3 to obtain the predicted target recognition result.

Images