Multispectral detection method

Abstract

The invention provides a multispectral detection method. The method comprises the following steps: firstly, acquiring visible light-infrared image pairs under severe weather conditions (dark light, rainy days, snowy days, foggy days and the like); secondly, inputting the visible light-infrared image pair into a double-flow feature extraction network to respectively extract multi-scale key features of each modal; thirdly, constructing a lightweight feature fusion network based on a hybrid expert system, evaluating the contribution degree of each modal feature to a detection effect, and extracting complementary features; and finally, designing a complementary feature multi-scale feature aggregation network and a target classification positioning detection head, and obtaining target position and category information. The method can be applied to target detection under extreme weather conditions.

Description

Technical Field

[0001] This invention relates to the field of multispectral image fusion and target detection technology, specifically to a multispectral detection method. Background Technology

[0002] In recent years, target detection technology based on deep neural networks has made significant progress. Under normal weather conditions, when using visible light images as input, existing methods can accurately classify and locate targets in the images. In well-lit, unobstructed environments, visible light images can clearly present key visual features of targets, such as color, texture, and contours, providing sufficient and discriminative information for neural networks to support high-precision detection tasks. However, under adverse lighting or weather conditions such as low light, rain, fog, and snow, the reliability of target detection relying solely on visible light images decreases significantly. These extreme environments lead to severe image quality degradation, specifically manifested as color distortion in the target area, loss of texture details, and reduced overall contrast. Furthermore, natural phenomena such as raindrops, fog, or snow introduce occlusion and noise during the imaging process, further weakening the discernibility of targets in the image. In such cases, neural networks struggle to extract effective discriminative features from degraded visible light images, resulting in classification errors and localization biases, severely impacting the practical application performance of the detection system.

[0003] To address the severe information loss in single-modal visible light images under extreme conditions such as low light, rain, fog, and snow, existing solutions introduce infrared images as a complementary input modality. Visible light and infrared images each have their advantages in target representation: visible light images provide rich color, texture, and detail information under normal lighting conditions, aiding in precise object differentiation and recognition; while infrared images, based on the thermal radiation characteristics of object surfaces, are less affected by changes in ambient light and interference factors such as rain, fog, and snow, and can stably present the contours and structural features of targets even under harsh conditions. By effectively fusing the image information from these two modalities, complementary advantages at the feature level can be achieved. Specifically, the fused features retain the rich appearance details of the visible light image while incorporating robust structural information from the infrared image, thus providing sufficient and discriminative visual cues for the neural network even in extreme environments. This multimodal fusion method significantly improves the robustness and recognition accuracy of target detection systems in complex scenes, effectively overcoming the performance bottleneck of single visible light modalities when information is lacking.

[0004] Multispectral target detection technology offers a new technical approach to solving the challenges of visual perception under adverse weather conditions. However, effectively extracting complementary features from multispectral image input and achieving efficient feature fusion remains a challenge. Images from different spectral modalities exhibit significant differences in feature distribution, semantic level, and sensitivity to noise. Simple overlay or shallow fusion is insufficient to fully exploit their complementary potential and may even lead to feature conflicts, affecting the overall model performance. Therefore, a fusion mechanism that adaptively balances the contributions of each modality and achieves deep feature complementarity and enhancement is needed to improve the robustness and accuracy of the detection system in complex environments.

[0005] Existing methods have evolved from early strategies based on simple overlay or shallow fusion to incorporating attention mechanisms for adaptive feature weighting, and further to employing the Transformer architecture, leveraging its cross-modal self-attention mechanism to achieve efficient fusion of global contextual semantics. Deevi et al. proposed a scene-aware modular feature fusion network that dynamically adjusts key features using spatiotemporal attention to predict weather conditions, achieving efficient fusion of multimodal features. Zhang et al. proposed a dual-stream SSD architecture that uses a cross-modal interactive attention network to encode the correlation between two modalities and adaptively recalibrates the channel responses of multi-scale intermediate feature maps to capture contextual enhancement features for visible-infrared target detection. Furthermore, Zhang et al. proposed AR-CNN based on Faster R-CNN, which designs a region feature alignment module to capture positional offsets in multispectral images and adaptively align region features from both modalities, selecting more reliable features and suppressing useless features through feature reweighting. With the advent of Transformers, Yuan et al. proposed a complementary fusion Transformer module that addresses the issues of weak misalignment and inaccurate fusion in multispectral images to achieve more advanced detection results. Furthermore, C2Former is a novel Transformer module that can be integrated into existing pre-trained models. This module acquires calibration and complementary features by learning the cross-attention relationship between visible and infrared modes. To reduce the computational cost of global attention, this method also introduces adaptive feature sampling to reduce the dimensionality of the feature map. Although such strategies can alleviate computational pressure to some extent, the inherent computational complexity of the self-attention mechanism in the Transformer architecture still results in significant computational overhead and resource consumption when processing multispectral high-dimensional features. This, to some extent, limits the practical application potential of this technology in scenarios with high real-time requirements or limited resources. Summary of the Invention

[0006] To address the challenge of balancing high accuracy and low computational cost in existing multispectral detection methods, this invention proposes a hybrid expert architecture that combines dynamic routing and gated collaborative optimization. A multi-scale feature-aware gated network is designed, employing lightweight convolutional modules and channel attention mechanisms to dynamically evaluate the modal characteristics and regional importance of the input multispectral data, thereby assigning appropriate computational weights to each expert network. A sparse activation and conditional computation mechanism is introduced, activating only some key expert modules during inference, significantly reducing overall computational overhead while preserving complementary multimodal information. This architecture effectively balances computational efficiency and model performance while maintaining high detection accuracy.

[0007] This invention provides a multispectral detection method, which includes three stages: dual-stream feature extraction, feature fusion, and target detection. A lightweight convolutional neural network extracts visible light and infrared light features respectively. A hybrid expert system uses a hybrid expert architecture to fuse complementary multispectral features. Multi-scale feature aggregation and convolutional networks predict the target location and category. Specifically, it includes the following steps:

[0008] S1. Collect visible light and infrared image pairs of the target to be detected under different weather conditions. Since fusion detection requires aligned visible light and infrared images, the XoFTR algorithm is used to align the image pairs, and the target category and location are labeled frame by frame to obtain a multispectral target detection dataset. The dataset is then divided into training and test sets in an 8:2 ratio.

[0009] S2. Batch input of visible light and infrared images from the training set into the CSPDarknet network pre-trained on the COCO dataset for feature extraction. The visible light image feature extraction branch and the infrared image feature extraction branch are independent of each other, ultimately obtaining multi-scale feature maps: in, Visible light images and infrared images are processed by their respective CSPDarknet backbone networks ψ R ,ψ T The resulting feature map of layer i ∈ {1,2,3,4,5}, θ R ,θ T It is the visible light and infrared image CSPDarknet backbone network ψ R ,ψ T The respective weights, which are obtained by pre-training on the COCO dataset, where W, H, and C refer to the width, height, and number of channels of the feature map, respectively, and I... R ,I T It is an input pair of visible light and infrared images.

[0010] S3, Given multi-scale features extracted from visible and infrared images. Cross-modal fusion is performed using a lightweight hybrid expert system (MoE), which consists of n expert networks and one gating network. Each expert network has the same structure and is built upon NIN (Network-in-Network) attention modules. The hybrid expert system fuses features processed by different expert networks at different scales. in, These are the fused features of the i-th layer ∈ {3,4,5}, with sizes H / 8×W / 8×256, H / 16×W / 16×512, and H / 32×W / 32×1024, respectively, and ψ f It is a feature fusion expert system, and its weights are θ. f The weights are obtained by training the entire detection framework on a real dataset. The feature fusion process can be represented by the following formula: Where j is the index number of the expert network, and k is the topk output of the gated network Gate. The hybrid expert system uses Top1 to select only the expert network with the highest probability output. The expert network uses the NIN attention module, which passes each 1×1 convolutional layer (with C / 2 output channels, where C is the number of input channels) to the original... Channel compression is performed, followed by weight allocation of the modulated features using a shared parameter weight allocator. This module consists of convolutions, ReLU activation functions, and more convolutions. The number of channels is halved after the first convolution and reduced to one dimension after the second convolution. Weights are generated for the modulated visible light and infrared features respectively, and finally, the generated weights are used to perform a weighted sum of the features for each modality. The gate network (Gate) uses four convolutional layers with kernel sizes of 3×3, 3×3, 1×1, and 1×1, plus global average pooling and a softmax layer, outputting a vector of length k. Each value in the vector represents the probability of each expert network being activated. Before the overall network training begins, all weights are randomly assigned; that is, the initial probability of each expert network being activated from the gate network (Gate) is random. As multiple rounds of forward calculation of the object detection loss and gradient backpropagation occur, the probability of each expert network being activated from the gate network (Gate) is gradually updated with the weights, minimizing the overall loss. The smaller the detection loss, the higher the classification and localization accuracy of the overall network. g It is a gated network, which uses... For input, θ g These are the weights of the gating network, which outputs the activation probability vectors of each expert network. It is the j-th expert network selected and activated by the gating network, and this network is... For input, For each expert network weight, output size and Same feature map.

[0011] S4. Obtain the recalibrated features. Then, the multi-scale feature aggregation detection head is input for classification and localization: Where D cls D loc These are the category label and the bounding box label for the detected target, respectively. ψ head This refers to the detection head, whose parameter is θ. h .

[0012] The proposed multispectral detection method consists of a hybrid expert system comprising a dual-stream feature extraction network (CSPDarknet), n NIN expert networks for fusing visible and infrared features, and a gate network (Gate) that assigns weights to each NIN expert network, along with a feature aggregation and object detection head. The weights of these components are randomly initialized, and the weights of each part of the model are updated as the gradient backpropagates. The final optimization objective is to minimize the loss function. The input visible and infrared image pairs are 768×768 pixels in size. Gradient updates are performed using the SGD optimizer with an initial learning rate of 0.01 and 60 iterations.

[0013] The advantage of this method is that the fusion module is lightweight and the detection accuracy is also taken into account. It can accurately detect targets with a small number of model parameters. The detection accuracy is mAP50≥75% on the FLIR dataset, mAP50≥75% on the Kaist dataset, and mAP50≥80% on the M3FD dataset. Attached Figure Description

[0014] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0015] Figure 1 This is a diagram of the overall framework for multispectral fusion detection based on a hybrid expert system according to the present invention.

[0016] Figure 2 This is a diagram showing the implementation effect of the present invention on the M3FD dataset;

[0017] Figure 3 This is a diagram showing the implementation effect of the present invention on the FLIR public dataset;

[0018] Figure 4 This is a diagram showing the implementation effect of the present invention on the Kaist public dataset; Detailed Implementation

[0019] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific examples. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0020] An overall framework diagram of a multispectral detection method is shown below. Figure 1 As shown, the entire process consists of four steps: data acquisition and annotation, dual-stream feature extraction, visible light and infrared feature fusion, and object classification and localization, as detailed below:

[0021] Step S1: First, based on the actual scene requirements, use visible light and infrared cameras to collect image pairs under different weather and climate conditions (low light, rain, snow, fog, etc.), and annotate the targets in the images one by one according to YOLO format. The proposed method is validated on three public datasets with aligned visible light and infrared data pairs: M3FD, FLIR, and Kaist. The M3FD dataset contains 4200 annotated image pairs, including different scenes such as low light, exposure, and rain, and the annotated data covers six categories: pedestrians, cars, buses, motorcycles, trucks, and streetlights. The FLIR dataset contains 5114 visible light-infrared image pairs, mainly including two typical scenes: campus and wilderness, and the annotated data includes three categories: people, vehicles, and bicycles. The Kaist dataset is a pedestrian detection dataset, covering road traffic scenes from different time periods from day to night, with 11215 infrared-visible light data pairs.

[0022] Step S2: Given a pair of visible light and infrared images, use the CSPDarknet backbone network to extract multi-scale visible light and infrared features respectively: Among them, I R ,I T It is the input pair of visible light and infrared images, θ R ,θ T It is a visible light and infrared image backbone network ψ R ,ψ T The weights of each branch are obtained by pre-training on the COCO dataset. The visible light feature extraction branch and the infrared feature extraction branch do not share weights, and each extracts key features based on the specific characteristics of the modality. Visible light images and infrared images are processed by their respective CSPDarknet backbone networks ψ R ,ψ T The resulting feature map of layer i ∈ {1,2,3,4,5} is where W, H, and C represent the width, height, and number of channels of the feature map, respectively.

[0023] Step S3: Given multi-scale features extracted from visible and infrared images. These features are fused from different scales using a lightweight hybrid expert system: in, It is the fused feature of the i-th layer ∈ {3,4,5}, ψ f It is a feature fusion expert system, and its weights are θ. f The weights include the gating network weights and the weights of each expert network, and the weights of each network are independent of each other. This feature fusion process can be specifically represented by the following formula: Where j is the index of the expert network, and k is the Topk selection parameter of the gating network. The gating network uses Top1 selection to sparsely select the expert network with the highest probability of output. The gating network uses four convolutional layers plus a softmax layer to output a vector of length k, where each value represents the probability of each expert network being activated. ψ g It is a gated network, which uses... For input, θ g The weights of the gating network are used to output the activation probability vectors of each expert network. It is the j-th expert network selected and activated by the gating network, and this network is... For input, For each expert network weight, output size and The same feature maps are used. The expert network uses the NIN attention module, which first applies channel attention to the extracted visible and infrared image features using a filter with a kernel size of 1, to modulate the visible and infrared features and obtain complementary properties. in, The modulated infrared and visible light features are represented respectively. Then, a shared parameter weight allocator is used to assign weights to the modulated features. The modulated features are then multiplied by their weights. Finally, the weighted visible light and infrared features are added together to obtain the final fused features. Where σ represents the Sigmoid activation function, ψ s The weight allocator, representing the shared parameters, consists of 1×1 convolutions, ReLU activation functions, and 1×1 convolutions. Its channel count is halved after the first convolution and reduced to one dimension after the second convolution. Weights are generated separately for modulated visible light and infrared features, and then summed to obtain the fused feature.

[0024] S4. Obtain the recalibrated features. Then, the multi-scale feature aggregation detection head is input for classification and localization: Where D cls D loc These are the category label and the bounding box label for the detected target, respectively. ψ head This represents the detection head, with a weight of θ. h .

[0025] The proposed multispectral detection method consists of a hybrid expert system comprising a dual-stream feature extraction network (CSPDarknet), n NIN expert networks for fusing visible and infrared features, and a gate network (Gate) that assigns weights to each NIN expert network, along with a feature aggregation and object detection head. The weights of these components are randomly initialized, and the weights of each part of the model are updated as the gradient backpropagates. The final optimization objective is to minimize the loss function. The input visible and infrared image pairs are 768×768 pixels in size. Gradient updates are performed using the SGD optimizer with an initial learning rate of 0.01 and 60 iterations.

[0026] like Figure 2-4 As shown, the proposed network's adaptability to different lighting and image degradation scenarios (rain, snow, fog, etc.) was tested on the M3FD, FLIR, and Kaist datasets. During the testing phase, aligned image pairs were input into the model, and the test results show that the proposed method has good detection performance under different lighting and weather conditions.

[0027] The advantage of this method is that it can balance computational speed and accuracy. Compared with Transformer, it can run stably on low computing power devices and achieve accurate detection results under extreme weather conditions (low light, rain, snow, fog, etc.). In multiple public datasets and real-world scenario tests, mAP@50>75%.

[0028] To simplify the explanation of this method, the above figures and text are described as a series of steps. However, it should be understood and understood that these methods are not restricted by the order of operations, because some actions may occur in different orders when implemented according to one or more steps, but those skilled in the art can understand the principle behind the actions.

[0029] Although illustrative specific embodiments of the invention have been described step by step to enable those skilled in the art to understand them, the invention is not limited to the scope of the specific embodiments. Those skilled in the art can make various modifications or variations within the scope of the claims, as long as such variations are within the spirit and scope of the invention as defined and determined by the appended claims.

Claims

1. A multispectral detection method, characterized in that, Includes the following steps: S1. Based on the actual scene requirements, use visible light cameras and infrared cameras to collect image pairs under different weather and climate conditions, and label the targets in the images one by one in the YOLO format. S2. Given a pair of visible light and infrared images, use a dual-stream backbone network with non-shared weights to extract multi-scale visible light and infrared features respectively. S3. Given multi-scale features extracted from visible and infrared images, use a lightweight hybrid expert system to fuse these features from different scales to obtain multimodal complementary features; S4. After obtaining the recalibrated features, input them into the multi-scale feature aggregation network and the target detection head for classification and localization. S5. Input the sample data from the test set into the final trained model and output the target detection results based on multispectral data under extreme weather conditions.

2. The multispectral target detection method according to claim 1, characterized in that: This includes image pair data acquisition and annotation, two-stream network feature extraction, visible light and infrared feature fusion, and object classification and localization.

3. The multispectral target detection method according to claim 2, characterized in that, We constructed a target detection dataset under different climate and lighting conditions, collected and aligned pairs of visible light and infrared images, and extracted complementary features from the visible light and infrared images using a non-shared two-stream backbone network. A lightweight hybrid expert system is adopted to integrate multi-scale visible light and infrared features; a multi-scale complementary feature aggregation module and a target detection head are designed to achieve object classification and localization.

4. The multispectral target detection method according to claim 1, characterized in that: In the dataset preparation phase, image pairs under different weather and climate conditions (low light, rain, snow, fog, etc.) were first collected using visible light and infrared cameras according to the actual scene requirements, and the targets in the images were labeled one by one according to the YOLO format. The proposed method was validated on three public datasets with aligned visible light and infrared light data pairs: M3FD, FLIR, and Kaist. The M3FD dataset contains multiple pairs of labeled images, including different scenes such as low light, exposure, and rain, and the labeled data covers six categories: pedestrians, cars, buses, motorcycles, trucks, and streetlights. The FLIR dataset contains multiple pairs of visible light-infrared image pairs, mainly including two typical scenes: campus and wilderness, and the labeled data includes three categories: people, vehicles, and bicycles. The Kaist dataset is a pedestrian detection dataset that covers road traffic scenes at different times from day to night and has multiple pairs of infrared-visible light data.

5. The multispectral target detection method according to claim 1, characterized in that: Given a pair of visible light and infrared images, multi-scale visible light and infrared features are extracted using the CSPDarknet backbone network: Among them, I R ,I T It is the input pair of visible light and infrared images, θ R ,θ T It is a visible light and infrared image backbone network ψ R ,ψ T The visible light feature extraction branch and the infrared feature extraction branch are independent of each other, and each performs key feature extraction based on the unique characteristics of the modality. Visible light images and infrared images are processed by their respective backbone networks ψ. R ,ψ T The resulting feature map of layer i ∈ {1,2,3,4,5} is where W, H, and C represent the width, height, and number of channels of the feature map, respectively.

6. The multispectral target detection method according to claim 1, characterized in that: Given multi-scale features extracted from visible and infrared images A lightweight hybrid expert system is used to fuse these features at different scales. This hybrid expert system consists of n expert networks and one gating network. Each expert network has the same structure, built on an NIN (Network-in-Network) attention module. The gating network uses four convolutional layers with kernel sizes of 3×3, 3×3, 1×1, and 1×1, plus global average pooling and a softmax layer to output a vector of length k. Each value in the vector represents the probability of each expert network being activated. The fusion process of the hybrid expert system is represented as follows: in, It is the fused feature of the i-th layer ∈ {3,4,5}, ψ f It is a feature fusion expert system, and its weights are θ. f The feature fusion process is specifically represented by the following formula: Where j is the index of the expert network, and k is the Topk selection parameter of the gated network. Top-1 selection is used to sparsely select the expert network with the highest probability of output. The gated network uses four convolutional layers + a softmax layer to output a vector of length k, where each value represents the probability of each expert network being activated. Before the overall network training begins, all weights are randomly assigned, meaning the initial activation probability of each expert network output by the gated network is random. As the target detection loss is calculated and gradient backpropagation occurs over multiple rounds, the activation probability of each expert network output by the gated network is gradually updated with the weights, minimizing the overall loss. The smaller the detection loss, the higher the classification and localization accuracy of the overall network. g It is a gated network, which uses... As input, θ g Assign weights to the gating network and output the activation probability vectors of each expert network. It is the expert network selected for activation by the j-th gated network, and this network is... For input, For each expert network, the weights, output size, and The same feature maps are used; the expert network uses the NIN attention module, which first applies channel attention to the extracted visible and infrared image features using a filter with a kernel size of 1, in order to modulate the visible and infrared features and obtain complementary properties. in, The modulated infrared and visible light features are represented respectively. Then, a shared parameter weight allocator is used to assign weights to the modulated features. The modulated features are then multiplied by their weights. Finally, the weighted visible light and infrared features are added together to obtain the final fused features. Where σ represents the Sigmoid activation function, ψ s This represents a weight allocator with shared parameters. The network consists of 1×1 convolutions, ReLU activation functions, and 1×1 convolutions. The number of channels is halved after the first convolution and reduced to one dimension after the second convolution. Weights are generated for modulated visible light and infrared features respectively, and then summed to obtain the fused features.

7. The multispectral target detection method according to claim 1, characterized in that: Obtain the recalibrated features Then, the multi-scale feature aggregation detection head is input for classification and localization: Where D cls D loc These are the category label and detection box label of the detected target, respectively; ψ head This represents the detection head, with a weight of θ. h .

8. The multispectral target detection method according to claim 1, characterized in that: The system consists of a hybrid expert system comprising a dual-stream feature extraction network (CSPDarknet), n NIN expert networks for fusing visible-infrared light features, and a gate network (Gate) that assigns weights to each NIN expert network. It also includes a feature aggregation and target detection head. The network weights are randomly initialized, and the gradient backpropagation loss function is minimized. The input visible and infrared image pairs are 768×768 pixels in size. Gradient updates are performed using the SGD optimizer with an initial learning rate of 0.01 and 60 iterations.

Images