Target detection method based on infrared visible light feature enhancement and fusion

Abstract

The invention relates to a visible light and infrared multi-mode target detection method, in particular to a target detection method based on infrared and visible light feature enhancement and fusion, which comprises the following steps of: firstly, constructing a double-branch feature encoder suitable for infrared and visible light double input, and respectively extracting double-mode features through a double-branch encoding network; secondly, in order to better utilize the bimodal features to carry out target detection in a complex scene, a DBE module is provided, and the internal feature attention is utilized to enhance information interaction between modals. In addition, a cross attention mechanism is introduced to further improve the expression ability of the network to complementarity information, so that the detection performance of the model is effectively improved. Experimental results show that compared with an existing method, the method disclosed by the invention has excellent performance on an SYUGV data set. For complex environments such as single-mode information loss caused by mutual shielding influence of targets and local transition exposure, the method disclosed by the invention still keeps good detection performance and has relatively high robustness.

Description

Technical Field

[0001] The present invention relates to a visible light and infrared multimodal target detection method, in particular to a target detection method based on infrared and visible light feature enhancement and fusion. Background Art

[0002] In recent years, unmanned aerial vehicles (UAVs) have been widely used in autonomous driving, security monitoring, and disaster relief due to their flexibility, strong detection capabilities, and low risk factors. The ability of UAVs to perform these tasks relies on target detection and recognition technology. [1] However, in complex scenes, the quality of single-sensor imaging will be affected by various environments and conditions, resulting in poor recognition results. To meet the application requirements of multiple scenarios, unmanned aerial platforms are usually equipped with dual-modal vision sensors, such as visible light and infrared sensors. This configuration has been widely used in various target detection tasks. [2] Visible light images have rich texture and color features, but are easily affected by changes in lighting and visibility; infrared images are based on the target's thermal radiation information and are not affected by lighting conditions, but they lack detailed information and have poor imaging quality. [3] Therefore, by combining visible light and infrared detectors, the information complementarity of visible light and infrared images can be achieved, thereby achieving effective target detection in a variety of scenarios.

[0003] Pedestrian and vehicle detection based on visible light and infrared images is a typical application of multimodal target detection. The challenges of dual-modal visual target detection are, on the one hand, the complexity of the actual scene, and the blurred features caused by the moving platform and moving targets; on the other hand, how to make full use of the complementary information of the dual modalities for target detection. [4] In this regard, some researchers use the method of fusing visible light images and infrared images and use the dual-modal fusion results for target recognition. [5] The infrared image is processed using a saliency detection network to obtain a target saliency map, and then the infrared and visible light images are fused using NSCT. Finally, the saliency map and the fusion result are sent to the detector for detection, which shows a high detection accuracy in target detection under poor light conditions. [6] Using the TarDAL network and the detection network, the visible light and infrared images are first fused into one image, and then the fused image is sent to YOLOv5 for detection; Tang [7] A parameter transfer model is proposed, and then a decision-level fusion detection model of infrared and visible light based on deep learning is proposed; Sun [8]A three-branch recognition network UA-CMDet consisting of a visible light branch, an infrared branch, and a fusion branch is proposed. ResNet-50 is used as the backbone network to extract image features, and then the three branch results are used for decision recognition. These front-end fusion methods retain the detailed information of the background and foreground in the fused modality, resulting in redundant information in the fused modality, which affects the subsequent recognition performance. The back-end fusion method cannot utilize cross-modal information, and the fusion at the result level requires two or more models, resulting in a large amount of calculation.

[0004] Feature-level fusion integrates bimodal data information while ensuring the lightweight of the model, so it has become the current mainstream fusion method. [9] Based on Faster R-CNN, a dual-branch detection network that fuses visible light and infrared images is designed. It sends paired visible light and infrared dual-modal images to the VGG-16 backbone network to extract features, and then fuses the dual-branch features for classification and bounding box regression.

[10] A real-time detection method based on image brightness attention, MAF-YOLO, is proposed. A multimodal feature extraction module is built based on the Darknet-53 framework, and then fused through the modality weighted fusion module. Cheng

[11] A lightweight dual-mode fusion network SLBAF-Net is proposed. In order to more effectively fuse dual-mode features, a dual-mode adaptive fusion module (BAFM) is proposed to perform adaptive fusion of visible light and infrared feature maps to improve detection robustness in complex environments. Since traditional CNN can only focus on local feature information, Shen

[12] A fusion method based on bidirectional attention-guided features is proposed. This method uses Transformer to perform global feature interaction and complementary information capture on visible light and infrared dual modalities, thereby improving the discrimination of target features and effectively solving the problem of image misalignment. However, the fusion method of splicing or stacking feature channels and allocating bimodal weights using mechanisms such as attention cannot fully utilize the complementary information of the bimodal modes, especially in the case of poor feature conditions. This mechanism suppresses the feature map and also leads to the loss of useful information. References:

[0005] [1]Ramachandran A,Sangaiah A KA review on object detection inunmanned aerial vehicle surveillance[J].International Journal of CognitiveComputing in Engineering,2021,2:215-228.

[0006] [2]Hu Y, Shi L, Yao L, et al.Dual Attention Feature Fusion for Visible-Infrared Object Detection[C] / / International Conference on Artificial NeuralNetworks. Cham: Springer Nature Switzerland, 2023:53-65.

[0007] [3] Ning Dahai, Zheng Sheng. Decision-level fusion target detection algorithm for visible light and infrared images[J]. Infrared Technology, 2023, 45(3): 282-291.

[0008] [4] Bustos N, Mashhadi M, Lai-Yuen SK, et al. A systematic literature review on object detection using near infrared and thermal images [J]. Neurocomputing, 2023: 126804.

[0009] [5] Yue G, Li Z, Tao Y, et al. Low-illumination traffic object detection using the saliency region of infrared image masking on infrared-visiblefusion image[J]. Journal of Electronic Imaging, 2022, 31(3):033029-033029.

[0010] [6]Liu J, Fan

[0011] [7]Cong T,Yongshun L,Hua Y,et al.Decision-level fusion detection forinfrared and visible spectra based on deep learning[J].Infrared and LaserEngineering,2019,48(6):626001-0626001(15).

[0012] [8]SUN Y M,CAO B,ZHU P F,et al.Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning[J].IEEETransactions on Circuitsand Systems for Video Technology,2022,32:6700-6713.

[0013] [9]GENG K K,ZOU W,YIN G D,et al.Low-observable targets detection forautonomous vehicles based on dual-modal sensor fusion with deep learningapproach[J].Journal of Automobile Engineering,2019,233(9):2270-2283.

[0014]

[10] Xue Y,Ju Z,Li Y,et al.MAF-YOLO:Multi-modal attention fusion basedYOLO for pedestrian detection[J].Infrared Physics&Technology,2021,118:103906.

[0015]

[11] Cheng

[0016]

[12] Shen J, Chen Y, Liu Y, et al.ICAFusion: Iterative cross-attentionguided feature fusion for multispectral object detection[J].PatternRecognition,2024,145:109913. Summary of the invention

[0017] In order to cope with the challenges of infrared and visible light dual-modal target detection in complex dynamic environments, especially the problems of insufficient target feature expression and the inability to fully utilize complementary features of infrared and visible light features in dual-modal fusion, resulting in missed detection and false detection, the present invention combines the attention mechanism with the Transformer model, and proposes a target detection method based on infrared and visible light feature enhancement and fusion. A dual-branch feature-level fusion method is adopted, and a feature interaction enhancement module and a cross-fusion network are embedded, so that the model can fully utilize the dual-modal information.

[0018] The present invention is implemented by adopting the following technical scheme: a target detection method based on infrared and visible light feature enhancement and fusion, the method comprises a dual-branch feature encoder, a cross fusion network and a detector, wherein the dual-branch feature encoder comprises a convolution feature extraction module and a feature interaction enhancement module, and the visible light image I vi and infrared image I ir The visible light features F are obtained through the convolution feature extraction module respectively. vi and infrared signature F ir , and then combined with F through the feature interaction enhancement module vi and F ir Assign spatially important feature maps to each channel to obtain enhanced features F f , enhanced feature F f After distribution according to channel dimension, the bimodal feature F is obtained ev and F ei ; F ev and F eiThrough the cross-fusion network, more effective feature representation is obtained to obtain the fusion feature F0, which realizes the complementarity of bimodal feature information; the enhanced feature F f And the fusion feature F0 is used as the input of the detector to complete the classification and positioning of the target.

[0019] In the above-mentioned target detection method based on infrared and visible light feature enhancement and fusion, the dual-branch feature encoder includes a CONV module, a CSP structure and a DBE module; the CONV module is used as a basic module for feature extraction, including a convolution layer, a batch normalization layer and a SILU activation function; the CSP structure introduces residual features to enhance the learning ability of the convolutional network, and the CONV module and the CSP structure constitute a convolutional feature extraction module. The visible light image I vi and infrared image I ir The visible light and infrared features obtained by the convolution feature extraction module are F vi ,F ir , F vi ,F ir Bimodal feature map F after Concat c As the input of the DBE module, the spatial attention weight W is calculated s and channel attention weight W c ,

[0020] W s =Conv 7×7 (Concat(MaxPool(F c ),AvgPool(F c )))

[0021] W c =Conv 1×1 (Max(0,Conv 1×1 (AvgPool(F c )))), where AvgPool represents the global average pooling across channel dimensions, MaxPool represents the global maximum pooling operation across channel dimensions, and AvgPool represents the global average pooling across spatial dimensions. s With W c Perform addition operation to obtain the initial attention weight W cs , that is, W = W + W, the bimodal feature map F c With the attention weight W cs Spell cs cs

[0022] The subsequent feature layer is used as the input of the convolutional layer, and the feature weight value is normalized by the sigmoid function to obtain W, W = σ (Conv 7×7 (Concat([F c ,W cs]))), W combines the bimodal feature information and assigns a unique spatially important feature map to each channel. c Combine and calculate the weighted feature information, and introduce residual connections to add bimodal feature maps F c , calculate the enhanced feature F f =F c ·W+F c , and finally enhance the feature F f The interactively enhanced bimodal features are obtained by allocating them according to the channel dimension

[0023] The above-mentioned target detection method based on infrared and visible light feature enhancement and fusion, the dual-modal feature F ev and F ei After average pooling downsampling and dimension conversion operations, it is converted into sequence I ev ,I ei , for the input sequence I ev , project it onto three weight matrices to calculate a set of Query, Key and Value vectors, namely Q i =I ei W Qi , K v =I ev W Kv , V v =I ev W Vv ,in,

[0024] W Qi , W Kv , W Vv is a learnable weight parameter, calculating the visible light attention weight Use multi-head attention mechanism to calculate the attention weight of visible light features Where h represents the number of heads, W O represents the parameter matrix;

[0025] For the input sequence I ei , project it onto three weight matrices to calculate a set of Query, Key and Value vectors, namely K i =I ei W Ki , V i =I ei W Vi , where W Qv , W Ki , W Vi is a learnable weight parameter, calculating the infrared attention weight Using multi-head attention mechanism to calculate infrared feature attention weights Where h represents the number of heads, W O Represents the parameter matrix; assign two weights to the bimodal features F ev and F ei After connecting Output the fusion feature F through the subsequent Transformer connection layer O :

[0026] FC1 is a linear layer, RELU represents a nonlinear activation, and FC2 represents a linear layer.

[0027] The present invention proposes a target detection method based on infrared and visible light feature enhancement and fusion. The method first constructs a dual-branch feature encoder suitable for dual input of infrared and visible light, and extracts dual-modal features respectively through a dual-branch encoding network. Secondly, in order to better utilize dual-modal features for target detection in complex scenes, a DBE module is proposed, which uses internal feature attention to enhance information interaction between modalities. In addition, the introduction of the cross-attention mechanism further enhances the network's ability to express complementary information, thereby effectively improving the detection performance of the model. Experimental results show that compared with existing methods, the method of the present invention performs well on the SYUGV dataset. Faced with complex environments such as the influence of mutual occlusion of targets and the loss of single-modal information due to local transition exposure, the method of the present invention still maintains good detection performance and has strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the overall architecture diagram of infrared-visible light target detection.

[0029] Figure 2 Schematic diagram of the dual-branch feature encoder structure.

[0030] Figure 3 Schematic diagram of the feature interaction enhancement module structure.

[0031] Figure 4 Schematic diagram of the cross-attention fusion structure.

[0032] Figure 5 Schematic diagram of the SYUGV dataset.

[0033] Figure 6 PR curves for different modal inputs.

[0034] Figure 7 Grad-CAM heat map of the dual-branch model.

[0035] Figure 8 This is the average detection accuracy curve of the ablation experiment.

[0036] Fig. 9 Feature attention visualization for different models.

[0037] Fig.10 Comparison chart for model training.

[0038] Fig.11 This is a comparison chart of the detection effects of different models on the SYUGV dataset. DETAILED DESCRIPTION

[0039] A target detection method based on infrared and visible light feature enhancement and fusion. Figure 1 The dual-branch feature enhancement and fusion (DBEF-Net) target detection framework in the paper consists of three parts: dual-branch feature encoder, cross fusion network and detector. The dual-branch feature encoder consists of a convolutional feature extraction module and a dual-branch interaction enhancement module (DBE). vi and infrared image I ir The visible light features F are obtained through the convolution feature extraction module respectively. vi and infrared signature F ir , and then combined with F through the feature interaction enhancement module vi and F ir Assign a spatial importance map (SIM) to each channel to obtain the enhanced feature F f , enhanced feature F f After distribution according to channel dimension, the bimodal feature F is obtained ev and F ei , F ev and F ei Through the cross-fusion network, more effective feature representation is obtained to obtain the fusion feature F0, which realizes the complementarity of bimodal feature information. f The fusion feature F0 is used as the input of the feature pyramid, and YOLOv8 is selected as the detector to complete the classification and positioning of the target.

[0040] Dual-branch feature encoder

[0041] Due to the imaging differences between infrared and visible light and their respective modality specificity, independent networks are used to extract the features of the two. Referring to the CSPDarknet53 backbone network, a dual-branch feature encoder is designed. Figure 2As shown in the figure, it includes CONV module, CSP structure and DBE module. CONV module is the basic module for feature extraction, including convolution layer, batch normalization layer and SILU activation function; CSP structure introduces residual features to enhance the learning ability of convolution network; DBE module combines dual-branch features to obtain enhanced representation of bimodal feature map. In order to balance bimodal information and limit network complexity, both branches use 640×640 pixels and 3-channel input, and finally are downsampled to 20×20×256 size feature map. This design aims to fully extract and enhance bimodal information and provide a favorable feature representation basis for subsequent feature fusion.

[0042] Most methods to improve performance are to fuse features using various types of attention, including channel attention (Squeeze-and-Excitation Networks, SENet) and convolutional attention (Convolutional BlockAttention Module, CBAM). However, attention within the modality can only solve the uneven distribution of single modal features, but ignores the uneven distribution of bimodal feature levels. Although the channel attention within the feature attention module integrates the differences between channels, it does not consider the contextual information between infrared and visible light, which leads to uneven distribution of infrared and visible light information for each feature channel in the spatial dimension. In addition, the attention weights of space and channels in feature attention usually lack information interaction. Therefore, it is necessary to combine the feature channels of infrared and visible light to construct a spatially important feature map to enhance the interaction of bimodal information. To solve the above problems, a feature interaction enhancement module (DBE) is designed as follows: Figure 3 shown.

[0043] I vi ,I ir After a series of convolution modules, the visible light and infrared features are obtained as follows: In order to utilize the bimodal information, F vi ,F ir After Concat As the input of the DBE module, the spatial attention and channel attention are calculated as shown in formula (1-2), and the spatial attention weight is obtained and channel attention weight

[0044] W s =Conv 7×7 (Concat(MaxPool(F c ),AvgPool(F c ))) (1)

[0045] W c =Conv 1×1(Max(0,Conv 1×1 (AvgPool(F c )))) (2)

[0046] Among them, AvgPool in formula (1) represents the global average pooling across channel dimensions, and MaxPool represents the global maximum pooling operation across channel dimensions. In formula (2), AvgPool represents the global average pooling across spatial dimensions. The two convolutions are used to reduce and increase dimensions respectively. The first convolution reduces the dimension 2C to r is the scale factor, and the second convolution expands it back to 2C. To limit the model complexity, r is set to 8, which keeps the amount of calculation and has a considerable effect. The activation function is introduced in the two convolutions to improve the feature representation ability and help the gradient flow. In order to make the two attentions interact, the Broadcast mechanism is used for W s With W c Perform addition operation to obtain the initial attention weight W cs ,Right now

[0047] W cs =W c +W s (3)

[0048] According to the initial attention weight W for each channel of the input bimodal feature map cs The refinement is performed to focus on the unique parts of the features in each channel to generate the final SIM. The input bimodal feature map F c With the attention weight W cs The concatenated feature layer is used as the input of the convolutional layer, and the feature weight is normalized by the sigmoid function to obtain W. The calculation is shown in formula (4). W combines the bimodal feature information and assigns a unique SIM to each channel to guide the model to focus on important areas in the infrared and visible light channels.

[0049] W=σ(Conv 7×7 (Concat([F c ,W cs ]))) (4)

[0050] F f =F c ·W+F c (5)

[0051] Combine W with the input features to calculate the weighted feature information, and introduce residual connections to add input features, as shown in formula (5), to enhance gradient transfer and prevent feature information loss due to too low weight allocation. Finally, the enhanced feature F f The interactively enhanced bimodal features are obtained by allocating them according to the channel dimension, as shown in formula (6).

[0052]

[0053] Cross-fusion network

[0054] The feature interaction enhancement module DBE based on the spatial and channel attention mechanism enables the model to focus on important features. However, simply assigning bimodal feature channel weights cannot effectively utilize the complementarity between the two modalities. In order to fully explore and utilize the complementary information of infrared and visible light features, the advantage of Transformers in modeling long-range sequence relationships and the idea of ​​Cross-Attention Transformers (CTF) are used to construct a cross-attention fusion network with the following structure: Figure 4 As shown in the figure. In the feature maps of infrared and visible light, useful information is retrieved from each other for extraction, while interference information does not work. This is actually a two-way complementary information extraction process, mutual information retrieval from visible light to infrared and from infrared to visible light to fuse dual-modal features.

[0055] Specifically, the input interaction enhances the bimodal features To utilize Transformer attention, the input is converted into a sequence by average pooling downsampling and dimension conversion operations. To limit the complexity of the model, For the input sequence, project it onto three weight matrices to calculate a set of Query, Key, and Value vectors, namely

[0056] Q i =I ei W Qi (7)

[0057] K v =I ev W Kv (8)

[0058] V v =I ev W Vv (9)

[0059] in, is a learnable weight parameter. In this model, the number of bimodal channels is C and the parameter D Q =D K =D V = C, use infrared feature information to retrieve visible light feature information to calculate channel correlation, calculate cross attention weight by scaling dot product, visible light feature information provides Key and Value, as shown in formula (10), output C V is the visible light attention weight fused with infrared features.

[0060]

[0061] In order to handle the problem of representing different complex relationships from different positions, a multi-head attention mechanism is adopted:

[0062]

[0063] Wherein, h represents the number of heads, which is set to 4 in the present invention. Concat(C1,…,C h ) is used to generate the attention weights of visible light features fused with infrared Similarly, for the infrared feature, the infrared feature attention weight can be obtained Assign the two weights to the initial branch features and then connect them to get Output the fusion feature F through the subsequent Transformer connection layer o :

[0064]

[0065] The Transformer connection layer includes two layers of neural network, which are successively passed through the linear layer FC1, RELU nonlinear activation and linear transformation FC2. Figure 4 As shown in Figure 1, a cross-fusion network contains 8 cross-attention blocks to achieve inter-modal fusion. Finally, the output corresponds to the input features, and the output F is interpolated by bilinear interpolation. o Upsampled to the original resolution. By leveraging the cross-attention mechanism of Transformer, the model can naturally perform inter-modal fusion and effectively capture the interactive correlation between visible light and infrared features, thereby improving the performance of multimodal object recognition.

[0066] Overall detection architecture

[0067] The backbone part of the network consists of the above-mentioned dual-branch feature encoder and the cross-fusion network. The dual-branch feature encoder enhances the dual-modal features step by step through multiple feature interaction blocks, and the cross-fusion network can achieve fusion between modalities at all scales. The backbone output of the present invention is: 32 times downsampled fusion feature F after 3 layers of DBE feature enhancement and feature fusion using the cross-fusion network o , after 2 layers of DBE feature enhancement, the 16-fold downsampled feature F is obtained by Concat f And the 8-fold downsampled feature F obtained by concat after 1 layer of DBE feature enhancement fThe features of these three scales are used as the input of the Feature Pyramid Network (FPN), and the target detection performance is improved by fusing high-level and low-level features. Finally, the three-level features are processed by YOLOv8-Head to output the final regression box and classification results. The loss function of this method is divided into two parts, namely the target box loss for target recognition and the improved cross entropy loss for classification.

[0068] The target box loss function uses CIoU (Complete-IoU Loss). CIoU fully considers the three important index parameters of overlapping area, center point distance, and aspect ratio. By constructing the loss function, the predicted box is closer to the real box, achieving a more comprehensive optimization. The calculation formula of CIoU is as follows:

[0069]

[0070] Among them, IoU represents the intersection-over-union ratio of the predicted box and the real box, ρ represents the Euclidean distance between the center points, c represents the diagonal distance of the minimum circumscribed rectangle of the two boxes, α is the balance coefficient, b, b gt are the center points of the predicted box and the true box respectively. w, h, w gt 、h gt Represent the width and height of the predicted box and the width and height of the real box respectively, and v is used to calculate the consistency of the aspect ratio of the two boxes. The classification loss uses Varifocal Loss (VFL), which adopts an asymmetric training sample weighting method to reduce the weight of negative samples and increase the weight of high-quality positive samples, thereby focusing on the training of high-quality positive samples. The calculation is as follows:

[0071]

[0072] Among them, p is the classification score of the model, q is the IoU between the predicted box and the real box, when q>0, the target is a positive sample, if the two boxes have no intersection, then q=0, it is a negative sample, β and γ are hyperparameters, this loss function pays more attention to the positive samples that are difficult to classify, thereby improving the overall target detection performance.

[0073] Experimental results and analysis

[0074] Experimental conditions

[0075] The experimental system is Windows 11-23H2, the hardware platform is Intel Core i7-13650, NVIDIA GeForceGTX 4060 (video memory 8GB), and the CUDA version is 11.6. The experiment is based on Python and PyTorch platforms, and SGD is selected as the optimizer for training. During the training process, BatchSize is set to 8, and visualization tools are used to observe the changes in the loss function during model training. After 200 Epochs, the loss function tends to be stable and the model converges. After the training is completed, the experimental results are compared based on different indicators of each model on the test set.

[0076] Datasets and evaluation metrics

[0077] In order to verify the advantages of this method in dual-modal data detection in complex scenes and apply unmanned platforms to actual scenes, a dataset SYUGV of complex actual scenes collected by the unmanned platform SYUGV-01 was constructed. Taking into account the environmental complexity that may be encountered in actual use, this dataset covers night scenes, overexposure, dynamic blur, and target occlusion. The resolution of dual-modal images is 720×576. In view of the fact that the dual-modal data is taken from continuous video frames and there are no valid targets in multiple frames, a certain degree of data cleaning is performed. In the end, a total of 6272 valid images were obtained, of which 3232 were taken during the day and 3040 were taken at night. Figure 5 As shown in the figure, it is divided into 5146 training sets and 1126 test sets. There is no overlap between the test set and the training set. The label categories mainly include pedestrians and vehicle targets. The bimodal data has been roughly aligned and shares one set of label data.

[0078] The evaluation indicators used in the experiment include accuracy (Precision, P), recall (Recall, R), mean average precision (Mean Average Precision, mAP), frames per second (Frames Per Second, FPS), etc., and the number of floating point operations (Flops) and the number of model parameters (Params) are used to test the number of parameters and model complexity of the algorithm. Among them, mAP0.50 is the average detection accuracy calculated with an IoU of 0.5 between the bounding box and the label, and mAP0.50-0.95 refers to the average AP of all categories calculated at intervals of 0.05 from 0.50 to 0.95.

[0079] Ablation experiment analysis

[0080] In order to evaluate the contribution of the dual-branch feature encoder and the cross-fusion network to the object detection performance, a series of ablation experiments were conducted on the SYUGV dataset and compared with the benchmark algorithm model to verify the effectiveness of the proposed method.

[0081] Dual-branch feature encoder

[0082] In order to limit the complexity of the model while maintaining accuracy and meet the real-time detection requirements of the bimodal model, Yolov8s was selected as the single-modal benchmark detection network. The bimodal benchmark network was constructed by modifying the backbone part to the above-mentioned dual-branch encoder structure and reducing the number of feature channels, thereby constructing Dual-Yolov8s, which will serve as the dual-branch benchmark model for subsequent experiments. The design of the dual-branch feature encoder structure enables the network to extract and learn the features of the two modalities separately, avoiding the information confusion and loss that may occur when a single-branch network processes bimodal information.

[0083] Through experimental comparison on the SYUGV dataset, as shown in Table 1, it is verified that the dual-branch feature encoder improves the detection performance. Specifically, when the YOLOv8s model only uses visible light images (VI) as input, mAP 0.5 The mAP is 86.7% when only infrared images (IR) are used as input. 0.5 The detection accuracy of dual-modal images based on the dual-branch feature encoder reaches 94.2%, which is 7.5% and 2.4% higher than that of single visible light and infrared image inputs, respectively. At the same time, by reducing the number of feature channels in the backbone network, the number of Flops and Params on the dual-modal target detection model Dual-Yolov8s is reduced by 5.8B and 2.3M respectively compared with the single-branch baseline model. In terms of detection speed, the proposed dual-branch model structure maintains the detection speed of the single-branch structure.

[0084] Table 1 Model detection performance of different modal inputs

[0085]

[0086] Figure 6 The PR (Precision-Recall) curves of different models are plotted. It can be seen intuitively that after using the dual-branch structure to extract infrared and visible light features respectively, the overall detection accuracy is significantly improved compared to the single mode.

[0087] The Grad-CAM method is used to identify and highlight the key areas in the input image for the network model to make category predictions. All target categories are selected and the feature map is used to calculate the weight features of different layers to generate a heat map. In the generated heat map, the red area indicates the area that the model focuses on when making predictions, that is, the area that has the greatest impact on the prediction results. The yellow area has the second highest degree of attention, and the blue area indicates the area that has a smaller impact on the prediction results. Taking night scene data as an example, the present invention uses the Grad-CAM method to select different network layers to realize feature attention visualization, and the effect is as follows: Figure 7 shown.

[0088] from Figure 7 As can be seen from the figure, when processing night images, especially when the local exposure (such as Figure 7 (a)) or insufficient light (such as Figure 7 (e)), it is difficult for the visible light image to display all the target information, and the visible light single-branch model only focuses on some weak features, such as Figure 7 (b) and 7(f). After introducing the infrared branch, the model significantly focuses on the brightness features of thermal targets that are difficult to detect in visible light images but prominent in infrared images, such as Figure 7 (d) and 7(h). The network uses a dual-branch feature encoder to process images of two modalities separately, utilizing the specific features of infrared and visible light, which verifies the effectiveness of the dual-branch structure in processing dual-modal information.

[0089] Feature interaction enhancement and fusion

[0090] In order to quantitatively analyze the contribution of the dual-modal feature interaction enhancement module DBE and the feature fusion network to the detection performance, further ablation experiments were conducted on the SYUGV dataset, and the results are shown in Table 2. By adding the feature interaction enhancement module DBE on the basis of the dual-branch encoder, the detection accuracy is improved from 94.2% to 95.2%, which is better than the effect of applying attention enhancement methods such as CBAM and ECA to the dual-branch network. The detection accuracy of the proposed feature interaction enhancement module DBE is improved by 1.6% and 0.7% compared with CBAM and ECA attention, respectively. The use of only the dual-branch encoder and cross-attention fusion does not enhance the specificity of infrared and visible light features, and the improvement effect is limited. The feature interaction enhancement module DBE enhances the useful information of infrared and visible light features, so that the cross-attention can fully fuse complementary features, thereby significantly improving the detection accuracy, which is 1.4% higher than the baseline model detection accuracy. Among them, the detection accuracy of the cross-attention mechanism is improved by 0.7% compared with the self-attention mechanism. Figure 8 This is the mAP0.5 curve of the ablation comparison experiment.

[0091] Table 2 Ablation experimental results of the dual branch detection model

[0092]

[0093] Select the same corresponding feature layer of different models after passing through the backbone network in the unmelted comparison experiment to generate a heat map, such as Fig. 9 As shown in the figure, the feature attention range of the baseline dual branch is relatively scattered, not only focusing on target areas such as pedestrians and vehicles, but also focusing on the surrounding area of ​​the target; after adding DBE, the focus feature further focuses on pedestrians, vehicles and other targets themselves; after adding CTF, the focus does not fall entirely on the target, and the cross attention does not play a full role due to the lack of feature-specific interaction enhancement. However, after feature interaction enhancement and attention fusion, the model can accurately focus on the dual-modal feature information of the target.

[0094] Comparative experimental analysis

[0095] To verify the effectiveness of this method, it is applied to the SYUGV dataset with recent dual-modal target detection algorithms, such as MAF-YOLO, SLBAF-Net, and ICAFusion, and the evaluation is compared using indicators. The specific indicators are shown in Table 3, and the comparison of different indicators is shown in Figure 3. Fig.10 shown.

[0096] Tab.3 Comparative experimental results of different models

[0097]

[0098] As shown in Table 3, compared with other methods, this method has higher detection accuracy and faster detection speed. Compared with MAF-YOLO and SLBAF-Net methods, the detection of this method is improved by 2.8% and 2.5% respectively. Compared with the ICAFusion method, under the condition of meeting high accuracy, the detection rate of this method is greatly improved, which can meet the needs of real-time detection of the platform.

[0099] The confidence threshold of inference of different model methods is set to 0.25, and the intersection-over-union threshold is set to 0.7. Some actual detection results are compared. Fig.11 As shown in the figure, the visualization results are flexible. Considering that infrared images are relatively rich in information, infrared images are used as detection results. It can be seen from the experimental results in actual scenes that this method has good detection capabilities, such as Fig.11(b) with Fig.11 In (c), there is a situation where the targets affect each other, and the proposed method identifies the partially obscured target. Through comprehensive comparison, this method can be better used for target detection in actual systems, allowing unmanned platforms to maintain high accuracy and complete multi-sensor real-time target detection tasks.

Claims

1. A target detection method based on infrared and visible light feature enhancement and fusion, characterized by: The method includes a dual-branch feature encoder, a cross-fusion network and a detector, wherein the dual-branch feature encoder includes a convolution feature extraction module and a feature interaction enhancement module, and the visible light image I vi and infrared image I ir The visible light features F are obtained through the convolution feature extraction module respectively. vi and infrared signature F ir , and then combined with F through the feature interaction enhancement module vi and F ir Assign spatially important feature maps to each channel to obtain enhanced features F f , enhanced feature F f After allocation according to channel dimension, the bimodal feature F is obtained ev and F ei ; F ev and F ei Through the cross-fusion network, more effective feature representation is obtained to obtain the fusion feature F0, which realizes the complementarity of bimodal feature information; the enhanced feature F f And the fusion feature F0 is used as the input of the detector to complete the classification and positioning of the target.

2. The target detection method based on infrared and visible light feature enhancement and fusion according to claim 1 is characterized in that: The dual-branch feature encoder includes a CONV module, a CSP structure and a DBE module; the CONV module is the basic module for feature extraction, including a convolution layer, a batch normalization layer and a SILU activation function; the CSP structure introduces residual features to enhance the learning ability of the convolutional network, and the CONV module and the CSP structure constitute a convolutional feature extraction module. vi and infrared image I ir The visible light and infrared features obtained by the convolution feature extraction module are F vi ,F ir , F vi ,F ir Bimodal feature map F after Concat c As the input of the DBE module, the spatial attention weight W is calculated s and channel attention weight W c , W s =Conv 7×7 (Concat(MaxPool(F c ),AvgPool(F c ))) W c =Conv 1×1 (Max(0,Conv 1×1 (AvgPool(F c )))), where AvgPool represents the global average pooling across channel dimensions, MaxPool represents the global maximum pooling operation across channel dimensions, and AvgPool represents the global average pooling across spatial dimensions. s With W c Perform addition operation to obtain the initial attention weight W cs , that is, W cs =W c +W s , the bimodal feature map F c With the attention weight W cs The concatenated feature layer is used as the input of the convolutional layer, and the feature weight value is normalized by the sigmoid function to obtain W, W = σ (Conv 7×7 (Concat([F c ,W cs ]))), W combines the bimodal feature information and assigns a unique spatially important feature map to each channel. c Combine and calculate the weighted feature information, and introduce residual connections to add bimodal feature maps F c , calculate the enhanced feature F f =F c ·W+F c , and finally enhance the feature F f The interactively enhanced bimodal features are obtained by allocating them according to the channel dimension 3. The target detection method based on infrared and visible light feature enhancement and fusion according to claim 2 is characterized in that: Bimodal feature F ev and F ei After average pooling downsampling and dimension conversion operations, it is converted into sequence I ev ,I ei , for the input sequence I ev , project it onto three weight matrices to calculate a set of Query, Key and Value vectors, namely Q i =I ei W Qi , K v =I ev W Kv , V v =I ev W Vv , where W Qi , W Kv , W Vv is a learnable weight parameter, calculating the visible light attention weight Use multi-head attention mechanism to calculate the attention weight of visible light features Where h represents the number of heads, W o represents the parameter matrix; For the input sequence I ei , project it onto three weight matrices to calculate a set of Query, Key and Value vectors, namely K i =I ei W Ki , V i =I ei W Vi , where W Qv , W Ki , W Vi is a learnable weight parameter, calculating the infrared attention weight Using multi-head attention mechanism to calculate infrared feature attention weights Where h represents the number of heads, W O Represents the parameter matrix; assign two weights to the bimodal features F ev and F ei After connecting Output the fusion feature F through the subsequent Transformer connection layer O : FC1 is a linear layer, RELU represents a nonlinear activation, and FC2 represents a linear layer.

Images