Visible light-infrared target detection method based on structured feature extraction and dynamic weight prediction

Abstract

The invention discloses a visible light-infrared target detection method based on structured feature extraction and dynamic weight prediction. The method comprises the following steps: carrying out target detection through a visible light-infrared target detection model based on structured feature extraction and dynamic weight prediction; the target detection model comprises a thermal radiation-motion feature extraction module, a multi-granularity visual representation module and a collaborative weight decision module; the thermal radiation-motion feature extraction module is used for capturing thermal radiation information in an infrared image and motion features of a target so as to generate infrared feature representation; the multi-granularity visual representation module is used for performing structured multi-granularity visual feature extraction on the visible light image so as to construct visible light feature representation; and the collaborative weight decision module is used for performing cross integration of infrared feature representation and visible light feature representation through a dynamic weight prediction mechanism. The target detection model provided by the invention has better performance in a visible light-infrared target detection task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visible light-infrared multi-modal target detection, and in particular to a visible light-infrared target detection method based on structured feature extraction and dynamic weight prediction, which is mainly used for improving the detection effect of targets in visible light-infrared fusion images in complex environments, and is especially suitable for target detection tasks in all-weather and multiple scenes. BACKGROUND

[0002] With the continuous development of science and technology, visible light and infrared imaging technology have been widely applied in the field of target detection. Visible light imaging technology can capture visual information such as texture and contour of objects, and performs well in daily lighting conditions. Infrared imaging technology, on the other hand, can still clearly image in the dark, bad weather and other environments where visible light imaging is limited, effectively avoiding the problem of general object occlusion to the target. Therefore, the combination of visible light and infrared imaging technology can fully play to the complementary advantages of both, providing a more comprehensive and reliable solution for target detection and recognition in complex environments. This multi-modal fusion technology has extremely important application value in many fields such as military target monitoring, security monitoring, aerospace, autonomous driving and industrial detection. However, in the field of visible light-infrared target detection, there are still some problems to be solved in the existing methods. On the one hand, in the feature extraction stage, the physical characteristics of visible light and infrared modalities are quite different, and the adaptability of existing methods to this difference is insufficient, making it difficult to fully exploit the complementary information of the two modalities. For example, visible light images focus on the expression of texture and contour, while infrared images reflect more the thermal radiation characteristics of objects. If this difference cannot be well handled, it will lead to insufficient feature extraction and affect the subsequent detection effect. On the other hand, there are also some problems in the cross-modal fusion mechanism. Most of the current fusion methods either have limited expression ability and cannot effectively integrate the features of the two modalities, or have redundant parameters, resulting in high computational complexity, low model training and inference efficiency. These problems together become the bottleneck of the performance improvement of visible light-infrared target detection models. These factors interweave each other, making the visible light-infrared target detection task more complex and difficult. Therefore, designing a visible light-infrared target detection method that can effectively solve the above problems is of great significance for improving the accuracy and robustness of target detection and meeting the needs of practical applications.

[0003] In summary, the existing target detection methods based on infrared and visible light fusion generally use a structurally homogeneous backbone network for dual-stream feature extraction. This approach ignores the inherent physical characteristic differences of different modalities, leading to difficulty in fully adapting the extracted feature representation to the unique advantages and information distribution characteristics of each modality, limiting the quality of source information available for subsequent fusion and becoming a bottleneck for model performance improvement. In addition, in the cross-modality feature fusion stage, existing methods have difficulty in efficiently modeling and fully utilizing the strong complementarity between modalities. Common fusion strategies (such as simple concatenation and element-wise addition) easily lead to insufficient information interaction or the introduction of redundant coupling, and cannot selectively integrate the most discriminative complementary clues. At the same time, to achieve effective interaction, existing fusion modules often involve a significant increase in parameter quantity or a sharp rise in computational complexity, leading to a contradiction between model efficiency and performance improvement, hindering their practical deployment and application. SUMMARY

[0004] The present application aims to address the problem of insufficient adaptability to the physical characteristic differences of modalities in the feature extraction stage and the bottleneck in cross-modality fusion mechanisms of existing methods. In order to improve the detection accuracy and robustness of visible light-infrared images and achieve efficient all-weather target detection in complex environments, a visible light-infrared target detection method based on structured feature extraction and dynamic weight prediction is proposed. This method fully utilizes the texture contour information of the visible light modality and the thermal radiation information of the infrared modality through structured feature extraction, and simultaneously achieves efficient integration of complementary features of the two modalities through dynamic weight prediction, thereby improving the detection accuracy and robustness of visible light-infrared targets and better meeting the needs of practical applications.

[0005] To achieve the above purpose, the present application adopts the following technical solutions: A visible light-infrared target detection method based on structured feature extraction and dynamic weight prediction, comprising: Performing target detection through a visible light-infrared target detection model based on structured feature extraction and dynamic weight prediction; the target detection model includes a thermal radiation-motion feature extraction module, a multi-granularity visual representation module, and a collaborative weight decision module; the thermal radiation-motion feature extraction module is used to capture thermal radiation information and motion features of the target in the infrared image, and then generate an infrared feature representation; the multi-granularity visual representation module is used to extract structured multi-granularity visual features from the visible light image, and then construct a visible light feature representation; the collaborative weight decision module is used to cross-integrate the infrared feature representation and the visible light feature representation through a dynamic weight prediction mechanism.

[0006] Further, the thermal radiation-motion feature extraction module includes a residual path, a thermal radiation path, a dynamic motion path, and a feature fusion and calibration process. The residual path processes the input infrared feature tensor through a 1x1 convolution and a batch normalization operation Projecting to the target dimension, using identity mapping if the number of input and output channels is the same, obtaining the processed feature of the residual path The thermal radiation path is used to obtain thermal radiation information in the infrared image The dynamic motion path is used to capture the motion features of the target in the infrared image The feature fusion and calibration process is used to fuse and calibrate , and to obtain the infrared feature representation.

[0007] Further, the thermal radiation path is specifically configured to perform the following steps: Average pooling is performed on to smooth the features and expand the receptive field; At the same time, the channel attention weight is calculated, which is generated through global average pooling and two fully connected layers, to calibrate the importance of each channel; The pooled features are multiplied by the channel attention weight to obtain the calibrated features Then, is divided into two parts and in the channel dimension, and the two parts are processed through two different down-sampling branches respectively; wherein is processed through a conditional routing convolution, which dynamically focuses on different thermal targets by introducing a dynamic weight generation mechanism and an expert routing strategy, so that the convolution kernel parameters are adaptively adjusted according to the content of the input infrared image; Finally, the outputs of the two branches are spliced in the channel dimension to obtain the output of the thermal radiation path .

[0008] Further, the dynamic motion path is specifically configured to perform the following steps: First, a convolution layer is used to perform feature transformation and channel compression on to obtain the transformed feature The fine motion attention submodule is used to perform the following calibration processing on : ​​​​​ wherein denotes a convolution operation, is a Sigmoid activation function, is a ReLU activation function, denotes an element-wise multiplication, denotes the number of input channels, denotes the c-th channel of the feature, and denote the horizontal and vertical direction convolution kernels of the Sobel operator respectively, and denote the gradient components of the image in the horizontal and vertical directions respectively, and M is the gradient magnitude map, is a smoothing factor, is a motion attention map, and are two standard convolution layers, is the calibrated feature; After the calibrated feature is processed in the following manner, the motion feature of the target is obtained wherein denotes a convolution layer with a kernel size of 3x3 and a stride of 2, and BN denotes batch normalization.

[0009] Further, the feature fusion and calibration process specifically comprises: The features extracted from the double paths are spliced in the channel dimension to obtain the preliminary fusion feature , denotes a splicing operation; The fusion gate network is processed by a lightweight fusion gate network, which first generates a spatial weight map through a spatial attention branch, and generates a channel weight vector through a channel attention branch; the spatial attention branch processes through a 5x5 convolution and a Sigmoid function in turn; the channel attention branch processes using global average pooling and two 1x1 convolutions; According to the channel splicing when is channel sliced, and the sliced data is weighted and summed to obtain the final fusion feature​​​​​ ; The refined features are processed by a calibration convolution layer with an ECA attention mechanism to enhance the representation ability of the features. The refined features are processed by a calibration convolution layer with an ECA attention mechanism to enhance the representation ability of the features. The features processed by the residual path are element-wise added to the features processed by the calibration convolution layer to obtain infrared feature representations.

[0010] Further, the multi-granularity visual representation module comprises an edge guiding sub-module, a texture enhancement sub-module, a refined attention sub-module, and a structured feature reorganization and interaction sub-module. The edge guiding sub-module is configured to extract edge information from the smoothed down-sampled visible light feature tensor. ; The texture enhancement sub-module is configured to extract texture features of different scales through a multi-branch parallel structure under the guidance of the edge information. The texture features of different scales are spliced, and then compressed and nonlinearly transformed by a fusion convolution layer, and added to the features processed by the residual connection to obtain features after the residual connection. ; The refined attention sub-module is configured to perform spatial and channel dimension collaborative calibration on the features based on fusion weights between spatial and channel attention. to obtain calibrated features. ; The structured feature reorganization and interaction sub-module is configured to perform structured reorganization on the features, cross-project the features after the structured reorganization, and splice and fuse the features after the cross projection to obtain visible light feature representations.

[0011] Further, in the edge guiding sub-module: a gradient feature is extracted using a lightweight learnable edge detector. The lightweight learnable edge detector is , wherein represents the number of input channels, The convolution kernel weights of are initialized to a pattern with a negative center and a positive periphery to enhance the inductive bias of edge detection. is compressed into a channelized edge saliency map through a conversion layer composed of convolution and a Sigmoid function. ​​​​​​​​As a spatial mask, the smoothed downsampled visible light feature tensor is modulated using residual connections to sharpen the response of edge regions and obtain edge information. .

[0012] Furthermore, in the refined attention submodule: Through a lightweight gating network To learn the fusion weight G between spatial and channel attention, the Depend on Convolution and the Sigmoid function constitute: in This indicates a splicing operation. It is a spatial weighted graph. It is the channel weight vector; based on The features obtained after calibration of G : .

[0013] Furthermore, in the structured feature recombination and interaction submodule: Will It is divided into two parts in terms of channel dimension. and ; Components Through a step size of 2 Convolution performs downsampling, which reduces resolution and transforms features while preserving local structure, resulting in a feature map. ; Components Firstly, through a Max pooling reduces the resolution, then passes through a... Convolution is performed to project the channels, resulting in feature maps. ; Will and Perform cross projection as follows: in and These are two feature components resulting from interaction. It is Convolution, and this convolution is an identity channel mapping. This refers to the number of output channels; Will and The visible light feature representation is obtained by splicing and fusing through a group convolution layer.

[0014] Further, the collaborative weight decision module is configured to have two operation modes: In the first mode, the infrared feature representation and the visible light feature representation are integrated in a manner of average fusion. In the second mode, global average pooling is performed on each input feature map to compress the spatial dimension of the feature map to 1*1 to obtain a global feature vector of each feature map, then, the global feature vectors of all input feature maps are spliced, a light weight fully connected layer is used to predict the weight of each input feature map, the input feature maps are weighted and summed according to the predicted weight to obtain a fused feature map, finally, the fused feature map is processed through a shared convolution, the shared convolution is composed of two 1*1 convolutions, the first 1*1 convolution reduces the channel number to half of the original, and the second convolution restores the channel number to the original channel number.

[0015] Compared with the prior art, the present application has the beneficial effects: The present application proposes a visible-infrared target detection method based on structured feature extraction and dynamic weight prediction. In view of the sensitivity of infrared modal thermal radiation, a thermal radiation-motion feature extraction module is constructed, which significantly enhances the capture ability of target features in infrared images by optimizing the feature extraction process. At the same time, a multi-granularity visual representation module is proposed, which explicitly guides the network to focus on and fuse multi-granularity visual features from pixel-level edges to regional textures and global context, thereby constructing rich and discriminative visible light feature representation. A collaborative weight decision module with a parameter overhead comparable to Concat operation is proposed, which dynamically generates optimal fusion weights by analyzing the global context information of the input features, realizing data-driven precise integration of cross-modal features. The model achieves mAP0.5 indexes of 73.4%, 82.3% and 75.4% on VEDAI, DroneVehicle and FLIR datasets respectively. These results clearly show that the proposed target detection model has better performance in visible-infrared target detection tasks. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A target detection model network structure diagram of a visible-infrared target detection method based on structured feature extraction and dynamic weight prediction provided by the embodiment of the present application is provided. Figure 2 A conditional routing convolution (CRConv) network structure diagram provided by the embodiment of the present application is provided. Figure 3A network structure schematic diagram of a thermal radiation-motion feature extraction (TKFE) module provided for an embodiment of the present application is shown in FIG. 1. Figure 4 A network structure schematic diagram of a multi-granularity visual representation (MVR) module provided for an embodiment of the present application is shown in FIG. 2. Figure 5 A network structure schematic diagram of a collaborative weight decision (CWD) module provided for an embodiment of the present application is shown in FIG. 3. Figure 6 A visual detection result on a VEDAI dataset provided for an embodiment of the present application is shown in FIG. 4, in which the yellow box is a local enlarged image. Figure 7 A visual detection result on a Drone Vehicle dataset provided for an embodiment of the present application is shown in FIG. 5. Figure 8 A visual detection result on a FLIR Aligned dataset provided for an embodiment of the present application is shown in FIG. 6, in which the red circle represents a false positive, the yellow represents a false positive (FP) detection result, and the green represents a false negative (FN) detection result. DETAILED DESCRIPTION

[0017] The present application will be further explained in conjunction with the accompanying drawings and specific embodiments: A visible-infrared target detection method based on structured feature extraction and dynamic weight prediction, comprising: performing target detection through a visible-infrared target detection model (DFDNet) based on structured feature extraction and dynamic weight prediction; the target detection model comprises a thermal radiation-motion feature extraction module, a multi-granularity visual representation module, and a collaborative weight decision module; the thermal radiation-motion feature extraction module is used to capture thermal radiation information and motion features of a target in an infrared image, and then generate an infrared feature representation; the multi-granularity visual representation module is used to perform structured multi-granularity visual feature extraction on a visible light image, and then construct a visible light feature representation; the collaborative weight decision module is used to cross-integrate the infrared feature representation and the visible light feature representation through a dynamic weight prediction mechanism.

[0018] Specifically, after cross-integrating the infrared feature representation and the visible light feature representation through the collaborative weight decision module, it further comprises performing target detection on the cross-integrated features through a detector (Detector), and outputting the final bounding box and class information.

[0019] To fully exploit the complementary information of visible light and infrared modalities, the application carefully designs a double-branch special feature extraction structure. In view of the thermal radiation sensitivity of the infrared modality, the application designs a conditional routing convolution (CRConv), and builds a thermal radiation-motion feature extraction module (TKFE module) based on it. The module can effectively capture the thermal radiation information and the motion features of the target in the infrared image, and provide accurate feature representation for the target detection of the infrared modality. At the same time, in view of the texture contour advantage of the visible light modality, the application proposes a multi-granularity visual representation module (MVR module). The module can construct rich and discriminative visible light feature representation through structured multi-granularity visual feature extraction, and fully utilize the texture and contour information in the visible light image, laying a solid foundation for target detection of the visible light modality. In terms of cross-modality fusion, the application designs a collaborative weight decision module (CWD module). The module realizes the efficient integration of complementary features of the double modalities (i.e. infrared modality and visible light modality) through a dynamic weight prediction mechanism with a parameter amount overhead comparable to that of the Concat operation. The dynamic weight prediction mechanism can adaptively adjust the weight according to the input features, so that the model can better balance the feature contribution of the visible light and infrared modalities, thereby improving the feature expression ability after fusion and enhancing the detection performance of the model for different modal targets. Figure 1 shows the overall network structure of DFDNet.

[0020] Specifically, the conditional routing convolution (Conditional Routing Conv2d, CRConv) is as shown in Figure 2 In the infrared image, the thermal radiation features of the target often show non-rigid changes, different intensities and are easily disturbed by environmental thermal noise, etc., which makes the static convolution kernel of the traditional convolutional neural network have inherent limitations in dealing with such complex and variable input patterns. The fixed and unchanged convolution kernel parameters are difficult to adaptively capture the feature variation of different targets in different thermodynamic states, and cannot accurately suppress the thermal noise in the complex background, thereby causing the discriminability of feature extraction to decrease. To cope with the above-mentioned core challenges in infrared image analysis, the application designs a conditional routing convolution (CRConv). The convolution realizes the adaptive adjustment of the convolution kernel parameters to the content of the input infrared image by introducing a dynamic weight generation mechanism and an expert routing strategy, so that it can dynamically focus on different thermal targets and patterns, thereby significantly improving the ability to capture the diversity of infrared features.

[0021] Given the input feature map , where is the batch size, is the number of input channels, and are the height and width respectively. In the conditional routing convolution, the global representation is first extracted through a spatial compression operation: (1) where is the pooling size (default value is 4), is the compressed feature vector.

[0022] The routing network composed of two fully connected layers generates the expert weights: (2) (3) (4) where , is the learnable parameter, is the ReLU activation function, is the number of experts (default is 4), is the normalized expert weight matrix.

[0023] The multiple expert convolution kernels are dynamically fused according to the expert weights: (5) where represents the convolution kernel parameters of the th expert.

[0024] The convolution operation can be represented as: (6) The convolution output is processed by batch normalization and parameterized activation function to obtain the final processed features of the conditional routing convolution: (7) (8) where and are learnable scaling and offset parameters, the activation function is defined as , is the function.

[0025] (1) TKFE module In order to fully exploit the inherent thermal radiation characteristics and potential motion information in infrared images, and overcome the defects of the existing homogenization feature extractor in modeling modal specificity, the present application constructs a TKFE module. The core idea of this module is to process and fuse features through a double-path parallel processing and feature fusion and calibration mechanism, respectively focusing on the robust thermal radiation representation and dynamic changing motion clues in infrared images, and finally integrating them to generate more discriminative infrared feature representation.

[0026] The structure of the TKFE module is as follows:Figure 3 Figure 1 shows the architecture of the proposed infrared feature extractor. Given an input infrared feature tensor To ensure the stability of gradient flow and alleviate the degradation problem that deep networks may bring, the module initializes a residual path. This path projects the input to the target dimension through a 1x1 convolution and batch normalization operation, and uses an identity mapping if the input and output channel numbers are the same. Its calculation process is shown in equation (9): (9) 1. Thermal radiation path: The thermal radiation information in the infrared feature is obtained using the thermal radiation path. First, the input is average-pooled to smooth the features and expand the receptive field: (10) At the same time, the channel attention weight is calculated, which is generated through global average pooling and two fully connected layers to calibrate the importance of each channel: (11) (12) where , is the weight of the convolution layer, is the ReLU activation function, is the Sigmoid activation function. The pooled features are multiplied by the attention weight to obtain the calibrated features: (13) Next, is divided into two parts in the channel dimension. The two parts are processed through different down-sampling branches: (14) (15) Finally, the outputs of the two branches are concatenated in the channel dimension to obtain the output of the thermal radiation path: (16) 2. Dynamic motion path: To explicitly capture the small apparent changes in the infrared image due to target or camera motion. The module needs to go through the dynamic motion path, which first performs preliminary feature transformation and channel compression through a convolution layer: (17) Refined Motion Attention (RMAtt). RMAtt generates a saliency motion energy map by calculating the gradient magnitude of the feature, which guides the network to focus on the motion region. The mathematical expression is: (18) (19) (20) (21) (22) where, denotes the convolution operation, is the Sigmoid activation function, denotes element-wise multiplication, denotes the number of input channels, denotes the feature of the c-th channel of , and denote the horizontal and vertical direction convolution kernels of the Sobel operator, and denote the gradient components in the horizontal and vertical directions of the image, and M is the gradient magnitude map, is the smoothing factor, is the motion attention map, and are two standard convolution layers, is the calibrated feature. After the RMAtt calibration, the feature is further extracted by the subsequent convolution layer to extract the motion information and downsample: (23) where denotes a convolution layer with a kernel size of 3x3 and a stride of 2, and BN denotes batch normalization.

[0027] 3. Feature Fusion and Calibration: The features extracted from the dual path and are concatenated in the channel dimension to obtain the preliminary fusion feature = . Subsequently, a lightweight fusion gating network is used to generate the spatial and channel joint attention weight. The network first generates a spatial weight map Meanwhile, a channel attention branch (using global average pooling and two 1x1 convolutions) is used to generate a channel weight vector .

[0028] (24) (25) After the fusion feature is modulated by the dual attention, the dual-path information is integrated by channel slicing and a weighted summation, where the thermal path weight is and the dynamic path weight is to slightly emphasize the dynamic information.

[0029] (26) Finally, a calibration convolution layer containing an ECA attention (sub) module is used to further refine the fused feature to enhance the representation ability of the feature.

[0030] (27) The refined feature is combined with the residual feature to obtain the final output of the module: (28) Through its internal careful design, the module realizes the deep and adaptive extraction and fusion of the thermal radiation properties and motion properties of the infrared image, providing a high-quality and high-discriminative infrared feature basis for subsequent cross-modal fusion.

[0031] (2) MVR module The essential information of the visible light image is contained in the multi-scale and multi-granularity visual patterns from fine edges to complex textures. To overcome the shortcomings of traditional homogeneous feature extractors in representing such hierarchical information, the present invention proposes a multi-granularity visual representation module (MVR module). This module aims to simulate the processing process of the human visual system, explicitly guiding the network to focus on and fuse multi-granularity visual features from pixel-level edges to region-level textures, and then to global context through a structured processing link, thereby constructing an extremely rich and discriminative visible light feature representation.

[0032] The structure diagram of the MVR module is shown in Figure 4 , given the input visible light feature tensor , first, to stabilize the gradient calculation and expand the receptive field, the input feature is first smoothed down-sampled: (29) 1. Pixel-level edge perception This module focuses on the finest-grained edge and contour information, which is the basis of object localization. This function is realized by the edge-guided sub-module. Gradient features are extracted using a lightweight learnable edge detector ( ). The weights of this convolution kernel are initialized to a pattern with a negative center and a positive periphery to enhance its inductive bias for edge detection: (30) Finally, the high-dimensional gradient features are compressed into a channelized edge saliency map by a transformation layer composed of convolution and Sigmoid function: (31) The generated edge map is used as a spatial mask to modulate the original features through a residual connection, sharpening the response in edge regions: (32) where denotes element-wise multiplication. This step ensures that subsequent processing can be performed in a feature space where edge information has been enhanced.

[0033] 2. Region-level texture enhancement Under the guidance of edge information, the module further extracts more complex texture patterns. This function is realized by the texture enhancement sub-module, which adopts a multi-branch parallel structure to capture texture information at different scales. Given the input , two parallel convolution branches extract texture features under specific receptive fields: (33) After the multi-scale texture features are concatenated, a fusion convolution layer ( convolution) is used for feature compression and nonlinear transformation, and the original input is added through a residual connection to effectively avoid the degradation of detailed information: (34) This design ensures the richness of texture information and enhances the model's ability to represent complex visible light patterns.

[0034] 3. Global context attention filtering To adaptively filter the most relevant multi-granularity features for the task, the module introduces a refined attention sub-module to collaboratively calibrate the spatial and channel dimensions of the features. The feature maps obtain corresponding weights through spatial attention and channel attention, respectively. To avoid information conflicts that may be caused by simple attention multiplication, we design a lightweight gating network (consisting of ​(Convolution and the sigmoid function are used to learn the fusion weights between spatial and channel attention:) (35) Finally, the calibrated features are calculated using the following formula: (36) This gating mechanism enables adaptive fusion of spatial and channel attention, resulting in more refined feature filtering.

[0035] 4. Structured Feature Reorganization and Interaction Submodule To preserve multi-granularity information to the maximum extent during downsampling, the module performs structured reorganization of the features. The calibrated features are then... It is divided into two parts in terms of channel dimension. and Quantity Through a step size of 2 Convolution performs downsampling, preserving local structure while reducing resolution and transforming features: (37) Quantity Firstly, through a Max pooling layer rapidly reduces resolution, then passes through a... Convolution performs channel projection, aiming to efficiently preserve broader contextual information: (38) To promote complementarity between information of different granularities, a path interaction submodule is used for cross-projection: (39) (40) in It is Convolution, and this convolution is an identity channel mapping. This is the number of output channels.

[0036] This operation allows detailed information and contextual information to enhance each other, forming a unified and powerful multi-granular representation. Finally, the two interacting feature components are concatenated and efficiently fused through a grouped convolutional layer to generate the module's final output. This operation further enhances the uniformity and non-linear expressive power of the features.

[0037] (41) Output It is a refined feature representation that deeply integrates multi-granular visual information from pixel-level edges to global context, providing the optimal visible light feature foundation for subsequent cross-modal fusion.

[0038] (3) CWD module To realize adaptive feature fusion between visible light and infrared modalities, and overcome the problem of modal conflict or information flooding caused by simple operations (such as addition or splicing), the present application proposes a collaborative weight decision module, the module structure diagram is shown in Figure 5 The core of the module is a lightweight weight predictor, which dynamically generates the optimal fusion weight by analyzing the global context information of the input features, and realizes the precise integration of cross-modal features driven by data. Compared with the conventional channel splicing operation followed by a convolution layer, the CWD module introduces strong adaptive ability while almost introducing no additional parameter overhead. Specifically, in the first few epochs of training (i.e. the warmup stage), in order to provide a stable initial optimization environment for the backbone feature extraction network and avoid immature weight prediction in the initial stage from interfering with gradient backpropagation, the module will use simple average fusion. If not in the warmup stage, the module will perform global average pooling on each input feature map, compressing the spatial dimension of the feature map to 1x1 to obtain the global feature vector of each feature map. Then, the global feature vectors of all input feature maps are spliced together, and a lightweight fully connected layer is used to predict the weight of each input feature map. According to the predicted weight, the input feature maps are weighted and summed to obtain the fused feature map. Finally, the fused feature map is further processed through a shared convolution, which consists of two 1x1 convolutions. The first 1x1 convolution reduces the number of channels to half of the original, and the second convolution restores the number of channels to the original to enhance the representation ability of the features.

[0039] Experimental results All experiments in the present application are implemented on Pytorch, and for VEDAI and FLIR Aligned datasets, run on an NVIDIA GeForce RTX 5090 D GPU, with Windows 10 as the operating system, CUDA version 12.9, and torch version torch-2.7.1+cu128. For the DroneVehicle dataset, run on an NVIDIA GeForce RTX3070 GPU, with CUDA version 11.8, and torch version torch-2.5.0+cu121. The present application selects the stochastic gradient descent (SGD) optimizer with a momentum of 0.937, and in the experiment, the learning rate is 0.01, and the weight decay is 0.0005. The training round is set to 600 cycles for the VEDAI dataset, and 300 cycles for the DroneVehicle and FLIR Aligned datasets. The batch size and thread size are both set to 8. In order to make the model more stable in the early stage of training and reach the optimal solution faster in subsequent training, the present application sets the warm-up round to 3, and the warm-up momentum to 0.8.

[0040] The present application performs overall ablation experiments based on the mid-fusion model in YOLOv11-RGBT, and the results are shown in Table 1. When the multi-granularity visual representation module works independently (II), the recall rate is increased to 73.0% (3.2% higher than the baseline of 69.8%), which proves that it significantly reduces low-light missed detection through texture enhancement. When the thermal radiation-motion feature extraction module works independently (III), the precision rate is increased to 62.3% (6.3% higher than the baseline of 56.0%), which indicates that its thermal noise suppression capability effectively reduces false detection. When the collaborative weight decision module works independently (IV), the increase in precision rate and mAP value indicates the basic fusion capability, but the sharp drop in recall rate of 59.2% (-10.6%) reveals that unoptimized features cause semantic deviation. Direct combination of the thermal radiation-motion feature extraction module and the multi-granularity visual representation module (V) results in a sharp drop in mAP0.5 to 67.1% (precision rate 58.9%) due to unaligned feature conflicts, exposing the modal gap. Only the thermal radiation-motion feature extraction module + collaborative weight decision module (VI) causes small target performance collapse (mAP0.5:0.95 47.7%) due to the lack of visible light texture correction, and only the multi-granularity visual representation module + collaborative weight decision module (VII) limits the precision rate (68.2%) due to the lack of thermal noise purification. Finally, the three-module collaboration (VIII) eliminates feature deviation and compensates for modal defects, achieving an optimal mAP0.5 of 73.4% and an optimal mAP0.5:0.95 of 50.5%.

[0041] Table 1 Overall ablation performance comparison on the VEDAI dataset To verify the effectiveness of the proposed DFDNet model, the present application compares it with various advanced target detection methods on the VEDAI dataset. As shown in Table 2, the present application method achieves 73.4% in the mAP0.5 index, significantly better than other comparison methods, showing excellent comprehensive detection performance. From the detection results of each category, the present application method performs particularly outstanding in the Tractor category, reaching a detection accuracy of 90.5%, which is 12.34 percentage points higher than the optimal comparison method MINIAOD (78.16%), which is due to the enhanced extraction capability of the differential perception backbone network for thermal radiation features in infrared images. In the Camping category, the present application method achieves a detection accuracy of 84.9%, which is 8.1 percentage points higher than ACDF-YOLO (76.80%), indicating the effectiveness of the multi-granularity visual representation module in feature extraction of complex texture targets. It is worth noting that although the present application method (58.3%) is lower than ACDF-YOLO (80.74%) in the Truck category, it maintains a leading or comparable performance level in the remaining five categories, especially in small targets and complex background scenes. Compared with MINIAOD (70.63%) based on attention mechanism and AMEA-YOLO (66.20%) that fuses multi-scale features, the present application method significantly improves the detection accuracy. Observing Figure 6 the results of the visualization, DFDNet produces fewer false positives and false negatives.

[0042] Table 2 Comparison results on the VEDAI dataset To verify the generalization ability of the DFDNet model, the present application performs a comparison experiment on the DroneVehicle dataset, and the detailed results are shown in Table 3. The experimental results show that the present application method achieves 82.3% in the mAP0.5 index, surpassing all comparison methods, showing excellent cross-dataset generalization ability. Figure 7 Some visualization results are shown, and it is found that in daytime scenarios, DFDNet has a lower false positive rate and can identify occluded targets at the edges. When the light is insufficient or densely arranged, DFDNet has a lower miss detection rate.

[0043] Table 3 Performance comparison on the DroneVehicle dataset To comprehensively evaluate the performance and generalization ability of the DFDNet model, the present application extends the experiment to natural scene images to verify the adaptability and robustness of the model under different shooting scenes, and the results are shown in Table 4. The advantages and disadvantages of the model are found. The present application uses the visible-infrared image pairs of the FLIR Aligned dataset for this extended experiment, and selects a number of relatively classic models and representative target detection models in 2024 and 2025 for comparison. These models cover a variety of architectures and methods, and have strong comparability. Figure 8 The visualization results on the FLIR Aligned dataset show that DFDNet performs better whether it is daytime or nighttime.

[0044] Table 4 Performance comparison on the FLIR Aligned dataset In summary: 1) In order to improve the target detection performance in visible-infrared images, we propose a visible-infrared target detection model based on structured feature extraction and dynamic weight prediction, named DFDNet.

[0045] 2) To address the sensitivity of infrared modal thermal radiation, we design a conditional routing convolution and build a thermal radiation-motion feature extraction module based on it. This module focuses on robust thermal radiation representation and dynamic motion cues in infrared images through dual-path parallel processing, and finally integrates them through a fusion strategy to generate more discriminative infrared feature representation.

[0046] 3) We propose a multi-granularity visual representation module that aims to simulate the processing flow of the human visual system. Through a structured processing link, the network is explicitly guided to focus on and fuse multi-granularity visual features from pixel-level edges to regional-level textures, and finally to global context, thereby constructing rich and discriminative visible light feature representation.

[0047] 4) We propose a collaborative weight decision module that dynamically generates optimal fusion weights by analyzing the global context information of the input features, and achieves data-driven cross-modal feature precise integration with equivalent parameter overhead of Concat operation.

[0048] 5) According to the experimental results, the overall performance of the DFDNet model designed by us is better than other advanced methods, and it can more accurately detect targets in visible-infrared images.

[0049] The above merely shows the preferred embodiments of the present application, and it should be noted that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A visible-infrared target detection method based on structured feature extraction and dynamic weight prediction, characterized in that, include: Target detection is performed using a visible-infrared target detection model based on structured feature extraction and dynamic weight prediction. The target detection model includes a thermal radiation-motion feature extraction module, a multi-granularity visual representation module, and a collaborative weight decision module. The thermal radiation-motion feature extraction module is used to capture thermal radiation information and target motion features in infrared images, thereby generating infrared feature representations. The multi-granularity visual representation module is used to perform structured multi-granularity visual feature extraction on visible light images, thereby constructing visible light feature representations. The collaborative weight decision module is used to cross-integrate infrared feature representation and visible light feature representation through a dynamic weight prediction mechanism.

2. The visible-infrared target detection method based on structured feature extraction and dynamic weight prediction according to claim 1, characterized in that, The thermal radiation-motion feature extraction module includes a residual path, a thermal radiation path, a dynamic motion path, and a feature fusion and calibration process. The residual path converts the input infrared feature tensor into a 1×1 convolution and a batch normalization operation. Projecting onto the target dimension, if the number of input and output channels is the same, an identity mapping is used to obtain the features after residual path processing. ; The thermal radiation path is used to acquire thermal radiation information from infrared images. ; The dynamic motion path is used to capture the motion features of targets in infrared images. ; The feature fusion and calibration process is used to... , and The infrared features are then fused and calibrated to obtain the representation.

3. The visible-infrared target detection method based on structured feature extraction and dynamic weight prediction according to claim 2, characterized in that, The thermal radiation path is specifically used to perform the following steps: right Average pooling is performed to smooth features and expand the receptive field; Simultaneously, channel attention weights are calculated, which are generated through global average pooling and two fully connected layers to calibrate the importance of each channel; The pooled features are multiplied by the channel attention weights to obtain the calibrated features. ; Next, It is divided into two parts in terms of channel dimension. and The two parts are processed by two different downsampling branches; among which After processing by conditional routing convolution, the convolution kernel parameters are adaptively adjusted according to the content of the input infrared image by introducing a dynamic weight generation mechanism and an expert routing strategy, thereby dynamically focusing on different thermal targets. Finally, the outputs of the two branches are concatenated along the channel dimension to obtain the output of the thermal radiation path. .

4. The visible-infrared target detection method based on structured feature extraction and dynamic weight prediction according to claim 2, characterized in that, The dynamic motion path is specifically used to perform the following steps: First, a convolutional layer is used... Perform feature transformation and channel compression to obtain the transformed features. ; Through the refined motion attention submodule Perform the following calibration process: in This represents the convolution operation. It is the Sigmoid activation function. It is the ReLU activation function. This represents element-wise multiplication. Indicates the number of input channels. express The characteristics of the c-th channel, and These represent the horizontal and vertical convolution kernels of the Sobel operator, respectively. and These represent the gradient components of the image in the horizontal and vertical directions, respectively, and M is the gradient magnitude map. It is a smoothing factor. It is a motion attention map. and They are two standard convolutional layers. These are the calibrated features; The calibrated features are then processed in the following manner to obtain the target's motion features. : in This represents a convolutional layer with a kernel size of 3×3 and a stride of 2. BN represents batch normalization.

5. The visible-infrared target detection method based on structured feature extraction and dynamic weight prediction according to claim 2, characterized in that, The feature fusion and calibration process specifically includes: Features extracted from dual paths and By splicing along the channel dimension, preliminary fusion features are obtained. = , Indicates a splicing operation; Through lightweight converged gating networks To process the data, the lightweight fusion gating network first generates a spatial weight graph through a spatial attention branch. Simultaneously, a channel weight vector is generated through a channel attention branch. The spatial attention branch will The process is sequentially performed using a 5×5 convolution followed by a sigmoid function; the channel attention branch uses global average pooling and two 1×1 convolution pairs. Process it; right Channel slicing is performed according to the channels used during splicing, and the sliced ​​data is then weighted and summed to obtain the final fused features. ; Use a calibrated convolutional layer with an ECA attention mechanism Refinement is performed to enhance the characterization ability of the features; Refined characteristics Features after residual path processing Element-by-element addition yields the infrared feature representation.

6. The visible-infrared target detection method based on structured feature extraction and dynamic weight prediction according to claim 1, characterized in that, The multi-granularity visual representation module includes an edge guidance submodule, a texture enhancement submodule, a refined attention submodule, and a structured feature recombination and interaction submodule. The edge guidance submodule is used to extract edge information from the smoothed downsampled visible light feature tensor. ; The texture enhancement submodule is used for edge information Guided by [previous framework], texture features at different scales are extracted using a multi-branch parallel structure. These features are then concatenated and subjected to feature compression and nonlinear transformation via a fusion convolutional layer, and finally connected to [other components] through residual connections. Add them together to obtain the features after residual connection. ; The refined attention submodule is used for fusion weights based on spatial and channel attention. right Perform co-calibration of spatial and channel dimensions to obtain calibrated features. ; The structured feature recombination and interaction submodule is used for... The structured features are recombined, cross-projected, and then stitched and fused to obtain a visible light feature representation.

7. The visible-infrared target detection method based on structured feature extraction and dynamic weight prediction according to claim 6, characterized in that, In the edge guidance submodule: Gradient features are extracted using a lightweight, learnable edge detector. The lightweight, learnable edge detector is ,in Indicates the number of input channels. The convolutional kernel weights are initialized to a pattern similar to the Sobel operator, with negative weights at the center and positive weights at the periphery, to enhance the inductive bias of edge detection. Through a The transformation layer, composed of convolution and the sigmoid function, will... Compressed into a channelized edge saliency map ; Will As a spatial mask, the smoothed downsampled visible light feature tensor is modulated using residual connections to sharpen the response of edge regions and obtain edge information. .

8. The visible-infrared target detection method based on structured feature extraction and dynamic weight prediction according to claim 6, characterized in that, In the refined attention submodule: Through a lightweight gating network To learn the fusion weight G between spatial and channel attention, the Depend on Convolution and the Sigmoid function constitute: in This indicates a splicing operation. It is a spatial weighted graph. It is the channel weight vector; based on The features obtained after calibration of G : 。 9. The visible-infrared target detection method based on structured feature extraction and dynamic weight prediction according to claim 6, characterized in that, In the structured feature recombination and interaction submodule: Will It is divided into two parts in terms of channel dimension. and ; Components Through a step size of 2 Convolution performs downsampling, which reduces resolution and transforms features while preserving local structure, resulting in a feature map. ; Components Firstly, through a Max pooling reduces the resolution, then passes through a... Convolution is performed to project the channels, resulting in feature maps. ; Will and Perform cross-projection as follows: in and These are two feature components resulting from interaction. It is Convolution, and this convolution is an identity channel mapping. This refers to the number of output channels; Will and The data are spliced ​​together and fused through a grouped convolutional layer to obtain a visible light feature representation.

10. The visible-infrared target detection method based on structured feature extraction and dynamic weight prediction according to claim 1, characterized in that, The collaborative weight decision module is configured to have two operating modes: In the first mode, the infrared feature representation and the visible light feature representation are integrated using an average fusion method; In the second mode, global average pooling is performed on each input feature map to compress the spatial dimension of the feature map to 1×1, resulting in a global feature vector for each feature map. Subsequently, the global feature vectors of all input feature maps are concatenated, and the weights of each input feature map are predicted through a lightweight fully connected layer. The input feature maps are then weighted and summed according to the predicted weights to obtain a fused feature map. Finally, the fused feature map is processed through a shared convolution, which consists of two 1×1 convolutions. The first 1×1 convolution reduces the number of channels to half of the original number, and the second convolution restores the number of channels to the original number.

Images