Visible light and infrared image bimodal target detection method

Abstract

The invention provides a visible light-infrared image bimodal target detection method and device, and belongs to the technical field of computer vision and deep learning, and the method comprises the steps: obtaining a bimodal alignment image data set, building a bimodal image data set, constructing a symmetric double-flow feature extraction network, and carrying out the multi-stage cross-modal information conduction. And executing dynamic weight distribution fusion, and finally generating a target detection result. According to the invention, gradient dependence of visible light and infrared double branches is decoupled; through staged freezing and activation, the model is forced to fully mine specific features in a single mode, so that the problem of insufficient extraction of secondary mode features caused by excessive dependence of a network on a dominant mode is avoided; multi-scale and progressive deep fusion of cross-modal information is realized, complementary advantages of features are enhanced, and the method has engineering practicability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, specifically relating to a method and apparatus for dual-modal target detection in visible light and infrared images based on symmetrical bidirectional alternating activation and dynamic weight fusion. Background Technology

[0002] Current target detection primarily utilizes visible light images, but the imaging quality of visible light sensors deteriorates significantly under complex lighting conditions such as nighttime, low illumination, or severe weather. Infrared images capture target thermal radiation information and are unaffected by lighting conditions, but lack texture details. Effectively fusing complementary information from two modalities has become crucial for improving the robustness of all-weather detection. Among existing technologies, Zhao et al. (Exchanging dual-encoder–decoder: A new strategy for changedetection with semantic guidance and spatial localization[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-16.) promoted modal fusion through a temporal fusion attention mechanism. Some studies have explored strategies based on frozen parameters. For example, Wan et al. (Towards a Comprehensive Single-Stage Multispectral Object Detection Framework[J]. arXiv preprint arXiv:2506.14696, 2025.) proposed a multispectral controllable fine-tuning strategy, using loaded visible light image features to fine-tune the base model to enhance it. There are also physical model-based illumination enhancement methods, such as CN117809202A, "A Dual-Modal Target Detection Method and System." This method targets remote sensing images and uses the Phong illumination model and the IEMBP model to extract illumination features from non-visible light data to enhance the depth features of the visible light portion, thus addressing uneven illumination and shadow issues. Additionally, there are network architectures based on attention mechanisms and denoising, such as patent CN202310321310, "A Dual-Modal Target Detection Model and Method Based on Residual Shrinkage Attention Network." These methods introduce residual shrinkage modules and attention mechanisms to eliminate infrared image noise and focus on key areas, improving feature quality through complex network modules.

[0003] The method proposed by Zhao et al. belongs to the traditional synchronous training strategy, which is prone to intermodal information interference and amplifies background redundancy. The method proposed by Wan et al. is an asymmetric strategy with static guidance. The freezing strategy restricts the comprehensive bidirectional interaction of information, making it difficult to adapt to dynamic scenarios and prone to "shortcut learning." This causes the model to over-rely on the dominant modality, resulting in insufficient feature extraction of the inferior branch. Whether based on residual shrinking networks (such as CN202310321310) or adaptive fusion (such as CN202410687752), most mainstream methods currently adopt synchronous training strategies. During backpropagation, the network tends to over-rely on the dominant modality (usually RGB), which has a large amount of information and is easy to learn, causing the gradient update of the other modality (IR) to be hindered, resulting in the "shortcut learning" phenomenon. Simply improving the network structure (such as adding attention or denoising modules) cannot fundamentally solve the problem of insufficient modality learning caused by this training mechanism.

[0004] In summary, existing bimodal target detection methods still face problems such as severe intermodal information interference, limited bidirectional interaction, and difficulty in balancing training stability and feature complementarity during feature fusion. Summary of the Invention

[0005] To overcome the problems of insufficient dual-modal feature fusion, intermodal information interference, and "shortcut learning" during training in existing technologies, this invention proposes a dual-modal target detection method for visible-infrared images based on symmetric bidirectional alternating activation and dynamic weight fusion. This method, through a staged activation strategy and dynamic weight allocation mechanism, enables the model to adaptively extract and fuse complementary features from both modalities in different lighting and environmental scenarios.

[0006] The basic principle of this invention is as follows: The baseline method YOLO11s is an efficient single-stage object detection algorithm, whose structure mainly consists of three parts: a backbone network, a feature fusion network (Neck), and a prediction head (Head). However, the original YOLO11s only supports single-modal image input, and its ability to mine infrared image features is insufficient in complex lighting environments (such as low light at night or strong light overexposure), making it difficult to utilize the complementary advantages of dual-modal information, resulting in limited detection performance and narrow application scenarios. Therefore, this invention makes in-depth improvements based on YOLO11s. Specifically, this invention constructs a symmetric dual-stream backbone network, decouples gradient dependence through alternating activation modules, and embeds gated variable gradient propagation convolutional modules in the multi-level feature layers of the backbone network to enhance cross-modal information propagation. Finally, the enhanced features, adaptively weighted by the dynamic weight allocation fusion module, are input into the detection network, enabling in-depth utilization of complementary features of visible light and infrared images, thereby obtaining more robust detection results in complex environments.

[0007] This invention discloses a method for dual-modal target detection using visible light and infrared images, comprising:

[0008] Step A: Obtain a bimodal aligned image dataset, i.e., acquire images of the same scene using infrared and visible light imaging devices, perform spatial registration, and construct a bimodal dataset containing visible light and infrared images.

[0009] Step B: Establish a bimodal image dataset, that is, perform data augmentation processing on the registered image dataset.

[0010] Step C: Construct a symmetric dual-stream feature extraction network, which involves building parallel backbone network branches using alternating activation modules and alternately activating gradient updates of different modal branches through a three-stage training strategy. This guides the dual-branch network to collaborate between modalities and extract high-quality modal features.

[0011] Step D: Implement multi-level cross-modal information transmission, that is, use gated variable gradient transmission convolution modules on the multi-level feature layers of the backbone network, and use learnable depth scanning factors to dynamically adjust the weights to achieve progressive injection and enhancement of cross-modal information from shallow to deep layers.

[0012] Step E: Perform dynamic weight allocation fusion, that is, use the dynamic weight allocation fusion module to regress the modality adjustment factor according to the input feature distribution to achieve adaptive weighting and deep interactive fusion of visible light and infrared features.

[0013] Step F: Generate target detection results, that is, output the target's category label and location prediction information.

[0014] This invention also discloses a visible-infrared dual-modal target detection device, comprising: a dual-mode image acquisition unit, a calculation unit, and a display unit. The dual-mode image acquisition unit acquires infrared and visible light images and then sends them to the calculation unit. The calculation unit performs dual-mode image recognition and then sends the results to the display unit. The dual-mode image acquisition unit includes a sealed enclosure, a beam splitter, an infrared camera, and a visible light camera. A light-collecting port is opened on the lower part of one side of the sealed enclosure. The beam splitter is fixedly installed on the bottom surface of the sealed enclosure. The infrared camera is connected to a fixed bracket of the sealed enclosure, pointing vertically downwards, with the infrared camera lens directly facing the center of the beam splitter. The visible light camera is installed to the side of the infrared camera. The calculation unit is configured with an implementation program corresponding to the method of this invention. The display unit displays the final results.

[0015] This invention employs a three-stage training strategy using symmetrical bidirectional alternating activation modules, successfully decoupling the gradient dependency between the visible light and infrared branches. Compared to traditional synchronous training, this method forces the model to fully exploit specific features within a single modality through phased freezing and activation, avoiding the problem of insufficient extraction of secondary modal features caused by excessive reliance on the dominant modality (e.g., visible light during the day and infrared at night). This invention achieves multi-scale, progressive deep fusion of cross-modal information, enhancing the complementary advantages of features and possessing engineering practicality. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention;

[0017] Figure 2 This is a diagram of the dual-stream alternating activation feature extraction network structure in this invention;

[0018] Figure 3 This is a structural diagram of the gated variable gradient propagation convolution module in this invention;

[0019] Figure 4 This is a structural diagram of the dynamic weight allocation module in this invention;

[0020] Figure 5 This is a structural diagram of the visible light-infrared image dual-modal target detection device of the present invention;

[0021] Figure 6 This invention compares the accuracy and speed of the method described in this paper with the YOLO11s method.

[0022] Figure 7 This is a visual comparison chart of the method of this invention and the YOLO11s method;

[0023] Explanation of reference numerals in the attached figures:

[0024] 1. Dual-mode image acquisition unit; 2. Calculation unit; 3. Display unit; 11. Sealed enclosure; 12. Beam splitter; 13. Infrared camera; 14. Visible light camera; 111. Light-collecting port. Detailed Implementation

[0025] In one specific embodiment of the present invention, such as Figure 1 As shown, the steps are as follows:

[0026] Step A: Obtain a bimodal aligned image dataset. This involves acquiring images of the same scene using infrared and visible light imaging devices, performing spatial registration, and constructing a bimodal dataset containing both visible light and infrared images.

[0027] Step A includes:

[0028] Step A1: Acquire infrared and visible light images using a visible light camera and an infrared camera. The visible light camera is a PointGrey Flea3, and the infrared camera is a FLIR-A35 with a spatial resolution of 640×480 pixels and a vertical field of view of 103.6 degrees. The thermal infrared camera has a spatial resolution of 320×256 pixels and a vertical field of view of 39 degrees.

[0029] Step A2: Perform image registration on the acquired infrared and visible light images by adopting an alignment strategy and adjusting the image size to achieve image registration.

[0030] Step B: Establish a bimodal image dataset, that is, perform data augmentation processing on the registered image dataset to improve the robustness and generalization ability of the model in complex and changing environments.

[0031] Step B includes:

[0032] Step B1: Perform coarse-grained processing on the image pair dataset to obtain a coarse-grained image pair dataset. This is achieved using the PIL class in Python: edge enhancement, sharpening, smoothing, and detail enhancement.

[0033] Step B2: Perform automated quality assessment and screening on the preprocessed images in the dataset, and use the OpenCV class in Python to extract high-gain feature paired images.

[0034] Step B1 includes:

[0035] Step B1-1: Image Edge Enhancement. Using edge detection operators or gradient enhancement algorithms, the texture edges of the visible light image and the heat source contours of the infrared image are salientized to improve the model's sensitivity to target boundaries. Specifically, the Laplacian operator in OpenCV is used, with an operator aperture size set to 3 or 5. A larger aperture introduces noise, while a smaller aperture results in insignificant enhancement. The scaling factor (Scale) is set to 1.2 to 1.5. The extracted edge components are multiplied by this factor and then superimposed back onto the original image.

[0036] Step B1-2: Structural Sharpening. Contour filtering or high-pass filtering methods are used to extract and enhance the structural feature information in the image. By suppressing low-frequency background redundancy, the geometric shape information of the target is fully extracted. The Unsharp Masking (USM) sharpening algorithm is used, with the Gaussian kernel size set to (5,5); the standard deviation set to 1.0 to 1.5; the sharpening intensity set to 0.5 to 0.8; and the threshold set to 0 to 5.

[0037] Step B1-3: Adaptive Smoothing Processing. Using smoothing filters or denoising algorithms, the grayscale distribution of the image is made to gradually flatten out, reducing abrupt gradient changes while effectively suppressing imaging noise introduced by the sensor and improving the signal-to-noise ratio. A bilateral filter algorithm is used, with the neighborhood diameter (d) set to 5 or 9; the color space standard deviation is set to 50 to 75 (a larger value results in more colors being mixed together); the coordinate space standard deviation is also set to 50 to 75.

[0038] Steps B1-4: Multi-scale detail enhancement. A detail enhancement filtering method is employed to dynamically adjust local contrast, making subtle small targets or weak texture features in the image more apparent, providing high-quality input data for subsequent deep feature extraction. Contrast-limited adaptive histogram equalization (CLAHE) is used, with the contrast limit set between 2.0 and 4.0; the block size is set to (8,8), dividing the image into 8×8 regions for local processing.

[0039] Step B2 includes:

[0040] Step B2-1: Spatial Registration Verification. The statistical correlation between visible and infrared images in pixel spatial distribution is measured based on mutual information (MI). The geometric alignment accuracy of the two modes is verified by calculating the pixel overlap of the visible and infrared images in the same spatial coordinate system, eliminating significantly mismatched image pairs caused by camera displacement or jitter. Alignment accuracy is judged by measuring the density of the grayscale statistical distribution; the histogram statistical binning is set to 256, and the normalized mutual information (NMI) formula is used, with a threshold of MI > 0.8.

[0041] Step B2-2: Comprehensive Image Quality Assessment. Based on the average gradient (AG), which reflects the rate of change of subtle detail contrast in an image and represents its sharpness, contrast, signal-to-noise ratio (SNR), and sharpness thresholds are set. Automatic image detection is performed to filter out invalid samples where target features are completely invisible due to sensor overexposure, extreme low-light noise, or severe motion blur. Sharpness is determined by the gradient change rate, and the level of environmental interference is determined by the SNR. The average gradient (AG) operator selected is the 3×3 Sobel operator, and the visible light threshold AG is... rgb >5.0, Infrared threshold AG ir >3.5, signal-to-noise ratio Where μ is the signal mean, The background standard deviation is used as the judgment threshold: SNR>20dB.

[0042] Step B2-3: Feature Saliency and Complementarity Screening. Using the Tenengrad gradient function algorithm, the horizontal and vertical gradient values ​​Gx and Gy are extracted using the Sobel operator. The root mean square of the gradient across the entire image is calculated to ensure that the selected samples possess rich texture details in the visible light mode and clear heat source contours in the infrared mode, thus guaranteeing the complementary value of the dual-modal information. The sum of squared gradient moduli is used to quantify texture richness, and local contrast is used to quantify heat source saliency. Tenengrad gradient function configuration, Sobel kernel size: 3×3, Sobel operator, calculating the root mean square of the gradient across all pixels. Final score ,in This represents the height of the image, i.e., the number of pixels in the vertical direction. This refers to the width of the image, i.e., the number of pixels in the horizontal direction. The final score is used to quantify the information content of the image and serves as a "filtering criterion." Only when both the texture score of the visible light image and the heat source score of the infrared image exceed a certain threshold will the image be considered acceptable. Only then will the image be preserved.

[0043] Step C: Construct a symmetric dual-stream feature extraction network. This involves building parallel backbone network branches using alternating activation modules, and then using a three-stage training strategy to alternately activate gradient updates of different modal branches. This guides the dual-branch network to collaborate between modalities and extract high-quality modal features.

[0044] Step C includes:

[0045] Step C1: Construct a dual-stream alternating activation feature extraction network.

[0046] Step C2: Input the enhanced bimodal dataset obtained from B2 and set the three-stage training strategy.

[0047] Step C1 includes:

[0048] Step C1-1: Using YOLO11s as the baseline network topology, the YOLO11 backbone network consists of five progressive stages (P1-P5). Stages 1 (P1) and 2 (P2) rapidly reduce image resolution and extract preliminary local spatial features. Stages 3 (P3) and 4 (P4) further extract complex texture and semantic information. Stage 5 (P5), the final layer of the backbone network, uses a positional attention mechanism to weight the features of the entire image, allowing the model to focus on the most important target regions. This invention's algorithm constructs a visible light branch and an infrared branch in parallel on the single-branch YOLO11s backbone network. Figure 2As shown, gated convolutional propagation (GCC) modules are inserted into the last three feature extraction layers (P3, P4, P5) of the backbone network, and a dynamic weight allocation fusion (DWM) module is added to the last feature extraction layer. The frozen parameters of the backbone network are set for use in subsequent training. The network structure diagram is shown below. Figure 1 Specifically, RGB and IR images are fed into a dual-branch backbone network for feature extraction. After feature propagation at layers P3, P4, and P5, they are fed into the GCC (Graphical Convolutional Network) for complementary information from another modality. The dual branches then enter the DWM (Dual Width Model) module at layer P5 for feature fusion. The DWM module contains a weight allocator (WA) to adjust weights during different training processes. Cross-multiplication is then performed to achieve information complementarity. Finally, the concatenated images are fed into the detection head to obtain the detection result. The hyperparameters for network construction are set as follows: stochastic gradient descent (SGD) is used to iteratively update network parameters, with an initial learning rate of 0.01, a 3.0 epoch warm-up period, a warm-up momentum of 0.8, and a warm-up bias learning rate of 0.1. All image resolutions are uniformly adjusted to 640×640 when loading data.

[0049] Step C1-2: Constructing and connecting multi-level gated variable gradient propagation convolutional modules. Three gated variable gradient propagation convolutional modules are embedded in the last three feature extraction layers (P3, P4, P5) of the visible light and infrared branches, providing... Figure 2 The GCC section. The GCC structure is as follows: Figure 3 As shown, the GCC module performs feature transfer on the incoming visible light or infrared information through Conv(1×1), CGG(Conv+GN+GELU), Conv(3×3), GN,GELU, and Conv(1×1). Based on this, GateNet, with its gating mechanism, is introduced.

[0050] The core functionality of GateNet is to control the convolutional weights of modules, reducing weights in the early stages of feature extraction to maximize the transmission of information from other branches. Weights are gradually increased in deeper layers of the network to transmit information from deeper levels. Specifically, within the convolutional layers of a module, a gated network called GateNet with a scan factor α that can be learned to determine the network depth is introduced. By setting differentiated α values ​​for different layers (0.05 for P3, 0.1 for P4, and 0.2 for P5), convolutional weights are dynamically generated. This is used to adjust the weights of the convolutional kernels within the GCC module. The calculation is as follows:

[0051]

[0052] It is the sigmoid activation function. FC1 and FC2 are fully connected layers, which are dynamic scanning factors. FC1 is usually used for dimensionality reduction to reduce computation, while FC2 is used for dimensionality increase to restore the original number of channels. GAP is global average pooling. The input feature map.

[0053] The complete calculation results of the GCC module are as follows:

[0054]

[0055] This is the output of the GCC module, where Representing input features Dynamic convolution weights generated by the GateNet network The transformed feature map is calculated as follows:

[0056]

[0057] Step C1-3: Integration of the dynamic weight allocation fusion module. After the output of the last layer (P5) of the visible light branch and the infrared branch, the dynamic weight allocation module is set as the feature convergence center. The structure of the dynamic weight allocation module is as follows: Figure 4 As shown.

[0058] The dynamic weight allocation module first considers visible light features. and infrared features Perform global average pooling to compress and encode the spatial dimension of global context information into a one-dimensional channel descriptor. and Subsequently, to achieve dynamic balance between modalities, a weight adaptive allocation unit (WA) was designed. This unit can dynamically regress two modality adjustment factors based on the current input feature distribution. and Regulatory factors and The initial settings are all 0.5. Finally, through cross-modal information interaction, attention weights for feature recalibration are generated. and .

[0059] The calculation process is as follows:

[0060]

[0061]

[0062] in This is a dot product operation. This is a cross product operation.

[0063] To obtain a weight vector that reflects cross-modal global interaction information. and Then, these weights are mapped back to the original feature space. The original features of the visible light branch are then mapped back to the original feature space. injection Similarly, the original features of the infrared branch will also be... injection This ensures that the network can absorb cross-modal complementary information while preserving the original intrinsic feature distribution of the current modality without loss, thus maintaining the original feature information. Finally, it is added to the information of another modality to output visible light enhanced features with rich texture details and clear heat source outlines. and infrared enhancement features .

[0064] The calculation process is as follows:

[0065]

[0066]

[0067] Finally and Perform Concat concatenation processing to prepare for input detection header.

[0068] Step C1-4: Prediction Head Input and Output. The stitched results of visible light enhancement features and infrared enhancement features are sent to the detection head. Figure 1 (Detection Head). The detection head adopts a decoupled head structure, which performs attribute recognition and bounding box localization of the target through classification branch and regression branch respectively.

[0069] Step C2 includes:

[0070] Step C2-1: Input the high-gain feature pairing image obtained from B2 into the network established in C1.

[0071] Step C2-2: Set the three-stage training parameters from C1-1.

[0072] Step C2-2 includes:

[0073] Step C2-2-1: Stage 1, the visible light branch activates the infrared branch. The infrared branch parameter updates are frozen by setting all module parameter updates for the infrared branch to Freeze, and the infrared branch stops passing convolutional modules through gated variable gradients.

[0074] Step C2-2-2: Stage Two, Infrared branch activates visible light branch. Freeze visible light branch parameter updates, set all module parameter updates of the visible light branch to Freeze, and stop the visible light branch from passing convolution modules through gated variable gradients.

[0075] Step C2-2-3: Phase Three, Dual-branch Co-tuning. Unfreeze the dual-branch parameter updates, cancel the Freeze setting, and perform joint fine-tuning across the entire network.

[0076] Step D: Implement multi-level cross-modal information transmission. This involves using gated variable gradient propagation convolutional modules on the multi-level feature layers of the backbone network, and dynamically adjusting the weights using a learnable depth scanning factor to achieve progressive injection and enhancement of cross-modal information from shallow to deep layers.

[0077] Step D includes:

[0078] Step D1: Input the information from the last three feature extraction layers of the visible light branch and infrared branch of C2 into the gated variable gradient propagation convolution module.

[0079] Step D2: The information input in D1 is used by the C1-2 module to progressively inject and enhance cross-modal information from shallow to deep layers. In the shallow layer (P3), the GCC module ensures the transmission of original information and retains more feature information. In the deep layer (P5), the GCC module transmits more refined information.

[0080] Step E: Perform dynamic weight allocation fusion. Before the features from the last feature extraction layer, P5, enter the C1-4 detector head, the C1-3 dynamic weight allocation fusion module uses the modality adjustment factor to regress based on the input feature distribution, thereby achieving adaptive weighting and deep interactive fusion of visible light and infrared features.

[0081] Step E includes:

[0082] Step E1: Input the last feature extraction layer of C1 into the dynamic weight allocation fusion module to perform dual-modal feature fusion.

[0083] Step E2: The dynamic weight allocation fusion module dynamically adjusts the fusion weights between modes based on the current stage. It adjusts the WA allocation value in the DWM, allocating it in stage C2-2-1. The weight value is 0.7. The weight value is 0.3; allocated in stage C2-2-2. The weight value is 0.3. The weight is 0.7; allocated in stage C2-2-3. and All weights are 0.5.

[0084] Step F: Generate target detection results. Input the E1 result into the C1-4 detection head, which outputs the target's category label and location prediction information.

[0085] like Figure 5 As shown, this invention also discloses a visible-infrared dual-modal target detection device using the above-described method, comprising: a dual-mode image acquisition unit 1, a calculation unit 2, and a display unit 3. The dual-mode image acquisition unit 1 acquires infrared and visible light images, and then sends them to the calculation unit 2. The calculation unit 2 performs dual-mode image recognition and then sends the results to the display unit 3. The dual-mode image acquisition unit 1 includes a sealed enclosure 11, a beam splitter 12, an infrared camera 13, and a visible light camera 14. A light-collecting port 111 is opened on the lower part of one side of the sealed enclosure 11. The beam splitter 12 is fixedly installed on the bottom surface of the sealed enclosure 11. The infrared camera 13 is connected to the fixed bracket of the sealed enclosure 11, pointing vertically downwards, with the lens of the infrared camera 13 directly facing the center of the beam splitter 12. The visible light camera 14 is installed on the side of the infrared camera 14. The calculation unit 2 is configured with an implementation program corresponding to the method of this invention. The display unit 3 displays the final result.

[0086] The dual-mode image acquisition unit 1 includes a sealed enclosure 11, a beam splitter 12, an infrared camera 13, and a visible light camera 14.

[0087] The sealed enclosure 11 measures 80cm in length, 40cm in width, and 80cm in height. A circular light-receiving opening 111, 15cm in diameter, is located on the lower side of one side of the enclosure. A beam splitter 12 is fixedly mounted on the bottom surface of the enclosure 11, with its lower part near the light-receiving opening 111 at a 40-degree angle to the bottom surface. The beam splitter 12 is made of zinc-coated silicon wafer. An infrared camera 13 is connected to the mounting bracket of the enclosure 11, pointing vertically downwards, with its lens directly facing the center of the beam splitter 12. The infrared camera 13 is a FLIR-A35 model. A visible light camera 14 is also connected to the mounting bracket of the enclosure 11, with its lens directly facing the center of the beam splitter 12. The visible light camera 14 is a PointGrey Flea3 model. The computing unit 2 is a computer or an image processor FPGA, configured with the implementation program corresponding to the method of this invention. The display unit 3 is an image display U2422H.

[0088] To verify the effectiveness of the method of the present invention, this method was compared with the existing YOLO11s method, and the entire process was carried out in the same test environment.

[0089] The specific testing process for this invention is as follows:

[0090] First, the bimodal dataset images established using this method are input into a dual-stream alternating activation network to obtain detection accuracy and speed, and then a visualization result is output.

[0091] The specific process of the comparison method is as follows:

[0092] Visible light image comparison: using only the visible light images from the bimodal dataset established in this invention,

[0093] Inputting the data into the YOLO11s network yields the detection accuracy and speed, and the results are visualized.

[0094] Infrared image comparison: The infrared images in the dual-modal dataset established in this invention are used alone and input into the YOLO11s network to obtain the detection accuracy and speed, and output the visualization results.

[0095] The evaluation metrics for the experiment were: precision 0.5:0.95 and precision 0.5. Precision 0.5:0.95 represents the average precision calculated at intervals of 0.05 from the confidence threshold between 0.50 and 0.95, while precision 0.5 represents the precision when the confidence threshold is set to 0.5. The experimental results are compared as follows... Figure 6 As shown, the horizontal axis represents the comparison between this invention and YOLO11s, and the vertical axis represents the results of precision 0.5:0.95 and precision 0.5. The experimental results show that precision 0.5 improves upon visible light and infrared by 8.9% and 2.3% respectively, while precision 0.5:0.95 improves upon visible light and infrared by 12.7% and 2.6% respectively.

[0096] Visual comparison results as follows Figure 7 As shown in these comparison charts, Figure 7 (a) is the actual bounding box in the image. Figure 7 (b) shows the detection using YOLO11s, which found one false positive and one false negative, as indicated by arrows in the figure. Figure 7 (c) shows the detection results of this invention. The comparison results clearly show that the algorithm in this paper effectively eliminates false detections and false negatives generated by the YOLO11s method.

[0097] This invention addresses the problems of modal interference and insufficient information utilization in visible-infrared dual-modal detection by proposing a detection framework based on symmetric bidirectional alternating activation. The invention designs an alternating activation module, which effectively avoids shortcut learning in synchronous training by decoupling the gradient dependency of the two branches. Simultaneously, a gated transmission module is introduced to achieve progressive injection of cross-modal information from shallow to deep layers. Furthermore, in conjunction with a dynamic weight allocation fusion module, the model of this invention achieves adaptive weighting of complementary information during the feature fusion stage, demonstrating strong engineering practicality.

Claims

1. A method for dual-modal target detection using visible light and infrared imaging, comprising: Step A: Obtain a dual-modal aligned image dataset, that is, use infrared and visible light imaging devices to acquire images of the same scene, perform spatial registration, and construct a dual-modal dataset containing visible light images and infrared images; Step B: Establish a bimodal image dataset, that is, perform data augmentation processing on the registered images to the dataset; Step C: Construct a symmetric dual-stream feature extraction network, that is, use alternating activation modules to build parallel backbone network branches, and use a three-stage training strategy to alternately activate gradient updates of different modal branches, guiding the dual-branch network to cooperate between modalities and extract high-quality modal features. Step D: Implement multi-level cross-modal information transmission, that is, use gated variable gradient transmission convolution modules on the multi-level feature layers of the backbone network, and use learnable depth scanning factors to dynamically adjust the weights to achieve progressive injection and enhancement of cross-modal information from shallow to deep layers; Step E: Perform dynamic weight allocation fusion, that is, use the dynamic weight allocation fusion module to regress the modality adjustment factor according to the input feature distribution to achieve adaptive weighting and deep interactive fusion of visible light and infrared features; Step F: Generate target detection results, that is, output the target's category label and location prediction information.

2. The visible-infrared dual-modal target detection method as described in claim 1, characterized in that: Step A includes: Step A1: Acquire infrared and visible light images using a visible light camera and an infrared camera; the visible light camera is a PointGrey Flea3, the infrared camera is a FLIR-A35 with a spatial resolution of 640×480 pixels and a vertical field of view of 103.6 degrees, and the thermal infrared camera has a spatial resolution of 320×256 pixels and a vertical field of view of 39 degrees. Step A2: Perform image registration on the acquired infrared and visible light images by adopting an alignment strategy and adjusting the image size to achieve image registration.

3. The visible-infrared dual-modal target detection method as described in claim 1, characterized in that: Step B includes: Step B1: Perform coarse-grained processing on the image pair dataset to obtain a coarse-grained image pair dataset. This is achieved using the PIL class in Python: edge enhancement, sharpening, smoothing, and detail enhancement. Step B2: Perform automated quality assessment and screening on the preprocessed images in the dataset, and use the OpenCV class in Python to extract high-gain feature paired images.

4. The visible-infrared dual-modal target detection method as described in claim 3, characterized in that: Step B1 includes: Step B1-1: Image edge enhancement processing, that is, using edge detection operators or gradient enhancement algorithms to saliency processing of texture edges in visible light images and heat source contours in infrared images; Step B1-2: Structural sharpening, which involves using contour filtering or high-pass filtering to extract and enhance the structural features in the image. By suppressing low-frequency background redundancy, the geometric shape information of the target can be completely extracted. Step B1-3: Adaptive smoothing processing, which uses smoothing filtering or denoising algorithms to make the gray-level distribution of the image tend to be smooth and gradual. Step B1-4: Multi-scale detail enhancement, which involves using detail enhancement filtering methods to dynamically adjust local contrast; Step B2 includes: Step B2-1: Spatial registration verification, which is to measure the statistical correlation between visible light and infrared images in pixel spatial distribution based on mutual information; Step B2-2: Comprehensive image quality assessment, which is based on the average gradient AG reflecting the rate of change of small detail contrast in the image, representing the image sharpness. Contrast, signal-to-noise ratio and sharpness thresholds are set, and the image is automatically detected to filter out invalid samples that are completely invisible due to sensor overexposure, extreme low-light noise or severe motion blur. Step B2-3: Feature saliency and complementarity screening, that is, using the Tenengrad gradient function algorithm, extracting the gradient values ​​Gx and Gy in the horizontal and vertical directions through the Sobel operator, calculating the root mean square of the gradient of the whole image, using the gradient magnitude sum to quantify the texture richness, and using local contrast to quantify the saliency of heat sources.

5. The visible-infrared dual-modal target detection method as described in claim 3, characterized in that: Step C includes: Step C1: Constructing a dual-stream alternating activation feature extraction network; Step C2: Input the enhanced bimodal dataset obtained from B2 and set the three-stage training strategy.

6. The visible-infrared dual-modal target detection method as described in claim 5, characterized in that: Step C1 includes: Step C1-1: Using YOLO11s as the baseline network topology, construct visible light and infrared branches in parallel on the YOLO11s single-branch backbone network; Step C1-2: Construct the embedding and connection of multi-level gated variable gradient propagation convolutional modules, that is, embed three gated variable gradient propagation convolutional modules in the last three feature extraction layers of the visible light branch and the infrared branch. Step C1-3: Integration of the dynamic weight allocation fusion module, that is, after the last layer output of the visible light branch and the infrared branch, the dynamic weight allocation module is set as the feature convergence center to output the visible light enhancement feature and the infrared enhancement feature. Step C1-4: Input and output of the prediction head, that is, the spliced ​​result of visible light enhancement feature and infrared enhancement feature is sent to the detection head, and the target attribute recognition and bounding box localization are performed through classification branch and regression branch respectively.

7. The visible-infrared dual-modal target detection method as described in claim 5, characterized in that: Step C2 includes: Step C2-1: Input the high-gain feature pairing image obtained in B2 into the network established in C1; Step C2-2: Set the three-stage training parameters from C1-1; Step C2-2 includes: Step C2-2-1: Stage 1, the visible light branch activates the infrared branch, that is, the infrared branch parameter update is frozen, all module parameter updates of the infrared branch are set to Freeze, and the infrared branch is stopped from passing convolution modules through gated variable gradients. Step C2-2-2: Phase 2, the infrared branch activates the visible light branch, that is, the visible light branch parameter update is frozen, all module parameter updates of the visible light branch are set to Freeze, and the visible light branch is stopped from passing through the gated variable gradient convolution module; Step C2-2-3: Phase 3, dual-branch joint debugging, that is, unfreezing the dual-branch parameter update, canceling the Freeze setting, and performing joint fine-tuning of the entire network.

8. The visible-infrared dual-modal target detection method as described in claim 5, characterized in that: Step D includes: Step D1: Input the information of the last 3 feature extraction layers of the visible light branch and infrared branch of C2 into the gated variable gradient propagation convolution module; Step D2: Use the C1-2 module to progressively inject and enhance cross-modal information from shallow to deep layers using the information input in D1.

9. The visible-infrared dual-modal target detection method as described in claim 5, characterized in that: Step E includes: Step E1: Input the last feature extraction layer of C1 into the dynamic weight allocation fusion module to perform dual-modal feature fusion; Step E2: The dynamic weight allocation fusion module dynamically adjusts the fusion weights between modes according to the current stage.

10. A visible-infrared dual-modal target detection device using the method as described in claim 1, comprising: A dual-mode image acquisition unit (1), a calculation unit (2), and a display unit (3) are characterized in that: the dual-mode image acquisition unit (1) acquires infrared images and visible light images, and then sends them to the calculation unit (2); the calculation unit (2) completes dual-mode image recognition, and then sends the results to the display unit (3); the dual-mode image acquisition unit (1) includes a sealed box (11), a beam splitter (12), an infrared camera (13), and a visible light camera (14); a light-collecting port (111) is opened on the lower part of one side of the sealed box (11); the beam splitter (12) is fixedly installed on the bottom surface of the sealed box (11); the infrared camera (13) is connected to the fixed bracket of the sealed box (11), and is vertically downward, with the lens of the infrared camera (13) directly facing the center position of the beam splitter (12); the visible light camera (14) is installed on the side of the infrared camera (14); the calculation unit (2) is configured with an implementation program corresponding to the method described in claim 1; and the display unit (3) displays the final result.

Images