A dual-branch multi-modal object detection method

Abstract

This invention belongs to the field of computer vision technology and proposes a dual-branch multimodal target detection method. It aims to solve the technical problem that single-modal detection is easily affected by illumination, occlusion, and background interference in complex environments, resulting in low accuracy and poor robustness. This method uses a general target detection model as its basic architecture, constructing a dual-branch feature extraction network for visible and infrared light. Dedicated feature enhancement modules are designed to address the deficiencies of each modality, and a mid-term multi-scale fusion strategy is employed to achieve cross-modal information complementarity. This invention effectively improves the accuracy and robustness of target detection in complex environments and can be applied to high-precision target detection in scenarios such as nighttime, strong light, and occlusion, demonstrating significant practical application value.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a dual-branch multimodal target detection method based on an improved YOLO11. Background Technology

[0002] Object detection, a core research direction in computer vision, directly determines the reliability of key applications such as autonomous driving, intelligent monitoring, emergency rescue, and industrial inspection, based on the detection accuracy, robustness, and real-time performance of algorithms. Object detection technology in complex scenes has become a research hotspot and challenge in the field of computer vision. In the development of object detection technology, single-modal object detection methods have long held a core position in research and application due to their high technological maturity and low deployment cost. However, single-modal object detection methods have limitations due to their single data source. To pursue higher detection accuracy and performance, researchers have shifted their focus to multimodal fusion object detection technology. Visible light images contain a large amount of texture and edge information, which can compensate for the lack of detail in infrared images. Infrared images rely on the thermal radiation of objects for imaging and are almost unaffected by light intensity. Even in complex environments such as backlight and low light, they can easily capture targets that are easily missed by visible light, alleviating the problem of target obscuration in visible light images under complex lighting conditions. This complementary nature provides a core approach to overcoming the technical bottlenecks of single-modal object detection.

[0003] Currently, multimodal fusion target detection technology based on the YOLO framework is still under development. Existing technologies mostly use simple feature stitching and weighted fusion to achieve feature fusion of visible light and infrared, which has many shortcomings. First, there is no dedicated feature extraction module designed for the inherent defects of visible light and infrared images, making it difficult to achieve targeted feature extraction. Second, the neck network of the traditional YOLO model adopts a unified channel design, which easily leads to redundant small-scale feature computation, insufficient large-scale feature expression ability, and weak fusion block expression ability, failing to fully aggregate the low-level spatial details of visible light and the high-level semantic features of infrared. Third, the design of multimodal fusion strategies lacks specificity, making it difficult to achieve efficient fusion and information complementarity of cross-modal features. Therefore, developing a dual-branch multimodal target detection method based on an improved YOLO model that combines targeted feature extraction and efficient multi-scale fusion capabilities has become the key to solving the problem of high-precision target detection. Summary of the Invention

[0004] The purpose of this invention is to provide a dual-branch multimodal target detection method in complex scenes, which solves the problems of low accuracy, easy missed detection and false detection of single-modal targets in complex scenes. By combining the advantages of visible light images and infrared light images, higher accuracy detection can be achieved.

[0005] To achieve the above objectives, this invention proposes a dual-branch multimodal target detection method, which includes the following steps:

[0006] Step 1: Preprocess the M3FD public dataset and divide it into training, validation, and test sets in a 7:1.5:1.5 ratio.

[0007] Step 2: Improve the original YOLO11 model by building a dual-branch feature extraction network. Use the MSLPEM module to improve the feature extraction process for visible light images; use the IR-FEGate module to improve the feature extraction process for infrared images; and perform multi-scale fusion processing on the features extracted from different levels of the dual branches.

[0008] Step 3: Train the improved network model using the training set and validation set;

[0009] Step 4: Test the trained network model using the test set to obtain evaluation metrics and complete the evaluation of detection accuracy;

[0010] Step 1 specifically refers to:

[0011] The preprocessing includes checking image integrity and deleting images where visible light and infrared light are misaligned;

[0012] The specific steps for building the dual-branch feature extraction network in Step 2 are as follows:

[0013] Two parallel feature extraction branches are built using the backbone of YOLO11, which are used for multi-scale feature extraction of visible light images and infrared light images, respectively. At the same time, a cross-modal feature fusion module is built based on the mid-term fusion strategy to fully integrate the features extracted from visible light images and infrared light images and put them into different levels of the dual-branch backbone multiple times.

[0014] Step 2 uses the MSLPEM module to improve the feature extraction process of visible light images, addressing issues such as background redundancy diluting target features in visible light images. Specifically:

[0015] First, the input feature map Channel calibration and dimensionality unification are performed, and residual features are obtained through 1×1 pointwise convolution. ,in B For batch size, C in Input the number of channels. H,W The feature map size;

[0016] Secondly, a label feature filtering mechanism should be constructed for... Use fixed size p × pPerform non-overlapping label partitioning to decompose the feature map into Local area By aggregating the channel-dimensional mean values ​​across all channels, the global statistical features of the region are obtained.

[0017] in, Subsequently, a two-layer MLP and layer normalization were applied:

[0018] This enables the mapping from statistical features to semantic features, enhancing feature discriminative power.

[0019] Then, a learnable task cue vector is introduced. The correlation between label features and task prompts is calculated using cosine similarity to generate a dynamic filtering mask. Simultaneously, the Softmax function is used to calculate the regional attention weights within the global scope. Enhanced region features are obtained through weighted averaging and masking. The original feature map size is then restored using bilinear interpolation, and the channel dimensions are calibrated using 1×1 convolution to obtain single-scale label-enhanced features. ;

[0020] Then, a parallelized dual-scale branching design is adopted, using parallel processing at p=2 and p=4 respectively to simultaneously capture the fine-grained features of small targets and the global structural features of large targets. All residual branch features are then added element-wise to form a six-branch fusion feature. ;

[0021] At the same time, construct the association matrix between tags. ,element Indicates the first i The first tag and the first j The semantic relevance of each label Based on this association matrix, the features of each label are complemented by a weighted sum of the global label features, thus reconstructing the complete target features. ; Next, a dual-level enhancement mechanism of efficient channel attention (ECA) and spatial attention is introduced. ECA learns channel dependencies through 1D convolution with adaptive kernel size to generate channel weights. ,in , =2, b =1, Features after channel enhancement , This represents element-wise multiplication followed by spatial attention:

[0022] Focus on the target area and suppress residual background interference;

[0023] After further processing including two-stage attention enhancement, Dropout regularization (p=0.1), batch normalization, and SILU activation, the final output features of the module are obtained.

[0024] Finally, the Bottleneck structure in the C3K2 module of the original YOLO11 model is replaced with the Multi-scale label perception enhancement mechanism (MSLPEM) module to achieve background redundancy suppression, multi-scale target representation enhancement, and robustness improvement in complex scenes of visible light images, thereby effectively improving feature extraction of visible light images.

[0025] The process of improving the feature extraction of infrared images using the IR-FEGate module in Step 2 is as follows:

[0026] First, large-kernel depthwise separable convolutions are used to enhance the discriminative power of target-background thermal distribution: the standard 3×3 convolutions of the traditional C3K module are replaced with 7×7 depthwise separable convolutions, and the receptive field calculation formula is as follows:

[0027]

[0028] Where k=7 is the kernel size, s i =1 is the step size, when the initial receptive field of the input feature map is RF. prev When RF=1, the receptive field expands to RF=7, which is 2.3 times that of a 3×3 convolution; meanwhile, the computational complexity of depthwise separable convolution is:

[0029] Compared to 3×3 standard convolution While maintaining C in =C out At that time, the complexity is reduced by approximately C. in It can capture the target's own thermal features and the surrounding background thermal distribution simultaneously, solving the problem of confusion between weak thermal features and background.

[0030] Subsequently, an adaptive thermal feature selection mechanism was constructed to suppress thermal noise and feature fragmentation:

[0031] The first step is input normalization: normalizing the input features. Execution layer normalization:

[0032]

[0033] The second step, channel expansion and splitting: The channels are expanded to 2C using a 1×1 convolution. hidden It is further divided into a filtering branch g, an identity branch i, and a convolution branch c based on the channel index:

[0034]

[0035] Where I is the channel split index, satisfying .

[0036] The third step is branch processing and filtering fusion: Convolutional branch c extracts local features through a 7×7 depthwise convolution, and filtering branch g generates adaptive weights through GELU activation. Finally, these weights are multiplied element-wise with the concatenated features from the identity branch and the convolutional branch. ,

[0037] Step 4, Residual Output: After 1×1 convolutional compression of the channels, the final feature is output by combining the residual connections.

[0038]

[0039] This mechanism suppresses the feature response of noisy spots through adaptive weights, while aggregating effective hot spots into continuous target contours, thus solving the feature fragmentation problem caused by thermal noise.

[0040] Next, linear complexity feature clustering is used to match low-resolution infrared features:

[0041] Feature aggregation is completed within O(C⋅H⋅W) complexity. Depthwise separable convolution performs spatial convolution only within a single channel, avoiding redundant channel computation. The adaptive thermal feature filtering mechanism focuses on the feature transfer of effective channels, reducing the propagation of invalid information. At the same time, the edge details in the low-resolution feature map are preserved through the splicing strategy of identity branch + convolution branch, and local features are enhanced through convolution branch. Finally, the aggregation of long-range weak thermal features is achieved through filtering and fusion, integrating the scattered thermal signals of different parts of the target into a concentrated strong feature representation.

[0042] Finally, the Bottleneck structure in the C3K2 module of the original YOLO11 model was replaced with an Infrared Thermal Feature Gated Convolution Block (IR-FEGate) module to avoid low-resolution feature degradation while keeping the model lightweight.

[0043] The multi-scale fusion process of the features extracted from different levels of the dual branches in Step 2 is as follows:

[0044] The neck structure in the traditional YOLO11 model is replaced with a re-parameterized generalized feature pyramid (RPGFP) network, and a multi-scale layer aggregation method is added, specifically:

[0045] First, a multi-scale feature channel differentiation allocation mechanism is constructed:

[0046] To address the computational redundancy and feature sparsity issues caused by unifying multi-scale feature channels in traditional FPN, the number of channels is dynamically allocated based on the FLOPs requirements of features at different scales. The formula for calculating the number of channels is as follows:

[0047] Where Cs is the number of channels at scale s (s∈{P3,P4,P5}), C base The baseline number of channels is set to 96, and αs is the scale adaptation coefficient (P3:α=1, P4:α=2, P5:α=4). Ultimately, differentiated configurations of CP3=96, CP4=192, and CP5=384 are achieved, enabling cost control with fewer channels for small-scale features and multi-channel expression for large-scale features, reducing ineffective computational overhead at small scales and enhancing the expressive power of effective features at high scales.

[0048] Then, construct the reparameterized enhanced fusion block:

[0049] To address the issues of weak expressive power and insufficient feature fusion in traditional FPN, the fusion block adopts a combined design of CSP structure + reparameterized RRB + multi-scale layer aggregation.

[0050] The CSP structure splits the input feature X into a main branch and a sub-branch, specifically using the formula: X1 = Conv 1×1 (X); X2=Conv 1×1 (X), the main branch directly retains the basic features, while the sub-branch strengthens them through residual block iteration to avoid feature degradation;

[0051] Reparameterized RRB residual blocks: Multi-branch convolutions are used during training, and merged into single convolutions during inference, balancing training effectiveness and inference speed; its forward propagation is as follows: ;

[0052] Among them, RepConv3×3 is a multi-branch reparameterized convolution. During training, it contains 1×1 and 3×3 branches, and during inference, it is merged into a single 3×3 convolution, achieving a balance between strong training expression and high inference efficiency.

[0053] MLAM (Multi-Scale Layer Aggregation Module): This module enhances the interaction between features at different depths through multi-scale layer aggregation, avoiding the isolation of local features. Its aggregation formula is as follows: ;

[0054] Among them, F i⨁ is the output of the residual block of the i-th layer of the sub-branch, and ⨁ is the channel splicing. Finally, all intermediate features of the main branch X1 and the sub-branch are merged to achieve deep aggregation of semantic and spatial features. This more fully integrates high-level semantics with low-level edge position details, avoiding missed detection of small targets due to loss of spatial details and false detection of large targets due to semantic ambiguity.

[0055] Step 3 specifically refers to:

[0056] The improved network model is trained using the training set and validated using the validation set. After training, the optimal weights are saved in the weights folder.

[0057] Step 4 specifically refers to:

[0058] The trained model is tested on the test set to evaluate its detection accuracy. The evaluation metrics are mAP50 and mAP50:95.

[0059] Wherein, mAP50 is the average AP of all categories when the IoU threshold is 0.5, and mAP50:95 refers to the average mAP from the IoU threshold from 0.5 to 0.95. The higher the mAP, the more accurate the model detection.

[0060] Compared with existing technologies, this invention has the following advantages: Based on the YOLO11 model, this invention constructs a dual-branch feature extraction network for visible and infrared light. The MSLPEM module improves the visible light image extraction process, solving the problems of high information redundancy and significant differences in local target features across multiple scales, thus enhancing robustness in complex scenes. The IR-FEGate module improves the feature extraction process for infrared images, solving the problems of insufficient detail information and fragmentation of local thermal features due to thermal noise. Dedicated feature enhancement extraction modules are designed to perform targeted feature mining on the dual-branch images. A mid-term fusion strategy is adopted to perform multi-scale fusion processing on the extracted features at different levels of the dual branches. The neck structure in the traditional YOLO11 model is replaced with a reparameterized general feature pyramid network. This solves the problems of computational redundancy and feature sparsity caused by the unification of multi-scale feature channels in traditional FPN, as well as the weak expressive power of the fusion block and insufficient feature fusion. This network structure fully leverages the complementary advantages of rich detail in visible light images and prominent target features in infrared images, achieving efficient fusion and information complementarity across modalities. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the process of the present invention;

[0062] Figure 2 This is a network model diagram of a dual-branch multimodal target detection method for complex scenarios, as illustrated in this invention.

[0063] Figure 3 This is a structural diagram of the MSLPEM module in an example of the present invention;

[0064] Figure 4 This is a structural diagram of the IR-FEGate module in an example of the present invention; Detailed Implementation

[0065] The invention will be further described below with reference to the accompanying drawings and specific examples, but this does not limit the scope of the invention.

[0066] like Figure 1 As shown, a dual-branch multimodal target detection method includes the following steps:

[0067] Step 1: Preprocess the M3FD public dataset by checking image integrity and deleting images where visible light and infrared light are not aligned; and divide it into training, validation and test sets in a ratio of 7:1.5:1.5.

[0068] Step 2: Improve the original YOLO11 model by building a dual-branch feature extraction network. Use the MSLPEM module to improve the feature extraction process for visible light images; use the IR-FEGate module to improve the feature extraction process for infrared images; and perform multi-scale fusion processing on the features extracted from different levels of the dual branches. The improved network model structure diagram is shown below. Figure 2 As shown;

[0069] Specifically, a dual-branch feature extraction network is constructed, such as... Figure 2 As shown:

[0070] Two parallel feature extraction branches are built using the backbone of YOLO11, which are used for multi-scale feature extraction of visible light images and infrared light images, respectively. At the same time, a cross-modal feature fusion module is built based on the mid-term fusion strategy to fully integrate the features extracted from visible light images and infrared light images and put them into different levels of the dual-branch backbone multiple times.

[0071] Specifically, the MSLPEM module is used to improve the feature extraction process of visible light images, such as... Figure 3 As shown;

[0072] This module first performs a 1×1 pointwise convolution on the input feature map to complete channel calibration and generate residual features, laying the foundation for dimensional alignment for multi-branch fusion. Then, it performs fixed-size non-overlapping label segmentation on the residual features, compresses redundant information through channel mean aggregation, and maps statistical features to semantic features through two layers of MLP and layer normalization. A learnable task cue vector is introduced, combined with cosine similarity masking and Softmax global attention weights to filter and enhance target-related label features. Single-scale label enhancement features are obtained through bilinear interpolation and 1×1 convolution. To adapt to multi-scale targets, the module employs parallel dual-scale branches with p=2 and p=4 to simultaneously capture fine-grained features of small targets and global structural features of large targets, fusing the residual branch, the three-level convolution branch, and the dual-scale label features element-wise. Subsequently, an inter-label correlation matrix is ​​constructed to achieve complementary reconstruction of fracture features. Finally, through dual-level enhancement using ECA channel attention and spatial attention, combined with Dropout regularization, batch normalization, and SiLU activation, robust features that accurately suppress background redundancy and enhance multi-scale target representation are output.

[0073] Specifically, the IR-FEGate module is used to improve the feature extraction process of infrared images, such as... Figure 4 As shown;

[0074] The IR-FEGate module focuses on infrared thermal feature enhancement. First, it performs layer normalization on the input features to stabilize the training distribution. Then, it expands the channels using 1×1 convolutions, splitting each channel into a selection branch, an identity branch, and a convolutional branch. The convolutional branch extracts local thermal features through 7×7 depthwise separable convolutions, expanding the receptive field and capturing the difference in thermal distribution between the target and background while maintaining linear computational complexity. The selection branch generates adaptive weights through GELU activation, which are then multiplied element-wise with the concatenated features from the identity and convolutional branches to achieve effective thermal feature selection and noise suppression, avoiding feature fragmentation caused by thermal noise. Finally, 1×1 convolutions compress the channels and combine them with residual connections for output, aggregating low-resolution, scattered thermal features while preserving edge details. This addresses infrared defects such as weak thermal feature confusion and feature dilution while maintaining a lightweight model suitable for real-time detection requirements.

[0075] Specifically, the process of multi-scale fusion of features extracted from different levels of the two branches is as follows: Figure 2 As shown:

[0076] To address the issues of channel redundancy, feature dilution, and insufficient fusion in traditional FPN, this paper proposes an improved RPGFP architecture that achieves efficient multi-scale feature fusion through two core mechanisms: First, a multi-scale feature channel differential allocation mechanism is constructed, dynamically allocating the number of channels based on the FLOPs requirements of features at each scale. Using a baseline of 96 channels as a base, the number of channels is configured according to adaptation coefficients of P3 (α=1), P4 (α=2), and P5 (α=4) as C.P3 =96、C P4 =192、C P5 =384, which reduces the ineffective computational overhead of small-scale features and enhances the semantic expressive power of large-scale features. Secondly, a reparameterized enhanced fusion block is designed, adopting a combination structure of CSP branch + reparameterized residual block + multi-scale hierarchical aggregation module (MLAM): CSP branch splits the input features into a main branch that retains the original features and a sub-branch that enhances convolution, avoiding feature degradation; the reparameterized residual block balances expressive power and inference efficiency with a mode of multi-branch convolution training and single convolution inference; the MLAM module aggregates all intermediate features of the sub-branch through channel concatenation, realizing deep fusion of high-level semantics and low-level spatial details, ultimately alleviating the problem of missed detection of small targets and false detection of large targets, and balancing detection accuracy and real-time performance.

[0077] Step 3: Train the improved network model using the training set and validation set;

[0078] Step 4: Test the trained network model using the test set to obtain evaluation index values ​​and complete the evaluation of detection accuracy;

[0079] Specifically, this invention uses precision, recall, mAP50, and mAP50:95 as evaluation metrics to evaluate the performance of the model's detection precision.

[0080] The formulas for calculating precision, recall, mAP50, and mAP50:95 are as follows:

[0081]

[0082]

[0083]

[0084]

[0085] Where P represents precision, R represents recall, TP represents the number of correctly predicted samples, FP represents the number of incorrectly predicted samples, FN represents the number of incorrectly predicted samples, and mAP50 is the average AP across all classes when the IoU threshold is 0.5. mAP50:95 refers to the average mAP across IoU thresholds from 0.5 to 0.95. As can be seen from the formula, changes in TP, FP, and FN directly affect precision and recall, and thus affect the mAP value. The higher the mAP, the more accurate the model detection.

[0086] To further demonstrate the performance advantages of this invention, several mainstream target detection models, such as YOLOv8, YOLOv10, YOLO11, YOLOv12, and YOLOv13, were introduced for comparative experiments. In the comparative experiments, the hyperparameters of all models were kept consistent, and they were trained for 300 epochs. The experimental results are shown in Table 1.

[0087] Table 1 Comparison of Experimental Results of Different Target Detection Models Model Dataset types Precision P (%) Recall rate R (%) mAP50 (%) mAP50: 95 (%) YOLOv8 RGB 82.1 68.9 76.2 49.4 YOLOv8 IR 78.1 66.9 72.6 47.3 YOLOv10 RGB 80.9 64.8 73 46.9 YOLOv10 IR 80.3 65.9 72.4 46.8 YOLO11 RGB 81.3 67.6 74.5 48.6 YOLO11 IR 78.9 66.2 72.4 47.1 YOLOv12 RGB 81.9 66.8 73.1 47.5 YOLOv12 IR 79.5 63.3 70.1 49 YOLOv13 RGB 81.4 68.2 74.4 48.5 YOLOv13 IR 77.4 66.5 71.4 46.9 Ours RGB+IR 85.8 81 84.4 59

[0088] As can be seen from the results in Table 1, the dual-branch multimodal target detection method of the present invention has the best performance, outperforming the comparative model in all four indicators. Moreover, the improved model improves by at least 9.9% and 10.4% in mAP50 and mAP50:95, respectively, compared with the original YOLO11.

[0089] The specific implementation of the present invention has been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above-described embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A dual-branch multimodal target detection method, characterized in that, Includes the following steps: Step 1: Preprocess the M3FD public dataset and divide it into training, validation and test sets in a ratio of 7:1.5:1.5; Step 2: Improve the original YOLO11 model by building a dual-branch feature extraction network and using the MSLPEM module to improve the feature extraction process of visible light images; The IR-FEGate module is used to improve the feature extraction process of infrared images, and multi-scale fusion processing is performed on the features extracted from different levels of the dual branches. Step 3: Train the improved network model using the training set and validation set; Step 4: Test the trained network model using the test set to obtain evaluation index values ​​and complete the evaluation of detection accuracy.

2. The dual-branch multimodal target detection method according to claim 1, characterized in that, Step 1 specifically refers to: The preprocessing includes checking image integrity and deleting images where visible light and infrared light are not aligned.

3. The dual-branch multimodal target detection method according to claim 1, characterized in that, The specific steps for building the dual-branch feature extraction network in Step 2 are as follows: Two parallel feature extraction branches were built using the backbone of YOLO11, which were used for multi-scale feature extraction of visible light images and infrared light images, respectively. At the same time, a cross-modal feature fusion module was built based on the mid-term fusion strategy to fully integrate the features extracted from visible light images and infrared light images and put them into different levels of the dual-branch backbone multiple times.

4. The dual-branch multimodal target detection method according to claim 1, characterized in that, The specific steps in Step 2, which use the MSLPEM module to improve the feature extraction process of visible light images, are as follows: The Bottleneck structure within the C3K2 module of the original YOLO11 model is replaced with a multi-scale label perception enhancement mechanism (MSLPEM) module. The construction and workflow of the MSLPEM module are as follows: S4-1, Input Feature Map Channel calibration and dimensionality unification are performed, and residual features are obtained through 1×1 pointwise convolution. ,in B For batch size, C in Input the number of channels. H,W The feature map size; S4-2, Construct a label feature filtering mechanism for... Use fixed size p × p Perform non-overlapping label partitioning to decompose the feature map into Local area By aggregating the channel-dimensional mean values ​​across all channels, the global statistical features of the region are obtained. ； Then, a two-layer MLP and layer normalization are applied: This enables the mapping from statistical features to semantic features, enhancing feature discriminative power. in S4-3, Introduce learnable task cue vectors The correlation between label features and task prompts is calculated using cosine similarity to generate a dynamic filtering mask. Simultaneously, the Softmax function is used to calculate the regional attention weights within the global scope. Enhanced region features are obtained through weighted averaging and masking. The original feature map size is then restored using bilinear interpolation, and the channel dimensions are calibrated using 1×1 convolution to obtain single-scale label-enhanced features. ; S4-4. A parallel dual-scale branching design is adopted, which simultaneously captures the fine-grained features of small targets and the global structural features of large targets through parallel processing at p=2 and p=4 respectively. All residual branch features are added element-wise to form a six-branch fusion feature. ; S4-5, Construct the tag association matrix , of which elements Indicates the first i The first tag and the first j Semantic relevance of each label: Based on this association matrix, the features of each label are complemented by a weighted sum of the global label features, thus reconstructing the complete target features. ; S4-6. Introduce a two-stage enhancement mechanism of efficient channel attention (ECA) and spatial attention. ECA learns channel dependencies through 1D convolution with adaptive kernel size to generate channel weights. ,in , =2, b =1, For the Sigmoid function, the enhanced features are... , This represents element-wise multiplication followed by spatial attention. Focus on the target area and suppress residual background interference; S4-7. After two-stage attention enhancement, Dropout regularization (p=0.1), batch normalization, and SILU activation function, the final output features of the module are obtained. This enables the suppression of background redundancy in visible light images, the enhancement of multi-scale target representation, and the improvement of robustness in complex scenes.

5. The dual-branch multimodal target detection method according to claim 1, characterized in that, The process of improving the feature extraction of infrared images using the IR-FEGate module in Step 2 is as follows: The Bottleneck structure within the C3K2 module of the original YOLO11 model is replaced with an Infrared Thermal Feature Gated Convolution Block (IR-FEGate) module. The construction and workflow of the IR-FEGate module are as follows: S5-1. Enhanced target-background heat distribution discrimination using large-kernel depthwise separable convolution: The traditional 3×3 standard convolution of the C3K module is replaced with a 7×7 depthwise separable convolution, and its receptive field calculation formula is as follows: Where k=7 is the kernel size, s i =1 is the step size, when the initial receptive field of the input feature map is RF. prev When RF=1, the receptive field expands to RF=7, which is 2.3 times that of a 3×3 convolution; meanwhile, the computational complexity of depthwise separable convolution is: Compared to 3×3 standard convolution While maintaining C in =C out At that time, the complexity is reduced by approximately C. in It can capture the target's own thermal features and the surrounding background thermal distribution simultaneously, solving the problem of confusion between weak thermal features and background. S5-2. Constructing an adaptive thermal feature selection mechanism to suppress thermal noise and feature fragmentation: Input normalization: normalization of input features Execution layer normalization: ; Channel expansion and splitting: Expanding channels by 2C using 1×1 convolution. hidden It is further divided into a filtering branch g, an identity branch i, and a convolution branch c based on the channel index: Where I is the channel split index, satisfying ;; Branch processing and filtering fusion: Convolutional branch c extracts local features through a 7×7 depthwise convolution, and filtering branch g generates adaptive weights through GELU activation. Finally, these weights are multiplied element-wise with the concatenated features from the identity branch and the convolutional branch. ; Residual output: After 1×1 convolutional compression of the channels, the final feature is output by combining the residual connections. This mechanism suppresses the feature response of noise spots through adaptive weights, while aggregating effective hot spots into continuous target contours, thus solving the feature fragmentation problem caused by thermal noise. S5-3. Use linear complexity feature aggregation to match low-resolution infrared features: feature aggregation is completed in O(C⋅H⋅W) complexity, and depthwise separable convolution is performed only in a single channel to avoid redundant channel calculations. The adaptive thermal feature filtering mechanism focuses on the feature transfer of effective channels and reduces the propagation of invalid information. At the same time, it retains the edge details in the low-resolution feature map through the splicing strategy of identity branch and convolution branch, enhances local features through convolution branch, and finally achieves the aggregation of long-range weak thermal features through filtering and fusion. It integrates the scattered thermal signals of different parts of the target into a concentrated strong feature representation. While avoiding the degradation of low-resolution features, it keeps the model lightweight (the number of parameters is only 12% of C3k2) and adapts to the requirements of real-time infrared detection.

6. The dual-branch multimodal target detection method according to claim 1, characterized in that, Step 2 involves multi-scale fusion of features extracted from different levels of the dual branches. Specifically, the neck structure in the traditional YOLO11 model is replaced with a re-parameterized generalized feature pyramid (RPGFP) network, and a multi-scale layer aggregation method is added. The construction and workflow of RPGFP are as follows: S6-1. Constructing a multi-scale feature channel differential allocation mechanism: Addressing the computational redundancy and feature sparsity issues caused by the uniformity of multi-scale feature channels in traditional FPN, a dynamic channel number allocation mechanism is implemented based on the FLOPs requirements of features at different scales. The formula for calculating the number of channels is as follows: Where Cs is the number of channels at scale s (s∈{P3,P4,P5}), C base The baseline number of channels (set to 96), α s The scale adaptation coefficients (P3:α=1, P4:α=2, P5:α=4) are used to achieve differentiated configurations of CP3=96, CP4=192, and CP5=384. This allows small-scale features to be expressed using fewer channels to control costs, while large-scale features are expressed using more channels. This reduces the ineffective computational overhead at small scales and enhances the expressive power of effective features at high scales. S6-2. Constructing a Reparameterized Enhanced Fusion Block: To address the issues of weak expressive power and insufficient feature fusion in traditional FPN's Neck fusion block, the fusion block adopts a combined design of CSP structure, reparameterized RRB residual block, and multi-scale layer aggregation. CSP structure: The input feature X is split into a main branch and a sub-branch to avoid feature degradation; X1=Conv1×1(X); X2=Conv1×1(X), the main branch directly retains the basic features, and the sub-branch is strengthened through residual block iteration to avoid feature degradation; Reparameterized RRB residual blocks: Multi-branch convolutions are used during training, and merged into single convolutions during inference, balancing training effectiveness and inference speed; its forward propagation is as follows: Among them, RepConv3×3 is a multi-branch reparameterized convolution. During training, it contains 1×1 and 3×3 branches, and during inference, it is merged into a single 3×3 convolution, achieving a balance between strong training expression and high inference efficiency. MLAM (Multi-Scale Layer Aggregation Module): This module enhances the interaction between features at different depths through multi-scale layer aggregation, avoiding the isolation of local features. Its aggregation formula is as follows: Among them, F i ⨁ is the output of the residual block of the i-th layer of the sub-branch, and ⨁ is the channel splicing. Finally, all intermediate features of the main branch X1 and the sub-branch are merged to achieve deep aggregation of semantic and spatial features. This more fully integrates high-level semantics with low-level edge position details, avoiding missed detection of small targets due to loss of spatial details and false detection of large targets due to semantic ambiguity.

Images