Multi-modal target detection method based on asymmetric double-branch and channel heterogeneous fusion

Abstract

The invention belongs to the technical field of computer vision and deep learning, and discloses a multi-modal target detection method based on asymmetric double-branch and channel heterogeneous fusion, and the method comprises the steps: receiving a visible light image and a thermal infrared image; inputting the two into an asymmetric double-branch backbone network to extract multi-scale features; the visible light features and the thermal infrared features of the corresponding scales are input into a channel heterogeneous weighted fusion module, cross-modal channel weights are obtained through convolution, and the channel weights act on different modal features to achieve selective enhancement according to channels; and the weighted features are fused and then sent to an FPN multi-scale detection head for target detection. Different modal feature expression differences are adapted through an asymmetric network structure, the cross-modal fusion capability is improved through a channel heterogeneous weighting mechanism, and the detection performance is remarkably improved on a multi-modal data set. The system is light in structure and effective in fusion, and can be widely applied to scenes such as night monitoring, pedestrian detection and automatic driving.

Description

A multimodal target detection method based on asymmetric dual-branch and channel heterogeneous fusion Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning technology, and specifically relates to a target detection method based on the fusion of visible light images and thermal infrared images, especially a mid-term fusion detection framework that combines an asymmetric dual-branch structure and a channel heterogeneous weighted fusion mechanism. Background Technology

[0002] Object detection is a crucial research area in computer vision. Its core task is to automatically identify and accurately locate target objects in images or videos, and it has been widely applied in practical scenarios such as autonomous driving, intelligent monitoring, aerospace, and emergency rescue. With the development of deep learning technology, object detection methods based on convolutional neural networks have made significant progress in both detection accuracy and processing efficiency. However, under complex environmental conditions, especially in scenarios with drastic changes in lighting, severe background interference, or poor target visibility, the information provided by a single modality image often has significant limitations, making it difficult to meet the practical requirements for highly robust object detection.

[0003] In real-world scenarios, infrared and visible light images exhibit significant complementary characteristics. Visible light images possess rich color, texture, and detail information, performing exceptionally well in daylight and under good lighting conditions; while infrared images, by capturing the thermal radiation characteristics of objects, retain clear contours and structural information even in harsh environments such as low light, backlight, and dense fog. Therefore, combining these two modalities for target detection can significantly improve environmental adaptability and detection stability, and has become one of the important research directions in multimodal vision.

[0004] Existing multimodal object detection methods are typically built upon deep learning frameworks and can be categorized into two-stage and single-stage methods based on their detection processes. Two-stage methods, exemplified by Faster R-CNN, offer high detection accuracy but suffer from high computational complexity, making them unsuitable for real-time applications. Single-stage methods, represented by the YOLO series, significantly improve inference speed while maintaining a certain level of accuracy, gradually becoming the mainstream approach in practical applications. Consequently, recent research on multimodal object detection has increasingly shifted towards single-stage detection frameworks.

[0005] In multimodal object detection, existing methods can be broadly categorized into three types based on the different modality fusion levels: early fusion, mid-stage fusion, and late-stage fusion. Early fusion methods typically stitch together images from different modalities at the input layer before feeding them into a unified network for processing. However, due to significant differences in information distribution and semantic characteristics among different modalities, these methods often struggle to learn effective cross-modal representations. Late-stage fusion methods combine the outputs of different modalities at the detection result level. While simple to implement, they fail to fully exploit the complementary information between modalities. In contrast, mid-stage fusion methods interactively model multimodal information at the feature level, achieving a balance between performance and efficiency to some extent, and have become the main technical approach in current multimodal object detection research.

[0006] However, current mainstream multimodal target detection methods still have many shortcomings, mainly in the following aspects:

[0007] First, different modalities of images exhibit inherent heterogeneity in imaging mechanisms, information distribution, and semantic representation capabilities. Visible light images excel in detail texture and color information, while infrared images are more prominent in structural contours and region perception. Baek J et al. proposed a method combining thermal position intensity histograms of oriented gradients with additive kernel support vector machines for nighttime pedestrian detection in thermal images. Herrmann C et al. first used preprocessing to make the thermal images look more like grayscale images converted from RGB images, and then used a pre-trained and fine-tuned SSD detector. Conversely, Guo T used generative adversarial networks to convert visible light images into thermal images as data augmentation to train a pedestrian detector for handling object detection tasks in thermal images. Although many two-stream network structures have been proposed in recent years to try to extract complementary features from the two modalities, most methods still adopt a backbone network design with a completely symmetrical structure and consistent parameter scale, treating the two modalities as "the same type of input," ignoring the significant semantic differences and task requirements between the modalities. This non-differentiated design makes it difficult for the model to fully utilize the structural advantages of infrared images or the detailed advantages of visible light images, resulting in insufficient feature representation capabilities and thus limiting further improvement in target detection performance.

[0008] Secondly, although existing research has enhanced visible light and infrared fusion through techniques such as weight sharing, attention mechanisms, and cross-modal interactive learning, most of these methods rely on the premise of consistent input feature channel numbers. Therefore, when the two modalities use backbone networks of different structures or sizes, the feature channel dimensions are often inconsistent, making existing fusion modules difficult to apply directly and severely limiting the flexibility and scalability of the model structure in multimodal scenarios. Furthermore, the lack of dynamic modeling of the semantic contributions of different modalities during the fusion process easily leads to problems such as intermodal interference, feature conflicts, or excessive information smoothing, resulting in unsatisfactory fusion effects.

[0009] Third, while existing research has attempted novel techniques such as diffusion models, complementary coding, and cross-modal generation to enhance the interaction between visible and infrared information, most methods still employ symmetrical network depth and channel configurations, failing to achieve a truly meaningful modal differentiation feature extraction strategy. In complex scenes, this setup may lead to the visible light image branch being affected by noise and background interference, while the infrared branch may be limited by insufficient resolution and texture representation capabilities, thus affecting the final detection performance.

[0010] Therefore, a novel multimodal feature extraction and fusion framework is needed that can explicitly model the differences between visible light and infrared modalities at the structural design level, and allow for differentiated configurations of the two modalities in terms of network size, channel capacity, and feature representation capabilities. Simultaneously, a new fusion mechanism needs to be designed to adapt to heterogeneous channel feature inputs, fully utilizing the structural advantages of the dominant modality (e.g., infrared) and the detailed information of the auxiliary modality (e.g., visible light) to achieve efficient collaborative representation of multi-source visual information.

[0011] Against this backdrop, this invention proposes a novel multimodal target detection method. By constructing an asymmetric dual-branch structure with "infrared dominance and visible light assistance," the infrared branch undertakes the main feature extraction task, capturing strong structural semantic information with a wider channel structure. Simultaneously, a lightweight visible light branch supplements high-frequency details and texture information. Furthermore, addressing the challenge of inconsistent multimodal feature channels, this invention designs a channel heterogeneous weighted fusion module (HCFM). This module achieves hierarchical fusion of multimodal features through three pathways: a high-weight path (infrared dominance), a low-weight path (visible light assistance), and an initial feature residual path. This enables the construction of stable modal competition and complementarity relationships in the feature space.

[0012] In summary, existing multimodal detection technologies still have significant room for improvement in handling modal differences, adaptability to heterogeneous structures, and fusion effectiveness. This invention addresses these issues by proposing a novel structural design and fusion mechanism, providing a new technical approach to enhance multimodal target detection performance in complex environments, low-light scenes, and high-dynamic backgrounds. Summary of the Invention

[0013] This invention discloses a multimodal target detection method based on modal differences and channel heterogeneous modeling, comprising the following steps. The entire process starts from cross-modal input, and through asymmetric bi-branch feature extraction, channel heterogeneous weighted fusion, hierarchical feature aggregation, and finally detection and prediction, forms a high-precision detection system that can be trained end-to-end.

[0014] A multimodal target detection method based on asymmetric dual-branch and channel heterogeneous fusion includes the following steps:

[0015] Step 1: Acquire multimodal input images and perform preprocessing;

[0016] Let the visible light image be Infrared images are Interpolation or cropping is performed on the two images to ensure that the visible light image and the infrared image are perfectly aligned in the pixel space, thus obtaining a preprocessed image set.

[0017] The preprocessed image set is used as the initial input and fed into an asymmetric dual-branch network for independent feature modeling.

[0018] Step 2: Feature extraction using an asymmetric dual-branch network based on modal differences;

[0019] Asymmetric dual-branch networks are divided into infrared network branches and visible light network branches;

[0020] The infrared network branch, targeting the clear target outlines and dominant detection modality of infrared images, adopts a YOLOv5-X-level architecture with high parameter count and deep structure. The network depth is set at a factor of 1.33, and the channel width at a factor of 1.25 to maximize the network's receptive field and semantic abstraction capabilities. The network structure consists of five layers: Layers 0 and 2 are shallow focusing and downsampling layers: lossless downsampling is performed through a slicing focusing module, followed by convolutional layers and the C3 residual module to extract basic edge features; Layer 3 is the P3 feature layer: after downsampling and multiple stacking of the C3 module, it has high spatial resolution and is used for locating small targets; Layer 4 is the P4 feature layer: further downsampling and deep residual calculation are performed for the feature response of medium-scale targets; Layer 5 is the P5 feature layer: after processing by a spatial pyramid pooling module, multi-scale local features are fused.

[0021] The visible light network branch, serving as an auxiliary modality, addresses the issue of high-frequency texture details in visible light images, which are susceptible to low-light noise. It employs a simplified and shallower YOLOv5-M-level architecture, with a network depth multiplier of 0.67 and a channel width multiplier of 0.75. Compared to the infrared branch, this branch significantly reduces the number of residual component stacks and convolutional channels, reducing computational redundancy and preventing overfitting to background noise. While still adhering to a five-layer structure, the number of Bottleneck repetitions within the C3 module of each layer is significantly reduced. The visible light branch outputs a feature set with a lower channel dimension than the infrared branch, providing feature information for subsequent channel heterogeneous weighted fusion modules.

[0022] The infrared network branch and the visible light network branch output infrared and visible light features at different levels, respectively:

[0023]

[0024] in, Indicates a branch of the visible light network. Indicates an infrared network branch. Indicates the number of layers in an asymmetric two-branch network;

[0025] Step 3: Construct a channel heterogeneous weighted fusion module to achieve hierarchical fusion of cross-modal features;

[0026] First, infrared and visible light features are bidirectionally stitched together to construct cross-modal combined features. :

[0027]

[0028] in, The infrared signature output by the infrared network branch. Visible light characteristics are the outputs of the visible light network branches; For the concatenation operation; subsequently, the cross-modal combined features are processed using deformable convolution and central difference convolution, normalization, and SiLU activation to obtain the cross-modal hybrid encoding:

[0029]

[0030]

[0031] in, It is a deformable convolution feature. Central difference convolution features;

[0032] Visible light characteristics Infrared features Feasible variable convolution features Convolution with center difference The fusion process is divided into three paths, each corresponding to a different semantic role;

[0033] (1) Spatial structure perception path:

[0034] infrared features Perform coordinate attention on the infrared features along the horizontal and vertical directions respectively. Perform one-dimensional global average pooling to generate orientation-aware features. and Subsequently through sharing Convolutional transformation and activation function generate spatial weights to obtain infrared structural localization features. ;

[0035]

[0036] in, This indicates element-wise multiplication. Represents convolution transformation, This represents the Sigmoid activation function;

[0037] Infrared structural localization features With deformable convolution features By splicing the data, a spatial structure perception path can be obtained. ;

[0038]

[0039] (2) High-frequency texture enhancement path:

[0040] Visible light characteristics Perform channel attention; for visible light features Global spatial average pooling is performed to obtain channel descriptors, followed by learning channel importance weights using a multilayer perceptron for visible light features. Perform channel recalibration to obtain pure texture features. ;

[0041]

[0042] Pure texture features in visible light Convolution features with center difference By stitching the data together, a high-frequency texture enhancement path can be obtained. ;

[0043]

[0044] (3) Initialize the residual path:

[0045] infrared features With visible light characteristics Perform concatenation to obtain the initial residual path. ;

[0046]

[0047] Finally, a fixed-weight approach is used in the fusion process to ensure the stability of the learning process:

[0048]

[0049] in, Features of fusion; ;

[0050] Subsequently, to reduce redundant information and enhance the discriminative power of the fused features, the fused features... Channel compression is performed using 1×1 convolution to obtain the compressed fused features. :

[0051]

[0052] in, express convolution, This indicates the number of channels in the feature map output by the fusion module. This indicates the number of visible light feature map channels input to the fusion module. This indicates the number of infrared feature map channels input to the fusion module. Indicates rounding down;

[0053] Step 4: Input the compressed fused features into the multi-scale detection head to perform target localization and classification;

[0054] The compressed fused features are fed into a multi-scale detection head for final prediction, enabling simultaneous localization and classification of multiple target types.

[0055] The output of the multi-scale detection head includes the target bounding box position, class label, and confidence score, which constitute the target detection result;

[0056] Step 5: Model training for the multimodal object detection network;

[0057] The multimodal object detection network is trained uniformly using a stochastic gradient descent optimizer. The network parameters are optimized by setting the initial learning rate, momentum factor, and weight decay coefficient. A cosine annealing learning rate decay strategy is also used to gradually converge the model during training. During the training phase, multiple iterations with a fixed batch size are performed to obtain stable multimodal object detection model parameters.

[0058] During the training and testing phases, the input image size is normalized to output the final object detection result.

[0059] Furthermore, the infrared network branch replaces the YOLOv5-X architecture with the YOLOv8-X architecture, while the visible light network branch replaces the YOLOv5-M architecture with the YOLOv8-M architecture.

[0060] Furthermore, by setting weights for channel heterogeneous weighted fusion to emphasize the dominant role of the infrared mode, the weights are set to... .

[0061] Furthermore, the multi-scale detection head is a replaceable structure, capable of employing YOLO-based target detection structures, FCOS-based anchor-free detection structures, or RetinaNet-based target detection structures.

[0062] The beneficial effects of this invention are as follows: By designing an asymmetric bi-branch feature extraction network based on modal differences, this invention significantly improves the performance of multimodal target detection under complex lighting conditions. This network utilizes differentiated modeling through an infrared-dominant, visible-light-assisted asymmetric architecture, fully leveraging the structural robustness of infrared images and the texture richness of visible-light images to perform complementary optimization of multimodal features. This design not only improves the feature representation quality of the model under nighttime or low-light conditions but also significantly enhances the robustness and accuracy of the target detection model in all-weather environments, enabling it to perform exceptionally well in a wide range of scenarios.

[0063] The core advantage of the Channel Heterogeneous Fusion Module (HCFM) lies in combining operator physics mechanisms with attention enhancement to form a dual guarantee mechanism of structural anchoring and texture filtering. It focuses on geometrically calibrating the infrared structure while maintaining the purity of the visible light texture. During the fusion process, the infrared branch utilizes deformable convolution and coordinate attention to adaptively adjust the sampling position and strengthen spatial anchors, providing precise skeletal support for the fused features. Meanwhile, the visible light branch utilizes differential convolution and channel attention to focus on extracting high-frequency gradients and suppressing low-light noise, ensuring that each fusion step focuses on preserving effective information and eliminating interfering information. This effectively avoids the shortcomings of traditional direct stitching methods, which easily introduce background noise or cause feature alignment deviations. It not only ensures the contour integrity of the detected target but also significantly reduces the feature aliasing side effects that may be caused by cross-modal fusion. Attached Figure Description

[0064] Figure 1 is a schematic diagram of the overall network structure of the infrared and visible light target detection method based on asymmetric dual-branch and channel heterogeneous fusion provided by the present invention.

[0065] Figure 2 is a flowchart of the infrared and visible light target detection method based on asymmetric dual-branch and channel heterogeneous fusion provided by the present invention. Detailed Implementation

[0066] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0067] As shown in Figure 1, this invention proposes a multimodal target detection method based on an asymmetric dual-branch structure and a channel heterogeneous fusion mechanism. This method is used for feature extraction, cross-modal information fusion, and final target detection from multimodal inputs that simultaneously contain visible light and infrared images. The method aims to fully utilize the complementarity of infrared and visible light, and improve detection performance in nighttime, low-light, and complex scenes through structural heterogeneity and semantic weighting strategies.

[0068] As shown in Figure 2, the method flow of this invention mainly includes steps such as multimodal image acquisition, modal differential feature extraction, channel heterogeneous weighted fusion and three-path semantic interaction, target detection, and model training. The specific implementation is as follows:

[0069] 1. Acquire multimodal input images and perform preprocessing;

[0070] First, visible light and infrared images are acquired from a dual-spectrum sensor. While these two images are typically acquired synchronously via hardware, spatial offsets may still exist. Therefore, a geometric alignment algorithm is needed to align the infrared image... Registered to visible light image In the coordinate space, to ensure that subsequent cross-modal features can be correctly mapped.

[0071] Secondly, the input images are normalized and their dimensions are standardized. For example, the images are uniformly scaled to 640×512 or 640×640, and basic data augmentation strategies are performed, including random cropping, random scaling, and random flipping, to improve the model's generalization ability and make it applicable to different viewpoints, lighting conditions, and scene changes.

[0072] 2. Feature extraction based on asymmetric dual-branch network with modal differences;

[0073] In infrared and visible light target detection, the infrared and visible light images are first input into their respective feature extraction backbone networks, as shown in Figure 1. The model contains two independent backbone networks: the visible light branch uses a medium-sized network structure. (e.g., YOLOv5-M), while the infrared branch uses a higher capacity model. (e.g., YOLOv5-L) is used to enhance the expression of infrared features.

[0074] Let the first The visible light branch output of the layer is: The infrared branch output is: , which usually satisfy: This means that there are more infrared branch channels, which can be used to carry stronger semantic expressions.

[0075] 3. Construct a channel heterogeneous weighted fusion module to achieve hierarchical fusion of cross-modal features;

[0076] First, the visible light and infrared features are bidirectionally stitched together to obtain: Subsequently, cross-channel semantics were extracted through convolution, and deformable convolution features were obtained for the infrared-dominant path. For visible light-assisted paths, central difference convolution is used to obtain central difference convolution features. :

[0077] Subsequently, visible light characteristics Infrared features Feasible variable convolution features Convolution with center difference The fusion process is divided into three paths, with the specific steps as follows;

[0078] To illustrate this more specifically, let's assume we are currently processing a feature map at the P3 level: infrared features. The dimension is Visible light characteristics The dimension is .

[0079] Spatial structure perception path: infrared features Perform coordinate attention. This module generates horizontal weights. and vertical weights The two are multiplied together to accurately locate the target center. Next, the processed features... With visible light characteristics splicing, outputting features .

[0080] High-frequency texture enhancement path: for visible light features Channel attention processing is performed. Assuming channels 10-50 contain nighttime noise and channels 51-180 contain clothing texture, the attention mechanism suppresses the noise channel weight to 0.05 and enhances the texture channel weight to 0.95. Next, the processed clean texture features are... With visible light characteristics splicing, outputting features .

[0081] Initialize residual path: set infrared features With visible light characteristics Perform concatenation to obtain the initial residual path. ;

[0082] Finally, a fixed-weight approach is used in the fusion process to ensure the stability of the learning process: Subsequently, to reduce redundant information and enhance the discriminative power of the fused features, the fused features... Channel compression was performed using 1×1 convolution, resulting in a fusion feature with 256 compressed channels. :

[0083] 4. Input the compressed fused features into the multi-scale detection head to perform target localization and classification;

[0084] The fusion features with dimensions of 80×80×256 obtained in step 3 The data is fed into a replaceable multi-scale detection head. Taking the YOLOv5 multi-scale detection head as an example, a prediction tensor containing bounding box coordinates, class probabilities, and confidence scores is generated through convolution operations. In the post-processing stage, a confidence threshold is first set to filter background noise, followed by non-maximum suppression. The final output is a optimized target detection result containing precise coordinates, class labels, and confidence scores.

[0085] 5. Model training for multimodal object detection networks;

[0086] The multimodal object detection network is trained uniformly using a stochastic gradient descent optimizer. The network parameters are optimized by setting the initial learning rate, momentum factor, and weight decay coefficient. A cosine annealing learning rate decay strategy is also used to gradually converge the model during training. During the training phase, multiple iterations with a fixed batch size are performed to obtain stable multimodal object detection model parameters.

[0087] This invention achieves deep alignment and complementarity of cross-modal semantics by constructing an infrared-dominated asymmetric dual-branch structure and combining it with the channel heterogeneous weighted fusion module HCFM. Experiments show that this structure significantly improves the model's ability to recognize medium and large targets and effectively alleviates the problem of information loss in visible light modalities in low-light scenes.

[0088] Furthermore, the three-path fusion mechanism proposed in this invention not only possesses high stability but also exhibits excellent adaptability to channel misalignment and modal differences, making it flexibly applicable to various detection frameworks and scenario requirements. This design achieves a balance between fusion quality, computational overhead, and model deployability, providing an efficient and reliable solution for multimodal vision tasks.

Claims

1. A multimodal target detection method based on asymmetric dual-branch and channel heterogeneous fusion, characterized in that, Includes the following steps: Step 1: Acquire multimodal input images and perform preprocessing; Let the visible light image be Infrared images are The two images are interpolated or cropped to ensure that the visible light image and the infrared image are perfectly aligned in the pixel space, resulting in a preprocessed image set. The preprocessed image set is then used as the initial input and fed into an asymmetric dual-branch network for independent feature modeling. step 2: Feature extraction based on asymmetric dual-branch network with modal differences; The asymmetric dual-branch network consists of an infrared network branch and a visible light network branch. The infrared network branch adopts the YOLOv5-X architecture, while the visible light network branch adopts the YOLOv5-M architecture. The infrared network branch and the visible light network branch output infrared and visible light features at different levels, respectively. ;in, Indicates a branch of the visible light network. Indicates an infrared network branch. Indicates the number of layers in the asymmetric dual-branch network; Step 3: Construct a channel heterogeneous weighted fusion module to achieve hierarchical fusion of cross-modal features; First, perform bidirectional splicing of infrared and visible light features to construct cross-modal combined features. ： ;in, The infrared signature output by the infrared network branch. Visible light characteristics are the outputs of the visible light network branches; For the concatenation operation; subsequently, the cross-modal combined features are processed using deformable convolution and central difference convolution, normalization, and SiLU activation to obtain the cross-modal hybrid encoding: ； ;in, It is a deformable convolution feature. Central difference convolution features; visible light features Infrared features Feasible variable convolution features Convolution with center difference The fusion process is divided into three paths, each corresponding to a different semantic role; (1) Spatial structure perception path: for infrared features Perform coordinate attention on the infrared features along the horizontal and vertical directions respectively. Perform one-dimensional global average pooling to generate orientation-aware features. and Subsequently through sharing Convolutional transformation and activation function generate spatial weights to obtain infrared structural localization features. ； ;in, This indicates element-wise multiplication. Represents convolution transformation, Indicates the Sigmoid activation function; locates infrared structural features. With deformable convolution features By splicing the data, a spatial structure perception path can be obtained. ； (2) High-frequency texture enhancement path: for visible light features Perform channel attention; for visible light features Global spatial average pooling is performed to obtain channel descriptors, followed by learning channel importance weights using a multilayer perceptron for visible light features. Perform channel recalibration to obtain pure texture features. ； ; Pure texture features in visible light Convolution features with center difference By stitching the data together, a high-frequency texture enhancement path can be obtained. ； (3) Initialize the residual path: set the infrared features With visible light characteristics Perform concatenation to obtain the initial residual path. ； Finally, a fixed-weight approach is used in the fusion process to ensure the stability of the learning process. ;in, Features of fusion; Subsequently, to reduce redundant information and enhance the discriminative power of the fused features, the fused features... Channel compression is performed using 1×1 convolution to obtain the compressed fused features. ： ;in, express convolution, This indicates the number of channels in the output feature map of the heterogeneous weighted fusion module. This indicates the number of visible light feature map channels input to the channel heterogeneous weighted fusion module. This indicates the number of infrared feature map channels input to the channel heterogeneous weighted fusion module. The text describes a process of rounding down to the nearest integer. Step 4 involves inputting the compressed fused features into a multi-scale detection head for target localization and classification. The compressed fused features are then fed into the multi-scale detection head for final prediction, achieving simultaneous localization and classification of multiple target classes. The output of the multi-scale detection head includes the target bounding box position, category label, and confidence score, constituting the target detection result. Step 5 involves training the multi-modal target detection network model. The multi-modal target detection network is trained uniformly using a stochastic gradient descent optimizer. The network parameters are optimized by setting the initial learning rate, momentum factor, and weight decay coefficient, combined with a cosine annealing learning rate decay strategy to gradually converge the multi-modal target detection network during training. Multiple iterations with a fixed batch size are used during the training phase to obtain stable network parameters for multi-modal target detection. The input image size is normalized during both the training and testing phases to output the final target detection result.

2. The multimodal target detection method based on asymmetric dual-branch and channel heterogeneous fusion according to claim 1, characterized in that, The infrared network branch replaces the YOLOv5-X architecture with the YOLOv8-X architecture, while the visible light network branch replaces the YOLOv5-M architecture with the YOLOv8-M architecture.

3. The multimodal target detection method based on asymmetric dual-branch and channel heterogeneous fusion according to claim 1, characterized in that, set up 。 4. The multimodal target detection method based on asymmetric dual-branch and channel heterogeneous fusion according to claim 1, characterized in that, The multi-scale detection head has a replaceable structure and can adopt a YOLO-based target detection structure, an FCOS-based anchorless detection structure, or a RetinaNet-based target detection structure.

Images