Cross-modal image information fusion enhanced small target detection device, method and application

Abstract

The present application relates to a small target detection device, a detection method and application enhanced by cross-modal image information fusion. The detection device comprises: an image acquisition module for acquiring a visible light image and an infrared thermal imaging image; a feature extraction module comprising a double-flow backbone network for extracting multi-scale features of the images, wherein a micro-domain enhancement convolution module (MECM) is integrated in the backbone network; a fusion module for adaptively fusing feature maps of two modalities through a cross-modal feature coupling module (CMFCM) at a specific level; a feature enhancement module for multi-scale enhancement of the fused features through a feature pyramid structure and a multi-scale aggregation pooling module (MAPM); and a detection module for target detection based on the enhanced features and outputting the results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and target detection technology, specifically relating to a small target detection device, detection method and application based on cross-modal image information fusion enhancement of visible light and infrared thermal imaging dual-modal image information fusion, suitable for scenarios such as drone aerial photography, security monitoring, autonomous driving and other scenarios. Background Technology

[0002] In the field of computer vision, small object detection (such as vehicles and pedestrians in long-distance aerial images taken by drones) is an extremely challenging task. Small objects occupy a very small percentage of pixels in an image, resulting in weak feature representation and easy obscuring by complex backgrounds. Single-modal vision sensors (such as RGB-only cameras) experience a significant drop in detection performance in environments with insufficient lighting (such as at night), low visibility (such as fog or rain), or cluttered backgrounds, leading to missed detections and false detections.

[0003] Infrared thermal imaging technology images objects by sensing their thermal radiation. It is less affected by ambient visible light, and the target and background typically have high thermal contrast. Therefore, combining the rich texture and color details of visible light (RGB) images with the significant target thermal contour information of infrared (thermal) images has become an important research direction for improving the robustness of small target detection in complex environments.

[0004] Currently, mainstream deep learning-based object detection frameworks (such as the YOLO series) are widely used for multimodal detection tasks. A common approach in existing technologies is to use a two-stream network to extract features from the RGB and Thermal modalities separately, and then fuse them. However, most existing fusion strategies are relatively simple, such as direct concatenation along the channel dimension or element-wise addition. This approach ignores the differences in contribution between the two modalities under different environments and the feature distribution shifts between modalities, failing to achieve adaptive and selective complementary advantages.

[0005] CN120564081A discloses a method and system for small target detection in UAVs based on dual-modal image fusion. Its purpose is to address the problems of low detection accuracy and high false negative rate in small target detection in UAVs due to the small target size and large differences in modal information. The technical solution includes: acquiring an image of the UAV to be tested; inputting the image of the UAV to be tested into a target detection model to obtain detection results; wherein the target detection model includes: a backbone network module, a neck network module, a detection module, and an output module; the backbone network module is used to acquire feature maps at different scales; the neck network module is used to perform feature fusion at different levels on the feature maps to obtain a fused feature map; the detection module is used to detect the fused feature map to obtain detection results; and the output module is used to output the detection results. Its shortcomings are: while it achieves detection through a structure of backbone network, neck network, and detection module, its fusion method is relatively simple and fails to fully consider the characteristic of easy loss of small target features, and how to efficiently balance model performance and computational overhead to adapt to the actual situation of limited resources on the UAV airborne platform.

[0006] Therefore, there is an urgent need for a novel small target detection method that can adaptively fuse dual-modal information, effectively enhance the feature representation of small targets, and simultaneously balance detection accuracy and computational efficiency. Summary of the Invention

[0007] This invention aims to overcome the aforementioned shortcomings of the prior art and provide a small target detection device, detection method, and application that enhances cross-modal image information fusion. The technical problem solved by this invention is: 1. To address the issue of single-modal detection failure in harsh environments, the model significantly improves its robustness under all-weather conditions by adaptively fusing complementary information from RGB and Thermal modal modes.

[0008] 2. To address the issues of small targets having few pixels and weak features, a dedicated feature enhancement module and fusion strategy are designed to effectively preserve and enhance the details and contextual information of small targets, thereby improving the detection accuracy of small targets.

[0009] 3. While ensuring high detection performance, optimize the network structure and computation process to reduce model complexity and computational overhead, making it suitable for deployment on embedded or mobile devices (such as drones) with limited computing resources.

[0010] The first technical solution of the present invention is the small target detection method enhanced by cross-modal image information fusion, which is characterized by including the following steps: S1. Construct and preprocess an RGB-T multimodal dataset, which includes spatially aligned pairs of visible light images and infrared thermal images; S2. The preprocessed visible light image and infrared thermal imaging image are respectively input into a dual-stream backbone network for feature extraction to obtain their respective multi-scale feature maps; the dual-stream backbone network introduces a micro-domain enhanced convolutional module (MECM) in each feature extraction stage to enhance the feature representation capability of small targets; S3. The visible light feature map and infrared feature map output by the dual-stream backbone network at a specific level are fused to obtain a cross-modal fused feature map; the fusion is achieved by the cross-modal feature coupling module CMFCM, which is used to adaptively interact and weight the features of the two modes; S4. Input the cross-modal fused feature map into the neck network, and perform multi-scale feature fusion and enhancement through the feature pyramid structure to obtain the enhanced multi-scale feature map; S5. Input the enhanced multi-scale feature map into the detection head and output the target's category, confidence level, and location information; The detection method calculates the Intersection over Union (IOU) between the predicted bounding box and the ground truth bounding box, and the accuracy P. The detection accuracy is evaluated using IOU and recall (R), and the average precision (AP) is ultimately calculated as a performance metric. The calculations of IOU, P, and R are based on the following relationships:

[0011] In the formula: Area_b∩Area_obj represents the intersection area of ​​the predicted bounding box and the ground truth bounding box; Area_b∪Area_obj represents the area of ​​the union of the predicted bounding box and the ground truth bounding box; P=TP / (TP+FP); P=TP / (TP+FN); In the formula: TP, FP, and FN represent the number of true positives, false positives, and false negatives, respectively.

[0012] Preferably, the structure of the Micro-Domain Enhanced Convolutional Module (MECM) is shown in Figure 2, including: A 1×1 convolutional layer is used to expand the number of channels in the input feature map; At least two parallel attention bottleneck branches are used to extract features with different levels of attention. One branch integrates the SE attention mechanism, and the other branch integrates the CBAM attention mechanism. A feature concatenation and fusion layer is used to concatenate the features output from each branch and perform channel compression and fusion through 1×1 convolution; A skip connection is used to add the module's input to the fused output.

[0013] Preferably, the cross-modal feature coupling module (CMFCM) includes: Feature splitting unit, used to split the input pre-fusion features into at least two branches; A multi-scale local perception branch is used to capture local contextual information of small targets by dividing non-overlapping image patches of different sizes; At least one bottleneck structure branch is used to expand the feature receptive field through stacked convolutional layers to obtain global spatial context information; The feature fusion unit is used to concatenate the output features of the multi-scale local perception branch and the bottleneck structure branch in the channel dimension, and perform channel compression and integration through a convolutional layer.

[0014] Preferably, the specific layer mentioned in step S3 is the layer with a downsampling factor of 8 in the backbone network, namely layer P3; the neck network mentioned in step S4 also includes a multi-scale aggregation pooling module MAPM, which is used to perform attention weighting on features of different layers to enhance the response of small target regions; the detection head mentioned in step S5 includes multiple sub-detection heads for feature maps of different scales, and each sub-detection head integrates a micro-domain enhancement convolution module MECM.

[0015] Preferably, the dual-stream backbone network adopts a lightweight design based on an inverted residual structure.

[0016] The second technical solution of the present invention is the small target detection device with cross-modal image information fusion enhancement, which is characterized in that, The image acquisition module is used to acquire visible light images and infrared thermal imaging images; The feature extraction module includes a dual-stream backbone network for extracting multi-scale features of the image, wherein the backbone network integrates a micro-domain enhanced convolutional module (MECM). The fusion module is used to adaptively fuse feature maps of two modalities at a specific level through the cross-modal feature coupling module CMFCM. The feature enhancement module is used to enhance the fused features at multiple scales through the feature pyramid structure and the multi-scale aggregation pooling module MAPM. The detection module is used to perform target detection based on the enhanced features and output the results.

[0017] Preferably, the fusion module is a cross-modal feature coupling module (CMFCM) that supports block perception and multi-scale feature fusion; the feature enhancement module includes a micro-domain enhanced convolution module (MECM) and a multi-scale aggregated pooling module (MAPM); and the detection device is deployed in a drone or an embedded edge device.

[0018] Preferably, the detection performance of the method or apparatus is quantitatively evaluated using average precision (AP). AP is obtained by calculating the area of ​​a precision (PR) curve plotted with recall (R) as the x-axis and precision (P) as the y-axis. The calculation method is as follows:

[0019] In the formula: K represents the number of points taken in the interval; the value of AP is between [0,1]; Pc(k)Rc(k) is the curve value of point k.

[0020] The third technical solution of the present invention is the cross-modal image information fusion and enhancement method for small target detection, which is characterized by including the following steps: S1. In each feature extraction stage of the dual-stream backbone network, the input feature map is enhanced using the Micro-Domain Enhanced Convolutional Module (MECM), specifically including: S1.1 uses a 1×1 convolutional layer to expand the number of channels in the input feature map from C to 2C; S1.2 The expanded feature map is evenly divided into two branches along the channel dimension, and input to the bottleneck branch Bottleneck_S integrating the SE attention mechanism and the bottleneck branch Bottleneck_CBAM integrating the CBAM attention mechanism, respectively. S1.3 concatenates the features output from the two branches, and then performs channel compression and fusion through a 1×1 convolutional layer to restore the number of channels C. S1.4 adds the module input to the fused output through skip connections to obtain the enhanced feature map; S2. At the layer with a downsampling factor of 8 in the backbone network, i.e., layer P3, the RGB and Thermal bimodal feature maps are concatenated and input into the cross-modal feature coupling module CMFCM for adaptive fusion. S3. Input the fused cross-modal feature map into the neck network, and perform feature enhancement and multi-scale aggregation through the multi-scale aggregation pooling module MAPM; S4. Input the enhanced multi-scale feature map into the detection head, perform target detection, and output the detection results.

[0021] Preferably, the cross-modal feature coupling module (CMFCM) includes the following steps: S2.1 divides the input feature map before fusion into at least two branches through a splitting operation; One branch of S2.2 is the multi-scale local perception branch, Patch-Aware, which is used to capture local contextual information of small targets by dividing non-overlapping image patches of different sizes. Specifically, it includes: S2.2.1 Divide the feature map into 4×4 non-overlapping blocks for coarse-grained local context extraction; S2.2.2 Divide the feature map into 2×2 non-overlapping blocks for fine-grained local detail extraction; Another branch of S2.3 is the bottleneck structure branch BottleneckC, which expands the receptive field by stacking convolutional layers to obtain global spatial context information; S2.4 The features output by the multi-scale local perception branch and the bottleneck structure branch are concatenated in the channel dimension; S2.5 uses convolutional layers to compress and integrate the concatenated features, outputting a fused cross-modal feature map.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Significantly Improved Detection Robustness: Through the adaptive dynamic fusion strategy of the CMFCM module, the model can automatically adjust the fusion weights of RGB and Thermal modalities based on the input image content (such as lighting conditions and background complexity). This allows the model to maintain stable and high-precision detection even under harsh conditions where single modalities are prone to failure, such as low lighting (<50 lux), fog, and rain. Experiments show that in such environments, the detection accuracy of the method in this invention is more than 12.5% ​​higher than that of the baseline method using simple stitching fusion.

[0023] (2) Significantly improved accuracy in small target detection: The MECM module and the Patch-Aware branch in CMFCM, designed specifically for small targets, effectively enhance the local features and contextual information of small regions. On authoritative small target detection datasets such as VEDAI and DroneVehicle, the present invention achieves an average accuracy (AP) of 82.8% and 84.9% for small targets, respectively, which is approximately 14.5% higher than existing state-of-the-art methods.

[0024] (3) High computational efficiency and easy deployment: The network adopts a lightweight inverted residual structure design and performs efficient one-time fusion at the mid-level (P3), avoiding information redundancy in early fusion and computational burden in later fusion. Compared with traditional dual-stream fusion networks, the computational cost (GFLOPs) of this invention is reduced by 23.7%, and the number of parameters is reduced by 18.4%, making it more suitable for resource-constrained embedded platforms such as UAVs and meeting real-time requirements.

[0025] (4) Module synergy generates a systemic effect: MECM, CMFCM, and MAPM do not work in isolation. MECM pre-extracts highly discriminative features for each modality in the backbone network; CMFCM then deeply fuses these high-quality features at the optimal intermediate level; and MAPM performs final optimization on the fused multi-scale features in subsequent stages. This progressive design of "feature enhancement → intelligent fusion → final optimization" creates a powerful synergistic effect, jointly ensuring a comprehensive improvement in the final detection performance from different dimensions.

[0026] (5) To verify the effectiveness of the present invention, experiments were conducted using the VEDAI[1] and DroneVehicle[2] datasets. Vehicle Detection in Aerial Imagery (VEDAI) serves as a benchmark dataset for vehicle detection in aerial images, which can evaluate the performance of automatic target recognition algorithms in complex environments. This dataset was provided by the GREYC laboratory in Grenoble, France, and involves 1210 high-resolution aerial images with an image resolution of approximately 0.125 meters per pixel. Each image contains three color channels (RGB) and one near-infrared channel, providing rich and diverse multispectral information. The DroneVehicle dataset is a large-scale visible-infrared vehicle detection dataset based on drones, constructed by the authors of the UA-CMDet paper. The drone collected 28439 pairs of visible and infrared images, covering various scenes such as residential areas and parking lots, and spanning different time periods from day to night. The hardware information of the test equipment is as follows: 8 NVIDIA 2080Ti GPUs and 12GB of memory. The test results are as follows. Figure 4 As shown in the figure, the detection visualization results demonstrate that the proposed scheme can not only accurately identify small-scale targets in remote sensing scenes, but also effectively detect targets in complex low-light environments, fully verifying the feasibility and effectiveness of the scheme. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall structure of the cross-modal image information fusion and enhancement small target detection network of the present invention; Figure 2-1 This is a schematic diagram of the structure of the Micro-Domain Enhanced Convolutional Module (MECM) of the present invention; Figure 2-2 This is a schematic diagram of the structure of the Bottleneck_SE module of this invention; Figure 2-3 This is a schematic diagram of the structure of the Bottleneck_CBAM module of the present invention; Figure 3-1 This is a schematic diagram of the cross-modal feature coupling module CMFCM of the present invention; Figure 3-2This is a schematic diagram of the structure of the Bottleneck C module of the present invention: Figure 4 This is a comparison of some detection visualization results of the method of the present invention on the VEDAI dataset (left is RGB, right is the detection result of the corresponding Thermal image); Figure 5 This is a comparison of partial detection visualization results applied to the DroneVehicle dataset using the method of this invention (left is RGB, right is the detection result of the corresponding Thermal image). Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings: The core of this invention lies in an innovative network architecture that integrates three key modules based on an improved YOLOv11: Micro-domain Enhanced Convolutional Module (MECM), Cross-modal Feature Coupling Module (CMFCM), and Multi-scale Aggregate Pooling Module (MAPM).

[0029] Example 1: Small target detection device and method enhanced by cross-modal image information fusion: Please see Figure 1 As shown, the small target detection method enhanced by cross-modal image information fusion includes the following steps: (1) Construct and preprocess the RGB-T multimodal dataset. Collect a small target detection dataset containing visible light and thermal imaging dual-modal images, align, register and label the images; normalize the labels, unify the target bounding box format, and divide the dataset into training set, validation set and test set; (2) Input RGB and Thermal dual-modal images and feed them into a shared backbone network to extract multi-scale features. The RGB image and the infrared thermal image are respectively input into two parallel convolutional backbone networks. Each branch uses multiple convolutional layers to extract features step by step, including Conv64, Conv128, Conv256, etc. The MECM module is introduced at each stage to enhance the ability to fuse cross-channel and spatial context information and improve the ability to represent small targets. (3) Cross-modal feature fusion and multi-scale feature aggregation are achieved at the Backbone output by using the Cross-Modal Feature Coupling Module (CMFCM) and the Micro-Domain Enhanced Convolutional Module (MECM). The feature maps of the RGB and Thermal branches are fused at the P3 level, and CMFCM is used for cross-modal feature stitching and interaction to fully explore the complementary information between RGB and infrared images. Subsequently, multiple MECM modules are used to further enhance the feature representation capability and output multi-scale feature maps (P3, P4, P5). (4) Construct a feature pyramid Neck structure to achieve feature upsampling and multi-scale fusion. The feature pyramid is constructed using the Neck part: the high-level features are upsampled to the low-level resolution through the Upsample operation, and concatenated with the downsampled features of the corresponding level. Then, the features are enhanced by the Micro-Domain Enhancement Convolutional Module (MECM). At the same time, the Multi-Scale Aggregate Pooling Module (MAPM) is introduced to perform attention weighting on features of different levels, highlighting the response of small target regions and improving detection accuracy. (5) In the Head part, multi-scale detection head output is performed. The multi-scale feature maps output by the Neck are sent to the Detect Head respectively. Each detection head contains a Conv layer, a Concat operation and a Micro-domain Enhanced Convolutional Module (MECM) module. Finally, the target detection results of multiple scales are output to achieve high-precision localization and recognition of small targets in RGB-T scenes. ⑹ Conduct lightweight network testing and deployment optimization. Perform inference tests on the trained model to evaluate its detection accuracy (mAP), inference speed, and memory usage in different scenarios. Use lightweight techniques such as model pruning, quantization, and knowledge distillation to compress the network size and adapt to the deployment requirements of edge devices.

[0030] To accurately evaluate the performance of small object detection methods, several evaluation metrics have been proposed, focusing on the two tasks of object detection: object category and object location. These metrics include precision, recall, accuracy, average precision, and mean average precision. Typically, the detector outputs the object's category, confidence score, and location information. This information is compared with the ground truth to determine the performance of the object detection method. In the field of object detection, average precision is often used to evaluate the detection accuracy of a specific object class, while mean average precision is used to evaluate the detection accuracy across all object classes. The calculation of average precision involves precision, recall, and IOU. Therefore, these basic concepts will be introduced first. IOU is calculated by measuring the overlap between the predicted bounding boxes and the ground truth values. The formula is as follows: (1) A true positive occurs when the Intersection over Union (IOU) between the predicted bounding box and the real object's location is greater than a set threshold; otherwise, it is called a false positive. A false negative occurs when a detected target is considered background and thus missed. Precision (P) and recall (R) are represented by letter abbreviations, and TP, FP, and FN represent the number of true positives, false positives, and false negatives, respectively. The calculation process is illustrated by formulas (2) and (3). (2) (3) In the formula, a larger P indicates more targets are correctly predicted; a larger R indicates fewer targets are missed. However, there is a contradiction: as one value increases, the other decreases. Therefore, to comprehensively evaluate model performance, researchers use the average precision (AP) to reflect both correct predictions and missed detections. Specifically, a curve is plotted with P on the ordinate and R on the abscissa; the area under the curve is AP, calculated as shown in equation (4): 4 In the formula, K represents the number of points taken in the interval; the value of AP is between [0,1]; Pc(k)Rc(k) is the curve value of point k. Because the PR curve is difficult to represent as a function and its ideal value is difficult to calculate, interpolation averaging accuracy is usually used for approximation.

[0031] Besides detection accuracy, detection speed is also a crucial aspect of this invention. To measure the model's computational speed, this invention uses the number of model parameters and floating-point operations (FLOPs), a measure of algorithm complexity, to evaluate the target detection model. Fewer model parameters indicate lower computational complexity. Smaller FLOPs mean fewer required operations and a theoretically faster detection speed. Due to the large number of floating-point operations in the model, GFLOPs, measured in billions of floating-point operations, are generally used as the evaluation metric.

[0032] A small target detection device enhanced by cross-modal image information fusion includes: Micro-domain Enhanced Convolutional Module (MECM): The MECM integrates a micro-domain attention mechanism and frequency domain feature optimization. The features extracted through convolution are fused at a mid-level in layer P3. In layer P3, the feature maps of the two modalities are concatenated and processed by a trainable cross-modal feature coupling module (CMFCM). The extracted cross-modal feature information is fed into the Neck network layer through a multi-scale aggregation pooling module (MAPM). Based on the fused features, the head part of the detection head performs target detection to generate the final detection result.

[0033] The specific network modules include: The Conv module is the basic unit for model feature extraction. Its core is the integrated encapsulation of "Conv2d+BN+SiLU". Batch normalization normalizes the input of each batch, so that the mean and variance of the input remain stable. This can accelerate the convergence of the network, improve the generalization ability of the model, and has a certain regularization effect.

[0034] Micro-domain Enhanced Convolutional Module (MECM) module: such as Figure 2-1 As shown, the input feature map expands the number of channels from C to 2C through a 1×1 convolutional layer, increasing the feature dimension and providing space for subsequent fine-grained feature expression; the expanded feature map is divided into two branches along the channel dimension, which are fed into the Bottleneck_SE and Bottleneck_CBAM modules respectively.

[0035] Bottleneck_SE module: such as Figure 2-2 As shown in the dashed box, the branch features are first convolved with 1×1 and then connected to the SE module (Squeeze-and-Excitation).

[0036] Bottleneck_CBAM module: such as Figure 2-3 As shown in the dashed box, the branch features are first convolved with 1×11×1, and then connected to the CBAM (Convolutional Block Attention Module) to perform channel attention and spatial attention in sequence.

[0037] After processing through multiple stacked bottleneck blocks, the MHRM output contains a combination of feature maps with receptive fields of different sizes. These processed subsets are then reassembled, and 1×1 convolutions are used to compress and fuse multi-scale feature information, while restoring the number of channels C to reduce the number of parameters. Finally, a skip connection is added to the outermost layer of the module to prevent gradient vanishing.

[0038] Please see Figure 3-1 As shown, the Cross-Modal Feature Coupling (CMFCM) module's main function is cross-modal feature coupling, i.e., information interaction and fusion between visible light and thermal infrared modal features to improve feature expressiveness and model robustness. The features received by the module are split into two channels by a Split operation, which are then fed into the Patch-Aware module and the BottleneckC module, respectively.

[0039] The Patch-Aware module models the spatial local dependencies of small targets by dividing the map into local patches of varying sizes. The p=4 branch divides the feature map into 4×4 non-overlapping patches, capturing coarse-grained local context through local feature interactions within each patch. The p=2 branch focuses on fine-grained 2×2 patches to extract detailed information. This dual-branch approach, by covering patches of different scales, adapts to variations in the size of small targets and enhances sensitivity to local features.

[0040] Please see Figure 3-2As shown, the BottleneckC module consists of two convolutional layers; the features reduce computational overhead through the Bottleneck structure, while expanding the receptive field through multi-layer stacking, helping small targets obtain a wider spatial context and making up for the problem of insufficient receptive field of shallow feature maps.

[0041] The three BottleneckC output features are concatenated along the channel dimension using a Concat operation, fusing feature information from different patch scales and levels. Subsequently, the convolutional layer compresses the number of channels back to C, integrating multi-scale features while maintaining feature compactness, providing a highly discriminative feature representation for subsequent detection heads.

[0042] Example 2: Vehicle Detection Based on UAV Aerial Images This embodiment uses vehicle small target detection in a drone aerial photography scenario as an application background to illustrate the implementation process of the present invention in detail.

[0043] (1) Data preparation: The publicly available RGB-T bimodal dataset DroneVehicle was selected. Strict geometric registration was performed on each pair of RGB and Thermal images in the dataset to ensure spatial alignment; bounding boxes were annotated for vehicle targets in the images using a labeling tool. The dataset was randomly divided into training, validation, and test sets in a 7:2:1 ratio. Data augmentation strategies were implemented, including random horizontal flipping, color dithering (only for RGB images), and random cropping, to increase data diversity and prevent overfitting. (2) Network construction and training: Please see Figure 1 As shown, the overall architecture of the detection network of this invention comprises five main parts: (a) Input: Paired RGB image and Thermal image; (b) Dual-stream backbone network: two parallel feature extraction streams, each stage is followed by a MECM module; (c) Fusion layer (P3 layer): At the feature map (P3) after downsampling by 8 times in the backbone network, dual-modal feature fusion is performed through the CMFCM module; (d) Neck network: includes Feature Pyramid (FPN) and MAPM modules for multi-scale feature aggregation and enhancement; (e) Detection Head: Contains multiple sub-detection heads and outputs the final detection result.

[0044] according to Figure 1 The detection network of this invention is constructed using the structure shown. Backbone Network: A lightweight inverted residual block is used to construct the dual-stream backbone. After each downsampling stage, a... Figure 2-1 The MECM module shown; Specifically, it includes: a dual-stream backbone network that receives RGB and thermal images in parallel. Each stream contains multiple convolutional stages (such as Conv64, Conv128, Conv256, etc.) to extract multi-scale features; after each stage, a micro-domain enhanced convolutional module (MECM) is introduced to enhance feature diversity and sensitivity to small targets. Fusion Layer: Set at the third scale of the Backbone output (P3 layer, feature map size is 1 / 8 of the input image); here, the P3 feature maps of the RGB and Thermal branches are input into... Figure 3-1 The CMFCM module shown; Specifically, this includes: mid-level cross-modal fusion: at the layer with a downsampling factor of 8 in the backbone network (denoted as layer P3), the feature maps extracted from the two branches are fused; the fusion process is completed by the cross-modal feature coupling module (CMFCM), which can adaptively interact and weight the features of the two modalities; Neck and Head: The neck is constructed according to the FPN (Feature Pyramid Network) structure, and a MAPM module (whose function can be achieved through a lightweight SE attention layer) is introduced after feature concatenation. The detection head adopts a decoupled head structure, and a MECM module is integrated into its convolutional layers to further enhance features; Specifically, this includes: Neck network and feature pyramid: receiving the fused features and constructing a feature pyramid through upsampling, concatenation and other operations to achieve multi-level feature fusion; in this process, a multi-scale aggregation pooling module (MAPM) is introduced to reweight the attention of features at different scales to highlight features related to small targets; Detection Head: Receives multi-scale feature maps from the Neck output and predicts target bounding boxes, categories, and confidence levels at different scales in parallel using multiple sub-detection heads.

[0045] The AdamW optimizer was used with an initial learning rate of 1e-3 and a cosine annealing strategy. Training was performed on eight NVIDIA 2080Ti GPUs with a batch size of 64 for a total of 300 epochs. (3) Functional example of the fusion process: Taking a single inference example, let's illustrate the adaptive function of the CMFCM module: When inputting a pair of nighttime images, the RGB image is almost entirely black, while the heated parts of the vehicle are clearly visible in the Thermal image. The attention mechanism within the CMFCM module learns that the features of the current Thermal modality have higher confidence and information content. During the fusion process, the module automatically assigns higher weights to the Thermal features while suppressing noisy features from the RGB modality, thereby generating a high-quality fused feature map dominated by the Thermal modality for accurate detection by subsequent networks. Conversely, in daytime scenes with rich textures, the weights of RGB features are correspondingly increased. (4) Testing and Lightweight Deployment: Please see Figure 4 As shown, the left side is a visualization of the detection results for the RGB image, and the right side is a visualization of the corresponding detection results for the thermal image. The figures clearly demonstrate that the method of this invention can effectively detect small-sized vehicle targets (marked with bounding boxes) in complex remote sensing scenes by simultaneously utilizing the texture details of the RGB image and the thermal contour information of the thermal image, thus verifying the effectiveness of the method.

[0046] Please see Figure 5 As shown, the left side displays the detection results of RGB images under daytime / normal lighting conditions, while the right side displays the detection results of Thermal images under the corresponding conditions (or a comparison of detection results under low lighting conditions). The results demonstrate that the method of this invention maintains stable high detection accuracy under different lighting conditions throughout the day and night, significantly reducing missed detections and false detections.

[0047] Evaluate the trained model on the test set. For example... Figure 4 and Figure 5 As shown, the method of this invention can accurately detect small targets such as vehicles in various scenarios, including daytime, nighttime, urban, and rural areas, with a significant reduction in missed detections and false detections. Evaluation metrics show that the mAP (maximum accuracy) reaches 84.9%. To deploy on embedded drone platforms (such as the NVIDIA Jetson TX2), post-training quantization is performed on the trained model, converting the model weights from FP32 to INT8 precision. After quantization, the model size is reduced by approximately 75%, inference speed is increased by approximately 2 times, and accuracy loss is less than 2%, perfectly balancing the requirements of accuracy and efficiency.

[0048] Detailed description of key modules: Please see Figure 2-1As shown, the input feature map of the MECM module is first expanded by a 1×1 convolution. The expanded features are divided into two branches, which are processed by bottleneck blocks integrating SE attention and CBAM attention, respectively. The output features of the two branches are concatenated, then compressed by a 1×1 convolution, and finally added to the module input through skip connections to form the output.

[0049] Micro-domain Enhanced Convolutional Module (MECM): such as Figure 2-1 As shown, the core idea of ​​MECM is to mine deep information of features through parallel dual attention branches. After the input features are expanded through 1×1 convolutions, they are evenly divided into two branches, which are fed into the bottleneck block integrating the SE (Squeeze-and-Excitation) attention mechanism and the bottleneck block integrating the CBAM (Convolutional Block Attention Module) attention mechanism, respectively. The SE branch focuses on the relationship between channels, while the CBAM branch focuses on both channel and spatial relationships. The two branches are concatenated after processing, compressed back to the original number of channels by 1×1 convolutions, and added to the input through skip connections. This design enhances the network's ability to capture subtle features and the robustness of feature representation.

[0050] Please see Figure 3-1 As shown, the input features of the CMFCM module are split into two paths. One path is fed into a multi-scale local perception (Patch-Aware) branch, which extracts coarse-grained and fine-grained local contextual information by dividing the feature map into non-overlapping blocks of different sizes (e.g., 4×4 and 2×2). The other path is fed into a bottleneck (BottleneckC) branch, which expands the receptive field to obtain global context through stacked convolutions. The two feature paths are finally concatenated along the channel dimension and compressed and integrated through convolutional layers to output a fused feature map.

[0051] Cross-modal feature coupling module (CMFCM): such as Figure 3-1 As shown, CMFCM is responsible for intelligently fusing RGB and Thermal features at layer P3. The module first splits the input features. One of the main branches is the multi-scale local perception (Patch-Aware) branch, which models local spatial dependencies at both coarse and fine granular levels by dividing the feature map into non-overlapping blocks of different sizes (such as 4×4 and 2×2), specifically for capturing the local context of small targets. Another branch is the bottleneck (BottleneckC) branch, which expands the receptive field and supplements global context information by stacking convolutional layers. Finally, the features output by each branch are concatenated and fused to form a cross-modal fused feature that contains both local details and a global perspective.

[0052] Multi-scale Aggregate Pooling Module (MAPM): This module is integrated into the neck network. By applying channel or spatial attention to features at different pyramid levels (such as P3, P4, P5), it adaptively filters and enhances the feature information most effective for small target detection and suppresses background noise.

[0053] The specific implementation steps of the method of the present invention include: S1. Dataset Construction and Preprocessing: Collect and register RGB-T image pairs, perform annotation and data augmentation, and divide the dataset into training, validation and test sets; S2. Dual-modal feature extraction: The registered RGB and Thermal images are input into the dual-stream backbone network, and the MECM module is used to enhance features at each stage; S3. Cross-modal feature fusion: In the P3 layer, the CMFCM module is used to adaptively interact and fuse the two-stream features; S4. Multi-scale feature aggregation: A feature pyramid is constructed through the neck network, and the MAPM module is used for feature optimization; S5. Target Detection and Output: The detection head predicts targets based on multi-scale feature maps; S6. (Optional) Model Lightweighting: Perform pruning, quantization, and other operations on the trained model to adapt it for edge deployment.

[0054] Technological contributions and creative arguments: The inventiveness of this invention is mainly reflected in the following aspects, which constitute a complete logical chain of "technical problem - technical means - technical effect": (1) To address the problem of poor adaptability to single-modal environments, this invention does not employ fixed-weight early or late-stage fusion. Instead, it creatively proposes a dynamic adaptive fusion technique at the mid-level (P3). By designing the CMFCM module and utilizing its internal learnable parameters, the contribution of each modality feature is evaluated and weighted in real time based on the input content. This directly leads to a significant improvement in the model's robustness in complex, all-weather environments.

[0055] (2) To address the problem of "weak and easily lost features of small targets," this invention does not simply deepen or widen the network. Instead, from a feature engineering perspective, it proposes a technique combining "micro-domain enhancement" and "multi-scale local perception." Specifically, this is achieved through the synergistic effect of the MECM module (enhanced local feature extraction) and the Patch-Aware branch in CMFCM (explicitly modeling local contexts at different granularities). This directly leads to a significant improvement in the accuracy of small target detection, especially for targets with very few pixels.

[0056] (3) To address the contradiction between "limited resources on the UAV platform" and "high-precision requirements," this invention solves the problem through system-level architectural optimization. A lightweight backbone network is adopted to reduce basic overhead; the optimal fusion layer (P3) is selected to avoid redundant computation; and highly efficient dedicated modules (MECM, CMFCM) are designed to achieve a significant performance improvement with minimal computational cost. These methods work synergistically to achieve the seemingly contradictory but highly valuable technical effect of significantly improving accuracy while reducing model complexity and computational load, breaking through the bottleneck of the difficulty in achieving both accuracy and efficiency in existing technologies.

[0057] Detailed experimental data and comparison tables for each example Table 1: Performance comparison of the method of the present invention and existing technologies on the VEDAI dataset.

[0058]

[0059] Note: The same dataset partitioning and training strategy were used in the experiments. Small targets were defined as having a pixel area less than 32×32. This invention significantly improves the mean accuracy (mAP) and small target detection accuracy while maintaining low computational complexity.

[0060] Table 2: Robustness test of the method of the present invention under different environmental conditions (DroneVehicle dataset)

[0061] Note: This table verifies that the present invention has significantly stronger robustness in harsh environments through cross-modal dynamic fusion.

[0062] Table 3: Evaluation of the Lightweight Model Effect (Edge Device Deployment)

[0063] Note: The inference speed test platform was an NVIDIA Jetson Xavier NX. The results show that, after model pruning and INT8 quantization, this invention achieves high frame rate real-time detection on embedded devices.

[0064] This invention provides an efficient, robust, and practical cross-modal small target detection solution through a series of interconnected and synergistically innovative module designs and system architecture optimizations.

[0065] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be covered by the claims of the present invention.

Claims

1. A method for small target detection enhanced by cross-modal image information fusion, characterized in that, Includes the following steps: S1. Construct and preprocess an RGB-T multimodal dataset, which includes spatially aligned pairs of visible light images and infrared thermal images; S2. The preprocessed visible light image and infrared thermal imaging image are respectively input into a dual-stream backbone network for feature extraction to obtain their respective multi-scale feature maps; the dual-stream backbone network introduces a micro-domain enhanced convolutional module (MECM) in each feature extraction stage to enhance the feature representation capability of small targets; S3. The visible light feature map and infrared feature map output by the dual-stream backbone network at a specific level are fused to obtain a cross-modal fused feature map; the fusion is achieved by the cross-modal feature coupling module CMFCM, which is used to adaptively interact and weight the features of the two modes; S4. Input the cross-modal fused feature map into the neck network, and perform multi-scale feature fusion and enhancement through the feature pyramid structure to obtain the enhanced multi-scale feature map; S5. Input the enhanced multi-scale feature map into the detection head and output the target's category, confidence level, and location information; The detection method evaluates detection accuracy by calculating the Intersection over Union (IOU), Precision (P), and Recall (R) between the predicted and ground truth bounding boxes, and finally calculates the Average Precision (AP) as a performance metric. The calculations of IOU, P, and R are based on the following relationships: In the formula: Area_b∩Area_obj represents the intersection area of ​​the predicted bounding box and the ground truth bounding box; Area_b∪Area_obj represents the area of ​​the union of the predicted bounding box and the ground truth bounding box; P=TP / (TP+FP); P=TP / (TP+FN); In the formula: TP, FP, and FN represent the number of true positives, false positives, and false negatives, respectively.

2. The small target detection method with cross-modal image information fusion enhancement according to claim 1, characterized in that, The structure of the Micro-Domain Enhanced Convolutional Module (MECM) includes: A 1×1 convolutional layer is used to expand the number of channels in the input feature map; At least two parallel attention bottleneck branches are used to extract features with different levels of attention. One branch integrates the SE attention mechanism, and the other branch integrates the CBAM attention mechanism. A feature concatenation and fusion layer is used to concatenate the features output from each branch and perform channel compression and fusion through 1×1 convolution; A skip connection is used to add the module's input to the fused output.

3. The small target detection method with cross-modal image information fusion enhancement according to claim 1, characterized in that, The cross-modal feature coupling module (CMFCM) includes: Feature splitting unit, used to split the input pre-fusion features into at least two branches; A multi-scale local perception branch is used to capture local contextual information of small targets by dividing non-overlapping image patches of different sizes; At least one bottleneck structure branch is used to expand the feature receptive field through stacked convolutional layers to obtain global spatial context information; The feature fusion unit is used to concatenate the output features of the multi-scale local perception branch and the bottleneck structure branch in the channel dimension, and perform channel compression and integration through a convolutional layer.

4. The small target detection method with cross-modal image information fusion enhancement according to claim 1, characterized in that, The specific layer mentioned in step S3 is the layer with a downsampling factor of 8 in the backbone network, namely layer P3; the neck network mentioned in step S4 also includes a multi-scale aggregation pooling module MAPM, which is used to perform attention weighting on features of different layers to enhance the response of small target regions; the detection head mentioned in step S5 includes multiple sub-detection heads for feature maps of different scales, and each sub-detection head integrates a micro-domain enhancement convolution module MECM.

5. The small target detection method with cross-modal image information fusion enhancement according to claim 1, characterized in that, The dual-stream backbone network adopts a lightweight design based on an inverted residual structure.

6. A small target detection device with cross-modal image information fusion enhancement, characterized in that, include: The image acquisition module is used to acquire visible light images and infrared thermal imaging images; The feature extraction module includes a dual-stream backbone network for extracting multi-scale features of the image, wherein the backbone network integrates a micro-domain enhanced convolutional module (MECM). The fusion module is used to adaptively fuse feature maps of two modalities at a specific level through the cross-modal feature coupling module CMFCM. The feature enhancement module is used to enhance the fused features at multiple scales through the feature pyramid structure and the multi-scale aggregation pooling module MAPM. The detection module is used to perform target detection based on the enhanced features and output the results.

7. The small target detection device with cross-modal image information fusion enhancement according to claim 6, characterized in that, The fusion module is a cross-modal feature coupling module (CMFCM), which supports block perception and multi-scale feature fusion. The feature enhancement module includes a micro-domain enhanced convolution module (MECM) and a multi-scale aggregated pooling module (MAPM); the detection device is deployed in a drone or an embedded edge device.

8. The method or apparatus according to claim 1 or 6, characterized in that, The detection performance of the method or device is quantitatively evaluated by the average precision (AP), which is obtained by taking the area of ​​a PR curve plotted with recall (R) on the x-axis and precision (P) on the y-axis. The calculation method is as follows: In the formula: K represents the number of points taken in the interval; the value of AP is between [0,1]; Pc(k)Rc(k) is the curve value of point k.

9. A method for small target detection enhanced by cross-modal image information fusion, characterized in that, Includes the following steps: S1. In each feature extraction stage of the dual-stream backbone network, the input feature map is enhanced using the Micro-Domain Enhanced Convolutional Module (MECM), specifically including: S1.1 uses a 1×1 convolutional layer to expand the number of channels in the input feature map from C to 2C; S1.2 The expanded feature map is evenly divided into two branches along the channel dimension, and input to the bottleneck branch Bottleneck_S integrating the SE attention mechanism and the bottleneck branch Bottleneck_CBAM integrating the CBAM attention mechanism, respectively. S1.3 concatenates the features output from the two branches, and then performs channel compression and fusion through a 1×1 convolutional layer to restore the number of channels C. S1.4 adds the module input to the fused output through skip connections to obtain the enhanced feature map; S2. At the layer with a downsampling factor of 8 in the backbone network, i.e., layer P3, the RGB and Thermal bimodal feature maps are concatenated and input into the cross-modal feature coupling module CMFCM for adaptive fusion. S3. Input the fused cross-modal feature map into the neck network, and perform feature enhancement and multi-scale aggregation through the multi-scale aggregation pooling module MAPM; S4. Input the enhanced multi-scale feature map into the detection head, perform target detection, and output the detection results.

10. The small target detection method with cross-modal image information fusion enhancement according to claim 9, characterized in that, The cross-modal feature coupling module (CMFCM) includes the following steps: S2.1 divides the input feature map before fusion into at least two branches through a splitting operation; One branch of S2.2 is the multi-scale local perception branch, Patch-Aware, which is used to capture local contextual information of small targets by dividing non-overlapping image patches of different sizes. Specifically, it includes: S2.2.1 Divide the feature map into 4×4 non-overlapping blocks for coarse-grained local context extraction; S2.2.2 Divide the feature map into 2×2 non-overlapping blocks for fine-grained local detail extraction; Another branch of S2.3 is the bottleneck structure branch BottleneckC, which expands the receptive field by stacking convolutional layers to obtain global spatial context information; S2.4 The features output by the multi-scale local perception branch and the bottleneck structure branch are concatenated in the channel dimension; S2.5 performs channel compression and integration on the concatenated features through convolutional layers, and outputs a fused cross-modal feature map.

Images