Lightweight small target detection method and system based on dual-modal channel reconstruction and adaptive fusion

Abstract

The application discloses a kind of light weight small target detection method and system based on bimodal channel reconstruction and adaptive fusion, the method includes: in RGB feature branch and IR branch respectively constructs a new light weight convolution module, and then obtains double branch light weight convolution module;Respectively extract visible light image and infrared image, input the obtained RGB and IR feature map to multimodal channel reconstruction and adaptive fusion module, generate bimodal fusion features for target detection;Introduce SD loss to construct total loss function to train light weight small target detection model, obtain final detection result for image test;The application can accurately extract small target features under different scales while maintaining light weight, highlight key target features in multimodal, suppress redundant information within modal, enhance the detection accuracy and efficiency of model to multimodal target in complex scene.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a lightweight small target detection method and system based on dual-modal channel reconstruction and adaptive fusion. Background Technology

[0002] In recent years, with the rapid popularization of low-altitude remote sensing platforms such as drones and satellites, as well as edge computing devices, the application of infrared and visible light fused images has become increasingly widespread in scenarios such as disaster relief, military reconnaissance, power line inspection, and smart healthcare. In remote sensing imagery, visible light images, with their higher spatial resolution and rich texture and color information, provide a wealth of semantic cues for small target detection. However, the acquisition of visible light images is limited by sensor imaging mechanisms and environmental factors, such as nighttime, fog, haze, and strong light / shadow conditions, leading to a severe loss of texture information. In contrast, infrared images can highlight thermal radiation targets, but they suffer from low spatial resolution, blurred edges, and insufficient signal-to-noise ratio. How to effectively fuse the complementary information of infrared and visible light while simultaneously ensuring reliable and efficient detection of extremely small targets in remote sensing scenarios has become a critical challenge that urgently needs to be addressed.

[0003] Due to the rapid development of deep learning technology, convolutional neural network-based methods have made significant progress in image fusion detection. These methods mainly fall into two categories: image-level fusion detection and feature-level fusion detection. Image-level fusion detection first fuses visible light and infrared images using techniques such as contrast, weighting, sparse representation, or autoencoders, and then feeds the fused result into a single-modal detection network. This type of method fails to efficiently reduce channel sparsity for small targets during the fusion stage, leading to feature redundancy and loss of key information. Feature-level fusion detection, on the other hand, directly fuses features extracted from the image using convolutional neural networks, aiming for better spatial consistency. Although these methods have achieved certain accuracy gains, most methods generally stack convolutional or Transformer structures in their model design, increasing the number of model parameters and computational complexity, making it difficult for the models to run in real-time on platforms with limited computing power, such as drones and handheld terminals. Therefore, balancing lightweight model design and detection accuracy has become a key challenge for multimodal fusion models. Summary of the Invention

[0004] To address the shortcomings of existing methods in remote sensing image detection in terms of accuracy, detection efficiency, and adaptability to practical deployment, this paper proposes a lightweight small target detection method and system based on multimodal channel reconstruction and adaptive fusion. This achieves the technical effects of integrating the advantages of each of the dual-branch lightweight modules, efficiently combining multimodal information, optimizing the loss function, and balancing model lightweighting and accuracy.

[0005] According to a first aspect of the present invention, a lightweight small target detection method based on dual-modal channel reconstruction and adaptive fusion is provided, the method comprising: A network framework containing RGB feature branches and IR feature branches is selected. The original C3K2 module in the RGB feature branch is fused with the detail enhancement convolution module to form the first lightweight convolution module. The original C3K2 module in the IR feature branch is fused with the windmill convolution module to form the second lightweight convolution module, resulting in a dual-branch lightweight convolution module. Visible light and infrared images are input into the aforementioned dual-branch lightweight convolution module, and the extracted RGB and IR feature maps are input into the multimodal channel reconstruction and adaptive fusion module to generate dual-modal fusion features for target detection. The dual-modal fusion features are input into a lightweight multimodal small target detection model to obtain the final target detection result; wherein, the detection model is trained using a total loss function of weighted fusion CloU loss and SD loss.

[0006] Furthermore, before extracting the RGB and IR feature maps, the method also includes dimensional alignment of the visible light image and the infrared image.

[0007] According to a second aspect of the present invention, a lightweight small target detection system based on dual-modal channel reconstruction and adaptive fusion is provided, comprising: The dual-branch lightweight convolutional module is used to select a network framework containing RGB feature branches and IR feature branches. The original C3K2 module in the RGB feature branch is fused with the detail enhancement convolutional module to form the first lightweight convolutional module; the original C3K2 module in the IR feature branch is fused with the windmill convolutional module to form the second lightweight convolutional module, thus obtaining the dual-branch lightweight convolutional module. The feature extraction and fusion module is used to input visible light images and infrared images into the aforementioned dual-branch lightweight convolution module, and input the extracted RGB and IR feature maps into the multimodal channel reconstruction and adaptive fusion module to generate dual-modal fusion features for target detection. The loss function optimization module is used to input the dual-modal fusion features into a lightweight multimodal small target detection model to obtain the final target detection result; wherein, the detection model is trained using a total loss function of weighted fusion CloU loss and SD loss.

[0008] Compared with existing technologies, this invention has the following significant advantages: 1. By introducing detail enhancement convolution and windmill convolution to lighten the convolution module, it can accurately extract rich texture and color information in visible light images and improve the extraction effect of low-level features of infrared small targets; 2. Through the multimodal channel reconstruction and adaptive fusion module, the information contribution of visible light and infrared modes is dynamically balanced, solving the heterogeneity problem of inconsistent channel numbers and spatial resolution differences between visible light and infrared images, while efficiently weakening redundant information in multimodal modes; 3. By constructing a weighted fusion loss function based on SD loss, the interference of infrared small target bounding box IoU fluctuations on the regression process is effectively reduced, and the localization error of small targets in complex backgrounds is significantly reduced, enhancing detection accuracy and training stability.

[0009] Furthermore, by setting standard data augmentation strategies, optimizers, and dynamic learning rate scheduling, we ensure that the model can maintain high accuracy under lightweight design, and the experimental results on the LLVIP and RGBT-Person datasets also verify the effectiveness of the method. Attached Figure Description

[0010] Figure 1 A schematic diagram of a lightweight small target detection method based on dual-modal channel reconstruction and adaptive fusion provided in an embodiment of the present invention;

[0011] Figure 2 A schematic diagram of a lightweight small target detection system architecture based on dual-modal channel reconstruction and adaptive fusion provided for an embodiment of the present invention;

[0012] Figure 3 This is a schematic diagram of the DACS-Det structure provided in an embodiment of the present invention;

[0013] Figure 4 This is a schematic diagram of the C3K2_DEConv structure provided in an embodiment of the present invention;

[0014] Figure 5 This is a schematic diagram of the C3K2_DPConv structure provided in an embodiment of the present invention;

[0015] Figure 6 The schematic diagram provided for the embodiments of the present invention shows the PR curve of the improved lightweight multimodal small target detection model on LLVIP;

[0016] Figure 7 The schematic diagram provided for the embodiments of the present invention shows the PR curve of the improved lightweight multimodal small target detection model on RGBT-Person. Detailed Implementation

[0017] Existing multimodal small target detection methods in remote sensing image detection suffer from shortcomings in accuracy, detection efficiency, and practical deployment adaptability. Specifically, image-level fusion detection fails to adequately consider the channel sparsity of small targets, leading to redundant fusion features and loss of key information; feature-level fusion detection often introduces a large amount of redundant computation due to direct channel superposition or global attention strategies, and has a high number of model parameters and inference latency, making it difficult to run in real time on edge devices with limited computing power. To address these issues, this application proposes a lightweight small target detection method and system based on dual-modal channel reconstruction and adaptive fusion.

[0018] For ease of understanding, the following explains some key terms in this embodiment: The RGB feature branch refers to the network pathway used to process visible light image data and extract feature information from it. The IR feature branch refers to the network pathway used to process infrared image data and extract feature information from it. These two branches are typically set up in parallel in deep learning networks to capture image characteristics of different modalities respectively.

[0019] Example 1 See Figure 1 This embodiment provides a lightweight small target detection method based on dual-modal channel reconstruction and adaptive fusion, specifically including the following steps: S1. Select a network framework that includes RGB feature branches and IR feature branches. Merge the original C3K2 module in the RGB feature branch with the detail enhancement convolution module to form the first lightweight convolution module. Merge the original C3K2 module in the IR feature branch with the windmill convolution module to form the second lightweight convolution module, thus obtaining a dual-branch lightweight convolution module.

[0020] Specifically, when constructing the network framework, a structure containing RGB feature branches and IR feature branches can be selected. For example, YOLOv11 can be used as the baseline detection network framework, which contains a dual-path backbone, divided into RGB feature branches and IR feature branches, processing visible light images and infrared images respectively. The two branches are isomorphic but have independent parameters. Each branch can consist of a series of standard convolutional layers, pooling layers, and activation functions to independently extract features of its respective modality.

[0021] In the cross-stage local network of the RGB feature branch, the original C3K2 module is merged with the detail enhancement convolution module to form the first lightweight convolution module. The detail enhancement convolution consists of five parallel convolutions: one ordinary convolution and four split convolutions. The four split convolutions specifically include CDC (Central Difference Center), ADC (Angle Difference Center), HDC (Horizontal Difference Center), and VDC (Vertical Difference Center). Each branch extracts the central difference, horizontal gradient, vertical gradient, diagonal gradient, and original detail information through different parameterized weights, capturing high-frequency details at different gradient levels. The five features are then fused element-wise to generate a feature enhancement map with rich texture details, effectively preserving the detailed texture features in the RGB image.

[0022] The first lightweight convolutional module includes: a first convolutional layer, a second C3K2_DEConv layer, a third splicing layer, a fourth convolutional layer, a fifth connecting layer, a sixth convolutional layer, a seventh connecting layer, an eighth SPPF layer, and a ninth C2PSA layer.

[0023] The introduction of detail-enhancing convolutions effectively solves the vanishing gradient problem in deep network training, enabling the model to construct deeper feature extraction paths and learn more complex and abstract feature representations without performance degradation. This is crucial for extracting subtle but critical texture details of small targets from visible light images. Feature attention blocks allow the network to dynamically learn and emphasize regions or channels in the RGB feature map that are more informative for object detection, ensuring that even minute texture and edge information is effectively preserved and highlighted, and given higher priority in subsequent multimodal fusion stages, thus significantly improving detection accuracy.

[0024] Simultaneously, in the cross-stage local network of the IR feature branch, the original C3K2 module and the windmill convolution module in the IR feature branch are fused to form a second lightweight convolutional module. This can be achieved by fusing a standard convolutional layer with a simple spatial attention module. This spatial attention module can be a module that generates a spatial weight map through global average pooling and fully connected layers, used to highlight high-response regions in the infrared image. For example, the output of a C3K2 module can be multiplied and fused with the output of a channel attention module to improve the feature extraction efficiency of the infrared image to some extent. Through the above methods, a dual-branch lightweight convolutional module for parallel processing of visible light and infrared images can be obtained.

[0025] The second lightweight convolutional module includes: a first convolutional layer, a second C3K2_DPConv layer, a third concatenation layer, a fourth convolutional layer, a fifth connection layer, a sixth convolutional layer, a seventh connection layer, an eighth SPPF layer, and a ninth C2PSA layer. The first layer of the windmill convolutional module performs parallel convolution operations. Through concatenation, normalization, and convolutional kernels with channel attention mechanisms, it analyzes the contributions of different convolutional directions to form a multi-dimensional feature enhancement map.

[0026] This embodiment employs the mature and efficient YOLOv11 as the baseline framework and integrates PAN-FPN (Path Aggregation Network-Feature Pyramid Network) to achieve robust multi-scale feature aggregation. This is crucial for detecting small targets of varying sizes in remote sensing scenarios, ensuring that targets of all sizes can be effectively captured at the appropriate feature level. The dual-stream isomorphic but parameter-independent backbone network design allows for parallel and independent feature extraction of visible and infrared modalities, better adapting to the characteristics of each modality. The introduction of specific layer structures such as SPPF (Spatial Pyramid Pooling Fast) and C2PSA (CSPNet with 2Conv and SPP-like Attention) in shallow cross-stage local networks further optimizes feature aggregation and representation capabilities, improving the accuracy and efficiency of capturing small target features while maintaining a lightweight model.

[0027] S2. Input the visible light image and infrared image into the aforementioned dual-branch lightweight convolution module, and input the extracted RGB and IR feature maps into the multimodal channel reconstruction and adaptive fusion module to generate dual-modal fusion features for target detection.

[0028] Specifically, the visible light image and the infrared image are first aligned in dimensions to ensure the consistency of the two modal images in terms of spatial size. Then, RGB and IR feature maps are extracted from the visible light image and the infrared image, respectively. The extracted RGB and IR feature maps are input into the multimodal channel reconstruction and adaptive fusion module to generate dual-modal fusion features for target detection.

[0029] S3. Input the dual-modal fusion features into the lightweight multimodal small target detection model to obtain the final target detection result; wherein, the detection model is trained using the total loss function of weighted fusion CloU loss and SD loss.

[0030] Specifically, when training a lightweight multimodal small object detection model, the overall loss function can be optimized using a weighted fusion of CloU loss and SD loss. The CloU loss focuses on the position and shape matching of the bounding box, while the SD loss is specifically optimized for small object detection. A warm-up strategy is introduced during the model training phase to stabilize the training process by gradually increasing the learning rate, while data augmentation strategies are used to improve the model's generalization ability. After training, the fusion weights are saved, resulting in a lightweight multimodal small object detection model. The dual-modal fusion features are input into the trained model, and after steps such as feature extraction, object localization, and classification, the final object detection result is output, including the object's location coordinates, category, and confidence information.

[0031] In the early stages of training, the learning rate is gradually increased linearly from a small value to a preset value. This helps to avoid model instability or getting stuck in local optima due to an excessively high learning rate in the early stages of training, especially for learning sensitive features such as small targets, which allows for smoother convergence.

[0032] S4. Export the trained lightweight multimodal small target detection model to ONNX format, perform INT8 quantization using TensorRT, and provide a visualization interface to overlay the fused feature map, SD loss weights, and detection results for quick location of false alarms and missed detection regions.

[0033] Specifically, ONNX (Open Neural Network Exchange) provides an open model representation format, enabling seamless migration and deployment of models across different deep learning frameworks and hardware platforms, greatly simplifying the process from model training to practical application. TensorRT INT8 quantization converts model weights and activation values ​​from floating-point numbers to lower-precision integers, significantly reducing model size and memory footprint. More importantly, on hardware supporting INT8 computation, inference speed can be greatly improved. This is crucial for achieving real-time, efficient small target detection on edge devices with limited computing power, such as drones and mobile robots. Visualization capabilities are provided, allowing developers and operators to intuitively view the fused feature maps, SD loss weights, and final detection results within the model. This is invaluable for debugging models, understanding their behavior in specific scenarios, and quickly identifying and correcting false alarms and missed detections in real-world deployments, thereby improving system reliability and availability.

[0034] The method proposed in this application, by constructing a lightweight dual-branch convolutional module, achieves accurate extraction of multi-scale small target features in visible and infrared images, and effectively suppresses intramodal redundant information. Simultaneously, the multimodal channel reconstruction and adaptive fusion module efficiently combines the advantages of dual-modal information, reducing the loss of key information during the fusion process. Furthermore, by optimizing the loss function, the model's localization accuracy and training stability for small targets in complex backgrounds are improved. Therefore, this method significantly improves the accuracy and efficiency of multimodal small target detection in remote sensing scenarios while maintaining a lightweight model, meeting the needs of edge device deployment.

[0035] In some of the solutions described above in this application, it is further proposed that, before extracting RGB and IR feature maps, the method further includes: dimensional alignment of the visible light image and the infrared image.

[0036] Dimension alignment refers to the preprocessing of input data from different modalities to achieve consistency in key dimensions when processing multimodal data. This operation aims to eliminate the inherent heterogeneity of different modalities, providing a unified and compatible data foundation for subsequent feature extraction and fusion. Specifically, dimension alignment can be achieved in several ways. For channel alignment, it can be achieved by adding zero-padding channels, copying existing channels, or using 1x1 convolutional layers for channel projection, mapping images with different channel numbers to the same channel space. This module can dynamically adjust its spatial size and number of channels based on the original dimensions of the input image. This approach can more flexibly adapt to the characteristics of different modalities and achieve efficient dimensionality unification. As a key preprocessing step before feature extraction, dimension alignment ensures that subsequent dual-branch lightweight convolutional modules receive structurally consistent inputs, thereby avoiding feature extraction bias and information loss caused by dimensionality mismatch.

[0037] The above technical solution involves dimensional alignment of visible light and infrared images before feature extraction from the dual-branch lightweight convolutional module. This preprocessing step effectively addresses the inherent differences in channel count and spatial size between visible light and infrared images. By unifying the dimensions of the input images, it ensures that the subsequent RGB and IR feature branches can perform efficient and accurate feature extraction based on standardized and compatible input when processing their respective modal data. This not only avoids information distortion and computational redundancy caused by dimensionality mismatch but also lays a solid foundation for the accurate fusion of multimodal channel reconstruction and adaptive fusion modules, thereby significantly improving the detection accuracy and robustness of the lightweight multimodal small target detection model for small targets in complex scenes.

[0038] In some of the embodiments described above in this application, a specific structure of the first lightweight convolutional module is further proposed, which includes: a first convolutional layer, a second C3K2_DEConv layer, a third splicing layer, a fourth convolutional layer, a fifth connection layer, a sixth convolutional layer, a seventh connection layer, an eighth SPPF layer, and a ninth C2PSA layer; wherein, the detail enhancement convolutional module consists of one ordinary convolution and four split convolutions, and the four split convolutions consist of CDC central difference, ADC angular difference, HDC horizontal difference, and VDC vertical difference.

[0039] Specifically, the first lightweight convolutional module, as a whole, aims to perform efficient and lightweight feature extraction from the input of the RGB feature branch. By integrating multiple functional layers, it enhances the ability to capture key details in visible light images while maintaining computational efficiency. The first convolutional layer performs preliminary feature extraction and dimensionality adjustment on the input RGB image or feature map. Its role is to capture low-level features of the image, such as edges and textures, and provide basic input for subsequent complex feature processing. This layer can use standard convolution operations, combined with stride and padding operations, to control the spatial size and number of channels of the feature map. The second C3K2_DEConv layer is one of the core parts of the first lightweight convolutional module. Its main function is to perform in-depth processing of features by introducing a detail enhancement convolutional module to more effectively capture and enhance the details in visible light images. While maintaining the efficiency advantages of the C3K2 structure, this layer compensates for the shortcomings of traditional convolution in processing fine textures and edges through the detail enhancement convolutional module, thereby improving the richness and expressive power of features. The third concatenation layer merges feature maps from different paths or stages along the channel dimension. Its role is to integrate multi-source information to retain more comprehensive contextual information and prevent information loss during deep network propagation. The fourth convolutional layer primarily compresses or adjusts the channel dimension of the concatenated feature maps. Its purpose is to reduce the number of channels in the feature maps, thereby reducing the complexity and number of parameters in subsequent calculations. It can also serve as a feature selection tool, focusing on more discriminative features. This layer is typically implemented using a 1x1 convolutional kernel. The fifth, sixth, and seventh connecting layers work together to optimize the feature flow path, ensuring unimpeded gradient movement within the network and further refining feature representations. The fifth and seventh connecting layers enable feature reuse or skip connections, helping to alleviate the vanishing gradient problem and allowing the network to learn features at different levels more effectively. The sixth convolutional layer performs further feature transformations and nonlinear mappings between connections to enhance the model's expressive power. The eighth SPPF layer aggregates multi-scale contextual information through pooling operations at different scales. Its function is to fuse receptive field information of different sizes into a fixed-size feature vector, thereby enhancing the model's robustness to changes in target scale and effectively expanding the receptive field of features to capture broader contextual information. This layer typically contains multiple parallel max-pooling operations with different pooling kernel sizes, and then concatenates the pooling results. The ninth C2PSA layer is used to introduce an attention mechanism to dynamically adjust the importance of different feature channels or spatial locations. Its function is to enable the model to pay more attention to key regions or features in the image, suppress redundant information, thereby improving the discriminative power of features and the detection accuracy of the model.

[0040] The detail enhancement convolution module is a core component of the second C3K2_DEConv layer, used to capture and enhance fine textures, edges, and high-frequency details in visible light images. This module, by processing different types of convolution operations in parallel, can extract features from multiple dimensions, overcoming the limitations of traditional single convolution kernels in capturing complex details and significantly improving feature richness and expressiveness. The ordinary convolution, as one path in the detail enhancement convolution module, is mainly responsible for extracting the image's intensity-level basic features. It captures basic visual information such as brightness and color through standard convolution operations, providing a basic feature representation for subsequent detail enhancement. The four-way split convolution is a key innovation of the detail enhancement convolution module; they work in parallel, each focusing on capturing high-frequency detail information in different dimensions. The CDC (Central Difference Center) is used to extract the central difference information of the feature map, effectively capturing subtle changes and textures within the target. The ADC (Angle Difference Center) is used to capture the gradient information in the diagonal direction of the feature map, helping to identify edges and textures with specific angular tendencies. The HDC (Horizontal Difference Center) is used to extract the horizontal gradient information of the feature map, effectively enhancing horizontal edges and lines in the image. Vertical differential (VDC) is used to extract vertical gradient information from feature maps, effectively enhancing vertical edges and lines in images. These split convolutions, through differential operations, can more precisely capture high-frequency gradient details that are easily overlooked by traditional convolutions, thus significantly enhancing feature richness and sensitivity to small targets.

[0041] Through the above technical solution, the first lightweight convolutional module proposed in this application effectively solves the problem of insufficient feature extraction in the RGB branch. The introduction of this first lightweight convolutional module enables the RGB branch to efficiently and accurately extract key details such as edges and textures of small targets, providing high-quality and rich feature inputs for subsequent multimodal fusion and small target detection, thereby significantly improving the accuracy and robustness of the overall detection method.

[0042] In some of the embodiments described above in this application, a lightweight small target detection method is further proposed, wherein the second lightweight convolutional module comprises: a first convolutional layer, a second C3K2_DPConv layer, a third splicing layer, a fourth convolutional layer, a fifth connection layer, a sixth convolutional layer, a seventh connection layer, an eighth SPPF layer, and a ninth C2PSA layer.

[0043] Specifically, the first convolutional layer is used to perform preliminary feature extraction on the input infrared image, laying the foundation for subsequent processing. This layer can use, for example, a 1x1 convolutional kernel, and the size and number of channels of the feature map can be controlled by adjusting the stride and padding. It can also be combined with batch normalization and activation functions to enhance the non-linear expressive power of the features. The second C3K2_DPConv layer is one of the core parts of this application. It combines a windmill convolution module and is specifically used to capture multi-dimensional directional gradient features in the infrared image to enhance the extraction of thermal radiation contours. This layer retains the CSP double branch, cross-stage stitching, and Bottleneck stacking, and cascades PConv at the end of the feature fusion to form a multi-dimensional enhanced feature extraction module. The third stitching layer is used to fuse feature maps from different branches or stages, usually stitching along the channel dimension to integrate multi-source information and enrich feature representation. The fourth convolutional layer further compresses and refines the fused features after stitching, and can use a 1x1 convolutional kernel to adjust the number of channels. The fifth connecting layer aims to ensure the smooth transmission of feature information, maintain gradient continuity, and prevent information loss in deep networks. The sixth convolutional layer further optimizes feature representation and enhances detail preservation. The seventh connecting layer further strengthens feature reuse and avoids target information loss due to excessive convolution. The eighth SPPF layer expands the receptive field and extracts multi-scale contextual information through fast spatial pyramid pooling, which is crucial for detecting small targets of different sizes. The ninth C2PSA layer introduces an attention mechanism, combining the C2 module with the PSA mechanism, enabling the network to focus on key regions and information in infrared images, suppress noise interference, and thus improve the expressive power of small target features.

[0044] This application addresses the problem of insufficient feature extraction caused by a lack of specific structure in infrared feature extraction by defining the specific layer structure of the second lightweight convolutional module. While maintaining lightweight design, it significantly optimizes the extraction quality of infrared features, providing high-quality infrared feature input for subsequent multimodal fusion, thereby improving the performance of the entire lightweight small target detection method in complex scenes.

[0045] In some of the above-mentioned solutions in this application, a further proposed method is proposed, which includes: using the adaptive fusion layer in the multimodal channel reconstruction and adaptive fusion module to perform global average pooling on the extracted RGB and IR feature maps respectively to obtain channel-level global statistical vectors, using channel-level Softmax operation to generate attention weights, and then weighted and fused to obtain the final bimodal fusion feature.

[0046] Specifically, the adaptive fusion layer in the multimodal channel reconstruction and adaptive fusion module is a key component for achieving dynamic balance of multimodal information. This layer aims to overcome the limitations of traditional fixed fusion methods by introducing an adaptive mechanism, enabling the fusion process to dynamically adjust according to the actual content and importance of the input features. Global average pooling is performed on the extracted RGB and IR feature maps respectively, aiming to extract channel-level global contextual information from the feature map of each modality. Global average pooling is an operation that averages all pixel values ​​across the spatial dimension of each channel in the feature map, thereby compressing the feature map of each channel into a single value. This helps capture the overall activation intensity of each channel, thus reflecting the importance of that channel in the entire feature map.

[0047] Global average pooling yields channel-level global statistical vectors. These vectors, serving as a generalized representation of the global information for each channel, quantify the overall response strength of each channel, providing the foundation for subsequent attention weight generation. Channel-level softmax operations are then used to generate attention weights, transforming the channel-level global statistical vectors of different modalities into attention weights with relative importance. This allows the model to dynamically highlight modalities or channels more important for object detection and suppress redundant information.

[0048] Finally, the final bimodal fusion feature is obtained through weighted fusion. Weighted fusion refers to linearly combining or element-wise multiplying and summing the RGB and IR feature maps according to the generated attention weights to generate a single fusion feature containing information from both modalities. The aim is to integrate the modal information with attention weights to form a more discriminative and robust bimodal fusion feature for subsequent object detection tasks.

[0049] In some of the solutions described above in this application, the method is further proposed to include: setting a model training strategy that includes at least standard data augmentation, warmup, and Mosaic data augmentation strategies.

[0050] Specifically, standard data augmentation aims to increase the diversity of training samples by applying a series of random transformations to the original training data, thereby improving the model's generalization ability and robustness. For example, operations such as random horizontal flipping, random vertical flipping, random rotation, random scaling, random cropping, or color jitter can be used. These transformations simulate changes in the target under different viewpoints, lighting, and backgrounds, enabling the model to learn more invariant feature representations and reducing its dependence on a specific training data distribution.

[0051] Warmup is a learning rate scheduling mechanism. Its core idea is to gradually increase the learning rate linearly or non-linearly from a very small value to a preset initial learning rate during the early stages of training. This aims to avoid drastic fluctuations in model parameters due to a large learning rate at the beginning of training, which could potentially lead to local optima or training instability. Through warmup, the model can start the training process more smoothly, laying a good foundation for subsequent optimization.

[0052] Mosaic data augmentation is an advanced data augmentation technique that generates new training samples by stitching multiple training images together into a single image. This strategy effectively increases the number of targets in a training batch while introducing richer contextual information and multi-scale target distribution. Mosaic data augmentation is particularly suitable for small target detection tasks because it can present more small target instances in a single training sample, thereby improving the model's ability to perceive and locate small targets.

[0053] By introducing the aforementioned model training strategies, this application effectively addresses the problems of unstable model convergence, insufficient generalization ability, and inadequate learning of features for extremely small targets in small target detection, caused by the target's small size, susceptibility to noise interference, and uneven data distribution. These strategies work synergistically to construct a more stable and efficient training framework, enabling the lightweight small target detection model to significantly enhance detection accuracy and training stability while maintaining its lightweight characteristics. Especially when dealing with small targets in complex backgrounds, it effectively reduces localization errors and improves overall detection performance.

[0054] Example 2

[0055] Traditional multimodal small target detection methods have shortcomings in accuracy, detection efficiency, and practical deployment adaptability in remote sensing image detection. Specifically, image-level fusion detection fails to fully consider the channel sparsity of small targets, resulting in redundant fusion features and loss of key information; feature-level fusion detection often introduces a large amount of redundant computation due to direct channel superposition or global attention strategies, and has a high number of model parameters and inference latency, making it difficult to run in real time on edge devices with limited computing power.

[0056] For this, please refer to Figure 2 This application proposes a lightweight small target detection system based on dual-modal channel reconstruction and adaptive fusion. The system includes a dual-branch lightweight convolutional construction module, a feature extraction and fusion module, and a loss function optimization module.

[0057] The dual-branch lightweight convolutional module is used to select a network framework containing RGB feature branches and IR feature branches. The original C3K2 module in the RGB feature branch is fused with the detail enhancement convolutional module to form the first lightweight convolutional module; the original C3K2 module in the IR feature branch is fused with the windmill convolutional module to form the second lightweight convolutional module, thus obtaining the dual-branch lightweight convolutional module.

[0058] The feature extraction and fusion module is used to input visible light images and infrared images into the aforementioned dual-branch lightweight convolution module, and input the extracted RGB and IR feature maps into the multimodal channel reconstruction and adaptive fusion module to generate dual-modal fusion features for target detection.

[0059] The loss function optimization module is used to input the dual-modal fusion features into the lightweight multimodal small target detection model to obtain the final target detection result; the detection model is trained using the total loss function of weighted fusion CloU loss and SD loss.

[0060] By fusing the detail enhancement convolutional module and the C3K2 module in the RGB feature branch to form a first lightweight convolutional module, and fusing the windmill convolutional module and the C3K2 module in the IR feature branch to form a second lightweight convolutional module, the feature extraction process is optimized for different modal image characteristics, improving feature expressiveness while reducing the number of parameters. Specifically, the detail enhancement convolutional module effectively captures texture and edge details in visible light images, while the windmill convolutional module optimizes the thermal radiation distribution characteristics of targets in infrared images, improving the targeting and efficiency of feature extraction. Simultaneously, the multimodal channel reconstruction and adaptive fusion module dynamically balances the information contributions of visible light and infrared modalities through channel alignment and adaptive mechanisms, suppressing redundant information and highlighting key target features, thus solving the information imbalance problem in multimodal feature fusion. Furthermore, by using a weighted fusion of CloU loss and SD loss in the total loss function, the jitter phenomenon in the localization process of small targets is effectively suppressed, improving the model's localization accuracy and training stability in complex backgrounds.

[0061] Through the above technical solutions, this application significantly improves the accuracy and efficiency of small target detection in infrared and visible light fusion scenarios while maintaining the lightweight nature of the model, thus meeting the real-time deployment needs of computing-constrained platforms such as drones and mobile robots.

[0062] In some of the solutions described above in this application, it is further proposed that the multimodal channel reconstruction and adaptive fusion module includes a dimension alignment layer, a transformation layer and an adaptive fusion layer.

[0063] Specifically, the dimension alignment layer consists of an RGB feature branch, an IR feature branch, and corresponding dimension alignment submodules. The RGB feature branch connects to the original RGB features, and the IR feature branch connects to the original IR features. A 1×1 convolutional layer is configured in each branch to unify the number of channels to C and to adjust the spatial dimension H. RGB ×W RGB With H ir ×W ir The standard is H×W, and the number of channels is controlled by compression ratio to reduce the amount of subsequent calculations.

[0064] The transformation layer provides a high-quality feature foundation for subsequent fusion. Differentiated feature extraction submodules are designed for the RGB and IR feature branches respectively. The RGB feature branch uses a GWC+PWC combination, while the IR feature branch uses a DWC+PWC+feature reuse unit combination. In the RGB feature branch, GWC captures local details of the visible light image through sparse channel connections, greatly reducing channel redundancy. PWC compensates for the channel fragmentation problem of GWC, ensuring the flow of dual-stream information across channels and preventing details from being interfered with by the original features. In the IR feature branch, DWC focuses on extracting the target thermal radiation contour of the infrared image, improving lightweight design. PWC compresses the number of channels and simultaneously enables the feature reuse unit to directly utilize the dimension-aligned original infrared features, avoiding excessive convolution that could lead to the loss of target contour information.

[0065] An adaptive fusion layer dynamically balances the information contributions of the visible light and infrared modes to generate the final bimodal fused features. This layer can intelligently adjust the fusion weights of different modal features based on the characteristics of the input features, avoiding information imbalance that may result from a fixed fusion strategy.

[0066] Through the above technical solutions, the dimension alignment layer within the multimodal channel reconstruction and adaptive fusion module effectively addresses the heterogeneity of visible light and infrared images in terms of channel count and spatial resolution, ensuring feature consistency before fusion and avoiding information misalignment due to dimensional differences. The transformation layer, by designing differentiated feature extraction sub-modules for different modalities, can specifically capture texture details in visible light images and thermal radiation contours in infrared images, thereby reducing redundant information within modalities and highlighting key target features. The adaptive fusion layer dynamically balances the information contributions of visible light and infrared modalities, avoiding modal information imbalance caused by traditional fixed fusion strategies, thus generating high-quality dual-modal fusion features. Overall, these layers work synergistically to significantly improve the fusion accuracy and efficiency of multimodal features while maintaining lightweight design, providing higher-quality input for subsequent lightweight multimodal small target detection models, thereby enhancing the model's ability to detect small targets in complex scenes.

[0067] Example 3 This embodiment provides a specific implementation method for a lightweight small target detection method based on dual-modal channel reconstruction and adaptive fusion. See reference [link to relevant documentation]. Figure 3 This includes the following steps: S1. Select YOLOv11, which includes RGB feature branches and IR feature branches, as the baseline detection network framework to construct a dual-stream infrared and visible light image fusion network.

[0068] S2. In the shallow cross-stage local network of the RGB feature extraction branch, please refer to Figure 4 The original C3K2 module in the RGB feature branch is merged with the detail enhancement convolutional module to form the first lightweight convolutional module C3K2_DEConv, which specifically includes a first convolutional layer, a second C3K2_DEConv layer, a third concatenation layer, a fourth convolutional layer, a fifth connection layer, a sixth convolutional layer, a seventh connection layer, an eighth SPPF layer, and a ninth C2PSA layer. The detail enhancement convolution consists of five parallel convolutions, including one ordinary convolution and four split convolutions. Each branch uses different parameterized weights to extract center differences, horizontal gradients, vertical gradients, diagonal gradients, and original details. The results of the five paths are then added together, and the output is a feature map with rich detail enhancement.

[0069] The feature output process of the section enhancement convolution is specifically extracted by five parallel convolutions. Among them, ordinary convolution extracts intensity-level basic features, and four differential convolutions, namely CDC central difference, ADC angular difference, HDC horizontal difference and VDC vertical difference, capture gradient-level high-frequency details in different dimensions respectively. The five features are fused by element-wise addition to obtain the final feature map with rich details as the output result.

[0070] For the first lightweight convolutional module C3K2_DEConv, its workflow is as follows: S201, Input Feature Map After being increased in dimension by 1×1 convolution and divided into two equal parts along the channel, we obtain ;in By sequentially passing through n Bottleneck or C3k submodules, cross-stage partial connections are completed, and features are output. ; S202, and The pieces are joined together along the channel to form a fusion feature. ; S203, Compressed to the target number of channels using 1×1 convolution. ,get The intensity and gradient information in four directions are extracted by five parallel convolutional kernels, and then the sums are processed by BN+SiLU to output the detail enhancement features. This is for use in subsequent Neck layers, fusion layers, and detection heads.

[0071] The output of the first lightweight convolutional module C3K2_DEConv is expressed by the formula: ; in, The proposed DEConv operation, F in T represents the features of the input. i=1：5 These represent the kernels of VC, CDC, ADC, HDC, and VDC, respectively. T represents the convolution operation. out This represents the transformed kernel, which combines parallel convolutions together.

[0072] S3. In the shallow cross-stage local network of the IR feature branch, the original C3K2 module in the IR feature branch is merged with the windmill convolution module to form a second lightweight convolution module block C3K2_DPConv. (See [link to documentation]). Figure 5 Specifically, it includes a first convolutional layer, a second C3K2_DPConv layer, a third splicing layer, a fourth convolutional layer, a fifth connecting layer, a sixth convolutional layer, a seventh connecting layer, an eighth SPPF layer, and a ninth C2PSA layer.

[0073] The second lightweight convolutional module block C3K2_DPConv, while retaining the original CSP's dual branches, cross-stage concatenation, and Bottleneck stacking, adds PConv at the end of the features to form a multi-dimensional enhanced feature extraction module. The specific workflow of the second lightweight convolutional module block C3K2_DPConv is as follows: S301, Input Feature Map The dimensionality is increased by 1×1 convolution and then divided into two equal parts along the channel; S302, obtained ,in, By sequentially passing through n Bottleneck or C3k submodules, cross-stage partial connections are completed, and features are output. ; S303, and The pieces are joined together along the channel to form a fusion feature. ; Compressed to the target number of channels using 1×1 convolution. ,get ; S304. Using four-way asymmetric zero-filling, fill maps P1-P4 are generated respectively; two 1×k convolution pairs P1 and P2 extract horizontal edges, and two k×1 convolution pairs P3 and P4 extract vertical edges, outputting four sets of directional feature maps. S305. Concatenate the four sets of directional features along the channels and fuse them through a 1×1 convolution to obtain the directional enhancement features. ; The output, after batch normalization and activation function, is used by subsequent network layers.

[0074] The output of the second lightweight convolutional module block C3K2_DPConv is expressed by the formula: ; ; ; ; in, , Input the width and height of x. It is the number of output channels for each convolutional layer. s represents the number of channels in the final output feature map of the PConv module, s represents the convolution stride; X, Xir, and W represent the input feature map, the feature map after concatenating the four branches, and the final fusion convolution kernel used to adjust the output size and number of channels, respectively; Cat, BN, and SiLU functions represent fusion, batch normalization, and activation functions, respectively.

[0075] The final result is a lightweight convolutional module with two branches.

[0076] S4. Input the visible light image and infrared image into the aforementioned dual-branch lightweight convolution module, and input the extracted RGB and IR feature maps into the multimodal channel reconstruction and adaptive fusion module to generate dual-modal fusion features for target detection.

[0077] S401. After extracting visible light and infrared features in the dual-stream backbone network, they are fed into the dimension alignment layer of the multimodal channel reconstruction and adaptive fusion module, respectively. The dimension alignment layer consists of two independent branches and corresponding dimension alignment sub-modules. The visible light branch receives the original visible light features, and the infrared branch receives the original infrared features. A 1×1 convolutional layer is configured in each branch to unify the number of channels in the multimodal system to C and the spatial size. and Unified as At the same time, the compression ratio controls the number of channels to reduce the amount of subsequent calculations.

[0078] S402, then transitions to the dedicated transformation layer within the multimodal channel reconstruction and adaptive fusion module to provide a high-quality feature foundation for subsequent fusion. A GWC+PWC combination is configured for the visible light branch, and a DWC+PWC+feature reuse unit combination is configured for the infrared branch. In the visible light branch, GWC captures local details of the visible light image through sparse channel connections, greatly reducing channel redundancy; PWC compensates for the channel fragmentation problem of GWC, ensuring the flow of dual-stream information across channels and preventing details from being interfered with by the original features; while in the infrared branch, DWC focuses on extracting the target thermal radiation contour of the infrared image, improving the lightweight nature; PWC compresses the number of channels, while enabling the feature reuse unit to directly utilize the dimension-aligned original infrared features, avoiding excessive convolution that could lose target contour information.

[0079] S403. Finally, the module's internal adaptive fusion layer dynamically balances the information contribution from the infrared modality. The specific process is as follows: right and Average pooling is performed to obtain the channel-level global statistical vector. and Then, these are stacked and subjected to a channel-level softmax operation to generate attention weights. and The final multimodal fusion features can be expressed by the formula: .

[0080] S5. Input the dual-modal fusion features into the lightweight multimodal small target detection model to obtain the final target detection result; wherein, the detection model is trained using the total loss function of weighted fusion CloU loss and SD loss.

[0081] S501. Combining the jitter suppression advantage of SD loss, a total loss function is constructed as a guide for training. Specifically, it is a weighted fusion total loss function based on CloU loss and SD loss, expressed by the formula: ; ; ; ; in, Indicates CloU loss, Indicates SD loss, Represents the empirical value coefficient. and These represent the prediction box and the truth box, respectively. p represents the aspect ratio penalty term. 2 / c 2 This indicates the center point distance penalty term. This represents the target area normalization coefficient. The default value is 0.5. and Indicates the weight.

[0082] S502. Set standard data augmentation and warm-up strategies, select optimizer and dynamic learning rate scheduling, and set the iteration rounds and batch size to train and save the model. In this embodiment, 640×640 pixels are used as the network input resolution; SGD with momentum is selected as the training optimizer, the initial learning rate is set to 0.01, the weight decay coefficient is 0.0005, the batch size is set to 16, the maximum number of training rounds is 300, and the convergence condition is reaching the preset maximum number of rounds; three rounds of linear warm-up are introduced during the training phase to slowly increase the learning rate and prevent gradient oscillations in the early stages of training. After training, the fusion weights are saved to obtain the lightweight multimodal small target detection model DACS-Det.

[0083] The experimental environment configured in this embodiment uses Python version 3.8.13. The computational experiments were conducted on a second-generation Intel Core i9-12900K processor (3.20 GHz) and an NVIDIA GeForce RTX A4000 GPU (16 GB VRAM). The CUDA version was 11.8, and the software environment was based on PyTorch 2.0.0.

[0084] The performance comparisons between the improved model and the original model are shown in Tables 1 and 2. Table 1 compares the experimental results on the publicly available LLVIP dataset under the same experimental conditions. Figure 6 As can be seen, the improved DACS-DET achieves improvements of 1.7%, 1.3%, and 3.2% in the three key metrics of Precision, mAP50, and mAP50-95, respectively, while having only 4.4M parameters and a computational cost of 12.0 GFLOPS. Table 2 also compares experimental results on the publicly available RGBT-Person dataset under the same experimental conditions. The improved DACS-DET achieves improvements of 3.4%, 4.1%, and 2.4% in the three key metrics of Precision, mAP50, and mAP50-95, respectively. Figure 7 It can be seen that the improved DACS-DET improves the accuracy of the person, rider and crowd categories by 5.2%, 2.5% and 4.4% respectively, further proving the performance of the improved model.

[0085] Table 1. Performance comparison of the improved model and the original model on the LLVIP dataset. Experimental methods Modal Precison mAP50 mAP50-95 FLOPs(G) Parameter(M) YOLO-Fusion RGB+IR 0.937 0.956 0.616 10.5 4.51 DACS-DET RGB+IR 0.954 0.969 0.648 12.1 4.41 Table 2. Performance comparison of the improved model and the original model on the RGBT-Person dataset. Experimental methods Modal Precison mAP50 mAP50-95 FLOPs(G) Parameter(M) YOLO-Fusion RGB+IR 0.711 0.737 0.323 10.5 4.51 DACS-DET RGB+IR 0.745 0.778 0.347 12.1 4.41 The proposed DACS-DET model was comprehensively compared with six existing state-of-the-art (SOTA) methods for fusion detection of classical visible and infrared light, and the results are shown in Table 3. The evaluation metrics used were mAP50 and mAP50-95, and the test data covered the publicly available LLVIP validation set. Under the same dual-modal image input conditions, the proposed DACS-DET model achieved an mAP50 of 0.969 and an mAP50-95 of 0.648, respectively outperforming the second-best methods by 0.6% and 1.3%. This demonstrates that the proposed method, while maintaining lightweight design, can not only accurately extract small target features at different scales in the infrared and visible light branches, but also highlight key target features in multimodal scenarios, suppress intramodal redundant information, and enhance the model's detection accuracy and efficiency for multimodal targets in complex scenes. This provides a novel multimodal small target detection method that balances detection accuracy, model lightweight design, and ease of practical edge deployment for subsequent multimodal data fusion research.

[0086] Table 3 Comparison of performance with SOTA algorithm algorithm Modal mAP50 mAP50-95 ICAFusion RGB+IR 0.873 0.534 PCMFNet RGB+IR 0.912 0.555 DPSNet RGB+IR 0.962 — CFT RGB+IR 0.963 0.635 ACDF RGB+IR 0.958 0.589 YOLOrs RGB+IR 0.943 0.553 DACS-DET(OURS) RGB+IR 0.954 0.969 Compared to traditional transformer models and two-stage detection frameworks such as Faster R-CNN, this invention achieves high efficiency in terms of both parameter count and computational complexity, with only 4.4M parameters and 12.0 GFLOPS of computation. At the same time, it balances the requirements of lightweight design and high accuracy on the LLVIP and RGBT-Person datasets, meeting the practical deployment needs of scenarios with limited computing power, such as drones and handheld terminals.

[0087] Example 4

[0088] This embodiment introduces a specific scenario: suppose a drone equipped with the DACS-Det system is carrying out a search and rescue mission in a forest at night, looking for a lost child.

[0089] The problems with existing technologies are as follows: First, due to insufficient light at night, visible light images are extremely dark, making it almost impossible to discern details such as a child's body and clothing, or they may even be completely obscured by darkness. Second, infrared images can capture a child's body temperature, making them appear as a heat source against the background. However, because children are small targets and far away, infrared images have low resolution and blurred edges, which may confuse the child's thermal signal with surrounding animals, heated rocks, or campfire embers, making accurate identification and location difficult. Simply fusing visible light and infrared images may introduce noise due to the lack of visible light information, or smooth out subtle features of the child during the fusion process, resulting in insufficient detection accuracy. To improve recall, models are often large and computationally expensive, which is difficult for the limited onboard computing units of drones to run in real time, leading to low search and rescue efficiency.

[0090] When a drone's visible light camera captures images of a forest at night, despite the overall darkness, the detail-enhancing convolutions of the C3K2_DEConv module strive to extract any possible minute texture and edge information from the dim light. For example, even if the color of a child's clothing is not obvious in the dark, its folds and subtle gradient changes in contours can still be captured by split convolutions such as CDC, ADC, HDC, and VDC, forming initial visual cues for the child's potential location. Simultaneously, the windmill convolutions of the C3K2_DPConv module efficiently extract the thermal radiation contours of the child's body. Due to its multidimensional enhancement properties, it can better distinguish the child's body features and differentiate them from other heat sources in the background, generating a clear infrared feature map.

[0091] The subtle texture features extracted by the RGB feature branch and the clear thermal contour features extracted by the IR feature branch are fed into the multimodal channel reconstruction and adaptive fusion module. In nighttime search and rescue scenarios, the adaptive fusion layer dynamically assigns higher attention weights to the infrared features based on the characteristics of the current image, providing more reliable target presence information. Simultaneously, it doesn't completely ignore RGB features but utilizes the subtle texture information captured by the RGB branch to supplement and refine the infrared features. For example, it helps distinguish a child's body from similarly shaped tree trunks in the background, or provides information about possible subtle reflections on a child's clothing. This intelligent weighted fusion ensures that the model maximizes the complementary advantages of the two modalities, generating a robust fused feature that includes both thermal source information and subtle visual details.

[0092] The fused feature maps are fed into a small object detection model for object recognition and localization. Because the detection model uses a jitter suppression total loss function combined with SD loss during training, it can more accurately predict the bounding boxes of child targets. SD loss is particularly helpful in suppressing jitter and instability during bounding box regression for small targets, such as children at a distance. Even if a child occupies only a small pixel area in the image, the model can still provide stable and high-precision localization results.

[0093] The technical solution provided by this invention enables drones to detect and accurately locate lost children in real time and in low-light forest environments, significantly improving search and rescue efficiency and success rates. Furthermore, its lightweight design allows for smooth operation on the drone's onboard computing platform. Compared to existing methods with large model parameters and high inference latency, this invention significantly improves the detection accuracy and efficiency of multimodal small targets in complex scenarios while maintaining a lightweight design, meeting the real-time operation requirements of edge computing devices.

[0094] Finally, it should be noted that the above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A lightweight small target detection method based on dual-modal channel reconstruction and adaptive fusion, characterized in that, The method includes: A network framework containing RGB feature branches and IR feature branches is selected. The original C3K2 module in the RGB feature branch is fused with the detail enhancement convolutional module to form the first lightweight convolutional module. The original C3K2 module in the IR feature branch is fused with the windmill convolutional module to form the second lightweight convolutional module, resulting in a dual-branch lightweight convolutional module. Visible light and infrared images are input into the aforementioned dual-branch lightweight convolution module, and the extracted RGB and IR feature maps are input into the multimodal channel reconstruction and adaptive fusion module to generate dual-modal fusion features for target detection. The dual-modal fusion features are input into a lightweight multimodal small target detection model to obtain the final target detection result; wherein, the detection model is trained using a total loss function of weighted fusion CloU loss and SD loss.

2. The lightweight small target detection method according to claim 1, characterized in that, Before extracting the RGB and IR feature maps, the method further includes dimensional alignment of the visible light image and the infrared image.

3. The lightweight small target detection method according to claim 1, characterized in that, The first lightweight convolutional module consists of: a first convolutional layer, a second C3K2_DEConv layer, a third splicing layer, a fourth convolutional layer, a fifth connection layer, a sixth convolutional layer, a seventh connection layer, an eighth SPPF layer, and a ninth C2PSA layer. The detail enhancement convolution module consists of one ordinary convolution and four split convolutions. The four split convolutions are composed of CDC central difference, ADC angular difference, HDC horizontal difference and VDC vertical difference.

4. The lightweight small target detection method according to claim 1, characterized in that, The second lightweight convolutional module consists of: a first convolutional layer, a second C3K2_DPConv layer, a third splicing layer, a fourth convolutional layer, a fifth connection layer, a sixth convolutional layer, a seventh connection layer, an eighth SPPF layer, and a ninth C2PSA layer.

5. The lightweight small target detection method according to claim 1, characterized in that, The method further includes: using the adaptive fusion layer in the multimodal channel reconstruction and adaptive fusion module to perform global average pooling on the extracted RGB and IR feature maps respectively to obtain channel-level global statistical vectors, using channel-level Softmax operation to generate attention weights, and then weighted and fused to obtain the final bimodal fusion feature.

6. The lightweight small target detection method according to claim 1, characterized in that, The method further includes setting a model training strategy that includes at least standard data augmentation, warmup, and Mosaic data augmentation strategies.

7. A lightweight small target detection system based on dual-modal channel reconstruction and adaptive fusion, characterized in that, include: The dual-branch lightweight convolutional module is used to select a network framework containing RGB feature branches and IR feature branches. The original C3K2 module in the RGB feature branch is fused with the detail enhancement convolutional module to form the first lightweight convolutional module; the original C3K2 module in the IR feature branch is fused with the windmill convolutional module to form the second lightweight convolutional module, thus obtaining the dual-branch lightweight convolutional module. The feature extraction and fusion module is used to input visible light images and infrared images into the aforementioned dual-branch lightweight convolution module, and input the extracted RGB and IR feature maps into the multimodal channel reconstruction and adaptive fusion module to generate dual-modal fusion features for target detection. The loss function optimization module is used to input the dual-modal fusion features into a lightweight multimodal small target detection model to obtain the final target detection result; wherein, the detection model is trained using a total loss function of weighted fusion CloU loss and SD loss.

8. The lightweight small target detection system according to claim 7, characterized in that, The multimodal channel reconstruction and adaptive fusion module includes a dimension alignment layer, a transformation layer, and an adaptive fusion layer.

9. The lightweight small target detection system according to claim 8, characterized in that, The dimension alignment layer consists of an RGB feature branch, an IR feature branch, and corresponding dimension alignment submodules. The RGB feature branch receives the original RGB features, and the IR feature branch receives the original IR features. A 1×1 convolutional layer is configured in each branch to unify the number of channels to C and adjust the spatial dimension H. RGB ×W RGB With H ir ×W ir The standard is H×W, and the compression ratio controls the number of channels to reduce the amount of subsequent calculations.

10. The lightweight small target detection system according to claim 8, characterized in that, The transformation layer is used to design differentiated feature extraction sub-modules for the RGB feature branch and the IR feature branch respectively. The RGB feature branch is configured with a GWC+PWC combination, and the IR feature branch is configured with a DWC+PWC+feature multiplexing unit combination.

Images