Unmanned aerial vehicle aerial photography-oriented all-weather multi-modal small target traffic detection method and system

Abstract

The application discloses a method and system for unmanned aerial vehicle (UAV) aerial all-weather multi-modal small target traffic detection, and the method is as follows: 1, acquiring RGB images and IR images, constructing a double-flow backbone network to independently extract multi-scale features of two modalities, and obtaining a hierarchical corresponding RGB and IR feature pyramid; 2, for the RGB and IR same scale feature pair, respectively generating a global light weight and a local difference weight through a light door and a difference door; the two weights are jointly used on the original feature to obtain adaptive fusion features; 3, flattening the fusion features into a sequence list representation, and performing global dependence coding through a bidirectional Mamba state space model along the forward and backward directions, and remodeling the bidirectional output into enhanced features; constructing a multi-scale detection feature pyramid; 4, inputting the multi-scale detection feature pyramid into a double-modal bidirectional Mamba small target detection network for training; 5, using a multi-scale detection head to predict the multi-scale feature pyramid, and outputting the result.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a small target detection method and system based on DualMamba-STDet (Dual-modal Bidirectional Mamba Small Target Detection Network), which is designed for UAV aerial photography scenarios. It adopts dual-modal adaptive fusion and a bidirectional Mamba state space model, and is suitable for the detection of extremely small targets under all weather conditions. Background Technology

[0002] With the development of the low-altitude economy, drone aerial photography is increasingly used in traffic inspection, road monitoring, and other fields. Target detection based on aerial images is a key technology for realizing traffic flow analysis and anomaly perception. However, unlike the ground perspective, drone aerial target detection has the following problems: the high flight altitude often results in vehicle target pixel ratios being less than 32×32, and detailed features are easily lost after multiple downsampling by deep networks; urban traffic scenes have dense occlusion, target overlap, and similar appearances; all-weather operation requires the algorithm to remain stable under conditions such as strong daylight, low nighttime illumination, and fog, and information degradation is severe in low-light environments for single visible light modes.

[0003] To address the aforementioned issues, existing technologies fall into two categories: single-modal detection methods and dual-modal fusion methods. Single-modal detection methods directly apply general detectors such as YOLO and Faster R-CNN (accelerated region convolutional neural networks) to UAV RGB or infrared data. These methods do not utilize the complementary information of the two modalities, resulting in limited detection accuracy in low light or complex backgrounds. Dual-modal fusion methods are categorized based on the timing of fusion: early fusion, mid-stage fusion, and decision-level fusion. Typical existing technologies include: early fusion achieved through feature map concatenation or element-wise addition, but intermodal interference is not suppressed; CrossModalNet (a cross-modal network) incorporating attention mechanisms uses bidirectional cross-attention to enhance cross-modal semantic interaction, but does not consider the global impact of scene illumination changes on modal reliability; lightweight frameworks such as RTMF-Net (RGB-TIR Multimodal Fusion Network) employ a two-stream structure and attention fusion, but still lack fine-grained modeling of local modal differences. Furthermore, while the Transformer (a deep neural network architecture based on self-attention) possesses global modeling capabilities, the quadratic complexity of its attention computation limits its efficiency under high-resolution features. A common shortcoming of the aforementioned existing technologies is that they neither establish modality selection strategies accommodating changes in ambient lighting nor explicitly model differences in modal dominance within local regions, resulting in insufficient adaptability in cross-modal fusion. Simultaneously, mainstream detection networks are limited by the local receptive field of convolutions, restricting their ability to model the global context of small, distant targets.

[0004] To address these issues, this invention proposes an all-weather multimodal small target traffic detection technology based on DualMamba-STDet for UAV aerial photography. It explicitly models the global illumination response and local difference response through a dual-gated cross-modal adaptive fusion mechanism and enhances global context awareness by combining a bidirectional Mamba state space model. This addresses the technical problems of insufficient multimodal fusion, easy loss of small target features, and insufficient robustness in all-weather scenarios in existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a fast multimodal small target detection method and system based on DualMamba-STDet (dual-modal bidirectional Mamba small target detection network) for all-weather UAV aerial photography scenarios, in order to solve the technical problems of low multimodal fusion efficiency, poor small target detection accuracy and weak environmental adaptability in the prior art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for all-weather multimodal small target traffic detection based on DualMamba-STDet (a dual-modal bidirectional Mamba small target detection network) for UAV aerial photography includes the following steps:

[0008] Step 1: Acquire aligned visible light RGB and infrared IR images from the same scene. Construct a dual-stream backbone network to independently extract multi-scale features from the two modalities (visible light RGB and infrared IR images), resulting in a hierarchically corresponding RGB feature pyramid and IR feature pyramid. RGB and IR features at the same level have completely identical spatial resolution, channel dimension, and downsampling rate, forming multiple sets of RGB and IR feature pairs at the same scale.

[0009] Step 2: For the RGB and IR feature pairs output in Step 1, generate global illumination weights through illumination gates to evaluate the overall reliability of the two modes, calculate feature differences through difference gates and generate local difference weights to characterize the complementary advantages of regional modes, and apply the illumination weights and difference weights together to the original features to obtain adaptive fusion features.

[0010] Step 3: Flatten the fused features output from Step 2 into a sequence representation, perform global dependency encoding along the forward and backward directions using a bidirectional Mamba state space model, and reshape the bidirectional outputs into enhanced features; adopt a top-down upsampling and cross-layer concatenation strategy to construct a multi-scale detection feature pyramid containing P2 to P5 (P2 to P5 are professional terms, which are standard hierarchical names for feature pyramids in the field of object detection, corresponding to high-resolution to low-resolution feature layers respectively), where the high-resolution layer focuses on small object detection.

[0011] Step 4: Build the multi-scale detection feature pyramid (i.e., ...) constructed in Step 3. The input is fed into the entire DualMamba-STDet network (including dual-stream backbone, dual-gated fusion, bidirectional Mamba, YOLO26 Neck, and multi-scale detection head) for end-to-end training. During training, a Small Target Attention Learning (STAL) mechanism is introduced, assigning higher loss weights to smaller samples based on the target pixel area. Training stops when certain conditions are met; for example, training stops when the number of training epochs reaches 100 and the loss function on the validation set tends to converge, and the detection accuracy no longer significantly improves.

[0012] Step 5: In the detection phase, a multi-scale detection head is used to process the multi-scale feature pyramid obtained in Step 3 (i.e., ... It performs classification, regression, and confidence prediction, and outputs the final detection results, including target category, bounding box location, and confidence score.

[0013] This invention also discloses an all-weather, multi-modal small target traffic detection system for UAV aerial photography, used to execute the above method, which includes the following modules:

[0014] RGB and IR co-scale feature pair construction module: acquire aligned visible light RGB images and infrared IR images in the same scene, construct a dual-stream backbone network to extract multi-scale features from the visible light RGB images and infrared IR images independently, and obtain RGB feature pyramids and IR feature pyramids with one-to-one correspondence at each level, forming multiple sets of RGB and IR co-scale feature pairs.

[0015] Adaptive fusion feature module: For the output RGB and IR feature pairs at the same scale, a global illumination weight is generated through an illumination gate to evaluate the overall reliability of the two images. A difference gate is used to calculate feature differences and generate local difference weights to characterize the regional modal complementarity advantages. The illumination weights and difference weights are applied together to the original features to obtain the adaptive fusion features.

[0016] Multi-scale detection feature pyramid construction module: flattens the output fused features into a sequence representation, performs global dependency encoding along the forward and backward directions through a bidirectional Mamba state space model, and reshapes the bidirectional outputs into enhanced features; adopts a top-down upsampling and cross-layer concatenation strategy to construct a multi-scale detection feature pyramid containing P2 to P5.

[0017] Training module: The constructed multi-scale detection feature pyramid is input into the bimodal bidirectional Mamba small object detection network for end-to-end training. The training process introduces a small object priority learning mechanism, which assigns loss weights higher than a set value to small samples based on the target pixel area. Training stops when the training meets the set conditions.

[0018] Detection module: The multi-scale detection head is used to classify, regress and predict the confidence of the obtained multi-scale feature pyramid, and output the final detection results.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0020] First, by using the dual-gating (illuminance gate + difference gate) cross-modal fusion mechanism in DualMamba-STDet, a two-level adaptive fusion from the global scene to the local area is achieved. Compared with the existing simple splicing or addition methods, it significantly improves the efficiency of multimodal information utilization and can still maintain high detection accuracy under poor lighting conditions such as night and shadow.

[0021] Second, the bidirectional Mamba structure is introduced into the dual-modal small target detection task. Its global receptive field effectively compensates for the shortcomings of convolutional networks in modeling long-distance contexts and enhances the semantic association ability of small targets and their surrounding environment.

[0022] Third, the STAL small target priority learning mechanism proposed in this invention solves the problem of small targets being dominated by large target samples in gradient updates from the training optimization level, and significantly improves the recall rate of small targets.

[0023] Fourth, experiments show that the DualMamba-STDet method of this invention achieves 77.9% mAP@0.5 on the DroneVehicle public dataset, with an inference speed of 40.2 FPS, meeting the requirements for real-time detection and demonstrating good engineering deployment value. Attached Figure Description

[0024] Figure 1 This is a flowchart of the overall DualMamba-STDet method provided in the embodiments of the present invention.

[0025] Figure 2 This is a diagram of the dual-stream backbone network and multi-scale feature extraction structure provided in the embodiments of the present invention.

[0026] Figure 3 This is a schematic diagram of the dual-gated cross-modal fusion mechanism provided in an embodiment of the present invention.

[0027] Figure 4 This is a flowchart of bidirectional Mamba global modeling and feature refinement provided in an embodiment of the present invention.

[0028] Figure 5 This is a structural diagram of the multi-scale feature fusion (i.e., the top-down upsampling and cross-layer stitching strategy in step 3) provided by the embodiments of the present invention.

[0029] Figure 6 This is a graph showing the loss function and accuracy during the training process provided in an embodiment of the present invention.

[0030] Figure 7 This is a block diagram of an all-weather, multi-modal small target traffic detection system for drone aerial photography, according to an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0032] To facilitate overall understanding, a brief description of the global flow of the method in this embodiment will be given first. For example... Figure 1As shown, the method in this embodiment unfolds along the main line of "modal independent feature extraction - cross-modal adaptive fusion - global context modeling - multi-scale feature aggregation - detection output", and proceeds through the following four main stages: First, multi-scale features are extracted independently from RGB and IR images using a dual-stream backbone network; then, the extracted same-scale feature pairs are fed into a dual-gated fusion module, where adaptive fusion is achieved for scene illumination changes and local regional differences through illumination gates and difference gates; next, the fused features are fed into a bidirectional Mamba global modeling module for global dependency encoding and feature refinement, and after channel shuffling, they are fed into the YOLO26 Neck for multi-scale feature aggregation, constructing a detection feature pyramid containing P2 to P5; finally, during the training phase, the learning weights of small targets are strengthened through the STAL small target priority learning mechanism; and during the detection phase, the multi-scale detection head outputs target category, bounding box position, and confidence information.

[0033] Specifically, this embodiment presents an all-weather, multi-modal small target traffic detection method based on DualMamba-STDet for UAV aerial photography, with detailed steps as follows:

[0034] Step 1: Acquire aligned visible light RGB images and infrared IR images from the same scene. Construct a dual-stream backbone network to perform independent multi-scale feature extraction on the two modalities (visible light RGB images and infrared IR images), resulting in RGB feature pyramids and IR feature pyramids. RGB features and IR features at the same level have completely identical spatial resolution, channel dimension, and downsampling rate, forming multiple sets of RGB and IR feature pairs at the same scale.

[0035] In dual-modal target detection tasks, visible light images and infrared images differ significantly in imaging mechanisms, information representation formats, and target characterization capabilities. To fully preserve the respective advantages of each modality and avoid information interference caused by direct fusion in the early stages of feature extraction, a dual-stream backbone network structure is adopted to independently encode and extract multi-scale features from the RGB and infrared images respectively. Specifically, let the input RGB and infrared images be represented as follows:

[0036]

[0037] in, and Let these represent the height and width of the input image, respectively. Based on the dual-branch feature extraction structure, the feature extraction process for both RGB and IR modalities can be defined as follows:

[0038]

[0039] in, and These represent the backbone feature extraction networks for the RGB and IR branches, respectively. and These represent the feature representations obtained after encoding the two modalities through the backbone network.

[0040] In actual network structures, the specific organizational forms and multi-scale feature interaction methods of the aforementioned dual-stream backbone are as follows: Figure 2 As shown. From Figure 2 As can be seen, the RGB image and the IR image enter the upper and lower symmetrical branches respectively, and then pass through the convolution module. to Features are extracted layer by layer to form to and to Two sets of multi-scale feature representations are used. As the network depth increases, the spatial resolution of the feature maps gradually decreases, while the semantic expressive power continuously increases. The two sets of features at the same level have completely consistent spatial resolution, channel dimension, and downsampling rate, forming multiple sets of RGB and IR feature pairs at the same scale, providing a matching input basis for subsequent cross-modal fusion.

[0041] Step 2: For the RGB and IR feature pairs output in Step 1, global illumination weights are generated through illumination gates to evaluate the overall reliability of the two modes. Feature differences are calculated through difference gates and local difference weights are generated to characterize the complementary advantages of regional modes. Illumination weights and difference weights are applied together to the original features to obtain adaptive fusion features.

[0042] The output after the first step is two sets of RGB and IR feature pairs at the same scale. However, simply concatenating or element-wise adding the two modal features cannot effectively address two key issues: First, the overall reliability of the two modalities varies under different lighting conditions; for example, in nighttime scenes, the infrared modality is significantly superior to the visible light modality. Second, even within the same scene, the modal complementarity relationships differ between different local regions; for example, RGB is more advantageous in textured areas, while IR is more discriminative in areas with obvious heat sources. Therefore, this step designs a dual-gated fusion mechanism to address these issues.

[0043] Specifically, for the RGB and IR feature pairs output in the first step, global illumination weights are generated through illumination gates to evaluate the overall reliability of the two modes. Feature differences are calculated through difference gates and local difference weights are generated to characterize the complementary advantages of regional modes. The illumination weights and difference weights are applied together to the original features to obtain adaptive fusion features.

[0044] The overall structure and information flow relationship of this dual-gating fusion mechanism are as follows: Figure 3 As shown. From Figure 3 As can be clearly seen, the entire module's processing flow is divided into three stages: the upper left section is global illumination response modeling (illumination gate), the lower left section is local difference information modeling (difference gate), and the right side is dual-gated adaptive fusion output. Each part will be explained in detail below.

[0045] 1. Light gate mechanism – global illumination response modeling

[0046] In complex scenarios, the RGB mode is more sensitive to changes in external lighting, while the IR mode typically exhibits stronger stability. To dynamically evaluate the reliability of both modes in the current scene, a lighting gate mechanism is introduced. Taking the RGB branch as an example, the IR branch follows the same logic, and its calculation process is expressed as follows:

[0047]

[0048]

[0049] Among them, L rgb L ir These represent the global illumination response values ​​(scalars) for the RGB and IR modes, respectively. This indicates a global average pooling operation. This represents the sigmoid function. These are learnable parameters. The process first extracts feature responses related to illumination changes using convolution, then compresses them into modality-level description vectors using global average pooling, and finally obtains the illumination weights based on the difference between the global responses of RGB and IR. .

[0050] from Figure 3 As can be seen from the illumination gate branch on the left, the illumination gate does not directly change the structure of the original feature map, but rather participates in subsequent feature fusion as a global modulation factor. When the scene lighting conditions are good, the RGB modality can usually provide clearer texture and edge information. The model tends to increase the weight of the RGB branch; however, when the environment is dark, there are many shadows, or the visible light imaging quality is degraded, the target salience of the IR mode is usually stronger, and the model will give a smaller weight to the RGB branch. This enhances the contribution of the IR branch in the fusion process. Therefore, the illumination gate mechanism essentially constructs a modality selection strategy oriented towards environmental changes, enabling the network to possess preliminary scene adaptation capabilities.

[0051] 2. Difference Gating Mechanism – Modeling Local Difference Information

[0052] Relying solely on global illumination information to weight the two modalities still has limitations, as the response differences between different target regions in RGB and IR images often exhibit significant local characteristics. Therefore, a difference gate component (DGC) is introduced to explicitly characterize the differences between the two modal features and generate finer-grained modal weights. Its mathematical expression is:

[0053]

[0054]

[0055] in, This represents the local difference response between RGB and IR features. , The number of input feature channels, For the intermediate hidden layer dimension This is a non-linear activation function. Furthermore, based on the differential attention vector... Generate the difference weights corresponding to the two modes:

[0056]

[0057]

[0058] in, , These are the learnable mapping functions used to convert difference attention into difference weights for RGB and IR, respectively.

[0059] Combination Figure 3 As seen in the middle difference gate branch, this module first performs an absolute difference operation on the RGB and IR feature maps to extract inconsistent response regions between the two modalities. Then, through pooling operations and a two-layer mapping network (MLP), the difference information is compressed into learnable attention weights. Through this process, the model no longer fuses the two modal features in a fixed ratio, but rather adaptively determines which modality has a greater discriminative advantage in the current local area based on the response differences between different regions. For regions with rich texture details and clear edges, the RGB branch is often more advantageous; while for nighttime vehicles, heat sources, or areas heavily affected by shadows, the IR branch is usually better at highlighting the target outline.

[0060] 3. Dual-gated fusion – adaptive weighted aggregation

[0061] After obtaining the outputs of the illumination gate and difference gate, the RGB and IR features are further fused using a dual-gated method, the expression of which is:

[0062]

[0063] in, This represents broadcast multiplication, which expands scalar or low-dimensional weights to the size of the feature map. This indicates element-wise multiplication.

[0064] Depend on Figure 3 As can be seen from the fusion output on the right, dual-gated fusion is not a single-level feature overlay, but a gradual adjustment process from global to local. The illumination gate is mainly used to determine "which modality is more reliable overall in the current scene," while the difference gate is used to determine "which modality is more discriminative in a specific local region." The combined effect of both gates results in fused features. It possesses both environmental adaptability and regional specificity. This design not only effectively reduces the adverse impact of single-modal failure on overall detection performance, but also provides a more stable and discriminative input representation for subsequent global feature modeling.

[0065] The aforementioned dual-gated fusion mechanism can also be defined as the Dual-Gated Complementary Mamba Fusion Module (DGC-MFM). While preserving the independent semantics of the two stream features, it completes global illumination adaptation and local difference modeling for the two modalities, realizes adaptive interactive enhancement of cross-modal features, and provides more discriminative input for subsequent global context modeling and multi-scale feature aggregation.

[0066] Step 3: Flatten the fused features output from Step 2 into a sequence representation, perform global dependency encoding along the forward and backward directions using a bidirectional Mamba state space model, and reshape the bidirectional outputs into enhanced features; adopt a top-down upsampling and cross-layer splicing strategy to construct a multi-scale detection feature pyramid containing P2 to P5, where the high-resolution layer focuses on small target detection.

[0067] The fusion features obtained in the second step Although complementary information from RGB and IR modalities has been integrated, the feature extraction process still primarily relies on local convolution operations. The receptive field is limited by the kernel size, resulting in limited ability to model long-range dependencies between distant small targets and their surroundings. In drone aerial photography of traffic scenes, the texture information of distant vehicles and other small targets is often very weak, making stable recognition difficult based solely on local features. A comprehensive judgment requires combining broader contextual information (such as road structure and the distribution of adjacent targets). Therefore, this step feeds the fused features into a bidirectional Mamba global modeling module for further enhancement, ultimately constructing a multi-scale detection feature pyramid.

[0068] Specifically, the fused features output from the second step are flattened into a sequence representation, and global dependency encoding is performed along the forward and backward directions using a state-space model. After fusing the bidirectional outputs, they are reshaped into enhanced features. A top-down upsampling and cross-layer concatenation strategy is adopted to construct a feature containing... to The multi-scale detection feature pyramid, in which the high-resolution layer focuses on small target detection.

[0069] The overall processing flow for this step is as follows: Figure 4 As shown. From Figure 4 It can be clearly seen that after dual-gated fusion, the features sequentially go through five stages: "flattening, bidirectional Mamba encoding, sequence restoration, feature refinement, and channel shuffling," ultimately outputting multi-scale features. The main processes of each stage are explained below.

[0070] 1. Bidirectional Mamba Global Context Modeling

[0071] To adapt the two-dimensional feature map to the modeling method of the state-space model, the fused features are first flattened into a sequence representation, which is as follows:

[0072]

[0073] in, and These represent the spatial dimensions of the feature map at the current scale. Indicates batch size. This represents the number of channels. Through this process, the original two-dimensional feature map is mapped into a one-dimensional sequence, which facilitates subsequent global dependency modeling using a state-space model.

[0074] Based on this, a bidirectional scanning method is used to model the sequence features, specifically expressed as follows:

[0075]

[0076]

[0077]

[0078] in, This represents the Mamba state-space model. This represents a sequence reversal operation. This indicates a learnable linear mapping layer. This represents the feature concatenation operation. The forward scan is used to capture contextual information propagating from front to back in the sequence, while the backward scan further supplements the dependencies propagating from back to front. Together, they enable the model to learn the global associations between different positions more comprehensively.

[0079] Finally, the updated sequence is restored to a two-dimensional feature map representation:

[0080]

[0081] Unlike traditional convolutions, which primarily focus on local receptive fields, the bidirectional Mamba module excels at characterizing large-scale contextual dependencies, making it particularly suitable for dual-modal detection scenarios with complex target distributions and strong background interference. Figure 4 As can be seen from the two Mamba paths in opposite directions, bidirectional scanning effectively reduces the information bias that may be caused by unidirectional scanning, enabling the model to capture the contextual dependencies in both directions simultaneously.

[0082] 2. Feature Refinement and Residual Connection

[0083] After completing the global context modeling, the enhanced semantic information needs to be fed back to the original RGB and IR branches. For example... Figure 4 As shown on the right, the fused features are processed through two parallel processes. The convolutional layers perform channel transformations to generate enhanced representations for the RGB and IR branches:

[0084]

[0085]

[0086] in, and These represent 1×1 convolution mappings applied to the RGB and IR branches, respectively.

[0087] To enhance high-level semantic representation while preserving local details in the original modality features, residual connections are further introduced to feed the enhanced features back into the original branch, in the form of:

[0088]

[0089]

[0090] from Figure 4 The arrows pointing towards the residual connections clearly demonstrate that this structure serves two main purposes: First, by sharing fused features, it semantically enhances the two modal branches, improving the discriminative power of the modal features. Second, the residual connections preserve the detailed information in the original features, thus avoiding the oversmoothing problem caused by relying solely on global modeling. In other words, the feature refinement module does not simply replace the original modal features, but rather superimposes a globally fused semantic compensation on top of the original features, thereby achieving a balance between "semantic enhancement" and "detail preservation."

[0091] 3. Channel mixed washing mechanism

[0092] To further enhance the interactive capabilities of bimodal features across the channel dimension, a channel shuffling mechanism is introduced after feature refinement:

[0093]

[0094]

[0095] in, These are the RGB and IR characteristics, respectively, after channel shuffling and recalibration. It is and The fusion feature obtained by adding elements one by one This represents a feature recombination operation consisting of group convolution and channel shuffling. This module first reduces computational overhead through group convolution, and then achieves cross-group and cross-modal information exchange through channel rearrangement. Figure 4 As can be seen, the channel shuffling mechanism is located after the feature refinement module and before the Neck network. Its main function is to reorganize the bimodal features after global modeling and residual compensation into a unified representation that is more suitable for subsequent multi-scale aggregation, thereby reducing the sense of separation between different modalities.

[0096] 4. YOLO26 Neck Multi-Scale Feature Fusion

[0097] After the above processing, enhanced multi-scale features have been obtained. to However, drone aerial photography of traffic scenes suffers from problems such as "large target scale span, weak features of extremely small targets, and insensitivity of deep features to small targets." Relying solely on features of a single scale is insufficient to simultaneously address the detection needs of both extremely small targets and large vehicles. Therefore, this embodiment introduces the YOLO26Neck module for multi-scale feature fusion. Its overall structure and key components are as follows: Figure 5 As shown.

[0098] from Figure 5 It is clear that the YOLO26 Neck employs a hierarchical approach to feature extraction, fusion, and output. First, the backbone extracts high-level semantic features through multiple Conv and C3k2 modules, and introduces SPPF and C2PSA modules to further enhance feature representation. Then, in the Neck section, Upsample and Concat operations are used to progressively fuse deep semantic features with shallow high-resolution features. C3k2 is the core feature processing unit in the Neck section, primarily used to further refine and reorganize the input features; the Upsample module upsamples the deep, low-resolution feature maps to a higher spatial resolution for concatenation and fusion with shallow features.

[0099] The specific fusion process follows the design philosophy of "upsampling high-level semantic features step by step and fusing them with shallow high-resolution features across layers." First, the highest-level input features... Semantic refinement was performed to obtain Subsequently, through upsampling and cross-layer splicing operations, high-level semantic information is sequentially passed to... , and Higher resolution feature layers are then added. Finally, the Neck module outputs... , , , Four-layer multi-scale feature maps. From Figure 5 The distribution of each detection branch can be seen from the position of the branch. and It is more inclined to represent high-resolution, small target features, making it suitable for detecting small targets such as vehicles at a distance; and It leans more towards strong semantic representation, making it suitable for detecting medium to large-sized targets and target regions with more complete structures. Through this multi-scale feature hierarchical design, the model can simultaneously meet the detection needs of extremely small targets and large vehicles.

[0100] Step 4: Utilize the multi-scale detection feature pyramid constructed in Step 3 (i.e., ...) The input is fed into the entire DualMamba-STDet network (including dual-stream backbone, dual-gated fusion, bidirectional Mamba, YOLO26 Neck, and multi-scale detection head) for end-to-end training. During training, a Small Target Attention Learning (STAL) mechanism is introduced, assigning higher loss weights to smaller samples based on the target pixel area. Training is stopped when the number of training epochs reaches 100 and the loss function on the validation set tends to converge, and the detection accuracy no longer significantly improves.

[0101] After the first three steps, the model has acquired the capabilities of multimodal adaptive fusion, global context modeling, and multi-scale feature representation. However, a prominent problem remains during the training phase: large targets and easily captured samples in drone aerial photography tend to dominate gradient updates, leading to insufficient learning of small targets and low recall rates for extremely small targets such as distant vehicles. To address this issue, this step introduces a Small Target Priority Learning (STAL) mechanism during the training phase.

[0102] STAL Small Goal Priority Learning Mechanism:

[0103] The core idea of ​​STAL can be summarized as "small target priority," which means increasing the learning weight of small targets in the model during the training phase, and combining it with high-resolution feature layers and multi-scale fusion strategies to improve the recall rate and detection stability of small traffic targets at long distances. This strategy mainly works in three ways: first, it increases sample attention by giving more attention to small target samples during the training phase; second, it strengthens the loss side by giving extra emphasis to the loss related to small targets during the overall loss calculation process, so that small target samples are not overwhelmed by the loss contribution of large target samples; and third, it enhances the feature layer synergy, combining STAL with Neck... , The high-resolution feature layers work together, combining a small target priority strategy with high-resolution feature representation, enabling the model to more effectively preserve the contour, edge, and local texture information of extremely small targets.

[0104] Let the first The pixel area of ​​each target is The threshold for determining small targets is defined as follows:

[0105]

[0106] Then the first The weights of the sub-objectives for each objective can be expressed as:

[0107]

[0108] in, This is the small target enhancement coefficient, used to control the degree of extra attention the model pays to small target samples.

[0109] Based on this, the total loss can be written as:

[0110]

[0111] Where M represents the total number of targets in a batch of training samples. Indicates the first The classification loss for each target. This represents the bounding box regression loss. Indicates the target confidence loss. and These are the balance coefficients for localization loss and target-oriented loss, respectively. The above explanation illustrates that the essence of STAL lies in introducing target scale-aware weights, allowing the loss optimization process to focus more on small target samples.

[0112] Step 5: In the detection phase, a multi-scale detection head is used to analyze the multi-scale feature pyramid obtained in Step 3 (i.e., ... It performs classification, regression, and confidence prediction, and outputs the final target category, bounding box location, and confidence score.

[0113] Scale inspection head:

[0114] The detector head is located after the YOLO26 Neck, and its main task is to receive multi-scale fused features. , , , Each scale feature layer corresponds to an independent detection branch, and the following steps are performed: target classification, bounding box regression, and target confidence prediction. Each scale input feature is The output of the detection head can then be expressed as:

[0115]

[0116]

[0117] in, This is a classification convolutional layer used to output the predicted object category. This is a regressive convolutional layer used to output bounding box coordinates. Indicates the output of the categorized branch. This indicates the output of the regression branch. Indicates the number of categories. , The first The height and width of the feature map at each scale. In engineering implementation, different scale detection branches predict high-resolution and low-resolution features respectively. The high-resolution branch is more suitable for small target detection, while the low-resolution branch is more suitable for medium and large target detection, so that the detection head module and Neck's multi-scale output can work together effectively.

[0118] To verify the effectiveness of the complete method comprised of the above steps of this invention, experiments were conducted on the DroneVehicle public dataset. This dataset contains 28,439 pairs of RGB-IR images, covering complex environments such as daytime, nighttime, shadows, and fog / haze, and includes five typical traffic targets: car, truck, bus, van, and freight car. The experimental platform was an NVIDIA GeForce RTX 4060 Laptop GPU, and the development environment was Python 3.9 and PyTorch 2.2.0. Training used a batch size of 8, an input size of 840×840, and 100 training epochs.

[0119] (1) Validation of the effectiveness of dual-modal fusion

[0120] To verify the practical effectiveness of the proposed dual-modal fusion strategy, it is necessary to compare and analyze the fusion model with the single-modal model. Based on this, three sets of experiments were set up: using only RGB, using only IR, and RGB+IR fusion, to evaluate the performance gain of dual-modal information in target detection in complex traffic scenarios. The results are shown in Table 1.

[0121] Table 1. Performance Evaluation Based on Dual Modality

[0122]

[0123] As shown in Table 1, the overall performance of the IR single-modal YOLOv11 is superior to that of the RGB single-modal model, with its mAP@50 increasing from 75.3% to 79.3%, indicating that the infrared modality has a stronger target saliency representation capability in UAV traffic scenarios. In contrast, the RGB+IR fusion result of CFT is only 76.9%, which does not exceed the baseline of the IR single-modal model, indicating that the simple fusion method cannot fully leverage the complementary advantages of the two modalities. The dual-stream backbone network used in DualMamba-STDet achieves 77.9% mAP@50 with RGB+IR input, which is 2.6 and 1.0 percentage points higher than the RGB single-modal and CFT models, respectively, indicating that the dual-stream independent feature extraction strategy can improve the efficiency of multimodal feature utilization to a certain extent.

[0124] (2) Training process analysis

[0125] like Figure 6As shown in the training curves, the train / box loss, train / cls loss, and train / dfl loss all decreased rapidly in the early stages, then gradually stabilized. The val / box loss, val / cls loss, and val / dfl loss on the validation set also showed a consistent decreasing trend with the training set, without significant oscillations or later rebounds. This indicates that the model training process was relatively stable, the loss function design and optimizer configuration were effective, and the network did not exhibit significant overfitting in the later stages of training. Meanwhile, metrics / precision, metrics / recall, metrics / mAP50, and metrics / mAP50-95 improved rapidly in the first 20 epochs, then the increase gradually decreased and approached saturation, indicating that the model had learned the main features in the early and mid-stages, and the later training focused more on refining the bounding boxes and stabilizing the classifier. From the results trend, the model has two distinct characteristics: first, the convergence speed is relatively fast, indicating that the current network structure has a good feature fitting ability for bimodal traffic targets; second, the trends of the training and validation sets are consistent, indicating that the collaboration between modules is stable, and the training did not become unstable due to an excessive number of modules.

[0126] (3) Ablation Experiment Analysis

[0127] To verify the independent effectiveness and collaborative contribution of each key module, an ablation experiment was conducted in a unified training environment, and the results are shown in Table 2.

[0128] Table 2 Ablation Experiments Based on the DroneVehicle Dataset

[0129]

[0130] The ablation results show that all three key modules contribute positively to the overall performance. Compared to the baseline model YOLOv11 (mAP@50: 75.3%), introducing the dual-stream backbone network alone improves performance to 76.1%, introducing YOLO26Neck alone to 75.7%, and introducing the STAL mechanism alone to 75.6%. This indicates that the three modules contribute to model performance from three different levels: modality modeling, multi-scale fusion, and small-objective training. Furthermore, the performance improvement is more significant when the two modules are combined: the combination of dual-stream backbone and YOLO26 Neck reaches 76.8%, the combination of dual-stream backbone and STAL reaches 76.5%, and the combination of YOLO26 Neck and STAL reaches 76.2%. Finally, when all three are used simultaneously, the model achieves the best mAP@0.5 at 77.9%. This result demonstrates that the designed key modules have a good synergistic relationship and are not independently superimposed, but rather complement each other in feature representation, modality fusion, and training optimization.

[0131] (4) Overall testing performance analysis

[0132] The overall detection results of DualMamba-STDet on the test set are shown in Table 3.

[0133] Table 3 Overall and Classification Detection Results of the Model

[0134]

[0135] The model achieves a good balance between accuracy and recall overall: Precision is 0.776, Recall is 0.714, mAP@0.5 is 0.779, mAP@0.5:0.95 is 0.555, and the inference speed is 40.2 FPS, meeting the requirements for real-time detection. Among these, the car category has the best performance with an mAP@0.5 of 96.8%, and the bus category has 95.0%, indicating that these two target classes have more abundant samples in the training data, more stable appearance features, and significant discriminative information in both RGB and IR modalities. In contrast, truck (70.6%), van (62.4%), and freight car (64.5%) are the weak categories of the current model.

[0136] like Figure 7 As shown, this embodiment discloses an all-weather multimodal small target traffic detection system for UAV aerial photography, used to execute the above method, which includes the following modules:

[0137] RGB and IR co-scale feature pair construction module: acquire aligned visible light RGB images and infrared IR images in the same scene, construct a dual-stream backbone network to extract multi-scale features from the visible light RGB images and infrared IR images independently, and obtain RGB feature pyramids and IR feature pyramids with one-to-one correspondence at each level, forming multiple sets of RGB and IR co-scale feature pairs.

[0138] Adaptive fusion feature module: For the output RGB and IR feature pairs at the same scale, a global illumination weight is generated through an illumination gate to evaluate the overall reliability of the two images. A difference gate is used to calculate feature differences and generate local difference weights to characterize the regional modal complementarity advantages. The illumination weights and difference weights are applied together to the original features to obtain the adaptive fusion features.

[0139] Multi-scale detection feature pyramid construction module: flattens the output fused features into a sequence representation, performs global dependency encoding along the forward and backward directions through a bidirectional Mamba state space model, and reshapes the bidirectional outputs into enhanced features; adopts a top-down upsampling and cross-layer concatenation strategy to construct a multi-scale detection feature pyramid containing P2 to P5.

[0140] Training module: The constructed multi-scale detection feature pyramid is input into the bimodal bidirectional Mamba small object detection network for end-to-end training. The training process introduces a small object priority learning mechanism, which assigns loss weights higher than a set value to small samples based on the target pixel area. Training stops when the training meets the set conditions.

[0141] Detection module: The multi-scale detection head is used to classify, regress and predict the confidence of the obtained multi-scale feature pyramid, and output the final detection results.

[0142] Other aspects of this embodiment can be found in the above method embodiments.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for all-weather, multimodal traffic detection of small targets based on UAV aerial photography, characterized by: Includes the following steps: Step 1: Obtain aligned visible light RGB images and infrared IR images in the same scene, construct a dual-stream backbone network to perform independent multi-scale feature extraction on the visible light RGB images and infrared IR images respectively, and obtain RGB feature pyramids and IR feature pyramids with one-to-one correspondence at each level, forming multiple sets of RGB and IR feature pairs of the same scale. Step 2: For the RGB and IR feature pairs output in Step 1, global illumination weights are generated through illumination gates to evaluate the overall reliability of the two images. Feature differences are calculated through difference gates and local difference weights are generated to characterize the regional modal complementarity advantages. Illumination weights and difference weights are applied together to the original features to obtain adaptive fusion features. Step 3: Flatten the fused features output from Step 2 into a sequence representation, perform global dependency encoding along the forward and backward directions using a bidirectional Mamba state space model, and reshape the bidirectional outputs into enhanced features; adopt a top-down upsampling and cross-layer concatenation strategy to construct a multi-scale detection feature pyramid containing P2 to P5. Step 4: Input the multi-scale detection feature pyramid constructed in Step 3 into the bimodal bidirectional Mamba small target detection network for end-to-end training. The training process introduces a small target priority learning mechanism, assigning loss weights higher than the set value to small samples based on the target pixel area; training stops when the training meets the set conditions. Step 5: Enter the detection stage. Use a multi-scale detection head to classify, regress, and predict the confidence of the multi-scale feature pyramid obtained in Step 3, and output the final detection results.

2. The all-weather, multi-modal small target traffic detection method for UAV aerial photography as described in claim 1, characterized in that, In step 1, a dual-stream backbone network is used to independently encode and extract multi-scale features from the visible light RGB image and the infrared IR image, respectively; specifically, let the input RGB image and infrared IR image be represented as follows: in, and These represent the height and width of the input image, respectively. The feature extraction process for both RGB and IR modes is defined as follows: in, and These represent the backbone feature extraction networks for the RGB and IR branches, respectively. and These represent the feature representations obtained after encoding the two modalities through the backbone network.

3. The all-weather, multi-modal small target traffic detection method for UAV aerial photography as described in claim 2, characterized in that, In step 2, a light gate mechanism is introduced. For the RGB branch, the calculation process is as follows: Among them, L rgb L ir These represent the global illumination response values ​​for the RGB and IR modes, respectively. This indicates a global average pooling operation. This represents the sigmoid function. For learnable parameters, ; Introducing the difference gate mechanism, the mathematical expression is: in, This represents the local difference response between RGB and IR features; , The number of input feature channels, For the intermediate hidden layer dimension It is a non-linear activation function; based on the differential attention vector Generate the difference weights corresponding to the two modes: in, , These are the learnable mapping functions used to convert difference attention into difference weights for RGB and IR, respectively; After obtaining the outputs of the illumination gate and difference gate, the RGB and IR features are further fused using a dual-gated method, expressed as follows: in, This represents broadcast multiplication. This indicates element-wise multiplication.

4. The all-weather, multi-modal small target traffic detection method for UAV aerial photography as described in claim 3, characterized in that, In step 3, global context modeling is performed using a bidirectional Mamba state-space model: First, flatten the fused features into a sequence representation as follows: in, and These represent the spatial dimensions of the feature map at the current scale. Indicates batch size. Indicates the number of channels; The sequence features are modeled using a bidirectional scanning method, specifically as follows: in, This represents the Mamba state-space model. This represents a sequence reversal operation. This indicates a learnable linear mapping layer. Indicates feature concatenation operation; Finally, the updated sequence is restored to a two-dimensional feature map representation: 。 5. The all-weather, multi-modal small target traffic detection method for UAV aerial photography as described in claim 4, characterized in that, In step 3, feature refinement and residual connection are performed: After completing the global context modeling, the enhanced semantic information needs to be fed back to the original RGB and IR branches; The fusion features are processed through two parallel processes. The convolutional layers perform channel transformations to generate enhanced representations for the RGB and IR branches: in, and These represent 1×1 convolution mappings applied to the RGB and IR branches, respectively. Further, residual connections are introduced to feed the enhanced features back into the original branch, in the form of: 。 6. The all-weather, multi-modal small target traffic detection method for UAV aerial photography as described in claim 5, characterized in that, In step 3, a channel mixing shuffling mechanism is used: A channel shuffling mechanism is introduced after feature refinement and residual joining: in, These are the RGB and IR characteristics, respectively, recalibrated after channel shuffling; It is and The fusion feature obtained by adding elements one by one This represents a feature recombination operation consisting of group convolution and channel shuffling.

7. The all-weather, multi-modal small target traffic detection method for UAV aerial photography as described in claim 6, characterized in that, In step 3, YOLO26 Neck is used for multi-scale feature fusion: The YOLO26 Neck employs a hierarchical approach to feature extraction, fusion, and output. First, the backbone extracts high-level semantic features through multiple Conv and C3k2 modules, and introduces SPPF and C2PSA modules to further enhance feature representation capabilities. Subsequently, in the Neck section, the deep semantic features and shallow high-resolution features are fused step by step through Upsample and Concat operations; C3k2 is used to further refine and reorganize the input features; the Upsample module is used to upsample the deep low-resolution feature map to a higher spatial resolution. Fusion process: First, process the highest layer input features Semantic refinement was performed to obtain Subsequently, through upsampling and cross-layer splicing operations, high-level semantic information is sequentially passed to... , and A higher resolution feature layer; ultimately, the Neck module outputs... , , , Four-layer multi-scale feature map.

8. The all-weather, multi-modal small target traffic detection method for UAV aerial photography as described in claim 7, characterized in that, In step 4, the STAL small-goal-first learning mechanism is as follows: Let the first The pixel area of ​​each target is The threshold for determining small targets is defined as follows: Then the first The weights of the sub-objectives for each objective are represented as follows: in, For small target enhancement coefficients; The total loss is written as: Where M represents the total number of targets in a batch of training samples, Indicates the first The classification loss for each target. This represents the bounding box regression loss. Indicates the target confidence loss. and These are the balance coefficients for positioning loss and targeting loss, respectively.

9. The all-weather multimodal small target traffic detection method for UAV aerial photography as described in claim 8, characterized in that, In step 5, the multi-scale detection head is specifically as follows: The task of the detection head is to receive multi-scale fused features. , , , Each scale feature layer corresponds to an independent detection branch, and the following steps are performed: target classification, bounding box regression, and target confidence prediction. Each scale input feature is The output of the detection head is then expressed as: in, For classification convolutional layers, For regressive convolutional layers, Indicates the output of the categorized branch. This indicates the output of the regression branch. Indicates the number of categories. , The first The height and width of the feature map at each scale.

10. An all-weather, multimodal small target traffic detection system for UAV aerial photography, used to perform the method as described in any one of claims 1-9, characterized in that, Includes the following modules: RGB and IR co-scale feature pair construction module: acquire aligned visible light RGB images and infrared IR images in the same scene, construct a dual-stream backbone network to extract multi-scale features from the visible light RGB images and infrared IR images independently, and obtain RGB feature pyramids and IR feature pyramids with one-to-one correspondence at each level, forming multiple sets of RGB and IR co-scale feature pairs. Adaptive fusion feature module: For the output RGB and IR feature pairs at the same scale, a global illumination weight is generated through an illumination gate to evaluate the overall reliability of the two images. A difference gate is used to calculate feature differences and generate local difference weights to characterize the regional modal complementarity advantages. The illumination weights and difference weights are applied together to the original features to obtain the adaptive fusion features. Multi-scale detection feature pyramid construction module: flattens the output fused features into a sequence representation, performs global dependency encoding along the forward and backward directions through a bidirectional Mamba state space model, and reshapes the bidirectional outputs into enhanced features; adopts a top-down upsampling and cross-layer concatenation strategy to construct a multi-scale detection feature pyramid containing P2 to P5. Training module: The constructed multi-scale detection feature pyramid is input into the bimodal bidirectional Mamba small object detection network for end-to-end training. The training process introduces a small object priority learning mechanism, which assigns loss weights higher than a set value to small samples based on the target pixel area. Training stops when the training meets the set conditions. Detection module: The multi-scale detection head is used to classify, regress and predict the confidence of the obtained multi-scale feature pyramid, and output the final detection results.

Images