Online Target Counting Method for UAVs Based on Multimodal Dynamic Neural Networks

Abstract

A method for online target counting on unmanned aerial vehicles (UAVs) based on a multimodal dynamic neural network includes the following steps: First, the UAV uses an onboard visible light camera and a thermal infrared camera to capture visible light and thermal infrared images, respectively. Then, the segmented images are used as input to a target counting model. The target counting model is a target counting model based on a multimodal dynamic neural network. When thermal infrared images are unavailable, the visible light images are converted into thermal infrared images using a mode conversion network. Multimodal gating analysis is performed on the scene illumination where the images are located. If the illumination is good, only the visible light mode branch of the target counting model is selected; if the illumination is insufficient, both the visible light mode branch and the thermal infrared mode branch are selected, with the thermal infrared images used as auxiliary data. A feature extraction network extracts the modal features of the selected modal branches and fuses the features from the multimodal inference flow. The features obtained in step S3 are decoded to obtain the output of the target counting model.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, specifically a method for online target counting of unmanned aerial vehicles based on a multimodal dynamic neural network. Background Technology

[0002] Most existing aerial image target counting methods assume that the drone is deployed in an environment with high visibility and clear field of view, and can capture rich details about the target's appearance and texture. [1] However, in reality, drones often encounter severe weather, overexposure, underexposure, or shadows, leading to a significant deterioration in the quality of visible light images. [2] As shown in Figure 1(a), counting methods relying solely on visible light are insufficient to accurately identify valid targets in images under such circumstances. [3] The advantage of thermal infrared imaging lies in its ability to capture surface thermal information of objects that cannot be directly perceived by the human eye. Thermal infrared cameras can be integrated into aerial drones. The acquired thermal infrared image, as shown in Figure 1(b), includes the target's location and spatial background, which can compensate for the target information missing in visible light images. [4] This multimodal target counting method for visible / thermal (RGB / Thermal, RGB-T) images exhibits stronger robustness compared to single-modal methods. [5] .

[0003] Researchers have explored cross-modal feature fusion and modality perception. Fang et al. [6] A cross-modal feature fusion strategy based on a self-attention mechanism is proposed, which improves the performance of multispectral target detection in aerial images by integrating rich information from different modalities. (Sun et al.) [7] A cross-modal vehicle detection framework with uncertainty awareness is developed. This framework integrates visible light and thermal infrared image data and reduces detection errors caused by uncertain targets through illumination quantization, thereby achieving robust target detection in complex environments.

[0004] While numerous multimodal target detection methods have been proposed by researchers, few have considered online multimodal target counting methods for aerial imagery. A practical problem is that complex multimodal models are difficult to deploy directly on drones with limited capabilities, especially consumer electronics drones. Dynamic Neural Network (DyNN) architectures offer a promising solution, with typical approaches including early termination. [8] Skip-level [9] and many experts

[10] Online drone counting has a natural dependence on multimodal and DyNN methods. Combining these two approaches, the resulting multimodal DyNN method can adapt to the diversity of detection environments and instances, balancing the accuracy of online counting with the cost of inference.

[0005] Despite its many advantages, the design of low-cost and highly available multimodal dynamic neural network counting models faces unique challenges due to the cluttered backgrounds, large target scale variations, and uneven distribution in drone aerial images, coupled with limited computational resources.

[0006] 1) Lightweight multimodal feature alignment. Because visible light and thermal infrared images exhibit inconsistent spatial distributions, blindly designing feature extraction networks is undesirable and can lead to uninterpretable extracted features. For multimodal feature learning, a common approach is to employ a two-stream network structure.

[11] In other words, features are extracted using two parallel network structures. However, the resulting model has a large number of parameters, making it difficult to deploy on resource-constrained drones. Furthermore, the two modalities are independent of each other. The two-stream structure does not fully exploit the complementarity of the modalities and ignores the parameters shared between the two modalities.

[0007] 2) Environmental Adaptation in Dynamic Deployment. During mission execution, the importance of different modal data for the UAV changes dynamically across time and space. In other words, the UAV's online counting requirements for different modal data are dynamically changing. For example, during daylight hours, the model simultaneously decodes both visible light and thermal infrared images. While this optimizes counting accuracy, the inference cost increases exponentially. For resource-constrained UAVs, trading high inference costs for a slight improvement in accuracy is not worthwhile. Pruning is a common method for lightweight models. However, while static pruning helps reduce inference costs, a loss of detection accuracy is unavoidable. Considering the diversity of environments during UAV mission execution, multimodal models must possess self-adjusting capabilities to achieve the optimal trade-off between counting performance and inference cost.

[0008] 3) Robustness optimization of multimodal models. During dynamic deployment, the quality of multimodal data acquired by airborne sensors inevitably degrades or even disappears. This is addressed in the image pair generation framework based on CycleGAN.

[12] In this process, visible light data is converted into thermal infrared data by a reversible neural network. This data, along with the visible light data, is used as input to the model for target counting. Because the loss function does not constrain local details, CycleGAN struggles to focus on local pattern and texture features. Summary of the Invention

[0009] To support online target counting of drone aerial images in complex scenarios such as insufficient lighting, rain, fog, haze, and nighttime operations, this invention proposes a framework based on the multimodal dynamic neural network DyNN for target counting of drone aerial images, taking into account the model's adaptability, robustness, and real-time performance.

[0010] An online target counting method for unmanned aerial vehicles (UAVs) based on a multimodal dynamic neural network includes the following steps: First, the UAV uses an onboard visible light camera and a thermal infrared camera to capture visible light images and thermal infrared images, respectively; then, these images are segmented based on scale estimation; the segmented images are used as input to a target counting model for target counting.

[0011] The target counting model is based on the multimodal dynamic neural network DyNN. The architecture of the target counting model includes a feature extraction network, a decoder, a multimodal gating network, and a mode transformation network. In the target counting model:

[0012] S1. When thermal infrared images are unavailable, the visible light image is converted into a thermal infrared image by a mode conversion network, providing the generated thermal infrared image for the thermal infrared mode branch of the target counting model; (the visible light image and the generated thermal infrared image are used together for counting in the target counting model).

[0013] S2. Multimodal gating analysis of the scene illumination where the image is located; if the illumination is good, only the visible light mode branch of the target counting model is selected; if the illumination is insufficient, the visible light mode branch and the thermal infrared mode branch are selected, with the thermal infrared image used as auxiliary data.

[0014] S3. The feature extraction network extracts the modal features of the selected modal branches and fuses the features of the multimodal inference flow;

[0015] S4. The features obtained in step S3 are decoded to obtain the output of the target counting model.

[0016] The main contributions of this invention include:

[0017] Lightweight multimodal feature fusion. Modal information is separated by an additional modality extractor and concatenated after shared features. A Conditional Generative Adversarial Network (CGAN) is used to establish complementarity between the feature representations of different modalities, facilitating model learning and capturing intermodal consistency information.

[0018] Decoder-assisted multimodal gating. The decoder provides training data for the multimodal gating. The multimodal gating guides the opening and closing of the thermal infrared branch network. The decoder upsamples features to restore resolution, generating high-quality density maps from different modalities. The multimodal gating is trained based on the comparison results of the density maps.

[0019] Density-Map Modal Transformation (DMMT) with density map loss. In adversarial learning, thermal infrared images and real thermal infrared images tend to be similar in spatial distribution based on density map counting. The model is gradually gaining the ability to generate high-quality thermal infrared data using visible light data to compensate for the lack of (thermal infrared) modal data.

[0020] Experiments on several authoritative datasets (including DroneRGBT and RGBT-CC) demonstrate that the proposed model is adaptive, achieving good counting accuracy and maintaining low inference latency even with unstable modal data inputs. Attached Figure Description

[0021] Figure 1(a) and Figure 1(b) show aerial images taken in dimly lit scenes, where:

[0022] Figure 1(a) is a visible light image.

[0023] Figure 1(b) is a thermal infrared image;

[0024] Figure 2 This represents a framework / model for UAV image target counting based on a multimodal dynamic neural network.

[0025] Figures 3(a) and 3(b) illustrate the decision-making process of multimodal gating under different lighting conditions.

[0026] Figure 3(a) shows the case with good lighting (PRRGB = 97, PRfinal = 106).

[0027] Figure 3(b) shows the case of insufficient lighting (PRRGB=4, PRfinal=16);

[0028] Figure 4 This diagram illustrates a DMMT-based mode transfer framework. The terms used in the diagram are explained below:

[0029] RGB image - a visible light image; Fake RGB image - a (generated) pseudo-visible light image.

[0030] Infrared image - an image with infrared radiation; Fake Infrared image - a generated, pseudo-infrared image.

[0031] Predicted density-Map – DM loss – density map loss

[0032] Pre-trained Counter (Shared and Frozen) - A pre-trained target counting model.

[0033] Cycle loss - Cycle consistency loss; GAN loss - Adversarial loss.

[0034] G R抗性 - The process of generating a thermal infrared image from a visible light image (RGB).

[0035] G T过程 - The process of generating visible light images from thermal infrared images.

[0036] D R -Visible light image discrimination process, D T - Thermal infrared image discrimination process;

[0037] Figure 5 This represents the mean absolute error (MAE) and mean square error (MSE) for different k values;

[0038] Figure 6 MAE and MSE represent different Q values ​​(large, medium, and small scales);

[0039] Figure 7 shows a visualization analysis of three samples from the DroneRGBT dataset (compared to a unimodal model), where...

[0040] Figure 7(a) is a visible light image.

[0041] Figure 7(b) is a thermal infrared image.

[0042] Figure 7(c) is the truth density plot.

[0043] Figure 7(d) is the prediction density map of comparison method-4.

[0044] Figure 7(e) is the prediction density map of the proposed method-1;

[0045] Figure 8 shows a visualization analysis of three samples from the DroneRGBT dataset (compared to a multimodal model), in which...

[0046] Figure 8(a) is a visible light image.

[0047] Figure 8(b) is a thermal infrared image.

[0048] Figure 8(c) is the truth density plot.

[0049] Figure 8(d) is the prediction density map of the proposed method-2.

[0050] Figure 8(e) is the prediction density map of comparison method-8.

[0051] Figure 8(f) is the prediction density map of the proposed method-1;

[0052] Figures 9(a) to 9(d) This section presents a visualization analysis of three samples from the RGBT-CC dataset.

[0053] Figure 9(a) is a visible light image.

[0054] Figure 9(b) is a thermal infrared image.

[0055] Figure 9(c) is the truth density plot.

[0056] Figure 9(d) is the prediction density map of the proposed method-1;

[0057] Figures 10(a) to 10(c) This is a diagram showing the modal transition effect, where:

[0058] Figure 10(a) is a visible light image.

[0059] Figure 10(b) is the actual thermal infrared image.

[0060] Figure 10(c) shows the generated thermal infrared image. Detailed Implementation

[0061] 1 Overview

[0062] Considering the insufficient counting capability of UAVs in complex environments (e.g., low light, severe weather), this invention designs an online target counting method for UAVs based on a multimodal dynamic neural network.

[0063] Modal information is fused through conditional adversarial generation to enhance multimodal feature representation and reduce redundant computation in feature extraction.

[0064] When scheduling modal branches, the accuracy of each modality density map is considered to optimize the decision-making for multimodal gating. Modality transformation is achieved using a CycleGAN that incorporates density map loss to address the issue of missing modality data.

[0065] Experiments on two authoritative datasets show that, compared with CMCRL, the method reduces the mean absolute error by 8.6%–19.5% and the mean inference latency by 79%.

[0066] 2 Scheme Design

[0067] like Figure 2As shown, the counting model based on multimodal DyNN proposed in this invention mainly includes the following three networks:

[0068] 1) A feature extraction network for modality fusion is used to extract and fuse multimodal information. A discriminator is used to supervise the generator in learning modality consistency information to achieve the purpose of modality feature fusion.

[0069] 2) A decoder-assisted multimodal gating network is responsible for scheduling modal information in complex scenes. When the lighting conditions in the scene of the input image are good, the thermal infrared branch is turned off, and only the visible light mode is used; when the image is relatively dark, the thermal infrared branch is activated to assist in counting.

[0070] 3) A mode conversion network based on density map loss is used to compensate for the lack of modal information. If thermal infrared data is unavailable, thermal infrared data is generated from visible light data to enhance counting performance under low-light conditions.

[0071] The main workflows in the proposed framework include:

[0072] a. When thermal infrared images cannot be obtained, a mode conversion network based on density map loss provides a thermal infrared branch to generate thermal infrared images.

[0073] b. Decoder-assisted multimodal gating analysis of scene lighting to select necessary modal branches.

[0074] c. The modal fusion feature extraction network extracts the modal features of the selected branches and fuses the features of the multimodal inference flow.

[0075] In this example, the image input to the target counting model is a segmented image based on scale estimation. Here, the scale estimation is obtained based on the UAV's flight altitude and elevation angle.

[0076] 2.1 Feature Extraction Network for Modality Fusion

[0077] like Figure 2 As shown, a multimodal feature fusion-dependent Conditional Generative Adversarial Network (CGAN) consists of a generator and a discriminator. The generator generates modality-related features, which are then processed by scale-aware convolutions before entering the discriminator. The discriminator determines the modality to which the features generated by the generator belong. During the adversarial process, the discriminator aligns the features of the optical image with those of the thermal infrared image into the same domain, enabling the generator to express modality consistency. The generator and discriminator are described below.

[0078] A. Generator

[0079] Since visible light images and thermal infrared images are not spatially consistent, this invention must reconstruct the feature extraction method.

[0080] Unlike dual-stream networks with a large number of parameters, this invention uses a general feature extractor to learn shared parameters. Mboilenet v3 is used to construct the backbone network. Two low-parameter modal extractors are added to the backbone network to extract modal features from visible light and thermal infrared data, respectively. The role of the modal extractors is to enhance the features extracted from the two modalities, ensuring that the mined shared features do not lose model information. After the backbone network extracts the modal consistency features, they are added pixel-by-pixel to the features of the modal extractors. Taking a thermal infrared data point as an example, its features are calculated as follows:

[0081]

[0082] Among them, F g and These represent a general feature extractor and a modal feature extractor for thermal infrared data, respectively. and Then represents the parameters of the relevant extractor. After substituting the convolution function, equation (1) is simplified to

[0083]

[0084] Among them, W g Parameters representing a general feature extractor The parameters represent the modality feature extractor; the convolution operation is represented by ×. The generated new matrix, M... T , representing parameters related to the mode.

[0085] B. Discriminator

[0086] Target counting based on adversarial learning mainly focuses on the game between the predicted count density map and the true density map. In contrast, this invention constructs an adversarial learning framework for the feature representation of visible light and thermal infrared modes, enabling the features of these two modes to complement each other and achieve intermodal alignment.

[0087] The characteristics of the visible light mode and the thermal infrared mode output by the generator are respectively represented by F RGB and F T After passing through a 1×1 convolutional layer, they are transformed into corresponding feature maps S by the Sigmoid function. RGB and S T Subsequently, these feature maps are merged with the RGB image (visible light image) into a 4-channel feature map, which is then passed to the discriminator to determine the original modality type of the input sample. The generator is able to learn consistent representations among multimodal features. The discriminator uses a SalGAN-like architecture.

[13] The network structure.

[0088] C. Competitive training

[0089] k-root optimized Euclidean loss and adversarial loss were used for model training. Features F of the two modalities involved in adversarial learning... RGB and F T The discriminators are mapped to the same feature space and processed through convolutional layers and a sigmoid function to obtain their discrimination results. During discriminator training, the generator is locked. During generator training, the generator's parameters are updated, and the discriminator's loss is backpropagated using gradients. It's important to note that the discriminator is only used during the training phase to support adversarial learning between modality features; that is, only the generator and decoder are used during the inference phase. The discriminator's loss function is expressed as...

[0090] L D =L(D(data) RGB ,S RGB ),1)+L(D(data RGB ,S T ),0) (3)

[0091] Where data RGB This represents the ground truth label used as auxiliary information in CGAN, i.e., the original visible light image. 1 and 0 represent the corresponding classifications for visible light and thermal infrared modes, respectively. The overall loss of the training model to obtain the final density map is expressed as...

[0092]

[0093] 2.2 Decoder-Assisted Multimodal Gating Network

[0094] The key to high-precision, low-redundancy counting lies in adaptively scheduling modal branches based on input instances. To this end, this invention proposes a decoder-assisted multimodal gating network. For instances captured under conditions such as low light or dust storms, this method achieves optimal matching of multiple modal information by dynamically invoking modal information. This method is jointly constructed by a decoder and a multimodal gating system. The multimodal gating system is a binary classification neural network used to decide whether to open or close branches. The decoder's output serves as the basis for training the multimodal gating system. The decoder and multimodal gating system are described below.

[0095] A. Decoder

[0096] The input raw image undergoes multiple convolutional downsampling operations, which, while extracting deep semantic information, results in a loss of target details. To address this, features are upsampled to restore resolution and generate a high-quality density map. The decoder first performs three upsampling operations on the features to restore the image to its initial size. Then, density map branches for the two modalities are separated. These two branches can output density maps for the visible light and thermal infrared modalities, respectively. Finally, a convolution operation is performed on the two stitched density maps to obtain a density map that fuses multimodal (two modalities) information.

[0097] Three density maps (density maps generated from visible light images, PR) RGB Density map PR generated from thermal infrared images T Density map PR after fusing information from two modalities final The comparison results determine the label values ​​of the multimodal gating system during the training process (details will be provided later).

[0098] When only visible light modal data is used, PR RGB As the final output of the network;

[0099] When using multimodal data, PR RGB and PR T Further fusion generates a more accurate density map.

[0100] The decoder provides flexible options for density map output to accommodate the decision results of pre-multimodal gating.

[0101] B. Multimodal gating

[0102] The input to the multimodal gating is a segmented visible light image. Features extracted by shallow convolutions are fed into a fully connected layer, which outputs a confidence score for modality selection, guiding the opening and closing of the thermal infrared branch. The convolutions use the first block of a ResNet50 array to analyze the illumination intensity in the image. The labels during training are set to...

[0103]

[0104] Among them, PR RGB PR represents the density map generated from a visible light image. final This represents the density map after the fusion of visible light and thermal infrared information. The threshold representing the label.

[0105] When visible light and thermal infrared images pass through the entire network, PR is obtained. RGB PR T and PR final According to equation (5), a label p = 1 means that the thermal infrared mode branch can detect more targets that the visible light branch did not identify, since PR at this timeRGB The image contains large shadowed areas; the thermal infrared branch was used to enhance the visible light branch. The label p=0 indicates "Turn off the thermal infrared branch".

[0106] When the UAV is in a well-lit environment (see Figure 3(a)), it can obtain relatively accurate counting results using only visible light data. Therefore, the UAV tends to choose visible light single-mode data. When the UAV is in a poorly lit environment (see Figure 3(b)), thermal infrared information is added as auxiliary data to the inference process.

[0107] 2.3 Density-Map Modal Transformation (DMMT) Network Based on Density Map Loss

[0108] While thermal infrared imagery is essential in low-light environments, it may be impossible for drones to capture it in certain conditions. In such cases, visible light data is used to generate thermal infrared data (i.e., "pseudo-thermal infrared imagery"), and both the visible light data and the generated thermal infrared data are input into the target counting model for counting.

[0109] CycleGANs are often used to solve the modality transformation problem, but considering that the loss function cannot constrain local details (local patterns and texture features), this invention proposes a density map loss-based modality transformation network (DMMT) to generate thermal infrared images. The aim is to minimize the spatial distance for target counting between the generated (pseudo) thermal infrared image and the real thermal infrared image.

[0110] During modality transformation, the pre-trained target counting model is treated as a spatial mapping transformer. The pseudo-thermal infrared images and real thermal infrared images generated by CycleGAN are transformed into density map space (density map space refers to the density distribution of certain features (such as objects, people, substances, etc.) in space). A density map mean square loss is used in the additionally introduced density map space to force the generated image to approximate the real image. This additional loss forces CycleGAN to focus its attention on the targets in the image.

[0111] The DMMT framework mainly consists of a CycleGAN and a pre-trained target counter model, such as Figure 4 As shown.

[0112] In CycleGAN, the generator and discriminator are denoted as G and D, respectively. R and T represent the visible light and thermal infrared domains, respectively, and r and t represent the data in the visible light domain and the thermal infrared domain, respectively. The adversarial loss L... GAN And cycle consistency loss L cycle Represented as

[14]

[0113]

[0114] and

[0115]

[0116] The density map mean square loss is expressed as

[0117]

[0118] Where ψ represents the spatial mapping performed by the pre-trained counting model, and M(·,·) represents the mean square error between the generated image and the real image in the density map space.

[0119] Based on equations (6), (7), and (8), the loss function of DMG (Density Management Graphic) is expressed as follows:

[0120]

[0121] Where λ and μ are the weights of the cycle consistency loss and density map loss, respectively.

[0122] 3. Experimental Design and Results Analysis

[0123] To enhance the reference value of the experimental results, two authoritative datasets containing complex scenes were selected: DroneRGBT.

[15] This is a multimodal crowd counting dataset collected from a drone's perspective, containing 3600 images covering various complex scenes from different altitudes, lighting conditions, and crowd densities. It includes images taken at three flight altitudes, which can be used to validate the effectiveness of the proposed multi-scale receptive field strategy. Regarding lighting conditions, the image pairs cover various scenarios such as dark, twilight, and bright light, simulating the complex environments encountered by drones in dynamic deployments. In terms of density, the dataset has a wide density range, from 1 to 401. The images in DroneRGBT were collected by a drone (DJIPhantom 4) and cover complex backgrounds such as parks, highways, streets, squares, subways, and shopping malls. RGBT

[16] This is a large-scale crowd counting dataset containing 2030 pairs of 640×480 resolution RGB thermal images taken in various scenes. The dataset covers a variety of scenes, such as shopping malls, streets, amusement parks, train stations, and subway stations. Of these, 1013 pairs were acquired under bright lighting conditions, while the other 1017 pairs were acquired under low-light conditions. Compared to some datasets sourced from the internet, the crowd density distribution of RGBT-CC more closely resembles real-world urban scenes.

[0124] The Adam optimizer was used for density map generation training, where the learning rate was set to 1×10. -5 Adversarial loss was used to alternately optimize the alignment and decision models to align visible light and thermal infrared features. The learning rate of the Adam optimizer was set to 1×10⁻⁶ during the adversarial training phase. -5 These settings help ensure the stability and convergence speed of the model, while taking into account the accuracy and robustness of the population count. In Equation (7), λ and μ are set to 0.1 and 0.1, respectively.

[0125] Comparative experiments on the DroneRGBT dataset aim to determine the value of k in the distribution-aware block loss. For example... Figure 5 As shown, since MAE and MSE perform best at k=3, this value was used in subsequent experiments. I obtained the optimal settings for Q at various scales through experiments on DroneRGBT, and summarized them as follows: Figure 6 When MAE and MSE reach their minimum values, Q = (1,2,3), (3,4,5), (5,6,7).

[0126] The discriminator's parameter settings are shown in Table 1. It mainly consists of six 3×3 convolutional layers, Conv1 to Conv6, with ReLU activation after each convolutional layer and max pooling after every two convolutional layers. Fc7 to Fc9 represent three fully connected layers, using Tanh and Sigmoid activation functions respectively.

[0127] Table 1: Structure of the discriminator

[0128] Table 1: Composition of discriminator

[0129]

[0130] This experiment categorizes the proposed method into the following six types to observe its counting performance under different inference flow patterns:

[0131] • Proposed Method-1: Dynamic inference using data from visible light and real thermal infrared modes;

[0132] • Proposed Method-2: Inference is performed using only visible light mode data;

[0133] • Proposed Method-3: Inference is performed using only data from real thermal infrared modes;

[0134] • Proposed Method-4: Inference is performed using data from both visible light and real thermal infrared modes simultaneously;

[0135] • Proposed Method-5: Dynamic inference using data from visible light and generated thermal infrared modes;

[0136] • Proposed Method-6: Dual-stream structure, using data from visible light and real thermal infrared modes for dynamic inference.

[0137] To verify the environmental adaptability and counting performance of the proposed method, this experiment selected some existing mainstream single-modal and multi-modal models as benchmark methods:

[0138] • Comparison Method-1 (derived from MCNN)

[17] A lightweight single-modal multi-column model that utilizes multiple sets of convolutional kernels to cover a large receptive field;

[0139] Comparison Method 2 (derived from SANET)

[18] ): Introduce a scale aggregation module into the encoder to improve the representational power and scale diversity of features;

[0140] Comparison Method 3 (from CSRNET)

[19] ): Utilize dilated convolutions to encode deeper features in crowded scenes;

[0141] Comparison Method 4 (derived from CANNET)

[20] ): A robust model that uses detailed information from features as attention-guided scale fusion;

[0142] • Comparison Method 5 (derived from BL)

[21] ): The expected count of each point label is monitored using the Bayesian Loss function;

[0143] Comparison Method 6 (from AACVCNET)

[22] ): Optimize counting accuracy and inference latency of UAV images by leveraging spatial information and multi-scale receptive fields;

[0144] Comparison Method 7 (derived from RMMCC)

[23] ): Add counting constraints during multimodal fusion and use a multi-scale token transformer to perceive targets at different scales;

[0145] • Comparison Method - 8 (derived from CMCRL)

[24] This framework integrates cross-modal collaborative representation learning, learning complementary information between different modalities through an information aggregation and distribution module.

[0146] 3.1 Comparative Experiment with Single-Mode Model

[0147] Table 2: Comparison of counting performance with unimodal models on the DroneRGBT dataset

[0148] Table 2: Counting performance comparison with mono-modality models onDroneRGBT

[0149]

[0150]

[0151] To demonstrate that the proposed multimodal model outperforms the single-modal model, experiments were conducted to test the performance of existing counting models on single-modal data. These models were trained on single-modal data and tested on corresponding test sets; the experimental results are shown in Table 2. When using single-modal data in visible light or thermal infrared for counting tasks, proposed methods-2 and-3 significantly outperformed comparative method-1 in counting accuracy, but slightly lagged behind comparative method-4, etc. Comparison with comparative method-6 shows that the proposed method maintains a similar single-modal information understanding capability after fusing multiple modal information. When using thermal infrared data to assist visible light target counting, the counting accuracy of proposed method-1 surpassed all single-modal models, indicating that the method designed in this chapter effectively understands and utilizes target information from multiple modalities. Figures 7(a) to 7(e) The density map visualization results show that the advantage of method-4 is that it is more accurate in counting the visible light portion, but it still cannot effectively identify objects in dark light areas. In contrast, the proposed method-1 achieves accurate counting in aerial images under various lighting conditions.

[0152] 3.2 Comparative Experiment with Multimodal Model

[0153] This section compares the proposed method with a representative multimodal counting model. The proposed method-6 is a two-stream network where both branches use the first four blocks of MobileNet V3 to extract features relevant to each modality. The feature maps output from each branch are then concatenated by channel and subjected to dimensionality reduction using a 1×1 convolutional kernel. Finally, the fused features are input into a regression module to obtain the final density map.

[0154] Table 3: Comparison of counting performance with multimodal models on the DroneRGBT and RGBT-CC datasets

[0155] Table 3: Counting comparison with multi-modality models on DroneRGBTand RGBT-CC

[0156]

[0157] As shown in Table 3, in the DroneRGBT dataset, the MAE of the proposed method-1 is 7.26, approaching 96.1% of the comparative method-7. However, it is lower than that of the proposed method-6, because the two-stream network improves the model's understanding ability by increasing the number of parameters. The proposed method-1 achieves acceptable counting performance with a much smaller number of parameters than the proposed method-6. When running the multimodal model, thermal infrared information helps to distinguish potential targets from cluttered backgrounds, while visible light information helps to eliminate negatively heated objects in thermal infrared images and eliminate interference from similar objects. This demonstrates that the model fully resolves the key features of targets in the image, and modal information is utilized efficiently. Furthermore, in the proposed method, the processing of the two modal information is decoupled, meaning it can work even with only visible light images. When there is sufficient lighting in the entire scene, the model can switch to a single-modal inference mode to achieve high-precision counting at a lower cost.

[0158] The proportion of dark regions is highly correlated with the model's inference latency. Table 3 summarizes the average inference latency on the DroneRGBT dataset. Notably, these average inference latencies are not linearly related to the number of modes. The proposed multimodal gating can dynamically activate and mobilize the thermal infrared branch for dark environments, thereby reducing inference costs. Figures 8(a) to 8(f) and Figures 9(a) to 9(d) The visualization inference results of the proposed method on the DroneRGBT and RGBT-CC datasets are presented. Due to the lack of clear features in visible light images under low light conditions, the proposed method-2 exhibits numerous false positives and false negatives. The baseline method, which simultaneously references thermal infrared and visible light images, captured most targets in low-light environments. In contrast, the proposed method-1 can distinguish targets under various lighting conditions, maintaining low false positive and false negative rates.

[0159] 3.2 Ablation Experiment

[0160] To verify the effectiveness of each network in the proposed framework, this section designs ablation experiments on the DroneRGBT dataset in modules.

[0161] This experiment first verifies whether a feature extraction network for modality fusion helps improve counting accuracy. As shown in Table 4, CGAN was not used for feature extraction; data from both modalities were processed through the same backbone network. The results readily demonstrate that the MAE without CGAN is significantly lower than the proposed method-1, because interference between features from different modalities hinders the correct representation of features. This result indirectly confirms the unique and crucial role of CGAN in understanding and extracting the common representations of various modalities.

[0162] Table 4: Comparison of ablation experimental results on the DroneRGBT dataset

[0163] Table 4: Comparison of results from ablation experiments on DroneRGBT

[0164]

[0165] Next, the impact of multimodal gating on counting performance and inference latency is examined. Table 4 shows that when multimodal gating is implemented (Proposed Method-1), the model's MAE is superior to the global unimodal model (Proposed Methods-2 and-3). When multimodal gating is removed (Proposed Method-4), the model consistently uses two modalities for counting. In this case, its MAE is reduced by only 2.1%, but it introduces many unnecessary computations. Therefore, multimodal gating plays a crucial role in maintaining counting stability and reducing the cost of counting inference.

[0166] Finally, the effectiveness of the mode transfer network was verified. Figures 10(a) to 10(c) In the generated thermal infrared images, compared to the real thermal infrared images, the object's outline and brightness are shown. Table 4 presents the experimental results on the DroneRGBT dataset. When all the real thermal infrared images in the dataset were replaced with thermal infrared images generated by the mode conversion network (proposed method-5), the MAE improved by only 9.4%, indicating that the mode conversion network can meet the model's requirements for thermal infrared data when modes are missing.

[0167] 4. Summary

[0168] This invention proposes an online target counting method for unmanned aerial vehicles (UAVs) based on a multimodal dynamic neural network (GNNN). This method improves the counting accuracy of UAV multimodal images while maintaining a low computational load by fusing multimodal information through CGAN. Multimodal gating dynamically schedules the inference of modal branches, balancing counting accuracy and inference latency. Furthermore, even with only visible light data, the model can perform modal transformation using DMMT to generate thermal infrared images from visible light images.

[0169] Experimental results show that the proposed model can perform efficient counting in various input scenarios. The proposed method can cope with harsh natural environments, adapting to changes in the UAV's flight altitude, tilt angle, and ambient lighting. The proposed multimodal DyNN framework is universal and customizable, and can be further applied to scenarios such as aerial target detection and target segmentation.

[0170] References

[0171] [1]Iftikhar S,Asim M,Zhang Z,et al.Target detection and recognitionfor traffic congestion in smart cities using deep learning-enabled UAVs:Areview and analysis[J].Applied Sciences,2023,13(6):3995.

[0172] [2]Sun Y,Cao B,Zhu P,et al.Detfusion:A detection-driven infrared andvisible image fusion network[C] / / Proceedings of the 30th ACM internationalconference on multimedia.2022:4003-4011.

[0173] [3]Zhang,X.,Ye,P.,Peng,S.,Liu,J.,Xiao,G.:Dsiammft:An rgb-t fusiontracking method via dynamic siamese networks using multi-layer featurefusion.Signal Processing:Image Communication(2020)115756.

[0174] [4]Li X,Chen H,Li Y,et al.MAFusion:Multiscale attention network forinfrared and visible image fusion[J].IEEE Transactions on Instrumentation andMeasurement,2022,71:1-16.

[0175] [5]Zhao W,Xie S,Zhao F,et al.Metafusion:Infrared and visible imagefusion via meta-feature embedding from object detection[C] / / Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:13955-13965.

[0176] [6]Qingyun F,Dapeng H,Zhaokui W.Cross-Modality Fusion Transformer forMultispectral Object Detection[J].2021.

[0177] [7]Sun Y,Cao B,Zhu P,et al.Drone-based RGB-infrared cross-modalityvehicle detection via uncertainty-aware learning[J].IEEE Transactions onCircuits and Systems for Video Technology,2022,32(10):6700-6713.

[0178] [8]G.Huang,D.Chen,T.Li,F.Wu,L.Van Der Maaten,and K.Q.Weinberger,Multi-scale dense networks for resource efficient image classification,arXivpreprint,2017,DOI 10.48550 / arXiv.1703.09844.

[0179] [9]A.Veit and S.Belongie,Convolutional networks with adaptiveinference graphs,Proc.of the European Conference on Computer Vision(ECCV),Munich,Germany),2018,pp.3–18.

[0180]

[10] D.Eigen,M.Ranzato,and I.Sutskever,Learning factoredrepresentations in a deep mixture of experts,arXiv preprint,2013,DOI10.48550 / arXiv.1312.4314.

[0181]

[11] Li G,Liu Z,Ling H.ICNet:Information conversion network for RGB-Dbased salient object detection.IEEE Trans.on Image Processing,2020,29:4873-4884.

[0182]

[12] Li F,Zha Y,Zhang L,et al.Information lossless multi-modal imagegeneration for RGB-T tracking[C] / / Chinese Conference on Pattern Recognitionand Computer Vision(PRCV).Cham:Springer Nature Switzerland,2022:671-683.

[0183]

[13] Pan J,Sayrol E,Nieto X G,et al.Salgan:Visual saliency predictionwith adversarial networks[C] / / CVPR scene understanding workshop(SUNw).2017.

[0184]

[14] Goodfellow I,Pouget-Abadie J,Mirza M,et al.Generative adversarialnetworks[J].Communications of the ACM,2020,63(11):139-144.

[0185]

[15] Peng T,Li Q,Zhu P.Rgb-t crowd counting from drone:A benchmark andmmccn network[C] / / Proceedings of the Asian conference on computervision.2020.

[0186]

[16] Liu Z,Wu W,Tan Y,et al.RGB-T Multi-Modal Crowd Counting Based onTransformer[J].arXiv preprint arXiv:2301.03033,2023.Shen H,Liu X,Shi N,etal.Blockchain-enabled solution for secure and scalable V2V video contentdissemination[J].Peer-to-Peer Networking and Applications,2022:1-17.

[0187]

[17] Yingying Zhang,Desen Zhou,Siqin Chen,Shenghua Gao,and Yi Ma,Single-image crowd counting via multi-column convolutional neural network,Proc.of the IEEE conference on computer vision and pattern recognition(2016),589–597.

[0188]

[18] Cao X,Wang Z,Zhao Y,et al.Scale aggregation network for accurateand efficient crowd counting[C].In Proceedings of the European Conference onComputer Vision(ECCV),2018:734–750.

[0189]

[19] Yuhong Li,Xiaofan Zhang,and Deming Chen,CSRNet:Dilatedconvolutional neural networks for understanding the highly congested scenes,Proc.of the IEEE conference on computer vision and pattern recognition(2018),1091–1100.

[0190]

[20] Liu W,Salzmann M,Fua P.Context-aware crowd counting[C].InProceedings of the IEEE Conference on Computer Vision and PatternRecognition,2019:5099–5108.

[0191]

[21] Ma Z,Wei X,Hong X,et al.Bayesian loss for crowd count estimationwith point supervision [C].In Proceedings of the IEEE InternationalConference on Computer Vision,2019:6142–6151.

[0192]

[22] Liu Y,Shen H,Wang T,et al.Vehicle counting in drone images:Anadaptive method with spatial attention and multiscale receptive fields[J].ETRI Journal,2024.

[0193]

[23] Liu Z,Wu W,Tan Y,et al.RGB-T Multi-Modal Crowd Counting Based onTransformer[J].arXiv preprint arXiv:2301.03033,2023.

[0194]

[24] Liu L,Chen J,Wu H,et al.Cross-Modal Collaborative RepresentationLearning and a Large-Scale RGBT Benchmark for Crowd Counting[J].2020.DOI:10.48550 / arXiv.2012.04529.

Claims

1. A method for online target counting of unmanned aerial vehicles (UAVs) based on a multimodal dynamic neural network, comprising the following steps: First, the drone uses its onboard visible light camera and thermal infrared camera to take aerial photos of visible light and thermal infrared images, respectively. Then, these images are segmented based on scale estimation; The segmented image is used as input to the target counting model for target counting; Its characteristic is that the target counting model is based on the multimodal dynamic neural network DyNN; the architecture of the target counting model includes a feature extraction network, a decoder, a multimodal gating network, and a mode transformation network; in the target counting model: S1. When thermal infrared images cannot be obtained, the visible light image is converted into a thermal infrared image by a mode conversion network, and the generated thermal infrared image is provided for the thermal infrared mode branch of the target counting model. S2. Multimodal gating network analysis of the scene illumination of the image; if the illumination is good, only the visible light mode branch in the target counting model is selected; if the illumination is insufficient, the visible light mode branch and the thermal infrared mode branch are selected, with the thermal infrared image as auxiliary data. S3. The feature extraction network extracts the modal features of the selected modal branches and fuses the features of the multimodal inference flow; S4. The features obtained in step S3 are decoded to obtain the output of the target counting model; In step S1, the modality transformation network is a density map loss-based modality transformation network DMMT; DMMT includes a recurrent generative adversarial network CycleGAN and a pre-trained target counter model; during the modality transformation process, the pre-trained target counter model is regarded as a spatial mapping converter; The thermal infrared images generated by CycleGAN and the real thermal infrared images are respectively transformed into density map space; density map mean square loss is used in the additionally introduced density map space to force the generated images to approximate the real images; In CycleGAN, the generator and discriminator are represented by G and D, respectively. R and T represent the visible light domain and the thermal infrared domain, respectively. r and t represent the data in the visible light domain and the data in the thermal infrared domain. Adversarial loss and cycle consistency loss They are represented as follows: ,and ； The density plot mean square loss is expressed as: ， in, This represents the spatial mapping performed using a pre-trained counting model. This represents the mean square error between the generated image and the real image in the density map space; The loss function for generating DMG from density maps is then expressed as: ， in, and These are the weights of the cycle consistency loss and the density map loss, respectively. In step S2, the multimodal gating network is a decoder-assisted multimodal gating network, wherein the multimodal gating network is a binary classification neural network used to decide whether to open or close the modal branch; the output of the decoder serves as the basis for training the multimodal gating network. S2.1 In the decoder, firstly, the features obtained in S3 are upsampled to restore the image to its initial size; then, the density map branches of the visible light mode and the thermal infrared mode are separated to obtain the density maps of the visible light mode and the thermal infrared mode; finally, the two density maps are convolved to obtain the density map of the fusion of the two modal information. The three density maps are represented as follows: the density map PR generated from the visible light image. RGB Density map PR generated from thermal infrared images T Density map PR after fusing information from two modalities final ;PR RGB PR T and PR final The comparison results determine the label values ​​of the multimodal gating network during the training process; When only visible light modal data is used, PR RGB As output; when using two modalities of data, PR final As output; S2.2 In the multimodal gating network, its input is a visible light image; the features extracted by the shallow convolution enter the fully connected layer, and the output is the confidence score of the modality selection, which guides the opening and closing of the thermal infrared branch; the convolution uses the first block of ResNet50 to analyze the light intensity in the image; Labels during training of multimodal gating networks ， in, The threshold representing the label; If the label p=1, it means that the thermal infrared mode branch can detect enough targets that the visible light branch did not recognize. If the label p=0, the thermal infrared mode branch is turned off.

2. The online target counting method for UAVs based on a multimodal dynamic neural network according to claim 1, characterized in that... In step S3, the feature extraction network is a modality fusion feature extraction network. Multimodal feature fusion relies on a generative adversarial network (GAN), which includes a generator and a discriminator. The generator generates modality-related features, which are then fed into the discriminator after undergoing scale-aware convolution operations. The discriminator determines the modality to which the features generated by the generator belong; During adversarial training, the discriminator aligns the features of the visible light image with the features of the thermal infrared image to the same domain, enabling the generator to have the ability to express modal consistency. The discriminator is used only during the training phase to support adversarial learning between modal features; The inference phase uses only the generator and decoder.

3. The online target counting method for unmanned aerial vehicles based on a multimodal dynamic neural network according to claim 2, characterized in that feature extraction... In the network, GAN stands for Conditional Generative Adversarial Network (CGAN).

4. The online target counting method for unmanned aerial vehicles based on a multimodal dynamic neural network according to claim 3, Its characteristics are found in CGAN: S3.1 Generator Mobilenet v3 was used to build the backbone network; two low-parameter modal extractors were added to the backbone network to extract modal features from visible light image data and thermal infrared image data, respectively. After the modality consistency features are extracted from the backbone network, they are added pixel by pixel to the features extracted by the modality extractor. For any given thermal infrared image data, its thermal infrared modal characteristics are calculated as follows: ， in, and These represent a general feature extractor and a modal feature extractor for thermal infrared image data, respectively. and These represent the parameters of the general feature extractor and the modal feature extractor, respectively; After substituting the convolution function, the above equation simplifies to: ， in, The parameters represent the general feature extractor. The parameters represent the modality feature extractor, the convolution operation is represented by ×, and the generated new matrix Represents parameters related to the mode; For any visible light image data, the visible light modal features are obtained using the same method as the feature extraction method for the thermal infrared image data described above. ; S3.1, Discriminator first, and After passing through a 1×1 convolutional layer, they are then transformed into corresponding feature maps using the Sigmoid function. and ; Subsequently, and The RGB image is merged into a 4-channel feature map and passed to a discriminator to determine the original modality type of the input sample. S3.3, Competitive Training k-root optimized Euclidean loss and adversarial loss were used for training; Participating in adversarial learning and They are mapped to the same feature space and pass through convolutional layers and the sigmoid function to obtain the discriminator's discrimination result; The generator is locked while the discriminator is being trained; the generator's parameters are updated while the discriminator's loss is backpropagated via gradients. The loss function of the discriminator is expressed as: ， in represents the true label used as auxiliary information in CGAN, i.e., the original visible light image; 1 and 0 represent the corresponding classifications of the visible light mode and the thermal infrared mode, respectively; the overall loss of the final density map obtained through training is expressed as... 。

Images