An unmanned aerial vehicle target positioning method based on heterogeneous data fusion and deep learning

Abstract

The application discloses a kind of unmanned aerial vehicle target positioning methods based on heterogeneous data fusion and deep learning, including steps as follows: S1, by data perception module, the geographic position information of unmanned aerial vehicle, attitude, acceleration information and image data are collected, and the data collected are preprocessed, and multi-modal input tensor is constructed;S2, the multi-modal deep fusion model constructed is trained;S3, in the process of flight, the multi-modal deep fusion model trained offline is used to carry out real-time inference, and online error correction and self-adaptive weighting mechanism are combined, realize centimeter-level high-precision positioning and stable control.The present application can judge model reliability in real time according to environmental change, and automatically switch source of confidence according to uncertainty, effectively solve the problem that traditional algorithm cannot correct error online.

Description

A UAV Target Localization Method Based on Heterogeneous Data Fusion and Deep Learning Technical Field

[0001] This invention relates to the field of intelligent control and target localization technology for unmanned aerial vehicles (UAVs), and in particular to a UAV target localization method based on heterogeneous data fusion and deep learning. Background Technology

[0002] With the widespread application of drones in reconnaissance and surveillance, counter-terrorism and bomb disposal, and emergency search and rescue missions, high-precision positioning and target identification in complex terrain have become crucial technologies. Taking disaster scenarios as an example, collapsed buildings, narrow streets, or tree obstructions can all reduce the accuracy of traditional positioning methods, impacting the operational safety and mission efficiency of drones. Compared to ground-based search and rescue forces, drones can quickly acquire a global perspective from the air and perform low-altitude searches in areas at risk of collapse, significantly improving search and rescue efficiency and reducing safety risks.

[0003] Currently, common UAV positioning methods are mainly based on a combination of Global Positioning System (GPS), Inertial Measurement Unit (IMU), and visual sensors. However, existing methods have the following shortcomings: GPS suffers from low position update rate and unstable accuracy in environments with building obstruction, multipath interference, or weak satellite signals; while IMU can provide high-frequency attitude and acceleration information, its drift error accumulates over time; visual measurement is prone to unstable recognition and feature drift in strong light, shadow, or low-texture scenes; traditional fusion algorithms often use fixed weights or linear filtering models, which cannot adapt to changes in sensor uncertainty, leading to increased positioning errors in dynamic environments.

[0004] Especially in low-altitude search and rescue and close-range inspection missions, UAVs need to hover, turn or descend rapidly frequently. Their attitude changes and wind disturbances cause nonlinear deviations in the sensor sampling timing, which further affects the positioning accuracy. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a UAV target localization method based on heterogeneous data fusion and deep learning. This method can combine GPS, IMU and visual sensor data, and improve the positioning accuracy and real-time performance in complex lighting, occlusion and attitude disturbance environments by dynamically adjusting the fusion weights and time-series compensation mechanism. This enhances the environmental adaptability and reliability of UAVs in search and rescue, inspection and other tasks.

[0006] Technical Solution: A UAV target localization method based on heterogeneous data fusion and deep learning, comprising the following steps:

[0007] S1, the data perception module collects the UAV's geographical location information, attitude, acceleration information and image data, and preprocesses the collected data to construct a multimodal input vector;

[0008] S2 uses multimodal input vectors to train the constructed multimodal deep fusion model;

[0009] During flight, S3 employs an offline-trained multimodal deep fusion model for real-time inference, combined with online error correction and adaptive weighting mechanisms, to achieve centimeter-level high-precision positioning and stable control.

[0010] Furthermore, the preprocessing of the collected data includes:

[0011] A11, GPS coarse positioning and speed compensation;

[0012] The GPS position is dynamically compensated by the linear velocity calculated from GPS and IMU to obtain continuous position estimates:

[0013] ,

[0014] in, The position coordinates in the global geographic coordinate system after compensation. To obtain and convert the location coordinate vectors to a global geographic coordinate system using a GPS receiver, Let be the linear velocity vector of the UAV in the same coordinate system. The average transmission and processing time delay of the GPS signal relative to the IMU time reference, which is pre-calibrated by the system.

[0015] A12, camera pixel offset calculation;

[0016] The camera determines the center position of the target in the image coordinate system using a target detection algorithm. With the image center For reference, the pixel offset is defined as:

[0017] ,

[0018] in, This represents the pixel offset along the x-axis. This is the pixel offset along the y-axis;

[0019] Combined with the camera's intrinsic parameter matrix The direction vector of the target in the camera coordinate system is calculated by back projection. :

[0020] ,

[0021] in, Let T denote the camera intrinsic parameter matrix;

[0022] A13, Attitude Angle Acquisition and Coordinate Transformation;

[0023] To align the UAV coordinate system with the geographic coordinate system, an Euler angle rotation matrix is ​​used for coordinate transformation:

[0024] ,

[0025] in, This represents the total rotation matrix used to transform the UAV's body coordinate system to the geographic coordinate system;

[0026] , , These represent the yaw, pitch, and roll angles of the UAV, respectively, collected by the IMU.

[0027] Indicates rotation about the Z-axis The basic rotation matrix of an angle;

[0028] Indicates rotation about the Y-axis The basic rotation matrix of an angle;

[0029] Indicates rotation about the X-axis The basic rotation matrix of an angle;

[0030] Next, the attitude matrix provided by the IMU is used. and camera external parameters Transform the line-of-sight vector to the geographic coordinate system:

[0031] ,

[0032] in, The line-of-sight vector in the geographic coordinate system;

[0033] Let the position of the UAV in the geographic coordinate system be... Assuming the ground height is 0, the approximate coordinates of the target are... The answer can be obtained by finding the intersection of the ray and the ground plane.

[0034] ,

[0035] in, X represents the vertical component of the line-of-sight vector; X, Y, and Z represent the three-dimensional position coordinates in the geographic coordinate system. This represents the approximate coordinates of the target detected by the camera.

[0036] Subsequently, time synchronization and feature combination are performed to align asynchronous multi-source data to the same timestamp and construct a multimodal input tensor.

[0037] Furthermore, the process of constructing the multimodal input tensor is as follows: using IMU high-frequency data as the time reference, linear interpolation is performed on GPS and camera frame data to align GPS position, visual offset, and IMU attitude under a unified timestamp, generating multimodal joint samples; the final constructed multimodal input vector includes: UAV attitude angle, camera pixel offset, compensated geographic location coordinates, target coarse coordinates, and target color features, which serve as the input to the multimodal deep fusion model.

[0038] Furthermore, the multimodal deep fusion model includes: a feature extraction module, a pose-related attention module, and a coordinate regression and uncertainty estimation module;

[0039] The feature extraction module, through a multi-layer fully connected structure, achieves nonlinear mapping of high-dimensional heterogeneous inputs, uncovers the deep correlation between UAV flight status and visual features, and transforms the raw data into a high-dimensional feature representation, defined as:

[0040] ,

[0041] ,

[0042] in, , These are the weight matrix and bias vector of the first layer of the feature extraction network, respectively; This is the output feature vector of the first hidden layer; For the first The ReLU activation function of the layer. For the first The output feature vector of the hidden layer;

[0043] The pose-related attention module introduces a pose angle-based attention mechanism to assign adaptive attention weights to features of different modalities; the expression for the adaptive attention weights is as follows:

[0044] ,

[0045] in, This represents the adaptive attention weight of the j-th modality in the current pose; It is a nonlinear mapping network containing two fully connected layers, used to map pose data to the same dimension as the feature vector to achieve feature space alignment; Let be the feature vector of the i-th sample and the j-th mode. The k-th modal eigenvector during summation of the denominator; Let be the roll angle, pitch angle, and yaw angle of the UAV corresponding to the i-th sample;

[0046] A unified global feature is obtained through weighted fusion. express:

[0047] ,

[0048] in, This is the global feature vector after attention-weighted fusion;

[0049] The coordinate regression and uncertainty estimation module, while predicting the target's geographical location, outputs the uncertainty of the prediction result, as shown in the following expression:

[0050] ,

[0051] in, This represents the two-dimensional location coordinates of the target in the geographic coordinate system predicted by the model. For parameterized regression networks, This represents the uncertainty estimate of the model.

[0052] Furthermore, an improved heteroscedasticity loss function is used during training, expressed as follows:

[0053] ,

[0054] ,

[0055] in, Let be the total loss function during network training. The batch size during training. To smooth the L1 loss function, For uncertainty regularization coefficients, for The threshold parameter of the loss function; The term acts as a precision weight; when the model predicts that the uncertainty of the current sample is high, the regression loss weight of that sample is automatically reduced. For regularization terms; This represents the two-dimensional position coordinate vector predicted by the model for the i-th sample. This represents the true geographical location coordinate vector of the i-th sample.

[0056] Furthermore, during training, a dynamic weighted correction mechanism is used for online adaptive correction. The expression for the dynamic weighted correction mechanism is as follows:

[0057] ,

[0058] ,

[0059] in, This is the final corrected position estimate. For the model to predict the location, The estimation results are from a single sensor. For the weighting function based on uncertainty, This is the adjustment coefficient for the hyperparameter. This represents the uncertainty of the model prediction.

[0060] If the model uncertainty increases, it will automatically increase. This enables real-time adaptive correction.

[0061] Furthermore, the steps to achieve centimeter-level high-precision positioning and stable control include:

[0062] S31, Standardize the input features;

[0063] S32, the standardized raw input The multimodal deep fusion model deployed in the flight control computing unit is fed into the system for real-time inference; forward computation is performed without enabling gradients.

[0064] ,

[0065] Among them, reasoning output This is the comprehensive prediction vector output by the model. For a trained multimodal deep fusion model, Represents fixed weight parameters; comprehensive prediction vector Including position coordinates and uncertainty ;

[0066] The multimodal deep fusion model incorporates a posture attention module, which dynamically adjusts the attention weights of each modality based on the posture angle at each moment.

[0067] S33 performs inverse normalization and coordinate restoration on the model output, recovering it to the true physical coordinate system:

[0068] ,

[0069] in, For the inverse normalization function, Predicted target location for UAV in geographic coordinate system;

[0070] S34, perform online error correction, if the error Then, dynamic fusion correction will be performed:

[0071] ,

[0072] in, The final target location is obtained after weighting. These are adaptive weighting coefficients, with a range of values. ;

[0073] ,

[0074] in, To calculate the Euclidean distance between the predicted results and the sensor results, For the threshold; Real-time observed coordinates of the sensor;

[0075] S35, Uncertainty based on the output of the multimodal deep fusion model Real-time weight adjustment:

[0076] ,

[0077] ,

[0078] The final correction output of the entire positioning system for:

[0079] ,

[0080] in, To integrate the weights of the prediction results, The weights of the sensor observations, This is an estimate of the uncertainty of sensor observations.

[0081] Compared with the prior art, the significant advantages of this invention are as follows:

[0082] 1. This invention uses an attitude-related attention mechanism and a heterogeneous data multimodal deep fusion model to perceive changes in the attitude of the UAV in real time and dynamically allocate weights. This effectively eliminates positioning drift caused by violent maneuvers, environmental occlusion, or visual blurring, solves the problem of single sensor failure in complex dynamic environments, and ensures that the positioning accuracy reaches the centimeter level in extreme scenarios such as jungles and ruins.

[0083] 2. This invention introduces uncertainty estimation and a self-confidence learning-based dynamic weighting strategy to construct a closed-loop system with self-perception and self-correction capabilities. It can judge the reliability of the multimodal deep fusion model in real time according to environmental changes and automatically switch the source of confidence according to the uncertainty. This effectively solves the problem that traditional algorithms cannot correct errors online and enables the system to have the ability to continuously adapt to new environments. Attached Figure Description

[0084] Figure 1 is a flowchart of the present invention;

[0085] Figure 2 shows the model architecture diagram;

[0086] Figure 3 shows the simulation scene.

[0087] Figure 4 is a schematic diagram of the search and rescue target;

[0088] Figure 5 shows a comparison of the simulation positioning accuracy. Detailed Implementation

[0089] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0090] The UAV target positioning system of this invention adopts a modular design, mainly consisting of a data perception module, a data preprocessing and synchronization module, and a fusion inference module.

[0091] The data sensing module includes an integrated GPS receiver, an IMU inertial measurement unit, and a vision sensor, used to collect geographic location information, attitude, acceleration information, and image data;

[0092] The data preprocessing and synchronization module performs noise reduction, timestamp alignment and coordinate system transformation on the data from each sensor to ensure that heterogeneous data can be compared and fused under a unified spatiotemporal reference.

[0093] The fusion inference module, based on a lightweight multimodal deep fusion model, achieves multimodal temporal feature learning and adaptive weight allocation. It uses dynamically compensated GPS location, IMU attitude information, and visual offset as inputs to establish nonlinear coupling relationships between modalities, outputting the target's geographic coordinates and uncertainty indicators. Internally, the fusion inference module deploys a multimodal deep fusion model (specifically HPTN). Its working logic is as follows: it receives preprocessed heterogeneous data and inputs it into the HPTN's feature extraction layer; HPTN uses a pose-related attention mechanism to weight the features; finally, based on the coordinates and uncertainty parameters output by HPTN, it performs dynamic error correction.

[0094] As shown in Figure 1, this invention provides a UAV target localization method based on heterogeneous data fusion and deep learning, comprising the following steps:

[0095] Step 1, Data Acquisition and Preprocessing;

[0096] The data sensing module collects geographic location information, attitude, acceleration information, and image data. To ensure the fusion of multi-source sensor data under a unified spatiotemporal reference, this invention designs a multi-stage preprocessing process for GPS, IMU, and visual information. The preprocessing process follows the logic of independent correction first, followed by spatiotemporal alignment. First, the UAV positioning system performs independent calibration for heterogeneous sensors in parallel, including: GPS dynamic compensation for the positioning module, pixel offset calculation for the vision module, and attitude angle acquisition and coordinate transformation for the inertial navigation module. Then, after completing the independent correction of each modality data, time synchronization and feature combination are performed to align the asynchronous multi-source data to the same timestamp and construct a multimodal input tensor.

[0097] Step 11, GPS coarse positioning and speed compensation;

[0098] Because GPS updates at a low frequency, position lag occurs during high-speed flight. Therefore, this invention dynamically compensates for GPS position using the linear velocity calculated by GPS and IMU, obtaining continuous position estimates.

[0099] (1)

[0100] in, The position coordinates in the global geographic coordinate system after compensation. To obtain and convert the location coordinate vectors to a global geographic coordinate system using a GPS receiver, Let be the linear velocity vector of the UAV in the same coordinate system. The average transmission and processing time delay of the GPS signal relative to the IMU time reference is pre-calibrated for the system.

[0101] This compensation method can effectively make up for the time domain delay caused by low-frequency GPS sampling, so that the location data can maintain temporal continuity under high-speed motion.

[0102] Step 12, Calculate camera pixel offset;

[0103] In the visual acquisition subunit, the camera determines the center position of the target in the image coordinate system through a target detection algorithm. Centered on the image For reference, the pixel offset is defined as:

[0104] (2)

[0105] in, This represents the pixel offset along the x-axis. This is the pixel offset along the y-axis.

[0106] Combined with the camera's intrinsic parameter matrix The direction vector of the target in the camera coordinate system can be calculated through back projection. :

[0107] (3)

[0108] in, Let T denote the camera intrinsic parameter matrix, and T denote the matrix transpose.

[0109] This direction vector Used to roughly calculate the physical space coordinates of the target, providing geometric position constraints for the system.

[0110] Step 13, Attitude angle acquisition and coordinate transformation;

[0111] The IMU outputs the UAV's attitude angles in real time, including roll, pitch, and yaw. To achieve alignment between the UAV coordinate system (Body Frame) and the geographic coordinate system (ENU or NED Frame), this invention uses an Euler angle rotation matrix for coordinate transformation:

[0112] (4)

[0113] in, This represents the total rotation matrix used to transform the UAV's body coordinate system to the geographic coordinate system;

[0114] , , These represent the yaw, pitch, and roll angles of the UAV, respectively, collected by the IMU.

[0115] Indicates rotation about the Z-axis The basic rotation matrix of an angle;

[0116] Indicates rotation about the Y-axis The basic rotation matrix of an angle;

[0117] Indicates rotation about the X-axis The basic rotation matrix of an angle.

[0118] Therefore, arbitrary sensor observation vector The matrix transformation can be performed using formula (4) to the geographic coordinate system: This step ensures that the modal data can be directly fused under the same spatial reference, avoiding the accumulation of errors caused by coordinate drift.

[0119] Step 14: Calculate the rough coordinates of the visual target;

[0120] Based on the pixel offset and camera intrinsic parameters obtained in step 12, and combined with the current flight altitude of the UAV, the rough geographic coordinates of the target are calculated using the Flat Earth Assumption model.

[0121] First, construct the line-of-sight vector in the camera coordinate system. .

[0122] Secondly, using the attitude matrix provided by the IMU in step 13 and camera external parameters Transform the line-of-sight vector to the geographic coordinate system:

[0123] (5)

[0124] in, This is the line-of-sight vector in the geographic coordinate system.

[0125] Let the position of the UAV in the geographic coordinate system be... Assuming the ground height is 0, the approximate coordinates of the target are... It can be obtained from the intersection of the ray and the ground plane:

[0126] (6)

[0127] in, X represents the vertical component of the line-of-sight vector; X, Y, and Z represent the three-dimensional position coordinates in the geographic coordinate system.

[0128] Will As prior spatial features, they are input into subsequent multimodal deep fusion models to help the network converge quickly.

[0129] Step 15: Data synchronization and feature combination;

[0130] To eliminate time shifts caused by differences in sampling frequencies among multiple sensors, this invention employs a timestamp alignment strategy for synchronization. Using IMU high-frequency data as a time reference, linear interpolation is performed on GPS and camera frame data to align all sensor data (GPS position, visual offset, IMU attitude) under a unified timestamp, generating multimodal joint samples. After the synchronized multimodal features are aligned, to enhance the model's perception of the environment and targets, this invention further combines the target's coarse coordinates and appearance color features to construct a unified multimodal input vector. :

[0131] (7)

[0132] in, This represents the approximate coordinates of the target detected by the camera; C is the color feature vector of the target region.

[0133] This multimodal unified feature vector It can be directly used as input to a multimodal deep fusion model, supporting subsequent nonlinear temporal modeling and multimodal feature fusion.

[0134] Step 2, Design and training of the multimodal deep fusion model;

[0135] To achieve high-precision positioning of personnel by UAVs in complex environments, this invention designs a deep learning-driven multimodal deep fusion model in the data fusion and algorithm inference sections. This multimodal deep fusion model can comprehensively process heterogeneous feature information from GPS, IMU, and visual sensors, improving positioning accuracy and robustness while ensuring real-time performance. The following section provides a detailed explanation of the model's design, training, and inference process, using a typical task scenario of UAVs identifying three trapped personnel in a jungle environment.

[0136] Step 21, Model Input Preparation;

[0137] Based on the multimodal input vector constructed in step 15 During the training phase, the input for the i-th sample is represented as:

[0138] (8)

[0139] Where the subscript i represents the sample number; This represents the RGB mean vector of pixels within the target bounding box, used to distinguish different measured objects.

[0140] The target's true geographical location Recorded as:

[0141] (9)

[0142] in, These represent the target's true horizontal coordinates and true vertical coordinates in the geographic coordinate system, respectively.

[0143] In jungle scenarios, complex lighting variations, vegetation obstruction, and GPS signal fluctuations can lead to temporal shifts and nonlinear errors in data across different modalities. This invention first standardizes each feature dimension:

[0144] (10)

[0145] (11)

[0146] in, and These represent the mean and standard deviation, respectively. It is the standardized input feature vector. It is the standardized truth value of the target.

[0147] The standardized data is then fed into a multimodal deep fusion model for training and inference.

[0148] Step 22, Overall network architecture design of the multimodal deep fusion model;

[0149] The network architecture of the multimodal deep fusion model in this embodiment consists of three main parts: a feature extractor, an attitude-aware attention block, and a regression and uncertainty estimation head, as shown in Figure 2.

[0150] The feature extraction module, through a multi-layer fully connected structure, achieves nonlinear mapping of high-dimensional heterogeneous inputs, uncovers the deep correlation between UAV flight status and visual features, and transforms the raw data into a high-dimensional feature representation, defined as:

[0151] (12)

[0152] (13)

[0153] in, , These are the weight matrix and bias vector of the first layer of the feature extraction network, respectively; This is the output feature vector of the first hidden layer; For the first The ReLU activation function of the layer. For the first The output feature vector of the hidden layer is designed with a layer structure of 256→512→512→256.

[0154] Numerical stability is maintained through batch normalization.

[0155] In a jungle environment, rapid changes in flight attitude can easily lead to visual feature distortion. Therefore, an attitude angle-based attention mechanism is introduced to adaptively assign weights to features of different modalities, enhancing the representation of reliable features in the current attitude while suppressing noise interference caused by drastic attitude fluctuations. The expression for the adaptive attention weights is as follows:

[0156] (14)

[0157] in, This represents the adaptive attention weights of the j-th modality in the current pose. It is a nonlinear mapping network containing two fully connected layers, used to map pose data to the same dimension as the feature vector to achieve feature space alignment; Let be the feature vector of the i-th sample and the j-th mode. The k-th modal eigenvector during the summation of the denominator. Let be the three-axis attitude angles of the UAV corresponding to the i-th sample: roll, pitch, and yaw.

[0158] A unified global feature is obtained through weighted fusion. express:

[0159] (15)

[0160] in, This is the global feature vector after attention-weighted fusion.

[0161] The coordinate regression and uncertainty estimation module outputs the final location data through multi-layer mapping. This module does not simply output coordinates; it simultaneously performs coordinate regression and uncertainty assessment: while predicting the target's geographical location, it outputs the uncertainty (variance) of the prediction results, providing a basis for subsequent adaptive loss weighting.

[0162] (16)

[0163] in, This represents the two-dimensional location coordinates of the target in the geographic coordinate system predicted by the model. For parameterized regression networks; This represents the uncertainty estimate of the model, used for subsequent adaptive correction.

[0164] Step 23, Loss Function and Optimization Strategy;

[0165] To balance positioning accuracy and robustness, this invention constructs an improved heteroscedasticity loss function. Loss Function Defined as an adaptive decay form:

[0166] (17)

[0167] (18)

[0168] in, Let be the total loss function during network training. The batch size during training. To smooth the L1 loss function, For uncertainty regularization coefficients, for Threshold parameter for the loss function (recommended value: 0.1). The first term acts as a precision weight. When the model predicts a high degree of uncertainty for the current sample, the regression loss weight for that sample is automatically reduced, thereby minimizing the interference of outliers on the model gradient; the second term... This is a regularization term to prevent the model from circumventing penalties by predicting infinitely large uncertainties.

[0169] loss function Mathematically, it is equivalent to an approximate maximum likelihood estimate of a mixture of Laplace and Gaussian noise, ensuring the convergence of the algorithm in complex environments such as jungles.

[0170] The optimizer uses the AdamW algorithm:

[0171] (19)

[0172] in, , These are the model parameters for step t and step t+1, respectively; For learning rate, This is the weight decay coefficient. , These are the first-order moment estimates and second-order moment estimates in the AdamW optimizer, respectively.

[0173] The learning rate is dynamically adjusted using the ReduceLROnPlateau strategy, automatically halving when the validation error fails to decrease for several consecutive rounds to accelerate convergence and avoid local optima. Furthermore, to reduce GPU memory usage and speed up convergence, a mixed-precision training mechanism is introduced. Forward propagation is performed using half-precision floating-point (FP16), combined with gradient scaling techniques to ensure numerical stability, thereby achieving efficient computation on the GPU.

[0174] Step 24, Training Process and Robustness Enhancement;

[0175] During the training phase, the data collected from the flight experiment was divided into a training set, a validation set, and a test set (in an 8:1:1 ratio). The attitude angle, pixel offset, and spatial coordinates of each sample were augmented using the following data augmentation strategies:

[0176] Attitude perturbation enhancement: ;

[0177] Pixel offset noise: ;

[0178] Coordinate compensation perturbation: ;

[0179] in, , , These represent the Gaussian noise variances superimposed on the pose angle, pixel offset, and spatial coordinates, respectively. These represent the three-axis attitude angles of the UAV collected by the IMU in the i-th sample, corresponding to the roll angle, pitch angle, and yaw angle, respectively. This represents the pixel offset of the target captured by the visual sensor relative to the image center in the image coordinate system in the i-th sample. Let represent the three-dimensional position coordinates of the UAV in the geographic coordinate system after GPS speed compensation in the i-th sample.

[0180] This mechanism can effectively simulate lighting flicker and attitude jitter during jungle flight, improving the model's generalization performance in complex environments.

[0181] Model training employs mini-batch iterations (Batch Size = 512), performing forward propagation, loss calculation, and backward gradient update in each round. A validation set is used to monitor the error profile. and The convergence trend. Among them, Indicates the first The loss function value of the model on the training dataset at the next iteration. This represents the loss function value of the model on the validation dataset during the same period. By comparing the changes in these two values, it can be determined whether the model is overfitting.

[0182] When the mean absolute error (MAE) of the validation set satisfies:

[0183] (20)

[0184] in, The total number of samples is denoted by m, where m is the unit of measurement in meters. This represents the two-dimensional position coordinate vector predicted by the model for the i-th sample. This represents the true geographic location coordinate vector of the i-th sample, indicating that the model has achieved centimeter-level positioning accuracy.

[0185] Step 25, Real-time Inference and Online Adaptive Correction;

[0186] During the flight mission execution phase, the UAV inputs standardized features in real time. The model outputs the predicted location. With uncertainty To address sudden visual obstruction or GPS signal drift, this invention designs a dynamic weighted correction mechanism:

[0187] (twenty one)

[0188] (twenty two)

[0189] in, This is the final corrected position estimate. For the model to predict the location, The estimation results are from a single sensor. For the weighting function based on uncertainty, This is the adjustment coefficient for the hyperparameter. This represents the uncertainty of the model prediction.

[0190] If the model uncertainty increases, it will automatically increase. This enables real-time adaptive correction.

[0191] Simulation results show that in a jungle environment, this mechanism can effectively suppress GPS drift and visual errors, and maintain stable system output.

[0192] Step 26, incremental model update and continuous learning;

[0193] When the drone collects new environmental data (such as different lighting or terrain conditions), the onboard processing system can perform incremental training. Let the parameters of the old model be... The new sample set is The goal of incremental training is:

[0194] (twenty three)

[0195] in, To optimize the objective, The loss function for the new task. The regularization coefficient; the second term To constrain knowledge and prevent catastrophic forgetting, a low learning rate is used. Fine-tuning the weights allows the model to adapt to new scene features while maintaining the original model's performance, enabling the model to self-evolve and continuously optimize.

[0196] Step 3, Model Reasoning and Application;

[0197] This invention employs an offline-trained multimodal deep fusion model for real-time inference during UAV flight, combined with online error correction and adaptive weighting mechanisms, to achieve centimeter-level high-precision positioning and stable control. The entire inference and application process includes five steps: input feature standardization, model forward inference, coordinate inverse normalization, online error correction, and uncertainty-driven dynamic weighting.

[0198] Step 31, Input features and standardization processing;

[0199] During actual flight, the UAV collects attitude angles, visual features, and spatial measurement data through multiple source sensors, and constructs a multimodal input vector consistent with that used in the training phase. :

[0200] (twenty four)

[0201] in,( The corresponding color feature C in the training phase, ( The vector is a coarse coordinate calculated in real time, containing full-dimensional information from pose, visual bias, physical location to prior and appearance.

[0202] To eliminate scale differences between different units, the original input Standardization transformation is required:

[0203] (25)

[0204] in, To normalize the prefitted normalizer during the training phase, linear normalization is performed on each dimension to make its mean 0 and variance 1.

[0205] Step 32, multimodal deep fusion model inference;

[0206] Standardized raw input The multimodal deep fusion model deployed in the flight control computing unit is fed into the system for real-time inference. Forward computation is performed without enabling gradients.

[0207] (26)

[0208] Among them, reasoning output This is the comprehensive prediction vector output by the model. For a trained multimodal deep fusion model, This represents a fixed weight parameter.

[0209] This multimodal deep fusion model employs a multi-branch attention fusion structure to model the dependency relationship between temporal and spatial features in a hierarchical manner. The temporal channel processes dynamic information from the IMU, capturing the impact of attitude changes on the position signal; the spatial channel fuses visual and GPS features, learning the nonlinear coupling between perception and measurement.

[0210] The model incorporates an Attitude-Aware Attention Block, which, at each time step, adjusts the attention based on the attitude angle (…). Dynamically adjust the attention weights for each modality:

[0211] (27)

[0212] Where A is the attention matrix, Represents the normalized exponential function, , These are the learnable weight matrices; , , These are feature data from a visual sensor, an inertial measurement unit (IMU), and GPS, respectively.

[0213] Through posture-related attention matrix The multimodal deep fusion model can adaptively suppress feature shifts caused by attitude disturbances such as tilting and rotation, thereby maintaining fusion stability under complex flight attitudes.

[0214] Ultimately, the model's comprehensive prediction vector It is decomposed into two parts: location coordinates and uncertainty. Specifically, the output layer adds an uncertainty estimation head to simultaneously predict the uncertainty of the positioning result, i.e.:

[0215] (28)

[0216] in, For the comprehensive prediction vector The specific unfolding form; These are the normalized predicted position coordinates, used for subsequent adaptive weighting and error correction; This is a function for deep fusion neural networks.

[0217] Step 33: Output the inverse normalization and coordinate restoration;

[0218] To facilitate direct use by the drone control module, the network output needs to be denormalized to restore it to the true physical coordinate system: (29)

[0219] in, For the inverse normalization function, This refers to the predicted target position of the UAV in the geographic coordinate system, which can be directly input into the flight controller for path planning and closed-loop navigation control, achieving centimeter-level positioning accuracy.

[0220] Step 34, online error correction and dynamic fusion mechanism;

[0221] During flight, the UAV's positioning system continuously receives real-time observation coordinates from sensors. To prevent error accumulation caused by model drift or external interference, this invention designs an online error correction mechanism, which calculates the Euclidean distance between the prediction result and the sensor result:

[0222] (30)

[0223] If error If the threshold is reached, then dynamic fusion correction will be performed.

[0224] (31)

[0225] in, The final target location is obtained after weighting. These are adaptive weighting coefficients, with a range of values. This weighting method enables a smooth transition between prediction results and sensor observations, effectively suppressing noise from a single sensor and improving positioning stability.

[0226] In typical implementations, the weight parameters use a fixed coefficient fusion method, for example: This strategy can maintain the continuity and safety of the output even when there is visual drift or high radar noise.

[0227] Step 35, Confident Learning-Based Adaptive Weighted Mechanism;

[0228] To further enhance the system's adaptive capability, this invention proposes a self-learning dynamic weighting mechanism based on the uncertainty output of a multimodal deep fusion model. To achieve real-time weight adjustment, define the uncertainty normalization function:

[0229] (32)

[0230] (33)

[0231] in, To integrate the weights of the prediction results, The weights of the sensor observations, This is an estimate of the uncertainty of the sensor observation. Its value is related to the hardware parameters of the sensor used. In this embodiment, it is preset to a fixed value based on the variance of the sensor measurement noise.

[0232] The final correction output of the entire positioning system for:

[0233] (34)

[0234] When the model uncertainty is low ( When the uncertainty of the model increases (e.g., in scenarios with changes in lighting, occlusion, or weakened GPS signals), the system will increase the weight of sensor measurements, realizing the dynamic adjustment principle of trusting the model when the uncertainty is low and trusting the sensors when the uncertainty is high.

[0235] Through this mechanism, the UAV positioning system utilizes formulas (32) to (34) to form a closed-loop feedback structure of fusion inference—uncertainty feedback—weight redistribution. Specifically, it utilizes the uncertainty output by the model in real time. Dynamically adjust fusion weights This enables drones to have self-sensing and self-correcting capabilities, significantly enhancing their robustness and stability in complex environments.

[0236] To verify the effectiveness and real-time performance of the multimodal deep fusion model of this invention, a drone simulation platform based on ROS (Robot Operating System) and Gazebo was constructed. The platform integrates multi-source sensing components such as visual sensors, inertial measurement units (IMUs), and GPS modules, and implements data synchronization and timestamp alignment mechanisms to ensure the fusion of multi-sensor information in a unified coordinate system. For a fair performance evaluation, the traditional GPS + visual positioning method was selected as a comparison algorithm, and the positioning accuracy and robustness performance under different environments were compared and analyzed.

[0237] The simulation scenario is based on a "dual-drone collaborative autonomous search and rescue" mission, as shown in Figure 3. The drone's flight speed ranges from 12 to 17 m / s, and the environment is a complex mixed scene of jungle and residential areas, including various lighting changes, occlusion, and attitude perturbation conditions. Three typical search and rescue targets are set in the scenario: critically injured persons (red), slightly injured persons (yellow), and healthy persons (camouflage white), as shown in Figure 4, corresponding to the visual features of different search and rescue states, used to test the adaptability and robustness of the algorithm in multi-target recognition and localization tasks.

[0238] To quantitatively evaluate system performance, the following metrics were selected: Position Error, which is the Euclidean distance between the predicted coordinates and the true target position; and Detection Error, which is the deviation between the pixel coordinates output by the vision detection module and the ground truth projection.

[0239] Figure 4 shows the error statistics for targets of different colors. As can be seen from Figure 5, traditional visual detection is prone to detection drift under complex textures or reflective surfaces, while the multimodal deep fusion model of this invention can maintain stable prediction output under pose perturbation and illumination changes. Specific numerical results are as follows:

[0240] For the camouflage_white target, the detection error is relatively large (about 0.29 m), but the multimodal deep fusion model still achieves a localization error of 0.20 m, indicating that the present invention can effectively correct visual errors under low contrast targets;

[0241] For the red target, the detection error is 0.29 m, while the localization error after fusion is reduced to 0.08 m, which demonstrates the high accuracy performance of the multimodal deep fusion model under low uncertainty visual feature conditions;

[0242] For yellow targets, visual detection fluctuates due to light reflection, resulting in a detection error of approximately 0.32 m. However, the localization error output by the multimodal deep fusion model is only 0.25 m, maintaining good stability.

[0243] Overall, the multimodal deep fusion model significantly reduced positioning errors in all three color scenarios, with an average error reduction of approximately 65–80%. In summary, the simulation results fully demonstrate that the present invention maintains high accuracy and strong robustness under different color and lighting conditions. Its attitude attention mechanism effectively reduces the impact of flight attitude disturbances on visual features; while the uncertainty estimation module dynamically adjusts the fusion weights, enabling the UAV positioning system to maintain continuous centimeter-level positioning output even in complex environments. These results verify the feasibility and practical value of the method of the present invention in real-time navigation and target tracking tasks for UAVs.

[0244] The multimodal deep fusion model proposed in this invention has advantages such as simple structure, high computational efficiency, and strong real-time performance. It can run efficiently on embedded GPU platforms or flight control main control chips, meeting the navigation and control requirements of UAVs in high-speed dynamic environments. Due to the small number of model parameters and low hardware dependence, the UAV positioning system can be directly embedded into the existing flight control system to achieve end-to-end intelligent positioning and autonomous decision-making.

[0245] By continuously collecting real-world flight data from UAVs, this invention can further perform incremental learning and online retraining, continuously optimizing model weights and feature distributions, thereby improving generalization ability and adaptive performance under complex climate, terrain, and lighting conditions. This incremental learning and online retraining mechanism enables the system to continuously evolve, providing technical support for long-term intelligent flight missions.

[0246] This invention's method is not only applicable to unmanned aerial vehicles (UAVs) for tasks such as inspection, surveying, emergency rescue, and precise material delivery, but can also be extended to various autonomous platforms such as unmanned vehicles, unmanned surface vessels, and mobile robots. Cross-platform migration and reuse can be achieved simply by changing the sensor adapter and coordinate mapping modules, greatly reducing system development and deployment costs. With its high precision, strong robustness, and universal architecture, this invention has broad application prospects and industrial promotion potential in intelligent transportation, sea-land-air collaborative sensing, and the intelligent equipment industry.

Claims

1. A method for UAV target localization based on heterogeneous data fusion and deep learning, characterized in that, The steps include: S1, the data perception module collects the UAV's geographical location information, attitude, acceleration information, and image data, and preprocesses the collected data to construct a multimodal input vector; S2, the multimodal deep fusion model is trained using the multimodal input vector; S3, during flight, the offline-trained multimodal deep fusion model is used for real-time inference, and combined with online error correction and adaptive weighting mechanisms, centimeter-level high-precision positioning and stable control are achieved; the multimodal deep fusion model includes: a feature extraction module, an attitude-related attention module, and a coordinate regression and uncertainty estimation module; the feature extraction module uses a multi-layer fully connected structure to achieve nonlinear mapping of high-dimensional heterogeneous inputs, mining the deep correlation between the UAV's flight state and visual features, and transforming the original data into a high-dimensional feature representation, defined as: in, 、 These are the weight matrix and bias vector of the first layer of the feature extraction network, respectively; This is the output feature vector of the first hidden layer; For the first The ReLU activation function of the layer. For the first The hidden layer outputs feature vectors; the pose-related attention module introduces a pose angle-based attention mechanism to assign adaptive attention weights to features of different modalities; the expression for the adaptive attention weights is as follows: ,in, This represents the adaptive attention weight of the j-th modality in the current pose; It is a nonlinear mapping network containing two fully connected layers, used to map pose data to the same dimension as the feature vector to achieve feature space alignment; Let be the feature vector of the i-th sample and the j-th mode. The k-th modal eigenvector during summation of the denominator; Let the roll angle, pitch angle, and yaw angle of the UAV corresponding to the i-th sample be denoted; a unified global feature is obtained through weighted fusion. express: ,in, This is the global feature vector after attention-weighted fusion; the coordinate regression and uncertainty estimation module, while predicting the target's geographical location, outputs the uncertainty of the prediction result, as expressed below: ,in, This represents the two-dimensional location coordinates of the target in the geographic coordinate system predicted by the model. For parameterized regression networks, This represents the uncertainty estimate of the model.

2. The UAV target localization method based on heterogeneous data fusion and deep learning according to claim 1, characterized in that, Preprocessing of the collected data includes: A11, coarse GPS positioning and velocity compensation; dynamic compensation of GPS position using linear velocity calculated from GPS and IMU to obtain continuous position estimates. ,in, The position coordinates in the global geographic coordinate system after compensation. To obtain and convert the location coordinate vectors to a global geographic coordinate system using a GPS receiver, Let be the linear velocity vector of the UAV in the same coordinate system. The system pre-calibrates the average transmission and processing time delay of the GPS signal relative to the IMU time reference; A12, camera pixel offset calculation; the camera determines the center position of the target in the image coordinate system using a target detection algorithm. With the image center For reference, the pixel offset is defined as: ,in, This represents the pixel offset along the x-axis. This represents the pixel offset along the y-axis; combined with the camera's intrinsic parameter matrix. The direction vector of the target in the camera coordinate system is calculated by back projection. ： ,in, Represents the camera intrinsic parameter matrix, T represents matrix transpose; A13, attitude angle acquisition and coordinate transformation; to achieve alignment between the UAV coordinate system and the geographic coordinate system, an Euler angle rotation matrix is ​​used for coordinate transformation: ,in, This represents the total rotation matrix used to transform the UAV's body coordinate system to the geographic coordinate system; 、 、 These represent the yaw, pitch, and roll angles of the UAV, respectively, collected by the IMU. Indicates rotation about the Z-axis The basic rotation matrix of an angle; Indicates rotation about the Y-axis The basic rotation matrix of an angle; Indicates rotation about the X-axis The basic rotation matrix of the angle; then, using the attitude matrix provided by the IMU. and camera external parameters Transform the line-of-sight vector to the geographic coordinate system: ,in, Let be the line-of-sight vector in the geographic coordinate system; let the position of the UAV in the geographic coordinate system be . Assuming the ground height is 0, the approximate coordinates of the target are... The answer can be obtained by finding the intersection of the ray and the ground plane. ,in, X represents the vertical component of the line-of-sight vector; X, Y, and Z represent the three-dimensional position coordinates in the geographic coordinate system. This represents the approximate coordinates of the target detected by the camera; subsequently, time synchronization and feature combination are performed to align asynchronous multi-source data to the same timestamp and construct a multimodal input tensor.

3. The UAV target localization method based on heterogeneous data fusion and deep learning according to claim 2, characterized in that, The process of constructing the multimodal input tensor is as follows: using IMU high-frequency data as the time reference, linear interpolation is performed on GPS and camera frame data to align GPS position, visual offset, and IMU pose under a unified timestamp, thereby generating multimodal joint samples; The final constructed multimodal input vector includes: UAV attitude angle, camera pixel offset, compensated geographic location coordinates, target coarse coordinates, and target color features. The multimodal input vector serves as the input to the multimodal deep fusion model.

4. The UAV target localization method based on heterogeneous data fusion and deep learning according to claim 1, characterized in that, An improved heteroscedasticity loss function is used during training, expressed as follows: ， ,in, Let be the total loss function during network training. The batch size during training. To smooth the L1 loss function, For uncertainty regularization coefficients, for The threshold parameter of the loss function; The term acts as a precision weight; when the model predicts that the uncertainty of the current sample is high, the regression loss weight of that sample is automatically reduced. For regularization terms; This represents the two-dimensional position coordinate vector predicted by the model for the i-th sample. This represents the true geographical location coordinate vector of the i-th sample.

5. The UAV target localization method based on heterogeneous data fusion and deep learning according to claim 1, characterized in that, During training, a dynamic weighted correction mechanism is used for online adaptive correction. The expression for the dynamic weighted correction mechanism is as follows: ， ,in, This is the final corrected position estimate. For the model to predict the location, The estimation results are from a single sensor. For the weighting function based on uncertainty, This is the adjustment coefficient for the hyperparameter. This represents the uncertainty of the model prediction; if the model uncertainty increases, it will be automatically increased. This enables real-time adaptive correction.

6. The UAV target localization method based on heterogeneous data fusion and deep learning according to claim 1, characterized in that, The steps to achieve centimeter-level high-precision positioning and stable control include: S31, standardizing the input features; S32, processing the standardized raw input... The multimodal deep fusion model deployed in the flight control computing unit is fed into the system for real-time inference; forward computation is performed without enabling gradients. Among them, inference output This is the comprehensive prediction vector output by the model. For a trained multimodal deep fusion model, Represents fixed weight parameters; comprehensive prediction vector Including position coordinates and uncertainty The multimodal deep fusion model incorporates a posture attention module, which dynamically adjusts the attention weights of each modality based on the posture angle at each time step; S33, the model output is denormalized and coordinates are restored to the real physical coordinate system. ,in, For the inverse normalization function, S34: Predict the target location of the UAV in the geographic coordinate system; S34: Perform online error correction, if the error... Then, dynamic fusion correction will be performed: ,in, The final target location is obtained after weighting. These are adaptive weighting coefficients, with a range of values. ； ,in, To calculate the Euclidean distance between the predicted results and the sensor results, For threshold; Real-time observation coordinates of the sensor; S35, uncertainty based on the output of the multimodal deep fusion model. Real-time weight adjustment: ， The entire positioning system eventually corrects its output. for: ,in, To integrate the weights of the prediction results, The weights of the sensor observations, This is an estimate of the uncertainty of sensor observations.

Images