Unmanned aerial vehicle target positioning method based on heterogeneous data fusion and deep learning

Abstract

The invention discloses an unmanned aerial vehicle target positioning method based on heterogeneous data fusion and deep learning, and the method comprises the following steps: S1, collecting the geographic position information, attitude, acceleration information and image data of an unmanned aerial vehicle through a data sensing module, carrying out the preprocessing of the collected data, and constructing a multi-modal input tensor; s2, training the constructed multi-modal deep fusion model; and S3, in the flight process, performing real-time reasoning by adopting an offline trained multi-modal deep fusion model, and realizing centimeter-level high-precision positioning and stable control in combination with an online error correction and self-adaptive weighting mechanism. According to the method, the model reliability can be judged in real time according to the environment change, the trust source is automatically switched according to the uncertainty, and the problem that a traditional algorithm cannot correct errors online is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control and target localization technology for unmanned aerial vehicles (UAVs), and in particular to a UAV target localization method based on heterogeneous data fusion and deep learning. Background Technology

[0002] With the widespread application of drones in reconnaissance and surveillance, counter-terrorism and bomb disposal, and emergency search and rescue missions, high-precision positioning and target identification in complex terrain have become crucial technologies. Taking disaster scenarios as an example, collapsed buildings, narrow streets, or tree obstructions can all reduce the accuracy of traditional positioning methods, impacting the operational safety and mission efficiency of drones. Compared to ground-based search and rescue forces, drones can quickly acquire a global perspective from the air and perform low-altitude searches in areas at risk of collapse, significantly improving search and rescue efficiency and reducing safety risks.

[0003] Currently, common UAV positioning methods are mainly based on a combination of Global Positioning System (GPS), Inertial Measurement Unit (IMU), and visual sensors. However, existing methods have the following shortcomings: GPS suffers from low position update rate and unstable accuracy in environments with building obstruction, multipath interference, or weak satellite signals; while IMU can provide high-frequency attitude and acceleration information, its drift error accumulates over time; visual measurement is prone to unstable recognition and feature drift in strong light, shadow, or low-texture scenes; traditional fusion algorithms often use fixed weights or linear filtering models, which cannot adapt to changes in sensor uncertainty, leading to increased positioning errors in dynamic environments.

[0004] Especially in low-altitude search and rescue and close-range inspection missions, UAVs need to hover, turn or descend rapidly frequently. Their attitude changes and wind disturbances cause nonlinear deviations in the sensor sampling timing, which further affects the positioning accuracy. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a UAV target localization method based on heterogeneous data fusion and deep learning. This method can combine GPS, IMU and visual sensor data, and improve the positioning accuracy and real-time performance in complex lighting, occlusion and attitude disturbance environments by dynamically adjusting the fusion weights and time-series compensation mechanism. This enhances the environmental adaptability and reliability of UAVs in search and rescue, inspection and other tasks.

[0006] Technical Solution: A UAV target localization method based on heterogeneous data fusion and deep learning, comprising the following steps:

[0007] S1, the data perception module collects the UAV's geographical location information, attitude, acceleration information and image data, and preprocesses the collected data to construct a multimodal input vector; S2 uses multimodal input vectors to train the constructed multimodal deep fusion model; During flight, S3 employs an offline-trained multimodal deep fusion model for real-time inference, combined with online error correction and adaptive weighting mechanisms, to achieve centimeter-level high-precision positioning and stable control.

[0008] Furthermore, the preprocessing of the collected data includes: A11, GPS coarse positioning and speed compensation; The GPS position is dynamically compensated by the linear velocity calculated from GPS and IMU to obtain continuous position estimates: , in, The position coordinates in the global geographic coordinate system after compensation. To obtain and convert the location coordinate vectors to a global geographic coordinate system using a GPS receiver, Let be the linear velocity vector of the UAV in the same coordinate system. The average transmission and processing time delay of the GPS signal relative to the IMU time reference, which is pre-calibrated by the system. A12, camera pixel offset calculation; The camera determines the center position of the target in the image coordinate system using a target detection algorithm. With the image center For reference, the pixel offset is defined as: , in, This represents the pixel offset along the x-axis. This is the pixel offset along the y-axis; Combined with the camera's intrinsic parameter matrix The direction vector of the target in the camera coordinate system is calculated by back projection. : , in, Let T denote the camera intrinsic parameter matrix; A13, Attitude Angle Acquisition and Coordinate Transformation; To align the UAV coordinate system with the geographic coordinate system, an Euler angle rotation matrix is ​​used for coordinate transformation: , in, This represents the total rotation matrix used to transform the UAV body coordinate system to the geographic coordinate system; , , These represent the yaw, pitch, and roll angles of the UAV, respectively, collected by the IMU. Indicates rotation about the Z-axis The basic rotation matrix of an angle; Indicates rotation about the Y-axis The basic rotation matrix of an angle; Indicates rotation about the X-axis The basic rotation matrix of an angle; Next, the attitude matrix provided by the IMU is used. and camera external parameters Transform the line-of-sight vector to the geographic coordinate system: , in, The line-of-sight vector in the geographic coordinate system; Let the position of the UAV in the geographic coordinate system be... Assuming the ground height is 0, the approximate coordinates of the target are... The answer can be obtained by finding the intersection of the ray and the ground plane. , in, X represents the vertical component of the line-of-sight vector; X, Y, and Z represent the three-dimensional position coordinates in the geographic coordinate system. This represents the approximate coordinates of the target detected by the camera. Subsequently, time synchronization and feature combination are performed to align asynchronous multi-source data to the same timestamp and construct a multimodal input tensor.

[0009] Furthermore, the process of constructing the multimodal input tensor is as follows: using IMU high-frequency data as the time reference, linear interpolation is performed on GPS and camera frame data to align GPS position, visual offset, and IMU attitude under a unified timestamp, generating multimodal joint samples; the final constructed multimodal input vector includes: UAV attitude angle, camera pixel offset, compensated geographic location coordinates, target coarse coordinates, and target color features, which serve as the input to the multimodal deep fusion model.

[0010] Furthermore, the multimodal deep fusion model includes: a feature extraction module, a pose-related attention module, and a coordinate regression and uncertainty estimation module; The feature extraction module, through a multi-layer fully connected structure, achieves nonlinear mapping of high-dimensional heterogeneous inputs, uncovers the deep correlation between UAV flight status and visual features, and transforms the raw data into a high-dimensional feature representation, defined as: , , in, , These are the weight matrix and bias vector of the first layer of the feature extraction network, respectively; This is the output feature vector of the first hidden layer; For the first The ReLU activation function of the layer. For the first The output feature vector of the hidden layer; The pose-related attention module introduces a pose angle-based attention mechanism to assign adaptive attention weights to features of different modalities; the expression for the adaptive attention weights is as follows: , in, This represents the adaptive attention weight of the j-th modality in the current pose; It is a nonlinear mapping network containing two fully connected layers, used to map pose data to the same dimension as the feature vector to achieve feature space alignment; Let be the feature vector of the i-th sample and the j-th mode. The k-th modal eigenvector during summation of the denominator; Let be the roll angle, pitch angle, and yaw angle of the UAV corresponding to the i-th sample; A unified global feature is obtained through weighted fusion. express: , in, This is the global feature vector after attention-weighted fusion; The coordinate regression and uncertainty estimation module, while predicting the target's geographical location, outputs the uncertainty of the prediction result, as shown in the following expression: , in, This represents the two-dimensional location coordinates of the target in the geographic coordinate system predicted by the model. For parameterized regression networks, This represents the uncertainty estimate of the model.

[0011] Furthermore, an improved heteroscedasticity loss function is used during training, expressed as follows: , , in, Let be the total loss function during network training. The batch size during training. To smooth the L1 loss function, For uncertainty regularization coefficients, for The threshold parameter of the loss function; The term acts as a precision weight; when the model predicts that the uncertainty of the current sample is high, the regression loss weight of that sample is automatically reduced. For regularization terms; This represents the two-dimensional position coordinate vector predicted by the model for the i-th sample. This represents the true geographical location coordinate vector of the i-th sample.

[0012] Furthermore, during training, a dynamic weighted correction mechanism is used for online adaptive correction. The expression for the dynamic weighted correction mechanism is as follows: , , in, This is the final corrected position estimate. For the model to predict the location, The estimation results are from a single sensor. For the weighting function based on uncertainty, This is the adjustment coefficient for the hyperparameter. The uncertainty of the model prediction; If the model uncertainty increases, it will automatically increase. This enables real-time adaptive correction.

[0013] Furthermore, the steps to achieve centimeter-level high-precision positioning and stable control include: S31, Standardize the input features; S32, the standardized raw input The multimodal deep fusion model deployed in the flight control computing unit is fed into the system for real-time inference; forward computation is performed without enabling gradients. , Among them, reasoning output This is the comprehensive prediction vector output by the model. For a trained multimodal deep fusion model, Represents fixed weight parameters; comprehensive prediction vector Including position coordinates and uncertainty ; The multimodal deep fusion model incorporates a posture attention module, which dynamically adjusts the attention weights of each modality based on the posture angle at each moment. S33 performs inverse normalization and coordinate restoration on the model output, recovering it to the true physical coordinate system: , in, For the inverse normalization function, Predicted target location for UAV in geographic coordinate system; S34, perform online error correction, if the error Then, dynamic fusion correction will be performed: , in, The final target location is obtained after weighting. These are adaptive weighting coefficients, with a range of values. ; , in, To calculate the Euclidean distance between the predicted results and the sensor results, For threshold; Real-time observed coordinates of the sensor; S35, Uncertainty based on the output of the multimodal deep fusion model Real-time weight adjustment: , , The final correction output of the entire positioning system for: , in, To integrate the weights of the prediction results, The weights of the sensor observations, This is an estimate of the uncertainty of sensor observations.

[0014] Compared with the prior art, the significant advantages of this invention are as follows: 1. This invention uses an attitude-related attention mechanism and a heterogeneous data multimodal deep fusion model to perceive changes in the attitude of the UAV in real time and dynamically allocate weights. This effectively eliminates positioning drift caused by violent maneuvers, environmental occlusion, or visual blurring, solves the problem of single sensor failure in complex dynamic environments, and ensures that the positioning accuracy reaches the centimeter level in extreme scenarios such as jungles and ruins. 2. This invention introduces uncertainty estimation and a self-confidence learning-based dynamic weighting strategy to construct a closed-loop system with self-perception and self-correction capabilities. It can judge the reliability of the multimodal deep fusion model in real time according to environmental changes and automatically switch the source of confidence according to the uncertainty. This effectively solves the problem that traditional algorithms cannot correct errors online and enables the system to have the ability to continuously adapt to new environments. Attached Figure Description

[0015] Figure 1 This is a flowchart of the present invention; Figure 2 This is a model architecture diagram; Figure 3 This is a simulation scene diagram; Figure 4 Diagram showing the search and rescue target; Figure 5 This is a comparison chart of simulated positioning accuracy. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0017] The UAV target positioning system of this invention adopts a modular design, mainly consisting of a data perception module, a data preprocessing and synchronization module, and a fusion inference module. The data sensing module includes an integrated GPS receiver, an IMU inertial measurement unit, and a vision sensor, used to collect geographic location information, attitude, acceleration information, and image data; The data preprocessing and synchronization module performs noise reduction, timestamp alignment and coordinate system transformation on the data from each sensor to ensure that heterogeneous data can be compared and fused under a unified spatiotemporal reference. The fusion inference module, based on a lightweight multimodal deep fusion model, achieves multimodal temporal feature learning and adaptive weight allocation. It uses dynamically compensated GPS location, IMU attitude information, and visual offset as inputs to establish nonlinear coupling relationships between modalities, outputting the target's geographic coordinates and uncertainty indicators. Internally, the fusion inference module deploys a multimodal deep fusion model (specifically HPTN). Its working logic is as follows: it receives preprocessed heterogeneous data and inputs it into the HPTN's feature extraction layer; HPTN uses a pose-related attention mechanism to weight the features; finally, based on the coordinates and uncertainty parameters output by HPTN, it performs dynamic error correction.

[0018] like Figure 1 As shown, this invention provides a UAV target localization method based on heterogeneous data fusion and deep learning, comprising the following steps: Step 1, Data Acquisition and Preprocessing; The data sensing module collects geographic location information, attitude, acceleration information, and image data. To ensure the fusion of multi-source sensor data under a unified spatiotemporal reference, this invention designs a multi-stage preprocessing process for GPS, IMU, and visual information. The preprocessing process follows the logic of independent correction first, followed by spatiotemporal alignment. First, the UAV positioning system performs independent calibration for heterogeneous sensors in parallel, including: GPS dynamic compensation for the positioning module, pixel offset calculation for the vision module, and attitude angle acquisition and coordinate transformation for the inertial navigation module. Then, after completing the independent correction of each modality data, time synchronization and feature combination are performed to align the asynchronous multi-source data to the same timestamp and construct a multimodal input tensor.

[0019] Step 11, GPS coarse positioning and speed compensation; Because GPS updates at a low frequency, position lag occurs during high-speed flight. Therefore, this invention dynamically compensates for GPS position using the linear velocity calculated by GPS and IMU, obtaining continuous position estimates. (1) in, The position coordinates in the global geographic coordinate system after compensation. To obtain and convert the location coordinate vectors to a global geographic coordinate system using a GPS receiver, Let be the linear velocity vector of the UAV in the same coordinate system. The average transmission and processing time delay of the GPS signal relative to the IMU time reference is pre-calibrated for the system.

[0020] This compensation method can effectively make up for the time domain delay caused by low-frequency GPS sampling, so that the location data can maintain temporal continuity under high-speed motion.

[0021] Step 12, Calculate camera pixel offset; In the visual acquisition subunit, the camera determines the center position of the target in the image coordinate system through a target detection algorithm. Centered on the image For reference, the pixel offset is defined as: (2) in, This represents the pixel offset along the x-axis. This is the pixel offset along the y-axis.

[0022] Combined with the camera's intrinsic parameter matrix The direction vector of the target in the camera coordinate system can be calculated through back projection. : (3) in, Let T denote the camera intrinsic parameter matrix, and T denote the matrix transpose.

[0023] This direction vector Used to roughly calculate the physical space coordinates of the target, providing geometric position constraints for the system.

[0024] Step 13, Attitude angle acquisition and coordinate transformation; The IMU outputs the UAV's attitude angles in real time, including roll, pitch, and yaw. To achieve alignment between the UAV coordinate system (Body Frame) and the geographic coordinate system (ENU or NED Frame), this invention uses an Euler angle rotation matrix for coordinate transformation: (4) in, This represents the total rotation matrix used to transform the UAV body coordinate system to the geographic coordinate system; , , These represent the yaw, pitch, and roll angles of the UAV, respectively, collected by the IMU. Indicates rotation about the Z-axis The basic rotation matrix of an angle; Indicates rotation about the Y-axis The basic rotation matrix of an angle; Indicates rotation about the X-axis The basic rotation matrix of an angle.

[0025] Therefore, arbitrary sensor observation vector The matrix transformation can be performed using formula (4) to the geographic coordinate system: This step ensures that the modal data can be directly fused under the same spatial reference, avoiding the accumulation of errors caused by coordinate drift.

[0026] Step 14: Calculate the rough coordinates of the visual target; Based on the pixel offset and camera intrinsic parameters obtained in step 12, and combined with the current flight altitude of the UAV, the rough geographic coordinates of the target are calculated using the Flat Earth Assumption model.

[0027] First, construct the line-of-sight vector in the camera coordinate system. .

[0028] Secondly, using the attitude matrix provided by the IMU in step 13 and camera external parameters Transform the line-of-sight vector to the geographic coordinate system: (5) in, This is the line-of-sight vector in the geographic coordinate system.

[0029] Let the position of the UAV in the geographic coordinate system be... Assuming the ground height is 0, the approximate coordinates of the target are... It can be obtained from the intersection of the ray and the ground plane: (6) in, X represents the vertical component of the line-of-sight vector; X, Y, and Z represent the three-dimensional position coordinates in the geographic coordinate system.

[0030] Will As prior spatial features, they are input into subsequent multimodal deep fusion models to help the network converge quickly.

[0031] Step 15: Data synchronization and feature combination; To eliminate time shifts caused by differences in sampling frequencies among multiple sensors, this invention employs a timestamp alignment strategy for synchronization. Using IMU high-frequency data as a time reference, linear interpolation is performed on GPS and camera frame data to align all sensor data (GPS position, visual offset, IMU attitude) under a unified timestamp, generating multimodal joint samples. After the synchronized multimodal features are aligned, to enhance the model's perception of the environment and targets, this invention further combines the target's coarse coordinates and appearance color features to construct a unified multimodal input vector. : (7) in, This represents the approximate coordinates of the target detected by the camera; C is the color feature vector of the target region. This multimodal unified feature vector It can be directly used as input to a multimodal deep fusion model, supporting subsequent nonlinear temporal modeling and multimodal feature fusion.

[0032] Step 2, Design and training of the multimodal deep fusion model; To achieve high-precision positioning of personnel by UAVs in complex environments, this invention designs a deep learning-driven multimodal deep fusion model in the data fusion and algorithm inference sections. This multimodal deep fusion model can comprehensively process heterogeneous feature information from GPS, IMU, and visual sensors, improving positioning accuracy and robustness while ensuring real-time performance. The following section provides a detailed explanation of the model's design, training, and inference process, using a typical task scenario of UAVs identifying three trapped personnel in a jungle environment.

[0033] Step 21, Model Input Preparation; Based on the multimodal input vector constructed in step 15 During the training phase, the input for the i-th sample is represented as: (8) Where the subscript i represents the sample number; This represents the RGB mean vector of pixels within the target bounding box, used to distinguish different measured objects.

[0034] The target's true geographical location Recorded as: (9) in, These represent the target's true horizontal coordinates and true vertical coordinates in the geographic coordinate system, respectively.

[0035] In jungle scenarios, complex lighting variations, vegetation obstruction, and GPS signal fluctuations can lead to temporal shifts and nonlinear errors in data across different modalities. This invention first standardizes each feature dimension: (10) (11) in, and These represent the mean and standard deviation, respectively. It is the standardized input feature vector. It is the standardized truth value of the target.

[0036] The standardized data is then fed into a multimodal deep fusion model for training and inference.

[0037] Step 22, Overall network architecture design of the multimodal deep fusion model; The network architecture of the multimodal deep fusion model in this embodiment consists of three main parts: a feature extraction module, an attitude-aware attention block, and a regression and uncertainty estimation head. See details in [link to documentation]. Figure 2 .

[0038] The feature extraction module, through a multi-layer fully connected structure, achieves nonlinear mapping of high-dimensional heterogeneous inputs, uncovers the deep correlation between UAV flight status and visual features, and transforms the raw data into a high-dimensional feature representation, defined as: (12) (13) in, , These are the weight matrix and bias vector of the first layer of the feature extraction network, respectively; This is the output feature vector of the first hidden layer; For the first The ReLU activation function of the layer. For the first The output feature vector of the hidden layer is designed with a layer structure of 256→512→512→256.

[0039] Numerical stability is maintained through batch normalization.

[0040] In a jungle environment, rapid changes in flight attitude can easily lead to visual feature distortion. Therefore, an attitude angle-based attention mechanism is introduced to adaptively assign weights to features of different modalities, enhancing the representation of reliable features in the current attitude while suppressing noise interference caused by drastic attitude fluctuations. The expression for the adaptive attention weights is as follows: (14) in, This represents the adaptive attention weights of the j-th modality in the current pose. It is a nonlinear mapping network containing two fully connected layers, used to map pose data to the same dimension as the feature vector to achieve feature space alignment; Let be the feature vector of the i-th sample and the j-th mode. The k-th modal eigenvector during the summation of the denominator. Let be the three-axis attitude angles of the UAV corresponding to the i-th sample: roll, pitch, and yaw.

[0041] A unified global feature is obtained through weighted fusion. express: (15) in, This is the global feature vector after attention-weighted fusion.

[0042] The coordinate regression and uncertainty estimation module outputs the final location data through multi-layer mapping. This module does not simply output coordinates; it simultaneously performs coordinate regression and uncertainty assessment: while predicting the target's geographical location, it outputs the uncertainty (variance) of the prediction results, providing a basis for subsequent adaptive loss weighting. (16) in, This represents the two-dimensional location coordinates of the target in the geographic coordinate system predicted by the model. For parameterized regression networks; This represents the uncertainty estimate of the model, used for subsequent adaptive correction.

[0043] Step 23, Loss Function and Optimization Strategy; To balance positioning accuracy and robustness, this invention constructs an improved heteroscedasticity loss function. Loss Function Defined as an adaptive decay form: (17) (18) in, Let be the total loss function during network training. The batch size during training. To smooth the L1 loss function, For uncertainty regularization coefficients, for Threshold parameter for the loss function (recommended value: 0.1). The first term acts as a precision weight. When the model predicts a high degree of uncertainty for the current sample, the regression loss weight for that sample is automatically reduced, thereby minimizing the interference of outliers on the model gradient; the second term... This is a regularization term to prevent the model from circumventing penalties by predicting infinitely large uncertainties.

[0044] loss function Mathematically, it is equivalent to an approximate maximum likelihood estimate of a mixture of Laplace and Gaussian noise, ensuring the convergence of the algorithm in complex environments such as jungles.

[0045] The optimizer uses the AdamW algorithm: (19) in, , These are the model parameters for step t and step t+1, respectively; For learning rate, This is the weight decay coefficient. , These are the first-order moment estimates and second-order moment estimates in the AdamW optimizer, respectively.

[0046] The learning rate is dynamically adjusted using the ReduceLROnPlateau strategy, automatically halving when the validation error fails to decrease for several consecutive rounds to accelerate convergence and avoid local optima. Furthermore, to reduce GPU memory usage and speed up convergence, a mixed-precision training mechanism is introduced. Forward propagation is performed using half-precision floating-point (FP16), combined with gradient scaling techniques to ensure numerical stability, thereby achieving efficient computation on the GPU.

[0047] Step 24, Training Process and Robustness Enhancement; During the training phase, the data collected from the flight experiment was divided into a training set, a validation set, and a test set (in an 8:1:1 ratio). The attitude angle, pixel offset, and spatial coordinates of each sample were augmented using the following data augmentation strategies: Attitude perturbation enhancement: ; Pixel offset noise: ; Coordinate compensation perturbation: ; in, , , These represent the Gaussian noise variances superimposed on the pose angle, pixel offset, and spatial coordinates, respectively. These represent the three-axis attitude angles of the UAV collected by the IMU in the i-th sample, corresponding to the roll angle, pitch angle, and yaw angle, respectively. This represents the pixel offset of the target captured by the visual sensor relative to the image center in the image coordinate system in the i-th sample. Let represent the three-dimensional position coordinates of the UAV in the geographic coordinate system after GPS speed compensation in the i-th sample.

[0048] This mechanism can effectively simulate lighting flicker and attitude jitter during jungle flight, improving the model's generalization performance in complex environments.

[0049] Model training employs mini-batch iterations (Batch Size = 512), performing forward propagation, loss calculation, and backward gradient update in each round. A validation set is used to monitor the error profile. and The convergence trend. Among them, Indicates the first The loss function value of the model on the training dataset at the next iteration. This represents the loss function value of the model on the validation dataset during the same period. By comparing the changes in these two values, it can be determined whether the model is overfitting.

[0050] When the mean absolute error (MAE) of the validation set satisfies: (20) in, The total number of samples is denoted by m, where m is the unit of measurement in meters. This represents the two-dimensional position coordinate vector predicted by the model for the i-th sample. This represents the true geographic location coordinate vector of the i-th sample, indicating that the model has achieved centimeter-level positioning accuracy.

[0051] Step 25, Real-time Inference and Online Adaptive Correction; During the flight mission execution phase, the UAV inputs standardized features in real time. The model outputs the predicted location. With uncertainty To address sudden visual obstruction or GPS signal drift, this invention designs a dynamic weighted correction mechanism: (twenty one) (twenty two) in, This is the final corrected position estimate. For the model to predict the location, The estimation results are from a single sensor. For the weighting function based on uncertainty, This is the adjustment coefficient for the hyperparameter. The uncertainty of the model prediction; If the model uncertainty increases, it will automatically increase. This enables real-time adaptive correction.

[0052] Simulation results show that in a jungle environment, this mechanism can effectively suppress GPS drift and visual errors, and maintain stable system output.

[0053] Step 26, incremental model update and continuous learning; When the drone collects new environmental data (such as different lighting or terrain conditions), the onboard processing system can perform incremental training. Let the parameters of the old model be... The new sample set is The goal of incremental training is: (twenty three) in, To optimize the objective, The loss function for the new task. The regularization coefficient; the second term To constrain knowledge and prevent catastrophic forgetting, a low learning rate is used. Fine-tuning the weights allows the model to adapt to new scene features while maintaining the original model's performance, enabling the model to self-evolve and continuously optimize.

[0054] Step 3, Model Reasoning and Application; This invention employs an offline-trained multimodal deep fusion model for real-time inference during UAV flight, combined with online error correction and adaptive weighting mechanisms, to achieve centimeter-level high-precision positioning and stable control. The entire inference and application process includes five steps: input feature standardization, model forward inference, coordinate inverse normalization, online error correction, and uncertainty-driven dynamic weighting.

[0055] Step 31, Input features and standardization processing; During actual flight, the UAV collects attitude angles, visual features, and spatial measurement data through multiple source sensors, and constructs a multimodal input vector consistent with that used in the training phase. : (twenty four) in,( The corresponding color feature C in the training phase, ( The vector is a coarse coordinate calculated in real time, containing full-dimensional information from pose, visual bias, physical location to prior and appearance.

[0056] To eliminate scale differences between different units, the original input Standardization transformation is required: (25) in, To normalize the prefitted normalizer during the training phase, linear normalization is performed on each dimension to make its mean 0 and variance 1.

[0057] Step 32, multimodal deep fusion model inference; Standardized raw input The multimodal deep fusion model deployed in the flight control computing unit is fed into the system for real-time inference. Forward computation is performed without enabling gradients. (26) Among them, reasoning output This is the comprehensive prediction vector output by the model. For a trained multimodal deep fusion model, This represents a fixed weight parameter.

[0058] This multimodal deep fusion model employs a multi-branch attention fusion structure to model the dependency relationship between temporal and spatial features in a hierarchical manner. The temporal channel processes dynamic information from the IMU, capturing the impact of attitude changes on the position signal; the spatial channel fuses visual and GPS features, learning the nonlinear coupling between perception and measurement.

[0059] The model incorporates an Attitude-Aware Attention Block, which, at each time step, adjusts the attention based on the attitude angle (…). Dynamically adjust the attention weights for each modality: (27) Where A is the attention matrix, Represents the normalized exponential function, , These are the learnable weight matrices; , , These are feature data from a visual sensor, an inertial measurement unit (IMU), and GPS, respectively.

[0060] Through posture-related attention matrix The multimodal deep fusion model can adaptively suppress feature shifts caused by attitude disturbances such as tilting and rotation, thereby maintaining fusion stability under complex flight attitudes.

[0061] Ultimately, the model's comprehensive prediction vector It is decomposed into two parts: location coordinates and uncertainty. Specifically, the output layer adds an uncertainty estimation head to simultaneously predict the uncertainty of the positioning result, i.e.: (28) in, For the comprehensive prediction vector The specific unfolding form; These are the normalized predicted position coordinates, used for subsequent adaptive weighting and error correction; This is a function for deep fusion neural networks.

[0062] Step 33: Output the inverse normalization and coordinate restoration; To facilitate direct use by the drone control module, the network output needs to be denormalized to restore it to the true physical coordinate system: (29)

[0063] in, For the inverse normalization function, This refers to the predicted target position of the UAV in the geographic coordinate system, which can be directly input into the flight controller for path planning and closed-loop navigation control, achieving centimeter-level positioning accuracy.

[0064] Step 34, online error correction and dynamic fusion mechanism; During flight, the UAV's positioning system continuously receives real-time observation coordinates from sensors. To prevent error accumulation caused by model drift or external interference, this invention designs an online error correction mechanism, which calculates the Euclidean distance between the prediction result and the sensor result: (30) If error If the threshold is reached, then dynamic fusion correction will be performed. (31) in, The final target location is obtained after weighting. These are adaptive weighting coefficients, with a range of values. This weighting method enables a smooth transition between prediction results and sensor observations, effectively suppressing noise from a single sensor and improving positioning stability.

[0065] In typical implementations, the weight parameters use a fixed coefficient fusion method, for example: This strategy can maintain the continuity and safety of the output even when there is visual drift or high radar noise.

[0066] Step 35, Confident Learning-Based Adaptive Weighted Mechanism; To further enhance the system's adaptive capability, this invention proposes a self-learning dynamic weighting mechanism based on the uncertainty output of a multimodal deep fusion model. To achieve real-time weight adjustment, define the uncertainty normalization function: (32) (33) in, To integrate the weights of the prediction results, The weights of the sensor observations, This is an estimate of the uncertainty of the sensor observation. Its value is related to the hardware parameters of the sensor used. In this embodiment, it is preset to a fixed value based on the variance of the sensor measurement noise.

[0067] The final correction output of the entire positioning system for: (34) When the model uncertainty is low ( When the uncertainty of the model increases (e.g., in scenarios with changes in lighting, occlusion, or weakened GPS signals), the system will increase the weight of sensor measurements, realizing the dynamic adjustment principle of trusting the model when the uncertainty is low and trusting the sensors when the uncertainty is high.

[0068] Through this mechanism, the UAV positioning system utilizes formulas (32) to (34) to form a closed-loop feedback structure of fusion inference—uncertainty feedback—weight redistribution. Specifically, it utilizes the uncertainty output by the model in real time. Dynamically adjust fusion weights This enables drones to have self-sensing and self-correcting capabilities, significantly enhancing their robustness and stability in complex environments.

[0069] To verify the effectiveness and real-time performance of the multimodal deep fusion model of this invention, a drone simulation platform based on ROS (Robot Operating System) and Gazebo was constructed. The platform integrates multi-source sensing components such as visual sensors, inertial measurement units (IMUs), and GPS modules, and implements data synchronization and timestamp alignment mechanisms to ensure the fusion of multi-sensor information in a unified coordinate system. For a fair performance evaluation, the traditional GPS + visual positioning method was selected as a comparison algorithm, and the positioning accuracy and robustness performance under different environments were compared and analyzed.

[0070] The simulation scenario is designed based on the "dual-drone collaborative autonomous search and rescue" mission, such as... Figure 3 As shown. The drone's flight speed ranges from 12 to 17 m / s, and the environment is a complex mix of jungle and residential areas, including various lighting variations, occlusion, and attitude disturbances. Three typical search and rescue targets are set in the scenario: critically injured individuals (red), lightly injured individuals (yellow), and healthy individuals (camouflage white), as shown. Figure 4 As shown, the visual features correspond to different search and rescue states, which are used to test the adaptability and robustness of the algorithm in multi-target recognition and localization tasks.

[0071] To quantitatively evaluate system performance, the following metrics were selected: Position Error, which is the Euclidean distance between the predicted coordinates and the true target position; and Detection Error, which is the deviation between the pixel coordinates output by the vision detection module and the ground truth projection.

[0072] Figure 4 The error statistics for targets of different colors are presented. Figure 5 As can be seen, traditional visual detection is prone to detection drift under complex textures or reflective surfaces, while the multimodal deep fusion model of this invention can maintain stable prediction output under pose perturbation and illumination changes. Specific numerical results are as follows: For the camouflage_white target, the detection error is relatively large (about 0.29 m), but the multimodal deep fusion model still achieves a localization error of 0.20 m, indicating that the present invention can effectively correct visual errors under low contrast targets; For the red target, the detection error is 0.29 m, while the localization error after fusion is reduced to 0.08 m, which demonstrates the high accuracy performance of the multimodal deep fusion model under low uncertainty visual feature conditions; For yellow targets, visual detection fluctuates due to light reflection, resulting in a detection error of approximately 0.32 m. However, the localization error output by the multimodal deep fusion model is only 0.25 m, maintaining good stability.

[0073] Overall, the multimodal deep fusion model significantly reduced positioning errors in all three color scenarios, with an average error reduction of approximately 65–80%. In summary, the simulation results fully demonstrate that the present invention maintains high accuracy and strong robustness under different color and lighting conditions. Its attitude attention mechanism effectively reduces the impact of flight attitude disturbances on visual features; while the uncertainty estimation module dynamically adjusts the fusion weights, enabling the UAV positioning system to maintain continuous centimeter-level positioning output even in complex environments. These results verify the feasibility and practical value of the method of the present invention in real-time navigation and target tracking tasks for UAVs.

[0074] The multimodal deep fusion model proposed in this invention has advantages such as simple structure, high computational efficiency, and strong real-time performance. It can run efficiently on embedded GPU platforms or flight control main control chips, meeting the navigation and control requirements of UAVs in high-speed dynamic environments. Due to the small number of model parameters and low hardware dependence, the UAV positioning system can be directly embedded into the existing flight control system to achieve end-to-end intelligent positioning and autonomous decision-making.

[0075] By continuously collecting real-world flight data from UAVs, this invention can further perform incremental learning and online retraining, continuously optimizing model weights and feature distributions, thereby improving generalization ability and adaptive performance under complex climate, terrain, and lighting conditions. This incremental learning and online retraining mechanism enables the system to continuously evolve, providing technical support for long-term intelligent flight missions.

[0076] This invention's method is not only applicable to unmanned aerial vehicles (UAVs) for tasks such as inspection, surveying, emergency rescue, and precise material delivery, but can also be extended to various autonomous platforms such as unmanned vehicles, unmanned surface vessels, and mobile robots. Cross-platform migration and reuse can be achieved simply by changing the sensor adapter and coordinate mapping modules, greatly reducing system development and deployment costs. With its high precision, strong robustness, and universal architecture, this invention has broad application prospects and industrial promotion potential in intelligent transportation, sea-land-air collaborative sensing, and the intelligent equipment industry.

Claims

1. A method for target positioning of a UAV based on heterogeneous data fusion and deep learning, characterized in that, The steps include the following: S1, collecting the geographic position information, attitude, acceleration information and image data of the unmanned aerial vehicle by the data perception module, pre-processing the collected data, and constructing a multi-modal input vector; S2, using the multi-modal input vector to train the constructed multi-modal deep fusion model; S3, during flight, using the offline trained multi-modal deep fusion model for real-time inference, and combining online error correction and adaptive weighting mechanism to realize centimeter level high-precision positioning and stable control. 2.The method of claim 1, wherein, The pre-processing of the collected data includes: A11, GPS coarse positioning and velocity compensation; The linear velocity calculated by GPS and IMU is used to dynamically compensate the GPS position to obtain continuous position estimation: ， wherein, is the position coordinate in the global geographic coordinate system after compensation, is the position coordinate vector in the global geographic coordinate system obtained and converted by the GPS receiver, is the linear velocity vector of the UAV in the same coordinate system, is the average transmission and processing time delay of the GPS signal relative to the IMU time reference pre-calibrated by the system; A12, camera pixel offset calculation; The camera determines the center position of the target in the image coordinate system through a target detection algorithm With the image center As a reference, the pixel offset is defined as: ， wherein is a pixel offset amount of the x-axis, is a pixel offset amount of the y-axis; Combining intrinsic matrix of camera , calculating direction vector of target in camera coordinate system through back projection : ， wherein, denotes the camera intrinsic matrix, and T denotes the matrix transpose; A13, attitude angle collection and coordinate conversion; In order to realize the alignment between the unmanned aerial vehicle coordinate system and the geographic coordinate system, the Euler angle rotation matrix is used for coordinate conversion: ， wherein, represents the total rotation matrix from the drone body coordinate system to the geographical coordinate system; , , yaw, pitch and roll angles of the drone acquired by the IMU, respectively; represents a rotation about the Z axis basic rotation matrix for an angle represents a rotation about the Y axis basic rotation matrix for an angle represents a rotation about the X axis by the basic rotation matrix for an angle Then, the pose matrix provided by the IMU and the camera extrinsic parameters is converted to the geographic coordinate system: ， wherein, is a line of sight vector in a geographic coordinate system; Let the position of the UAV in the geographic coordinate system be , and assume that the ground height is 0, then the target rough coordinates are obtained by the intersection of the ray and the ground plane. ， wherein, is the component of the line-of-sight vector in the vertical direction; X, Y, Z represent three-dimensional position coordinates in a geographic coordinate system; represents the target rough coordinate detected by the camera; Then, time synchronization and feature combination are performed to align the asynchronous multi-source data to the same timestamp and construct a multi-modal input tensor.

3. The method of claim 2, wherein, The implementation process of constructing the multi-modal input tensor is as follows: taking the IMU high-frequency data as the time reference, linearly interpolating the GPS and camera frame data, aligning the GPS position, visual offset and IMU attitude under the unified timestamp, and generating a multi-modal joint sample; The finally constructed multi-modal input vector includes: unmanned aerial vehicle attitude angle, camera pixel offset, compensated geographic position coordinates, target rough coordinates and target color features, which are used as the input of the multi-modal deep fusion model.

4. The method of claim 1, wherein, The multi-modal deep fusion model includes: a feature extraction module, an attitude related attention module and a coordinate regression and uncertainty estimation module; The feature extraction module realizes the nonlinear mapping of high-dimensional heterogeneous input through a multi-layer fully connected structure, excavates the deep correlation between the flight state of the unmanned aerial vehicle and the visual features, and converts the original data into high-dimensional feature representation, which is defined as: wherein, , are a weight matrix and a bias vector of the first layer of the feature extraction network, respectively; is an output feature vector of the first hidden layer; is a ReLU activation function of the first hidden layer, is an output feature vector of the first hidden layer; is a ReLU activation function of the second hidden layer, is an output feature vector of the second hidden layer; The attitude related attention module introduces an attention mechanism based on attitude angle to assign adaptive attention weights to different modal features; the expression of adaptive attention weight is as follows: ， wherein, denotes the adaptive attention weight of the j-th modality at the current pose; is a nonlinear mapping network containing two fully connected layers, which is used to map the pose data to the same dimension as the feature vector to realize the feature space alignment; is the feature vector of the i-th sample and the j-th modality, is the k-th modality feature vector when traversing the denominator sum; is the roll angle, pitch angle and yaw angle of the unmanned aerial vehicle corresponding to the i-th sample; A unified global feature is obtained by weighted fusion Indicates: ， wherein, is the global feature vector after attention-weighted fusion; The coordinate regression and uncertainty estimation module outputs the uncertainty of the prediction result while predicting the target geographic position, and the expression is as follows: ， wherein, represents the two-dimensional position coordinates of the target predicted by the model in the geographical coordinate system, is a parametric regression network, represents the uncertainty estimate of the model.

5. The method of claim 1, wherein, An improved heteroscedastic loss function is used in the training process, and the expression is as follows: ， ， wherein, is the total loss function for network training, is the batch size for training, is the smooth L1 loss function, is the uncertainty regularization coefficient, is the is the threshold parameter of the loss function; The term acts as an accuracy weight, when the model predicts a high uncertainty for the current sample, the regression loss weight of this sample is automatically reduced; is the regularization term; represents the two-dimensional position coordinate vector predicted by the model for the i-th sample, represents the real geographical position coordinate vector of the i-th sample.

6. The method of claim 1, wherein, In the training process, a dynamic weighting correction mechanism is used for online adaptive correction, and the expression of the dynamic weighting correction mechanism is as follows: ， ， wherein, is the final corrected position estimate, is the model predicted position, is the estimate result of a single sensor, is the weight function based on uncertainty, is the adjustment coefficient of hyperparameters, is the uncertainty of the model prediction; If the model uncertainty rises, then automatically increase , to achieve real-time adaptive correction.

7. The method of claim 1, wherein, The steps of realizing centimeter level high-precision positioning and stable control include: S31, standardizing the input features; S32, the normalized original input is sent to a multi-modal deep fusion model deployed in the flight control computing unit for real-time inference; forward calculation is performed under the condition that the gradient is not enabled: ， wherein the inference output is a comprehensive prediction vector output by the model, is a trained multi-modal deep fusion model, represents fixed weight parameters; the comprehensive prediction vector includes position coordinates and uncertainty ; The multi-modal deep fusion model internally introduces an attitude attention module to dynamically adjust the attention weights of each modality according to the attitude angle at each time; S33, the model output is de-normalized and the coordinates are restored to the real physical coordinate system: ， wherein, is the inverse normalization function, is the predicted target position of the UAV in the geographical coordinate system; S34, online error correction is performed, if error Dynamic fusion correction is performed: ， wherein, is the final position target after weighting; is an adaptive weighting coefficient, taking a value range ; ， wherein, is a Euclidean distance between the predicted result and the sensor result, is a threshold value; a real-time observation coordinate of the sensor; S35, uncertainty based on the multi-modal deep fusion model output Real-time weight adjustment is achieved: ， ， Final positioning system correction output is: ， wherein, is a weight for the fusion prediction result, is a weight for the sensor observation result, is an uncertainty estimate for the sensor observation.

Images