A fusion perception self-positioning method

Abstract

The application provides a fusion perception self-positioning method, which constructs an end-to-end fusion perception neural network model, and takes a continuous frame image pair of a UAV, inertial measurement unit (IMU) data and global positioning system (GPS) original measurement values as multi-modal inputs, wherein a double-task regression head is used to generate visual-inertial position estimation and relative attitude estimation relative to the last time, a GPS validity detection and dynamic fusion module is used to adaptively adjust the fusion weight between visual-inertial prediction results and GPS measurement values according to real-time GPS signal quality, a coordinated pose loss function is used during training, and the final absolute position error and relative attitude error are jointly optimized, so that the three-degree-of-freedom absolute position of the UAV in the east-north-up (ENU) coordinate system and the relative attitude relative to the last time are finally obtained. The application can maintain positioning continuity in the GPS intermittent, shielding or interference scene, and improve positioning accuracy and trajectory smoothness.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of autonomous navigation for unmanned aerial vehicles (UAVs), computer vision, and state estimation, and particularly to a fusion perception self-localization method. Background Technology

[0002] Accurate self-localization of unmanned aerial vehicles (UAVs) is fundamental for stable flight, path planning, and mission execution. Traditional pose estimation methods mainly rely on the fusion of the Global Positioning System (GPS) and inertial navigation systems, such as the Extended Kalman Filter (EKF). However, such methods have significant limitations: in environments where signals are blocked or reflected, positioning accuracy drops sharply or even fails; furthermore, the inherent drift error of inertial sensors accumulates over time, leading to divergence in attitude estimation.

[0003] To compensate for the limitations of GPS, Visual Inertial Odometry (VIO) has been widely studied as a supplementary solution. VIO estimates the relative motion and attitude of a vehicle through data fusion from cameras and inertial measurement units (IMUs). However, pure VIO systems inherently suffer from scale uncertainty and cumulative drift problems. Although loop closure detection and graph optimization can eliminate drift to some extent, their long-term stability remains challenging in scenarios lacking salient features.

[0004] In recent years, deep learning-based methods have provided new ideas for multi-sensor fusion localization. These methods can learn the mapping relationship between complex sensor noise models and environmental features from massive amounts of data, showing strong potential. However, existing deep learning-based fusion localization schemes still have the following limitations: (1) Most of them only output one of absolute position or relative pose, failing to make full use of the complementary constraints between absolute position supervision and inter-frame pose supervision for joint optimization; (2) GPS is usually used as a constant and reliable input source, without considering the dynamic changes in its signal quality. When GPS is intermittent or interfered with, the model performance will deteriorate significantly; (3) The network structure is insufficient in mining motion change information between continuous images, which limits its performance in high-speed maneuvering scenarios. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a fusion-sensing self-localization method. This method reconstructs the UAV pose estimation problem into an end-to-end regression problem based on deep learning and multi-sensor fusion. A fusion-sensing self-localization model is constructed to jointly estimate the absolute position and relative attitude of the UAV at any given time. By utilizing deep learning methods, multi-modal sensor information from vision, inertial, and satellite data is fused to achieve high-precision and robust self-localization of the UAV.

[0006] A fusion-sensing self-localization method, the method comprising the following steps:

[0007] Step S1: Acquire and synchronize the multimodal sensor data of the UAV, including continuous image pairs, inertial measurement unit (IMU) sequences and global positioning system (GPS) raw coordinates, and calculate the corresponding absolute position true value and relative attitude true value;

[0008] Step S2: Construct a fusion sensing neural network model. The model receives the multimodal sensor data sequence, performs feature fusion on continuous images, IMU sequences and GPS position coordinates, and generates the visual-inertial position estimate, position correction amount and relative attitude estimate relative to the previous moment at the current moment.

[0009] Step S3: Design a coordinated pose loss function to supervise both the final absolute position estimate output from Step S4 and the relative pose estimate output from Step S2 during model training, thereby achieving joint optimization of the dual tasks of absolute position and relative pose.

[0010] Step S4: Introduce a GPS validity detection and dynamic fusion module into the fusion sensing neural network model to evaluate the reliability of the current GPS signal and adaptively fuse the visual-inertial prediction results with the GPS measurement values ​​to generate the final absolute position output.

[0011] Step S5: Train the fusion sensing neural network model using the dataset obtained in step S1, and verify the model's performance in absolute positioning accuracy and relative pose consistency through evaluation metrics.

[0012] Further, step S1 specifically includes:

[0013] Step S11: Extract image timestamps, IMU time series, GPS time series and ground truth pose time series from the UAV's multimodal sensor data source. The data source includes camera images, inertial measurement unit data, GPS raw coordinates and pose ground truth records for supervised training.

[0014] Step S12: Based on the image timestamp, select adjacent IMU, GPS and ground truth samples before and after each image timestamp; use linear interpolation for triaxial acceleration, triaxial angular velocity, GPS position and ground truth position, and use spherical linear interpolation for ground truth attitude quaternions to synchronize each modal data to the image frame rate.

[0015] Step S13: Perform coordinate transformation on the synchronized GPS latitude, longitude and altitude coordinates. First, convert the longitude, latitude and altitude in the WGS-84 coordinate system to geocentric and geofixed coordinates. Then, construct a rotation matrix based on the latitude and longitude of the starting point and transform it to the local northeast-sky (ENU) coordinate system with the starting point as the origin to obtain the GPS position coordinates.

[0016] Step S14: Based on the true value of the absolute attitude quaternion from the previous time step after synchronization. The truth value of the absolute attitude quaternion at the current moment Calculate the true values ​​of the inter-frame relative attitude quaternions:

[0017]

[0018] in, The quaternion q = [q_w, q_x, q_y, q_z] represents the truth value of the relative attitude at the current time relative to the previous time. The quaternion q = [q_w, q_x, q_y, q_z] consists of a real part q_w and three imaginary parts q_x, q_y, and q_z. This represents quaternion multiplication. It represents the inverse of a quaternion.

[0019] Furthermore, the fusion perception neural network model includes a motion perception visual feature extractor, an IMU feature construction unit, a feature fusion layer, a temporal modeling module, a dual-task regression head, and a GPS validity detection and dynamic fusion module connected to the absolute position regression path.

[0020] Further, step S2 specifically includes:

[0021] Step S21: Motion-aware visual feature extractor, input is two consecutive RGB images. and The 6-channel image tensor obtained by stitching along the channel dimension The motion-aware visual feature extractor first extracts the visual features through a shared or corresponding convolutional encoder. and Spatial feature extraction is performed to obtain the image features of adjacent frames. Then, the features of adjacent frames are subtracted to obtain inter-frame variation features. The current frame image features are then concatenated with the inter-frame variation features. The concatenated features are further processed through convolutional layers, pooling layers, and fully connected layers to finally output a visual feature vector containing motion change information and scene structure information. ;

[0022] Step S22: The IMU feature construction unit extracts the IMU sampling sequence between the timestamps of two adjacent image frames, and performs normalization and time window statistics on the three-axis acceleration and three-axis angular velocity, respectively, to obtain the IMU feature vector. ;

[0023] Step S23: The feature fusion layer combines the visual feature vectors IMU feature vectors and GPS location coordinates The concatenation process yields a joint feature vector, which is then dimensionality-reduced and fused using a multilayer perceptron consisting of fully connected layers, activation functions, and dropout layers to obtain fused features. ;

[0024] Step S24: The temporal modeling module receives the fused features from L consecutive time steps. The sequence is encoded using a Long Short-Term Memory (LSTM) network to obtain the hidden states at each time step. A gated recurrent unit further recursively updates the hidden states. Then, the final hidden state is used as the query vector, and the hidden states at each historical time step are used as the key and value vectors. Scaling dot product attention weights are calculated, and finally, the historical hidden states are weighted and summed to obtain the contextual features at the final time step. ;

[0025] Step S25: The dual-task regression head includes an absolute position generation path and a relative pose regression head. The absolute position generation path uses contextual features. As input, output is a visual-inertial position estimate. and position correction amount and will and The data is then passed to the GPS validity detection and dynamic fusion module, which generates the final absolute position estimate. Relative pose regression head with contextual features As input, a four-dimensional vector is regressed through a fully connected layer, and after normalization, the relative pose quaternion estimate is obtained. .

[0026] Furthermore, the coordinated pose loss function Loss due to absolute position and relative attitude loss The weighted composition is expressed as follows:

[0027]

[0028] in, and To balance the weighting coefficients of the two losses, The absolute position at the current moment is the true value. The true value of the relative attitude quaternion calculated in step S14. The final absolute position estimate is output from step S4. This is the relative attitude quaternion estimate output from step S25.

[0029] Furthermore, absolute position loss Using the Huber loss function, it can be expressed as:

[0030]

[0031] Where N is the sample size. Indicates the first training samples, Indicates the first The absolute position prediction error of each sample. Here, δ is the Huber loss function, and δ is the threshold parameter for the Huber loss.

[0032] Furthermore, the relative attitude loss is based on relative attitude quaternion estimation. relative attitude truth quaternion To construct the angle error between the two, first calculate the angle error between them, then apply the Smooth L1 loss function to the angle error, specifically as follows:

[0033]

[0034] Where N is the sample size. Indicates the first There are training samples, where · represents the quaternion dot product and |·| represents the absolute value. The threshold parameter for Smooth L1 loss.

[0035] Further, step S4 specifically includes:

[0036] Step S41: GPS validity check, obtain the current GPS location coordinates. Input a neural network, and the network output is mapped to a validity score by a sigmoid activation function. ;

[0037] Step S42: Dynamic weight generation, incorporating contextual features and GPS location coordinates The data is then concatenated again and used to generate a 3D fusion weight vector through another neural network. ;

[0038] Step S43: Adaptive fusion based on validity score and 3D fusion weight vector Calculate the final absolute position estimate :

[0039]

[0040] in, This represents the final absolute position estimate. Indicates GPS location coordinates, This represents visual-inertial position estimation. ⊙ indicates positional correction, and ⊙ indicates element-wise multiplication.

[0041] Furthermore, the evaluation indicators in step S5 include absolute position error, relative attitude error, and trajectory consistency. The absolute position error includes the overall root mean square error RMSE and the root mean square errors of the three coordinate axis components RMSE_X, RMSE_Y, and RMSE_Z. The relative attitude error is the relative rotation error RRE, and the trajectory consistency is the relative position error RPE.

[0042] The fusion sensing self-localization method proposed in this invention has at least the following beneficial technical effects:

[0043] First, by uniformly synchronizing images, IMU, GPS, and ground truth poses based on image timestamps and transforming GPS coordinates to the local ENU coordinate system, supervision bias caused by inconsistencies in time and coordinates among multi-source data can be reduced. Second, the motion-aware visual feature extractor uses feature difference and stitching between adjacent frames to obtain inter-frame motion change information, which, combined with IMU features, enhances the motion representation capability in high-speed maneuvers or texture-changing scenarios. Third, by coordinating the pose loss function to simultaneously constrain the final absolute position and relative pose, the model utilizes both the global constraints provided by GPS or ground truth position and the inter-frame pose changes. The invention provides local motion consistency constraints, thereby reducing the drift risk caused by a single supervised target; fourth, the GPS validity detection and dynamic fusion module adaptively switches between GPS-dominated and vision-inertial-dominated modes based on real-time GPS reliability, and sets fusion weights for the X, Y, and Z directions respectively. Therefore, when GPS is reliable, it can use absolute coordinates to suppress accumulated errors, and when GPS is blocked, intermittent, or interfered with, it can still maintain positioning continuity by relying on vision-inertial estimation. Thus, the invention can improve the accuracy, robustness, and trajectory smoothness of UAV self-localization in complex environments, providing a reliable solution for stable navigation of UAVs in complex environments. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a fusion sensing self-localization method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the structure of the motion-sensing visual feature extractor provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the overall framework of the fusion perceptual neural network model provided in the embodiments of the present invention;

[0048] Figure 4 This is a schematic diagram of the internal structure of the LSTM unit provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram illustrating the calculation of the attention mechanism (scaled dot product) provided in an embodiment of the present invention;

[0050] Figure 6 This is a comparison diagram of the trajectories of the model prediction, EKF fusion, and ground truth in three-dimensional space provided in the embodiments of the present invention.

[0051] Figure 7 This is a comparison diagram of the trajectories of the model prediction, EKF fusion, and ground truth in the X, Y, and Z coordinate axes provided in the embodiments of the present invention.

[0052] Figure 8 This is a comparison chart of the 3D position error cumulative distribution function of the model prediction (Model) and the extended Kalman filter fusion method (EKF) provided in the embodiments of the present invention;

[0053] Figure 9 This is a comparison diagram of the error distribution of the model prediction (Model) and extended Kalman filter (EKF) fusion method provided in the embodiments of the present invention in each coordinate axis direction;

[0054] Figure 10 This is a comparison diagram of the relative attitude error between the model of the present invention and other models provided in the embodiments of the present invention;

[0055] Figure 11 This is a comparison diagram of the absolute position error of the model of the present invention with other models provided in the embodiments of the present invention;

[0056] Figure 12 This is a comparison chart of the absolute positional errors of the model of the present invention and other models in terms of the X, Y, and Z coordinate axes, provided in the embodiments of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] like Figure 1 As shown, the fusion sensing self-localization method provided by the present invention includes the following steps:

[0059] S1: Acquire and synchronize multimodal sensor data of the UAV, including continuous image pairs, inertial measurement unit (IMU) sequences and global positioning system (GPS) raw coordinates, and calculate the corresponding absolute position true value and relative attitude true value.

[0060] In UAV flight experiments or simulations, the following data streams are simultaneously acquired: images from the left eye camera, inertial measurement unit (IMU) data (including three-axis acceleration and three-axis angular velocity), raw GPS coordinates, and ground truth pose data provided by a high-precision motion capture system. Image timestamps, IMU time series, GPS time series, and ground truth pose time series are then extracted from these data. Using the image timestamp as a reference, for each image timestamp t, two adjacent sampling times before and after t are found in the IMU, GPS, and ground truth time series. For three-axis acceleration, three-axis angular velocity, GPS position, and ground truth position, linear interpolation is used to obtain the corresponding data at time t. For the attitude quaternion ground truth, spherical linear interpolation is used to obtain the attitude at time t, thus synchronizing the modal data to the image frame rate. The synchronized GPS latitude, longitude, and altitude coordinates are transformed. First, the longitude, latitude, and altitude in the WGS-84 coordinate system are converted to geocentric and geofixed coordinates. Then, a rotation matrix is ​​constructed based on the latitude and longitude of the starting point to transform it to the local northeast-sky (ENU) coordinate system with the starting point as the origin, thus obtaining the GPS position coordinates. Subsequently, the true values ​​of the absolute attitude quaternions after synchronization of two adjacent frames are used to obtain the GPS position coordinates. and Calculate the true value of relative attitude quaternions The calculation method is as follows: , The quaternion q = [q_w, q_x, q_y, q_z] represents the truth value of the relative attitude at the current time relative to the previous time. The quaternion q = [q_w, q_x, q_y, q_z] consists of a real part q_w and three imaginary parts q_x, q_y, and q_z. This represents quaternion multiplication. The inverse of a quaternion is represented by the inverse of its conjugate quaternion for a unit quaternion; for a non-unit quaternion, the inverse is the conjugate quaternion divided by the square of the quaternion's modulus. Finally, a data sample is constructed using consecutive image pairs, corresponding IMU feature vectors, and GPS coordinates as input, and the current absolute position and relative pose as supervision labels.

[0061] S2: Constructing a fusion-sensory neural network model. The fusion-sensory neural network model of this invention is an end-to-end deep architecture, such as... Figure 3 As shown, the model specifically includes a motion-aware visual feature extractor, an IMU feature construction unit, a feature fusion layer, a temporal modeling module, a dual-task regression head, and a GPS validity detection and dynamic fusion module connected to the absolute position regression path. The model receives the multimodal sensor data sequence, performs feature fusion on continuous images, IMU sequences, and GPS position coordinates, and generates the current visual-inertial position estimate, position correction, and relative attitude estimate relative to the previous moment.

[0062] First, a motion-aware visual feature extractor is used to extract motion change features from consecutive image pairs, with input being two consecutive RGB images. and The 6-channel image tensor obtained by stitching along the channel dimension The 6-channel image tensor is only the input to the visual branch and is not equivalent to the complete multimodal sensor data sequence. The complete multimodal sensor data sequence also includes the IMU sequence and GPS location coordinates. The network structure of the motion-aware visual feature extractor is as follows: Figure 2 As shown, it encodes spatial features of adjacent frames through convolutional and pooling layers, then differs and concatenates the features of adjacent frames to extract motion change information between consecutive frames. The concatenated features are then processed by subsequent convolutional layers and adaptive average pooling to output a visual feature vector. Specifically, the motion-aware visual feature extractor first uses a shared or corresponding convolutional encoder to encode spatial features of adjacent frames. and Spatial feature extraction is performed to obtain the image features of adjacent frames. Then, the features of adjacent frames are subtracted to obtain inter-frame variation features. The current frame image features are then concatenated with the inter-frame variation features. The concatenated features are further processed through convolutional layers, pooling layers, and fully connected layers to finally output a visual feature vector containing motion change information and scene structure information. .

[0063] Subsequently, the IMU feature construction unit extracts the IMU sampling sequence between the timestamps of two adjacent image frames, normalizes and performs time window statistics on the triaxial acceleration and triaxial angular velocity, including at least mean calculation, to obtain the IMU feature vector. .

[0064] Then, the feature fusion layer combines the visual feature vectors IMU feature vectors and GPS location coordinates The concatenation results are combined into a joint feature vector, which is then subjected to dimensionality reduction and fusion using a multilayer perceptron consisting of fully connected layers, activation functions, and dropout layers to obtain the fused features. .

[0065] Subsequently, the temporal modeling module receives the fused feature sequence over L consecutive time steps. The sequence is composed of elements, and temporal dependencies are extracted using LSTM and GRU. Then, a scaled dot product attention mechanism is used to weight and sum the historical hidden states to obtain the context features at the final time step. Specifically, a Long Short-Term Memory (LSTM) network is used to encode the sequence to obtain the hidden states at each time step; the structure of the LTM network is as follows: Figure 4 As shown, the gated recurrent unit then recursively updates the hidden state; secondly, the hidden state at the final time step is used as the query vector, and the hidden states at each historical time step are used as the key vector and value vector, respectively, to calculate the scaled dot product attention weights. The calculation process of the scaled dot product attention mechanism is as follows: Figure 5 As shown, the calculation method is as follows: ,in, These are query, key, and value matrices, respectively. The dimension of the key vector, and the scaling factor. To prevent gradient loss due to excessively large dot products; finally, a weighted summation of the historical hidden states is performed to obtain the context features at the final time step. .

[0066] Finally, the dual-task regression head includes an absolute position generation path and a relative pose regression head. The absolute position generation path is based on contextual features. Output visual-inertial position estimation and position correction amount and will and The data is passed to the GPS validity detection and dynamic fusion module described in step S4, where the final absolute position estimate is generated. The absolute position generation path is implemented by a position regression subnetwork, which can be composed of fully connected layers and nonlinear activation functions, with context features as its input. The output is a three-dimensional position vector; the position correction amount Used to represent the deviation of visual-inertial estimation relative to GPS measurements in the X, Y, and Z directions; relative attitude regression head with contextual features As input, a four-dimensional vector is regressed through a fully connected layer, and after normalization, the relative pose quaternion estimate is obtained. Therefore, the absolute position generation path in this step provides visual-inertial position information before dynamic fusion, and the final absolute position estimate is output by step S4.

[0067] S3: Design and apply a coordinated pose loss function to supervise both the final absolute position estimate output by step S4 and the relative pose estimate output by step S2 during model training, thereby achieving joint optimization of the dual tasks of absolute position and relative pose.

[0068] To simultaneously optimize both absolute position and relative pose, the coordinated pose loss function designed in this invention is composed of a weighted sum of absolute position loss and relative pose loss, wherein the absolute position loss is the final absolute position estimate output in step S4. and absolute position truth value As input, the relative attitude loss is estimated using the relative attitude quaternion output from step S2. The true values ​​of the relative attitude quaternions obtained in step S1 The input is [value]. The absolute position loss employs Huber loss to reduce the impact of GPS outliers or ground truth sampling noise on the training process; the relative pose loss is constructed based on quaternion angle errors, calculating the angle error through the dot product between the relative pose quaternion estimate and the ground truth, and applying SmoothL1 loss to the angle error. This joint loss function guides the model to accurately regress the global position while maintaining consistency in pose changes between frames.

[0069] Specifically, the pose coordination loss function Loss due to absolute position and relative attitude loss The weighted composition is expressed as follows:

[0070]

[0071] in, and To balance the weighting coefficients of the two losses, The absolute position at the current moment is the true value. The true value of the relative attitude quaternion calculated in step S14. The final absolute position estimate is output from step S4. The relative attitude quaternion is estimated from the output of step S25;

[0072] Absolute position loss The Huber loss function is used to enhance robustness to outliers:

[0073]

[0074] Where N is the sample size. Indicates the first training samples, Indicates the first The absolute position prediction error of each sample. Here, δ is the Huber loss function, and δ is the threshold parameter for the Huber loss.

[0075] Relative attitude loss is based on relative attitude quaternion estimation relative attitude truth quaternion To construct the angle error between the two, first calculate the angle error between them, and then apply the Smooth L1 loss function to the angle error:

[0076]

[0077] Where N is the sample size. Indicates the first There are training samples, where · represents the quaternion dot product and |·| represents the absolute value. The threshold parameter for Smooth L1 loss.

[0078] S4: A GPS validity detection and dynamic fusion module is introduced into the fusion sensing neural network model to evaluate the credibility of the current GPS signal and adaptively fuse the visual-inertial prediction results with the GPS measurement values ​​to generate the final absolute position output. This module is embedded in the absolute position generation path.

[0079] First, GPS validity is checked by taking the current GPS location coordinates. The input is a small neural network, which is not a single sigmoid function, but a multilayer perceptron consisting of at least one fully connected layer and a non-linear activation function; the output of the multilayer perceptron is then mapped to a validity score using a sigmoid activation function. It is used to characterize the reliability of GPS signals. A value close to 1 indicates that the GPS signal is reliable. A value close to 0 indicates that the GPS is unreliable; the sigmoid function is only used to map network outputs to a value between 0 and 1.

[0080] Then, dynamic weight generation incorporates contextual features. GPS location coordinates The data is then concatenated again, and a three-dimensional fusion weight vector is generated using another small neural network. This is used to apply different levels of weighting to the position correction amounts in the X, Y, and Z directions, thereby controlling the position correction intensity in each of the three directions. The other small neural network can also employ a multilayer perceptron structure, with its output mapped to a range of 0 to 1 via a sigmoid function.

[0081] Finally, adaptive fusion is performed based on the validity score. and 3D fusion weight vector Calculate the final absolute position estimate :

[0082]

[0083] in, This represents the final absolute position estimate. Indicates GPS location coordinates, This represents visual-inertial position estimation. This represents the position correction amount, and ⊙ represents element-wise multiplication. When GPS is reliable, When the value is close to 1, the output primarily uses GPS location coordinates, supplemented by fine-tuning corrections based on visual-inertial predictions; when GPS is unreliable, The value is close to 0, and the output mainly relies on visual-inertial position estimation, thereby ensuring system continuity in the event of GPS failure. Steps S41 to S44 together constitute the function of the aforementioned GPS validity detection and dynamic fusion module.

[0084] The fusion sensing neural network model provided in this invention first synchronizes the time and unifies the coordinates of multimodal heterogeneous data from the camera, inertial measurement unit (IMU), and global positioning system (GPS), constructing a dual-supervised dataset containing absolute position and inter-frame relative attitude. Then, a deep fusion network is designed, using a motion-aware visual feature extractor to parse motion change information of continuous image pairs and fusing it with IMU features and GPS position features. A temporal modeling module then captures long-range dependencies. The network adopts a dual-task output structure, generating visual-inertial position information and relative attitude estimation respectively. A GPS validity detection and dynamic fusion module adaptively fuses visual-inertial predictions and GPS measurements based on real-time signal quality. Simultaneously, a coordinated pose loss function is used to jointly optimize the final absolute position error and relative attitude angle error, driving the model to achieve high-precision and robust self-localization in complex environments.

[0085] S5: Train the fused perceptual neural network model using the dataset obtained in step S1, and verify the model's performance in absolute positioning accuracy and relative pose consistency using evaluation metrics. The model is trained using the dataset constructed in step S1. The optimizer is Adam, the initial learning rate is set to 0.001, the batch size is determined based on the GPU memory configuration, and the loss function is the aforementioned coordinated pose loss. After training, evaluation is performed on an independent test set. Evaluation metrics include absolute position error, relative pose error, and trajectory consistency. The absolute position error includes the root mean square error (RMSE) and the root mean square errors of the X, Y, and Z coordinate axis components RMSE_X, RMSE_Y, and RMSE_Z, used to evaluate global positioning accuracy. The overall RMSE is obtained by taking the square root of the mean square of the Euclidean distance between the final absolute position estimate and the true absolute position, while the component RMSEs are obtained by taking the square root of the mean square of the prediction errors on the corresponding coordinate axes.

[0086] The relative attitude error index, also known as the relative rotation error (RRE), is measured in radians (rad) and is used to assess the accuracy of attitude change prediction. This index calculates the rotation angle error based on the dot product of the relative attitude quaternion estimate and the true relative attitude quaternion.

[0087] The trajectory consistency index, or relative position error (RPE), is calculated by the root mean square error between the predicted displacement of adjacent frames and the actual displacement of adjacent frames. It is used to evaluate the accuracy of displacement prediction between adjacent frames.

[0088] The comparative experiment compares the method of this invention (denoted as Model) with the traditional extended Kalman filter fusion method (EKF), with the ground truth provided by the motion capture system.

[0089] Figure 6 The Model predicted trajectory, EKF fused trajectory, and ground truth trajectory were intuitively compared in three-dimensional space. Figure 7Further, the 3D trajectory was decomposed into three independent coordinate axis components (X, Y, and Z) for time series comparison: In the X-direction, the Model prediction curve (red) closely matched the true curve (blue), with only minor deviations in a few places (maximum error 0.351m), while the EKF prediction curve (green) had a maximum error of 1.115m; in the Y-direction, similar to the X-direction, the Model prediction was stable and accurate (maximum error 0.306m), while the EKF prediction deviated, with a maximum error exceeding 1.2m; in the Z-direction, the EKF showed significant oscillations, while the Model prediction curve had a higher degree of overlap with the true value, smaller fluctuations, and a maximum error (0.025m) smaller than the EKF maximum error (0.053m), demonstrating more refined height control capabilities.

[0090] pass Figure 6 and Figure 7 Comprehensive analysis shows that the method of the present invention not only has a high trajectory restoration accuracy in the overall 3D space, but also achieves more accurate and smoother predictions than the traditional extended Kalman filter fusion method (EKF) in each coordinate axis component (X, Y, Z), proving the effectiveness of the fusion perception model and GPS dynamic fusion mechanism proposed in this invention.

[0091] This invention also quantifies the difference in 3D global localization accuracy between the proposed model and the EKF method from a statistical probability perspective using a Cumulative Distribution Function (CDF) plot. The CDF curve represents the probability that the localization error is less than the value on the horizontal axis, such as... Figure 8 As shown, the CDF curve (red) of the Model lies entirely to the upper left of the CDF curve (green) of the EKF. This means that for any given error threshold, the probability of the Model's error being less than that threshold is higher than that of the EKF. Specifically, when the error threshold is 0.25m, approximately 75% of the samples from the Model meet the requirements, while only about 10% from the EKF. When the error threshold is 0.315m, the Model's satisfaction rate reaches 90%, while the EKF needs to relax the threshold to 0.957m to achieve the same probability. This figure strongly demonstrates that the method of this invention not only has a lower average error but also a better overall error distribution, significantly improving reliability.

[0092] Figure 9The position error distribution of the model of this invention and the EKF method in the X, Y, and Z coordinate axes is shown. Box plots visually illustrate the median, quartiles, and dispersion of the errors of the two methods. It is clearly visible that in the X and Y directions (horizontal plane), the error box of the Model (red) is significantly smaller than that of the EKF (green), and the distribution is more concentrated. Specifically, the median and upper limit of the Model's error are much smaller than those of the EKF. For example, in the Y direction, the maximum error of the Model is approximately 0.3m, while the maximum error of the EKF exceeds 1.2m. This demonstrates that the method of this invention has a certain advantage in horizontal positioning accuracy. In the Z direction (height), both errors are relatively small, but the error distribution of the Model is still more compact than that of the EKF, further demonstrating the stability of the method of this invention.

[0093] To conduct a more in-depth comparative analysis, this invention also designed an ablation experiment, constructing three comparative models: the model of this invention (denoted as Model A), the model with GPS correction removed (denoted as Model B), and the model with the attention mechanism removed (denoted as Model C). All models were trained on the same training set and evaluated on the same validation set, with evaluation metrics including relative pose error and absolute position error.

[0094] Figure 10 The figures show the relative pose error curves of the three models as a function of training epochs. Model A (red) shows the fastest convergence and eventually stabilizes at the lowest level (approximately 0.006 to 0.061 radians), indicating that its predicted relative pose is the most accurate and stable. Model B (green) has a higher error throughout the training, with a maximum error of 0.081 radians, and its curve fluctuates more significantly. This demonstrates that GPS correction is crucial for learning accurate and smooth relative pose changes. In contrast, the error curve of Model C (blue) falls between the two, because the multi-head attention temporal modeling module allows the model to focus on the moments in the historical sequence that are most relevant to the current rotation estimate.

[0095] Figure 11A comparison of the overall 3D absolute position errors of the three models shows that Model A (red) has the lowest 3D RMSE value, demonstrating excellent accuracy and stability. Model B (green) has the largest error, proving the core role of the GPS intelligent correction module. When GPS signals are available, this module effectively utilizes absolute coordinate information to constrain position estimation and prevent cumulative drift; when GPS signals are unreliable, it smoothly switches to a vision-inertial dominant mode. Removing this module causes the model to lose its ability to utilize global absolute references and dynamically evaluate, resulting in severe position estimation bias. Model C (blue) has a 3D RMSE between Model A and Model B, but higher than Model A, indicating that the attention mechanism indirectly improves the accuracy and robustness of position estimation by enhancing temporal modeling capabilities.

[0096] Figure 12 The 3D error is further decomposed into three coordinate axis components (X, Y, and Z) for analysis. Figure 12 In the diagram (a), which represents the X-axis direction, it can be seen that Model A has the lowest and most stable X-axis RMSE, while Model B has the largest X-axis error. This further confirms the crucial role of GPS correction in preventing horizontal divergence (especially along the X-axis). Model C has a higher error than Model A. Figure 12 In (b), the Y-axis direction shows a similar trend to the X-axis. Model A is the best, followed by Model B, and Model C is the worst. This indicates that the attention mechanism has a positive impact on the smoothness of the Y-axis trajectory. Figure 12 In the Z-axis direction, (c) represents the height estimation. The error comparison of the three models is more obvious. Model A still maintains a significant advantage, with its Z-axis RMSE remaining stable at a very low level. Model B is slightly better than Model A, but Model C has greater fluctuations and the worst performance. This indicates that the model designed in this invention brings a significant improvement in height estimation.

[0097] Based on the results of the ablation experiment, we can conclude that: (1) The GPS correction module is the decisive component to ensure the high accuracy and robustness of the model in absolute position. Its absence will directly lead to a serious deviation of the position estimation from the true value; (2) The attention temporal modeling module makes an important contribution to improving the overall performance of the model, especially the accuracy of attitude estimation and the stability of position estimation; (3) The complete model (model A) designed in this invention is superior to its variant in all aspects of position and attitude, which verifies the rationality and effectiveness of the overall architecture design.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fusion-sensing self-localization method, characterized in that, The method includes: Step S1: Acquire and synchronize the multimodal sensor data of the UAV, including continuous image pairs, inertial measurement unit (IMU) sequences and GPS raw coordinates, and calculate the corresponding absolute position true value and relative attitude true value; Step S2: Construct a fusion sensing neural network model. The model receives the multimodal sensor data sequence, performs feature fusion on continuous images, IMU sequences and GPS position coordinates, and generates the visual-inertial position estimate, position correction amount and relative attitude estimate relative to the previous moment at the current moment. Step S3: Design a coordinated pose loss function to supervise both the final absolute position estimation and relative pose estimation during model training, thereby achieving joint optimization of the two tasks of absolute position and relative pose. Step S4: Introduce a GPS validity detection and dynamic fusion module into the fusion perception model to evaluate the reliability of the current GPS signal and adaptively fuse the visual-inertial prediction results with the GPS measurement values ​​to generate the final absolute position output. Step S5: Train the fusion sensing neural network model using the dataset obtained in step S1, and verify the model's performance in absolute positioning accuracy and relative pose consistency through evaluation metrics.

2. The fusion sensing self-localization method according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Extract image timestamps, IMU time series, GPS time series, and ground truth pose time series from the UAV's multimodal sensor data source; Step S12: Based on the image timestamp, select adjacent IMU, GPS and ground truth samples before and after each image timestamp, and use linear interpolation for triaxial acceleration, triaxial angular velocity, GPS position and ground truth position, and use spherical linear interpolation for ground truth attitude quaternions to synchronize each modal data to the image frame rate. Step S13: Perform coordinate transformation on the synchronized GPS latitude, longitude and altitude coordinates. First, convert the longitude, latitude and altitude in the WGS-84 coordinate system to geocentric and geofixed coordinates. Then, construct a rotation matrix based on the latitude and longitude of the starting point and transform it to the local northeast ENU coordinate system with the starting point as the origin to obtain the GPS position coordinates. Step S14: Based on the true value of the absolute attitude quaternion from the previous time step after synchronization. The truth value of the absolute attitude quaternion at the current moment Calculate the true values ​​of the inter-frame relative attitude quaternions: in, This represents the truth value of the relative attitude quaternion between the current time step and the previous time step. This represents quaternion multiplication. It represents the inverse of a quaternion.

3. The fusion sensing self-localization method according to claim 1, characterized in that, The fusion perception neural network model includes a motion perception visual feature extractor, an IMU feature construction unit, a feature fusion layer, a temporal modeling module, a dual-task regression head, and a GPS validity detection and dynamic fusion module connected to the absolute position regression path.

4. The fusion sensing self-localization method according to claim 3, characterized in that, Step S2 specifically includes: Step S21: Motion-aware visual feature extractor, input is two consecutive RGB images. and The 6-channel image tensor obtained by stitching along the channel dimension The motion-aware visual feature extractor first extracts the visual features through a shared or corresponding convolutional encoder. and Spatial feature extraction is performed to obtain the image features of adjacent frames. Then, the features of adjacent frames are subtracted to obtain inter-frame variation features. The current frame image features are then concatenated with the inter-frame variation features. The concatenated features are further processed through convolutional layers, pooling layers, and fully connected layers to finally output a visual feature vector containing motion change information and scene structure information. ; Step S22: The IMU feature construction unit extracts the IMU sampling sequence between the timestamps of two adjacent image frames, and performs normalization and time window statistics on the three-axis acceleration and three-axis angular velocity, respectively, to obtain the IMU feature vector. ; Step S23: The feature fusion layer combines the visual feature vectors IMU feature vectors and GPS location coordinates The concatenation process yields a joint feature vector, which is then subjected to dimensionality reduction and fusion using a multilayer perceptron consisting of fully connected layers, activation functions, and dropout layers to obtain fused features. ; Step S24: The time series modeling module receives the fused features from L consecutive time steps. The sequence is encoded using a Long Short-Term Memory (LSTM) network to obtain the hidden states at each time step. A gated recurrent unit further recursively updates the hidden states. Then, the final hidden state is used as the query vector, and the hidden states at each historical time step are used as the key and value vectors. Scaling dot product attention weights are calculated, and finally, the historical hidden states are weighted and summed to obtain the contextual features at the final time step. ; Step S25: The dual-task regression head includes an absolute position generation path and a relative pose regression head. The absolute position generation path uses contextual features. As input, output is a visual-inertial position estimate. and position correction amount and will and The data is then passed to the GPS validity detection and dynamic fusion module, which generates the final absolute position estimate. Relative pose regression head with contextual features As input, a four-dimensional vector is regressed through a fully connected layer, and after normalization, the relative pose quaternion estimate is obtained. .

5. The fusion sensing self-localization method according to claim 4, characterized in that, The coordinated pose loss function Loss due to absolute position and relative attitude loss The weighted composition is expressed as follows: in, and To balance the weighting coefficients of the two losses, The absolute position at the current moment is the true value. The true value of the relative attitude quaternion calculated in step S14. The final absolute position estimate is output from step S4. This is the relative attitude quaternion estimate output from step S25.

6. The fusion sensing self-localization method according to claim 5, characterized in that, Absolute position loss Using the Huber loss function, it can be expressed as: Where N is the sample size. Indicates the first training samples, Indicates the first The absolute position prediction error of each sample. Here, δ is the Huber loss function, and δ is the threshold parameter for the Huber loss.

7. The fusion sensing self-localization method according to claim 5, characterized in that, Relative attitude loss is based on relative attitude quaternion estimation relative attitude truth quaternion To construct the angle error between the two, first calculate the angle error between them, then apply the Smooth L1 loss function to the angle error, specifically as follows: Where N is the sample size. Indicates the first There are training samples, where · represents the quaternion dot product and |·| represents the absolute value. The threshold parameter for Smooth L1 loss.

8. The fusion sensing self-localization method according to claim 5, characterized in that, Step S4 specifically includes: Step S41: GPS validity check, obtain the current GPS location coordinates. Input a neural network, and the network output is mapped to a validity score by a sigmoid activation function. ; Step S42: Dynamic weight generation, incorporating contextual features and GPS location coordinates The data is then concatenated again and used to generate a 3D fusion weight vector through another neural network. ; Step S43: Adaptive fusion based on validity score and 3D fusion weight vector Calculate the final absolute position estimate : in, This represents the final absolute position estimate. Indicates GPS location coordinates, This represents visual-inertial position estimation. ⊙ indicates positional correction, and ⊙ indicates element-wise multiplication.

9. The fusion sensing self-localization method according to claim 1, characterized in that, The evaluation indicators in step S5 include absolute position error, relative attitude error and trajectory consistency. The absolute position error includes the overall root mean square error RMSE and the root mean square errors of the three coordinate axis components RMSE_X, RMSE_Y and RMSE_Z. The relative attitude error is the relative rotation error RRE and the trajectory consistency is the relative position error RPE.

Images