A Dynamic Path Control Method for Unmanned Aerial Vehicles Based on Multi-Source Sensing Fusion

Abstract

This invention discloses a dynamic path control method for unmanned aerial vehicles (UAVs) based on multi-source perception fusion, relating to the field of UAV path control technology. The method includes the following steps: acquiring raw perception data; interpolating and resampling the data acquired by the visual perception unit to achieve time alignment; inputting the time-aligned multimodal perception data into a time-interpolated perception fusion network model; estimating the system delay from perception data acquisition to control command output, constructing a feedforward prediction model based on the current state, predicting the future state, and generating a delay-compensated control signal; inputting the delay-compensated control signal into the flight controller, calculating and outputting attitude, velocity, and acceleration commands to drive the UAV to perform path control tasks, and adaptively adjusting parameters based on flight control feedback information. This invention enhances the spatiotemporal accuracy of path estimation under high-dynamic maneuvering conditions.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) path control technology, specifically to a UAV dynamic path control method based on multi-source perception fusion. Background Technology

[0002] When drones perform highly dynamic maneuvers such as high-speed dives and sharp turns for evasion, their multimodal perception systems (such as vision, inertial measurement units, and lidar) exhibit a dual failure phenomenon in the time domain:

[0003] On the one hand, the timestamp drift of the sensors themselves causes synchronization errors in multi-source data before fusion; on the other hand, the fusion algorithm has overall system-level lag when responding to these highly dynamic actions (such as visual frame rate delay and inertial measurement unit data lag), resulting in additional response delay between perception and action.

[0004] This combined effect of synchronization drift and fusion lag can cause the path estimation to deviate significantly from the actual trajectory in time and space, leading to control oscillations or flight control system instability, which is particularly prominent in flight missions that require a rapid real-time perception-decision-control closed loop. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a dynamic path control method for unmanned aerial vehicles (UAVs) based on multi-source perception fusion.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a dynamic path control method for unmanned aerial vehicles (UAVs) based on multi-source perception fusion, comprising the following steps:

[0008] The multimodal sensing data synchronous acquisition step involves the inertial measurement unit, visual sensing unit, and lidar sensing unit simultaneously acquiring raw sensing data during the UAV's flight.

[0009] The time alignment step for sensing data involves using the time series of the inertial measurement unit as a unified main time axis to interpolate and resample the data collected by the visual sensing unit to achieve time alignment.

[0010] The perceptual feature fusion modeling step involves inputting time-aligned multimodal perceptual data into a time-interpolated perceptual fusion network model, extracting feature representations from the inertial measurement unit, visual perception unit, and lidar perception unit, and then performing feature fusion on a unified time axis.

[0011] The steps for generating a delay-compensated control signal are as follows: estimate the system delay from the acquisition of sensing data to the output of control commands, build a feedforward prediction model based on the current state, predict the future state, and then generate a delay-compensated control signal.

[0012] The dynamic path control execution steps involve inputting delay compensation control signals into the flight controller, calculating and outputting attitude, velocity, and acceleration commands to drive the UAV to perform path control tasks, and adaptively adjusting parameters based on flight control feedback information.

[0013] As a preferred embodiment of the present invention, the interpolation resampling of the data collected by the visual perception unit in the sensing data time alignment step includes:

[0014] The inertial measurement unit time series is set as the main time axis and used as the alignment reference for visual image frames;

[0015] Based on the original timestamps of the visual image frames and the time points of adjacent inertial measurement unit data, a time interpolation function is constructed to resample the visual image data;

[0016] Choose a linear interpolation, Lagrange interpolation, or spline interpolation strategy based on the sampling density;

[0017] Generate a sequence of visual perception unit features corresponding to the time nodes of the inertial measurement unit;

[0018] Output a sequence of visual perception unit feature data that is consistent with the timestamp of the inertial measurement unit.

[0019] As a preferred embodiment of the present invention, the nonlinear time mapping of the data collected by the lidar sensing unit in the sensing data time alignment step includes:

[0020] Extract the original timestamp sequences of the inertial measurement unit and the lidar sensing unit;

[0021] Construct a cost matrix to measure local temporal differences;

[0022] The minimum cost path is searched in the cost matrix using the dynamic time warping algorithm;

[0023] Output the nonlinear time mapping index between the inertial measurement unit and the lidar sensing unit;

[0024] The time index of the lidar sensing unit is adjusted according to the mapping index to achieve alignment with the main time axis of the inertial measurement unit.

[0025] As a preferred embodiment of the present invention, the perceptual feature fusion modeling step includes:

[0026] The time-aligned data from the inertial measurement unit, visual perception unit, and lidar perception unit are input into the time interpolation perception fusion network model.

[0027] Construct deep neural network structures for processing time series data, including multilayer perceptrons or Transformer networks;

[0028] A temporal continuity constraint mechanism is introduced to complete the missing features in low-frequency sampling modes;

[0029] Multimodal fusion feature output results are jointly generated on a unified time axis.

[0030] As a preferred embodiment of the present invention, the perceptual feature fusion modeling step further includes a main modality recognition process, the steps of which include:

[0031] The fused features are then fed into the modal analysis module;

[0032] Statistical analysis was performed on the residual values, information entropy, and related distribution characteristics of each sensing mode;

[0033] A modal confidence evaluation function is constructed based on residual values, information entropy, and related distribution characteristics;

[0034] Dynamically select the perceptual mode with the highest current confidence level as the dominant mode;

[0035] When the confidence level of the primary mode falls below the threshold, the system switches to the secondary mode and updates the fusion network structure.

[0036] As a preferred embodiment of the present invention, the step of generating the sensing delay compensation control signal includes:

[0037] Calculate real-time control signals based on current fused sensing features;

[0038] Input the current state sequence into the feedforward prediction model to predict the state values ​​at multiple future time points;

[0039] Estimate the total system delay from the acquisition of sensing data to the activation of control commands;

[0040] Calculate the delay compensation control signal by combining the current control quantity, the predicted state, and the estimated delay value;

[0041] The control signal is output to the flight control system to achieve path tracking control.

[0042] As a preferred embodiment of the present invention, the method for estimating system delay includes the following steps:

[0043] Calculate the error between the desired flight control output and the feedback signal;

[0044] Extract the current fused perception state vector;

[0045] An extended Kalman filter or a neural network-based function structure is used as the delay estimator;

[0046] The control error and the current state are input into the delay estimator to model the delay value;

[0047] The system delay estimation results are updated in each control cycle and fed back to the sensing delay compensation control signal generation step.

[0048] As a preferred embodiment of the present invention, the construction of the feedforward prediction model includes:

[0049] Collect fused sensing state data from the current moment and several historical moments to form a time series;

[0050] Based on this sequence, construct recurrent neural networks, gated recurrent units, or Transformer prediction structures;

[0051] Time series data are input into the prediction network to establish state transition relationships;

[0052] Output the state estimation results within a specified future time window;

[0053] The predicted state is used as a feedforward control variable in the generation step of the sensing delay compensation control signal.

[0054] As a preferred embodiment of the present invention, the method further includes a model self-calibration step, which includes:

[0055] The control deviation is obtained by comparing the output of the flight control system with the expected control quantity of the fusion sensing system.

[0056] Using the control deviation as input, a loss function is constructed to evaluate the error;

[0057] Calculate the gradient of the loss function with respect to the parameters of the time interpolation-aware fusion network model and the delay modeling module;

[0058] Update the model parameters according to the set learning rate;

[0059] When the error exceeds the set threshold, the model is updated and a closed-loop optimization mechanism is formed.

[0060] The beneficial effects of this invention are:

[0061] 1. In this invention, by setting the inertial measurement unit time series as the main time axis, interpolation and resampling are performed on the visual perception unit respectively, and the timestamp of the lidar perception unit is nonlinearly mapped using a dynamic time warping algorithm, thus achieving the unification of the three perception modes in the time domain. This perception data time alignment step effectively eliminates the synchronization error caused by sensor clock drift between perception sources, providing a unified time reference for the subsequent perception feature fusion modeling step. This design significantly reduces the accumulation of fusion error and enhances the spatiotemporal accuracy of path estimation under high dynamic maneuvering conditions.

[0062] 2. In this invention, a modal analysis module is used to statistically analyze the residual values, information entropy, and distribution characteristics of each sensing mode, establish a modal confidence evaluation function, and dynamically select the dominant mode. When the confidence of the dominant mode drops below a threshold, the system automatically switches to the suboptimal mode and updates the fusion network structure. This mechanism ensures that the fusion output remains usable even if a sensing mode experiences signal interruption, accuracy degradation, or other failures, effectively enhancing the system's sensing robustness in complex flight missions.

[0063] 3. In this invention, by combining the perception delay compensation control signal generation step with the control compensation execution module, the system can estimate the total system delay from perception data acquisition to control signal execution in real time, and predict the future state based on the feedforward prediction model, thereby generating a delay-compensated control signal. The perception delay compensation control signal generation step also combines the estimation of system delay with the construction of the feedforward prediction model, and adaptively updates the delay in each control cycle. This link-level response optimization significantly reduces control lag under high dynamic actions, enabling the UAV to maintain stable flight control performance in tasks such as high-speed dives and sharp obstacle avoidance. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0065] Figure 1 This is a schematic diagram illustrating the workflow of the UAV dynamic path control method of the present invention.

[0066] Figure 2 This is a schematic diagram comparing the actual trajectory of the aircraft of the present invention with the trajectory after delay compensation.

[0067] Figure 3 This is a schematic diagram comparing the trajectory after compensation and the trajectory without delay compensation according to the present invention. Detailed Implementation

[0068] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0069] Example 1:

[0070] like Figure 1 As shown, a dynamic path control method for unmanned aerial vehicles (UAVs) based on multi-source perception fusion includes the following steps:

[0071] The multimodal perception data synchronous acquisition step involves the inertial measurement unit (IMU), visual perception unit, and lidar perception unit synchronously acquiring raw perception data during the UAV's flight. The IMU acquires acceleration, angular velocity, and attitude angle data at a frequency typically of 200~1000Hz, providing high temporal accuracy. The visual perception unit acquires continuous image frames, primarily for obtaining forward or downward view image frames, at a frequency typically of 30~60Hz, for environmental understanding and attitude estimation. The lidar perception unit acquires 3D point cloud data of the environment and adds timestamp information to each set of data at a frequency typically of 10~20Hz.

[0072] The time alignment step for sensing data uses the time series of the inertial measurement unit (IMU) as a unified master time axis. Based on its high sampling rate and low jitter characteristics, it serves as an alignment reference. The data collected by the visual sensing unit is interpolated and resampled to achieve time alignment. Linear interpolation, Lagrange interpolation, or spline interpolation methods are used to match the image frame data with the time nodes of the IMU. Then, a dynamic time warping algorithm is used to map the timestamp of the LiDAR sensing unit to the time axis of the IMU, realizing the unification of the three sensing modes in the time domain. This algorithm searches for the optimal path by constructing a cost matrix, thereby mapping the original timestamp of the LiDAR sensing unit to the time axis of the IMU, achieving alignment under non-uniform sampling.

[0073] The perceptual feature fusion modeling step involves inputting time-aligned multimodal perceptual data into a time-interpolated perceptual fusion network model. Feature representations from the inertial measurement unit, visual perception unit, and lidar perception unit are extracted, and feature fusion is performed on a unified time axis. Deep neural network structures (such as MLP or Transformer) can be used to process different modal inputs. End-to-end feature fusion is achieved through temporal alignment. Shared encoders and modality-specific sub-channels can be designed in the network structure to balance modal differences and joint representation capabilities. During the fusion process, the current main information source is dynamically identified based on a confidence mechanism. By introducing a confidence mechanism, the reliability of different modalities is dynamically evaluated by analyzing indicators such as residuals and information entropy in the current scene. The current "dominant modality" is selected and used to dominate the fusion weights.

[0074] The step of generating a delay-compensated control signal involves estimating the system delay from the acquisition of sensing data to the output of control commands, constructing a feedforward prediction model based on the current state, predicting the future state, and then generating a delay-compensated control signal. In this step, the system-level delay existing in the sensing control link is considered, and a feedforward prediction and hysteresis compensation mechanism is designed to ensure the timeliness of the control signal.

[0075] The dynamic path control execution steps involve inputting delay compensation control signals into the flight controller, calculating and outputting attitude, velocity, and acceleration commands to drive the UAV to perform path control tasks, and adaptively adjusting parameters based on flight control feedback information.

[0076] This invention can be widely applied in the following typical scenarios: urban low-altitude logistics flight, high-speed drone racing flight, etc.

[0077] Furthermore, in the time alignment step of the perceived data, the interpolation resampling of the data collected by the visual perception unit includes:

[0078] The inertial measurement unit time series is set as the main time axis and used as the alignment reference for visual image frames;

[0079] Considering that inertial measurement units typically have high frequencies (generally above 100Hz) and stable sampling periods, as well as low transmission delay and system time drift, the timestamps of the data collected by the inertial measurement units are selected as the main time axis of the entire system. This main time axis serves as the reference for aligning all subsequent sensing modal data.

[0080] Based on the original timestamp of the visual image frame and the time points of adjacent inertial measurement unit data, a time interpolation function is constructed to resample the visual image data. Linear interpolation, Lagrange interpolation or spline interpolation strategy is selected according to the sampling density.

[0081] For visual perception units, which typically have a low sampling frequency, the timestamps corresponding to each frame may not match the main time axis of the inertial measurement unit (IMU). This results in time intervals between adjacent frames being much larger than the IMU's sampling period, making direct mapping impossible. Therefore, a time interpolation function needs to be constructed to remap the visual image data onto the IMU's time axis. The construction process of the interpolation function includes:

[0082] Original image frame timestamp extraction: Extract the timestamp sequence of consecutive image frames, denoted as {t}. (i)};

[0083] Inertial measurement unit time point selection: at every two consecutive visual frame time points t (i) With t (i+1) Between, select all inertial measurement unit time points {t} located within this interval. (j)IMU};

[0084] Interpolation strategy selection: Based on the time interval and system real-time requirements, select the interpolation method:

[0085] If the data changes linearly, linear interpolation can be used;

[0086] If the image frame has non-linear displacement characteristics, Lagrange interpolation can be used;

[0087] To improve fitting accuracy and smooth the curve, cubic spline interpolation can be used.

[0088] Generate a sequence of visual perception unit features corresponding to the time nodes of the inertial measurement unit;

[0089] After the interpolation function is constructed, the visual image frame data is resampled according to the time nodes of the main time axis of the inertial measurement unit using the selected interpolation strategy to obtain the "pseudo-synchronous" visual frame feature value sequence corresponding to each inertial measurement unit timestamp. This feature value sequence may include the pixel intensity distribution, edge extraction features, key point matching features, etc. of the original image frame, which are used as inputs in the subsequent feature fusion modeling steps.

[0090] In particular, to improve the robustness of the interpolation results, a smoothing factor or weighting term can be introduced during the interpolation process to suppress interpolation anomalies caused by image quality jitter or frame loss. At the same time, a mapping table between the original timestamp and the interpolation results needs to be established during the resampling process for traceability and system debugging.

[0091] The output is a sequence of visual perception unit feature data that is consistent with the timestamp of the inertial measurement unit. Finally, the output is a sequence of visual perception features that is completely consistent with the time axis of the inertial measurement unit. This sequence is time-synchronized with the original data such as acceleration and angular velocity of the inertial measurement unit, ensuring that each modal feature is under a consistent time reference in the subsequent time interpolation perception fusion network, thereby improving the fusion accuracy and temporal continuity.

[0092] Furthermore, in the sensing data time alignment step, the nonlinear time mapping of the data collected by the lidar sensing unit includes:

[0093] Extract the original timestamp sequences of the inertial measurement unit and the lidar sensing unit;

[0094] The inertial measurement unit timestamp sequence is as follows:

[0095] ;

[0096] The lidar timestamp sequence is as follows:

[0097] ;

[0098] Where N > M, the sampling frequency of the inertial measurement unit is much higher than that of the lidar.

[0099] Construct a cost matrix to measure local temporal differences;

[0100] A cost matrix is ​​constructed for the time series of the inertial measurement unit and the lidar:

[0101] ;

[0102] Where D represents a real matrix of size N×M, D i,j Indicates the timestamp t of the inertial measurement unit i IMU With lidar timestamp t i LIDAR The absolute values ​​of the time difference between them are shown below:

[0103] ;

[0104] This cost matrix reflects the temporal differences between the sensing modes at each time step and is used for subsequent matching path search.

[0105] The minimum cost path is searched in the cost matrix using the dynamic time warping algorithm;

[0106] Based on the aforementioned cost matrix, the Dynamic Time Warping (DTW) algorithm is used to find a minimum-cost path P={(i1,j1),(i2,j2),...,(i k ,j k The path satisfies the following constraints:

[0107] The path starts at (1,1) and ends at (N,M);

[0108] Path continuity: The step between adjacent nodes is (1,0), (0,1), or (1,1);

[0109] Minimize cumulative cost:

[0110] ;

[0111] DTW path search can achieve optimal matching between lidar sampling points and inertial measurement unit time nodes, adapting to nonlinear differences between different modal time series.

[0112] Output the nonlinear time mapping index between the inertial measurement unit and the lidar sensing unit;

[0113] Based on the matching path P obtained by the DTW algorithm, the lidar timestamp sequence {t jLiDAR Mapping this to the main time axis of the inertial measurement unit generates a set of nonlinear time mapping index tables:

[0114] ;

[0115] Each lidar data point corresponds to the optimal inertial measurement unit time node, achieving a unified modal time reference.

[0116] The time index of the lidar sensing unit is adjusted according to the mapping index to achieve alignment with the main time axis of the inertial measurement unit. Based on the mapping index table, the three-dimensional point cloud data sequence {P} of the lidar sensing unit is... j LiDAR Rearrange and interpolate to match the inertial measurement unit timeline sequence, outputting a lidar feature data sequence {P} that matches the inertial measurement unit timeline sequence. ij LiDAR This serves as the input for subsequent perceptual feature fusion modeling steps.

[0117] Furthermore, the perceptual feature fusion modeling step includes:

[0118] The time-aligned data from the inertial measurement unit, visual perception unit, and lidar perception unit are input into the time interpolation perception fusion network model.

[0119] Construct deep neural network structures for processing time series data, including multilayer perceptrons or Transformer networks;

[0120] A temporal continuity constraint mechanism is introduced to complete the missing features in low-frequency sampling modes;

[0121] Multimodal fusion feature output results are jointly generated on a unified time axis.

[0122] The temporal interpolation-aware fusion network model mainly includes a Transformer-based feature interpolation module and a modality confidence evaluation submodule. The interpolation module uses a standard multi-head self-attention mechanism to process the time-series input of three types of modal data (inertial measurement unit, vision, and LiDAR) on a unified time axis. Each modality uses an independent encoder for feature extraction, and an attention mechanism is used in the fusion layer to integrate cross-modal information. The modality confidence evaluation module calculates confidence scores based on the output residuals, entropy indices, and feature variability of each modality, and performs master mode selection and dynamic switching. The overall structure supports parallel inference and can dynamically adjust the information dominance for different modalities with missing or distorted information.

[0123] This step constructs a temporal interpolation-aware fusion network model to perform the following functions:

[0124] Intermodal time series interpolation and structure completion:

[0125] For modes with low sampling frequency or sampling gaps, a time series interpolation mechanism is introduced. The interpolation structure can be built based on Transformer, Gated Recurrent Neural Network (GRU) or Multilayer Perceptron (MLP). The interpolation model combines contextual time information and the state of adjacent modes to achieve reasonable estimation of missing features.

[0126] Generate a unified fusion feature representation:

[0127] By fusing feature representations from different modalities through a shared attention mechanism or parallel sub-networks, a structured feature vector is output, which has uniform dimensions, temporal consistency, and modal complementarity. The output is used as the input to the path control and decision-making module.

[0128] Introducing a temporal continuity constraint mechanism:

[0129] The interpolation results are smooth and interpretable by using a time-domain regularization term, avoiding drastic fluctuations in perception results caused by mode switching or frame loss. During training, the prediction stability is improved by minimizing the feature differences between time series.

[0130] The generated unified time feature output will be passed as input to the perception delay compensation control signal generation step to construct the feedforward control quantity. The modal confidence identification result can also be fed back to the time alignment step to adaptively adjust the interpolation weights for low confidence modes.

[0131] Furthermore, the perceptual feature fusion modeling step further includes a main modality recognition process, the steps of which include:

[0132] The fused features are then fed into the modal analysis module;

[0133] Statistical analysis was performed on the residual values, information entropy, and related distribution characteristics of each sensing mode;

[0134] A modal confidence evaluation function is constructed based on residual values, information entropy, and related distribution characteristics;

[0135] Dynamically select the perceptual mode with the highest current confidence level as the dominant mode;

[0136] When the confidence level of the primary mode falls below the threshold, the system switches to the secondary mode and updates the fusion network structure.

[0137] Specifically, for each perceptual modality i, its confidence function is defined as follows:

[0138] ;

[0139] Among them, En i (Entropy)i Re represents the entropy value of the prediction output of the i-th mode, reflecting the model uncertainty. i (Residual) i ) represents the residual between the modality-aware feature and the fusion reference model, Dr i (DropRate) i The expression represents the proportion of recent frame loss / unavailability for this modality, used to measure its stability. f is a trainable nonlinear mapping function, which can also be approximated using a weighted linear combination.

[0140] ;

[0141] Where α represents the entropy weight, and since entropy can effectively measure uncertainty, the default value of α is set to 0.4. β represents the residual weight, and since the residual can reflect the accuracy of observation, the default value of β is set to 0.4. γ represents the frame loss weight, and since frame loss is a minor but important factor, the default value of γ is set to 0.2.

[0142] Within each sensing cycle of the system, the dominant mode M is identified. t The steps are as follows:

[0143] ;

[0144] That is, select the mode with the highest current confidence as the dominant mode, and set a threshold θ. If the dominant mode confidence M t If <θ, then the mode switching mechanism is triggered:

[0145] ;

[0146] That is, select the mode with the highest confidence other than the current mode. Preferably, θ is set in the range of [0.6, 0.75] and fine-tuned according to the overall signal-to-noise ratio of the system and the actual use scenario. The higher θ is, the easier it is for the system to trigger mode switching (to be conservative).

[0147] Furthermore, such as Figures 2-3 As shown, the step of generating the sensing delay compensation control signal includes:

[0148] Calculate real-time control signals based on current fused sensing features;

[0149] The system first constructs a unified sensing feature vector based on time-aligned multimodal sensing data. This feature vector is output by the time-interpolated sensing fusion network model and represents the flight state x at the current time t. t This state input is fed into a conventional controller (such as a proportional-integral-derivative (PID) controller or a model predictive controller (MPC)) to generate an instantaneous control signal C(x) based on the current state. t ).

[0150] Input the current state sequence into the feedforward prediction model to predict the state values ​​at multiple future time points;

[0151] Considering the non-negligible system delay from the generation of the control signal to its application to the aircraft, the system needs to predict the state x after the delay window. t+Δ Therefore, a feedforward prediction network with temporal modeling capabilities, such as a recurrent neural network, is used for modeling and outputs state estimation results for a future period. The output P of the prediction module... feedforward (x t+Δ ) represents the delay compensation feedforward control signal component.

[0152] Estimate the total system delay from the acquisition of sensing data to the activation of control commands;

[0153] The system needs to detect the perception-decision-execution delay δ in the current link. t Real-time estimation can be performed using an extended Kalman filter (EKF) or a neural network estimator. Based on the error between the desired flight control output and sensor feedback, and combined with the current fused sensing state vector, the current delay value is dynamically modeled. The delay estimator structure takes the control error and sensing state as input and outputs the delay value δ. t It is continuously updated in each control cycle through a feedback mechanism.

[0154] Calculate the delay compensation control signal by combining the current control quantity, the predicted state, and the estimated delay value;

[0155] The immediate control signal C(x) of the current state t ), delay-compensated feedforward control signal component P feedforward (x t+Δ and output delay value δ t The signal is synthesized to generate the final control signal after delay compensation:

[0156] ;

[0157] Among them, L lag (δ t The lag error compensation term is used to offset the execution error caused by the delay. The lag compensation module can adopt a forward back-pushing method, combining historical control results and the current prediction state, to adjust the control command so that it can still maintain the target control performance in the lag interval.

[0158] The control signal is output to the flight control system to achieve path tracking control;

[0159] Final control signal u tAfter being calculated by the flight controller, attitude angle, velocity, and acceleration commands are generated for execution and applied to the UAV's motors or control surfaces. The flight control system will return execution feedback in each control cycle. This feedback is used to further correct the prediction model, delay estimator, and control gain, enabling the system to update adaptively.

[0160] Among them, by Figures 2-3 It is evident that by improving the control signal through perception delay compensation and applying the P-value of the prediction module's output, feedforward (x t+Δ () represents the delay compensation feedforward control signal component, which can significantly improve trajectory tracking accuracy.

[0161] Specifically, the horizontal axis in the figure represents the flight time, which ranges from 0 to 10 seconds, with 20 sampling points per second (a total of 200 data points). The horizontal axis shows the complete time history of the aircraft during high-speed flight, with each time point corresponding to a specific action during the flight (such as acceleration, turning, deceleration, etc.). The vertical axis represents the deviation between the actual position and the expected position (true trajectory) of the aircraft. A positive error indicates that the aircraft position is ahead of the expected position, while a negative error indicates that the aircraft position is behind the expected position. The error value directly reflects the accuracy of the control system.

[0162] Figure 2 The black curve in the figure represents the ideal flight path, which serves as a benchmark. The changes are most significant in the 2-4 second (turn) and 6-8 second (acceleration) phases. In the path estimation without delay compensation (dark gray dashed line), the maximum positive error (lead) reaches 8 cm at 2.5 seconds, the maximum negative error (lag) reaches 12 cm at 4 seconds, and the peak error reaches 10 cm at 7 seconds. The average error is 18.7 cm, which means that the error increases significantly in the region with large acceleration changes and exhibits a systematic deviation (overall higher than the actual trajectory).

[0163] Figure 3 After mid-delay compensation, the trajectory estimation closely matches the actual trajectory, with an average error of only 3.2cm, a maximum error of 6.8cm (81.9% lower than without compensation), and a standard deviation of 1.5cm (81.9% lower than without compensation). The most significant performance improvements are seen in the areas where the error decreases from 15cm to 2cm at 3-4 seconds (sharp turn) and from 12cm to 3cm at 7-8 seconds (acceleration phase).

[0164] In summary, by accurately modeling the system delay value (δ) tBy applying delay compensation to the feedforward control signal components, the path error was successfully reduced by 82.9%, and the maximum error was reduced by 81.9%. Experimental data verified the effectiveness of this technology in high-speed dynamic scenarios (150-250 km / h), especially in critical maneuvers such as sharp turns and acceleration / deceleration, where the trajectory overlap rate was increased to 96.7%, significantly improving the control accuracy and safety of the aircraft.

[0165] Furthermore, the method for estimating the system delay includes the following steps:

[0166] The error between the desired output of flight control and the feedback signal is calculated. This error information typically includes: attitude error (difference between pitch angle, roll angle, and yaw angle), velocity error (difference between target velocity and feedback velocity), and position error (Euclidean distance between the predicted trajectory and the current position).

[0167] Extract the current fused perception state vector, including the 3D position and attitude after fusion of inertial measurement unit, vision and lidar, current velocity and acceleration, and confidence index of each mode in the perception system (for weighting). This state input and error together constitute the input variables for modeling system delay.

[0168] An extended Kalman filter or a neural network-based function structure can be used as the delay estimator. Optionally, an extended Kalman filter can be selected and designed. The state variables include delay parameters. The control output residual is used as the observation and the delay estimate is calculated recursively. This can achieve robust modeling of small-scale nonlinear delay changes.

[0169] The control error and the current state are input into the delay estimator to calculate the delay value δ. t The delay value δ t The core parameters of the subsequent control compensation execution module are used in the feedforward prediction and compensation process of the control command, and the system delay estimation results are updated in each control cycle and fed back to the perception delay compensation control signal generation step.

[0170] Furthermore, the construction of the feedforward prediction model includes:

[0171] Collect fused sensing state data from the current moment and several historical moments to form a time series;

[0172] Based on this sequence, construct recurrent neural networks, gated recurrent units, or Transformer prediction structures;

[0173] Time series data are input into the prediction network to establish state transition relationships;

[0174] Output the state estimation results within a specified future time window;

[0175] The predicted state is used as a feedforward control variable in the generation step of the sensing delay compensation control signal.

[0176] Specifically, we first define the current time as t, and to predict the state value at the future time t+Δ, we need to construct a time series input based on historical data:

[0177] ;

[0178] Where N represents the length of the time window, the size of which should balance the expressive power of historical information with the computational burden of the model. Too small a value results in weak predictive ability, while too large a value introduces redundancy and increases latency. Generally, it is 5~20. t The fused sensing state vector at time t contains features from the inertial measurement unit, vision, and LiDAR fusion. All states are aligned with the main time axis of the inertial measurement unit to ensure time consistency.

[0179] Considering the limited edge computing capabilities of the UAV platform, the feedforward prediction module preferentially adopts a lightweight gated recurrent unit (GRU) structure for state modeling. Its computational expression is as follows:

[0180] ;

[0181] ;

[0182] Among them, h t This represents the hidden state of a GRU, with dimensions that can be set from 64 to 128, W. o b o The output layer weights and biases are given by T, which represents the final time step after the GRU expansion, and is equal to t, x t+Δ This represents the predicted future state.

[0183] To enhance feedforward compensation capabilities, the model can be designed to output state estimates for multiple future time steps at once:

[0184] ;

[0185] This structure enables more flexible delay compensation strategies (e.g., dynamically selecting the optimal prediction time point), with an output dimension of [Δ×d], where d is the dimension of a single state vector. In addition, the prediction model must undergo standardization and regularization to avoid numerical explosion during prediction.

[0186] Furthermore, the method further includes a model self-calibration step, which includes:

[0187] The control deviation is obtained by comparing the output of the flight control system with the expected control quantity of the fusion sensing system.

[0188] Using the control deviation as input, a loss function is constructed to evaluate the error;

[0189] Calculate the gradient of the loss function with respect to the parameters of the time interpolation-aware fusion network model and the delay modeling module;

[0190] Update the model parameters according to the set learning rate;

[0191] When the error exceeds the set threshold, the model is updated and a closed-loop optimization mechanism is formed.

[0192] Specifically, during the model self-calibration process, the system constructs a loss function L to measure the difference between the current fused control output and the actual flight control feedback, as shown in the following formula:

[0193] ;

[0194] Where N represents the dimension of the control command (such as 3D attitude + thrust). The predictive control signal output by the perception fusion module, u t For the actual flight control system feedback output, when L > ϵ (where ϵ ranges from [0.05, 0.5]), the system will trigger online updates of the parameters of the fusion model and delay modeling module, using the following gradient descent formula:

[0195] ;

[0196] Where, η∈

[10] -5 10 -2 ] represents the learning rate, θ includes the parameter set of the fusion network model and the latency modeling module, ∇ θ L represents the gradient with respect to the loss function, expressed as:

[0197] ;

[0198] This process can be triggered once every 10 to 20 frames to ensure that the operating load of the embedded platform is controllable and improve the stability of the closed-loop control of the overall system in highly dynamic task scenarios.

[0199] Furthermore, this method is deployed on an embedded flight control platform with edge AI inference capabilities. This platform has the following core features: it supports parallel acquisition of data from multiple sensors and high-frequency data processing; it has a tensor computation acceleration module (such as an NPU or GPU) to run deep learning models; it supports shared memory and real-time communication mechanisms to coordinate data flow between modules; it has a flight control output interface that can directly control attitude, speed, and trajectory commands; and it includes the following functional modules:

[0200] Time alignment processing module: performs interpolation alignment and resampling between the inertial measurement unit and the visual perception unit, and completes the nonlinear time mapping of the lidar perception unit through the dynamic time warping algorithm to achieve a unified modal time reference;

[0201] Perceptual fusion inference module: Based on the Transformer or multilayer perceptron structure, a temporal interpolation perceptual fusion network model is constructed to generate joint features and identify the main modality, supporting dynamic switching of the main information source;

[0202] Control compensation execution module: integrates delay modeling, feedforward prediction and sensing delay compensation control signal generation functions, and is used to output corrective control commands that take into account system delay;

[0203] Each module collaborates to process unified time series and fused feature data through a shared memory mechanism, ensuring the real-time performance and consistency of the overall system.

[0204] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic path control method for unmanned aerial vehicles (UAVs) based on multi-source perception fusion, characterized in that, Includes the following steps: The multimodal sensing data synchronous acquisition step involves the inertial measurement unit, visual sensing unit, and lidar sensing unit simultaneously acquiring raw sensing data during the UAV's flight. The sensing data time alignment step uses the time series of the inertial measurement unit as a unified main time axis, interpolates and resamples the data collected by the visual sensing unit to achieve time alignment, and then uses the dynamic time warping algorithm to map the timestamp of the lidar sensing unit to the time axis of the inertial measurement unit. The perceptual feature fusion modeling step involves inputting time-aligned multimodal perceptual data into a time-interpolated perceptual fusion network model, extracting feature representations from the inertial measurement unit, visual perception unit, and lidar perception unit, and then performing feature fusion on a unified time axis. The steps for generating a delay-compensated control signal are as follows: estimate the system delay from the acquisition of sensing data to the output of control commands, build a feedforward prediction model based on the current state, predict the future state, and then generate a delay-compensated control signal. The dynamic path control execution steps involve inputting delay compensation control signals into the flight controller, calculating and outputting attitude, velocity, and acceleration commands to drive the UAV to perform path control tasks, and adaptively adjusting parameters based on flight control feedback information.

2. The UAV dynamic path control method based on multi-source perception fusion according to claim 1, characterized in that, In the time alignment step of the perception data, the interpolation resampling of the data collected by the visual perception unit includes: The inertial measurement unit time series is set as the main time axis and used as the alignment reference for visual image frames; Based on the original timestamps of the visual image frames and the time points of adjacent inertial measurement unit data, a time interpolation function is constructed to resample the visual image data; Choose a linear interpolation, Lagrange interpolation, or spline interpolation strategy based on the sampling density; Generate a sequence of visual perception unit features corresponding to the time nodes of the inertial measurement unit; Output a sequence of visual perception unit feature data that is consistent with the timestamp of the inertial measurement unit.

3. The UAV dynamic path control method based on multi-source perception fusion according to claim 1, characterized in that, In the sensing data time alignment step, the nonlinear time mapping of the data collected by the lidar sensing unit includes: Extract the original timestamp sequences of the inertial measurement unit and the lidar sensing unit; Construct a cost matrix to measure local temporal differences; The minimum cost path is searched in the cost matrix using the dynamic time warping algorithm; Output the nonlinear time mapping index between the inertial measurement unit and the lidar sensing unit; The time index of the lidar sensing unit is adjusted according to the mapping index to achieve alignment with the main time axis of the inertial measurement unit.

4. The UAV dynamic path control method based on multi-source perception fusion according to claim 1, characterized in that, The perceptual feature fusion modeling steps include: The time-aligned data from the inertial measurement unit, visual perception unit, and lidar perception unit are input into the time interpolation perception fusion network model. Construct deep neural network structures for processing time series data, including multilayer perceptrons or Transformer networks; A temporal continuity constraint mechanism is introduced to complete the missing features in low-frequency sampling modes; Multimodal fusion feature output results are jointly generated on a unified time axis.

5. The UAV dynamic path control method based on multi-source perception fusion according to claim 1, characterized in that, The perceptual feature fusion modeling step further includes a main modality recognition process, the steps of which include: The fused features are then fed into the modal analysis module; Statistical analysis was performed on the residual values, information entropy, and related distribution characteristics of each sensing mode; A modal confidence evaluation function is constructed based on residual values, information entropy, and related distribution characteristics; Dynamically select the perceptual mode with the highest current confidence level as the dominant mode; When the confidence level of the primary mode falls below the threshold, the system switches to the secondary mode and updates the fusion network structure.

6. The UAV dynamic path control method based on multi-source perception fusion according to claim 1, characterized in that, The step of generating the sensing delay compensation control signal includes: Calculate real-time control signals based on current fused sensing features; Input the current state sequence into the feedforward prediction model to predict the state values ​​at multiple future time points; Estimate the total system delay from the acquisition of sensing data to the activation of control commands; Calculate the delay compensation control signal by combining the current control quantity, the predicted state, and the estimated delay value; The control signal is output to the flight control system to achieve path tracking control.

7. The UAV dynamic path control method based on multi-source perception fusion according to claim 1, characterized in that, The method for estimating system delay includes the following steps: Calculate the error between the desired flight control output and the feedback signal; Extract the current fused perception state vector; An extended Kalman filter or a neural network-based function structure is used as the delay estimator; The control error and the current state are input into the delay estimator to model the delay value; The system delay estimation results are updated in each control cycle and fed back to the sensing delay compensation control signal generation step.

8. A dynamic path control method for unmanned aerial vehicles based on multi-source perception fusion according to claim 6, characterized in that, The construction of the feedforward prediction model includes: Collect fused sensing state data from the current moment and several historical moments to form a time series; Based on this sequence, construct recurrent neural networks, gated recurrent units, or Transformer prediction structures; Time series data are input into the prediction network to establish state transition relationships; Output the state estimation results within a specified future time window; The predicted state is used as a feedforward control variable in the generation step of the sensing delay compensation control signal.

9. The UAV dynamic path control method based on multi-source perception fusion according to claim 1, characterized in that, The method further includes a model self-calibration step, which includes: The control deviation is obtained by comparing the output of the flight control system with the expected control quantity of the fusion sensing system. Using the control deviation as input, a loss function is constructed to evaluate the error; Calculate the gradient of the loss function with respect to the parameters of the time interpolation-aware fusion network model and the delay estimator; Update the model parameters according to the set learning rate; When the error exceeds the set threshold, the model is updated and a closed-loop optimization mechanism is formed.

Images