Aircraft complex terrain dynamic path planning method and system based on reinforcement learning

Abstract

The invention discloses an aircraft complex terrain dynamic path planning method and system based on reinforcement learning, and the method comprises the steps: S1, carrying out the multi-source heterogeneous perception fusion, and generating a state vector through the multi-source perception fusion; s2, element reinforcement learning intelligent decision making: dynamically adjusting the weight of a reward function by utilizing an element reinforcement learning agent and outputting an action instruction; s3, a safety control instruction is generated, model prediction control is adopted to conduct rolling optimization on the instruction under the condition that dynamics and safety constraints are met, and a final control instruction is generated; and S4, online learning and strategy updating: storing interaction experience through an online learning mechanism, preferentially sampling high-complexity scene data, and finely adjusting decision network parameters. According to the method, the dynamic adaptive path planning of the aircraft in the complex terrain is realized, the defects of poor adaptability and dependence on prior data in the prior art are overcome, and the autonomy and safety of path planning are improved.

Description

Technical Field

[0001] This invention relates to the field of flight control technology for low-altitude aircraft, and in particular to a method and system for dynamic path planning of aircraft in complex terrain based on reinforcement learning. Background Technology

[0002] Low-altitude logistics aircraft, when performing last-mile delivery and inter-regional transportation tasks, often need to autonomously fly over complex terrains such as urban canyons, mountains, hills, and dense forests. These environments are highly unstructured and dynamic, containing not only static obstacles but also numerous unpredictable dynamic obstacles, accompanied by complex wind fields and airflow disturbances. When performing path planning in complex terrain environments, low-altitude logistics aircraft must ensure real-time performance, safety, and adaptability. Existing path planning methods, such as those based on preset rules or global search algorithms, typically have the following limitations: 1. Existing path planning methods, such as the A* algorithm (also known as the A-star algorithm, which is one of the more popular heuristic search algorithms and is widely used in the field of path optimization) and Dijkstra's algorithm, rely on pre-generated global maps. The parameters and strategies of these methods are often statically preset and cannot be dynamically adjusted according to the complex terrain perceived in real time (such as undulating mountains and dense buildings). This leads to a significant decrease in planning ability in unknown or unstructured environments, and they are prone to getting trapped in local optima or planning failure.

[0003] 2. In traditional reinforcement learning path planning methods, the sub-objective weights of the reward function are mostly preset manually and remain unchanged. They cannot be dynamically adjusted according to terrain characteristics, resulting in poor adaptability in complex and ever-changing terrains and difficulty in balancing the needs of multiple objectives.

[0004] 3. Existing methods lack online learning and evolution capabilities. Their performance is fixed at deployment, making it impossible to learn from actual flight experience and continuously optimize to adapt to the unique environmental characteristics of specific operational airspaces, thus limiting their reliability and efficiency in long-term missions. Summary of the Invention

[0005] The purpose of this invention is to solve the above-mentioned problems by proposing a dynamic path planning method and system for aircraft in complex terrain based on reinforcement learning.

[0006] To achieve the above objectives, the following technical solution was adopted: A method and system for dynamic path planning of aircraft in complex terrain based on reinforcement learning. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain specifically includes the following steps: S1: Multi-source heterogeneous sensing fusion; Environmental information is acquired using airborne sensors. A deep learning network based on an attention mechanism is used to dynamically evaluate the confidence weights of the airborne sensors. The environmental information is then fused based on the confidence weights to generate a feature map containing obstacle and terrain information. Based on the feature map, a state vector is generated. ; The system employs one or more of LiDAR, visual sensors, and GPS / IMU; the environmental information includes one or more of 3D laser point clouds, visual images, positioning information, or attitude.

[0007] Specifically, the attention-based deep learning network dynamically evaluates the confidence weights by calculating the normalized function values ​​of the reliability indices of each sensor. The specific calculation formula is as follows: in, This is a reliability indicator for lidar. This is a reliability indicator for vision sensors. The reliability index for GPS / IMU is τ=0.3, used to balance weight differences.

[0008] Specifically, the steps for generating feature maps are as follows: Environmental information collected by LiDAR, visual sensors, and GPS / IMU is transformed into feature information using feature extraction algorithms, and then, under the same spatial reference, feature maps are generated based on confidence weights. The weighted fusion formula is as follows: in, This represents the feature map generated after fusion; , representing the confidence weights of airborne sensors; This refers to the feature information obtained by using feature extraction algorithms based on environmental information collected by airborne sensors.

[0009] Based on the feature map, it is encoded to generate a state vector. The state vector It should include at least the aircraft's own status, local terrain features, dynamic obstacle information, and mission context. Among them, local terrain features should include at least the average slope, surface roughness, and obstacle density.

[0010] Specifically, the encoding steps include: extracting local terrain features from the feature map, standardizing the aircraft's own state, dynamic obstacle information, mission context, and local terrain features to the [0,1] interval, and concatenating them sequentially to form a state vector. .

[0011] S2: Meta-reinforcement learning intelligent decision-making; Includes the following sub-steps: S201: Generate weight vector; The weight vector , For path efficiency weights, For terrain-adaptive weights, For safety distance weighting, To optimize energy consumption weights, Weights for cargo stability; S202: Dynamic adaptation of the reward function; Based on the weight vector Construct and adjust reward function preferences based on the state vector. and weight vector It outputs a preliminary action command through a pre-trained deep neural network. .

[0012] Among them, the action command This includes at least one of the following: the expected rate of change of heading angle, climb angle, velocity increment, yaw angle, or acceleration vector.

[0013] Specifically, the expression for the reward function is: in, For path rewards, As a reward for terrain safety, As an obstacle avoidance reward, As an energy consumption reward, Rewards are given for the condition of the goods.

[0014] By changing the weight vector The reward function is adjusted by varying the weights of different components. Preferences.

[0015] Specifically, in addition to the common Path rewards Obstacle avoidance rewards and Energy consumption reward, this invention adds Terrain safety rewards and Cargo integrity bonus, including the terrain safety bonus. Configured such that its value decreases as terrain complexity increases; the cargo integrity reward It is configured such that when the acceleration or angular velocity of the aircraft cargo exceeds a preset safety threshold, the value decreases as the degree of exceedance increases.

[0016] S3: Generation of security control commands; Includes the following sub-steps: S301: Obtain the action command output by S202 Parse action instructions It transforms action commands into a continuous state sequence, generating a reference trajectory within a preset future time domain. The reference trajectory This includes the aircraft's desired position sequence, velocity sequence, and attitude sequence.

[0017] The reference trajectory Represented as: in, The expected state of itself at time t+k is predicted for the current time t, including the expected position, expected velocity, and expected attitude angle.

[0018] The value range of the future preset time domain is 1 to 2 seconds, the prediction step size is 5 to 10 steps, and the time interval between each step is 0.2 to 0.5 seconds.

[0019] S302: Invoke a pre-set simplified dynamic model that reflects the dynamic characteristics of the aircraft, based on the current state vector. Based on the aircraft's own state and a simplified dynamic model, the system state is predicted within a predetermined time domain, resulting in a predicted state sequence. .

[0020] The predicted state sequence Represented as: in, This represents the self-state at time t+k predicted based on a simplified dynamics model.

[0021] S303: To "make the predicted state sequence" Tracking reference trajectory "To optimize the objective, under preset constraints, the optimal control sequence U for the next N steps is calculated by solving a quadratic programming problem; the constraints include physical limit constraints and safety distance constraints."

[0022] S304: Extract the control command u corresponding to the current time t from the optimal control sequence U, and input the control command u to the aircraft actuators. After entering the next control cycle, based on the new state vector and the new action command... Repeat steps S301 to S304 to achieve scrolling optimization.

[0023] Specifically, the implementation of the rolling optimization includes: the optimal control sequence U contains control commands u for multiple future steps; the control command u at the current time t is extracted and input to the aircraft's actuators; control commands u at other times are discarded; and the optimal control sequence U is represented as: Where N is the prediction step size, and t is the current time. These are candidate control instructions for the future time t+k.

[0024] S4: Online learning and strategy updates; The interaction experience of each step of the aircraft is stored in a pre-established experience database. High-reward experiences in high terrain complexity scenarios are sampled first. The parameters of the deep neural network and meta-learner are fine-tuned through reinforcement learning algorithms to optimize the decision-making strategy.

[0025] Specifically, the expression for the interactive experience is ( , , , ),in, This is a state vector that includes local terrain features. For action commands, The result of the reward function calculation. To execute action instructions The new state vector is then obtained.

[0026] The formula for calculating the priority of the priority sampling is as follows: Wherein, λ is the balance coefficient, with a value ranging from 0.9 to 0.98; TD error is used to reflect the value deviation of current experience; State vector The local terrain complexity is calculated using the following formula: , , , These are the average slope, surface roughness, and obstacle density, respectively. The result is the calculation result of the reward function; the priority p of the priority sampling is determined by the TD error. With local terrain features A joint decision.

[0027] Specifically, after calculating the priority p of the priority sampling, the following steps are also performed: setting a reward value threshold. For base priority p that is higher than the reward threshold Interactive experiences are identified as high-value experiences and ultimately prioritized. ,in The reward bonus factor is a constant greater than 1.

[0028] This invention also discloses a system for implementing the above-mentioned reinforcement learning-based dynamic path planning method for aircraft in complex terrain. This system provides an execution platform for the method and specifically includes: Perception fusion module: This module is used to execute step S1 of the above method. It is equipped with airborne sensors such as LiDAR, vision camera, and GPS / IMU to collect multi-source heterogeneous data. It dynamically adjusts the weights based on the sensor reliability index and fuses the data to generate a feature map.

[0029] State Construction Module: This module executes step S1 of the above method, connects to the perception fusion module, and extracts local terrain features from the feature map. It then integrates and encodes these features with the UAV's own state, dynamic obstacle information, and task context, outputting a comprehensive state vector. ; Meta-reinforcement learning decision module: This module executes step S2 of the above method, employs an Actor-Critic architecture, and includes a built-in meta-learner and connection state construction module. The meta-learner is used to... The weight vector of the reward function generated from local terrain features. The Actor network is used for... and Adjusted reward preferences, outputting high-level action commands ; The secure execution module executes step S3 of the above method, connects to the meta-reinforcement learning decision module, and integrates the MPC algorithm for parsing. Generate a reference trajectory, predict the state sequence, solve for the optimal control sequence, and output the low-level control command u at the current moment. Disturbances are offset through rolling optimization. Online learning module: Used to execute step S4 of the above method, it is connected to the state construction module, the meta-reinforcement learning decision module and the secure execution module respectively. It is used to store interaction experience, prioritize sampling high-value experience, and fine-tune the parameters of the meta-reinforcement learning decision module through reinforcement learning algorithm.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention employs an attention-based deep learning network to dynamically adjust the confidence weights of multi-source sensors. This allows for real-time response to reliability fluctuations in LiDAR, visual sensors, and GPS / IMU under complex terrain, avoiding misjudgments of environmental features caused by existing fixed-weight fusion methods. It further improves the accuracy of environmental feature extraction, reduces obstacle localization errors, and enhances the accuracy of terrain feature extraction, providing a reliable environmental modeling foundation for subsequent decision-making.

[0031] 2. This invention introduces terrain safety and cargo integrity bonuses into the reward function, so that the aircraft's path planning is no longer limited to avoiding obstacles, but can also consider the optimal path that balances safety and cargo transport stability in addition to conventional flight efficiency and energy consumption.

[0032] 3. This invention uses a meta-learner to dynamically generate function weight vectors based on local terrain features in the state vector. It can adjust reward preferences according to the core needs of different terrains such as mountains and canyons, and incorporates multi-dimensional objectives such as cargo stability and energy consumption optimization. This effectively overcomes the fundamental defects of path planning methods based on fixed rules or parameters, which are poorly adaptable to unknown, unstructured, and complex terrains and are prone to getting trapped in local optima.

[0033] 4. This invention utilizes a model predictive control framework, first predicting future states and verifying command feasibility based on a simplified dynamic model, then embedding constraints, and responding to environmental disturbances in real time through rolling optimization, ensuring that the control commands finally output to the aircraft actuators are smooth, stable, and physically absolutely feasible, avoiding the safety risk of instability caused by control commands exceeding the aircraft's limits.

[0034] 5. This invention prioritizes the sampling of interactive experience based on the dual indicators of "TD error + terrain complexity", so that high-value experience in complex terrain is sampled first, avoiding the dilution of complex terrain experience caused by random sampling, and enabling the aircraft to quickly adapt to complex terrain. Attached Figure Description

[0035] Figure 1 This is a flowchart of the reinforcement learning-based dynamic path planning method for aircraft in complex terrain according to the present invention. Figure 2 This is a flowchart of the S2 sub-step of the reinforcement learning-based dynamic path planning method for aircraft in complex terrain according to the present invention. Figure 3 This is a flowchart of the S3 sub-step of the reinforcement learning-based dynamic path planning method for aircraft in complex terrain according to the present invention. Detailed Implementation

[0036] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0037] A method and system for dynamic path planning of aircraft in complex terrain based on reinforcement learning. In this embodiment, the present invention provides a dynamic path planning system for aircraft in complex terrain based on reinforcement learning, which specifically includes a multi-dimensional perception module, a meta-reinforcement learning decision module, a safety control module, and an online learning module.

[0038] Perception fusion module: Equipped with airborne sensors such as LiDAR, vision camera, GPS / IMU, etc., to collect multi-source heterogeneous data, dynamically adjust weights based on sensor reliability indicators, and fuse them to generate feature maps.

[0039] State Construction Module: This module executes step S1 of the above method, connects to the perception fusion module, and extracts local terrain features from the feature map. It then integrates and encodes these features with the UAV's own state, dynamic obstacle information, and task context, outputting a comprehensive state vector. .

[0040] The meta-reinforcement learning decision module executes step S2 of the above method. It employs an Actor-Critic architecture, includes a built-in meta-learner, and a connection state construction module. The meta-learner is used to... The weight vector of the reward function generated from local terrain features. Actor networks are used for... and Adjusted reward preferences, outputting high-level action commands .

[0041] The secure execution module executes step S3 of the above method, connects to the meta-reinforcement learning decision module, and integrates the MPC algorithm for parsing. Generate a reference trajectory, predict the state sequence, solve for the optimal control sequence, and output the low-level control command u at the current moment. Disturbances are offset through rolling optimization.

[0042] Online learning module: Used to execute step S4 of the above method, it is connected to the state construction module, the meta-reinforcement learning decision module and the secure execution module respectively. It is used to store interaction experience, prioritize sampling high-value experience, and fine-tune the parameters of the meta-reinforcement learning decision module through reinforcement learning algorithm.

[0043] like Figure 1 As shown, the reinforcement learning-based dynamic path planning method for aircraft in complex terrain includes the following steps: S1: Multi-source heterogeneous sensing fusion; In this embodiment, the low-altitude aircraft is a drone. The drone is equipped with airborne sensors such as lidar, visual sensors, and GPS / IMU. The airborne sensors are used to synchronously collect external environmental information, which includes, but is not limited to, 3D laser point clouds, visual images, positioning information, or the drone's own attitude.

[0044] First, the collected environmental information is sent to the perception fusion module, which uses a deep learning network based on an attention mechanism to dynamically evaluate the confidence weights of the airborne sensors.

[0045] In this embodiment, the deep learning network uses a convolutional neural network (CNN) for feature extraction. In addition, depending on the type of sensor data, it can be replaced with a visual Transformer, PointNet, or a CNN-LSTM hybrid network. All of the above networks can provide support for the weight calculation of the attention mechanism through feature extraction. The specific network can be selected according to the terrain complexity and onboard computing power requirements.

[0046] Specifically, environmental information is transformed into high-dimensional features using a convolutional neural network (CNN). Then, the normalized exponential function value of the reliability index for each sensor is calculated. Finally, the confidence weights of the airborne sensors are dynamically evaluated according to a confidence weight formula. The confidence weight allocation formula is as follows: in, This is a reliability indicator for lidar. This is a reliability indicator for vision sensors. The reliability index for GPS / IMU is τ=0.3, used to balance weight differences.

[0047] Furthermore, the reliability index of each sensor is jointly determined by the sensor's intrinsic quality index and the consistency index with data from other sensors: Reliability indicators of lidar The density of the point cloud is determined by counting the number of valid points. Simultaneously, the stability of the point cloud data is evaluated by registering the current frame's point cloud with the previous frame's point cloud and calculating the registration error. Reliability indicators of vision sensors The texture richness and blurriness of a convex shape are quantified by calculating its average gradient. Simultaneously, the temporal consistency of the visual data is verified by comparing the inter-frame uniqueness calculated by visual odometry with the uniqueness measured by the inertial measurement unit (IMU). GPS reliability indicators The signal reception quality is evaluated by calculating the signal-to-noise ratio (SNR) of the GPS receiver and the number of satellites in the data set. The consistency of positioning is assessed by comparing the instantaneous positioning results of GPS with the short-term trajectory extrapolation results based on the inertial navigation system (INS). If the deviation between the two is too large, it indicates that the GPS data may be inaccurate. in, , , These are the weighting coefficients for each item, and their specific values ​​can be calibrated according to the actual sensor performance and scenario requirements. All intermediate calculation results are normalized to the interval [0, 1].

[0048] Convolutional neural networks employing attention mechanisms can dynamically adapt to fluctuations in sensor reliability. For example, when visual overexposure occurs, their weights will be appropriately reduced to avoid misjudgment of features with fixed weights.

[0049] Subsequently, environmental information is fused based on confidence weights to generate a feature map containing obstacle and terrain information.

[0050] This fusion process is achieved through the following steps: First, the environmental information collected by various airborne sensors is converted into feature information suitable for path decision-making. For lidar point clouds, rasterization and geometric calculations can be used to convert them into feature information representing terrain and obstacles; for visual images, visual perception technologies such as semantic segmentation can be used to convert them into feature information representing obstacles; while GPS / IMU data mainly provides reference feature information for spatial alignment.

[0051] Specifically, the feature information obtained by converting environmental information collected by LiDAR, vision sensors, and GPS / IMU is then analyzed under the same spatial reference based on confidence weights. The weighted fusion formula is as follows: in, This represents the feature map generated after fusion; , representing the confidence weights of airborne sensors; This refers to the feature information obtained by using feature extraction algorithms based on environmental information collected by airborne sensors.

[0052] Finally, the state construction module obtains the feature map and extracts local terrain features from it, including at least the average slope, surface roughness, and obstacle density. The average slope is obtained by processing lidar point cloud data, the surface roughness is obtained by calculating the standard deviation of the local grid, and the obstacle density is obtained by counting the number of obstacle grids per unit volume. All of these can be obtained using existing technologies.

[0053] The state construction module standardizes the aircraft's own state, dynamic obstacle information, mission context, and local terrain features to the [0,1] interval, and then concatenates them in the order of "own state → terrain features → obstacle information → mission context" to form a state vector. Constructing the state vector The goal is to create a standardized data structure that contains all the necessary decision-making information. This allows the meta-reinforcement learning decision module to focus on these high-level features instead of directly processing raw, high-dimensional perceptual data, greatly reducing the dimensionality of the decision space and improving computational efficiency and decision quality.

[0054] Wherein, the state vector It is a multidimensional vector i, specifically composed of the following terms concatenated: The spacecraft's state is represented by a 9-dimensional vector, comprising its three-dimensional position: \left [ {x, y, z} \right ] 3D velocity \left [ {{v}_{x},{v}_{y},{v}_{y}} \right ] And the three-axis attitude angles \left [ {roll, pitch, yaw} \right ] .

[0055] Local terrain features are M-dimensional vectors, which are predefined network hyperparameters whose dimensions are determined by the structure of the convolutional neural network used for feature extraction.

[0056] The dynamic obstacle information is a K×P matrix, where K is the maximum number of obstacles considered, and each obstacle is described by P-dimensional data, specifically including its relative position: \left [ {\Delta x, \Delta y, \Delta z} \right ] Relative velocities: [{{\Delta v}_{x},\Delta {v}_{y},\Delta {v}_{y}} ] And the type of obstacle.

[0057] The task context is a 4-dimensional vector.

[0058] Furthermore, the mission context specifically includes, but is not limited to, the mission destination location, the remaining battery power of the UAV, the mission urgency, and the type of cargo. This information is typically obtained in real time through the UAV's flight management system or by ground control station via data link communication. The specific sensing, measurement, and communication processes involved are common knowledge in the field and are not the innovation of this invention; therefore, they will not be elaborated upon in this specification. The core of this invention lies in how to utilize this mission context information, combined with real-time environmental perception, to achieve dynamic and adaptive path decision-making through a meta-reinforcement learning mechanism.

[0059] S2: Meta-reinforcement learning intelligent decision-making; like Figure 2 As shown, it includes the following sub-steps: S201: Generate weight vector; The state vector obtained in step S1 The input is fed into the meta-learner built into the meta-reinforcement learning decision module, and the meta-learner extracts the state vector. The local terrain features are analyzed, and a weight vector is generated based on these features. By obtaining the weight vector This enables dynamic adaptation of the reward function, thereby achieving context-aware and refined decision-making.

[0060] Furthermore, the meta-learner is essentially a parameterized data processing model. It undergoes offline co-training with the Actor and Critic networks in the Actor-Critic architecture within a simulation environment library containing various complex terrains such as mountains, canyons, and plains, using a meta-reinforcement learning framework. Its specific task is to perform numerous training rounds. In each round, the aircraft performs path planning from a random starting point to a random target point within a randomly sampled specific terrain. During this process, at each decision step, the meta-learner generates and outputs a weight vector adapted to the current terrain based on real-time perceived local terrain features. The training objective is to enable the meta-learner to generate weight vectors based on terrain features, thereby maximizing the agent's long-term cumulative reward. Specifically, it aims to ensure the Actor network, guided by the Critic network, achieves the highest possible value evaluation. The convergence criterion is that training is considered complete when the system's average task success rate and cumulative reward on the validation terrain set stabilize. Through this training process, the meta-learner ultimately acquires the core capability of adaptively generating weight vectors based on real-time terrain features.

[0061] Weight vector , For path efficiency weights, For terrain-adaptive weights, For safety distance weighting, To optimize energy consumption weights, The weights are the stability weights for the goods, and each weight satisfies the following conditions: .

[0062] S202: Dynamic adaptation of the reward function; Based on weight vector The pre-trained Actor network constructs and adjusts reward function preferences based on the state vector. and weight vector Output a preliminary action command. .

[0063] Specifically, the expression for the reward function is: in, For path rewards, The negative sign indicates that the closer you are to the target point, the higher the reward. Location of the drone. The target location.

[0064] As an obstacle avoidance reward, , Location of the drone. Location of the obstacle. =5m, the farther the obstacle is, the higher the reward.

[0065] As an energy consumption reward, , The motor speed is 0.1. The lower the speed, the higher the reward. 0.1 is a coefficient used to scale the reward level.

[0066] This invention also introduces and : As a reward for terrain safety, , For slope, This represents terrain roughness; the negative sign indicates slope. Lower roughness results in a higher reward. 0.5 and 0.3 are empirical coefficients used to balance the influence of the two features. Terrain safety reward. It is configured such that its value decreases as terrain complexity increases. It can provide continuous gradient penalties, rather than binary collision signals. This allows the aircraft to actively prefer flat areas during planning, achieving "bypassing" of terrain rather than "emergency collision avoidance," resulting in smoother and safer generated paths.

[0067] Rewards for the integrity of the goods. , , These are the preset acceleration threshold and angular velocity threshold, respectively. , / s, , These represent the cargo's acceleration and angular velocity, respectively. The reward decreases when either acceleration or angular velocity exceeds a threshold. Coefficients of 0.8 and 0.4 are used to emphasize the importance of cargo stability. Cargo integrity reward. It is configured such that when the acceleration or angular velocity of the aircraft cargo exceeds a preset safety threshold, the value decreases as the degree of exceedance increases. It can directly punish harmful accelerations and integrate the special needs of logistics transportation directly into the decision-making objectives, making the planned route not only safe but also commercially valuable, which is something that general path planning algorithms do not have.

[0068] In this embodiment, the Actor-Critic architecture used in the meta-reinforcement learning decision module is a complete decision-making and evaluation system. The Actor network employs the PPO algorithm and is responsible for decision-making and evaluation based on state vectors. Directly output high-level action commands The Critic network, on the other hand, learns a value function to evaluate the long-term value of the Actor network's decisions under specific states and reward preferences. The common optimization objective of both the Actor and Critic networks is to maximize the total reward function. By training on a large amount of simulation experience, the initial decision-making strategy is ensured to have basic obstacle avoidance and path planning capabilities. Finally, by leveraging the powerful sequential decision-making capabilities of reinforcement learning, an implicit and efficient global planning strategy is learned in a complex high-dimensional state space, rather than just performing local and reactive obstacle avoidance.

[0069] Specifically, the Actor network will use the state vector With the current weight vector Adjusted reward preferences are combined to output action commands. ,in, Let α be the desired rate of change of heading angle, and α be the desired climb angle. This represents the desired speed increment.

[0070] The Actor network, as a pre-trained deep neural network, functions to map from the state space to the action space. Through an offline co-training process, the optimized decision policy is parameterized and stored in its network weights. During online execution, the Actor network receives the state vector. With weight vector As input, it directly outputs a preliminary action command through its internal forward propagation calculation. Those skilled in the art will understand that using deep neural networks and policy functions to generate control commands based on state inputs is a well-known technique in the field of reinforcement learning. The innovation of this invention lies in providing dynamic, terrain-context-dependent reward preference guidance for the decision-making process through a meta-learner, rather than the specific implementation of the decision network itself.

[0071] S3: Generation of security control commands; like Figure 3 As shown, it includes the following sub-steps: S301: Obtain the action command output by S202 Parse action instructions , will action command Transform into a continuous state sequence to generate a reference trajectory in the future preset time domain. Reference trajectory This includes the aircraft's desired position sequence, velocity sequence, and attitude sequence.

[0072] In this embodiment, since a time domain that is too long can easily lead to prediction errors, and a time domain that is too short cannot cope with sudden obstacles, the future preset time domain is set to 1.5 seconds, the prediction step size N is set to 7 steps, and the time interval between each step is set to 0.2 seconds.

[0073] Specifically, reference trajectory Represented as:

[0074] in, The expected state of itself at time t+k is predicted for the current time t. The expected state includes the expected position, expected velocity, and expected attitude angle.

[0075] S302: Invoke a pre-set simplified dynamic model that reflects the dynamic characteristics of the aircraft, based on the current state vector. Based on the aircraft's own state and a simplified dynamic model, the system state is predicted within a predetermined time domain, resulting in a predicted state sequence. .

[0076] Specifically, in step S1 The simplified dynamic model includes the current actual state input, and the predicted state sequence for the next 7 steps is calculated using the Euler integral method (step size 0.2 seconds). Predicting state sequence Represented as: in, This represents the self-state at time t+k predicted based on a simplified dynamic model. It is necessary to ensure that each step... and Consistent dimensions are used for subsequent trajectory tracking optimization.

[0077] S303: The safe execution module uses the MPC algorithm to "predict the state sequence". Tracking reference trajectory "To optimize the objective, under the preset constraints, the optimal control sequence U for the next N=7 steps is calculated by solving a quadratic programming problem."

[0078] The quadratic programming solver used is the OSQP lightweight quadratic programming solver, and the objective function is expressed as: The first term is the trajectory tracking error term. It is a predicted state. Q represents the desired state in the reference trajectory, and Q is a weighted matrix, which includes weights for position, velocity, and attitude angles that can be set differently.

[0079] The second item is the smoothing control item. R is the control command for the k-th step, and R is a diagonal matrix used to constrain the fluctuation range of the control command.

[0080] The third item is a penalty for controlling the rate of change. It is the amount of change in the control command, that is , It is the control command from the previous moment, and L is a diagonal matrix used to control the rate of change of the penalty control.

[0081] N is the prediction step size, and in this case, N=7; Weight matrix used to optimize the objective function , , .

[0082] Furthermore, in this embodiment, the present invention uses tracking the reference trajectory and minimizing the control quantity as optimization objectives, and introduces a control change rate penalty term as a third optimization objective. This is sufficient to prove that the optimization objectives of the MPC of the present invention can be dynamically added or deleted according to the task performance indicators. The optimization objectives that can be introduced include, but are not limited to, comfort optimization, energy consumption optimization, or perceived quality optimization, and other optimization objectives.

[0083] Specifically, the constraints include physical limit constraints and safety distance constraints. The physical limit constraints are: attitude angle ≤ 30°, three-dimensional acceleration ≤ 5m / s². 2The motor speed is 1000~6000rpm; the safety distance constraint is: the distance between the UAV position and the obstacle at all times in the predicted state sequence is ≥5m.

[0084] Furthermore, in this embodiment, the model prediction control module of the present invention uses physical limit constraints and safety distance constraints as the main constraints to clearly illustrate the core working principle of the present invention. The specific constraints considered are not limited to physical limit constraints and safety distance constraints. Those skilled in the art will understand that, depending on specific task requirements and application scenarios, other types of constraints can be flexibly introduced, including but not limited to: comfort constraints related to cargo integrity (such as acceleration limits), communication delay constraints under communication link quality, and line-of-sight constraints to ensure effective sensor perception.

[0085] By introducing constraints, MPC can generate a safe, feasible, and smooth control sequence that is closest to the instruction, regardless of the instruction output by the upper-level reinforcement learning, while satisfying the constraints of aircraft dynamics and obstacle safety distance. This fundamentally solves the safety risk problem faced by directly applying reinforcement learning to physical systems.

[0086] Furthermore, to address the constraint conflicts that may arise from multiple dynamic obstacles, the MPC algorithm of this invention can set the critical safety distance constraint as a soft constraint. That is, a larger penalty weight is added to the constraint violation in the optimization objective, so that when there is no solution that strictly satisfies all constraints, the quadratic programming solver can automatically and slightly violate the constraints and achieve safe avoidance at minimal cost, thereby always being able to compute a feasible and safe control sequence U.

[0087] Furthermore, this invention includes a pre-defined degradation strategy to address situations where the optimization problem is unsolvable in extreme cases. As a preferred emergency measure, the specific steps for switching to the backup PID controller and performing a hover are as follows: Trigger: The quadratic programming solver is triggered to indicate a downgrade after failing to find a feasible solution for several consecutive cycles. Switching: The safety execution module cuts off action instructions from the meta-reinforcement learning decision module. The input is immediately switched to another pre-packaged, independently backed-up PID controller; Execution: The PID controller takes the current state of the aircraft as input, ignores or disregards environmental obstacles, and takes maintaining the current flight altitude and zero speed as the control objective to generate and execute hovering control commands; Alarm and Replanning: While switching the PID controller, an emergency status signal is sent to the meta-reinforcement learning decision module and the online learning module, triggering the aircraft to replan its strategy based on the new safety status.

[0088] The aforementioned emergency measures ensure that the aircraft can quickly enter a stable and safe hovering state when control decisions fail, creating favorable conditions for system fault diagnosis or manual intervention. Furthermore, in addition to the hovering strategy described above, those skilled in the art can also employ other known emergency measures based on actual mission requirements, such as reverting along a historical safe trajectory or executing previously feasible control commands. All safety backup solutions adopted based on the core ideas of this invention to address unsolvable control problems fall within the protection scope of this invention.

[0089] S304: Extract the control command u corresponding to the current time t from the optimal control sequence U, and input the control command u to the aircraft actuators. After entering the next control cycle, based on the new state vector... And new action instructions Repeat steps S301 to S304 to achieve scrolling optimization.

[0090] Specifically, for rolling optimization, the optimal control sequence U contains control commands u for the next 7 steps. The control command u at the current time t is extracted and input into the aircraft's actuators, while the remaining control commands u for future times are discarded. After entering the next control cycle (t+0.2s), the new state vector from step S1 is reacquired. Repeat steps S301~S303 to generate a new U and execute the current instruction.

[0091] The optimal control sequence U is represented as:

[0092] Where N is the prediction step size, and t is the current time. These are candidate control instructions for the future time t+k.

[0093] Furthermore, if the GPS / IMU detects gust disturbances, i.e. attitude angle fluctuations > 5°, it automatically shortens the preset time domain to 1 second, reduces the prediction step size N to 5 steps, and reduces the solution time to 25ms, thereby improving the disturbance response speed.

[0094] This invention chooses "rolling optimization" instead of a one-time solution in order to effectively overcome the problem of prediction error accumulation caused by model mismatch and environmental disturbances such as gusts of wind and obstacle movement by re-predicting and optimizing based on the latest real state in each control cycle. This ensures that the control system always makes decisions based on the most accurate information at present, and that the control commands are always safe and feasible, thus solving the rigidity problem of the existing "one-time planning to the end".

[0095] S4: Online learning and strategy updates; An experience database is constructed to store the interaction experience of the aircraft at each step. High-reward experiences in high terrain complexity scenarios are sampled first. The parameters of the Actor network and meta-learner are fine-tuned through reinforcement learning algorithms to optimize the decision-making strategy.

[0096] Specifically, the expression for interactive experience is ( , , , ),in, This is a state vector that includes local terrain features. For action commands, The result of the reward function calculation. To execute action instructions The new state vector is then obtained.

[0097] First, for newly collected or replayed interactive experiences, the priority is calculated based on the absolute value of their temporal differential error (TD error).

[0098] Specifically, the formula for calculating the priority of priority sampling is as follows: Wherein, λ is the balance coefficient, with a value of 0.95, which is used to balance the influence of TD error and terrain complexity; TD error is used to reflect the value deviation of current experience. The larger the value of experience, the greater the deviation, and the more it needs to be learned first. State vector The local terrain complexity is calculated using the following formula: , , , These are the average slope, surface roughness, and obstacle density, respectively. This is the result of the reward function calculation.

[0099] The priority p of the priority sampling is determined by the TD error. With local terrain features A joint decision.

[0100] Then, after calculating the priority p for priority sampling, a reward value threshold needs to be set to select high-reward experiences for priority sampling. The reward value threshold This can be a static or dynamic threshold. If set as a static threshold, it can be set as a multiple of the average of all experienced baseline priorities; if set as a dynamic threshold, it can be set as a fixed quantile of the baseline priorities in the experience pool. For baseline priorities p higher than the reward threshold... Interactive experiences are identified as high-value experiences and ultimately prioritized. ,in The reward bonus factor is a constant greater than 1.

[0101] Furthermore, the experience database has an experience pool management mechanism. When the capacity reaches the limit, a strategy of "first-in, first-out" combined with "low-priority elimination" is adopted. Low experience values ​​with priority p < 0.1 are eliminated first. If all current experiences in the experience database are ≥ 0.1, the earliest experience stored in the experience database is eliminated to ensure that the experience pool always retains high-value experiences.

[0102] While the aircraft is hovering or charging, the flight stability of the drone is ensured. At the same time, the PPO algorithm is used to fine-tune the deep neural network in S2 based on these experiences, and elastic weight consolidation technology is used to protect the existing knowledge.

[0103] The above are merely preferred examples of this application and are not intended to limit this application. Those skilled in the art may have other optimizations and additional functions. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A dynamic path planning method for aircraft in complex terrain based on reinforcement learning, characterized in that: Includes the following steps: S1: Multi-source heterogeneous sensing fusion; By utilizing airborne sensors to acquire information about the external environment, a deep learning network based on an attention mechanism is employed to dynamically evaluate the confidence weights of the airborne sensors. And based on confidence weights The environmental information is fused to generate a feature map that includes obstacle and terrain information; Based on the feature map, it is encoded to generate a state vector. The state vector It should include at least the aircraft's own status, local terrain features, dynamic obstacle information, and mission context; The airborne sensors are one or more of lidar, vision sensors, and GPS / IMU; the environmental information includes one or more of 3D laser point cloud data, visual image data, positioning information data, or attitude data. S2: Meta-reinforcement learning intelligent decision-making; Includes the following sub-steps: S201: Generate weight vector; The state vector obtained in step S1 Input a meta-learner, and dynamically generate weight vectors using the meta-learner. ; S202: Dynamic adaptation of the reward function; Based on the weight vector Construct and adjust reward function preferences based on the state vector. and weight vector It outputs a preliminary action command through a pre-trained deep neural network. ; Among them, the action command Including at least one of the following: the expected rate of change of heading angle, climb angle, velocity increment, yaw angle, or acceleration vector; S3: Generation of security control commands; Includes the following sub-steps: S301: Obtain the action command output by S202 Parse action instructions , will action command Transform into a continuous state sequence to generate a reference trajectory in the future preset time domain. The reference trajectory This includes the aircraft's desired position sequence, velocity sequence, and attitude sequence; S302: Invoke a pre-set simplified dynamic model that reflects the dynamic characteristics of the aircraft, based on the current state vector. Based on the aircraft's own state and a simplified dynamic model, the system state is predicted within a predetermined time domain, resulting in a predicted state sequence. ; S303: To make the predicted state sequence Tracking reference trajectory "To achieve the optimization objective, under pre-defined constraints, the optimal control sequence U for the next N steps is calculated by solving a quadratic programming problem; the constraints include physical limit constraints and safety distance constraints." S304: Extract the control command u corresponding to the current time t from the optimal control sequence U, and input the control command u to the aircraft actuator; Upon entering the next control cycle, based on the new state vector And new action instructions Repeat steps S301 to S304 to achieve scrolling optimization; S4: Online learning and strategy updates; The interaction experience of each step of the aircraft is stored in a pre-established experience database. High-reward experiences in high terrain complexity scenarios are sampled first. The parameters of the deep neural network and meta-learner are fine-tuned through reinforcement learning algorithms to optimize the decision-making strategy.

2. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain as described in claim 1, characterized in that: In step S1, the deep learning network based on the attention mechanism dynamically evaluates the confidence weights by calculating the normalized function values ​​of the reliability indicators of each sensor. The specific calculation formula is as follows: in, This is a reliability indicator for lidar. This is a reliability indicator for vision sensors. The reliability index for GPS / IMU is τ=0.3, used to balance weight differences.

3. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain as described in claim 1, characterized in that: In step S1, the environmental information collected by LiDAR, vision sensors, and GPS / IMU is transformed into feature information using a feature extraction algorithm, and then, under the same spatial reference, based on confidence weights... The weighted fusion formula is as follows: in, This represents the feature map generated after fusion; , representing the confidence weights of airborne sensors; This refers to the feature information obtained by using feature extraction algorithms based on environmental information collected by airborne sensors.

4. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain as described in claim 1, characterized in that: In step S1, the specific steps of the encoding are as follows: extracting local terrain features from the feature map, standardizing the aircraft's own state, dynamic obstacle information, mission context, and local terrain features to the [0,1] interval respectively, and concatenating them in a fixed order to form a state vector. Local terrain features include at least average slope, surface roughness, and obstacle density.

5. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain as described in claim 1, characterized in that: In step S2, the weight vector , For path efficiency weights, For terrain-adaptive weights, For safety distance weighting, To optimize energy consumption weights, Weights for cargo stability; Based on the weight vector The reward function can be adjusted, and the reward function is expressed as: in, For path rewards, As a reward for terrain safety, As an obstacle avoidance reward, As an energy consumption reward, Rewards are given for the condition of the goods.

6. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain as described in claim 5, characterized in that: Terrain safety reward Configured such that its value decreases as terrain complexity increases; the cargo integrity reward It is configured such that when the acceleration or angular velocity of the aircraft cargo exceeds a preset safety threshold, the value decreases as the degree of exceedance increases.

7. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain as described in claim 1, characterized in that: In step S3, the reference trajectory Represented as: in, The expected state of itself at time t+k is predicted for the current time t, including the expected position, expected velocity, and expected attitude angle; The predicted state sequence Represented as: in, This is the self-state at time t+k predicted based on a simplified dynamic model; The value range of the future preset time domain is 1 to 2 seconds, the prediction step size is 5 to 10 steps, and the time interval between each step is 0.2 to 0.5 seconds.

8. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain as described in claim 1, characterized in that: In step S304, the specific implementation of the rolling optimization includes: the optimal control sequence U contains control instructions u for multiple future steps; the control instruction u at the current time t is extracted and input to the aircraft's actuators; control instructions u at other times are discarded; the optimal control sequence U is represented as: Where N is the prediction step size, and t is the current time. These are candidate control instructions for the future time t+k.

9. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain as described in claim 1, characterized in that: In step S4, the expression for the interactive experience is ( , , , ),in, This is a state vector that includes local terrain features. For action commands, The result of the reward function calculation. To execute action instructions The new state vector is then obtained; The formula for calculating the priority of the priority sampling is as follows: Wherein, λ is the balance coefficient, with a value ranging from 0.9 to 0.98; TD error is used to reflect the value deviation of current experience; State vector The local terrain complexity is calculated using the following formula: , , , These are the average slope, surface roughness, and obstacle density, respectively. The result is the calculation result of the reward function; the priority p of the priority sampling is determined by the TD error. With local terrain features A joint decision.

10. The reinforcement learning-based dynamic path planning method for aircraft in complex terrain as described in claim 1, characterized in that: In step S4, after calculating the priority p of the priority sampling, the following step is also performed: setting a reward value threshold. For base priority p that is higher than the reward threshold Interactive experiences are identified as high-value experiences and ultimately prioritized. ,in The reward bonus factor is a constant greater than 1.

Images