Autonomous Mission Path Planning Method for Unmanned Aerial Vehicles Based on LCAEG-TD3 Algorithm

Abstract

This invention relates to the field of path planning methods, specifically disclosing a path planning method for autonomous missions of unmanned aerial vehicles (UAVs) based on the LCAEG-TD3 algorithm. The method includes: S1. Constructing a single-aircraft virtual path planning scenario; S2. Determining the nonholonomic motion constraints and environmental range constraints in the path planning scenario for the firefighting aircraft, and building a detection model for the firefighting aircraft; S3. Constructing a reward function for the firefighting aircraft; the reward function is used to evaluate the execution of a certain action by the firefighting aircraft in a specific state and output a reward value; S4. Introducing the LCAEG-TD3 algorithm and combining it with the nonholonomic motion constraints, environmental range constraints, and the reward function to construct an initial path planning model for the firefighting aircraft; S5. Training the initial path planning model for the firefighting aircraft and obtaining interactive data during the training process. The technical solution of this invention can consider kinematic constraints, has high stability, and can also obtain a more real-time intelligent path planning strategy.

Description

Technical Field

[0001] This invention relates to the field of path planning methods, and in particular to a path planning method for autonomous missions of unmanned aerial vehicles based on the LCAEG-TD3 algorithm. Background Technology

[0002] In today's aviation field, especially when firefighting aircraft perform water-dropping firefighting missions, path planning plays a crucial role. The essence of path planning is to accurately explore an optimal or at least feasible flight path from the starting point to the target point within a planning space, given a series of evaluation indicators and various constraints. When complete and accurate environmental information on influencing factors is available, traditional one-time global planning methods have the potential to calculate the optimal or suboptimal path. For example, route planning modules have often planned firefighting flight paths for a single aircraft based on calculations of no-fly zones and the selected aircraft firefighting strategy. Single-aircraft mission path planning, as a classic mission scenario within the scope of aircraft mission execution, is the foundation for studying complex multi-aircraft collaborative missions.

[0003] However, path planning is essentially a typical optimization problem, and traditional algorithms exhibit numerous shortcomings when dealing with real-world environments characterized by vast state spaces and complex real-time decision-making. On one hand, the modeling difficulty increases exponentially. The numerous variables in real-world scenarios, such as terrain, weather, and dynamic changes in no-fly zones, make it difficult for traditional algorithms to construct accurate and practical models to guide firefighting aircraft. On the other hand, the transferability of agent interaction behavior is extremely poor. Path planning strategies trained in a specific scenario cannot be effectively transferred and applied once the environment changes slightly, such as sudden wind shifts or the addition of temporary no-fly zones, leading to a significant reduction in firefighting efficiency.

[0004] In recent years, reinforcement learning (RL), as a cutting-edge artificial intelligence technique, has been widely used to describe and solve problems involving agents learning optimal strategies through continuous interaction with their environment, thereby maximizing rewards or achieving specific goals. Building on this foundation, Shauharda Khadka et al. innovatively integrated evolutionary algorithms (EAs) with reinforcement learning, incorporating sparse rewards using fitness metrics and employing a population-based approach to explore diversity. This led to the development of evolutionary reinforcement learning (ERL), which significantly improves training efficiency and stability, bringing new hope to the solution of path planning problems.

[0005] However, it cannot be ignored that despite the significant progress made by the ERL framework, some problems still exist. In genetic algorithms, deep neural networks (DNNs) use simple genetic encoding. When traditional biologically inspired mutation operators are applied to these encodings, the stability is poor, which can lead to catastrophic forgetting of the features acquired by the network. When multiple agents interact with the environment for evaluation, the cost is too high, affecting the applicability.

[0006] Therefore, there is an urgent need for a path planning method for UAV autonomous missions based on the LCAEG-TD3 algorithm that can take into account kinematic constraints, has high stability, and can obtain more real-time intelligent path planning strategies. Summary of the Invention

[0007] This invention provides an autonomous mission path planning method for unmanned aerial vehicles based on the LCAEG-TD3 algorithm. It can take into account kinematic constraints, has high stability, and can obtain a more real-time intelligent path planning strategy.

[0008] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0009] The UAV autonomous mission path planning method based on the LCAEG-TD3 algorithm includes:

[0010] S1. Construct a standalone virtual path planning scenario;

[0011] S2. Determine the nonholonomic motion constraints and environmental range constraints in the path planning scenario of the firefighting aircraft, and build a detection model for the firefighting aircraft;

[0012] S3. Construct the reward function for the firefighting aircraft; the reward function is used to evaluate the performance of the firefighting aircraft in a specific state for a certain action and output the reward value.

[0013] S4. Introduce the LCAEG-TD3 algorithm and combine it with nonholonomic motion constraints, environmental range constraints, and reward functions to construct an initial path planning model for the firefighting aircraft;

[0014] S5. Train the initial path planning model of the firefighting aircraft, obtain the interaction data during the training process, and use the interaction data to continuously train and update the neural network of the firefighting aircraft. After the training is completed, the final model of single-aircraft water drop path planning is obtained.

[0015] S6. The final model of single-aircraft water drop path planning can output the path planning strategy of firefighting aircraft in real time.

[0016] Furthermore, the expression for the nonholonomic motion constraint condition of the firefighting aircraft in the firefighting water-dropping scenario in S2 is as follows:

[0017] ,

[0018] ,

[0019] ,

[0020] ,

[0021] ,

[0022] ,

[0023] In the above formula: A matrix representing the information of the firefighting aircraft at time t; and Let be the position coordinates of the firefighting aircraft at time t; Let be the speed of the firefighting aircraft at time t; Let be the azimuth angle of the firefighting aircraft at time t; and Let be the linear acceleration and angular acceleration of the firefighting aircraft at time t; This is for simulating step size; A matrix representing the information of firefighting aircraft at time t+1; and The coordinates of the firefighting aircraft at time t+1; The speed of the firefighting aircraft at time t+1; The azimuth of the firefighting aircraft at time t+1.

[0024] Furthermore, the expression for the environmental range constraint condition in S2 is:

[0025] ,

[0026] in, The minimum and maximum values ​​of the horizontal axis of the virtual environment; The minimum and maximum values ​​of the vertical axis of the virtual environment; Four points form a rectangular virtual environment in which the firefighting aircraft is generated and moves. This indicates the location of the firefighting aircraft; the mission is considered a failure if the firefighting aircraft flies out of any environmental boundary.

[0027] Furthermore, the firefighting aircraft detection model in S2 is as follows: since individual firefighting aircraft cannot obtain global state information, each individual firefighting aircraft is set to carry a detection radar payload, and the radar detection range is... The detection range is a 360° circular area around the fuselage; the radar can obtain the relative distance and angle between the aircraft and neighboring firefighting aircraft within the detection range, as well as the no-fly zone and the aircraft itself. This information will be used for the firefighting aircraft's decision learning.

[0028] When the radar carried by the firefighting aircraft detects a target, it is necessary to determine the threat level of the target relative to the firefighting aircraft. The threat level is measured by relative distance. When the detected target is less than [a certain distance] from the firefighting aircraft... At this point, it is considered that the target is very close to the firefighting aircraft, and the firefighting aircraft needs to take precautions to avoid a collision; when the detected target is less than... At this point, it is assumed that the firefighting aircraft has collided with the target and has crashed;

[0029] Furthermore, the reward function in S3 includes a long-term reward function and an immediate reward function; the long-term reward function is mainly for the reward or penalty given to the agent when it reaches some special state in the task; in the water drop mission path planning of the fire-fighting aircraft swarm, the long-term reward includes: fire-fighting aircraft arrival reward, fire-fighting aircraft going out of bounds reward, and fire-fighting aircraft crash reward.

[0030] Furthermore, the formula for the long-term reward function is as follows:

[0031] ,

[0032] in, Firefighting aircraft Long-term rewards; Firefighting aircraft The reward for arriving, when the firefighting plane Enter the designated water-dropping task area, that is... When the mission is completed, the firefighting aircraft will receive a certain amount of reward points. Firefighting aircraft The penalty for going out of bounds, when the firefighting plane Flying out of the simulated firefighting environment, that is or When this occurs, the firefighting aircraft is considered to have lost contact and crashed, and a certain penalty value is obtained; Firefighting aircraft The crash reward for the firefighting aircraft Flying into a no-fly zone poses a danger of detection targets entering the firefighting aircraft's range. At that time, it is considered that the firefighting aircraft has collided and crashed, and a certain penalty value is obtained.

[0033] Furthermore, the firefighting aircraft Instant reward function Defined as:

[0034] ,

[0035] ,

[0036] ,

[0037] in, Firefighting aircraft The instant pathfinding reward function, , These are weight parameters, and all are positive numbers. Firefighting aircraft Instant collision rewards. Firefighting aircraft The single-step time penalty value. In order to enable the firefighting aircraft to reach the water drop mission area as soon as possible, a certain penalty value is given for each decision step before the firefighting aircraft reaches the water drop mission area, so as to guide the firefighting aircraft to find the optimal strategy to reach the water drop mission area as soon as possible.

[0038] Furthermore, the training process of the initial path planning model in S5 includes:

[0039] At the beginning of each round, the position, speed, and direction of the firefighting aircraft are randomly initialized, and the positions and areas of the no-fly zone and the mission area are randomly initialized according to a specified number.

[0040] The firefighting aircraft selects its actions using an initial strategy.

[0041] Information on each step of the pursuit of the firefighting aircraft Saved for network training, among which This is the status information of the firefighting aircraft at this moment. The actions taken by the firefighting aircraft at this moment The observation information from the firefighting aircraft at this moment The bonus value for the firefighting aircraft at this moment. The status information of the firefighting aircraft at the next moment after it takes action;

[0042] go through After rounds of game training, the final water-dropping path planning model of the agent is obtained.

[0043] Furthermore, the training and updating process of the neural network in S5 is as follows:

[0044] The TD3 algorithm is built upon double Q-learning. By selecting the smaller of two evaluation functions, it limits the overestimation of Q-values ​​and uses a double-delay update strategy to reduce the error in each update, thereby further improving the algorithm's performance. The TD3 algorithm contains a total of six neural networks, including two policy neural networks (actor-critic type networks within the "actor-critic" framework, called the "actor_eval" policy estimation network and the "actor_target" policy realization network, respectively), and an additional pair of value neural networks compared to the DDPG algorithm, namely "critic" type networks (critic_eval value estimation network and criticic_target value realization network, respectively).

[0045] The TD3 algorithm uses two critic networks to evaluate the value of the actor network, and selects the Q value of the smaller evaluation network as the update parameter of the actor network.

[0046] Design a critic network-assisted evaluation mechanism in the TD3 algorithm. , Used as an auxiliary model; the fitness values ​​of actors in the genetic population are obtained by averaging the predicted Q values ​​of all states using information from the k latest state vectors extracted from the mixed experience pool.

[0047] Design candidate control strategies that combine real fitness and virtual fitness; assuming a population size of N, before evaluating the population in the real environment, generate a candidate population with M offspring, where M>N. After evaluation by an auxiliary mechanism, the top N individuals in fitness are presented in the real environment for interaction; in addition to the original population, MN additional individuals are generated by adding Gaussian noise to the RL agent.

[0048] The design incorporates a learning crossover operator. During crossover, the buffer is only partially inherited, and the crossover offspring fills its buffer with the most recent half of the experience from the genetic memory of both parents. The new agent's genetic code is filled with the latest experience from the genetic code of both parents. The child agent is trained through imitation learning to selectively imitate the actions taken by the parents in the newly created genetic code, and the behavior of the parents is incorporated into the offspring's policy. That is, the SGD algorithm is used to minimize L(c) and optimize the offspring's policy.

[0049] The policy population is updated using the above operators, and the TD3 network is trained using sample data until convergence, thus obtaining the optimal policy network.

[0050] Furthermore, the LCAEG-TD3 algorithm in S4 is as follows:

[0051] (1) Initialize n policy networks as a population, initialize the RL algorithm actor network, critic network, and GA algorithm parameters; initialize the genetic memory of the population individuals, i.e., the empty replay buffer; initialize the hybrid experience pool R;

[0052] (2) Initialize the state and fitness;

[0053] (3) Assess the population, adopt an auxiliary assessment mechanism, obtain individual fitness, and store the strategy network and environment interaction data into the individual's genetic code and replay buffer R;

[0054] (4) Rank the policy network according to fitness, and select the policy with the highest fitness as the elite policy. ;

[0055] (5) Based on the obtained fitness, use the improved learning crossover operator and mutation operator to perform crossover and mutation operations on the population to update the population;

[0056] (6) Train the RL network using sample data;

[0057] (7) If the termination condition is not met, go to (2) until convergence is achieved and the optimal policy network is obtained;

[0058] (8) After training, a water drop path planning model for firefighting aircraft is obtained for direct use:

[0059] After training, the final water-dropping path planning model for the firefighting aircraft is obtained. The state information of the mission area and some environmental conditions is input into the neural network of the model in real time to obtain the action selection of the firefighting aircraft and control the firefighting aircraft to complete the path planning task of the water-dropping mission area.

[0060] The principle and beneficial effects of this scheme are as follows: This invention considers nonholonomic constraints, velocity constraints, and acceleration constraints in the kinematics of firefighting aircraft. The LCAEG-TD3 algorithm provided by this invention is an improvement and development based on the TD3 algorithm. It combines evolutionary algorithms, introduces an evolutionary reinforcement learning framework, and proposes a learning-type crossover operator, enabling offspring to integrate parent features through training. To avoid excessive interaction costs, a critic network-assisted evaluation mechanism is proposed, and response control decisions are designed. By reasonably combining real fitness and virtual fitness, accuracy is improved, and the convergence speed of the model is increased. The trained water drop path planning model has good transferability and can be directly applied to water drop path planning tasks for firefighting aircraft in various scenarios and situations, with a high success rate for firefighting aircraft.

[0061] In the stage of constructing a single-machine virtual path planning scenario (S1), a basic framework is built for subsequent algorithm training and strategy formulation by simulating real fire fighting or mission environments, setting key elements such as no-fly zones and mission areas, and restoring complex and ever-changing real-world scenarios.

[0062] The nonholonomic motion constraints (S2) are determined based on the UAV's own physical characteristics and flight mechanics principles. These constraints precisely define the changes in the UAV's position, velocity, acceleration, and azimuth at each moment, ensuring stable and controllable flight attitude and preventing flight accidents caused by unreasonable motion commands. Environmental range constraints are also based on the defined mission area boundaries. If the UAV exceeds the preset rectangular virtual environment boundary, the mission is considered a failure, ensuring the UAV executes the mission along the planned path. The firefighting aircraft's detection model, utilizing its onboard omnidirectional radar payload and the principle of radar wave reflection, collects real-time relative distance and angle information of neighboring aircraft, no-fly zones, and other targets within a 360° circular area around the aircraft. This provides real-time data for decision-making and assesses the threat level of targets based on relative distance, allowing for proactive risk avoidance.

[0063] The reward function (S3) construction step is based on the principles of behavioral psychology and reinforcement learning incentive mechanisms. The long-term reward function provides positive or negative rewards for the drone to achieve key mission milestones (such as reaching the water drop area, avoiding going out of bounds, and staying away from no-fly zones), guiding the drone towards the optimal strategy. The immediate reward function focuses on the current decision-making, rewarding the progress of pathfinding to encourage rapid approach to the target, and penalizing potential collision risks. At the same time, a single-step time penalty is imposed if the target area is not reached at each decision step, prompting the drone to make optimal decisions as quickly as possible.

[0064] The LCAEG-TD3 algorithm (S4) is introduced, deeply integrating the advantages of evolutionary algorithms and reinforcement learning. Firstly, it uses the TD3 algorithm as its foundation. TD3 is based on the double-Q learning principle, utilizing two evaluation functions to select the smaller Q value to limit overestimation, and combining a double-delay update strategy to reduce update errors. Its two policy neural networks and two pairs of value neural networks work together to achieve accurate evaluation and update of the policy network. Simultaneously, a critic network-assisted evaluation mechanism is designed, extracting the latest state vector information from a mixed experience pool to average the predicted Q value and obtain the fitness value of the actors in the genetic population, simulating the process of environmental evaluation of population fitness in natural selection. A learning crossover operator simulates gene crossover and recombination in biological heredity, allowing offspring to inherit the latest experience from both parents. Through imitation learning, the offspring's strategy is optimized. The population is iteratively updated through the above operators, and trained to convergence using sample data to obtain the optimal policy network.

[0065] When training the initial path planning model (S5), the drone state and mission environment elements are randomly initialized at the beginning of each game to simulate the uncertainty of starting a real mission. An initial strategy is used to explore the action space. The interaction information of the chasing drone is recorded throughout the process and fed back to the neural network for training. Through multiple rounds of game training, the drone gradually learns the optimal path decision mode in a complex environment and finally outputs a real-time usable path planning strategy.

[0066] This invention fully considers the non-holonomic motion constraints of UAVs and strictly follows the laws of physical motion to plan paths, greatly improving flight safety and stability. It can ensure precise flight attitude adjustment regardless of complex terrain undulations or airflow changes. Closely conforming to environmental range constraints and real-time detection information, the UAV can dynamically perceive and adapt to environmental boundaries, no-fly zones, and the dynamics of neighboring aircraft, flexibly avoid obstacles, and adjust its path in real time.

[0067] The LCAEG-TD3 algorithm integrates evolutionary thinking with reinforcement learning feedback regulation, resulting in an exponential improvement in learning efficiency compared to traditional algorithms. Through population iteration and crossover mutation, it rapidly explores the policy space and uncovers potential high-quality paths. Dual-delay updates and a critic network-assisted evaluation effectively suppress Q-value overestimation, accelerate convergence, reduce training resource consumption, and achieve a more accurate and efficient path planning model faster.

[0068] The real-time feedback mechanism of the reward function enables the drone to respond instantly to environmental changes. Each decision is based on a comprehensive reward balance of multiple objectives such as pathfinding, obstacle avoidance, and timeliness. In rapidly changing scenarios (such as fire spread or the appearance of temporary obstacles), the drone quickly determines the optimal path, ensuring mission continuity and efficiency, and significantly improving the mission success rate.

[0069] The trained model exhibits excellent generalization ability across different task areas, no-fly zone layouts, and initial states. It can adapt to new scenarios without large-scale retraining, and the algorithm framework is easily extended to complex multi-drone collaborative and multi-task systems, providing solid technical support for complex drone swarm operations in the future.

[0070] In summary, this invention clearly constructs nonholonomic motion constraints for UAVs in path planning scenarios, precisely limiting the UAV's position coordinates, velocity magnitude, azimuth angle, linear acceleration, and angular acceleration at each moment. Through detailed constraint expressions, it ensures that the UAV's flight strictly follows the laws of physical motion, maintaining a stable flight attitude even in complex environments. This effectively avoids flight accidents caused by unreasonable motion commands, greatly improving flight safety and controllability.

[0071] The LCAEG-TD3 algorithm is based on the TD3 algorithm. It reduces the error of each update through a dual-delay update strategy and uses two critic networks to evaluate the actor network. By selecting a smaller Q-value as the update parameter, it effectively suppresses overestimation of the Q-value and improves the algorithm's stability. Furthermore, the design of the critic network-assisted evaluation mechanism and the learning crossover operator makes the policy population more stable during iterative updates, avoiding problems such as catastrophic feature forgetting that may occur in traditional algorithms, thus ensuring the stability and reliability of the network training process.

[0072] The carefully designed reward function, including both long-term and immediate reward functions, enables real-time evaluation of the drone's actions under different conditions, providing corresponding rewards or penalties. This allows the drone to quickly make optimal decisions and adjust its flight path based on real-time environmental information. Simultaneously, the drone's onboard radar acquires real-time environmental information, and combined with the algorithm's rapid learning and decision-making capabilities, it can quickly plan the best path in rapidly changing scenarios, achieving truly real-time intelligent path planning and significantly improving mission efficiency and success rate. Attached Figure Description

[0073] Figure 1 A flowchart illustrating an embodiment of an autonomous mission path planning method for unmanned aerial vehicles based on the LCAEG-TD3 algorithm;

[0074] Figure 2 This is a schematic diagram of a simulated scenario for an autonomous mission path planning method for unmanned aerial vehicles based on the LCAEG-TD3 algorithm.

[0075] Figure 3 A schematic diagram of aircraft motion modeling for an autonomous mission path planning method for unmanned aerial vehicles based on the LCAEG-TD3 algorithm;

[0076] Figure 4 A schematic diagram of an aircraft detection model for an autonomous mission path planning method for unmanned aerial vehicles based on the LCAEG-TD3 algorithm;

[0077] Figure 5 This is the random initial situation diagram of the present invention;

[0078] Figure 6 The flowchart of the LCAEG-TD3 algorithm provided by this invention;

[0079] Figure 7 The graph showing the change of some parameters of the neural network during training is provided by this invention.

[0080] Figure 8a The trajectory diagram for a no-fly zone with a limit of 6;

[0081] Figure 8b The trajectory diagram with the number of no-fly zones set to 10;

[0082] Figure 8c The trajectory diagram with the number of no-fly zones set to 15;

[0083] Figure 8d To remove the no-fly zone trajectory map;

[0084] Figure 9 A comparison chart showing the success rates of the LCAEG-TD3 algorithm and the TD3 algorithm provided in this invention for planning water droplet routes for firefighting aircraft. Detailed Implementation

[0085] The following detailed description illustrates the specific implementation method:

[0086] A method for autonomous mission path planning of unmanned aerial vehicles based on the LCAEG-TD3 algorithm (e.g., Figure 1 (as shown), including:

[0087] S1. Construct a standalone virtual path planning scenario;

[0088] S2. Determine the nonholonomic motion constraints and environmental range constraints in the path planning scenario of the firefighting aircraft, and build a detection model for the firefighting aircraft;

[0089] S3. Construct the reward function for the firefighting aircraft; the reward function is used to evaluate the performance of the firefighting aircraft in a specific state for a certain action and output the reward value.

[0090] S4. Introduce the LCAEG-TD3 algorithm and combine it with nonholonomic motion constraints, environmental range constraints, and reward functions to construct an initial path planning model for the firefighting aircraft;

[0091] S5. Train the initial path planning model of the firefighting aircraft, obtain the interaction data during the training process, and use the interaction data to continuously train and update the neural network of the firefighting aircraft. After the training is completed, the final model of single-aircraft water drop path planning is obtained.

[0092] S6. The final model of single-aircraft water drop path planning can output the path planning strategy of firefighting aircraft in real time.

[0093] The steps described above are described in detail below.

[0094] S1: Construct a virtual path planning scenario for firefighting aircraft. The scenario is as follows: Figure 2 As shown, the black firefighting aircraft is the one that performs water-dropping missions, the red area is the no-fly zone, which simulates dangerous environments such as burned areas, mountains, high-temperature areas, and areas with high carbon dioxide concentrations, and the green area is the mission area.

[0095] S2: Apply nonholonomic motion constraints and environmental range constraints to the firefighting aircraft. The motion modeling of the firefighting aircraft is as follows: Figure 3 As shown, via position coordinates speed magnitude and velocity direction To describe the movement of firefighting aircraft, the firefighting aircraft detection model is as follows: Figure 4 As shown. The motion expression is as follows:

[0096] ,

[0097] ,

[0098] ,

[0099] ,

[0100] ,

[0101] ,

[0102] In the above formula: A matrix representing the information of the firefighting aircraft at time t; and Let be the position coordinates of the firefighting aircraft at time t; Let be the speed of the firefighting aircraft at time t; Let be the azimuth angle of the firefighting aircraft at time t; and Let be the linear acceleration and angular acceleration of the firefighting aircraft at time t; This is for simulating step size; A matrix representing the information of firefighting aircraft at time t+1; and The coordinates of the firefighting aircraft at time t+1; The speed of the firefighting aircraft at time t+1; The azimuth of the firefighting aircraft at time t+1.

[0103] The expression for the environmental range constraint is:

[0104] ,

[0105] in, The minimum and maximum values ​​of the horizontal axis of the virtual environment; The minimum and maximum values ​​of the vertical axis of the virtual environment; Four points form a rectangular virtual environment in which the firefighting aircraft is generated and moves. This indicates the location of the firefighting aircraft; the mission is considered a failure if the firefighting aircraft flies out of any environmental boundary.

[0106] State space of firefighting aircraft This includes: the firefighting aircraft's own status information. Status information of the water-dropping mission area of ​​firefighting aircraft and the detection status information of firefighting aircraft :

[0107] ,

[0108] ,

[0109] ,

[0110] ,

[0111] in: and The coordinates of the firefighting aircraft's location; The speed of the firefighting aircraft; This refers to the azimuth angle of the firefighting aircraft. The relative distance between the firefighting aircraft and the water-dropping mission area; The relative angle between the firefighting aircraft's heading and the water-dropping mission area; Firefighting aircraft To detect the target The relative distance, ; For firefighting aircraft to detect targets The relative angle, The detection targets were the five no-fly zones closest to the firefighting aircraft.

[0112] When modeling firefighting aircraft, use linear acceleration. and angular acceleration Controlling firefighting aircraft The motion of the firefighting aircraft is controlled by two factors: the magnitude and direction of its velocity. Linear acceleration and angular acceleration are output as a single action pair, meaning the neural network output dimension is 1. The action space of the firefighting aircraft is then defined as follows:

[0113] .

[0114] S3: Construct the reward function for the firefighting aircraft; the reward function is used to evaluate the performance of the firefighting aircraft in a specific state for a certain action and output the reward value.

[0115] The reward functions are divided into two categories: long-term and immediate. Long-term reward functions primarily reward or penalize agents when they reach specific states during a task. In the task path planning of a firefighting aircraft swarm, long-term rewards include: firefighting aircraft arrival rewards, firefighting aircraft going out of bounds rewards, and firefighting aircraft crash rewards. The specific formulas are shown below:

[0116] ,

[0117] in, Firefighting aircraft Long-term rewards; Firefighting aircraft The reward for arriving, when the firefighting plane Enter the designated water-dropping task area, that is... When the firefighting aircraft completes its water drop path planning, it will receive a certain reward value. Firefighting aircraft The penalty for going out of bounds, when the firefighting plane Flying out of the simulated firefighting environment, that is or When this occurs, the firefighting aircraft is considered to have lost contact and crashed, and a certain penalty value is obtained; Firefighting aircraft The crash reward for the firefighting aircraft Flying into a no-fly zone poses a danger of detection targets entering the firefighting aircraft's range. At that time, it is considered that the firefighting aircraft has collided and crashed, and a certain penalty value is obtained.

[0118] firefighting aircraft Instant reward function Defined as:

[0119] ,

[0120] ,

[0121] ,

[0122] in, Firefighting aircraft The instant pathfinding reward function, , These are weight parameters, and all are positive numbers. Firefighting aircraft Instant collision rewards. Firefighting aircraft The single-step time penalty value. In order to enable the firefighting aircraft to reach the water drop mission area as soon as possible, a certain penalty value is given for each decision step before the firefighting aircraft reaches the water drop mission area, so as to guide the firefighting aircraft to find the optimal strategy to reach the water drop mission area as soon as possible.

[0123] S4: Introduce the LCAEG-TD3 algorithm and combine it with nonholonomic motion constraints, environmental range constraints and reward functions to construct the initial path planning model for the firefighting aircraft;

[0124] The LCAEG-TD3 algorithm provided by this invention is an improved TD3 algorithm, and the algorithm flowchart is as follows. Figure 6 As shown;

[0125] The training process for the initial water drop path planning model of the firefighting aircraft includes:

[0126] At the start of each round, the positions of the firefighting aircraft are randomly initialized (e.g., ...). Figure 5 As shown), the speed magnitude and direction are determined, and the positions and areas of the no-fly zone and water drop zone are randomly initialized according to the prescribed number;

[0127] The firefighting aircraft selects its actions using an initial strategy.

[0128] Information on each step of the pursuit of the firefighting aircraft Saved for network training, among which This is the status information of the firefighting aircraft at this moment. The actions taken by the firefighting aircraft at this moment The observation information from the firefighting aircraft at this moment The bonus value for the firefighting aircraft at this moment. The status information of the firefighting aircraft at the next moment after it takes action;

[0129] go through After rounds of game-based training, the final water-dropping path planning model of the agent is obtained. The value is 3000.

[0130] The update process of the LCAEG-TD3 algorithm is as follows:

[0131] The TD3 algorithm is based on double Q learning. By selecting the smaller value of the two evaluation functions, it limits the overestimation of Q value and uses a double delay update strategy to reduce the error of each update

[12] , thereby further improving the performance of the algorithm. The TD3 algorithm contains a total of six neural networks, including two policy neural networks, namely the "actor" type network in the "actor-critic" framework. These two networks are called "actor_eval" policy estimation network and "actor_target" policy realization network, respectively. Compared with the DDPG algorithm, it has an additional pair of value neural networks, namely "critic" type networks, namely "critic_eval" value estimation network and "critic_target" value realization network, respectively

[13] . The TD3 algorithm uses two critic networks to evaluate the value of the actor network and selects the Q value of the smaller evaluation network as the update parameter of the actor network, which can effectively alleviate the overestimation of Q value.

[0132] Design a critical network assisted evaluation mechanism (AE) in the TD3 algorithm. , Used as an auxiliary model. The fitness of actors in the genetic population is obtained by averaging the predicted Q-values ​​of all states using information from the k latest state vectors extracted from the mixed experience pool. Candidate control strategies are designed to reasonably combine real and virtual fitness to improve accuracy. Assuming a population size of N, a candidate population with M offspring is generated before evaluating the population in the real environment, where M>N. After evaluation by the auxiliary mechanism, the top N individuals in fitness are presented to interact in the real environment. In addition to the original population, MN additional individuals are generated by adding Gaussian noise to the RL agent.

[0133] A learning crossover operator is proposed. During crossover, the buffer is only partially inherited, and the crossover offspring fill its buffer with the most recent half of the experience from both parents' genetic memories. The new agent's genome is filled with the latest experience from both parents' genomes. The child agent is trained through imitation learning to selectively mimic the actions taken by the parents in the newly created genome state, "refining" the parents' behavior into the offspring policy. That is, the SGD algorithm is used to minimize L(c) and optimize the offspring policy.

[0134] The policy population is updated using the above operators, and the TD3 network is trained using sample data until convergence, thus obtaining the optimal policy network.

[0135] LCAEG-TD3 algorithm steps:

[0136] (1) Initialize n policy networks as a population, initialize the RL algorithm actor network, critic network, and GA algorithm parameters; initialize the genetic memory of the population individuals, i.e., the empty replay buffer; initialize the hybrid experience pool R;

[0137] (2) Initialize the state and fitness;

[0138] (3) Assess the population, adopt an auxiliary assessment mechanism, obtain individual fitness, and store the strategy network and environment interaction data into the individual's genetic code and replay buffer R;

[0139] (4) Rank the policy network according to fitness, and select the policy with the highest fitness as the elite policy. ;

[0140] (5) Based on the obtained fitness, use the improved learning crossover operator and mutation operator to perform crossover and mutation operations on the population to update the population;

[0141] (6) Train the RL network using sample data;

[0142] (7) If the termination condition is not met, go to (2) until convergence is achieved and the optimal policy network is obtained;

[0143] S5: Train the initial path planning model of the firefighting aircraft, obtain the interaction data during the training process, and use the interaction data to continuously train and update the neural network of the firefighting aircraft. After the training is completed, the final model of the firefighting aircraft's water drop path planning is obtained.

[0144] The present invention will now conduct a simulation experiment on the final model of the firefighting aircraft water drop path planning obtained by the above method:

[0145] Extensive training was conducted on the water drop path planning of firefighting aircraft in the constructed simulated firefighting environment. The main parameters during the training process are shown in Table 1:

[0146] Table 1 Training parameters for water drop path planning of firefighting aircraft swarm

[0147]

[0148] During training, the changes in some parameter values ​​of the neural network are as follows: Figure 7 As shown in the figure, in the early stages of training, the actor network parameters are unstable and change significantly. As the number of training rounds increases, the network parameters gradually stabilize, and the network model tends to converge.

[0149] S6: The final model for water-dropping path planning using firefighting aircraft can output the path planning strategy for firefighting aircraft in real time. After training, the trained LCAEG-TD3 algorithm model is used for simulation verification to test the effectiveness of the algorithm. In a randomly generated scenario, there is 1 firefighting aircraft and 6 no-fly zones. The complete flight path of the firefighting aircraft cluster is as follows: Figure 8a As shown in the diagram, the obstacle avoidance effect of the firefighting aircraft can be clearly seen from the flight path. Ultimately, the firefighting aircraft safely reached the water-dropping mission area within the specified time limit.

[0150] To verify the generalization ability of the LCAEG-TD3 algorithm, simulated firefighting environments with varying numbers of no-fly zones were constructed, including a no-fly zone scenario. The firefighting aircraft used the algorithm model trained in the no-fly zone environment to make flight decisions, verifying the performance of the trained model in the new environment. The initial state and flight path of the firefighting aircraft in the new environment are as follows: Figure 8b , Figure 8c , Figure 8d As shown, it is entirely feasible to use the trained LCAEG-TD3 model for firefighting aircraft flight decisions.

[0151] To verify the superiority of the LCAEG-TD3 algorithm, both the LCAEG-TD3 and TD3 algorithms were used to train the water drop path planning of firefighting aircraft for 3000 training epochs. The network models of both algorithms at different iteration epochs were retained, and the final comparison of firefighting aircraft mission success rates is shown in the figure below. Figure 9 As shown in the figure, it is clear that the LCAEG-TD3 algorithm converges after about 1200 training epochs, while the TD3 algorithm converges after about 1800 training epochs. Furthermore, the LCAEG-TD3 algorithm has a slightly higher task success rate after convergence, demonstrating that the LCAEG-TD3 algorithm has a faster convergence speed and performs better in completing tasks.

[0152] The above are merely embodiments of the present invention. The invention is not limited to the fields covered by these embodiments. Commonly known structures and characteristics in the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are able to access all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A method for autonomous mission path planning of unmanned aerial vehicles based on the LCAEG-TD3 algorithm, characterized in that, include: S1. Construct a standalone virtual path planning scenario; S2. Determine the nonholonomic motion constraints and environmental range constraints in the path planning scenario of the firefighting aircraft, and build a detection model for the firefighting aircraft; S3. Construct the reward function for the firefighting aircraft; the reward function is used to evaluate the performance of the firefighting aircraft in a specific state for a certain action and output the reward value. S4. Introduce the LCAEG-TD3 algorithm and combine it with nonholonomic motion constraints, environmental range constraints, and reward functions to construct an initial path planning model for the firefighting aircraft; S5. Train the initial path planning model of the firefighting aircraft, obtain the interaction data during the training process, and use the interaction data to continuously train and update the neural network of the firefighting aircraft. After the training is completed, the final model of single-aircraft water drop path planning is obtained. S6. The final model of single-aircraft water drop path planning can output the path planning strategy of firefighting aircraft in real time; The expression for the environmental range constraint in S2 is as follows: ， in, The minimum and maximum values ​​of the horizontal axis of the virtual environment; The minimum and maximum values ​​of the vertical axis of the virtual environment; Four points form a rectangular virtual environment in which the firefighting aircraft is generated and moves. The location of the firefighting aircraft is indicated; the mission is considered a failure if the firefighting aircraft flies out of any environmental boundary. The firefighting aircraft detection model in S2 is as follows: Since individual firefighting aircraft cannot obtain global status information, each individual firefighting aircraft is assumed to carry a detection radar payload, and the radar detection range is... The detection range is a 360° circular area around the fuselage; the radar can obtain the relative distance and angle between the aircraft and neighboring firefighting aircraft within the detection range, as well as the no-fly zone and the aircraft itself. This information will be used for the firefighting aircraft's decision learning. When the radar carried by the firefighting aircraft detects a target, it is necessary to determine the threat level of the target relative to the firefighting aircraft. The threat level is determined by relative distance. At this point, it is considered that the target is very close to the firefighting aircraft, and the firefighting aircraft needs to take precautions to avoid a collision; when the detected target is less than... At this point, it is assumed that the firefighting aircraft has collided with the target and has crashed; The LCAEG-TD3 algorithm in S4 is as follows: (1) Initialize n policy networks as a population, initialize the RL algorithm actor network, critic network, and GA algorithm parameters; initialize the genetic memory of the population individuals, i.e., the empty replay buffer; initialize the hybrid experience pool R; (2) Initialize the state and fitness; (3) Evaluate the population, adopt an auxiliary evaluation mechanism, obtain individual fitness, and store the strategy network and environment interaction data into the individual's genetic code and replay buffer R; (4) Rank the policy network according to fitness, and select the policy with the highest fitness as the elite policy. ; (5) Based on the obtained fitness, use the improved learning crossover operator and mutation operator to perform crossover and mutation operations on the population to update the population; (6) Train the RL network using sample data; (7) If the termination condition is not met, go to (2) until convergence is achieved and the optimal policy network is obtained; (8) After training, a water drop path planning model for firefighting aircraft is obtained for direct use: After training, the final water-dropping path planning model for the firefighting aircraft is obtained. The state information of the mission area and some environmental conditions is input into the neural network of the model in real time to obtain the action selection of the firefighting aircraft and control the firefighting aircraft to complete the path planning task of the water-dropping mission area.

2. The UAV autonomous mission path planning method based on the LCAEG-TD3 algorithm according to claim 1, characterized in that, The expression for the nonholonomic motion constraint condition of the firefighting aircraft in the firefighting and water-dropping scenario in S2 is as follows: ， ， ， ， ， ， In the above formula: A matrix representing the information of the firefighting aircraft at time t; and Let be the position coordinates of the firefighting aircraft at time t; Let be the speed of the firefighting aircraft at time t; Let be the azimuth angle of the firefighting aircraft at time t; and Let be the linear acceleration and angular acceleration of the firefighting aircraft at time t; This is for simulating step size; A matrix representing the information of firefighting aircraft at time t+1; and The coordinates of the firefighting aircraft at time t+1; The speed of the firefighting aircraft at time t+1; The azimuth of the firefighting aircraft at time t+1.

3. The UAV autonomous mission path planning method based on the LCAEG-TD3 algorithm according to claim 2, characterized in that, The reward function in S3 includes a long-term reward function and an immediate reward function. The long-term reward function is mainly for the reward or penalty given to the agent when it reaches some special state in the task. In the water drop mission path planning of the fire-fighting aircraft swarm, the long-term rewards include: fire-fighting aircraft arrival reward, fire-fighting aircraft going out of bounds reward, and fire-fighting aircraft crash reward.

4. The UAV autonomous mission path planning method based on the LCAEG-TD3 algorithm according to claim 3, characterized in that, The formula for the long-term reward function is as follows: ， in, Firefighting aircraft Long-term rewards; Firefighting aircraft The reward for arriving, when the firefighting plane Enter the designated water-dropping task area, that is... When the mission is completed, the firefighting aircraft will receive a certain amount of reward points. Firefighting aircraft The penalty for going out of bounds, when the firefighting plane Flying out of the simulated firefighting environment, that is or When this occurs, the firefighting aircraft is considered to have lost contact and crashed, and a certain penalty value is obtained; Firefighting aircraft The crash reward for the firefighting aircraft Flying into a no-fly zone poses a danger of detection targets entering the firefighting aircraft's range. At that time, it is considered that the firefighting aircraft has collided and crashed, and a certain penalty value is obtained.

5. The UAV autonomous mission path planning method based on the LCAEG-TD3 algorithm according to claim 4, characterized in that, For It is represented and defined as: ， ， ， in, Firefighting aircraft The instant pathfinding reward function, , These are weight parameters, and all are positive numbers; Firefighting aircraft Instant collision rewards; Firefighting aircraft The single-step time penalty value; in order to enable the firefighting aircraft to reach the water drop mission area as soon as possible, a certain penalty value is given for each decision step before the firefighting aircraft reaches the water drop mission area, so as to guide the firefighting aircraft to find the optimal strategy to reach the water drop mission area as soon as possible.

6. The UAV autonomous mission path planning method based on the LCAEG-TD3 algorithm according to claim 1, characterized in that, The training process of the initial path planning model in S5 includes: At the beginning of each round, the position, speed, and direction of the firefighting aircraft are randomly initialized, and the positions and areas of the no-fly zone and the mission area are randomly initialized according to a specified number. The firefighting aircraft selects its actions using an initial strategy. Information on each step of the pursuit of the firefighting aircraft Saved for network training, among which This is the status information of the firefighting aircraft at this moment. The actions taken by the firefighting aircraft at this moment The observation information from the firefighting aircraft at this moment The bonus value for the firefighting aircraft at this moment. The status information of the firefighting aircraft at the next moment after it takes action; go through After rounds of game training, the final water-dropping path planning model of the agent is obtained.

7. The UAV autonomous mission path planning method based on the LCAEG-TD3 algorithm according to claim 1, characterized in that, The training and updating process of the neural network in S5 is as follows: The TD3 algorithm is built on double Q-learning. By selecting the smaller of the two evaluation functions, it limits the overestimation of Q-values ​​and uses a double-delay update strategy to reduce the error of each update, thereby further improving the algorithm's performance. The TD3 algorithm contains a total of six neural networks, including two policy neural networks, namely the "actor" type networks in the "actor-critic" framework. These two networks are called the "actor_eval" policy estimation network and the "actor_target" policy realization network, respectively. Compared with the DDPG algorithm, it has an additional pair of value neural networks, namely the "critic" type networks, namely the "critic_eval" value estimation network and the "critic_target" value realization network, respectively. The TD3 algorithm uses two critic networks to evaluate the value of the actor network, and selects the Q value of the smaller evaluation network as the update parameter of the actor network. Design a critic network-assisted evaluation mechanism in the TD3 algorithm. , Used as an auxiliary model; By averaging the predicted Q-values ​​of all states using the information from the k latest state vectors extracted from the mixed experience pool, the fitness value of the actors in the genetic population is obtained. Design a candidate control strategy that combines real fitness and virtual fitness. Assuming the population size is N, before evaluating the population in the real environment, generate a candidate population with M offspring, where M>N. After evaluation by an auxiliary mechanism, the top N individuals in terms of fitness are presented in the real environment for interaction. In addition to the original population, MN additional individuals are generated by adding Gaussian noise to the RL agent. The design incorporates a learning crossover operator. During crossover, the buffer is only partially inherited, and the crossover offspring fills its buffer with the most recent half of the experience from the genetic memory of both parents. The new agent's genetic code is filled with the latest experience from the genetic code of both parents. The child agent is trained through imitation learning to selectively imitate the actions taken by the parents in the newly created genetic code, and the behavior of the parents is incorporated into the offspring's policy. That is, the SGD algorithm is used to minimize L(c) and optimize the offspring's policy. The policy population is updated using the above operators, and the TD3 network is trained using sample data until convergence, thus obtaining the optimal policy network.

Images