EVTOL safety obstacle avoidance method based on maximum trajectory entropy SAC reinforcement learning

Abstract

The invention provides an eVTOL safety obstacle avoidance method based on maximum trajectory entropy SAC reinforcement learning, which is used for solving the safety obstacle avoidance problem of flying eVTOL in urban dynamic and static environments, and comprises the following steps: designing a constraint set by using an eVTOL kinetic equation, and constraining a selection strategy of an MDP model based on the constraint set to obtain an initial CMDP model; performing iterative training on the initial CMDP model, and optimizing the initial CMDP model by using a maximum trajectory entropy SAC algorithm in each iterative process to obtain a trained CMDP model; and selecting an optimal current action by using the trained CMDP model. Compared with a traditional maximum entropy reinforcement learning algorithm based on a strategy entropy item, the method is higher in convergence speed, more stable in performance and high in safety obstacle avoidance task completion rate, and a new thought is provided for eVTOL automatic driving function design.

Description

Technical Field

[0001] The present invention belongs to the technical field of safe obstacle avoidance, and specifically relates to an eVTOL safe obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning. Background Art

[0002] Electric Vertical Takeoff and Landing Aircraft (eVTOL) is a new type of aircraft that has developed rapidly in recent years. It combines vertical takeoff and landing capabilities, electric propulsion systems and advanced autonomous driving technology, and is regarded as the core carrier of future urban air mobility (UAM).

[0003] The core application scenarios of eVTOL are: urban air travel, solving traffic congestion, and providing "air taxi" services (such as the Uber Elevate program). Emergency rescue, quickly transporting medical supplies or injured people, especially when disasters cause ground transportation to be interrupted. Logistics and transportation, an upgraded version of drone delivery, such as Amazon Prime Air and SF Express's large-scale freight eVTOL test. Regional commuting, connecting urban agglomerations, a 1-hour flight covering a traditional 3-4 hour drive. Driving automation is an important feature of eVTOL. Most eVTOLs are designed to be autonomous or remotely controlled, without the need for passengers to perform complex operations. This lays the foundation for the widespread application of eVTOL. Compared with ordinary drones, eVTOLs have higher requirements for autonomous driving functions and place greater emphasis on flight safety, making safe obstacle avoidance of eVTOL a research focus.

[0004] Reinforcement learning algorithms are widely used in path planning and safe obstacle avoidance tasks. Compared with classical algorithms, they are more efficient in dynamic path planning of aircraft. Lan et al. used the improved DDPG algorithm to solve the path planning problem of gliders in time-varying ocean current in the paper "Lan W, Jin X, Chang X, et al. Pathplanning for underwater gliders in time-varying ocean current using deep reinforcement learning [J]. Ocean Engineering, 2022, 262: 112226.". Wang et al. used drones as base station nodes in the paper "Wang X, Gursoy MC, Erpek T, et al. Learning-based UAV path planning for data collection within integrated collision avoidance [J]. IEEE Internet of Things Journal, 2022, 9(17): 16663-16676." to solve the UAV path planning problem with collision avoidance at a fixed altitude.

[0005] The obstacle avoidance solutions in the prior art have the following defects:

[0006] 1. Current obstacle avoidance algorithm research primarily focuses on multi-rotor drones and traditional fixed-wing aircraft, while the development of safe obstacle avoidance for eVTOLs is still an emerging field. While eVTOLs share structural similarities with traditional aircraft, they possess unique kinematic models, which directly impact the design of obstacle avoidance algorithm constraints. Therefore, a dedicated obstacle avoidance algorithm theory suitable for eVTOLs is needed.

[0007] 2. Path planning and obstacle avoidance algorithms based on deep reinforcement learning demonstrate significant performance advantages over traditional rule-based algorithms. Existing research generally employs policy entropy regularization to enhance policy exploration capabilities. However, the classic policy entropy mechanism lacks temporal correlation, which can lead to a decrease in exploration efficiency. This not only reduces the agent's behavioral space coverage but also leads to suboptimal policy convergence, ultimately significantly reducing mission success rates. Summary of the Invention

[0008] In order to solve the above problems existing in the prior art, the present invention provides an eVTOL safe obstacle avoidance method based on maximizing trajectory entropy (SAC) reinforcement learning. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0009] An eVTOL safe obstacle avoidance method based on maximizing trajectory entropy (SAC) reinforcement learning includes:

[0010] S100, determining an obstacle avoidance area for the eVTOL and constructing an eVTOL dynamics equation;

[0011] S200, designing a constraint set using the eVTOL dynamics equation, and constraining a selection strategy of an MDP model based on the constraint set to obtain an initial CMDP model;

[0012] S300, iteratively training the initial CMDP model, and optimizing the initial CMDP model using a SAC algorithm based on maximum trajectory entropy in each iterative process to obtain a trained CMDP model;

[0013] S400, inputting the current state of the eVTOL in the obstacle avoidance area into the trained CMDP model to obtain an optimal current action, and executing the optimal current action;

[0014] S500, repeat S400 until the destination is reached.

[0015] Beneficial effects:

[0016] The present invention provides an eVTOL safe obstacle avoidance method based on maximizing trajectory entropy SAC reinforcement learning, which is used to address the safe obstacle avoidance problem of eVTOL in dynamic and static environments when operating in cities. A constraint set is designed using the eVTOL dynamic equation, and the selection strategy of the MDP model is constrained based on the constraint set to obtain an initial CMDP model, which is then iteratively trained. In each iteration, the initial CMDP model is optimized using the maximizing trajectory entropy SAC algorithm to obtain a trained CMDP model; the trained CMDP model is used to select the optimal current action. Compared with the traditional maximum entropy reinforcement learning algorithm based on the policy entropy term, the present invention has a faster convergence speed, more stable performance, and a high completion rate for safe obstacle avoidance tasks, providing a new approach to the design of eVTOL autonomous driving functions.

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of an eVTOL safe obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning provided by the present invention;

[0019] Figure 2 This is a schematic diagram of the route planning task provided by the present invention;

[0020] Figure 3 This is a simulation effect diagram provided by the present invention;

[0021] Figure 4 It is a convergence curve diagram of the maximum trajectory entropy SAC algorithm provided by the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0023] like Figure 1 As shown, the present invention provides an eVTOL safety obstacle avoidance method based on maximum trajectory entropy SAC reinforcement learning, including:

[0024] S100, determining an obstacle avoidance area for the eVTOL and constructing an eVTOL dynamics equation; wherein, within the obstacle avoidance area, the eVTOL has two types of subtasks, the first type of subtask is: avoiding two obstacles, avoiding collisions, and ensuring flight safety; the second type of subtask is reaching the destination in the shortest possible time.

[0025] The application scenarios of eVTOL focus on manned and cargo flights. Its flight speed range is about 120-300km / h, and its take-off weight is more than 200kg. Compared with drones and light rotorcraft, it has a greater speed and payload weight. It cannot change course immediately at the turning point, and is subject to maximum turning performance constraints when turning during high-speed cruising flight. At the same time, based on its application prospects of "urban air travel", eVTOL emphasizes safety and comfort during flight, requiring the speed to be appropriate and change smoothly, so it is subject to maximum speed and acceleration constraints. Based on this, the range limitations of speed, acceleration, angular velocity and yaw angle should be considered in the dynamic modeling of eVTOL. The position and attitude of the eVTOL aircraft at any moment can be expressed as (x, y, v, ψ, u, w). The position of the eVTOL aircraft is (x, y), the yaw angle is ψ, the eVTOL aircraft moves at a cruising speed v, its acceleration at any moment is u, and its angular velocity is w. The eVTOL dynamic equation is expressed as follows:

[0026]

[0027] Where, v min ,v max is the speed range, ψ min ,ψ max is the heading angle range, u min ,u max is the acceleration range, ω min ,ω max is the angular velocity range.

[0028] A single eVTOL flies in a low-altitude urban environment. It needs to reach the set destination from an initial position. There are two types of obstacles during the flight: static obstacles mainly consisting of buildings (building logos in the picture) and dynamic obstacles mainly consisting of unknown aircraft (red aircraft in the picture). The eVTOL needs to avoid static obstacles and dynamic obstacles that move in various ways and reach its destination. From this, we can derive two types of subtasks that need to be completed for eVTOL to avoid obstacles safely: 1. Avoid two types of obstacles to avoid collisions and ensure flight safety; 2. Reach the destination in the shortest possible time.

[0029] refer to Figure 2 Based on the aforementioned task requirements and the characteristics of deep reinforcement learning, the present invention constructs a static uncertain environment with turn-based braking. In each training round, the eVTOL's initial state—position (x, y), yaw angle ψ, and cruising speed v—is randomly generated. The coordinates of the target location, dynamic and static obstacles are also randomly generated. To simplify the complexity of the environment, the eVTOL's flight environment is considered a bounded airspace, and the eVTOL is assumed to move at a constant altitude (ignoring changes in the eVTOL's altitude), and the unknown aircraft also moves at this altitude. The target location is communicated to the eVTOL via ground radar at the start of the round. During flight, the eVTOL uses its own radar ranging to obtain distance information from the target location and obstacles. The entire drone path planning simulation environment is defined as a two-dimensional plane of length*width. If the friendly eVTOL approaches the unknown aircraft too close or enters the area containing a building, a collision is detected, the obstacle avoidance mission fails, and the round ends. If the friendly eVTOL reaches the target location, it is considered to have successfully reached the target location, and the round ends.

[0030] S200: Designing a constraint set using the eVTOL dynamics equation, and constraining the selection strategy of the MDP model based on the constraint set to obtain an initial CMDP model; wherein the constraint set includes the eVTOL dynamics equation constraint and the eVTOL flight path constraint; the dynamics equation constraint uses the eVTOL dynamics equation as a constraint condition, and the eVTOL flight path constraint uses the distance between the eVTOL and an obstacle as a constraint condition. The initial CMDP model includes an actor network and a critic network; the actor network is used to output a selected action, and the critic network is used to evaluate the selected action.

[0031] Reinforcement learning is essentially a Markov decision process (MDP). MDP is mathematically defined as a five-tuple MDP = (S, A, P, R, γ). Where s = {s1, s2, ...s t} is the state space; a={a1,a2,...a t} is the action space; P(s t+1 |s t ,a t )=P(S'=s t+1 |S=s t ,A=a t ) is the state transfer function; R is the reward function; γ is the discount factor. However, in the present invention, we should not only pay attention to the reward R obtained by the intelligent agent (eVTOL), but also consider its dynamic and environmental constraints. Therefore, the present invention adopts a safe reinforcement learning CMDP process, which takes into account that in addition to pursuing maximization of rewards when selecting actions, the intelligent agent must also meet some additional constraints. These constraints may be restrictions on state transfer, reward functions or strategies. Therefore, the initial CMDP model is defined as a five-tuple CMDP = (S, A, P, R, C), where S, A, P, R are defined as the definition form of the MDP model, C is a constraint set used to constrain the CMDP model to select actions in the action set; S, A, P, R represent the state space, action space, state transfer function and reward function, respectively.

[0032] In the initial CMDP model, S={s1,s2,...s t} is represented as state space; A={a1,a2,...a t} is represented as action space; P(s t+1 |s t ,a t )=P(S'=s t+1 |S=s t ,A=a t ) is represented as the state transfer function; R is represented as the reward function; s=[x,y,v,ψ,x goal ,y goal ,O flag ], (x, y) is the position coordinate of eVTOL, v is the speed of eVTOL, ψ is the yaw angle of eVTOL, (x goal ,y goal ) is the location coordinate of the target location, O flag The obstacle flag is set to 1 when the distance between the eVTOL and any obstacle is less than a certain threshold, and it is considered that there is an obstacle near the eVTOL. Otherwise, it is set to 0.

[0033] The eVTOL action decision is to select the appropriate acceleration and angular velocity to execute for Δt time to achieve the desired speed and desired heading angle. The action control input of the eVTOL is a two-dimensional vector, and the action information is a continuous quantity, expressed as a = [u, ω] T , u represents the acceleration of eVTOL, and ω represents the angular velocity.

[0034] The reward function in the initial CMDP is expressed as:

[0035] R=ω1R edge +ω2R og

[0036] Where R edge represents the boundary reward, R og represents obstacle avoidance and target rewards, ω1, ω2 are the weights of the corresponding rewards;

[0037]

[0038] Where r edge Indicates the penalty given when the eVTOL reaches the border, otherwise no reward or penalty is given;

[0039]

[0040] When the distance between the eVTOL and the obstacle is less than the threshold D2, it is considered that the eVTOL has the risk of potential collision with the obstacle, and the r o Penalty; when the distance between the eVTOL and the obstacle is less than the radius D of the obstacle o When the eVTOL collides with an obstacle, the eVTOL is damaged and the r fail When the distance between the eVTOL and the target point is less than the threshold D1, it is considered to have reached the target point and is given an r success Otherwise, k1d is given based on the distance between eVTOL and the target point. G Punishment; d G Indicates the distance between eVTOL and the target point, d O represents the distance between eVTOL and the obstacle, k1 is the distance influence coefficient, r success 、r o 、r fail The points represent the reward given for successful mission, the penalty given for approaching an obstacle, and the penalty given for mission failure.

[0041] The eVTOL dynamic equation constraints are as follows:

[0042]

[0043] Where x, y, v, ψ are the flight state variables of the own eVTOL, [u, ω] T is the control vector, v min ,v max is the speed range, ψ min ,ψ max is the heading angle range, u min ,u max is the acceleration range, ω min ,ω max is the angular velocity range. Select velocity v min =100m / s,v max =200m / s, acceleration u min =-6m / s 2 ,u max =6m / s 2 , angular velocity ω min =-1.2rad / s,ω max =1.2rad / s.

[0044] eVTOL flight path constraints: eVTOL needs to avoid buildings and unknown aircraft during flight. Once a collision occurs, the agent will be destroyed. For ease of handling, the collision detection of all obstacles is set to a standard circle, which can be described as

[0045]

[0046] Where (x O ,y O ) is the coordinate of the center of the circular obstacle, D O is the radius of the circular obstacle.

[0047] S300, iteratively training the initial CMDP model, and optimizing the initial CMDP model using a SAC algorithm based on maximum trajectory entropy in each iterative process to obtain a trained CMDP model;

[0048] In this step, the network is trained using the Soft Actor Critic (SAC) algorithm that maximizes trajectory entropy.

[0049] A. Definition and calculation of trajectory entropy and its advantages:

[0050] Introducing policy entropy term into traditional maximum entropy reinforcement learning algorithm

[0051]

[0052] where π(·|s) is the policy function. A higher entropy results in a more uniform distribution of actions in state s, i.e., higher uncertainty and stronger exploratory behavior. This term encourages the policy to select actions more randomly, avoiding local optima.

[0053] However, due to the state transition p(s t+1 |s t ,a t ) may cause the state sequence to be highly correlated due to dynamic constraints (such as inertia and physical continuity), showing the characteristics of time correlation. At this time, since the policy entropy does not directly constrain the statistical characteristics of state transition, it is impossible to actively intervene in the dynamic process of state transition and can only passively rely on the randomness of actions, resulting in invalid exploration (such as repeated visits to similar states).

[0054] And introduce the trajectory entropy term:

[0055] H(x t )=logdetC[x t ], formula (1)

[0056] The calculation of C is as follows: t At any time point x*∈x t ,

[0057]

[0058] In the formula, Z is the regularization constant, and the matrix C[x] calculates the local autocovariance K XX .

[0059] Taking the trajectory entropy term into consideration instead of the policy entropy term, the general optimization objective for the reinforcement learning algorithm is obtained as

[0060]

[0061] At this point, the algorithm tends to maximize the reward while maximizing the trajectory entropy of the path.

[0062] This idea comes from the paper Berrueta, TA, Pinosky, A. & Murphey, TD Maximum diffusion reinforcement learning. Nat Mach Intell 6, 504–514 (2024).

[0063] The actual physical meaning of trajectory entropy is the quantification of path correlation. Maximizing trajectory entropy is actually equivalent to maximizing the trajectory probability density, that is, finding the trajectory with minimum correlation under the constraint of path continuity. It can be shown that this result is equivalent to the path distribution of an anisotropic, spatially inhomogeneous diffusion process.

[0064] Trajectory entropy proactively intervenes in the dynamics of state transitions, rewarding highly diffuse state transitions and guiding the agent to prioritize exploration in less correlated directions. Furthermore, the algorithm ensures that a single round of path exploration covers the entire reachable state space, guaranteeing the traversability of a single trajectory and enabling trajectory entropy to achieve global coverage. Overall, the trajectory entropy algorithm achieves high exploratory power even when states exhibit temporal correlation.

[0065] Based on the goal of optimizing the eVTOL safe obstacle avoidance algorithm of the present invention, the trajectory entropy term has great advantages over the policy entropy term: First, the eVTOL obstacle avoidance process is obviously a process of continuous path change, which conforms to the basic application field of the trajectory entropy algorithm mentioned above; second, because the eVTOL's own dynamic equations have range limitations such as speed and acceleration, and there are obstacle constraints, it is easy for the state sequence to be highly correlated, which reduces the effect of the policy entropy term and causes the algorithm to fall into a local optimum. Introducing the trajectory entropy term at this time can avoid this problem, enhance the algorithm's exploratory nature, and further improve the efficiency of obstacle avoidance.

[0066] It should be noted that theoretically, in the trajectory entropy H(x t ), x t Defined as starting from the turn To the current time step However, considering the characteristics of the eVTOL safe obstacle avoidance task with a large number of round steps and a high path repetition rate, in order to reduce the algorithm training time and improve the algorithm convergence efficiency, in the actual algorithm design x t N steps can be taken, that is, only considering to The trajectory entropy is calculated at the time steps between .

[0067] Trajectory = x t -x t-N

[0068] After experiments, the optimal parameter N=100 is obtained.

[0069] B. Maximizing Trajectory Entropy SAC Algorithm

[0070] The SAC model based on maximizing trajectory entropy consists of two networks, one with parameters θ and the other with state s as input, and the other with policy probability distribution π(·|s t ) Actor network with parameter w, input state-action pair (s, a), and output Q value Critic network.

[0071] The target strategy formula of this method

[0072]

[0073] where rt (s t ,a t ) is the reward at time t, α is the temperature coefficient, γ is the attenuation factor, logdetC[x t ] is the trajectory entropy at time t.

[0074] For the Actor network, updates are performed by minimizing the following loss function, expressed as:

[0075]

[0076] Among them, M is the experience pool, which is responsible for collecting training data and can break the correlation between samples.

[0077] For the update of the Critic policy network, minimize the loss function of the following formula:

[0078]

[0079] S300 includes:

[0080] (1) Initialize the Actor network, Critic network, experience pool M and Gaussian noise;

[0081] (2) Set the number of loops to Episode = 1;

[0082] (3) Initialize the obstacle avoidance environment and obtain the current state s of the eVTOL;

[0083] (4) Let the current step of the drone be Step = 1;

[0084] (5) Use Gaussian noise to select an action from the experience pool and output the selected action a;

[0085] (6) limiting the amplitude of the action a to [-1, 1];

[0086] (7) Based on action a, and using this action a to return reward r, the next moment state s', the round end flag Done, and the current trajectory Trajectory;

[0087] (8), store the sample (s,a,r,s',Trajectory) in the experience pool M;

[0088] (9) If the experience pool M is full, execute (10). If not, set Step = Step + 1 and return to step (4). If full, execute step (10).

[0089] (10) Randomly sample a batch of data from M, use the sampled data to calculate the trajectory entropy Entrophy, and use the trajectory entropy Entrophy to update the weight parameters of the Critic network and the Actor network;

[0090] The process of updating the weight parameters in this step is as follows: randomly sample a batch of data from M, and use the sampled data to calculate the trajectory entropy Entrophy; perform the gradient descent of the Actor network with the purpose of maximizing the trajectory entropy Entrophy, maximizing the reward, and minimizing the loss function of the Actor network, and update the weight parameters of the Actor network; perform the gradient descent of the Critic network with the purpose of maximizing the trajectory entropy Entrophy, maximizing the reward, and minimizing the loss function of the Critic network, and update the weight parameters of the Critic network.

[0091] (11), update the current state of the drone to s';

[0092] (12) Determine whether Episode has reached the maximum number of iterations. If not, set Episode = Episode + 1 and return to step (4) until the maximum number of iterations is reached to obtain the trained CMDP model.

[0093] See the following algorithm code for details:

[0094] Table 1

[0095]

[0096] After experiments, the parameters with the best training effect are:

[0097] Table 2 CMDP parameter selection

[0098] <![CDATA[r edge =-2]]> <![CDATA[r success =1000]]> <![CDATA[r o =-2]]> <![CDATA[r fail =-500]]> <![CDATA[k1=-0.001]]> <![CDATA[ω1=1]]> <![CDATA[ω2=1]]> <![CDATA[D1=40]]> <![CDATA[D2=40]]> <![CDATA[D o =20]]>

[0099] S400, inputting the current state of the eVTOL in the obstacle avoidance area into the trained CMDP model to obtain an optimal current action, and executing the optimal current action;

[0100] S500, repeat S400 until the destination is reached.

[0101] The effects of the present invention can be further illustrated by the following simulation experiments.

[0102] 1. Simulation conditions

[0103] The simulation was performed using Python, based on Pycharm Community Edition 2023.3.4. The versions of the main installation packages used in the simulation compilation environment were: pygame version 2.1.2; gym version 0.19.0; pytorch version 1.10.0+cu113; and numpy version 1.23.1.

[0104] The following table lists all the parameters and hyperparameters of the SAC algorithm for maximizing trajectory entropy. It should be noted that Gaussian noise is added to the algorithm in the first 50 rounds to allow the agent to explore as many states as possible.

[0105] Table 3 Parameters and hyperparameters for maximizing trajectory entropy SAC

[0106]

[0107] In the eVTOL path planning simulation environment, select length = 7000m, width = 6000m. One pixel represents 10 meters, so the actual simulation interface size is 700 pixels * 600 pixels. The animation refresh rate of the environment is 60 FPS. The simulation time step Δt is set to 1 frame. The simulation effect is shown in the figure below. Figure 3 As shown, the blue aircraft represents eVTOL, the red aircraft represents unknown aircraft, the black solid circle represents the building, and the green five-pointed star represents the target location.

[0108] The methods compared in the experiment are as follows:

[0109] One is the deep deterministic policy gradient (DDPG) algorithm, referenced by T.P.Lillicrap, J.J.Hunt, A.Pritzel, et al., “Continuous control with deep reinforcement learning,” in International Conference on Learning Representations, 2016.

[0110] One is the SAC method based on policy entropy. The reference is T.Haarnoja, A.Zhou, P.Abbeel, et al., “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in International Conference on Machine Learning, 2018, pp.1861-1870.

[0111] 2. Simulation content

[0112] 1) First, the training process of the maximum trajectory entropy SAC algorithm is carried out.

[0113] During the algorithm training phase, the number of buildings and unknown aircraft in the airspace scene is set to 1. The total number of training rounds is set to 500. The single-step reward obtained in each round of eVTOL is accumulated and calculated, and then the total reward curve of the 500-round maximum trajectory entropy SAC algorithm training is drawn. The training of 500 rounds is repeated multiple times to calculate the average value. Through experiments, the convergence curve of the maximum trajectory entropy SAC algorithm can be drawn as follows Figure 4 shown.

[0114] The total reward ranged from -1500 to -500 for the first 100 rounds, indicating that the eVTOL was unable to reach the destination and was instead randomly exploring the environment. The algorithm's exploration outweighed its utilization. After 100 rounds, the reward began to rise rapidly, and around round 125, the algorithm converged, with the average total reward reaching around 950. This indicates that the eVTOL was able to complete the mission and reach the designated destination while avoiding obstacles. After 100 rounds, although some rounds still had negative rewards, this was due to collision penalties caused by the agent's failure to avoid obstacles in certain special circumstances. This means that the algorithm has found a near-optimal strategy, but it cannot achieve a 100% success rate for the mission.

[0115] 2) Based on the trained approximate optimal network, test the algorithm performance under different numbers of obstacles.

[0116] During the algorithm testing phase, simulations were conducted based on the previously trained, near-optimal network weight parameters. The number of obstacles was expanded to: 1 building, 1 unknown aircraft; 2 buildings, 2 unknown aircraft; 2 buildings, 3 unknown aircraft; and 3 buildings, 3 unknown aircraft. The success rate R (number of successes / total number of attempts), average flight time T (accumulated number of time steps / total number of attempts), average flight distance S (accumulated acceleration per time step / total number of attempts), and average energy consumption E (sum of the absolute values of the accumulated actions per time step / total number of attempts) for each scenario were recorded. 100 rounds of testing were performed on each scenario, yielding the following results:

[0117] Test results show that algorithm performance is closely related to scene complexity: the success rate decreases significantly with the increase in the number of obstacles and aircraft, mainly due to the increased difficulty of obstacle avoidance conflicts and multi-aircraft collaborative decision-making; the average flight time fluctuates between 21.97 and 23.7, and in complex scenes, path optimization may shorten the time or obstacle avoidance waiting time may be extended; the average flight distance has little to do with scene complexity, averaging around 19; the average energy consumption is positively correlated with complexity, and the increased difficulty of eVTOL obstacle avoidance in complex scenes leads to increased energy consumption.

[0118] 3) Compare with SAC and DDPG methods.

[0119] In an environment with 2 buildings and 2 aircraft, the task execution effects of the algorithm of the present invention are compared with those of the SAC and DDPG algorithms, and the results are recorded in the following table:

[0120]

[0121] In this scenario, the proposed method comprehensively outperforms the SAC and DDPG algorithms in mission execution efficiency: its success rate reaches 79.0%, significantly higher than both SAC and DDPG methods. Furthermore, by introducing a trajectory entropy mechanism that increases the diversity of path exploration, the proposed method can guide the eVTOL to find the optimal path, significantly reducing average flight time, flight trajectory, and energy consumption.

[0122] It is worth noting that the terms "first" and "second" in this disclosure are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of this disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0123] Although the present application is described herein with reference to various embodiments, those skilled in the art will be able to understand and implement other variations of the disclosed embodiments in practicing the claimed application by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0124] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A safe obstacle avoidance method for eVTOL based on maximizing trajectory entropy (SAC) reinforcement learning, characterized by: include: S100, determining an obstacle avoidance area for the eVTOL and constructing an eVTOL dynamics equation; S200, designing a constraint set using the eVTOL dynamics equation, and constraining a selection strategy of an MDP model based on the constraint set to obtain an initial CMDP model; S300, iteratively training the initial CMDP model, and optimizing the initial CMDP model using a SAC algorithm based on maximum trajectory entropy in each iterative process to obtain a trained CMDP model; S400, inputting the current state of the eVTOL in the obstacle avoidance area into the trained CMDP model to obtain an optimal current action, and executing the optimal current action; S500, repeat S400 until the destination is reached.

2. The eVTOL safety obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning according to claim 1 is characterized in that: There are two types of subtasks for eVTOL in the obstacle avoidance area. The first subtask is to avoid two obstacles, avoid collision, and ensure flight safety; the second subtask is to reach the destination in the shortest possible time.

3. The eVTOL safety obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning according to claim 1 is characterized in that: The eVTOL dynamic equation is expressed as: Where, the position of the eVTOL is (x, y), the yaw angle is ψ, it moves at a cruising speed v, the acceleration at any moment is u, the angular velocity is ω, and the position and posture at any moment can be expressed as (x, y, v, ψ, u, ω), v min ,v max is the speed range, ψ min ,ψ max is the heading angle range, u min ,u max is the acceleration range, ω min ,ω max is the angular velocity range.

4. The eVTOL safety obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning according to claim 1 is characterized in that: The constraint set includes an eVTOL dynamic equation constraint and an eVTOL flight path constraint; the dynamic equation constraint uses the eVTOL dynamic equation as a constraint condition, and the eVTOL flight path constraint uses the distance between the eVTOL and an obstacle as a constraint condition.

5. The eVTOL safety obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning according to claim 1 is characterized in that: The initial CMDP model is defined as a five-tuple CMDP = (S, A, P, R, C), where S, A, P, R are defined as the definition form of the MDP model, C is a constraint set used to constrain the CMDP model to select actions in the action set; S, A, P, R represent the state space, action space, state transfer function and reward function, respectively.

6. The eVTOL safety obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning according to claim 5 is characterized in that: In the initial CMDP model, S={s1,s2,...s t } is represented as state space; A={a1,a2,...a t } is represented as action space; P(s t+1 |s t ,a t )=P(S'=s t+1 |S=s t ,A=a t ) is represented as the state transfer function; R is represented as the reward function; s=[x,y,v,ψ,x goal ,y goal ,O flag ], (x, y) is the position coordinate of eVTOL, v is the speed of eVTOL, ψ is the yaw angle of eVTOL, (x goal ,y goal ) is the location coordinate of the target location, O flag is the obstacle flag. When the distance between the eVTOL and any obstacle is less than a certain threshold, it is considered that there is an obstacle near the eVTOL. At this time, the obstacle flag position is set to 1, otherwise it is set to 0; a=[u,ω] T , u represents the acceleration of eVTOL, and ω represents the angular velocity.

7. The eVTOL safety obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning according to claim 5, characterized in that: The reward function in the initial CMDP is expressed as: R=ω1R edge +ω2R og Where R edge represents the boundary reward, R og represents obstacle avoidance and target rewards, ω1, ω2 are the weights of the corresponding rewards; Where r edge Indicates the penalty given when the eVTOL reaches the border, otherwise no reward or penalty is given; When the distance between the eVTOL and the obstacle is less than the threshold D2, it is considered that the eVTOL has the risk of potential collision with the obstacle, and the r o Penalty; when the distance between the eVTOL and the obstacle is less than the radius D of the obstacle o When the eVTOL collides with an obstacle, the eVTOL is damaged and the r fail When the distance between the eVTOL and the target point is less than the threshold D1, it is considered to have reached the target point and is given an r success Otherwise, k1d is given based on the distance between eVTOL and the target point. G Punishment, d G Indicates the distance between eVTOL and the target point, d O represents the distance between eVTOL and the obstacle, k1 is the distance influence coefficient, r success 、r o 、r fail The points represent the reward given for successful mission, the penalty given for approaching an obstacle, and the penalty given for mission failure.

8. The eVTOL safety obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning according to claim 1, characterized in that: The initial CMDP model includes an Actor network and a Critic network; the Actor network is used to output the selected action, and the Critic network is used to evaluate the selected action.

9. The eVTOL safety obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning according to claim 8, characterized in that: S300 includes: (1) Initialize the Actor network, Critic network, experience pool M and Gaussian noise; (2) Set the number of loops to Episode = 1; (3) Initialize the obstacle avoidance environment and obtain the current state s of the eVTOL; (4) Let the current step of the drone be Step = 1; (5) Use Gaussian noise to select an action from the experience pool and output the selected action a; (6) limiting the amplitude of the action a to [-1, 1]; (7) Based on action a, and using this action a to return reward r, the next moment state s', the round end flag Done, and the current trajectory Trajectory; (8), store the sample (s,a,r,s',Trajectory) in the experience pool M; (9) If the experience pool M is full, execute (10). If not, set Step = Step + 1 and return to step (4). If full, execute step (10). (10) Randomly sample a batch of data from M, use the sampled data to calculate the trajectory entropy Entrophy, and use the trajectory entropy Entrophy to update the weight parameters of the Critic network and the Actor network; (11), update the current state of the drone to s'; (12) Determine whether Episode has reached the maximum number of iterations. If not, set Episode = Episode + 1 and return to step (4) until the maximum number of iterations is reached to obtain the trained CMDP model.

10. The eVTOL safety obstacle avoidance method based on maximum trajectory entropy (SAC) reinforcement learning according to claim 9, characterized in that: The randomly sampling a batch of data from M, calculating the trajectory entropy Entrophy using the sampled data, and updating the weight parameters of the Critic network and the Actor network using the trajectory entropy Entrophy include: Randomly sample a batch of data from M and use the sampled data to calculate the trajectory entropy Entrophy; Perform gradient descent of the Actor network with the goal of maximizing trajectory entrophy, maximizing rewards, and minimizing the loss function of the Actor network, and update the weight parameters of the Actor network; With the goal of maximizing trajectory entrophy, maximizing rewards, and minimizing the loss function of the Critic network, the gradient descent of the Critic network is performed and the weight parameters of the Critic network are updated.

Images