Air-land combined transport task-oriented aircraft path online planning method

Abstract

The application relates to a method for online planning of a carrier path for a combined air and land transportation task, and relates to the technical field of route planning. The application is aimed at solving the problem of low planning accuracy and efficiency of the existing carrier path planning method. The application comprises the following steps: obtaining the state of a current waypoint, inputting the state of the current waypoint into an optimized reinforcement learning network, and obtaining the decision variable of the current waypoint; and determining the carrier planning path according to the decision variables of all the waypoints. The application divides the waypoints into grades and takes into account the advantages of air transportation and automobile transportation, thereby improving the transportation efficiency and the accuracy of the optimal planning route. The application can enable the carrier to perform multiple delivery tasks at a time and improve the transportation efficiency. The application is used for planning the carrier transportation route.

Description

Technical Field

[0001] This invention relates to the field of route planning technology, and in particular to an online method for planning aircraft routes for combined air and land transport missions. Background Technology

[0002] With the rapid development of the low-altitude economy and the Internet of Things, the timeliness and economic efficiency of cargo transportation are becoming increasingly important. Against this backdrop, an unmanned aerial vehicle (UAV) has emerged, launched from a carrier aircraft and featuring a continuous power source, high lift-to-drag ratio, and a large internal cabin design. Compared to traditional fixed-wing UAVs, this aircraft is faster, flies farther, and has a larger payload. If a single carrier aircraft can carry multiple UAVs simultaneously, it can complete multiple transport missions in a single takeoff. Furthermore, by combining this with land transportation methods, it can achieve complementary air and land transport, enabling the transport of a wider variety of goods, significantly reducing costs and improving efficiency.

[0003] Currently, for the multi-target delivery mission planning problem of carrier aircraft, existing technologies first provide performance indicators for the delivery scheme through performance evaluation, including the analytic hierarchy process (AHP) and fuzzy evaluation method. However, this process is greatly affected by subjective human factors, and the solution efficiency of this method is limited, resulting in low accuracy and efficiency in path planning, making it unsuitable for online planning. Then, nonlinear programming algorithms are used to solve for the optimal delivery strategy. However, linear programming algorithms are insufficient in dealing with actual mission interference and unexpected situations, such as delivery interference, sudden weather, and detection interference, resulting in low accuracy in obtaining the optimal planned route. Neural network models are prone to getting trapped in local optima when solving such problems, making it difficult to obtain the optimal planned path. Summary of the Invention

[0004] The purpose of this invention is to address the problems of low accuracy and efficiency in existing aircraft path planning methods, and to propose an online aircraft path planning method for air-land joint transportation missions.

[0005] The online aircraft path planning method for joint air-land transport missions is as follows:

[0006] S1. Obtain the state of the current waypoint and input the state of the current waypoint into the optimized reinforcement learning network to obtain the decision variables of the current waypoint;

[0007] The status of the waypoints includes: the current waypoint number, the number of remaining transport aircraft, the air traffic control restrictions of the current waypoint, the deployment difficulty of the current waypoint, the weather conditions of the current waypoint, the communication conditions of the current waypoint, the remaining flight distance, and the historical deployment waypoints;

[0008] The reinforcement learning network includes: a classification agent and a ranking agent;

[0009] The classification agent is used to set waypoints as level 2 or level 3 waypoints, and to decide whether to travel from level n waypoint a to level n+1 waypoint b.

[0010] The sorting intelligent machine is used to determine the order in which the carrier aircraft passes through secondary waypoints;

[0011] The decision variable for the waypoint is 0 or 1; a decision variable of 0 indicates that the route from waypoint a to waypoint b is not selected; a decision variable of 1 indicates that the route from waypoint a to waypoint b is selected.

[0012] Where n takes the value of 1 or 2;

[0013] Among them, Level 1 waypoints are the starting points for aircraft, Level 2 waypoints are the target points for aircraft deployment, and Level 3 waypoints are waypoints that aircraft do not pass through.

[0014] S2. Determine the aircraft planning path based on the decision variables of all waypoints.

[0015] Furthermore, the optimized reinforcement learning network is obtained through the following method:

[0016] Step 1: Establish an optimization model for aircraft drop points, specifically as follows:

[0017] min J=J1+J2+J3

[0018] Wherein, J1 is the air traffic control situation encountered by the aircraft at the waypoint, J2 is the distance traveled by the car, J3 is the total flight distance of the aircraft, and J is the overall optimization index.

[0019] Step 2: Train a multi-agent reinforcement learning network based on the aircraft drop point optimization model until the multi-agent reinforcement learning network converges, thus obtaining the optimized reinforcement learning network.

[0020] Furthermore, the air traffic control J1 received by the aircraft at the waypoint is as follows:

[0021]

[0022] Where n is the waypoint class designation, N is the total number of waypoint classes, and M is the total number of waypoint classes. n F is the number of n-level waypoints. n ∈[0,1], F n These are the restrictions imposed on the aircraft at n-level waypoints;

[0023] The F n The value is the score for the current restriction; the current restriction is the flight time interval, flight altitude, flight range, or flight speed.

[0024] When restricted to a flight time interval, F nA larger value indicates a smaller flight time range; when limited by flight altitude, F n A larger value indicates a smaller flight altitude range; when restricted to a flight range, F n A larger value indicates a smaller flight range; when limited by flight speed, F n A higher value indicates a lower maximum flight speed; if there are multiple restrictions, the average score of each restriction is taken.

[0025] Furthermore, the driving distance J2 of the vehicle is as follows:

[0026]

[0027] Where i,j are the waypoint numbers, M n P is the number of n-level waypoints. n i,j ∈[0,1], P n i,j This refers to the efficiency of automobile transportation from waypoint i (n-level) to waypoint j (n+1-level). It is the distance from waypoint i (level n) to waypoint j (level n+1). It is a decision variable;

[0028] The vehicle transport efficiency refers to transport speed or cargo capacity.

[0029] Furthermore, the total range J3 of the carrier aircraft is as follows:

[0030]

[0031] Among them, U n+1,i ∈[0,1],U n+1,i It is the difficulty of deploying waypoint i at level n. Indicates the number of remaining aircraft transport vehicles.

[0032] The difficulty of deployment is a score based on wind force; the stronger the wind, the greater the difficulty of deployment.

[0033] Further, in step two, the multi-agent reinforcement learning network is trained based on the aircraft drop point optimization model until the multi-agent reinforcement learning network converges, thus obtaining the optimized reinforcement learning network. Specifically:

[0034] Step 2: 1. Initialize the agents and randomly initialize the reinforcement learning network for each agent;

[0035] Step 22: Set the initial state for the agent;

[0036] The state set S includes: the current waypoint number, the number of remaining air transport aircraft, the air traffic control restrictions of the current waypoint, the deployment difficulty of the current waypoint, the deployment difficulty of the current waypoint, the communication conditions of the current waypoint, the remaining flight distance, and the historical deployment waypoints;

[0037] Steps 2 and 3: Select and execute an action for each agent, and then calculate the state-action value function Q(s,a);

[0038] Step 24: The agent trains the reinforcement learning network according to the KL divergence based on the state, action, reward and new state until the reinforcement learning network converges, obtains the optimal reinforcement learning network parameters and the optimal policy, and obtains the trained reinforcement learning network.

[0039] Furthermore, the state-action value function Q(s) j' ,a j' Specifically:

[0040] Q(s j' ,a j' )=Ε[R(s j' ,a j' )|S t =s j' A t =a j' ]

[0041] Where Q(s) j' ,a j' ) is state s j' Next action a j' State-action value function, R(s) j' ,a j' ) is state s j' Next, execute action a j' The reward, s j' It is the joint state of the j'th iteration, a j' It is the action of the j'th iteration, S t It is the state of the agent at time t, A t It refers to the action of the intelligent agent at time t.

[0042] Furthermore, in step two and four, the agent trains a reinforcement learning network based on the state, action, reward, and new state using KL divergence, employing the following objective function:

[0043]

[0044] R(s j' ,a j' ) = J

[0045] Among them, K HHere are hyperparameters: N' is the total number of agents, H represents entropy, and π... i' Let H(π) be the current policy of agent i'. i' (·|s j' )) is the state of the agent s j' The policy entropy at time ζ j' It is the discount coefficient of the j'th iteration, π is the set of agent policies, J(π(·|s j' )) is the objective function, and A is the set of actions of the agent.

[0046] Furthermore, the agent's policy is updated using the following formula:

[0047]

[0048] Among them, D KL For KL divergence, π j'+1 It is the strategy at step j'+1, i' p Here, Q represents the specific index of agent i' in the agent sorting sequence, Q represents the state-action value function, and Z is the partition function. It is state s j' The next number is i' p The agent's strategy, It is except for the action The set of all other actions. The sequence number is i' p The actions of the intelligent agent.

[0049] Furthermore, the convergence condition of the reinforcement learning network is as follows:

[0050]

[0051] in, It is the optimal strategy for all agents except agent i', π i' It is the optimal strategy for agent i'. Is the agent i' adopting strategy π? i' The joint objective function when other agents adopt their corresponding optimal strategies, a -i' It is the action space excluding action a i' All other action sets besides b i' It's a hyperparameter, A i' It is the set of actions of agent i'.

[0052] The beneficial effects of this invention are as follows:

[0053] This invention proposes an online aircraft path planning method for air transport delivery missions. It establishes an optimal drop point model for multiple air transport missions, and combines this model with a maximum entropy-improved multi-agent reinforcement learning algorithm to enable aircraft to depart from any airport and deliver air transports to all ground targets, thus achieving online optimal drop point planning. The invention considers air traffic control, distance, and range factors when establishing the optimal drop point model, and classifies targets into secondary and tertiary waypoints, taking into account the combined transportation advantages of air transport and automobiles. This allows the aircraft to perform multiple delivery missions simultaneously, thereby improving transportation efficiency and the accuracy of the optimal planned route. This method, based on a maximum entropy-improved multi-agent reinforcement learning algorithm, achieves aircraft path planning, improves planning efficiency, and avoids getting trapped in local optima, thus obtaining the optimal planned route. Attached Figure Description

[0054] Figure 1 A schematic diagram of an optimized model for aircraft drop-off points in combined air-land transport.

[0055] Figure 2 The flowchart shows the reinforcement learning algorithm for maximum entropy agents. Detailed Implementation

[0056] Specific Implementation Method 1: The specific process of the online aircraft path planning method for combined air-land transport missions in this implementation method is as follows:

[0057] S1. Obtain the state of the current waypoint and input the state of the current waypoint into the optimized reinforcement learning network to obtain the decision variables of the current waypoint;

[0058] The status of the waypoints includes: the current waypoint number, the number of remaining transport aircraft, the air traffic control restrictions of the current waypoint, the deployment difficulty of the current waypoint, the weather conditions of the current waypoint, the communication conditions of the current waypoint, the remaining flight distance, and the historical deployment waypoints;

[0059] The reinforcement learning network includes: a classification agent and a ranking agent;

[0060] The classification agent is used to set waypoints as level 2 or level 3 waypoints, and to decide whether to travel from level n waypoint a to level n+1 waypoint b.

[0061] The sorting intelligent machine is used to determine the order in which the carrier aircraft passes through secondary waypoints;

[0062] The decision variable being 0 indicates that the route from waypoint a to waypoint b is not selected;

[0063] Among them, waypoint a is a waypoint of class n, and waypoint b is a waypoint of class n+1;

[0064] The decision variable being 1 indicates that the route is from waypoint a to waypoint b.

[0065] The optimized reinforcement learning network is obtained through the following method:

[0066] Step 1, such as Figure 1 As shown, an optimization model for the aircraft launch point is established:

[0067] min J=J1+J2+J3

[0068]

[0069] Where J1 is the air traffic control situation the aircraft encounters at the waypoint, J2 is the distance the vehicle travels, J3 is the total flight distance of the aircraft, J is the overall optimization index, n is the waypoint class number, N is the total number of waypoint classes, and M... n F is the number of n-level waypoints. n This refers to the restrictions imposed on the aircraft at level n waypoints, where i and j are the waypoint numbers, and P... n i,j This refers to the efficiency of automobile transportation from waypoint i (n-level) to waypoint j (n+1-level). It is the distance from waypoint i of class n to waypoint j of class (n+1) It is a decision variable. Y takes the value 0 or 1; when waypoint i of class n reaches waypoint j of class (n+1) n i,j =1, otherwise Y n i,j =0;U n+1,i It is the difficulty of deploying waypoint i at level n. Indicates the number of remaining aircraft transport vehicles.

[0070] The vehicle transportation efficiency P n i,j P represents the normalized transport speed or cargo capacity. n i,j ∈[0,1];

[0071] The difficulty of deployment U n+1,i This refers to the current rating of wind strength by technical personnel; U n+1,i ∈[0,1];U n+1,i The larger the value, the stronger the wind; for example, wind: level 1-3 wind is relatively light, one score; level 4-6 wind is slightly stronger, one score; level 7-10 wind is very strong, one score.

[0072] Where M1 = 1 and F1 = 0; the carrier aircraft is a first-order waypoint, and the carrier aircraft is subject to the restriction F nThe score is given for the flight time interval, flight altitude, flight range, or flight speed; F n ∈[0,1], when restricted to a flight time interval, F n A larger value indicates a smaller flight time range; when limited by flight altitude, F n A larger value indicates a smaller flight altitude range; when restricted to a flight range, F n A larger value indicates a smaller flight range; when limited by flight speed, F n A higher value indicates a lower maximum flight speed; if there are multiple restrictions, the average of the scores for each restriction is taken.

[0073] Automobile transportation efficiency mainly refers to transportation speed or cargo capacity, which is a relative value and dimensionless. If normalized to 0-1, this value may be 0.11 or 0.98.

[0074] In this step, only values ​​of n ranging from 1 to N-1 are added to formula J2, without considering the case where n = N;

[0075] The aircraft launch rules are set in this step as follows:

[0076] 1) The carrier aircraft departs from an airport and drops air transport vehicles to targets along the route;

[0077] 2) The carrier aircraft is the starting point of the entire flight path, and is called the first-order waypoint;

[0078] 3) Divide the track into secondary and tertiary waypoints based on the target;

[0079] 4) The aircraft only passes through secondary waypoints;

[0080] 5) The carrier aircraft drops the air transport vehicle to the target near the secondary waypoint. The transport from the secondary waypoint to the tertiary waypoint is completed by vehicle. Therefore, the carrier aircraft does not pass directly through the tertiary waypoint.

[0081] 6) Each secondary waypoint may have zero to multiple tertiary waypoints.

[0082] Step Two, as follows Figure 2 As shown, a multi-agent reinforcement learning network based on maximum entropy improvement is trained based on the aircraft drop point optimization model until the multi-agent reinforcement learning network converges, thus obtaining the optimized reinforcement learning network:

[0083] Step 2.1: Initialize the classification agent and ranking agent in the reinforcement learning network:

[0084] The classification agent is used to set waypoints as level 2 or level 3 waypoints, and to decide whether to travel from level n waypoint a to level n+1 waypoint b.

[0085] The sorting intelligent machine is used to determine the order in which the carrier aircraft passes through secondary waypoints;

[0086] The parameters of the reinforcement learning network include: joint policy, hyperparameters, discount factor, and number of agents;

[0087] Step 22: Set the initial state for the agent:

[0088] The state set S includes: the current waypoint number, the number of remaining transport aircraft, the air traffic control restrictions of the current waypoint, the deployment difficulty of the current waypoint, the weather conditions of the current waypoint, the communication conditions of the current waypoint, the remaining flight distance, and the historical deployment waypoints;

[0089] Steps two and three: Select and execute actions for each agent, then calculate the state-action value function Q(s). j' ,a j' ):

[0090] Q(s j' ,a j' )=Ε[R(s j' ,a j' )|S t =s j' A t =a j' ]

[0091] Where Q(s) j' ,a j' ) is state s j' Next action a j' State-action value function, R(s) j' ,a j' ) is state s j' Next, execute action a j' The reward, s j' It is the joint state of the j'th iteration, a j' It is the action of the j'th iteration, S t It is the state of the agent at time t, A t It is the action of the agent at time t;

[0092] The action is a decision variable.

[0093] Step 24: The agent trains the reinforcement learning network according to the KL divergence based on the state, action, reward and new state until the reinforcement learning network converges, obtains the optimal reinforcement learning network parameters and the optimal policy, and obtains the trained reinforcement learning network.

[0094] The multi-agent training model is as follows:

[0095]

[0096] Among them, J(π(·|s j' )) is the objective function, j' is the iteration number label, s j' Let R be the joint state in the j'-th iteration of training, and let a be the joint reward function. j' Here, π is the joint action in the j'th iteration of training, A is the finite joint action space, S is the finite joint state space, and R(s) is the joint action. j' ,a j' ) is the state of the agent s j' When performing action a j' The reward, ζ j' is the discount coefficient of the j'th iteration, and P is the state transition matrix;

[0097] The challenge of multi-agent training lies in coordinating the optimization processes of individual and multiple agents to ensure the objective function monotonically increases and reaches the global optimum. For example, a single agent might "sacrifice" itself to achieve this goal, gathering with other agents and getting trapped in a local optimum. To address this, a maximum entropy term is introduced to improve the training objective function, allowing each agent to approximate the optimal policy based on KL divergence, thus achieving joint policy optimization.

[0098]

[0099] R(s j' ,a j' ) = J

[0100] Among them, K H Here are hyperparameters: N' is the total number of agents, H represents entropy, and π... i' Let H(π) be the current policy of agent i'. i' (·|s j' )) is the state of the agent s j' The policy entropy at time, where A is the set of agent actions;

[0101] The agent's policy is updated in the following ways:

[0102] Let each agent continuously approach the optimal strategy using the KL divergence method, and denote the sequence i' 1:n For a multi-agent sorted sequence, i' p (p = 1, 2, ..., n) is the specific index of agent i'. Then, the new policy of the agent in the next step, i.e., the (j'+1)th step, is:

[0103]

[0104] Among them, D KL For KL divergence, π j'+1 It is the strategy at step j'+1, i' pis the specific index of agent i' in the agent sorting sequence, Q represents the state-action value function, and Z is the partition function that normalizes the data distribution. It is state s j' The next number is i' p The agent's strategy, It is except for the action The set of all other actions. The sequence number is i' p The actions of the intelligent agent.

[0105] The convergence conditions for reinforcement learning networks are as follows:

[0106] If all agents cannot improve the joint objective by adjusting their own policies, then the joint policy is considered to have reached response equilibrium, denoted as π. * :

[0107]

[0108] in, π is the optimal policy of all agents except agent i', where π is the set of agent policies. i' It is the optimal strategy for agent i';

[0109] The optimal strategy for agent i' is:

[0110]

[0111] Among them, a -i' In the action space, besides action a i' All other action sets besides b i' It's a hyperparameter, A i' It is the set of actions of agent i'.

[0112] S2. Determine the aircraft planning path based on the decision variables of all waypoints;

[0113] The planned path for the aircraft is the path where the decision variable for all waypoints in the path is 1.

[0114] The optimization variables in this invention can be categorized into two types: classification problems, i.e., dividing secondary and tertiary waypoints according to the objective, and ranking problems, i.e., determining the order in which secondary waypoints are flown. For these two types of problems, this invention employs two reinforcement learning agents to solve them separately, forming a multi-agent reinforcement learning architecture. The multiple agents execute joint actions based on a joint policy and state, obtaining discounted joint rewards. The joint policy is iterated with the objective of maximizing the expected cumulative reward until all agents can no longer improve the objective function. This invention addresses the multiple types of nonlinear programming problems involved in secondary / tertiary waypoint optimization, employing different reinforcement learning agents to solve them, forming a multi-agent reinforcement learning architecture. Simultaneously, a maximum entropy multi-agent learning improvement strategy is proposed to increase the randomness of training and thus avoid getting trapped in local optima.

Claims

1. An online aircraft path planning method for combined air-land transport missions, characterized in that... The specific process of the method is as follows: S1. Obtain the state of the current waypoint and input the state of the current waypoint into the optimized reinforcement learning network to obtain the decision variables of the current waypoint; The status of the waypoints includes: the current waypoint number, the number of remaining transport aircraft, the air traffic control restrictions of the current waypoint, the deployment difficulty of the current waypoint, the weather conditions of the current waypoint, the communication conditions of the current waypoint, the remaining flight distance, and the historical deployment waypoints; The reinforcement learning network includes: a classification agent and a ranking agent; The classification agent is used to set waypoints as secondary or tertiary waypoints, and to decide whether to use an nth-level waypoint. Heading to n+1 level waypoint and output the decision variables; The sorting agent is used to determine the order in which the carrier aircraft passes through secondary waypoints; The decision variable for the waypoint is either 0 or 1; a decision variable of 0 indicates that the waypoint is not selected. to waypoint The decision variable being 1 indicates that the choice is made from the waypoint. to waypoint ; Where n takes the value of 1 or 2; Level 1 waypoints are the aircraft's starting point, Level 2 waypoints are the aircraft's launch target points, and Level 3 waypoints are waypoints that the aircraft will not pass through. S2. Determine the aircraft planning path based on the decision variables of all waypoints.

2. The online aircraft path planning method for joint air-land transport missions according to claim 1, characterized in that: The optimized reinforcement learning network is obtained through the following method: Step 1: Establish an optimization model for aircraft drop points, specifically as follows: in, J1 represents the air traffic control conditions encountered by the aircraft at the waypoint, J2 represents the distance traveled by the car, J3 represents the total flight distance of the aircraft, and J represents the overall optimization index. Step 2: Train a multi-agent reinforcement learning network based on the aircraft drop point optimization model until the multi-agent reinforcement learning network converges, and obtain the optimized reinforcement learning network.

3. The online aircraft path planning method for joint air-land transport missions according to claim 2, characterized in that: The aircraft is subject to air traffic control at waypoints. as follows: in, These are waypoint classification numbers. This is the total number of waypoints. It is the number of n-level waypoints. , These are the restrictions imposed on the aircraft at n-level waypoints; The The value is the score for the current restriction; the current restriction is the flight time interval, flight altitude, flight range, or flight speed. When restricted to a flight time range A larger value indicates a smaller flight time range; when limited by flight altitude, A larger value indicates a smaller flight altitude range; when restricted to a specific flight range, A larger value indicates a smaller flight range; when the limitation is flight speed, A higher value indicates a lower maximum flight speed; if there are multiple restrictions, the average score of each restriction is taken.

4. The online aircraft path planning method for joint air-land transport missions according to claim 3, characterized in that: The travel distance J2 of the vehicle is as follows: in, These are waypoint numbers. It is the number of n-level waypoints. , This refers to the efficiency of automobile transportation from waypoint i (n-level) to waypoint j (n+1-level). It is the distance from waypoint i (level n) to waypoint j (level n+1). It is a decision variable; The vehicle transport efficiency refers to transport speed or cargo capacity.

5. The online aircraft path planning method for joint air-land transport missions according to claim 4, characterized in that: The total range J3 of the carrier aircraft is as follows: in, , It is the difficulty of deploying waypoint i at level n. Indicates the number of remaining aircraft transport vehicles. The difficulty of deployment is a score based on wind force; the stronger the wind, the greater the difficulty of deployment.

6. The online aircraft path planning method for air-land joint transportation missions according to claim 5, characterized in that: Step two involves training a multi-agent reinforcement learning network based on the aircraft drop point optimization model until the multi-agent reinforcement learning network converges, thus obtaining the optimized reinforcement learning network. Specifically: Step 2: Initialize the classification agent and ranking agent in the reinforcement learning network; Step 22: Set the initial state for the agent; The state set S includes: the current waypoint number, the number of remaining air transport aircraft, the air traffic control restrictions of the current waypoint, the deployment difficulty of the current waypoint, the deployment difficulty of the current waypoint, the communication conditions of the current waypoint, the remaining flight distance, and the historical deployment waypoints; Steps 2 and 3: Select and execute an action for each agent, then calculate the state-action value function. ; Step 24: The agent trains the reinforcement learning network according to the KL divergence based on the state, action, reward and new state until the reinforcement learning network converges, obtains the optimal reinforcement learning network parameters and the optimal policy, and obtains the trained reinforcement learning network.

7. The online aircraft path planning method for joint air-land transport missions according to claim 6, characterized in that: The state-action value function Specifically: in, It is a state Next action State-action value function It is a state Next action The reward It is the first The joint state of the step iterations, It is the first Step-by-step iterative action, It is the state of the agent at time t. It refers to the action of the intelligent agent at time t.

8. The online aircraft path planning method for air-land joint transportation missions according to claim 7, characterized in that: In step two-four, the agent trains a reinforcement learning network based on the state, action, reward, and new state according to the KL divergence, using the following objective function: Among them, K H For hyperparameters, Let H represent the total number of agents, and H represent entropy. For intelligent agents The current strategy, The agent's state is Strategy entropy at time It is the first Discount factor for each iteration It is a set of agent policies. It is the objective function. It is a set of actions of an intelligent agent.

9. The online aircraft path planning method for joint air-land transport missions according to claim 8, characterized in that: The agent's policy is updated using the following formula: Among them, D KL For KL divergence, It is the first Step strategy, It is an agent in the sorted sequence of agents. The specific sequence number, Q represents the state-action value function, and Z is the partition function. It is a state The next number is The agent's strategy, It is except for the action The set of all other actions. The serial number is The actions of the intelligent agent.

10. The online aircraft path planning method for air-land joint transportation missions according to claim 9, characterized in that: The conditions for the convergence of the reinforcement learning network are as follows: in, It is except intelligent agents The optimal strategy for other intelligent agents, It is an intelligent agent The optimal strategy It is an intelligent agent Take strategy The joint objective function when other agents adopt their corresponding optimal strategies. It is the space of motion excluding the action All other action sets, It's a hyperparameter. It is an intelligent agent A set of actions.

Images