A Multi-Agent Simulation Method for Air Taxi Combining Double-DQN Network

Abstract

This invention discloses a multi-agent simulation method for air taxis combining a Double-DQN network. The method includes constructing a low-to-high-level composite urban air taxi traffic network model based on urban geographic information data; then building a fixed-time-step multi-agent simulation system based on the constructed air taxi agent model and station passenger behavior model; constructing an air taxi scheduling model; training the air taxi scheduling model within the multi-agent simulation system; deploying the trained air taxi scheduling model into the multi-agent simulation system; running the simulation based on the low-to-high-level composite urban air taxi traffic network model; and outputting the simulation results of air taxi operation. This invention constructs a two-layer urban air traffic network, employs multiple air taxi agents to finely simulate the entire process behavior, and combines optimized scheduling strategies to achieve realistic simulation and efficiency improvement of air taxi operation.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, and specifically relates to a multi-agent simulation method for air taxis that combines a Double-DQN network. Background Technology

[0002] With the rapid advancement of technology, electric vertical takeoff and landing (eVTOL) aircraft are gradually moving from technology verification to commercial application. Air taxis, as an important application of eVTOL, can, to some extent, alleviate the inconvenience caused by traffic congestion in large cities, meeting the needs of urban residents for convenient and rapid travel. Against this backdrop, developing a multi-agent simulation method for air taxis to accurately simulate their operational mechanisms and objectively and comprehensively reflect their operational status will become an important tool for enterprise decision-making.

[0003] Currently, multi-agent simulation technology has been widely applied in various scenarios. In the transportation sector, it has been applied to multiple areas such as manned / driverless ride-hailing, taxis, and low-altitude logistics. These simulation methods can directly model the basic elements constituting the transportation system, simulating macroscopic traffic phenomena through local interactions between individuals, thus more realistically reflecting traffic conditions. Furthermore, multi-agent simulation can test technologies that have not yet been implemented risk-free and at low cost, avoiding the costs of accidents and waste caused by decision-making errors in real-world experiments. Multi-agent simulation technology can also provide decision-makers with multi-level, visualized analytical results, facilitating their analysis of traffic conditions.

[0004] In the field of air traffic simulation, existing research mainly focuses on air traffic control systems and UAV route planning, while research on multi-agent simulation of urban air taxis is still in its early stages. Most existing methods use simplified arrival service queuing models, treating air taxis as a "black box" and ignoring their actual behaviors such as takeoff and landing, low-altitude obstacles, multi-pathway orders, charging and maintenance, leading to significant deviations between simulation results and reality. In addition, traditional methods are difficult to adaptively learn and optimize scheduling strategies, and cannot cope with complex decision-making problems such as dynamic order allocation, multi-vehicle coordination, and the balance between revenue and efficiency. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention provides a multi-agent simulation method for air taxis that combines a Double-DQN network, solving the technical problems of neglecting the low-altitude environment and the difficulty in describing the individual behavior of air taxis in the prior art.

[0006] The technical solution adopted in this invention is:

[0007] A multi-agent simulation method for air taxis combining Double-DQN networks:

[0008] S1. Obtain urban geographic information data and construct a low- to high-rise composite urban air taxi transportation network model based on the urban geographic information data.

[0009] S2. Construct an intelligent agent model for air taxis and a passenger behavior model for stations, and build a multi-agent simulation system with a fixed time step based on the intelligent agent model for air taxis and the passenger behavior model for stations.

[0010] S3. A Double-DQN network is used to construct an air taxi scheduling model, and the air taxi scheduling model is trained in a multi-agent simulation system to obtain a trained air taxi scheduling model.

[0011] S4. Deploy the trained air taxi dispatching model to the multi-agent simulation system, run the simulation based on the low-level-high-level composite urban air taxi traffic network model, and output the simulation results of air taxi operation.

[0012] Step S1 specifically involves:

[0013] S11. Divide the urban low-altitude airspace into low-level airspace and high-level airspace according to the preset altitude threshold, and obtain environmental data at the cruising altitude of the low-level airspace and high-level airspace respectively, and generate a first matrix describing the environment of the low-level airspace and a second matrix describing the environment of the high-level airspace.

[0014] The environmental data includes urban building and airspace control GIS data (no-fly zone data).

[0015] S12. Based on the first and second matrices, a path planning algorithm is used to plan the shortest route for any two take-off and landing stations in the lower and upper airspace, respectively.

[0016] S13. Optimize the nodes of each shortest route, extract the nodes where the path direction changes as valid nodes, and construct the first undirected connected graph in the lower airspace and the second undirected connected graph in the upper airspace based on all valid nodes and connecting edges.

[0017] S14. Supplement the altitude coordinates of each node in the first and second undirected connected graphs with their respective cruising altitudes, and connect the nodes with the same horizontal coordinates to generate a low-to-high-level composite urban air taxi transportation network model.

[0018] The multi-agent simulation system includes multiple air taxi agents, each generated by instantiating an air taxi agent model. The multi-agent simulation system uses a low- to high-level composite urban air taxi traffic network model to provide a basis for path planning for the air taxi agents.

[0019] Step S2 specifically involves:

[0020] S21. A hierarchical finite state machine is used to model the behavior of air taxis throughout their entire lifecycle, and an intelligent agent model of air taxis is constructed.

[0021] S22. Use a queuing model to model the behavior of passengers at the station and build a passenger behavior model for the station.

[0022] S23. Based on the constructed air taxi intelligent agent model and the constructed station passenger behavior model, build a multi-agent simulation system with a fixed time step, and load the low-level-high-level composite urban air taxi traffic network model into the multi-agent simulation system.

[0023] The hierarchical finite state machine in step S21 includes two main states: air and ground. The air main state includes sub-states such as takeoff, heading to the destination, and landing. The ground main state includes sub-states such as dropping off passengers, picking up passengers, charging and maintenance, standby, and scheduling.

[0024] The passenger arrival events corresponding to the station passenger behavior in step S22 are generated by Poisson distribution, and the destination station is assigned to the passenger according to the travel distribution matrix between the take-off and landing stations to form an order; the generated order enters the queuing system of the station and is cancelled with a preset probability according to the waiting time;

[0025] The travel distribution matrix is ​​an n×n matrix, where n is the number of departure and arrival stations. The value in the i-th row and j-th column of the matrix represents the number of passengers departing from station i and traveling to station j. The queuing system is a list of unlimited length that stores order data in the order of order arrival.

[0026] Step S3 specifically involves:

[0027] S31. Construct an air taxi scheduling model using a Double-DQN network and set the reward function for the air taxi scheduling model.

[0028] S32. Train the air taxi dispatching model in the multi-agent simulation system to obtain the trained air taxi dispatching model.

[0029] The air taxi dispatching model is deployed to the multi-agent simulation system, and the network parameters are iteratively updated using an experience replay mechanism and a mean square error loss function.

[0030] In step S31:

[0031] The input to the air taxi scheduling model is a state matrix; the output of the air taxi scheduling model is a Q-value vector representing the actions of the air taxi agent in selecting and transferring to various take-off and landing stations.

[0032] The state matrix contains at least the following information: the shortest distance between the current air taxi and each take-off and landing station, the number of passengers queuing at each take-off and landing station, the estimated total time required to complete the current order at each take-off and landing station, the number of air taxis currently heading to each take-off and landing station, and the number of air taxis currently waiting at each take-off and landing station.

[0033] The reward function is set according to the following formula:

[0034]

[0035] in, Indicates the reward value; , and All are weighting coefficients; This indicates the total waiting time for passengers; Indicates the order cancellation rate; This represents the total length of the flight paths of all air taxis. and These correspond to the mean and standard deviation of the variables, respectively.

[0036] Step S4 specifically involves:

[0037] S41. Load the low-level-high-level composite urban air taxi traffic network model constructed in step S1 to initialize the simulation environment of the multi-agent simulation system and initialize the initial state of all air taxi agents.

[0038] S42. The simulation proceeds according to a fixed time step. In each time step, the trained air taxi scheduling model provides scheduling decisions for the air taxi agents in the standby sub-state, updates the status of all air taxi agents and system order data, until the simulation ends, and outputs statistical reports of various preset indicators.

[0039] The present invention has the following beneficial effects:

[0040] 1. This invention utilizes topology optimization to construct a low- to high-level composite urban air taxi transportation network model. By performing layered processing of urban low-altitude airspace and extracting effective nodes to construct a connected graph, the path search space is significantly compressed while ensuring flight safety (avoiding buildings and no-fly zones), greatly improving the route-finding efficiency of air taxis. At the same time, this network model can comprehensively reflect the topological structure of urban low-altitude transportation.

[0041] 2. This invention can describe more complex individual behaviors. It employs a hierarchical finite state machine to perform refined modeling of the entire lifecycle behavior of each air taxi, decomposing the operation process into two main states—air and ground—and multiple sub-states. Each air taxi is simulated as an independent intelligent agent with independent state variables, enabling realistic simulation of complex behaviors such as multi-pathway order execution and charging / maintenance timing selection, significantly improving the realism and credibility of the simulation results.

[0042] 3. This invention fully considers the scheduling optimization problem of air taxis. An air taxi scheduling optimization model is constructed using a Double-DQN network. By designing a reward function that comprehensively considers passenger waiting time, order cancellation rate, and flight path length, the agent can autonomously learn and optimize scheduling strategies in a simulation environment. This effectively simulates and optimizes the dynamic scheduling behavior of air taxis between take-off and landing stations, improving the overall system operating efficiency.

[0043] 4. This invention provides a complete set of trainable and scalable simulation methods that support multi-scenario simulation by connecting to actual or synthetic order flows. It can also output multi-dimensional statistical indicators such as passenger waiting time, order completion rate, and vehicle utilization rate and generate visualization results, providing quantitative decision support for air taxi operation strategy research and infrastructure planning. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention.

[0045] Figure 2 It is an undirected connected graph of the lower and upper airspace.

[0046] Figure 3 This is an example diagram of a low-rise-high-rise complex urban air taxi transportation network.

[0047] Figure 4 This is a schematic diagram of a hierarchical finite state machine used to describe the behavioral logic of the air taxi intelligent agent of the present invention.

[0048] Figure 5 This is a structural diagram of the Double-DQN network used in Example 1.

[0049] Figure 6 This is a diagram showing the order data structure for each order within the air taxi simulation system.

[0050] Figure 7 This is a sequence diagram of event execution within each time step of a multi-agent simulation system. Detailed Implementation

[0051] The present invention will now be described in more detail with reference to the accompanying drawings and embodiments. However, the present invention is not limited thereto. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

[0052] Air taxis are low-altitude air transportation vehicles that use electric vertical takeoff and landing technology. They operate in urban low-altitude airspace, taking off and landing from fixed takeoff and landing platforms, providing passengers with transportation services from one takeoff and landing station to another.

[0053] Example 1:

[0054] like Figure 1 As shown, the multi-agent simulation method for air taxis in this embodiment is implemented according to the following steps:

[0055] S1. Obtain urban geographic information data and construct a low- to high-rise composite urban air taxi transportation network model based on the urban geographic information data.

[0056] Urban geographic information data includes urban building and airspace control GIS data (no-fly zone data).

[0057] S11, using a preset height threshold The urban low-altitude airspace (in this embodiment, the urban low-altitude airspace is the flight area below 1000 meters above ground level) is divided into low-level airspace and high-level airspace using a dividing line; the cruising altitudes of air taxis in the low-level and high-level airspaces are set as follows: and The altitudes were obtained respectively. and Using GIS data on urban buildings and airspace control (no-fly zone data), the operating area of ​​urban air taxis is divided into grids. Based on the presence or absence of urban buildings or no-fly zones in each grid, a first matrix describing the lower-level airspace environment is obtained. The second matrix describing the high-altitude airspace environment ,in:

[0058]

[0059]

[0060] in, and All are indexes; Represents the first matrix The Middle Line 1 The values ​​of the grid (whether there are buildings or no-fly zones); Represents the second matrix The Middle Line 1 The values ​​of the column grid (whether there are buildings or no-fly zones). In this embodiment, The setting is 300 meters.

[0061] S12. Obtain the coordinates of all take-off and landing stations of the city's air taxis. Any two take-off and landing stations form a take-off and landing station pair.

[0062] For each takeoff and landing station pair, based on the first matrix Second matrix The A* algorithm was used to calculate the shortest routes between takeoff and landing station pairs in both the lower and upper airspace, respectively, resulting in path matrices representing the corresponding takeoff and landing station pairs in the lower airspace. Path matrix between corresponding take-off and landing station pairs in high-altitude airspace .

[0063] All takeoff and landing station pairs are processed to obtain a path matrix representing the relationships between several takeoff and landing station pairs in the low-altitude airspace. Path matrix between several take-off and landing station pairs in high-altitude airspace .

[0064] Each path matrix is ​​an ordered array that stores a sequence of path point coordinates. ,in and The coordinates of the origin and destination stations connected by this path. The coordinates of the intermediate path points. These are collectively referred to as path point coordinates.

[0065] S131. For each path matrix, in the path point coordinate sequence The coordinates of three consecutive path points are retrieved by sliding the traversal from left to right. Then calculate the vector and The angle between :

[0066]

[0067] like If it is 0, then Invalid node; if If it is not 0, then This is a valid node.

[0068] Extract path point coordinate sequence Except for the starting point and the end point All valid nodes except the starting point. All valid nodes and endpoints By connecting the coordinates of the points in the path in sequence, a route based on graph representation is obtained, which connects the edges between two valid nodes.

[0069] S132. Process all path matrices using the same method as in step S131 to obtain several graph-based routes. Overlay all graph-based routes, remove duplicate valid nodes and edges, and obtain two undirected connected graphs, one of which is the first undirected connected graph in the lower spatial domain. This represents the topology of the air taxi transportation network in the lower airspace, such as... Figure 2 As shown on the left. Another undirected connected graph is the second undirected connected graph in the upper spatial domain. This illustrates the topology of the air taxi transportation network in high-altitude airspace, such as... Figure 2 As shown on the right.

[0070] S14 is the first undirected connected graph. Second undirected connected graph Each node is supplemented with its corresponding altitude coordinates (i.e., z-axis coordinates) at the cruising altitude, connecting the first undirected connected graph. Second undirected connected graph Nodes with the same horizontal coordinates are used to obtain a complete low- to high-rise urban air taxi transportation network. ,like Figure 3 As shown.

[0071] S2. Construct an intelligent agent model for air taxis and a passenger behavior model for stations, and build a multi-agent simulation system with a fixed time step based on the intelligent agent model for air taxis and the passenger behavior model for stations.

[0072] The multi-agent simulation system includes multiple air taxi agents, each generated by instantiating an air taxi agent model. The multi-agent simulation system uses a low- to high-level composite urban air taxi traffic network model to provide a basis for path planning for the air taxi agents.

[0073] In practice, each air taxi is treated as an intelligent agent. During the simulation system initialization phase, based on the preset total number of air taxis and the initial distribution of takeoff and landing stations, a corresponding number of intelligent agent objects are generated by calling the air taxi intelligent agent model instantiation interface. Each instantiated air taxi intelligent agent has its own independent memory space to store its own state variables, including current location, current state, remaining battery power, target station, etc. However, all intelligent agents share the same set of behavioral logic defined by a hierarchical finite state machine. The intelligent agents interact indirectly through the shared simulation environment, for example, by influencing the station queuing status or competing for takeoff and landing platform resources, thus collectively forming a complete multi-agent simulation system for air taxis.

[0074] S21. A hierarchical finite state machine is used to model the behavior of air taxis throughout their entire lifecycle, and an intelligent agent model of air taxis is constructed.

[0075] The hierarchical finite state machine includes two main states: air and ground. The air main state includes sub-states such as takeoff, heading to the destination, and landing. The ground main state includes sub-states such as dropping off passengers, picking up passengers, charging and maintenance, standby, and scheduling. A sub-state refers to the new state to which a process transitions after certain conditions are met.

[0076] In this embodiment, the following is adopted: Figure 4 The hierarchical finite state machine shown describes the behavior of the air taxi agent.

[0077] Specifically, the state transition rules of a hierarchical finite state machine are as follows: when a state ends, a transition action is executed. If the current state has one and only one selectable substate, then the transition occurs to that substate. For example, after the takeoff substate ends, the transition proceeds to the destination substate; after the destination substate ends, the transition proceeds to the landing substate.

[0078] If the current state has multiple selectable sub-states, the sub-state is selected according to preset rules. The transition rules for sub-states with multiple selectable sub-states are as follows: After the landing sub-state ends, the air taxi selects to transfer to either the passenger drop-off or passenger pick-up sub-state based on the requirements of the current service order; after the passenger drop-off sub-state ends, if the order is not completed, it transfers to the takeoff sub-state, otherwise it transfers to the charging and maintenance sub-state; after the charging and maintenance sub-state ends, it transfers to the standby sub-state; after the standby sub-state ends, if there is an order at the current takeoff and landing station, it accepts the order and transfers to the passenger pick-up sub-state, otherwise it transfers to the dispatch sub-state.

[0079] The scheduling sub-state executes the scheduling model of the air taxi agent (Double-DQN network), determines the target station based on the output of the scheduling model, and transitions to the destination sub-state after the end of this sub-state.

[0080] The destination sub-state uses the A* algorithm to plan the route from the current location to the target station in the low-to-high-level composite urban air taxi transportation network obtained in step S1, and instructs the air taxi agent to travel along the planned route.

[0081] Except for the destination sub-state, the execution process of the other sub-states is simulated by calculating the degree of completion. Specifically, it is assumed that the total time required to fully execute the sub-state is... The current step has been executed for [duration]. If within the current time step Then Set as ,in Indicates the time step; conversely, if within the current time step... If the substate is completed, the next substate will be executed.

[0082] S22. Use a queuing model to model passenger behavior at the station and construct a passenger behavior model for the station.

[0083] The passenger behavior model at the stations is a queuing model, and passenger arrival behavior is characterized by a Poisson distribution. Air taxi take-off and landing stations are defined. ( The average passenger arrival rate (indexed) is (Unit: people / hour). The time step of the multi-agent simulation system is a fixed value T (unit: hours). Then, within each time step, the number of people arriving at the station... The number of passengers is a random variable Its probability distribution is:

[0084]

[0085] in, It means that exactly one time step has Passengers arrived at the station The probability, This site was obtained based on historical data statistics. The average passenger arrival rate, where T is the simulation time step. Represents the natural constant.

[0086] Within each simulation time step, random sampling is performed according to the Poisson distribution described above to obtain the arrival times at each takeoff and landing station within that time step. Number of passengers For each takeoff and landing station, the destination will be... The passengers are randomly divided into m groups, with the number of passengers in each group being... The following conditions must be met:

[0087] ;

[0088] in, Also an index, Indicates the first The number of passengers in each group This represents the maximum passenger capacity of the air taxi. A multi-agent simulation system dispatches m orders to this takeoff and landing station, with each order containing the following number of passengers: For each order, a destination station is randomly assigned to each passenger within the order based on the travel distribution matrix (OD matrix) between the departure and arrival stations. This results in order data containing multiple passengers, the destination station for each passenger, and the corresponding station behavior for each passenger. The order data structure is as follows: Figure 6 As shown.

[0089] The generated orders will be placed into the queuing system of the respective departure and arrival stations. The queuing system is an unlimited-length list that stores order data in the order of arrival. At each time step, each order in the queuing system is assigned a probability... The probability of cancellation depends on the order's waiting time. Decide.

[0090]

[0091] in, The length of time passengers wait in the queuing system. The probability of cancellation is The constants between, This is a preset threshold.

[0092] The travel distribution matrix (OD matrix) between takeoff and landing stations is an N×N matrix, where N is the total number of air taxi takeoff and landing stations. The element value in the i-th row and j-th column of the matrix represents the number of passengers departing from takeoff and landing station i and destined for takeoff and landing station j within a given statistical period. This travel distribution matrix can be obtained by statistically analyzing historical order data from an actual air taxi operating platform or by predicting it using other mathematical models. It is used to randomly assign destination stations to generated orders in each simulation time step.

[0093] S23. Based on the constructed air taxi intelligent agent model and the constructed station passenger behavior model, build a multi-agent simulation system with a fixed time step, and load the low-level-high-level composite urban air taxi traffic network model into the multi-agent simulation system.

[0094] The multi-agent simulation system is a discrete event simulation system built using Python 3.12. Within each fixed time step, it sequentially executes: handling passenger arrival events, handling passenger queuing events, the air taxi agent's actions, and data statistics, such as... Figure 7 The diagram shows the sequence of events in a multi-agent simulation system at each time step.

[0095] S3. A Double-DQN network is used to construct an air taxi scheduling model, and the air taxi scheduling model is trained in a multi-agent simulation system to obtain a trained air taxi scheduling model.

[0096] S31. Construct an air taxi scheduling model using a Double-DQN network and set the reward function for the air taxi scheduling model.

[0097] In this embodiment, a Double-DQN network is used as the scheduling model for the air taxi agent. The Double-DQN network is an improved form of the classic DQN network, comprising two structurally identical fully connected neural networks: an action selection network and a value evaluation network. The action selection network selects actions based on the current state, and the value evaluation network evaluates the Q-value of each action. This design avoids the problem of DQN networks overestimating Q-values.

[0098] like Figure 5 As shown, the Double-DQN network in this embodiment adopts a 4-layer fully connected network structure, with the ReLU function used for the first three layers and the Sigmoid function used for the output layer.

[0099] The input to the Double-DQN network is the state matrix, which is set according to the following formula:

[0100]

[0101] in, This represents the input to the Double-DQN network. This represents the shortest distance vector between the current air taxi and each take-off and landing station; This represents a vector of the number of passengers queuing at each departure and arrival station. This represents a vector indicating the estimated total time required to complete all current orders at each takeoff and landing station; This represents the vector of the number of air taxis currently heading to various take-off and landing stations; This represents the vector of the number of air taxis currently in a standby state at each take-off and landing station.

[0102] In this embodiment, the size of the state matrix is ​​set to 128×5. If the number of stations is less than 128, the data in the input matrix is ​​filled with 0.

[0103] The output of the Double-DQN network is a Q-value vector:

[0104]

[0105] in, This is the output of the Double-DQN network; , and These represent the Q values ​​of the actions taken by the air taxi agent to transfer to stations numbered 1, 2, and h, respectively.

[0106] In this embodiment, the Double-DQN network also introduces an experience replay mechanism. A replay buffer is set up to store the experience data of the agent's interaction with the environment. Each piece of experience data includes (current state) The selected action The rewards received Next state During training, a batch of empirical data is randomly sampled from the replay buffer for learning, in order to break the correlation between empirical data generated by continuous interaction and improve data utilization.

[0107] In this embodiment, the reward function is designed to take into account the system's operating efficiency and service quality, and is constructed based on the following three indicators: the total waiting time of passengers in the system, the ratio of the number of canceled orders to the total number of orders, and the total length of the flight paths of all air taxis.

[0108] The reward function is set according to the following formula:

[0109]

[0110] in, Indicates the reward value; , and All are weighting coefficients; This indicates the total waiting time for passengers; This represents the ratio of canceled orders to the total number of orders (order cancellation rate). This represents the total length of the flight paths of all air taxis. and The mean and standard deviation of the corresponding variables are determined using the Monte Carlo method.

[0111] The design idea of ​​this reward function is: when the system performs better than the average level, it receives a positive reward; conversely, when the system performs worse than the average level, it receives a negative reward, thereby guiding the agent to learn a scheduling strategy that optimizes the overall performance of the system.

[0112] S32. Train the air taxi dispatching model in a multi-agent simulation system.

[0113] The air taxi dispatching model constructed in step S31 is deployed to the multi-agent simulation system built in step S2 for training. The training process is as follows:

[0114] At the start of each training round, the simulation environment is initialized, all air taxi agents are set to standby sub-state, and the queuing systems at each station are cleared. The simulation then proceeds according to a fixed time step T.

[0115] Within each time step:

[0116] 1. Generate passenger arrival events based on the station passenger behavior model in step S22.

[0117] 2. Update the queuing system at each site to handle order cancellation events.

[0118] 3. For each air taxi agent in the standby sub-state, its current state information (including distance to each station, number of people queuing at each station, number of vehicles waiting at each station, etc.) is input into the action selection network of the Double-DQN network. The network outputs the Q value of each action. The agent selects actions according to the ε-greedy policy (i.e., randomly selects actions with probability ε, and selects the action with the largest Q value with probability 1-ε). After determining the target station, it transitions to the scheduling sub-state. After the end of this sub-state, it enters the destination driving sub-state.

[0119] 4. For each air taxi agent in other states, update its state according to the S21 state transition rule.

[0120] 5. After all agent behaviors have been executed, the various indicators within that time step are statistically analyzed, and the global reward value is calculated.

[0121] 6. For each piece of empirical data Store to the playback buffer.

[0122] After a certain number of time steps, a batch of empirical data is randomly sampled from the replay buffer, and the parameters of the action selection network are updated using the following loss function:

[0123]

[0124] in, The loss function; Indicates mean square error; Indicates the current reward; Indicates the discount factor; This represents the current state predicted by the action selection network – the action Q-value. The maximum Q-value for the next state predicted by the target network; Represents the current state; This represents the currently selected action; Represents the next state; This represents the action to be chosen in the next state.

[0125] Every preset number of training steps (C rounds), the parameters of the action selection network are adjusted. Copy to target network To stabilize the training process.

[0126] Repeat the above process until the reward function value converges or reaches the preset maximum number of training rounds to obtain the trained air taxi dispatch model.

[0127] S4. Deploy the trained air taxi dispatching model to the multi-agent simulation system, run the simulation based on the low-level-high-level composite urban air taxi traffic network model, and output a statistical report on air taxi operation.

[0128] S41. Load the low-level-high-level composite urban air taxi transportation network model constructed in step S1 to initialize the simulation environment and initialize the initial state of all air taxi agents.

[0129] In this embodiment, the initialization settings are first performed during the simulation run phase, which specifically includes the following steps:

[0130] The low- to high-rise composite urban air taxi traffic network model constructed in step S1 is loaded into a multi-agent simulation system as the basis for path planning.

[0131] Input the parameters required for the simulation system to run, including but not limited to:

[0132] Simulation time step T, total simulation duration (or total number of time steps), initial number of air taxis at each takeoff and landing station, number of takeoff and landing platforms at each takeoff and landing station, horizontal flight speed of the air taxis, climb / descent speed of the air taxis, and maximum passenger capacity of the air taxis. Maximum range of air taxis, maximum charging time of air taxis, time required for passengers to board and disembark, and average passenger arrival rate at each takeoff and landing station during each time period. , Departure / Arrival Distribution Matrix (OD Matrix), Order Cancellation Threshold and cancellation probability and the weighting coefficients of each indicator in the reward function , and .

[0133] Initialize the state of all air taxi agents: Based on the initial number and distribution of air taxis, generate the corresponding number of air taxi agents at the corresponding take-off and landing stations, set the sub-state of all air taxi agents to standby, and initialize process variables such as "execution duration" to 0.

[0134] Initialize the queuing system for each take-off and landing station: Clear the order list for all stations and set the number of passengers in the queue at each station to 0.

[0135] After completing the above initialization, the multi-agent simulation system enters the ready state and is ready to start running.

[0136] S42. The simulation proceeds according to a fixed time step. In each time step, the trained air taxi scheduling model provides scheduling decisions for the air taxi agents in the standby sub-state, updates the status of all air taxi agents and system order data, until the simulation ends, and outputs statistical reports of various preset indicators.

[0137] In this embodiment, the multi-agent simulation system supports accessing historical order data exported from the actual operation platform as the actual order flow, and also supports randomly generating synthetic order flows based on preset passenger arrival rates and travel distribution matrices. By flexibly configuring the order input method, various scenarios can be simulated, such as peak-hour demand surge scenarios, holiday travel pattern changes scenarios, and network adjustment scenarios after adding take-off and landing stations, thereby comprehensively evaluating the performance of air taxis under different operating conditions.

[0138] In this embodiment, the simulation run phase proceeds according to a fixed time step T, and the specific process is as follows:

[0139] Within each time step, the multi-agent simulation system performs the following operations sequentially:

[0140] (1) Handling passenger arrival events:

[0141] Based on the station passenger behavior model in step S22, for each take-off and landing station The number of passengers arriving within that time step is obtained by random sampling based on a Poisson distribution. .Will Passengers are randomly divided into several groups, and a corresponding number of orders are generated. Each order is then randomly assigned a destination station based on a travel distribution matrix. The generated orders are then added to the end of the queuing system for their respective stations.

[0142] (2) Handling passenger queuing incidents:

[0143] The queuing system iterates through all departure and arrival stations. For each order, it calculates its waiting time. Determine whether to cancel based on the following rules:

[0144] like If so, the order will be held; if Then, based on probability Cancel the order; the canceled order will be removed from the queue system.

[0145] (3) Executing the behavior of the air taxi intelligent agent:

[0146] Iterate through all air taxi agents and execute corresponding actions based on their current state:

[0147] For an air taxi agent in the standby sub-state, its current state information is input into the trained Double-DQN network (i.e., the trained air taxi scheduling model). The network outputs the Q value of each target station. The agent selects the station with the largest Q value as the target station and the state transitions to the scheduling sub-state. After the end of this sub-state, it enters the destination-bound sub-state.

[0148] For an agent in the destination-bound sub-state, based on its current location and target station, the A* algorithm is used to plan a specific flight route in the low-to-high-level composite urban air taxi transportation network constructed in step S1, and its location is updated along the route; when the air taxi agent arrives at the target station, the sub-state transitions to the landing sub-state.

[0149] For agents in other states, update their state and process variables according to the state transition rules of the hierarchical finite state machine.

[0150] (4) Data statistics:

[0151] Record various operational metrics within the current time step, including but not limited to: the number of passengers queuing at each station, the location and status of each air taxi, the number of completed orders and the average waiting time, the number of cancelled orders and the cancellation rate, the total path length of all air taxis that have flown, and the total passenger waiting time.

[0152] After the above operations are completed, the simulation time advances by one step, entering the next time step. Repeat the above process until the preset total simulation time is reached.

[0153] Simulation End and Result Output:

[0154] After the simulation is complete, the system outputs all simulation results.

[0155] Furthermore, all simulation results are summarized and statistically analyzed to generate statistical reports for various preset indicators. The statistical data includes, but is not limited to: average passenger waiting time, total time required for each order from placement to completion (order response time), average utilization rate of air taxis, maximum number of air taxis taking off and landing simultaneously at each take-off and landing station, passenger order completion rate (proportion of orders not cancelled), total number of passengers served by the system, and total flight mileage of all air taxis.

[0156] The multi-agent simulation system can generate relevant charts as needed, such as curves showing the change of queue length over time at each station, histograms of order response time distribution, and heatmaps of air taxi utilization, to visualize the simulation results and provide data support for the operation evaluation of the air taxi network and corporate decision-making.

[0157] Example 2:

[0158] This embodiment uses the same method as in Embodiment 1 to simulate and evaluate the urban air taxis in the given city of H.

[0159] During the data input phase, GIS information data of buildings, no-fly zones, and urban air taxi take-off and landing sites in City H are collected and input into the multi-agent simulation system. The method in step S1 of Example 1 is used to obtain the urban low-rise-high-rise composite urban air taxi traffic network model. Various parameters of the multi-agent simulation system are input, including time step T, total simulation duration, initial number of air taxi agents at each take-off and landing site, number of take-off and landing platforms at each take-off and landing site, horizontal flight speed of air taxis, climb / descent speed of air taxis, maximum passenger capacity of air taxis, maximum range of air taxis, maximum charging time of air taxis, and time required for passengers to board and alight. The average passenger arrival rate of each take-off and landing site in each time period and the travel distribution matrix between take-off and landing sites are also input. After the above data input is completed, the multi-agent simulation system will execute the simulation.

[0160] During the simulation phase, the multi-agent simulation system advances the clock according to the set time step T. Each time the clock advances, the system processes passenger arrival events occurring within the current time step based on the station passenger behavior model from step S22, executes the behavioral logic defined by the hierarchical finite state machine for each air taxi agent, and utilizes the Double-DQN network trained in step S3 to provide scheduling decisions for air taxi agents in the standby sub-state.

[0161] At the end of the simulation, all required data were collected and corresponding charts were generated. The collected data included average passenger waiting time, total time required for each order from order placement to completion, average utilization rate of air taxis, maximum number of simultaneous takeoffs and landings at each takeoff and landing station, and passenger order completion rate, which comprehensively evaluated the operational performance of the urban air taxi system in City H.

[0162] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A multi-agent simulation method for air taxis combining a Double-DQN network, characterized in that, Includes the following steps: S1. Obtain urban geographic information data and construct a low-level-high-level composite urban air taxi transportation network model based on the urban geographic information data; S2. Construct an intelligent agent model for air taxis and a passenger behavior model at stations, and build a multi-agent simulation system with a fixed time step based on the intelligent agent model for air taxis and the passenger behavior model at stations. Step S2 specifically involves: S21. A hierarchical finite state machine is used to model the entire lifecycle behavior of air taxis, and an intelligent agent model of air taxis is constructed. The hierarchical finite state machine in step S21 includes two main states: air and ground. The air main state includes sub-states such as takeoff, heading to the destination, and landing. The ground main state includes sub-states such as dropping off passengers, picking up passengers, charging and maintenance, standby, and scheduling. S22. Use a queuing model to model passenger behavior at the station and construct a passenger behavior model for the station. S23. Based on the constructed air taxi intelligent agent model and the constructed station passenger behavior model, build a multi-agent simulation system with a fixed time step, and load the low-level-high-level composite urban air taxi traffic network model into the multi-agent simulation system. S3. A Double-DQN network is used to construct an air taxi dispatching model, and the air taxi dispatching model is trained in a multi-agent simulation system to obtain a trained air taxi dispatching model. S4. Deploy the trained air taxi dispatching model to the multi-agent simulation system, run the simulation based on the low-level-high-level composite urban air taxi traffic network model, and output the simulation results of air taxi operation.

2. The multi-agent simulation method for air taxis combined with Double-DQN networks according to claim 1, characterized in that, Step S1 specifically involves: S11. Divide the urban low-altitude airspace into low-level airspace and high-level airspace according to the preset altitude threshold, and obtain environmental data at the cruising altitude of the low-level airspace and the high-level airspace respectively, and generate a first matrix describing the low-level airspace environment and a second matrix describing the high-level airspace environment. S12. Based on the first and second matrices, a path planning algorithm is used to plan the shortest route for any two take-off and landing stations in the lower and upper airspace, respectively. S13. Optimize the nodes of each shortest route, extract the nodes where the path direction changes as effective nodes, and construct the first undirected connected graph in the lower airspace and the second undirected connected graph in the upper airspace based on all effective nodes and connecting edges. S14. Supplement the altitude coordinates of each node in the first and second undirected connected graphs with their respective cruising altitudes, and connect the nodes with the same horizontal coordinates to generate a low-to-high-level composite urban air taxi transportation network model.

3. The multi-agent simulation method for air taxis combined with Double-DQN networks according to claim 1, characterized in that: The multi-agent simulation system includes multiple air taxi agents, each of which is generated by instantiating an air taxi agent model. The multi-agent simulation system uses a low- to high-level composite urban air taxi traffic network model to provide a basis for path planning for air taxi agents.

4. The multi-agent simulation method for air taxis combined with Double-DQN networks according to claim 1, characterized in that: The passenger arrival events corresponding to the station passenger behavior in step S22 are generated by Poisson distribution, and the destination station is assigned to the passenger based on the travel distribution matrix between the take-off and landing stations to form an order; The generated orders enter the queuing system of the site and are cancelled with a preset probability based on the waiting time; The travel distribution matrix is ​​an n×n matrix, where n is the number of departure and arrival stations. The value in the i-th row and j-th column of the matrix represents the number of passengers departing from station i and traveling to station j. The queuing system is a list of unlimited length that stores order data in the order of order arrival.

5. The multi-agent simulation method for air taxis combined with Double-DQN networks according to claim 1, characterized in that, Step S3 specifically involves: S31. Construct an air taxi dispatching model using a Double-DQN network and set the reward function for the air taxi dispatching model; S32. Train the air taxi dispatching model in the multi-agent simulation system to obtain the trained air taxi dispatching model.

6. The multi-agent simulation method for air taxis combined with a Double-DQN network according to claim 5, characterized in that, In step S31: The input to the air taxi scheduling model is a state matrix; the output of the air taxi scheduling model is a Q-value vector representing the actions of the air taxi agent in selecting and transferring to various take-off and landing stations. The state matrix contains at least the following information: the shortest distance between the current air taxi and each take-off and landing station, the number of passengers queuing at each take-off and landing station, the estimated total time required to complete the current order at each take-off and landing station, the number of air taxis currently heading to each take-off and landing station, and the number of air taxis currently waiting at each take-off and landing station.

7. The multi-agent simulation method for air taxis combined with Double-DQN networks according to claim 5, characterized in that: The reward function is set according to the following formula: in, Indicates the reward value; , and All are weighting coefficients; This indicates the total waiting time for passengers; Indicates the order cancellation rate; This represents the total length of the flight paths of all air taxis. and These correspond to the mean and standard deviation of the variables, respectively.

8. The multi-agent simulation method for air taxis combined with Double-DQN networks according to claim 1, characterized in that, Step S4 specifically involves: S41. Load the low-level-high-level composite urban air taxi traffic network model constructed in step S1 to initialize the simulation environment of the multi-agent simulation system and initialize the initial state of all air taxi agents. S42. The simulation proceeds according to a fixed time step. In each time step, the trained air taxi scheduling model provides scheduling decisions for the air taxi agents in the standby sub-state, updates the status of all air taxi agents and system order data, until the simulation ends, and outputs statistical reports of various preset indicators.

Images