Virtual aircraft multi-agent co-simulation and optimization method and system based on reinforcement learning

Abstract

The invention relates to the technical field of traffic engineering, and discloses a reinforcement learning-based virtual aircraft multi-agent co-simulation and optimization method and system, and the system comprises a virtual environment construction module which is used for constructing a virtual city environment through UE4 and AirSim simulation software; the aircraft model construction module is used for designing an eVTOL virtual aircraft agent model by using 3ds Max; the agent state sensing module is used for sensing the state of the agent in real time through an AirSim plug-in; the reinforcement learning training module is used for applying a DDQN algorithm to train an agent to make an autonomous decision; and the collaborative task design module is used for multi-agent collaborative task execution, such as traffic planning, route optimization and the like. Through a virtual simulation method, an innovative perspective of cooperative operation of a large number of virtual aircraft agents in a complex urban environment is provided. The method has remarkable advancement in urban air traffic (UAM) system design and verification research, and dynamic characteristics of urban air traffic can be truly represented and analyzed in a simulation environment.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic engineering, and in particular to a virtual aircraft multi-agent collaborative simulation and optimization method and system based on reinforcement learning. Background Art

[0002] With the acceleration of urbanization and the continuous expansion of population, traditional ground transportation systems are unable to meet the needs of modern cities for efficient and environmentally friendly travel, and urban transportation problems are becoming increasingly prominent. Urban Air Mobility (UAM), as an emerging transportation mode, has received widespread attention since it was proposed by NASA in 2017. The core concept of UAM is to alleviate urban traffic congestion, reduce environmental pollution, and provide convenient and efficient travel options for urban residents through the application of air transportation tools.

[0003] Compared with traditional ground transportation, UAM has significant three-dimensional transportation advantages and can open up new transportation routes in the air, thereby reducing the pressure on ground transportation. At the same time, the promotion of UAM will also help reduce exhaust emissions and promote the green development of cities. However, the implementation of UAM faces a series of technical challenges. The complex urban environment and dense buildings cause the navigation system to face high interference problems, including multipath effects and shielding problems of GNSS signals. These factors may affect the navigation accuracy and stability of the aircraft, thereby affecting safety.

[0004] In order to solve these technical problems, virtual simulation technology is widely used in the research and development of UAM systems. By establishing a virtual scene that is highly similar to the actual urban environment, researchers can conduct comprehensive tests on the navigation, control and communication systems of the aircraft, reducing the risks and R&D costs in actual operations.

[0005] However, although the current UAM system has certain navigation and control technologies, in complex urban environments, the existing navigation system faces high interference problems, such as signal multipath effects and building obstructions, resulting in insufficient navigation accuracy and increased flight safety risks. At the same time, the operating mechanism is not yet mature and lacks effective management and optimization solutions, which makes it difficult to achieve efficient and safe air traffic in practical applications. For this reason, technical personnel in this field propose a virtual aircraft multi-agent collaborative simulation and optimization method and system based on reinforcement learning to solve the above problems. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a virtual aircraft multi-agent collaborative simulation and optimization method and system based on reinforcement learning, which solves the problems raised in the above-mentioned background technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning, comprising: Virtual environment construction module, used to build virtual city environment and aircraft models using UE4 and AirSim simulation software; Aircraft model building module, used to design eVTOL virtual aircraft intelligent model using 3ds Max; The agent state perception module is used to realize real-time perception of the agent state through the AirSim plug-in; Reinforcement learning training module, used to apply the DDQN algorithm to train the agent to make autonomous decisions; Collaborative task design module, used for multi-agent collaborative execution of tasks, such as traffic planning, route optimization, etc. Distributed reinforcement learning module, used for collaborative training and optimization of multiple agents using distributed learning methods.

[0008] Preferably, the virtual environment construction module further includes: The 3D city scene simulation subunit uses UE4 to build a highly realistic 3D city environment model, which includes terrain, buildings and weather conditions, and simulates real urban air traffic scenarios.

[0009] Preferably, the aircraft model building module further comprises: The aircraft model construction subunit uses 3ds Max modeling to design the eVTOL virtual aircraft intelligent model.

[0010] Preferably, the agent state perception module includes: The location sensing subunit is used to sense the location information of the intelligent agent; The speed perception subunit is used to perceive the speed information of the intelligent agent; The height perception subunit is used to perceive the height information of the intelligent agent; The flight direction perception subunit is used to perceive the flight direction information of the intelligent agent.

[0011] Preferably, the reinforcement learning training module includes: DDQN algorithm implementation subunit, used to train the autonomous decision-making ability of the aircraft agent; The policy network design subunit, including the Actor and Critic networks, is used for action selection and value estimation of the agent.

[0012] Preferably, the collaborative task design module includes: Traffic planning subunit, used to collaboratively execute urban air traffic planning tasks; The route optimization subunit is used to collaboratively perform route optimization tasks.

[0013] Preferably, the distributed reinforcement learning module includes: The global reward sharing subunit is used for multiple agents to share the global reward; The back-propagation optimization subunit is used to optimize the policy of a single agent.

[0014] Preferably, it also includes an information sharing mechanism for achieving real-time transmission of flight intention and status, and developing collaborative strategies to enable intelligent agents to make collaborative decisions based on global information.

[0015] Preferably, it also includes a multi-agent collaborative decision-making mechanism module for realizing the collaborative operation of multiple aircraft agents, and realizing an intelligent and autonomous unmanned aerial agent by optimizing the autonomous decision-making and collaborative capabilities of the aircraft agents.

[0016] The virtual aircraft multi-agent collaborative simulation and optimization method based on reinforcement learning includes the following steps: S1. Use UE4 and AirSim software to build a virtual city environment; S2. Design the eVTOL aircraft model using 3ds Max; S3, achieve real-time perception of the agent's state through the AirSim plug-in; S4. Apply DDQN reinforcement learning algorithm to train intelligent agents to make autonomous decisions; S5. Design multi-agent collaborative tasks, such as traffic planning, route optimization, etc. S6. Use distributed learning methods to conduct collaborative training and optimization of multiple agents.

[0017] The present invention provides a virtual aircraft multi-agent collaborative simulation and optimization method and system based on reinforcement learning. It has the following beneficial effects: 1. This invention proposes an innovative perspective on the collaborative operation of a large number of virtual aircraft agents in a complex urban environment through a virtual simulation method. This method has significant advancement in the design and verification research of urban air traffic systems, and can truly reproduce and analyze the dynamic characteristics and challenges of urban air traffic in a simulated environment. By using reinforcement learning algorithms, this system not only realizes the autonomous decision-making of virtual aircraft agents, but also promotes the collaborative operation between agents, thereby optimizing flight paths. This innovative platform will provide efficient and intelligent route planning and management solutions for future urban air traffic systems, improve the efficiency and safety of air traffic, and strongly support the full implementation of UAM systems.

[0018] 2. This invention fully integrates cutting-edge technologies such as multi-agent collaboration and distributed reinforcement learning. Through the integration of these technologies, it not only improves the operating efficiency and safety of the UAM system, but also provides novel solutions to existing challenges. Especially in terms of the collaborative decision-making mechanism of intelligent agents, the project has conducted in-depth research on how to optimize the strategies of intelligent agents through distributed reinforcement learning. This method significantly improves the overall performance and stability of the UAM system, making it possible to effectively deal with flight path optimization and conflict resolution in complex urban environments. This technological innovation provides a strong competitive advantage for the UAM field and promotes the development of intelligent transportation systems.

[0019] 3. The present invention can effectively support the evaluation, optimization and decision-making process of UAM system architecture design, and help promote the normalized operation of UAM. With the increasing severity of urban traffic congestion and environmental problems, UAM, as a new mode of transportation, has shown huge market potential. By providing an efficient and convenient travel experience, UAM can not only effectively alleviate the pressure of ground transportation, but also improve the overall operating efficiency of urban transportation. The research and development of this project not only responds to the society's urgent need for efficient transportation solutions, but also has broad market prospects and helps the sustainable development of cities. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the overall framework diagram of the present invention; Figure 2 It is a schematic diagram of reinforcement learning training of the present invention; Figure 3 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings of the present invention specification to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example

[0022] Please see attached Figure 1 - Attachment Figure 3 The embodiment of the present invention provides a virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning, including: Virtual environment construction module, used to build a virtual city environment using UE4 and AirSim simulation software; The virtual environment building blocks include: The 3D city scene simulation subunit uses UE4 to build a highly realistic 3D city environment model, which includes terrain, buildings and weather conditions, and simulates real urban air traffic scenarios.

[0023] Specifically, 3D urban scene simulation technology plays a vital role in air traffic management systems. This system uses Unreal Engine 4 (UE4) to build a highly realistic 3D urban environment to simulate real urban air traffic scenarios. This environment includes UAM urban environments and complex terrain, providing a comprehensive and realistic test platform for urban air traffic management.

[0024] Software Tools: Use Unreal Engine 4 (UE4) in conjunction with the city’s Geographic Information System (GIS) data to generate highly realistic urban environment models.

[0025] Detail processing: During the modeling process, attention is paid to the detailed expression of urban features (buildings, roads, bridges, etc.) to ensure the consistency between the model and the real environment.

[0026] Sensor data: Use technologies such as drone aerial photography to obtain terrain and building data to improve the realism of the model.

[0027] Dynamic elements: Introduce dynamic models, such as traffic flow, pedestrians, and weather changes, to enhance the realism and complexity of simulation scenarios.

[0028] Environmental simulation: Use platforms such as AirSim to perform environmental simulation and support actual operation and testing of aircraft in virtual cities.

[0029] Interactive Experience: Develop interactive user interfaces that allow researchers and designers to modify models and observe changes in real time.

[0030] Aircraft model building module, used to design eVTOL aircraft models using 3ds Max; Aircraft model building modules include: The aircraft model construction subunit uses 3ds Max modeling to design the eVTOL virtual aircraft intelligent model.

[0031] Specifically, the main goal of the aircraft model building subunit is to create a high-precision eVTOL (electric vertical take-off and landing aircraft) virtual aircraft agent model. The model inherits all the physical characteristics of the eVTOL, including the size and weight of the aircraft. The model provides an accurate experimental object for the research and planning of urban air traffic, making the simulation results closer to reality and improving the validity of the data.

[0032] The agent state perception module is used to realize real-time perception of the agent state through the AirSim plug-in; The agent state perception module includes: The location sensing subunit is used to sense the location information of the intelligent agent; The speed perception subunit is used to perceive the speed information of the intelligent agent; The height perception subunit is used to perceive the height information of the intelligent agent; The flight direction perception subunit is used to perceive the flight direction information of the intelligent agent.

[0033] Specifically, the agent state perception module aims to achieve real-time state monitoring of the eVTOL (electric vertical take-off and landing vehicle) model to ensure safe and efficient operation in a dynamic urban environment.

[0034] Sensor integration: Through the AirSim plug-in, use the API interface to connect to various sensors (such as IMU, GPS, camera, etc.) to obtain the real-time position information, speed, attitude and other data of the aircraft.

[0035] Real-time monitoring: In the AirSim environment, real-time visual monitoring of the aircraft status is achieved to help operators instantly understand the operating status of the aircraft.

[0036] Anomaly Detection: Through machine learning technology, the aircraft status data is analyzed to detect abnormal situations (such as failures, deviations from the route, etc.) in real time and issue alarms.

[0037] Path planning: Based on the current environment status and aircraft position, reinforcement learning algorithms are used to implement dynamic path planning to ensure the safe flight of the aircraft in complex urban environments.

[0038] Decision support: Develop intelligent decision-making systems that combine environmental information and aircraft status to provide operators with flight operation suggestions and optimize routes and mission execution efficiency.

[0039] Reinforcement learning training module, used to apply the DDQN algorithm to train the agent to make autonomous decisions; The reinforcement learning training module includes: DDQN algorithm implementation subunit, used to train the autonomous decision-making ability of the aircraft agent; The policy network design subunit, including the Actor and Critic networks, is used for action selection and value estimation of the agent.

[0040] Collaborative task design module, used for multi-agent collaborative execution of tasks, such as traffic planning, route optimization, etc. The collaborative task design module includes: Traffic planning subunit, used to collaboratively execute urban air traffic planning tasks; The route optimization subunit is used to collaboratively perform route optimization tasks.

[0041] Specifically, collaborative task design: In addition to the autonomous decision-making capabilities of a single agent, this system also designs collaborative tasks that enable multiple virtual aircraft agents to jointly perform UAM air traffic planning, conflict resolution, and route optimization tasks. These collaborative tasks not only improve the operating efficiency of the entire air traffic system, but also enhance the stability and safety of the system. The collaborative decision-making mechanism is one of the key links for the successful implementation of the UAM system. In order to achieve the collaborative operation of multiple aircraft agents, we need to conduct in-depth research on advanced technologies such as distributed reinforcement learning methods. By optimizing the autonomous decision-making and collaborative capabilities of aircraft agents, intelligent and autonomous unmanned aerial vehicle agents are realized, providing strong support for the successful implementation of the UAM system.

[0042] Distributed reinforcement learning module, used for collaborative training and optimization of multiple agents using distributed learning methods.

[0043] The distributed reinforcement learning module includes: The global reward sharing subunit is used for multiple agents to share the global reward; The back-propagation optimization subunit is used to optimize the policy of a single agent.

[0044] It also includes information sharing mechanisms to achieve real-time transmission of flight intentions and status, and develop collaborative strategies to enable agents to make collaborative decisions based on global information. It also includes a multi-agent collaborative decision-making mechanism module to achieve collaborative operation of multiple agents in aircraft, and to achieve intelligent and autonomous unmanned aerial agents by optimizing the autonomous decision-making and collaborative capabilities of aircraft agents.

[0045] Agent state perception: In the 3D urban simulation environment of this system, through the AirSim plug-in and the eVTOL (electric vertical take-off and landing) aircraft model designed with 3ds Max, the system can accurately perceive the state of the virtual aircraft agent. These perception data include key information such as the position, speed, altitude and flight direction of the aircraft, which provides an important basis for the subsequent autonomous decision-making of the virtual aircraft agent.

[0046] Autonomous decision-making training: In order to train the virtual aircraft agent to make autonomous decisions, this system uses reinforcement learning algorithms, especially DDQN (Dual Deep Q Network). Through a large number of simulation training and iterative optimization, the agent can gradually learn how to perform tasks such as dynamic path planning and autonomous obstacle avoidance in complex and dynamic environments. This autonomous decision-making ability is of great significance for responding to emergencies, optimizing routes, and improving flight efficiency.

[0047] The virtual aircraft multi-agent collaborative simulation and optimization method based on reinforcement learning includes the following steps: S1. Use UE4 and AirSim software to build a virtual city environment; Specifically, we used Unreal Engine 4 (UE4) and AirSim software to build a high-fidelity virtual city environment and simulate urban air traffic scenarios. The combination of UE4's powerful graphics rendering capabilities and AirSim's flight simulation capabilities enabled us to create a dynamic and interactive simulation platform.

[0048] S2. Design the eVTOL aircraft model using 3ds Max; S3, achieve real-time perception of the agent's state through the AirSim plug-in; S4. Apply DDQN reinforcement learning algorithm to train intelligent agents to make autonomous decisions; S5. Design multi-agent collaborative tasks, such as traffic planning, route optimization, etc. Specifically, advanced path planning algorithms are used to optimize traffic flow to ensure efficient and safe aircraft routes. Based on real-time traffic data, routes are dynamically adjusted to cope with emergencies (such as obstacles, weather changes, etc.).

[0049] S6. Use distributed learning methods to conduct collaborative training and optimization of multiple agents.

[0050] Specifically, through the experience sharing mechanism, each intelligent agent can learn from the success and failure experiences of other intelligent agents, thereby improving the overall learning effect.

[0051] Performance optimization: Continuously optimize training algorithms and model parameters to ensure that intelligent agents demonstrate better decision-making capabilities and execution efficiency in complex environments.

[0052] In summary, this paper adopts virtual simulation technology and proposes an innovative perspective to achieve the coordinated operation of a large number of virtual aircraft agents in a complex urban environment. This method has significant advancement in the design and verification research of urban air traffic systems, and can truly reproduce and analyze the dynamic characteristics and challenges of urban air traffic in a simulated environment.

[0053] Through reinforcement learning algorithms, this system not only realizes autonomous decision-making of virtual aircraft agents, but also promotes collaborative operations between them, thereby optimizing flight paths. This innovative platform will provide efficient and intelligent route planning and management solutions for future urban air traffic systems, improve the efficiency and safety of air traffic, and support the full implementation of urban air traffic (UAM) systems.

[0054] As urban traffic congestion and environmental problems become increasingly serious, UAM, as a new mode of transportation, shows great market potential. By providing an efficient and convenient travel experience, UAM can effectively alleviate the pressure of ground transportation and improve the overall operating efficiency of urban transportation. The research and development of this project not only responds to the society's urgent need for efficient transportation solutions, but also has broad market prospects and helps the sustainable development of cities.

[0055] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning, characterized by: include: Virtual environment construction module, used to build a virtual city environment using UE4 and AirSim simulation software; Aircraft model building module, used to design eVTOL virtual aircraft intelligent model using 3ds Max; The agent state perception module is used to realize real-time perception of the agent state through the AirSim plug-in; Reinforcement learning training module, used to apply the DDQN algorithm to train the agent to make autonomous decisions; Collaborative task design module, used for multi-agent collaborative execution of tasks, such as traffic planning, route optimization, etc. Distributed reinforcement learning module, used to train and optimize the collaborative capabilities of multiple agents using distributed learning methods.

2. The virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning according to claim 1 is characterized in that: The virtual environment construction module further includes: The 3D city scene simulation subunit uses UE4 to build a highly realistic 3D city environment model, which includes terrain, buildings and meteorological conditions, and simulates real urban air traffic scenes.

3. The virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning according to claim 1 is characterized in that: The aircraft model building module includes: The aircraft model construction subunit uses 3ds Max modeling to design the eVTOL virtual aircraft intelligent model.

4. The virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning according to claim 1 is characterized in that: The agent state perception module comprises: The location sensing subunit is used to sense the location information of the intelligent agent; The speed perception subunit is used to perceive the speed information of the intelligent agent; The height perception subunit is used to perceive the height information of the intelligent agent; The flight direction perception subunit is used to perceive the flight direction information of the intelligent agent.

5. The virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning according to claim 1 is characterized in that: The reinforcement learning training module includes: DDQN algorithm implementation subunit, used to train the autonomous decision-making ability of the aircraft agent; The policy network design subunit, including the Actor and Critic networks, is used for action selection and value estimation of the agent.

6. The virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning according to claim 1 is characterized in that: The collaborative task design module includes: Traffic planning subunit, used to collaboratively execute urban air traffic planning tasks; The route optimization subunit is used to collaboratively perform route optimization tasks.

7. The virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning according to claim 1 is characterized in that: The distributed reinforcement learning module includes: The global reward sharing subunit is used for multiple agents to share the global reward; The back-propagation optimization subunit is used to optimize the policy of a single agent.

8. The virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning according to claim 1 is characterized in that: It also includes information sharing mechanisms to achieve real-time transmission of flight intentions and status, as well as the development of collaborative strategies that enable agents to make collaborative decisions based on global information.

9. The virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning according to claim 1, characterized in that: It also includes a multi-agent collaborative decision-making mechanism module, which is used to realize the collaborative operation of multiple aircraft agents, and realize intelligent and autonomous unmanned aerial vehicle agents by optimizing the autonomous decision-making and collaborative capabilities of aircraft agents.

10. A virtual aircraft multi-agent collaborative simulation and optimization method based on reinforcement learning, according to the virtual aircraft multi-agent collaborative simulation and optimization system based on reinforcement learning according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1. Use UE4 and AirSim software to build a virtual city environment; S2. Design the eVTOL aircraft model using 3ds Max; S3, achieve real-time perception of the agent's state through the AirSim plug-in; S4. Apply DDQN reinforcement learning algorithm to train intelligent agents to make autonomous decisions; S5. Design multi-agent collaborative tasks, such as traffic planning, route optimization, etc. S6. Use distributed learning methods to conduct collaborative training and optimization of multiple agents.

Images