A virtual-reality combined simulation system and method supporting large-scale UAV cluster flight tests

Abstract

The present invention provides a virtual-reality combined simulation system and method supporting large-scale unmanned aerial vehicle cluster flight tests, belonging to the field of unmanned aerial vehicle cluster simulation. To solve the problems of low fidelity of large-scale unmanned aerial vehicle clusters and multiple limitations of virtual-reality combined simulation in existing software simulation systems. Combining virtual unmanned aerial vehicle clusters with real real-time unmanned aerial vehicle clusters, simulation support for the underlying environment is provided, covering cluster scheduling algorithms, task allocation algorithms, formation control algorithms, dynamic control algorithms, and three-dimensional visualization frameworks, to achieve task requirement data analysis, task allocation and optimization scheduling, cluster command and control formation flight, cluster measurement and control and status monitoring, and intelligent collaboration of human-machine terminals. The present invention realizes effective verification of large-scale unmanned aerial vehicle clusters in performing tasks through a parallel simulation method that combines virtual unmanned aerial vehicle flight nodes with real unmanned aerial vehicle flight nodes, which has important theoretical significance and application value for promoting the verification of key technologies of unmanned aerial vehicle clusters and the practical application of unmanned aerial vehicle clusters.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle cluster application simulation, and in particular to a virtual-reality combined simulation system and method for supporting large-scale unmanned aerial vehicle cluster flight tests. Background Art

[0002] With the rapid development and widespread application of drone technology, drone swarms are becoming increasingly important in the military, civil and commercial fields. By building virtual drones and real drone swarm flights through simulation systems, and completing task-oriented drone swarm flights, it has important evaluation significance for the drone swarm application service system in a real environment.

[0003] The hardware verification platform can conduct experimental verification of the cluster, but it is difficult to organize, costly, and limited in scale by the actual number of drones, which greatly limits the experimental verification of the cluster system. Although the existing software simulation verification platform can simulate large-scale drone clusters, its fidelity is still far behind that of the hardware verification platform.

[0004] Existing patents, such as a virtual drone and physical drone hybrid cluster mission control system (application number: CN201911382101.9), include a fixed visual camera system with closed communication connection, an environmental control system, a drone simulation system, a ground control system, and a physical drone cluster. The fixed visual camera system includes environmental boundary markers and a visual camera group. The environmental control system includes environmental editing software and a virtual terrain database. The drone simulation system includes drone simulation software and drone formation software. The ground control system includes a flight control computer, data acquisition equipment, a graphic display system, and wireless communication equipment. The physical drone cluster includes several drones. The present invention is flexible to use and rich in functions, while significantly reducing the cost of training and exercise. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] In order to solve the problem of low fidelity of existing software simulation systems for large-scale drone swarms.

[0007] In order to solve the problems of low fidelity of large-scale UAV swarms and many limitations of virtual-reality combined simulation in existing software simulation systems.

[0008] The present invention adopts the following technical solutions to solve the above technical problems:

[0009] The present invention provides a virtual-reality combined simulation system supporting large-scale UAV cluster flight tests, comprising:

[0010] Real UAV data acquisition module, used to collect the position and attitude data of the UAV and the image data of the optoelectronic pod in real time;

[0011] Virtual drone creation module, used to create virtual drone dynamics and dynamics solution, and output the drone's position, attitude, speed, angular velocity, and timestamp data;

[0012] The geographic real scene data acquisition module is used to load the pre-collected real geographic elevation data and the corresponding satellite image maps;

[0013] Data transmission module, used for data exchange, mutual access and service request sending among modules based on the data distribution service DDS mechanism;

[0014] The data processing and analysis module is used to process and analyze real UAV data, virtual UAV data and geographic scene data, including cluster scheduling algorithm submodule, task allocation algorithm submodule, formation control algorithm submodule and dynamics control algorithm submodule;

[0015] A service distribution module is used to execute corresponding instructions according to the information obtained by the data processing and analysis module;

[0016] The 3D vision module is used to establish geographic simulation scenes and UAV simulation models through UE4, and realize the display of a virtual and real UAV simulation experimental platform;

[0017] The application module is used to realize the interaction between the user and the simulation system. The user can send command and control instructions through the application module to dispatch and control the drone cluster, including cluster scheduling, task allocation, formation flight, trajectory planning and trajectory tracking. The user can observe the execution status through the three-dimensional vision module and obtain the corresponding analysis conclusions of the execution status according to the data processing and analysis module;

[0018] The ground station module is used for users to directly monitor the status of the drone cluster and send command and control instructions directly to the drone cluster; receive drone cluster command and control instructions from the application module and send command and control instructions to the drone cluster; send task scheduling instructions to the application service module.

[0019] Furthermore, in the data transmission module, based on the DDS distributed communication mechanism, any module can publish any data topic, and any module can subscribe to obtain the topic data published by other modules, thereby realizing a distributed communication mechanism.

[0020] Furthermore, the application module adopts customized App development, and the user customizes the App application according to the simulation task scenario; the user designs and implements the functions of the application according to specific needs and goals; the user develops the application module based on the DDS distributed mechanism and customizes the App by obtaining the required data from other subsystems.

[0021] Furthermore, in the application module, the App calls the interface of the data processing and analysis module according to the specific needs of the simulation to realize large-scale cluster simulation scenarios; users can customize a variety of drone cluster application simulations, including reconnaissance, mapping, identification and exploration, and cluster strike scenarios.

[0022] Furthermore, users send command and control instructions to the simulation system through the App in the terminal connected to the network architecture to realize the scheduling and control of the drone cluster; the command and control instructions include adding new task requirements, allowing the drone cluster to independently plan tasks, and specifically specifying the execution path of the task, so that the drone cluster can perform the task according to the preset path.

[0023] Furthermore, in the real drone data acquisition module and the virtual drone data creation module, the output drone data includes state quantity information of the drone's position, attitude, speed, angular velocity, and timestamp.

[0024] Furthermore, in the virtual drone data creation module, an attribute assignment mode is adopted; for any virtual drone node, it can be selected as a specified configuration, specified dynamic properties and specified mission payload to implement the role definition of different network nodes, establish the properties and functions of different roles, and form a script management of the drone swarm.

[0025] The present invention provides a virtual-real combination simulation method supporting large-scale UAV cluster flight test, comprising the following steps:

[0026] S100, establishing a virtual-reality combined simulation system supporting large-scale UAV swarm flight tests as described in any one of claims 1 to 7;

[0027] S200, collecting real-time real drone data, establishing a virtual drone and collecting virtual drone data, loading pre-collected geographic real environment data, using the above data to process and analyze, construct a virtual combined drone three-dimensional model and display it in the corresponding geographic three-dimensional model to achieve three-dimensional visual display;

[0028] S300. The user inputs task instructions to the drone cluster through the application module. The data processing and analysis module calls the corresponding algorithm submodule to solve and calculate various instructions and send them to the drone cluster, so that the drone cluster performs corresponding command and dispatch, formation flight control, trajectory planning and trajectory tracking control according to the task instructions. The user can observe the command execution status of the drone cluster in the application module and obtain the corresponding task execution results; or the user directly monitors the status of the drone cluster through the ground station module and directly sends command and control instructions to the drone cluster; receives drone cluster command and control instructions from the application module, and sends command and control instructions to the drone cluster; sends task scheduling instructions to the application service module.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) Designed a modular simulation system architecture that combines virtual and real elements

[0031] The virtual drone cluster is combined with the real real-time drone cluster to provide simulation support for the underlying environment of the entire simulation system, covering cluster scheduling algorithm, task allocation algorithm, formation control algorithm, and dynamic control algorithm. It realizes task demand data analysis, task allocation and optimization scheduling, cluster command and control formation flight, cluster measurement and control and status monitoring, and intelligent collaboration of human-machine terminals based on cluster scheduling, task allocation, formation flight, trajectory planning, and trajectory tracking.

[0032] (2) Real-time communication between virtual nodes and real nodes based on the DDS communication mechanism was established, realizing the simulated interconnection between real drones and virtual drones; any drone can publish various data topics, such as current position, attitude, and speed; other drones can subscribe to these topics according to needs, and can also automatically process subscription requests for these data and provide corresponding data responses; overcoming the problem of hardware upper limit and inability to easily expand on a large scale;

[0033] (3) A visualization simulation system driven by real terrain was established. The map data uses real geographic elevation data to describe the ups and downs of the mountains on the map. Combined with 0.6-meter high-precision satellite image mapping, the real geographical features can be fully restored. Through its powerful physics engine, a high-simulation scene and a high-simulation drone model can be established to realize the display and simulation of real virtual drones in three-dimensional visualization scenes. Compared with obtaining the drone's posture information and external environment information through visual motion capture, this overcomes the problem of large time delay in the overall simulation implementation.

[0034] (4) The application module uses custom APP software to develop applications, allowing users to customize various scenarios according to simulation requirements and develop corresponding customized App software. This approach not only enhances the flexibility of simulation, but also improves its practicality and application scope;

[0035] (5) Wide-area ubiquitous human-machine terminal intelligent collaborative application mode. Facing the challenge of large-scale cluster command and control, users can send command instructions to the business logic layer through any terminal connected to the network architecture to accurately dispatch and control the drone cluster. This intelligent collaborative mode greatly improves the flexibility of operation and the efficiency of execution, achieving a wide-area ubiquitous intelligent collaborative application mode.

[0036] (6) A large-scale network node role custom attribute mode is implemented. For heterogeneous clusters, an attribute assignment mode is adopted. For any virtual drone node, it can be selected as a specified configuration, dynamic attribute, and mission payload to realize the role definition of different network nodes, establish the attributes and functions of different roles, and form script management of roles, so as to achieve flexible definition of the rapid role deployment of any drone node in a large-scale cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The present invention is a flowchart of a virtual-reality combined simulation method for supporting large-scale UAV cluster flight tests in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0039] Specific implementation plan 1: Combine Figure 1 As shown, the present invention provides a virtual-real combination simulation system supporting large-scale UAV cluster flight tests, comprising:

[0040] Real UAV data acquisition module, used to collect and send the UAV's position, attitude data and optoelectronic pod image data in real time;

[0041] Since the real drone hardware is equipped with a self-organizing data link radio and an optoelectronic pod, the real drone fleet composed of the self-organizing data link radio can transmit the position and attitude data of the drone and the image data of the optoelectronic pod to the data link radio on the ground side respectively;

[0042] Virtual drone creation module, used to create virtual drone dynamics and dynamics solution, and output the drone's position, attitude, speed, angular velocity, and timestamp data;

[0043] The virtual drone is developed based on ROS2 and PX4 software. PX4 is responsible for dynamics calculation. Multiple virtual drones communicate in a network under the ROS2 framework. The position and attitude information of the virtual drone is sent to the 3D vision module through the local area network for display.

[0044] The geographic real scene data acquisition module is used to load the pre-collected real geographic elevation data and the corresponding satellite image maps;

[0045] Adopting real geographic elevation data, the ups and downs of the mountains on the map are described through real geographic elevation data, and combined with 0.6-meter high-precision satellite image mapping, the real geographical features can be fully restored;

[0046] Data transmission module, used for data exchange, mutual access and service request sending among modules based on the data distribution service DDS mechanism;

[0047] Based on the ROS2 framework, the DDS communication protocol is used to realize real-time communication between virtual drone nodes and real drone nodes, and the simulation interconnection between real drones and virtual drones can be realized;

[0048] The data processing and analysis module is used to process and analyze real UAV data, virtual UAV data and geographic scene data, including cluster scheduling algorithm submodule, task allocation algorithm submodule, formation control algorithm submodule and dynamics control algorithm submodule;

[0049] The cluster scheduling algorithm submodule is used to schedule the drone cluster. According to the task instructions to be executed, the cluster scheduling algorithm is called to select the appropriate type and number of drone clusters;

[0050] The task allocation algorithm submodule is used to allocate specific tasks to each drone in the drone cluster. According to the task sequence to be executed, the task allocation algorithm is called to allocate a reasonable execution task sequence to the drone.

[0051] The formation control algorithm submodule is used to control the UAV cluster to perform formation flight;

[0052] The dynamics control algorithm submodule is used for the low-level flight control of the individual drone itself;

[0053] The 3D vision module is used to establish geographic simulation scenes and UAV simulation models through UE4, and realize the display of a virtual and real UAV simulation experimental platform;

[0054] The ground station module is used for users to directly monitor the status of the drone cluster and send command instructions to the drone cluster directly; receive the drone cluster command instructions from the application module and send command instructions to the drone cluster; send task scheduling instructions to the application service module;

[0055] The application module is used to realize the interaction between the user and the simulation system. The user can send command and control instructions to the drone through the application module to realize the scheduling and control of the drone cluster, including cluster scheduling, task allocation, formation flight, trajectory planning and trajectory tracking. The user can observe the execution status through the three-dimensional vision module and obtain the corresponding analysis conclusions of the execution status according to the data processing and analysis module;

[0056] Customized App development allows users to customize executable App applications for simulation task scenarios. Users can design and implement applications with different functions according to specific needs and goals, thereby maximizing work efficiency and simulation accuracy. Users can develop application modules and obtain required data from other subsystems based on the DDS distributed mechanism to customize the App.

[0057] In the application module, the App can analyze the necessary business logic according to the specific needs of the simulation and call the interface of the data processing and analysis module to realize complex large-scale cluster simulation scenarios; users can flexibly customize a variety of drone cluster application simulations, covering but not limited to reconnaissance, mapping, identification exploration and cluster strike scenarios; the setting of this module not only enhances the flexibility of the simulation, but also improves its practicality and application scope.

[0058] In the real UAV data acquisition module, since the real UAV has a complete flight control autopilot and self-organizing network data link radio; the flight control autopilot can accept instructions from the service distribution module, and the data link radio can transmit the status information of the UAV's position, attitude, speed, angular velocity, and timestamp back to the data link radio on the ground; the data link radio on the ground publishes various data topics to the data processing and analysis module.

[0059] In the virtual drone data creation module, the dynamic simulation of the virtual drone runs on a professional workstation computer. The workstation and the virtual drone data creation module are connected by a network cable, and the state quantity information of the virtual drone's position, attitude, speed, angular velocity, and timestamp is published to the data processing and analysis module through the local area Ethernet.

[0060] In the virtual drone data creation module, an attribute assignment mode is adopted for heterogeneous clusters (drone swarms); for any virtual drone node, it can be selected as a specified configuration (multi-rotor, fixed-wing, vertical take-off), and its dynamic properties (mass, inertia, propeller parameters, aerodynamic data) can be set. Set its mission payload (pod, ammunition), realize the role definition of different network nodes, establish the attributes and functions of different roles, and form a script management of the drone swarm.

[0061] In the data processing and analysis module, any virtual or real flight node can subscribe to the data topics published by other nodes (obtain the state information of the position, attitude, speed, angular velocity, and timestamp of other nodes). According to the business logic, the node calls its own algorithm sub-module, including the cluster scheduling algorithm sub-module, the task allocation algorithm sub-module, the formation control algorithm sub-module, and the dynamics control algorithm sub-module, to perform automatic command and control of the node.

[0062] In the 3D vision module, the position, attitude, speed, angular velocity, and timestamp status information of all drones can be subscribed to from the real drone data acquisition module and the virtual drone creation module. Based on this information, the real drones and virtual drones can be simulated and demonstrated in the 3D vision module together.

[0063] In the above modules, any module can publish data topics to the data service network, and any module can subscribe to and obtain topic data published by other modules through the data service network.

[0064] Specific implementation plan 2: Combination Figure 1 As shown, the present invention provides a virtual-real combination simulation method supporting large-scale UAV cluster flight tests, comprising:

[0065] S100, establish a virtual-reality combined simulation system to support large-scale UAV swarm flight tests;

[0066] S200, collecting real drone data in real time, establishing a virtual drone and collecting virtual drone data, loading pre-collected real geographic environment data, and constructing a virtual combined drone three-dimensional model after processing and analyzing the above data and displaying it in the corresponding geographic three-dimensional model to achieve three-dimensional visual display;

[0067] S300. The user inputs task instructions to the drone cluster through the application module. The data processing and analysis module calls the corresponding algorithm submodule to solve and calculate various instructions and send them to the drone cluster, so that the drone cluster performs corresponding command and dispatch, formation flight control, trajectory planning and trajectory tracking control according to the task instructions. The user can observe the command execution status of the drone cluster in the application module and obtain the corresponding task execution results; or the user directly monitors the status of the drone cluster through the ground station module and directly sends command and control instructions to the drone cluster; receives drone cluster command and control instructions from the application module, and sends command and control instructions to the drone cluster; sends task scheduling instructions to the application service module.

[0068] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0069] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A virtual-real combination simulation system supporting large-scale UAV swarm flight tests, characterized by: include, Real UAV data acquisition module, used to collect the position and attitude data of the UAV and the image data of the optoelectronic pod in real time; Virtual drone creation module, used to create virtual drone dynamics and dynamics solution, and output the drone's position, attitude, speed, angular velocity, and timestamp data; The geographic real scene data acquisition module is used to load the pre-collected real geographic elevation data and the corresponding satellite image maps; Data transmission module, used for data exchange, mutual access and service request sending among modules based on the data distribution service DDS mechanism; The data processing and analysis module is used to process and analyze real UAV data, virtual UAV data and geographic scene data, including cluster scheduling algorithm submodule, task allocation algorithm submodule, formation control algorithm submodule and dynamics control algorithm submodule; The 3D vision module is used to establish geographic simulation scenes and UAV simulation models through UE4, and realize the display of a virtual and real UAV simulation experimental platform; The application module is used to realize the interaction between the user and the simulation system. The user can send command and control instructions through the application module to dispatch and control the drone cluster, including cluster scheduling, task allocation, formation flight, trajectory planning and trajectory tracking. The user can observe the execution status through the three-dimensional vision module and obtain the corresponding analysis conclusions of the execution status according to the data processing and analysis module; The ground station module is used for users to directly monitor the status of the drone cluster and send command instructions directly to the drone cluster; Receive UAV cluster command and control instructions from the application module, send command and control instructions to the UAV cluster; send task scheduling instructions to the application service module.

2. The virtual-reality combined simulation system supporting large-scale UAV swarm flight tests according to claim 1 is characterized by: In the data transmission module, based on the DDS distributed communication mechanism, any module can publish any data topic, and any module can subscribe to obtain the topic data published by other modules, realizing a distributed communication mechanism.

3. The virtual-reality combined simulation system supporting large-scale UAV swarm flight tests according to claim 2 is characterized by: The application module adopts customized App development, and users customize App applications according to simulation task scenarios; users design and implement the functions of the application according to specific needs and goals; Users develop application modules based on the DDS distributed mechanism and customize the App by obtaining the required data from other subsystems.

4. The virtual-reality combined simulation system supporting large-scale UAV swarm flight tests according to claim 3 is characterized by: In the application module, the App calls the interface of the data processing and analysis module according to the specific needs of the simulation to realize large-scale cluster simulation scenarios; users can customize a variety of drone cluster application simulations, including reconnaissance, mapping, identification and exploration, and cluster strike scenarios.

5. The virtual-reality combined simulation system supporting large-scale UAV swarm flight tests according to claim 4 is characterized by: Users send command and control instructions to the simulation system through the App in the terminal connected to the network architecture to realize the scheduling and control of the drone cluster; the command and control instructions include adding new task requirements, allowing the drone cluster to plan tasks autonomously, and specifically specifying the execution path of the task, so that the drone cluster can perform the task according to the preset path.

6. The virtual-reality combined simulation system supporting large-scale UAV swarm flight tests according to claim 5 is characterized by: In the real drone data acquisition module and the virtual drone data creation module, the output drone data includes the state quantity information of the drone's position, attitude, speed, angular velocity, and timestamp.

7. The virtual-reality combined simulation system supporting large-scale UAV swarm flight tests according to claim 6 is characterized by: In the virtual drone data creation module, the attribute assignment mode is adopted; for any virtual drone node, it can be selected as a specified configuration, specified dynamic properties and specified mission payload to realize the role definition of different network nodes, establish the properties and functions of different roles, and form a script management of the drone swarm.

8. A virtual-real combination simulation method supporting large-scale UAV swarm flight tests, characterized in that: The following steps are involved: S100, establishing a virtual-reality combined simulation system supporting large-scale UAV swarm flight tests as described in any one of claims 1 to 7; S200, collecting real drone data in real time, establishing a virtual drone and collecting virtual drone data, loading pre-collected geographic real environment data, using the above data to process and analyze, construct a virtual combined drone three-dimensional model and display it in the corresponding geographic three-dimensional model, to achieve three-dimensional visual display; S300, the user inputs the task instruction to the drone cluster through the application module, the data processing and analysis module calls the corresponding algorithm submodule, solves and calculates various instructions and sends them to the drone cluster, so that the drone cluster performs the corresponding command and dispatch, formation flight control, trajectory planning and trajectory tracking control according to the task instruction. The user can observe the command execution of the drone cluster in the application module and obtain the corresponding task execution results; or the user directly monitors the status of the drone cluster through the ground station module and directly sends the command and control instructions to the drone cluster; receives the drone cluster command and control instructions from the application module and sends the command and control instructions to the drone cluster; Send task scheduling instructions to the application service module.

Images