Unmanned aerial vehicle simulation system

Abstract

The invention relates to an unmanned aerial vehicle simulation system, relates to the technical field of digital twinning, and solves the technical problem that an existing unmanned aerial vehicle simulation method is insufficient in simulation flexibility, and the system comprises a first script which comprises configuration of a simulation unmanned aerial vehicle and a simulation environment; the analog simulation platform is configured with a first script interpreter to operate the first script, and the analog simulation platform is further adapted to configure the unmanned aerial vehicle simulation plug-in as simulation middleware of the analog simulation platform; the unmanned aerial vehicle simulation plug-in is integrated with unmanned aerial vehicle flight control firmware; and the unmanned aerial vehicle control software is used for sending an unmanned aerial vehicle flight control instruction to the unmanned aerial vehicle flight control firmware through a flight control communication protocol, so that a simulation unmanned aerial vehicle in the analog simulation platform executes the unmanned aerial vehicle flight control instruction in a simulation environment.

Description

Technical Field

[0001] The present invention relates to the field of digital twin technology, and in particular to an unmanned aerial vehicle simulation system. Background Art

[0002] Currently, drones, especially rotary-wing drones, are very popular. When researchers are developing drone algorithms, they often face the difficulty of not being able to fly drones in their location, which seriously affects the development efficiency. In addition, in the process of developing algorithms, researchers often need to constantly debug and optimize the algorithms. However, if they have to test them on real drones every time, this will increase the time investment and manpower costs of research and development, and face flight safety accidents caused by insufficient algorithm performance.

[0003] This inspired researchers to use digital twins to build a simulation environment for drones through simulation methods, and then conduct experiments in the simulation environment. Existing drone simulation methods have problems such as low fidelity and insufficient simulation flexibility. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a UAV simulation system to solve the technical problem of insufficient simulation flexibility of existing UAV simulation methods.

[0005] In a first aspect, an embodiment of the present invention provides a drone simulation system, comprising:

[0006] The first script includes the configuration of the simulated drone and the simulation environment;

[0007] A simulation platform, configured with a first script interpreter to run the first script, and further adapted to configure the drone simulation plug-in as a simulation middleware of the simulation platform;

[0008] The drone simulation plug-in is integrated with the drone flight control firmware; and

[0009] The drone control software is used to send the drone flight control instructions to the drone flight control firmware through the flight control communication protocol so that the simulated drone in the simulation platform executes the drone flight control instructions in the simulation environment.

[0010] Based on further improvement of the above simulation system, the drone control software is a second script, and the second script includes a drone intelligent algorithm, and the drone intelligent algorithm is executed to obtain the drone flight control instructions.

[0011] Based on the further improvement of the above simulation system, the UAV control software is ground control station software.

[0012] Based on the further improvement of the above-mentioned simulation system, the first script also includes the configuration of sensor data and label data, and the drone simulation system is also adapted to train the drone intelligent algorithm using the sensor data and the label data.

[0013] Based on the further improvement of the above-mentioned simulation system, the first script also includes the configuration of sensor data, and the UAV simulation device is also adapted to use the UAV intelligent algorithm to infer the sensor data to obtain the UAV flight control instructions.

[0014] Based on the further improvement of the above simulation system, the first script also includes the configuration of the robot operating system.

[0015] The simulation platform is configured such that when the first script is executed, the sensor data or the tag data is published in the form of a topic message in the robot operating system.

[0016] Based on further improvement of the above simulation system, the flight control communication protocol is the MAVLINK protocol, and the second script includes UAV flight control instructions of a robot operating system node in the form of MAVROS.

[0017] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0018] 1. The solution of the present invention adopts a more flexible script form to complete the construction of the UAV simulation environment, thereby improving the flexibility of the UAV simulation method.

[0019] 2. The solution of the present invention adopts a simulation platform with sufficient computing power to achieve a more realistic UAV virtual simulation method.

[0020] 3. The solution of the present invention realizes the training and reasoning of UAV intelligent algorithms based on a simulation environment.

[0021] 4. The solution of the present invention saves the time of UAV algorithm development and reduces the manpower cost and debugging cost of UAV algorithm development.

[0022] 5. The solution of the present invention establishes a simulation association with the existing flight control and ground control stations.

[0023] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0025] Figure 1 A structural block diagram of a UAV simulation system according to an embodiment of the present invention is shown.

[0026] Figure 2 A schematic diagram of a process of performing drone simulation by a drone simulation system according to an embodiment of the present invention is shown.

[0027] Figure 3 A structural diagram of a UAV simulation system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0029] Figure 1 FIG. 2 shows a structural block diagram of a UAV simulation system according to an embodiment of the present invention. Figure 1 As shown, the UAV simulation system includes:

[0030] 1. Simulation platform

[0031] The simulation platform can be a 3D robot simulation platform, which can provide a fully functional simulation environment and support accurate simulation of robots, sensors and complex three-dimensional worlds. In this embodiment, the simulation platform needs to support a first script interpreter of a programming language such as Python, C++, Java, etc. to run the first script (described in detail below). For example, the simulation platform supports a Python script interpreter to run Python scripts. The existing drone simulation method uses the Gazebo simulation platform, but because its fidelity is not high and because it does not support a Python script interpreter to run the Python script, the simulation flexibility is not enough.

[0032] In some embodiments, the simulation platform can be the Isaac Sim simulation platform developed by NVIDIA. Isaac Sim provides computing power support based on NVIDIA's graphics card, which improves the performance of the simulation. Isaac Sim supports navigation and operation applications through ROS (Robot Operating System) / ROS2, allowing researchers and engineers to develop and test in the familiar ROS framework.

[0033] The Isaac Sim simulation platform includes scenes, simulated aircraft, cameras, action graphs, replicators, physics engines, rendering engines, etc. The scene is the background model in the Isaac Sim simulation platform. The simulated aircraft is the simulated aircraft model in the Isaac Sim simulation platform. The camera is a virtual camera in the Isaac Sim simulation platform, which can collect images, depth maps, semantic segmentation maps, and can set various acquisition parameters. The action graph is a graphical programming method in the Isaac Sim simulation platform. The replicator is a software tool in the Isaac Sim simulation platform for synchronously generating simulation label data. The physics engine is a physical simulation engine in the Isaac Sim simulation platform, which can simulate gravity, contact, collision, gas flow, etc. The rendering engine is a software tool for rendering and visualization in the Isaac Sim simulation platform.

[0034] Since the Isaac Sim simulation platform supports a script interpreter to run the script, the scenes, simulated aircraft, cameras, action graphs, replicators, physics engines, rendering engines, etc. in the Isaac Sim simulation platform can all be called through scripts, such as Python scripts.

[0035] 2. First Script

[0036] A script is a computer program that usually consists of a series of commands or instructions to automate specific tasks. Scripting languages ​​include but are not limited to Python, C++, Java and other programming languages.

[0037] In this embodiment, the first script includes the configuration of the simulated drone and the simulation environment. In order to start the entire simulation environment, thanks to the fact that the simulation platform can be started by script, the first script can be written here, and the scene definition, the simulated aircraft definition, the camera definition, and the physical engine and rendering engine settings are performed therein.

[0038] After the first script is written, the first script can be run through a first script interpreter supported by the simulation platform, thereby starting the simulation platform and executing the content in the first script.

[0039] In the embodiment where the simulation platform is Isaac Sim, thanks to the fact that Isaac Sim can be started by means of Python scripts, a Python script can be written here, and the scene, the simulation aircraft, the camera, the action graph, and the replicator, the physics engine, and the rendering engine can be defined therein. After the Python script is written, the Python script can be run here by the Python interpreter supported by Isaac Sim, thereby starting the entire Isaac Sim software and executing the contents in the script.

[0040] 3. Drone simulation plug-in

[0041] The drone simulation plug-in here can simulate the aerodynamics of the drone. The drone simulation plug-in is integrated with the drone flight control firmware, which provides a bridge between the drone flight control firmware and the simulation platform.

[0042] UAV flight control firmware refers to the software system used to control the flight of the UAV. It is responsible for processing the flight control algorithm and motion planning of the UAV. Open source flight control firmware includes PX4 and AutoPilot. PX4 firmware is suitable for a variety of aircraft such as multi-rotor and fixed-wing aircraft, while AutoPilot firmware is suitable for fixed-wing flight.

[0043] In some embodiments, the drone simulation plug-in can support a second script interpreter of a programming language such as Python, C++, Java, etc. to run the second script (described in detail below). For example, the drone simulation plug-in supports a Python script interpreter to run a Python script. The drone simulation plug-in can be Pegasus Simulator. PegasusSimulator is a modular framework that is an extended implementation of Isaac Sim. It can simulate multiple multi-rotor aircraft in real time in a realistic environment, and provides plug-and-play integration with PX4, Autopilot, and ROS2 through its modular implementation and intuitive graphical user interface. Pegasus Simulator also provides a custom Python control interface.

[0044] 4. Drone control software

[0045] The drone control software here can provide flight control and mission planning for drones that support flight control communication protocols. The flight control communication protocol can be the MAVlink protocol. The MAVLink protocol is a lightweight, efficient, and reliable communication protocol designed specifically for drones and robotic systems. It is mainly used for two-way communication between drones (UAVs) and ground control stations (GCS). The MAVLink protocol is widely supported due to its efficiency, reliability, flexibility, scalability, and compatibility, and has become the standard communication protocol for PX4 and ArduPilot.

[0046] In some embodiments, the drone control software may be QGroundControl, which is a ground station software that can connect to the aircraft, view the aircraft status, and send commands.

[0047] In some embodiments, the drone control software may be a second script. The second script includes a drone intelligent algorithm. The drone intelligent algorithm refers to a neural network algorithm, including target recognition, autonomous flight, autonomous navigation planning, etc. A second script may be written here, mainly including setting the drone control mode to Offboard mode in the script and writing a series of codes to extract data from camera data, inertial navigation data and / or lidar data, and calling the drone intelligent algorithm to obtain the drone flight control instruction, and then sending the drone flight control instruction to the drone flight control firmware through the flight control communication protocol so that the simulated drone in the simulation platform executes the drone flight control instruction in the simulation environment, and the flight of the simulated drone will be displayed in real time in the simulation platform. Offboard mode is a PX4 flight mode for handing over control of the drone to an external computer program. The second script here may also be a Python script.

[0048] In some embodiments, the first script also includes the configuration of sensor data and label data. In this embodiment, the definition of sensors and the setting of replicators can be performed in the first script. The sensors here can include cameras, inertial navigation, laser radars, etc. The labels here can include depth image labels, semantic segmentation labels, etc. In the training phase of the drone intelligent algorithm, Isaac Sim is started and the content in the first script is executed, and the data and label data collected by the drone sensor can be published in Isaac Sim. Label data, for example, includes depth images, semantic segmentation labels, etc. These labels are automatically calculated by the Isaac Sim engine and have the characteristics of high accuracy, high frame rate, and high resolution. After collecting enough sensor data and label data, these data can be saved locally, and then the drone intelligent algorithm can be trained or fine-tuned. The training or fine-tuning of the drone intelligent algorithm can be implemented in IsaacSim through the second script.

[0049] In some embodiments, the first script also includes the configuration of sensor data. In this embodiment, the definition of the sensor can be performed in the first script. The sensors here may include cameras, inertial navigation, lidar, etc. In the drone intelligent algorithm reasoning stage, Isaac Sim is started and the content in the first script is executed, and the data collected by the drone sensor can be published in Isaac Sim. After collecting the sensor data, the drone intelligent algorithm is applied to infer the information in the sensor data. The application of the drone intelligent algorithm can be implemented in Isaac Sim through the second script. At this time, the trained drone intelligent algorithm can be run through the script, and the effect of the drone flight will be directly reflected in the Isaac Sim visual interface.

[0050] In some embodiments, the first script also includes the configuration of ROS. In this embodiment, the definition of ROS-related environment and parameters can be performed in the first script. Start Isaac Sim and execute the content in the first script. The content published by Isaac Sim can be published through ROS topic messages. For example, in the drone intelligent algorithm training stage, the ROS topic message output by Isaac Sim contains the collected sensor data and label data; in the drone intelligent algorithm reasoning stage, the ROS topic message output by Isaac Sim contains the collected sensor data. By defining the ROS-related environment and parameters, Isaac Sim can communicate with the external ROS system.

[0051] In some embodiments, MAVROS middleware can be inserted into the second script. MAVROS is a middleware used to convert MAVLink messages into ROS2 topic messages, and it is a ROS2 node. Running the second script starts MAVROS, and at this time, the MAVLink message can be converted into a ROS2 topic message. A Python script can also be written here, and the control mode of the drone is set to Offboard mode in the script, and a series of codes can be written, so that data can be extracted from the camera images, IMU data, and lidar data in the ROS2 topic published by the IsaacSim platform, and the drone intelligent algorithm is called to obtain the drone flight control instruction, and then the topic related to drone control in MAVROS is called to control the drone to fly in the simulation environment. In this embodiment, the instruction form of the drone flight control instruction is a ROS2 node in the form of MAVROS.

[0052] Figure 2 A schematic diagram of a process of performing drone simulation by a drone simulation system according to an embodiment of the present invention is shown.

[0053] like Figure 2 As shown, the UAV simulation process can be implemented through NVIDIA's Isaac Sim 4.1, and Pegasus can be configured as the simulation middleware in the software interface of Isaac Sim 4.1. In Pegasus, the fluid mechanics parameters of the environment to be simulated are configured and defined. The flight control firmware can also be defined in Pegasus, and PX4-AUTOPILOT can be configured as the flight control firmware during the UAV simulation process. In order to start the entire simulation environment, thanks to the fact that IsaacSim can be started through a python script, a python script can be written here, and the scene definition, simulation aircraft definition, camera definition, ROS2 related environment and parameters definition, action graph definition, replicator settings, physics engine and rendering engine settings are successively performed.

[0054] When writing a python script, in the scene definition stage, you can load the pre-simulated city scene model into the simulation environment, and then configure the simulated aircraft supported by Pegasus. The simulated aircraft supports camera settings by default, and here you also need to set the position and posture of the camera on the drone. After that, you can make relevant settings for ROS2 to communicate between the Isaac Sim environment and the external ROS2 system. Then you can write an action graph to allow the position of the drone in the simulation scene and simple flight to be controlled by the keyboard. If it is the training data collection stage, here you can add and configure the replicator supported in Isaac Sim to the scene, and then set the physics engine and rendering engine separately, so that the objects and aircraft in the simulation environment have physical properties and realistic visual effects.

[0055] After writing the Python script, you can run the Python script through the python interpreter supported by Isaac Sim to start the entire Isaac Sim software and execute other contents in the script. At this point, the city scene, drone, light source, etc. have been configured, and the drone handles the ground stop state. PX4 has been started at this time, and the image data collected by the drone camera has been published through the ROS2 topic message.

[0056] Here you can install and configure the QGroundControl software, which is used to configure the status of the simulated drone and issue commands. The communication protocol between it and PX4 is mavlink. Any command issued to the drone in QGroundControl will be directly reflected in the execution of the drone action in the simulation environment. You can also connect an external remote control here to control the drone flight in the simulation environment through the manual mode of QGroundControl. The whole process is the same as operating a real drone.

[0057] Then, you can start the ROS2 node mavros. At this time, the mavlink message has been converted to ros2 message. Here you can also write a python script, set the drone control mode to Offboard mode in the script, and write a series of codes to extract data from the camera image data in the ROS2 topic, call the drone intelligent algorithm to get the drone control command, and then call the drone control related topics in mavros to control the drone to fly in the simulation environment. The drone flight will be displayed in real time.

[0058] In the case of label data collection, that is, when adding the Isaac Sim replicator mentioned above, the ros2 message output by Isaac Sim contains the collected label data, specifically including depth images, semantic segmentation labels, etc. These labels are automatically calculated by the Isaac Sim engine and have the characteristics of high accuracy, high frame rate, high resolution, etc. After collecting enough images, depth maps, and semantic segmentation labels, these data can be saved locally, and then the drone intelligent algorithm can be trained or fine-tuned.

[0059] Figure 3 FIG. 2 shows a structural diagram of a UAV simulation system according to an embodiment of the present invention. Figure 3 As shown, first, obtain the scene model and introduce the model of the simulated aircraft into the scene model. Then, add the camera model or other sensor models to the simulated aircraft model. Subsequently, configure the data input and output interfaces of these sensors to ROS2, and complete the definition of ROS2. The ROS2 mentioned here is integrated in the simulation tool, and the simulation tool can be a simulation platform such as Isaac Sim and AirSim. These simulation platforms provide a physics engine and a rendering engine, where the physics engine is responsible for the aerodynamic simulation of the drone.

[0060] Drone simulation plug-ins (such as Pegasus Simulator) use the aerodynamic simulation capabilities of the physics engine to convert algorithms from flight control programs (such as PX4 or Autopilot) into controls for simulated aircraft in the simulation scene. The ROS2 definitions integrated into the simulation tool can communicate directly with external computer programs, so that external computer programs can obtain sensor data in the simulation scene and process or manipulate these data. At the same time, the external computer program can also control the flight control program to control the aircraft in the simulation environment to fly. In addition, the external computer program can also call the labeled data generated by the rendering engine to achieve custom flight or intelligent flight.

[0061] Compared with the prior art, the embodiments of the present invention can achieve at least one of the following beneficial effects:

[0062] 1. The solution of the present invention adopts a more flexible script form to complete the construction of the UAV simulation environment, thereby improving the flexibility of the UAV simulation method.

[0063] 2. The solution of the present invention adopts a simulation platform with sufficient computing power to achieve a more realistic UAV virtual simulation method.

[0064] 3. The solution of the present invention realizes the training and reasoning of UAV intelligent algorithms based on a simulation environment.

[0065] 4. The solution of the present invention saves the time of UAV algorithm development and reduces the manpower cost and debugging cost of UAV algorithm development.

[0066] 5. The solution of the present invention establishes a simulation association with the existing flight control and ground control stations.

[0067] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A drone simulation system, characterized in that: include: The first script includes the configuration of the simulated drone and the simulation environment; A simulation platform, configured with a first script interpreter to run the first script, and further adapted to configure the drone simulation plug-in as a simulation middleware of the simulation platform; The drone simulation plug-in is integrated with the drone flight control firmware; as well as The drone control software is used to send the drone flight control instructions to the drone flight control firmware through the flight control communication protocol so that the simulated drone in the simulation platform executes the drone flight control instructions in the simulation environment.

2. The unmanned aerial vehicle simulation system according to claim 1, characterized in that: The drone control software is a second script, which includes a drone intelligent algorithm. The drone intelligent algorithm is executed to obtain the drone flight control instructions.

3. The unmanned aerial vehicle simulation system according to claim 1, characterized in that: The drone control software is ground control station software.

4. The unmanned aerial vehicle simulation system according to claim 2, characterized in that: The first script also includes configuration of sensor data and label data, and the drone simulation device is also adapted to train the drone intelligent algorithm using the sensor data and the label data.

5. The unmanned aerial vehicle simulation system according to claim 2, characterized in that: The first script also includes configuration of sensor data, and the drone simulation system is also adapted to use the drone intelligent algorithm to infer the sensor data to obtain the drone flight control instructions.

6. The unmanned aerial vehicle simulation system according to claim 4 or 5, characterized in that: The first script also includes configuration of the robot operating system, The simulation platform is configured such that when the first script is executed, the sensor data or the tag data is published in the form of a topic message in the robot operating system.

7. The unmanned aerial vehicle simulation system according to claim 2, characterized in that: The flight control communication protocol is the MAVLINK protocol, and the second script includes UAV flight control instructions of a robot operating system node in the form of MAVROS.

Images