A multi-UAV perception and avoidance verification system and method combining virtual and real

Abstract

The present invention discloses a multi-UAV perception avoidance verification system and method combining virtual and real, and the system includes: a physical platform, which includes a positioning system, a physical UAV and a communication interconnection device, a simulation platform, which is used to construct an experimental scenario, a simulated UAV and a mirrored UAV in a simulation environment, and the pose of the mirrored UAV is obtained by real-time mapping according to the pose of the physical UAV; an interaction module, which is used to realize data interaction between the physical platform and the simulation platform; the physical platform obtains the data of the sensors carried by the mirrored UAV in real time, and calculates the control quantity of the UAV according to the preset perception avoidance verification task, so as to control the movement of the physical UAV. The present invention realizes multi-UAV perception avoidance verification based on the combination of virtual and real, can fully simulate the complex wind field in the physical environment, and has the advantages of low implementation cost, strong real-time performance, high experimental fidelity and reliability, etc.

Description

Technical Field

[0001] The present invention relates to the technical field of UAV swarm performance simulation and verification, and particularly to a multi-UAV perception and avoidance verification system and method combining virtual and real scenarios. Background Art

[0002] With the continuous expansion of the application fields of UAVs, the current requirements for the automation level and execution efficiency of UAV missions are constantly increasing. Compared with single UAVs, multi-UAVs can carry more mission payloads and have higher mission efficiency. Perception and avoidance is one of the basic technologies for UAVs to perform missions. For UAV swarms, perception and avoidance not only include avoiding obstacles in the environment, but also include avoiding collisions between aircraft, so it is more challenging.

[0003] Currently, perception and avoidance verification methods are mainly divided into three categories: simulation experiments, semi-physical experiments, and physical experiments. Among them, simulation experiments are further divided into two types: Software In The Loop experiments and numerical simulation experiments. Semi-physical experiments are also known as Hardware In The Loop experiments. Among them, the verification effect of physical experiments is the most sufficient but the cost is high. The cost of simulation experiment methods is the lowest but the fidelity is the worst. The semi-physical simulation method is between the two. The Hardware In The Loop can fully verify the real-time performance and reliability of flight control algorithms.

[0004] For the verification of multi-UAV perception and avoidance, the above-mentioned various experimental methods will specifically have the following problems:

[0005] (1) Simulation experiment methods usually use simulation experiment platforms such as AirSim, Aerostack, and XTDrone. However, such experimental platforms cannot verify the feasibility and reliability of algorithms on actual UAV platforms, and limited by the fidelity of the simulation platform, simulation experiments cannot fully simulate complex wind fields in the physical environment, such as the computing power of the on-board computing platform and complex wind fields during actual flight.

[0006] (2) The semi-physical experiment method cannot verify the real-time performance and reliability of algorithms on actual on-board computing platforms and cannot fully simulate complex wind fields in the physical environment.

[0007] (3) Physical experiments (especially multi-aircraft physical experiments) require a large amount of manpower and material resources to prepare the experimental platform and maintain the experimental aircraft, so the experimental cost is high and the flexibility is poor.

[0008] Therefore, there is an urgent need to provide a multi-UAV perception and avoidance verification system and method, so as to combine the advantages of simulation experiments and physical experiments, while being able to avoid the defects of simulation experiments and physical experiments, and taking into account the cost and reliability of experimental verification. Summary of the Invention

[0009] The technical problem to be solved by the present invention is: aiming at the technical problems existing in the prior art, the present invention provides a virtual-real combined multi-UAV perception and avoidance verification system and method that can fully simulate the complex wind field in the physical environment, and has low implementation cost, strong real-time performance, high experimental fidelity and reliability.

[0010] A virtual-real combined multi-UAV perception and avoidance verification system, comprising:

[0011] A physical platform, including a positioning system, physical UAVs and required communication and interconnection devices;

[0012] A simulation platform, used to construct an experimental scenario, simulation UAVs and mirror UAVs in a simulation environment. The simulation UAVs are constructed using a dynamic model to simulate the physical UAVs. The mirror UAVs are configured as mirrors of the physical UAVs, and the poses of the mirror UAVs are obtained by real-time mapping according to the poses of the physical UAVs;

[0013] An interaction module, used to realize data interaction between the physical platform and the simulation platform;

[0014] The physical platform obtains the operation data of the mirror UAVs in real time, and calculates the control quantity of the UAVs according to the preset perception and avoidance verification tasks to control the movement of the physical UAVs.

[0015] Further, sensors are mounted on both the simulation UAVs and the mirror UAVs. The mirror UAVs and the simulation UAVs perceive each other through the sensors mounted on them respectively. The physical platform obtains the operation data of the mirror UAVs in real time through the sensors mounted on the mirror UAVs.

[0016] Further, the sensor is a collision detection sensor.

[0017] Further, the interaction module communicates with the physical platform and the simulation platform respectively through communication based on ROS (Robot Operating System), and the control programs in the physical platform and the simulation platform and the UAVs adopt the MAVLink (Micro Air Vehicle Link) protocol to control the movement of the UAVs through MAVROS (MAVLink extendable communication node for ROS).

[0018] Further, the parameters of the simulated unmanned aerial vehicle (UAV) are the same as those of the physical UAV. Both the mirror UAV and the simulated UAV have visual volume and physical collision volume. The mirror UAV follows the real-time pose of the physical UAV through a pose follower.

[0019] Further, the mirror UAV has the same shape as the simulated UAV. The rotors of the mirror UAV are configured not to rotate and the mirror UAV does not have dynamic characteristics. The number and position of the mirror UAVs are configured according to the number and position of the physical UAVs.

[0020] Further, a computing platform is also carried on the physical platform, which is used to obtain the operation data of the mirror UAV in real time and to realize the real-time control of the physical UAV.

[0021] Further, the physical platform also includes a binocular vision or motion capture system for obtaining the pose information of the UAV in an indoor test environment.

[0022] A method for verifying multi-UAV perception and avoidance in a virtual-real combination, the steps include:

[0023] S1. Verification system setup: A physical platform is constructed by the physical UAV and the required communication and interconnection devices, and an experimental scenario, a simulated UAV, and a mirror UAV are constructed in a simulation environment to construct a simulation platform. The mirror UAV is the mirror image of the physical UAV. The simulated UAV is constructed using a dynamic model to simulate the physical UAV. The mirror UAV is configured as the mirror image of the physical UAV, and the pose of the mirror UAV is obtained by real-time mapping according to the pose of the physical UAV. A communication connection is established between the physical platform and the simulation platform;

[0024] S2. Perception and avoidance verification: Start the physical UAV and the simulation environment, and control the physical UAV and the simulated UAV according to a preset perception and avoidance verification task. The physical UAV obtains the operation data of the mirror UAV in real time, calculates the control quantity of the UAV according to the preset perception and avoidance verification task, and the physical UAV moves according to the calculated control quantity.

[0025] S3. Verification result output: Obtain and output the operation data of the physical UAV and the simulated UAV during the verification test.

[0026] Further, in step S1, the number and position of the corresponding mirror UAVs are configured according to the number and position of the physical UAVs.

[0027] To solve the above technical problems, the technical solution proposed by the present invention is:

[0028] Compared with the prior art, the advantages of the present invention are as follows: By constructing a physical platform and a simulation platform, the system includes both physical drones and simulated drones. At the same time, a mirror drone of the physical drone is also constructed in the simulation environment, and the pose of the mirror drone is obtained by real-time mapping according to the pose of the physical drone. The physical drone determines the real-time control amount by perceiving the mirror drone, which can realize the perception avoidance verification of the combination of virtual and real, and can give full play to the advantages of low cost of simulation experiments and more sufficient verification of physical experiments. Only a small number of physical drones are needed to verify the feasibility, reliability, real-time performance, etc. of various perception avoidance algorithms on actual drone platforms, autopilots, etc., ensuring the accuracy of verification, solving the problem of insufficient fidelity in the verification of traditional pure simulation systems, and it is convenient to add virtual drones in the simulation platform to realize large-scale scalable experiments, so as to verify various perception avoidance algorithms for large-scale drone swarms, while saving a large amount of manpower and material resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 FIG. is a schematic structural diagram of the multi-drone perception avoidance verification system combining virtual and real in this embodiment.

[0030] Figure 2 FIG. is a schematic diagram of the deployment principle of the multi-drone perception avoidance verification system in this embodiment.

[0031] Figure 3 FIG. is a schematic flowchart of the implementation process of the multi-drone perception avoidance verification method combining virtual and real in a specific embodiment of the present invention.

[0032] Figure 4 FIG. is a schematic diagram of the effect of the simulation environment constructed in a specific embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The present invention will be further described below in conjunction with the accompanying drawings of the specification and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0034] As Figure 1 shown, the multi-drone perception avoidance verification system combining virtual and real in this embodiment includes:

[0035] A physical platform, including a positioning system, physical drones, and the required communication and interconnection equipment;

[0036] A simulation platform, used to construct an experimental scenario, simulated drones, and mirror drones in a simulation environment. The simulated drones are constructed using a dynamic model to simulate the physical drones, and the mirror drones are configured as mirrors of the physical drones, and the pose of the mirror drones is obtained by real-time mapping according to the pose of the physical drones;

[0037] An interaction module for realizing data interaction between the physical platform and the simulation platform;

[0038] The physical platform obtains the operation data of the mirror unmanned aerial vehicle (UAV) in real time, calculates the control quantity of the UAV according to the preset perception and avoidance verification task, and controls the movement of the physical UAV.

[0039] In this embodiment, by constructing a physical platform and a simulation platform, both a physical UAV and a simulation UAV are included in the system. In addition to constructing the simulation UAV in the simulation platform, a mirror UAV of the physical UAV is also constructed. The pose of the mirror UAV is obtained by real-time mapping according to the pose of the physical UAV. The physical UAV determines the real-time control quantity by perceiving the mirror UAV, and can realize the perception and avoidance verification of the combination of virtual and real, and can give full play to the advantages of low cost of simulation experiments and more sufficient verification of physical experiments. Only a small number of physical UAVs are needed to verify the feasibility, reliability, real-time performance, etc. of various perception and avoidance algorithms on the actual UAV platform, autopilot, etc., ensure the accuracy of verification, solve the problem of insufficient fidelity in the verification of traditional pure simulation systems, and it is convenient to add virtual UAVs in the simulation platform to realize large-scale scalable experiments, so as to verify various perception and avoidance algorithms for large-scale UAV clusters, while saving a large amount of costs.

[0040] In this embodiment, sensors are installed on both the simulation UAV and the mirror UAV, and mutual perception between UAVs is realized based on the sensors: mutual perception between the mirror UAV and the simulation UAV is realized through the sensors installed on each of them, and the physical platform obtains the operation data of the mirror UAV in real time through the sensors installed on the mirror UAV to realize perception and avoidance between multiple UAVs. The above sensors can specifically use a two-dimensional lidar as a collision detection sensor for collision detection. Of course, other types of radars or other types of collision detection sensors can also be used.

[0041] In this embodiment, the interaction module communicates with the physical platform and the simulation platform respectively based on ROS, and the MAVLink protocol is used between the control programs and the UAVs in the physical platform and the simulation platform to control the movement of the UAVs through MAVROS. The MAVLink communication protocol is an information marshalling library composed only of header files designed for micro air vehicles.

[0042] In this embodiment, the simulation method of the simulated unmanned aerial vehicle (UAV) adopts the PX4 software-in-the-loop simulation method. The parameters of the simulated UAV are the same as those of the physical UAV. Both the mirrored UAV and the simulated UAV have a visual volume and a physical collision volume, and the external configurations of the mirrored UAV and the simulated UAV are the same, and both are equipped with a two-dimensional lidar. However, the mirrored UAV itself does not have any dynamic characteristics, that is, it has no dynamic model. In the simulation environment, the key point that the mirrored UAV is different from the virtual UAV is that its rotors do not rotate. The pose of the mirrored UAV is directly determined by the physical aircraft and mapped in real time, and this mapping relationship is realized by the pose follower by calling the pose setting topic of the simulation software.

[0043] In this embodiment, a computing platform is also carried on the physical platform, which is used to obtain the operation data of the mirrored UAV in real time and to realize the real-time control of the physical UAV. Specifically, the above computing platform can adopt Nvidia TX2, etc. Under the communication mechanism of the robot operating system, the on-board computing platform of the physical aircraft can obtain the data of the two-dimensional lidar carried by the mirrored UAV in real time through a topic, calculate the control amount of the UAV according to the pre-loaded perception and avoidance program, and control the movement of the physical UAV through MAVROS.

[0044] As Figure 1 shown, the multi-UAV perception and avoidance verification system combining virtual and real in this embodiment is divided into three parts: a physical platform, a simulation platform, and an interaction module (including a human-computer interaction interface). The deployment effect is as Figure 2 shown, where the physical platform mainly includes a positioning system, a physical UAV, and various communication and interconnection devices, etc. The simulation platform mainly includes an experimental scene, a simulated UAV, and a mirror image of the physical UAV, etc. The interaction module uniformly schedules all the aircraft.

[0045] In this embodiment, the physical platform also includes a binocular vision or motion capture system, which is used to obtain the UAV pose information in an indoor test environment. Specifically, a WIFI antenna can be installed on the physical UAV to realize the real-time control of the UAV by the ground station. When conducting experiments outdoors, a satellite navigation and positioning system can be used for positioning, and when conducting experiments indoors, a binocular vision or motion capture system can be used to obtain the UAV pose information.

[0046] The steps of the multi-UAV perception and avoidance verification method combining virtual and real in this embodiment include:

[0047] S1. Verification system setup: Build a physical platform with a physical drone and the required communication and interconnection devices, and build an experimental scenario, a simulation drone, and a mirrored drone in a simulation environment to build a simulation platform. The mirrored drone is a mirror of the physical drone. The simulation drone is built using a dynamic model to simulate the physical drone. The mirrored drone is configured as a mirror of the physical drone, and the pose of the mirrored drone is obtained by real-time mapping according to the pose of the physical drone. Establish a communication connection between the physical platform and the simulation platform;

[0048] S2. Perception and avoidance verification: Start the physical drone and the simulation environment, control the physical drone and the simulation drone according to a preset perception and avoidance verification task. The physical drone obtains the running data of the mirrored drone in real time, calculates the control quantity of the drone according to the preset perception and avoidance verification task, and the physical drone moves according to the calculated control quantity;

[0049] S3. Output of verification results: Obtain and output the running data of the physical drone and the simulation drone during the verification test.

[0050] In this embodiment, through the above steps, a preset perception and avoidance verification program is loaded in the computing platform of the physical drone. By obtaining the running data of the mirrored drone in real time and calculating the control quantity of the drone according to the perception and avoidance verification algorithm, the perception and avoidance verification based on the combination of virtual and real can be quickly realized. By adjusting the perception and avoidance verification program, the verification of various different perception and avoidance algorithms can be flexibly realized, and the advantages of the virtual-real experimental method can be fully exerted.

[0051] In step S1 of this embodiment, specifically configure the number and position of the corresponding mirrored drones according to the number and position of the physical drones. By following the number and position of the physical drones and using virtual mirrored drones to replace the simulation drones, the mirroring of the physical drones in the simulation environment is realized.

[0052] As Figure 3 shown, the detailed steps for the multi-drone perception and avoidance verification using the above method in a specific application embodiment of the present invention are:

[0053] Step (1) Build a software-in-the-loop simulation verification environment based on the PX4 open-source flight controller

[0054] For multi-drone perception and avoidance, design a software-in-the-loop simulation environment based on the PX4 open-source flight control software and the Gazebo platform. The simulation environment built in a specific application embodiment is as Figure 4 shown, where the parameters of the simulation drone and the physical drone are as consistent as possible. The simulation drone is equipped with a two-dimensional lidar to detect obstacles and other drones.

[0055] Step (2) Build a physical drone platform

[0056] Install an airborne computing platform (such as Nvidia TX2) on the physical UAV platform to achieve automated program control. The UAV is also equipped with a WIFI antenna to enable real-time control of the UAV by the ground station. A positioning system and a binocular vision or motion capture system are also installed on the physical UAV platform. Satellite navigation positioning is used when conducting experiments outdoors, and binocular vision or motion capture system is used to obtain the pose information of the UAV during indoor experiments.

[0057] Step (3) Build a communication framework based on the Robot Operating System

[0058] Design a unified UAV communication control program based on the Robot Operating System and MPI (Message Passing Interface).

[0059] Step (4) Load the human-machine interface and control program

[0060] The human-machine interface program is implemented in the form of a command line or a graphical user interface. The interface program has functions such as one-key start / stop and batch control, and can output necessary monitoring information in real time. The human-machine interface and control program that meet the above functions are recorded in the interaction module.

[0061] Step (5) Construct the virtual mapping of the physical UAV

[0062] According to the quantity and position of the physical UAVs, use virtual mirror UAVs to replace the simulated UAVs. The mirror UAVs have no kinematic and dynamic models, and their positions and postures are directly mapped from the poses of the physical UAVs. The shapes and sensors of the mirror UAVs are the same as those of the simulated UAVs.

[0063] Both types of UAVs (simulated UAVs and mirror UAVs) in the simulation environment have visual volumes and physical collision volumes. The simulated UAVs and mirror UAVs equipped with two-dimensional lidar can achieve mutual perception at the sensor layer through sensors. The physical UAVs can achieve mutual perception through the sensors on the corresponding mirror UAVs without additional collision detection sensors such as two-dimensional lidar. The environmental information obtained by the mirror UAVs through two-dimensional lidar is used as the basis for the decision-making of the physical UAVs.

[0064] Step (6) Verify the experiment execution

[0065] Start the simulation environment and the physical drone, and control the physical and simulated drones through the interaction module. During the verification process, the physical drone obtains the operation data of the mirrored drone through the sensors carried by the mirrored drone, calculates the control amount of the drone according to the preset perception and avoidance program, and the physical drone operates according to the calculated control amount. Record the data of the physical drone and the simulated drone and conduct analysis to complete the verification of the perception and avoidance algorithm.

[0066] The present invention combines the advantages of simulation experiments and physical experiments to construct a method for verifying multi-drone perception and avoidance that combines virtual and real scenarios. By constructing a mirror image of the physical drone in the simulation platform, it realizes mutual perception and interaction between machines based on sensors, can fully verify the algorithm reliability under airborne computing capabilities, drone dynamics, and various disturbances, enables large-scale scalable experiments in the simulation platform, and only requires a small number of physical drone platforms to verify the feasibility of the algorithm on the actual drone platform, which can greatly reduce the experimental cost.

[0067] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention shall fall within the scope of the protection of the technical solution of the present invention.

Claims

1. A multi-UAV perception and avoidance verification system combining virtual and physical elements, characterized in that, it includes: A physical platform, including a positioning system, physical UAVs, and required communication and interconnection devices; A simulation platform, used to construct experimental scenarios, simulated UAVs, and mirrored UAVs in a simulation environment. The simulated UAVs are constructed using dynamic models to simulate the physical UAVs. The mirrored UAVs are configured as mirrors of the physical UAVs. The poses of the mirrored UAVs are obtained by real-time mapping according to the poses of the physical UAVs. The mirrored UAVs follow the real-time poses of the physical UAVs through pose followers. The mirrored UAVs have the same appearance as the simulated UAVs. The rotors of the mirrored UAVs are configured not to rotate and the mirrored UAVs do not have dynamic characteristics. The number and positions of the mirrored UAVs are configured according to the number and positions of the physical UAVs; An interaction module, used to realize data interaction between the physical platform and the simulation platform; The physical platform controls the movement of the physical UAVs by obtaining the operation data of the mirrored UAVs in real time and calculating the control quantities of the UAVs according to the preset perception and avoidance verification tasks.

2. The multi-UAV perception and avoidance verification system combining virtual and physical elements according to claim 1, characterized in that, Sensors are installed on both the simulated UAVs and the mirrored UAVs. The mirrored UAVs and the simulated UAVs perceive each other through the sensors installed on them respectively. The physical platform obtains the operation data of the mirrored UAVs in real time through the sensors installed on the mirrored UAVs.

3. The multi-UAV perception and avoidance verification system combining virtual and physical elements according to claim 2, characterized in that, The sensor is a collision detection sensor.

4. The multi-UAV perception and avoidance verification system combining virtual and physical elements according to claim 1, characterized in that, The interaction module communicates with the physical platform and the simulation platform respectively based on ROS. The control programs in the physical platform and the simulation platform and the UAVs adopt the MAVLink protocol to control the movement of the UAVs through MAVROS.

5. The multi-UAV perception and avoidance verification system combining virtual and physical elements according to claim 1, characterized in that, The parameters of the simulated UAVs are the same as various parameters of the physical UAVs. Both the mirrored UAVs and the simulated UAVs have visual volumes and physical collision volumes.

6. The multi-UAV perception and avoidance verification system combining virtual and physical elements according to any one of claims 1 to 5, characterized in that, A computing platform is also installed on the physical platform, used to obtain the operation data of the mirrored UAVs in real time and realize the real-time control of the physical UAVs.

7. The multi-UAV perception and avoidance verification system combining virtual and physical elements according to any one of claims 1 to 5, characterized in that, The physical platform also includes a binocular vision or motion capture system, used to obtain the pose information of the UAVs in an indoor test environment.

8. A multi-UAV perception and avoidance verification method combining virtual and physical elements, characterized in that the steps include: S1. Verification system construction: construct a physical platform with a physical drone and the required communication and interconnection equipment, and construct an experimental scene, a simulated drone and a mirror drone in a simulation environment to construct a simulation platform, wherein the mirror drone is a mirror image of the physical drone, and the simulated drone is constructed using a dynamic model to simulate the physical drone, and the mirror drone is configured as a mirror image of the physical drone, and the posture of the mirror drone is obtained by real-time mapping according to the posture of the physical drone, and the mirror drone follows the real-time posture of the physical drone through a posture follower, and the mirror drone has the same appearance as the simulated drone, and the rotor of the mirror drone is configured to be non-rotatable and the mirror drone has no dynamic characteristics, and the number and position of the mirror drone are configured according to the number and position of the physical drone, and the physical platform establishes a communication connection with the simulation platform; S2. Perception avoidance verification: starting the physical drone and the simulation environment, controlling the physical drone and the simulation drone according to the preset perception avoidance verification task, the physical drone acquires the operation data of the mirror drone in real time, and calculates the control amount of the drone according to the preset perception avoidance verification task, and the physical drone moves according to the calculated control amount; S3. Verification result output: obtain the operation data output of the physical UAV and simulated UAV during the verification test.

9. The virtual-real multi-UAV perception and avoidance verification method according to claim 8, It is characterized in that In the step S1, the number and position of the corresponding mirror drones are configured according to the number and position of the physical drones.

Images