A hybrid scenario simulation system for heterogeneous multi-unmanned systems

Abstract

A heterogeneous multi-unmanned system mixed scene simulation system, which belongs to the field of robot simulation and semi-physical simulation. The present invention solves the problem that the existing simulation technology cannot simultaneously have the ability to model and simulate multiple types of unmanned platforms, run on multiple computers, and combine virtual and real. The digital simulation module of the present invention can include digital simulation models such as quad-rotor drones, fixed-wing drones, unmanned vehicles, bipedal robots, and quadrupedal robots to achieve simulation of different digital simulation models, thereby improving the scalability and versatility of the system. Moreover, the system of the present invention can perform multi-machine simulation on multiple computers, and therefore, is suitable for use in large-scale scenes. At the same time, the simulation technology of combining virtual and real in the present invention can well guarantee the effect of digital simulation. The method of the present invention can be applied to the field of heterogeneous multi-unmanned system mixed scene simulation.

Description

Technical Field

[0001] The present invention belongs to the field of robot simulation and semi-physical simulation, and specifically relates to a heterogeneous multi-unmanned system mixed scene simulation system. Background Art

[0002] Ground experiments and simulation verification of unmanned systems are important links in the development process of unmanned systems. An efficient and accurate simulation environment can greatly improve the efficiency of unmanned system development and reduce development costs.

[0003] The patent application with publication number CN112906245A proposes a multi-robot simulation platform for multiple tasks, and monitors the interaction data between the simulation platform and the simulation robot in real time. The patent application with publication number CN110262283A provides a multi-scenario visual robot simulation platform and method. In this method, the interactive information of the real operation of the robot is obtained by positioning and tracking device, and the AR image simulation device is linked to obtain the real robot information, and the simulation data of the same trajectory and different scenes are simulated, realizing the AR virtual and real combination simulation method. The patent application with publication number CN113888512A provides a simulation platform suitable for quickly calculating the area of ​​traversable areas, and proposes a method for evaluating the accuracy of the traversed area area, which improves the evaluation efficiency of the robot effect. The patent application with publication number CN104950695A proposes a scalable and universal unmanned aerial vehicle visual simulation platform, which overcomes the shortcomings of the existing unmanned aerial vehicle simulation platform, which has a single object, poor scalability and versatility. However, this method is only for vision-based unmanned aerial vehicle platforms, and is not applicable to other types of unmanned systems and multi-modal unmanned aerial vehicle systems. The paper "Implementation of UAV CoordinationBased on a Hierarchical Multi-UAV Simulation Platform" proposed a multi-UAV / unmanned vehicle simulation environment based on ROS1 and Gazebo. However, this environment only considers the simulation of multiple machines, and does not consider the combination of virtual and real. At the same time, the environment can only run on one computer or server, and cannot run across platforms, which limits the use of large-scale scenarios.

[0004] In summary, existing simulation technology cannot simultaneously model and simulate multiple types of unmanned platforms, run on multiple computers, and combine virtual and real capabilities, making it difficult to meet actual needs. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the existing simulation technology cannot simultaneously have the ability to model and simulate multiple types of unmanned platforms, run on multiple computers and combine virtual and real, and to propose a heterogeneous multi-unmanned system hybrid scene simulation system.

[0006] The technical solution adopted by the present invention to solve the above technical problems is:

[0007] A heterogeneous multi-unmanned system mixed scene simulation system, the system includes a ground command module, a simulation scene generation module, a digital simulation module, a semi-physical simulation module, a virtual communication and a DDS communication module;

[0008] The simulation scene generation module is used to generate various types of environment models;

[0009] The ground command module is used to adjust the model parameters and input limits of each unmanned system digital simulation model, and send tasks to the digital simulation module and the semi-physical simulation module. The ground command module communicates control information and environmental perception information with the digital simulation module and the semi-physical simulation module using virtual communication and DDS communication modules.

[0010] The digital simulation module includes the unmanned system digital simulation models loaded on each computer terminal, as well as the data storage and forwarding unit, decision unit and perception unit corresponding to each unmanned system digital simulation model, and the perception unit is configured with a sensor node;

[0011] The sensor node is used to sense the environmental data of the current environmental model and send the sensed environmental data to the corresponding data storage and forwarding unit;

[0012] The data storage and forwarding unit is used to store the perceived environmental data and the position, posture, and speed data of each unmanned system digital simulation model, and send the stored data to the corresponding decision-making unit;

[0013] The decision-making unit is used to make decisions and plans based on the perceived environmental data, the position, attitude, speed data of each unmanned system digital simulation model, and the tasks issued by the ground command module, and send the decision-making planning results to the unmanned system digital simulation model through the data storage and forwarding unit to control the unmanned system digital simulation model;

[0014] After the unmanned system digital simulation model is running, the output state perception data and underlying control data are sent to the data storage and forwarding unit;

[0015] The semi-physical simulation module includes a physical unmanned system corresponding to each unmanned system digital simulation model, and a perception planning unit, a data storage and forwarding unit corresponding to each physical unmanned system, and the physical unmanned system is provided with the same sensors as the digital simulation model;

[0016] The sensor is used to sense the environmental data corresponding to the physical unmanned system and send the sensed environmental data to the data storage and forwarding unit;

[0017] The data storage and forwarding unit is used to store the perceived environmental data and the position, attitude, and speed data of each physical unmanned system, and send the data to the perception planning unit;

[0018] The perception planning unit is used to make decisions and plans based on the perceived environmental data, the position, posture, speed data of each physical unmanned system, and the tasks issued by the ground command module, and send the decision-making planning results to the physical unmanned system through the data storage and forwarding unit to control the physical unmanned system;

[0019] After the physical unmanned system is running, the output status perception data and underlying control data are sent to the data storage and forwarding unit.

[0020] The beneficial effects of the present invention are:

[0021] The digital simulation module of the present invention may include digital simulation models such as quad-rotor drones, fixed-wing drones, unmanned vehicles, bipedal robots, and quadrupedal robots, so as to realize simulation of different digital simulation models, thereby improving the scalability and versatility of the system. Moreover, the system of the present invention can perform multi-machine simulation on multiple computers, and therefore, is suitable for use in large-scale scenarios. At the same time, the present invention uses the simulation technology that combines the virtual and the real to well ensure the effect of digital simulation, and solves the problem that the existing simulation system does not have the ability to simultaneously model and simulate multiple types of unmanned platforms, run on multiple computers, and combine the virtual and the real. The simulation system of the present invention is compatible with the functions of being able to model and simulate multiple types of unmanned platforms, being able to run on multiple computers, and being able to combine the virtual and the real. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is an architecture diagram of a heterogeneous multi-unmanned system hybrid scenario simulation system of the present invention. DETAILED DESCRIPTION

[0023] Specific implementation method 1: Combination Figure 1 This embodiment is described. A heterogeneous multi-unmanned system hybrid scene simulation system described in this embodiment includes a ground command module, a simulation scene generation module, a digital simulation module, a semi-physical simulation module, a virtual communication and DDS communication module;

[0024] The simulation scene generation module is used to generate various types of environment models;

[0025] The ground command module is used to adjust the model parameters and input limits of each unmanned system digital simulation model, and send tasks to the digital simulation module and the semi-physical simulation module. The ground command module communicates control information and environmental perception information with the digital simulation module and the semi-physical simulation module using virtual communication and DDS communication modules.

[0026] The digital simulation module includes the unmanned system digital simulation models loaded on each computer terminal, as well as the data storage and forwarding unit, decision unit and perception unit corresponding to each unmanned system digital simulation model, and the perception unit is configured with a sensor node;

[0027] That is, for each unmanned system digital simulation model, there is a corresponding data storage and forwarding unit, decision-making unit and perception unit;

[0028] The sensor node is used to sense the environmental data of the current environmental model and send the sensed environmental data to the corresponding data storage and forwarding unit;

[0029] The data storage and forwarding unit is used to store the perceived environmental data and the position, posture, and speed data of each unmanned system digital simulation model, and send the stored data to the corresponding decision-making unit;

[0030] The decision-making unit is used to make decisions and plans based on the perceived environmental data, the position, attitude, speed data of each unmanned system digital simulation model, and the tasks issued by the ground command module, and send the decision-making planning results to the unmanned system digital simulation model through the data storage and forwarding unit to control the unmanned system digital simulation model;

[0031] After the unmanned system digital simulation model is running, the output state perception data and underlying control data are sent to the data storage and forwarding unit;

[0032] State perception data includes images, radar data, ranging data, IMU data, etc.

[0033] The semi-physical simulation module includes a physical unmanned system corresponding to each digital simulation model of the unmanned system (through data interaction between the digital simulation module and the semi-physical simulation module, a physical unmanned system corresponding to each digital simulation model can be set in the semi-physical simulation module), and a perception planning unit, a data storage and forwarding unit corresponding to each physical unmanned system, and the physical unmanned system is provided with the same sensor as the digital simulation model;

[0034] The sensor is used to sense the environmental data corresponding to the physical unmanned system (according to the correspondence between the physical unmanned system and the digital simulation model of the unmanned system, the digital simulation model corresponding to the physical unmanned system is obtained, and then the environmental data corresponding to the digital simulation model is used as the environmental data of the physical unmanned system and sensed by the sensor of the physical unmanned system. In this way, the physical unmanned system is directly planned according to the digital simulation environmental data, which on the one hand verifies the simulation effect of the digital simulation model, and on the other hand does not need to establish a physical simulation environment, which is easy to operate), and sends the sensed environmental data to the data storage and forwarding unit (here, the data storage and forwarding unit of the physical unmanned system);

[0035] The data storage and forwarding unit (here, the data storage and forwarding unit of the physical unmanned system) is used to store the perceived environmental data and the position, posture, and speed data of each physical unmanned system, and send the data to the perception planning unit;

[0036] The perception planning unit is used to make decisions and plans based on the perceived environmental data, the position, attitude, speed data of each physical unmanned system, and the tasks issued by the ground command module, and send the decision-making planning results to the physical unmanned system through the data storage and forwarding unit (here, the data storage and forwarding unit of the physical unmanned system) to control the physical unmanned system;

[0037] After the physical unmanned system is running, the output state perception data and underlying control data are sent to the data storage and forwarding unit (here is the data storage and forwarding unit of the physical unmanned system).

[0038] The ground command module of the present invention is mainly used for tasks, monitoring, navigation data inspection and other functions. It has the functions of upper-level planning and dynamic node adjustment. It can dynamically change the number of nodes in the system and display the system status and sensor perception data in real time. The specific functional design includes the following parts:

[0039] Mission issuance: Assigning unmanned system groups to reach the set target in a specified formation;

[0040] Data verification: Display the position, speed, posture and other navigation data information of the unmanned vehicle, drone and legged robot in the system in real time, and return it at the set frequency;

[0041] Upper-level planning: Plans paths or trajectories based on the virtual environment, and has the capabilities of unmanned systems such as unmanned vehicles, drones, and legged robots.

[0042] Dynamically adjust system scale: Dynamically add and delete unmanned system node information, including physical unmanned systems and digital simulation model systems, and set unmanned system nodes to be real or virtual. Adjust model parameters, input restrictions, and sensor configurations of various unmanned system nodes.

[0043] Sensor and map information: obtain perception sensor information, draw IMU and other sensor data information curves, and display maps in real time.

[0044] Save data: Save the above data and have the functions of reading and playing back.

[0045] Virtual communication and DDS communication module: To reduce communication congestion and disconnection problems, the communication adopts the DDS self-organizing network mode. The software layers use the DDS architecture based on the second-generation robot operating system for communication, and the hardware uses a self-organizing communication module to complete the communication. In this architecture, information interaction can be achieved as long as there is a communication path. In order to be more in line with the actual situation, it is necessary to set up a virtual communication architecture, that is, to build a virtual communication topology diagram between unmanned system nodes, so that the simulation platform is more in line with the operation of the actual physical system.

[0046] Simulation scene generation module: The present invention supports mixed simulation of multiple scenes. In the digital simulation environment, the present invention can simultaneously set up large-scale underground environment digital models (underground tunnels, underground mines), outdoor woods, mountainous areas, urban digital models, and digital models of multi-layer indoor scenes. The scenes can be mixed and superimposed to achieve seamless switching, and the environmental granularity is higher than 1mm. In order to test different lighting environments, a lighting editor is used. The present invention realizes the addition, editing and adjustment of light properties, such as position, color, intensity, etc. For scenes that combine virtuality and reality, the present invention maps the environmental map information (position, shape, etc.) perceived by the physical nodes into the digital simulation environment. At the same time, when the physical nodes are running, they will also perceive and avoid obstacles according to their positions in the digital simulation environment. Therefore, in the present invention, the digital simulation model of the scene changes continuously over time, and a mixed linkage between virtuality and reality can be achieved.

[0047] Specific implementation method 2: The difference between this implementation method and specific implementation method 1 is that the digital simulation models of the unmanned systems loaded on each computer terminal include a digital simulation model of a quadrotor drone, a digital simulation model of a fixed-wing drone, a digital simulation model of an unmanned vehicle, a digital simulation model of a bipedal robot and a digital simulation model of a quadrupedal robot.

[0048] The other steps and parameters are the same as those in the first embodiment.

[0049] The models that can be simulated by the present invention include but are not limited to the above-mentioned models. Moreover, in the actual simulation process, according to specific simulation requirements, the digital simulation model in the digital simulation module can be a combination of one or several models of the present embodiment.

[0050] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the digital simulation model of the unmanned system is constructed based on the dynamics and kinematics model of the unmanned system, the rigid body components of different parts, the physical properties of each component and the connection relationship.

[0051] The other steps and parameters are the same as those in the first or second embodiment.

[0052] The present invention adopts URDF file form to describe and design a digital simulation model of a high-precision unmanned system motion platform and sensor, and provides a corresponding control interface. According to the dynamics and kinematics model of the unmanned system platform, the physical properties and connection relationship of each component, and the rigid body components of different parts, an accurate unmanned system digital simulation model is constructed. Correspondingly, according to the actual physical properties of the sensor and the position of the system, the digital simulation model of different sensors is constructed. The control interface of the unmanned system and the sensor is implemented in a plug-in manner to interact with external data. Specifically, the "collision" tag is used to describe the specified collision detection properties of each rigid structure component, the "joint" and "link" tags are used to describe the connection relationship of each component, the "plugin" tag is used to build a control plug-in, and the "visual" tag is used to describe the specified visual properties of each rigid structure component, so that the constructed digital simulation model can be displayed in gazebo. In order to distinguish the same type of platforms in the unmanned system, the "prefix" parameter is designed as the prefix of each component and sensor in different nodes, so as to ensure the uniqueness of the connection relationship between components of different nodes. The unmanned system nodes included in the present invention include but are not limited to quadrotor drones, fixed-wing drones, unmanned vehicles, quadruped robots, bipedal robots and hexapod robots, etc.

[0053] Specific implementation method four: This implementation method is different from any one of specific implementation methods one to three in that: the sensor node includes an inertial sensor, a visual sensor, an RGB camera, a depth camera, a single-line laser radar and a multi-line laser radar.

[0054] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0055] The sensor node types of the present invention include but are not limited to inertial sensors, visual sensors, RGB cameras, depth cameras, single-line laser radars, and multi-line laser radars.

[0056] Specific implementation five: This implementation differs from any one of specific implementations one to four in that: the ground command module is also used to obtain sensor-perceived environmental data, the position, attitude, speed data of the digital simulation model of the unmanned system, and the position, attitude, speed data of the physical unmanned system from the data storage and forwarding unit of the digital simulation module and the data storage and forwarding unit of the semi-physical simulation module, and to monitor and analyze the acquired data in real time.

[0057] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0058] After the monitoring and analysis results are obtained by the method of this embodiment, the model parameters and input limits of each digital environment model are adjusted by using a preset algorithm and taking into account the external interference of the physical unmanned system, and then the new task is sent to the digital simulation module and the semi-physical simulation module, and the digital simulation module and the semi-physical simulation module are re-simulated. Repeat the above process, and then obtain and analyze data from the data storage unit of the digital simulation module and the semi-physical simulation module until the digital simulation model is consistent with the experimental reproduction effect of the physical unmanned system.

[0059] Moreover, the present invention can also perform fault detection on physical unmanned systems, specifically: comparing the position, posture, and speed data of the physical unmanned system fed back by the semi-physical simulation module with the data predicted by the prediction method. When the deviation between the feedback data and the predicted data is large, or the deviation between the feedback data and the predicted data tends to increase, it can be determined that the unmanned system is faulty.

[0060] Specific implementation method six: This implementation method is different from any one of specific implementation methods one to five in that: the model parameters include the mass, moment of inertia and motor pulling force of the unmanned system, and the input limit is the control input constraint.

[0061] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0062] Specific implementation method seven: This implementation method is different from any one of specific implementation methods one to six in that: the simulation scene generation module is also used for switching between various digital environment models.

[0063] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0064] By switching the environment, simulation in different environments can be achieved.

[0065] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A heterogeneous multi-unmanned system mixed scene simulation system, characterized in that: The system includes a ground command module, a simulation scene generation module, a digital simulation module, a semi-physical simulation module, a virtual communication and a DDS communication module; The simulation scene generation module is used to generate various types of environment models; The ground command module is used to adjust the model parameters and input limits of each unmanned system digital simulation model, and send tasks to the digital simulation module and the semi-physical simulation module. The ground command module communicates control information and environmental perception information with the digital simulation module and the semi-physical simulation module using virtual communication and DDS communication modules. The digital simulation module includes the unmanned system digital simulation models loaded on each computer terminal, as well as the data storage and forwarding unit, decision unit and perception unit corresponding to each unmanned system digital simulation model, and the perception unit is configured with a sensor node; The sensor node is used to sense the environmental data of the current environmental model and send the sensed environmental data to the corresponding data storage and forwarding unit; The data storage and forwarding unit is used to store the perceived environmental data and the position, posture, and speed data of each unmanned system digital simulation model, and send the stored data to the corresponding decision-making unit; The decision-making unit is used to make decisions and plans based on the perceived environmental data, the position, attitude, speed data of each unmanned system digital simulation model, and the tasks issued by the ground command module, and send the decision-making planning results to the unmanned system digital simulation model through the data storage and forwarding unit to control the unmanned system digital simulation model; After the unmanned system digital simulation model is running, the output state perception data and underlying control data are sent to the data storage and forwarding unit; The semi-physical simulation module includes a physical unmanned system corresponding to each unmanned system digital simulation model, and a perception planning unit, a data storage and forwarding unit corresponding to each physical unmanned system, and the physical unmanned system is provided with the same sensors as the digital simulation model; The sensor is used to sense the environmental data corresponding to the physical unmanned system and send the sensed environmental data to the data storage and forwarding unit; The data storage and forwarding unit is used to store the perceived environmental data and the position, attitude, and speed data of each physical unmanned system, and send the data to the perception planning unit; The perception planning unit is used to make decisions and plans based on the perceived environmental data, the position, posture, speed data of each physical unmanned system, and the tasks issued by the ground command module, and send the decision-making planning results to the physical unmanned system through the data storage and forwarding unit to control the physical unmanned system; After the physical unmanned system is in operation, the output state perception data and underlying control data are sent to the data storage and forwarding unit; The ground command and control module is also used to obtain environmental data perceived by sensors, position, attitude, and speed data of the digital simulation model of the unmanned system, and position, attitude, and speed data of the physical unmanned system from the data storage and forwarding unit of the digital simulation module and the data storage and forwarding unit of the semi-physical simulation module, and to monitor and analyze the acquired data in real time.

2. A heterogeneous multi-unmanned system mixed scene simulation system according to claim 1, characterized in that: The digital simulation models of the unmanned systems loaded on each computer terminal include a digital simulation model of a quad-rotor unmanned aerial vehicle, a digital simulation model of a fixed-wing unmanned aerial vehicle, a digital simulation model of an unmanned vehicle, a digital simulation model of a bipedal robot and a digital simulation model of a quadrupedal robot.

3. A heterogeneous multi-unmanned system mixed scene simulation system according to claim 2, characterized in that: The unmanned system digital simulation model is constructed based on the unmanned system's dynamics and kinematics models, the rigid body components of different parts, and the physical properties and connection relationships of each component.

4. A heterogeneous multi-unmanned system mixed scene simulation system according to claim 3, characterized in that: The sensor nodes include inertial sensors, visual sensors, RGB cameras, depth cameras, single-line laser radars and multi-line laser radars.

5. A heterogeneous multi-unmanned system mixed scene simulation system according to claim 4, characterized in that: The model parameters include the mass, moment of inertia and motor pulling force of the unmanned system, and the input limit is the control input constraint.

6. A heterogeneous multi-unmanned system mixed scene simulation system according to claim 5, characterized in that: The simulation scene generation module is also used for switching between various digital environment models.

Images