Robot simulation system and method with multi-engine cooperation

Abstract

The present invention provides a multi-engine collaborative robot simulation method and system according to claim 1, the system comprising: developing a simulation management system based on the UE5 three-dimensional simulation engine to realize effective management and display of various information and data during the simulation process; developing a high-precision map, simulating sensors such as lidar to realize scene data acquisition and interaction, and optimizing the authenticity and credibility of the simulation environment for actual application simulation; based on the Vortex dynamics modeling software, through the VHL_Interface data interaction method, performing robot and scene interaction dynamics modeling to obtain robot simulation results; based on the FastDDS communication middleware, establishing data communication between UE5 and ROS2, being compatible with the ROS2 message format, and performing simulation tests of the controller hardware in the loop. The present invention solves the technical problems of being difficult to balance diverse control algorithms and fine dynamics models, difficult data interaction in the simulation system, high complexity and cost of the simulation system, and low reliability.

Description

Technical Field

[0001] The present invention relates to the field of robot simulation, and particularly to a robot simulation system and method with multi-engine collaboration. Background Art

[0002] Robot simulation systems are generated to meet the research, development, and testing requirements of robots with the support of computer technology and virtual simulation technology. With the rapid development and wide application of robot technology, there is a need for an efficient, safe, and cost-effective method to verify functions such as robot perception, planning, and control. Robot simulation systems provide a virtual environment that enables researchers and engineers to test and optimize robots in simulated scenarios, reducing experimental costs and risks, and improving development efficiency and technological innovation. Through robot simulation systems, various different environments, tasks, and robot types can be simulated, accelerating the development and popularization of robot technology, and promoting the integration and application of robots with human society.

[0003] Currently, there are many simulation tools for different system levels. For example, the tool Simulink that focuses on system modeling has advantages in control system design, dynamic system modeling, and simulation, and is particularly suitable for the design, analysis, and optimization of complex systems in engineering fields such as electrical, mechanical, aerospace, and automotive. However, its disadvantage is that the 3D rendering ability is relatively simple and cannot provide a realistic and intuitive simulation effect. In addition, there are also simulation platforms such as Gazebo. Gazebo is more focused on robot modeling, simulation, and control, and can also be used for sensor simulation, with an improvement in 3D rendering effects. However, it is difficult to obtain high-precision images of the sensors and 3D scenes simulated by Gazebo.

[0004] To meet more diverse simulation requirements, some robot simulation systems for joint simulation on multiple platforms have emerged. For example, in the existing invention patent application document "A Simulation Test and Verification Platform for UAV Multi-Sensor Fusion" with publication number CN116185074A, this existing simulation system uses the Unreal Engine to jointly simulate UAVs with algorithm modules on the ROS side. A three-dimensional scene and simulated sensors are deployed in the Unreal Engine. The simulated data is sent to the ROS side through network communication. The model inference module and multi-sensor fusion module on the ROS side use the simulated data for calculations, and the output motion instructions are sent to the UAV in the Unreal Engine. This simulation system leverages the excellent three-dimensional rendering capabilities of the Unreal Engine and the powerful function library of ROS to improve the authenticity of the simulation scene, provide higher-precision camera image data, and can give full play to the roles of various algorithms and function packages in ROS. However, this simulation system is more suitable for UAV simulation because it cannot provide a high-precision dynamics model, so it is difficult to be used for the simulation of ground robots.

[0005] To combine a high-precision dynamics model with the powerful three-dimensional rendering capabilities of a three-dimensional engine, in the existing invention patent application document "A Co-Simulation Method of Modelica Platform and UE4 Based on Opendds" with publication number CN115422723A, the robot is simulated and modeled in OpenModelica, and a three-dimensional scene and sensors such as cameras are simulated in the Unreal Engine. The Opendds publish / subscribe mode is used to complete data interaction between the two platforms. This simulation system can provide a more refined and customizable dynamics model and a realistic three-dimensional scene, but it is also difficult to provide a refined simulation function for road surface robots because the model information of the three-dimensional scene in the Unreal Engine cannot be obtained by Modelica. When Modelica performs dynamics calculations, it is difficult to consider the impact of road conditions on the robot model and can only rely on the model collision of the Unreal Engine itself.

[0006] Currently, three-dimensional simulation technology faces multiple challenges in the field of ground robots: it is difficult to simultaneously meet the realistic three-dimensional rendering effect, the performance of complex control logic, and the refined dynamics model for interaction with the scene, especially when simulating ground robots in a complex ground environment. At the same time, the processing, transmission delay of sensor data, and the complexity of multi-terminal transmission together lead to data interaction problems in the simulation system. In addition, the need for hardware-in-the-loop increases the complexity and cost of the simulation system. These problems affect the development of ground robot simulation technology, and innovative solutions are urgently needed to improve the overall performance and reliability of the simulation system.

[0007] In summary, the prior art has technical problems such as difficulty in balancing diverse control algorithms and fine dynamic models, difficult data interaction in the simulation system, complexity of the simulation system, high cost, and low reliability. Summary of the Invention

[0008] The technical problem to be solved by the present invention is: how to solve the technical problems in the prior art, such as difficulty in balancing diverse control algorithms and fine dynamic models, difficult data interaction in the simulation system, complexity of the simulation system, high cost, and low reliability.

[0009] The present invention solves the above technical problems by adopting the following technical solutions: A multi-engine collaborative robot simulation system includes: a dynamics module, a 3D simulation module, and a general controller module;

[0010] The dynamics module is used to implement the dynamics module using Vortex, take the dynamics module as a plugin, and import it into the 3D simulation module to perform mechanical interaction operations with the static model in the preset 3D simulation environment;

[0011] The 3D simulation module is used to create patrol tasks and obstacle avoidance tasks using the preset graphical programming interface, jump to the corresponding levels, load the Vortex dynamics model, send task instructions corresponding to the patrol tasks and obstacle avoidance tasks, and the 3D simulation module is connected to the dynamics module;

[0012] The general controller module is implemented in ROS, used to receive task start commands by subscribing, start the perception algorithm program and the planning algorithm program in sequence, obtain sensor information from the 3D simulation environment and the Vortex dynamics model in the UE5 3D simulation engine, set the driving target point of the wheeled tracked robot, and publish it through the Topic using the FastDDS protocol; subscribe to the sensor and the form target point according to the Topic of the preset protocol, use the perception algorithm program and the perception planning program to calculate and publish the desired linear velocity and desired angular velocity of the wheeled tracked robot, and accordingly adjust the pose of the wheeled tracked robot in the 3D simulation environment. The general controller module is connected to the 3D simulation module.

[0013] The present invention provides a robot hardware-in-the-loop motion control simulation system that combines the ROS2 control system based on the Fastdds communication middleware, the UE5 3D simulation environment, and the Vortex dynamics modeling. Based on the UE5 3D simulation engine, a simulation management system is developed to effectively manage and display various information and data during the simulation process, thereby improving the overall efficiency and visualization effect of the simulation system.

[0014] In a more specific technical solution, a three-dimensional simulation module is built using the UE5 three-dimensional simulation engine. The three-dimensional simulation module and the general controller module in ROS2 establish data communication between UE5 and ROS2 through a pre-set communication middleware, perform interactive operations on environmental information, model information, and control information, be compatible with the message format of ROS2, and conduct simulation tests of the controller hardware in the loop.

[0015] The ROS adopted by the present invention has powerful algorithms and function libraries, is convenient to use the DDS communication protocol, and can be combined with physical robots in the later stage. The present invention is based on the FastDDS communication middleware, establishes data communication between UE5 and ROS2, is compatible with the ROS2 message format, realizes the simulation test of the controller hardware in the loop, and provides reliable support and verification for the development of the actual control system. These technical combinations provide an important technical foundation and support for the comprehensive development of the simulation system, and will show great potential and value in various actual application scenarios.

[0016] In a more specific technical solution, the three-dimensional simulation module includes: an interactive display component, a sensor component, a three-dimensional model component, a dynamics interface component, and a communication component;

[0017] The interactive display component is used to perform user interactive operations;

[0018] The sensor component is used to load component data from a pre-set sensor component library to simulate the sensor function during the operation of the robot. The sensor component is connected to the interactive display component;

[0019] The three-dimensional model component is used to import a three-dimensional model library from the outside and set the three-dimensional model of the three-dimensional simulation system. The three-dimensional model component is connected to the sensor component;

[0020] The dynamics interface component is used to perform plug-in data exchange between the three-dimensional simulation module and the dynamics module. During the external dynamics modeling library expansion process, interface addition settings are performed in the pre-set dynamics modeling software, and the dynamics interface component is used for interface expansion. The dynamics interface component is connected to the three-dimensional model component;

[0021] The communication component is used to perform information interaction operations between the three-dimensional simulation module and the external modules of the wheeled tracked robot. Among them, the information sent by the communication component also includes: multi-dimensional sensor information, simulation instructions, and received control information. The communication component is connected to the dynamics interface component, the three-dimensional model component, and the dynamics interface component.

[0022] In a more specific technical solution, the interactive display component further includes:

[0023] A graphical programming component is used to obtain a 3D selection model, a sensor selection type, and corresponding algorithms from 3D model components and sensor components; through task selection, dragging robot components, and operating algorithm modules, a simulation task and task instructions are automatically generated and sent to the sensor components, 3D model components, and dynamics interface components.

[0024] A data display component is used to generate and display robot control information received from the communication component according to the task instructions. The data display component is connected to the graphical programming component.

[0025] In a more specific technical solution, the sensor components include:

[0026] A lidar sensor component is installed at the lidar installation position of the wheeled tracked robot to simulate a lidar in a preset 3D simulation environment, generate 3D point cloud data, encapsulate the 3D point cloud data into a preset protocol format, and send it to the communication component.

[0027] An IMU sensor component obtains the linear acceleration and angular velocity of the wheeled tracked robot body from the dynamics module, encapsulates the linear acceleration and angular velocity into a preset protocol format, and sends it to the communication component.

[0028] A magnetometer sensor component is used to add the body position of the wheeled tracked robot to the 3D model component to process and obtain the body angle quaternion.

[0029] The Fastdds communication middleware adopted by the present invention supports various platforms, is convenient for transmitting large amounts of data, has a fast transmission speed and a stable transmission effect. When these four components are jointly simulated, good 3D rendering effects can be obtained, and multiple control algorithms and fine dynamics models can be taken into account. The simulation system framework designed by the present invention has good flexibility and scalability and can be used for simulation tasks of various types of robots in various scenarios such as sea, land, air, and space.

[0030] In a more specific technical solution, the 3D model component further includes:

[0031] A static object model component includes: ground Figure 3 3D models, 3D scene models, and 3D models of static objects in the scene, as well as natural special effects; among them, the natural special effects further include: level lighting data and level wind data.

[0032] A dynamic object model component includes: a wheeled tracked robot model, which performs data exchange operations with the dynamics interface component.

[0033] The present invention develops a high-precision map, simulates sensors such as lidar to achieve scene data collection and interaction, enhances the authenticity and credibility of the simulation environment, and provides a more accurate basis for the simulation of practical applications. The framework of the simulation system of the present invention is also applicable to the simulation requirements of other types of robots such as unmanned aerial vehicles and robotic arms, and has good flexibility and compatibility.

[0034] In a more specific technical solution, the general controller module includes:

[0035] An algorithm control module for subscribing to task instructions in the UE5 three-dimensional simulation engine, where the task instructions include: a task start instruction and a task end instruction;

[0036] Use the dynamics modeling software Vortex to build a dynamics module, use the dynamics modeling software Vortex as a plug-in of the UE5 three-dimensional simulation engine, and through the VHL_Interface data interaction method, perform dynamics modeling of the interaction between the robot and the scene, construct the Vortex dynamics model of the wheeled tracked robot, and import the Vortex dynamics model into the UE5 three-dimensional simulation engine in a preset import method, and use the blueprint of the UE5 three-dimensional simulation engine to call the input and output interfaces in the Vortex dynamics model to realize model data exchange; among them, the preset import method includes:.Mechanism resource method.

[0037] Based on the Vortex dynamics modeling software, the present invention realizes high-precision dynamics modeling of the interaction between the robot and the scene through the VHL_Interface data interaction method, and obtains more accurate and realistic simulation results. Vortex provides fine and highly customizable dynamics modeling.

[0038] The present invention uses the rich algorithms and function libraries in ROS2, and builds a real and high-definition three-dimensional simulation scene in UE5; the method of integrating Vortex into UE5 allows the high-precision dynamics model to interact well with the three-dimensional simulation scene, and simulates the road surface undulation effect during the driving process of the wheeled tracked robot.

[0039] In a more specific technical solution, when receiving the task start instruction, the perception algorithm program and the planning algorithm program in the algorithm module are started in sequence to subscribe to radar point cloud data and IMU and magnetometer data through the Topic of the preset protocol; when the algorithm control module receives the task end instruction, the perception algorithm program and the planning algorithm program are closed.

[0040] In a more specific technical solution, by using the UE5 three-dimensional simulation engine, the desired linear velocity and the desired angular velocity are received through Topic subscription. Through the input and output interfaces in the Vortex dynamics model, the desired linear velocity and the desired angular velocity are transmitted to the dynamics model, and the dynamics model calculates the desired linear velocity and the desired angular velocity to adjust the pose of the wheel-track robot in the three-dimensional simulation environment.

[0041] The Vortex dynamics model and the UE5 three-dimensional simulation engine adopted in the present invention jointly provide high-precision sensor simulation data for the ROS2 algorithm, enabling the algorithms in ROS2 to play a role similar to that of a physical robot. UE5 provides powerful three-dimensional rendering effects and is highly compatible with Vortex plugins.

[0042] In a more specific technical solution, the robot simulation method with multi-engine collaboration includes:

[0043] S1. Use Vortex to implement a dynamics module, import the dynamics module as a plugin into the three-dimensional simulation module to perform mechanical interaction operations with the static models in the preset three-dimensional simulation environment;

[0044] S2. Use the preset graphical programming interface to create a patrol task and an obstacle avoidance task, jump to the corresponding levels, load the Vortex dynamics model, and send task instructions corresponding to the patrol task and the obstacle avoidance task;

[0045] S3. Receive a task start command through subscription, start the perception algorithm program and the planning algorithm program in sequence. In the UE5 three-dimensional simulation engine, obtain sensor information from the three-dimensional simulation environment and the Vortex dynamics model, set the driving target point of the wheel-track robot, and publish it through the Topic using the FastDDS protocol; subscribe to the sensors and the form target point according to the Topic of the preset protocol, use the perception algorithm program and the perception planning program to calculate and publish the desired linear velocity and the desired angular velocity of the wheel-track robot, and accordingly adjust the pose of the wheel-track robot in the three-dimensional simulation environment.

[0046] The present invention has the following advantages compared with the prior art:

[0047] The present invention provides a robot hardware-in-the-loop motion control simulation system that combines a ROS2 control system based on the FastDDS communication middleware, a UE5 three-dimensional simulation environment, and Vortex dynamics modeling. Based on the UE5 three-dimensional simulation engine, a simulation management system is developed to effectively manage and display various types of information and data during the simulation process, thereby improving the overall efficiency and visualization effect of the simulation system. A high-precision map is developed to simulate sensors such as lidar to achieve scene data collection and interaction, enhancing the authenticity and credibility of the simulation environment and providing a more accurate basis for the simulation of practical applications. The framework of the simulation system of the present invention is also applicable to the simulation requirements of other types of robots such as unmanned aerial vehicles and robotic arms, with good flexibility and compatibility.

[0048] ROS adopted by the present invention has powerful algorithms and function libraries, which facilitates the use of the DDS communication protocol and can be combined with physical robots in the later stage. Based on the FastDDS communication middleware, the present invention establishes data communication between UE5 and ROS2, is compatible with the ROS2 message format, and realizes the simulation test of the controller hardware-in-the-loop, providing reliable support and verification for the development of the actual control system. These technical combinations provide an important technical foundation and support for the comprehensive development of the simulation system, and will demonstrate great potential and value in various actual application scenarios.

[0049] The FastDDS communication middleware adopted by the present invention supports various platforms, facilitates the transmission of large amounts of data, and has a fast transmission speed and stable transmission effect. When these four components are jointly simulated, good three-dimensional rendering effects can be obtained, and diverse control algorithms and fine-grained dynamics models can be taken into account. The designed simulation system framework of the present invention has good flexibility and scalability and can be used for simulation tasks of various types of robots in multiple scenarios on land, sea, air, and space.

[0050] Based on the Vortex dynamics modeling software, the present invention realizes high-precision robot and scene interaction dynamics modeling through the VHL_Interface data interaction method, obtaining more accurate and realistic simulation results. Vortex provides fine-grained and highly customizable dynamics modeling.

[0051] While using the rich algorithms and function libraries in ROS2, the present invention builds a real and high-definition three-dimensional simulation scene in UE5; the integration of Vortex into UE5 allows the high-precision dynamics model to interact well with the three-dimensional simulation scene, simulating the road undulation effect during the driving process of the wheeled tracked robot.

[0052] The Vortex dynamics model and the UE5 three-dimensional simulation engine adopted by the present invention jointly provide high-precision sensor simulation data for the ROS2 algorithm, enabling the algorithms in ROS2 to play a role similar to that of a physical robot. UE5 provides powerful three-dimensional rendering effects and is highly compatible with Vortex plugins.

[0053] The present invention solves the technical problems existing in the prior art, such as the difficulty in balancing multiple control algorithms and a fine dynamics model, difficult data interaction in the simulation system, the complexity of the simulation system, high cost, and low reliability. Brief Description of the Drawings

[0054] Figure 1 Schematic diagram of the module connection of the multi-engine collaborative robot simulation system in Embodiment 1 of the present invention;

[0055] Figure 2 Schematic diagram of the information flow between three-dimensional simulation modules in Embodiment 1 of the present invention;

[0056] Figure 3 Schematic diagram of the collaborative data flow processing of each module when the wheeled tracked robot performs patrol / obstacle avoidance tasks in Embodiment 2 of the present invention;

[0057] Figure 4 Schematic diagram of the specific steps of the collaboration of each module when the wheeled tracked robot performs patrol / obstacle avoidance tasks in Embodiment 2 of the present invention;

[0058] Figure 5 Schematic diagram of the data flow processing of the components of the wheeled tracked robot patrol and obstacle avoidance task in Embodiment 2 of the present invention;

[0059] Figure 6 Schematic diagram of the specific steps of the UE5 process of the wheeled tracked robot patrol and obstacle avoidance task in Embodiment 2 of the present invention;

[0060] Figure 7 Schematic diagram of the patrol scene of the wheeled tracked robot in Embodiment 2 of the present invention;

[0061] Figure 8 Schematic diagram of the obstacle avoidance scene of the wheeled tracked robot in Embodiment 2 of the present invention;

[0062] Figure 9 Schematic diagram of the specific steps of the ROS process of the wheeled tracked robot patrol and obstacle avoidance task in Embodiment 2 of the present invention;

[0063] Figure 10 Flow chart of the Vortex multi-body dynamics modeling in Embodiment 2 of the present invention;

[0064] Figure 11 Rendering effect diagram of the lidar sensor in Embodiment 2 of the present invention;

[0065] Figure 12 This is the 3D model diagram of the 2-wheel tracked robot in Embodiment 2 of the present invention. Detailed implementation manners

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0067] Embodiment 1

[0068] As Figure 1 shown, the basic modules of the multi-engine collaborative robot simulation system include: a dynamics module 1, a 3D simulation module 2, and a general controller module 3.

[0069] In this embodiment, the dynamics module 1 includes but is not limited to: a tracked vehicle dynamics modeling component 11 and other dynamics modeling components 12; for example, it can be implemented using Vortex, imported into the 3D simulation module 2 as a plugin, and perform mechanical interactions with static models in the 3D simulation environment.

[0070] The 3D simulation module 2 is built on the UE5 platform. The 3D simulation module 2 is connected to the general controller module 3 in ROS2, and interacts with environmental information, model information, and control information through a communication middleware. For the specific interaction principle, refer to Figure 1 .

[0071] As Figure 2 shown, in this embodiment, the 3D simulation module 2 includes but is not limited to: an interaction display component 21, a sensor component 22, a 3D model component 23, a dynamics interface component 24, and a communication component 25.

[0072] In this embodiment, the interaction display component 21 performs user interactions; in this embodiment, the interaction display component 21 includes but is not limited to the following sub-components:

[0073] A graphical programming component 211, which is used to obtain selectable 3D models, sensor types, and corresponding algorithms from the 3D model component 23 and the sensor component 22; automatically generate a simulation task through task selection and dragging of robot components and algorithm modules, and send corresponding task instructions to other components.

[0074] A data display component 212, which is used to display the data received through the communication component 25, including but not limited to: robot control information.

[0075] In this embodiment, the sensor component 22 is used to simulate the sensor function during the operation of the robot, and a new component can be loaded from the sensor component library. The sensor component 22 is connected to the interactive display component 21;

[0076] In this embodiment, the sensor assembly 22 includes but is not limited to the following subassemblies;

[0077] The laser radar sensor component 221 can be placed at any position of the wheeled robot, simulates the laser radar in a three-dimensional environment, generates 3D point cloud data and encapsulates it into a protocol format for transmission;

[0078] The IMU sensor component 222 obtains the linear acceleration and angular velocity information of the wheel-track robot body from the dynamics module 1, and encapsulates it into a protocol format for transmission;

[0079] The magnetometer sensor component 223 is used to add the body position of the wheeled robot to the three-dimensional model component 23 and obtain the quaternion of the body angle.

[0080] In this embodiment, all three-dimensional models of the three-dimensional simulation system are set in the three-dimensional model component 23, and a new three-dimensional model library can be imported from the outside. The three-dimensional model component 23 is connected to the sensor component 22;

[0081] In this embodiment, the three-dimensional model component 23 includes but is not limited to the following subcomponents:

[0082] Static object model component 231, including: Figure 3 3D model, 3D scene model, 3D model of static objects in the scene, and natural special effects; in this embodiment, natural special effects include but are not limited to: lighting and wind in the level;

[0083] The dynamic object model component 232 includes: a wheel-track robot model, which exchanges data with the dynamic interface component 233;

[0084] In this embodiment, the dynamic interface component 24 is used for data exchange between the plug-in of the three-dimensional simulation module 2 and the dynamic module 1. When the external dynamic modeling library is expanded, the interface is added and set in the dynamic modeling software, and the aforementioned dynamic interface component 24 performs the corresponding interface expansion, and the dynamic interface component 24 is connected to the three-dimensional model component 23.

[0085] In this embodiment, the communication component 25 is responsible for the information exchange between the three-dimensional simulation module 2 and the external module. The information sent by the communication component 25 includes but is not limited to: multi-dimensional sensor information, simulation instructions and receiving control information. The communication component 24 is connected with the interactive display component 21, the sensor component 22 and the three-dimensional model component 23;

[0086] In this embodiment, the general controller module 3 can be implemented in, for example, ROS;

[0087] In this embodiment, the general controller module 3 includes but is not limited to: an algorithm control module 31 and an algorithm module 32. Taking the perception and planning algorithms required for the patrol and obstacle avoidance tasks of the wheel-track robot in the embodiments of the present invention as an example, the algorithm control module 31 subscribes to task instructions in UE5 through Fastdds. If a task start instruction is received, it sequentially starts the perception and planning algorithm programs in the algorithm module 32, and these two programs respectively subscribe to radar point cloud data and IMU and magnetometer data through the Topic of the protocol. When the algorithm control module 31 receives an end instruction, it closes the perception and planning algorithm programs.

[0088] In this embodiment, the dynamics module 32 can be built using, for example, the dynamics modeling software Vortex. Use Vortex as a plugin for the UE5 engine, import the dynamics modeling of the wheel-track robot into UE5 in the form of a.Mechanism resource, and UE can call the input and output interfaces in the dynamics model through blueprints to achieve data exchange.

[0089] Embodiment 2

[0090] As Figure 3 and Figure 4 shown, in this embodiment, in the robot simulation method with multi-engine collaboration, the process of cooperation among various modules when implementing the patrol / obstacle avoidance task of the wheel-track robot further includes the following specific implementation steps:

[0091] S1. Enter the 3D simulation module 2 on the UE side, create a patrol / obstacle avoidance task using the graphical programming interface. After successful creation, it will jump to the corresponding level, load the corresponding Vortex dynamics model, and send the corresponding task instructions;

[0092] S2. After the ROS-side algorithm control module receives the start command through subscription, it sequentially starts the perception and planning algorithm programs;

[0093] S3. The UE side obtains sensor information from the environment or the Vortex dynamics model, sets the target point for the wheel-track robot to travel, and publishes it through FastDDS using the Topic of the protocol;

[0094] S4. The algorithm control module 31 in the general controller module 3 in ROS2 subscribes to sensor and target point information according to the Topic of the protocol. The perception and planning algorithm calculates based on this information and publishes the expected linear velocity and angular velocity of the wheel-track robot;

[0095] S5. The UE receives the desired linear velocity and angular velocity through Topic subscription and transmits this information to the dynamics model through the input interface provided by the Vortex plugin.

[0096] S6. After calculation, the dynamics model directly changes the pose of the wheeled tracked robot in the three-dimensional environment.

[0097] S7. Repeat the aforementioned steps S3 to S5 until the UE determines that the task is completed and issues a task completion command.

[0098] S8. After receiving the task completion command, the ROS side closes the perception planning algorithm program, and the algorithm control program remains.

[0099] For the functional components in the system, see Figure 5 , the UE side includes four parts: a magnetometer sensor component, a tracked vehicle model component, a lidar sensor component, and a Vortex dynamics component. The tracked vehicle model component, as the core component, obtains quaternion information from the magnetometer sensor component, lidar point cloud information from the lidar sensor component, and IMU information from the Vortex dynamics component through interfaces. At the same time, it feeds back the desired linear velocity and angular velocity of the tracked vehicle to drive the operation of the Vortex dynamics component. In this embodiment, the UE obtains the aforementioned information and sends it to the communication component through the interface, and obtains the desired linear velocity and angular velocity of the tracked vehicle from the communication component. The communication component and the ROS side also perform data interaction through the interface, sending IMU, quaternion, and lidar point cloud information to the perception algorithm component of the ROS side. The perception algorithm component generates target points and global lidar point clouds based on this information data and sends them to the planning algorithm component. The planning algorithm component feeds back the desired speed and angular velocity of the tracked vehicle to the communication component. In the patrol and obstacle avoidance task of the wheeled tracked robot in this embodiment, the UE completes the closed-loop of ROS-UE-Vortex based on Fastdds.

[0100] As Figure 6 shown, the UE5 process of the 3-wheeled tracked robot patrol and obstacle avoidance task in this embodiment further includes the following specific steps:

[0101] S11. Determine whether to enter the patrol task level.

[0102] S12. If so, enter the training task level.

[0103] S13. If not, determine whether to enter the obstacle avoidance task level.

[0104] S14. If so, enter the obstacle avoidance task level.

[0105] S15. If not, determine to enter other task levels.

[0106] S16. Create a publisher and a subscriber for communication;

[0107] S17. Determine whether the communication is successfully initialized;

[0108] S18. When it is determined that the communication is successfully initialized, use the publisher to publish the command to start the grasping task;

[0109] S19. Publish the current target point;

[0110] S110. Use the subscriber to receive the desired linear velocity and angular velocity of the tracked vehicle;

[0111] S111. Set the tracked vehicle to the desired linear velocity and angular velocity;

[0112] S112. Obtain the quaternion information of the tracked vehicle;

[0113] S113. Obtain the IMU information of the tracked vehicle;

[0114] S114. Use the publisher to publish the IMU and quaternion information;

[0115] S115. Obtain the radar point cloud information;

[0116] S116. Use the publisher to publish the radar point cloud information;

[0117] S117. Use the subscriber to receive the ROS target point arrival instruction;

[0118] S118. Determine whether the tracked vehicle has reached the current target position;

[0119] S119. If so, determine whether the current target point is the last target point;

[0120] S120. If so, determine that the task is successful;

[0121] S121. When the tracked vehicle has not reached the current target position, determine whether the tracked vehicle has encountered an obstacle; if so, determine that the task fails; if not, jump to the aforementioned steps S110, S112, S115, S117;

[0122] S122. When it is determined that the communication initialization is not successful, determine that the task fails;

[0123] S123. Determine whether the publisher is successfully initialized; if not, end the task;

[0124] S124. If so, use the publisher to publish the task end command.

[0125] The patrol mission scenario in this embodiment is shown in Figure 7 , and the obstacle avoidance mission scenario is shown in Figure 8 .

[0126] As Figure 9 shown, in this embodiment, the ROS process of the wheel-track robot's patrol and obstacle avoidance mission further includes the following basic steps:

[0127] S41. Receive the target point;

[0128] S42. Use the subscriber to receive the radar point cloud data;

[0129] S43. Determine whether the point cloud data is valid; if not, jump to the previous step S42;

[0130] S44. If so, draw the radar point cloud map and identify obstacles;

[0131] S45. Use the subscriber to receive the IMU and quaternion information;

[0132] S46. Perform path planning based on the target point, quaternion, IMU, and obstacles;

[0133] S47. Output the desired linear velocity and angular velocity according to the planned target point;

[0134] S48. Use the publisher to publish the desired linear velocity and angular velocity of the tracked vehicle;

[0135] S49. Use the subscriber to receive the end instruction;

[0136] S410. Determine whether the end instruction is received; if so, end the ROS process of the wheel-track robot's patrol and obstacle avoidance mission;

[0137] S411. If not, determine whether the tracked vehicle has reached the target point; if not, jump to the previous steps S42, S45, and S49 for execution;

[0138] S412. If so, use the publisher to publish the arrival instruction and jump to the previous step S41.

[0139] As Figure 10 shown, in this embodiment, the Vortex multi-body dynamics modeling operation further includes the following specific steps:

[0140] S31. Create a Mechanism project, name it, and save the.vxmechanism file;

[0141] S32. Import the.fbx 3D model, create a Graphics Gallery file and name it to save the.vxgraphicgallery file;

[0142] In this embodiment, create a project and import the.fbx 3D model of the robot to be modeled;

[0143] S33. Open the gallery model file and check each node and axis of the model;

[0144] S34. Determine whether the nodes and axes of the model are correct;

[0145] S35. If not, adjust the nodes of the model and align the axis to the correct node;

[0146] S36. If so, select the model nodes for which you want to create dynamics, create an Assembly dynamics module using Create partin, and name it to save the.vxassembly file;

[0147] S37. Enter the Assembly file and add appropriate collision bodies to each node of the model;

[0148] S38. Add collision rules to the collision bodies. For example, in this model, no collision occurs between the component nodes;

[0149] S39. Whether the materials in the existing material list meet the requirements;

[0150] S310. If not, add new materials to the material list and set basic parameters such as friction and stiffness;

[0151] S311. Set parameters such as materials and contact types for the collision bodies; set basic parameters such as mass and confidence for each component;

[0152] In this embodiment, create a dynamics file, add appropriate collision bodies to each component, and set parameters such as collision rules, mass, and materials;

[0153] S312. Add appropriate connection constraints between each collision body, and adjust the coordinate system of the constraints for the movement mode between the collision bodies;

[0154] S313. Adjust the connection constraints to add parameters such as motion limits, friction, and control methods;

[0155] S314. Add logical control methods such as connection containers and control scripts according to the model control requirements;

[0156] S315. Add an interface according to the requirements of the external input and output parameters of the dynamics model.

[0157] In this embodiment, complex control logics such as control scripts are added to the kinetic modeling. Input and output parameters are added to the interface.

[0158] IMU sensor simulation: In Vortex, mounting points are added to the body of the wheel-track robot, and kinematics sensors are added to the mounting points. The relative linear acceleration and relative angular velocity in the sensors are connected to the output parameters of the interface and transmitted to UE in real time. UE encapsulates these parameters in accordance with the protocol format, thus completing the simulation of the IMU sensor.

[0159] The process of obtaining one circle of lidar data is as Figure 11 shown:

[0160] S31’: Initialize the technical variable xn of the horizontal scanning times to 0, obtain the initial position and initial angle of the radar component, and initialize the point cloud data storage array SendList;

[0161] S32’: Determine whether xn is less than the total number of horizontal scans. If so, jump to S33’; otherwise, jump to S34’;

[0162] S33’: Initialize the array for storing the distance from the point to the radar, with a length of the number of radar lines * 2. Initialize the counting variable yn of the vertical scanning times to 0, and increment the xn count by 1;

[0163] S34’: Send the SendList array, and then end this process;

[0164] S35’: Determine whether yn is less than the number of radar lines 32. If so, go to S36’; otherwise, go to S37’;

[0165] S36’: Obtain the distance value obtained by the radar on the yn-th vertical line of the xn-th horizontal line and put it into the array. Increment the yn count by 1, and go to S320;

[0166] S37’: Put the timestamp and the radar scan distance array into SendList, and jump to S32’;

[0167] The lidar effect is as Figure 12As shown, in the laser radar sensor simulation process of this embodiment, the laser radar used is a remote sensing technology that detects targets by sending laser pulses and measuring their return time. The laser radar system generally includes a laser, a receiver, a positioning / scanning system, and a data processing unit. The general working principle of laser radar is: emit laser pulses to the target surface, and by measuring the return time of the laser pulse and its reflection intensity, the distance and shape between the targets can be calculated. Laser radar is usually able to achieve high-precision measurement and is widely used in unmanned vehicles, robots, map making, environmental monitoring and other fields. In the UE-side three-dimensional simulation system, rays are used to simulate laser pulses. When the rays are emitted to the surface of an object, a collision will occur. The straight-line distance between the ray emission source and the collision surface can be directly obtained, thereby achieving the effect of simulating laser pulses.

[0168] In this embodiment, a 32-line laser radar is used as an example, and the simulation parameters are: range: 200m, horizontal field of view: 360°, horizontal angle resolution: 0.2°, vertical field of view: 40° (-25°-15°), and transmission frequency 10HZ. The process of simulating a circle of laser radar is as follows: Figure 10 , a circle of point cloud data is encapsulated and published every 100ms. The data contains timestamp information. The size of the data sent at one time is about 133200 bytes. Fastdds can undertake the task of sending such large data.

[0169] In the magnetometer sensor simulation process of this embodiment, you can use, for example: the method in the UE blueprint to obtain the angle information of the wheeled robot in real time, convert it into quaternion, package it according to the format of the protocol and publish it, thus completing the simulation of the magnetometer sensor.

[0170] In this embodiment, Fastdds is integrated into UE5 as a plug-in, and the data to be sent to ROS is encapsulated into the data format of ROS itself. Finally, the subscription / publishing function is created through the method called by the blueprint. The ROS side uses its own publish / subscribe mode function to directly send, receive and parse the data received from UE5.

[0171] In summary, the key point of the present invention lies in the design concept of using the Fastdd communication middleware to connect several modules of Vortex, UE5, and ROS only in a data coupling manner; in the existing co-simulation technology, simulation tools such as Gazebo are used to combine with a 3D engine, or the combination of Modelica and a 3D engine is used. The former cannot ensure a high-precision dynamic model, and the latter cannot use a powerful control algorithm library, and the dynamic model cannot interact with the 3D environment, so neither is applicable to road robots. The present invention takes into account the use of control algorithms, high-precision multi-body dynamics, and good 3D rendering effects. Moreover, the Fastdds communication middleware is open-source and has excellent performance, which can ensure the running speed of the simulation system.

[0172] The present invention provides a robot hardware-in-the-loop motion control simulation system for the co-simulation of a ROS2 control system, a UE5 3D simulation environment, and Vortex dynamic modeling based on the Fastdds communication middleware. Based on the UE5 3D simulation engine, a simulation management system is developed to effectively manage and display various types of information and data during the simulation process, thereby improving the overall efficiency and visualization effect of the simulation system. A high-precision map is developed to simulate sensors such as lidar to achieve scene data collection and interaction, strengthening the authenticity and credibility of the simulation environment and providing a more accurate basis for the simulation of actual applications. The framework of the simulation system of the present invention is also applicable to the simulation requirements of other types of robots such as unmanned aerial vehicles and robotic arms, and has good flexibility and compatibility.

[0173] ROS adopted by the present invention has powerful algorithms and function libraries, is convenient to use the DDS communication protocol, and can be combined with physical robots in the later stage. Based on the FastDDS communication middleware, the present invention establishes data communication between UE5 and ROS2, is compatible with the ROS2 message format, and realizes the simulation test of the controller hardware-in-the-loop, providing reliable support and verification for the development of the actual control system. The combination of these technologies provides an important technical foundation and support for the comprehensive development of the simulation system, and will show great potential and value in various actual application scenarios.

[0174] The Fastdds communication middleware adopted by the present invention supports various platforms, is convenient for transmitting large amounts of data, has a fast transmission speed and stable transmission effect. When these four components are co-simulated, a good 3D rendering effect can be obtained, and diverse control algorithms and fine dynamic models can be taken into account. The designed simulation system framework of the present invention has good flexibility and scalability and can be used for the simulation tasks of various types of robots in multiple scenarios on land, sea, air, and space.

[0175] Based on the Vortex dynamics modeling software, the present invention realizes high-precision dynamic modeling of robot and scene interaction through the VHL_Interface data interaction method, obtaining more accurate and realistic simulation results. Vortex provides fine and highly customizable dynamic modeling.

[0176] While using the rich algorithms and function libraries in ROS2, the present invention constructs a real and high-definition three-dimensional simulation scene in UE5; integrating Vortex into UE5 allows the high-precision dynamic model to interact well with the three-dimensional simulation scene, simulating the road undulation effect during the driving process of the wheeled-tracked robot.

[0177] The Vortex dynamic model and the UE5 three-dimensional simulation engine adopted by the present invention jointly provide high-precision sensor simulation data for the ROS2 algorithm, enabling the algorithms in ROS2 to play a role similar to that of the physical robot. UE5 provides powerful three-dimensional rendering effects and is highly compatible with Vortex plugins.

[0178] The present invention solves the technical problems existing in the prior art, such as the difficulty in balancing multiple control algorithms and fine dynamic models, the difficulty in data interaction of the simulation system, the complexity of the simulation system, the high cost, and the low reliability.

[0179] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot simulation system with multi-engine collaboration, characterized in that Including: A dynamics module, a 3D simulation module, and a general controller module; The dynamics module is used to implement the dynamics module using Vortex. The dynamics module is imported into the 3D simulation module as a plug-in to perform mechanical interaction operations with the static model in the preset 3D simulation environment; The 3D simulation module is used to create patrol tasks and obstacle avoidance tasks using the preset graphical programming interface, jump to the corresponding levels, load the Vortex dynamics model, send task instructions corresponding to the patrol tasks and obstacle avoidance tasks. The 3D simulation module is connected to the dynamics module. The 3D simulation module is built using the UE5 3D simulation engine; The general controller module is implemented in ROS2. It is used to receive task start commands by subscribing, start the perception algorithm program and the planning algorithm program in sequence. In the UE5 3D simulation engine, it obtains sensor information from the 3D simulation environment and the Vortex dynamics model, sets the driving target point of the wheeled tracked robot, and publishes it through the Topic using the FastDDS protocol; subscribes to the sensor and the driving target point according to the Topic of the preset protocol, uses the perception algorithm program and the perception planning program to calculate and publish the desired linear velocity and desired angular velocity of the wheeled tracked robot, and adjusts the pose of the wheeled tracked robot in the 3D simulation environment accordingly. The general controller module is connected to the 3D simulation module.

2. The multi-engine collaborative robot simulation system according to claim 1, wherein The simulation system includes: the 3D simulation module and the general controller module in ROS2. UE5 and ROS2 data communication is established through the preset communication middleware to perform interactive operations on environmental information, model information, and control information, and is compatible with the message format of ROS2 to perform hardware-in-the-loop simulation testing of the controller.

3. The multi-engine collaborative robot simulation system according to claim 1, wherein The 3D simulation module includes: an interactive display component, a sensor component, a 3D model component, a dynamics interface component, and a communication component; The interactive display component is used to perform user interaction operations; The sensor component is used to load component data from the preset sensor component library to simulate the sensor functions during the operation of the robot. The sensor component is connected to the interactive display component; The 3D model component is used to import a 3D model library from the outside and set the 3D model of the 3D simulation system. The 3D model component is connected to the sensor component; The dynamics interface component is used to perform plug-in data exchange between the 3D simulation module and the dynamics module. During the extension of the external dynamics modeling library, interface addition settings are made in the preset dynamics modeling software, and the dynamics interface component is used for interface extension. The dynamics interface component is connected to the 3D model component; The communication component is used to perform information interaction operations between the 3D simulation module and the external module of the wheeled tracked robot. Among them, the information sent by the communication component also includes: multi-dimensional sensor information, simulation instructions, and received control information. The communication component is connected to the dynamics interface component, the 3D model component, and the dynamics interface component.

4. The multi-engine collaborative robot simulation system according to claim 3, wherein The interactive display component further includes: A graphical programming component is used to obtain a 3D selection model, a sensor selection type, and corresponding algorithms from the 3D model component and the sensor component; through task selection, dragging robot components, and algorithm module operations, a simulation task and task instructions are automatically generated and sent to the sensor component, the 3D model component, and the dynamics interface component. A data display component is used to generate and display robot control information received from the communication component according to the task instructions, and the data display component is connected to the graphical programming component.

5. The multi-engine collaborative robot simulation system according to claim 3, wherein, The sensor component includes: A lidar sensor component is installed at the lidar installation position of the wheeled tracked robot to simulate a lidar in a preset 3D simulation environment, generate 3D point cloud data, encapsulate the 3D point cloud data into a preset protocol format, and send it to the communication component. An IMU sensor component obtains the linear acceleration and angular velocity of the wheeled tracked robot body from the dynamics module, encapsulates the linear acceleration and the angular velocity into the preset protocol format, and sends it to the communication component. A magnetometer sensor component is used to add the body position of the wheeled tracked robot to the 3D model component to process and obtain the body angle quaternion.

6. The multi-engine collaborative robot simulation system according to claim 3, wherein The 3D model component further includes: A static object model component includes a map 3D model, a 3D scene model, 3D models of static objects in the scene, and natural special effects; among them, the natural special effects further include level lighting data and level wind data. A dynamic object model component includes a wheeled tracked robot model, which performs data exchange operations with the dynamics interface component.

7. The multi-engine collaborative robot simulation system according to claim 1, wherein The general controller module includes: An algorithm control module is used to subscribe to the task instructions in the UE5 3D simulation engine, and the task instructions include a task start instruction and a task end instruction. The dynamics module is built using the dynamics modeling software Vortex, and the dynamics modeling software Vortex is used as a plugin of the UE5 3D simulation engine. Through the VHL_Interface data interaction method, robot and scene interaction dynamics modeling is carried out to construct the Vortex dynamics model of the wheeled tracked robot. The Vortex dynamics model is imported into the UE5 3D simulation engine in a preset import method, and the input and output interfaces in the Vortex dynamics model are called using the blueprint of the UE5 3D simulation engine to achieve model data exchange; among them, the preset import method includes the.Mechanism resource method.

8. The multi-engine collaborative robot simulation system according to claim 7, wherein, The includes: When the task start instruction is received, the perception algorithm program and the planning algorithm program in the algorithm module are started in sequence to subscribe to lidar point cloud data and IMU and magnetometer data through the Topic of the preset protocol. When the algorithm control module receives the task end instruction, the perception algorithm program and the planning algorithm program are closed.

9. The multi-engine collaborative robot simulation system according to claim 7, characterized in that, Using the UE5 three-dimensional simulation engine, the desired linear velocity and the desired angular velocity are received through the Topic subscription, and the desired linear velocity and the desired angular velocity are transmitted to the dynamics model through the input-output interface in the Vortex dynamics model. The dynamics model calculates the desired linear velocity and the desired angular velocity to adjust the pose of the wheel-track robot in the three-dimensional simulation environment.

Images