Multi-engine collaborative robot simulation system and method

Abstract

The invention provides a multi-engine collaborative robot simulation method and system according to the claim 1, and the system comprises the steps: developing a simulation management system based on a UE5 three-dimensional simulation engine, and achieving the effective management and display of various information and data in a simulation process; a high-precision map is developed, a laser radar and other sensors are simulated to realize scene data acquisition and interaction, and the authenticity and credibility of a simulation environment are optimized for practical application simulation; based on Vortex dynamics modeling software, through a VHLInterface data interaction mode, robot and scene interaction dynamics modeling is carried out, and a robot simulation result is obtained; based on FastDDS communication middleware, data communication between the UE5 and the ROS2 is established, an ROS2 message format is compatible, and a hardware-in-the-loop simulation test of the controller is carried out. The technical problems that a multivariate control algorithm and a fine dynamic model are difficult to consider, data interaction of a simulation system is difficult, complexity and cost of the simulation system are high, and reliability is low are solved.

Description

Technical Field

[0001] The present invention relates to the field of robot simulation, and in particular to a multi-engine collaborative robot simulation system and method. Background Art

[0002] The robot simulation system was created with the support of computer technology and virtual simulation technology to meet the needs of robot research, development and testing. With the rapid development and widespread application of robot technology, an efficient, safe and low-cost method is needed to verify the robot's perception, planning and control functions. The robot simulation system provides a virtual environment that enables researchers and engineers to test and optimize robots in simulated scenarios, reduce experimental costs and risks, and improve development efficiency and technological innovation. Through the robot simulation system, a variety of different environments, tasks and robot types can be simulated, accelerating the development and promotion of robot technology and promoting the integration and application of robots with human society.

[0003] There are many simulation tools for different system levels, such as Simulink, a tool that focuses on system modeling. Its advantages lie in control system design, dynamic system modeling and simulation, and it is particularly suitable for the design, analysis and optimization of complex systems in engineering fields such as electrical, mechanical, aerospace, and automotive. However, its disadvantage is that its 3D rendering capabilities are relatively simple and cannot provide realistic and intuitive simulation effects. In addition, there are simulation platforms such as Gazebo, which focuses more on robot modeling, simulation and control, and can also be used for sensor simulation, with improved 3D rendering effects. However, it is difficult to obtain high-precision images of sensors and 3D scenes simulated using Gazebo.

[0004] In order to achieve more diverse simulation requirements, some multi-platform joint simulation robot simulation systems have emerged. For example, the existing invention patent application document "A simulation test and verification platform for multi-sensor fusion of unmanned aerial vehicles" with publication number CN116185074A, this existing simulation system uses the Unreal Engine engine and the algorithm module on the ROS side to simulate the unmanned aerial vehicle, deploys three-dimensional scenes and simulated sensors in the Unreal Engine engine, and sends the simulated data to the ROS side through network communication. The model reasoning module and multi-sensor fusion module on the ROS side use the simulated data for calculation, and the output motion instructions are then sent to the unmanned aerial vehicle in the Unreal Engine. This simulation system uses the excellent three-dimensional rendering capabilities of the Unreal Engine engine and the powerful function library of ROS to improve the realism of the simulation scene, provide higher-precision camera image data, and at the same time, can give full play to the role of various algorithms and function packages in ROS. However, this simulation system is more suitable for the simulation of unmanned aerial vehicles. Because it cannot provide a high-precision dynamic model, it is difficult to use it for the simulation of ground robots.

[0005] In order to combine the high-precision dynamic model with the powerful 3D rendering capability of the 3D engine, in the existing invention patent application document "Cooperative Simulation Method of Modelica Platform and UE4 Based on Opendds" with publication number CN115422723A, the robot is simulated and modeled in OpenModelica, and the 3D scene and sensors such as cameras are simulated in the Unreal Engine engine, and the data interaction between the two platforms is completed using the Opendds publish / subscribe mode. This simulation system can provide more sophisticated and customizable dynamic models and realistic 3D scenes, but it is also difficult to provide sophisticated road robot simulation functions, because the model information of the 3D scene in the Unreal Engine engine cannot be obtained by Modelica. When Modelica performs dynamic calculations, it is difficult to consider the impact of road conditions on the robot model, and can only rely on the model collision of the UnrealEngine engine itself.

[0006] At present, 3D simulation technology faces multiple challenges in the field of ground robots: it is difficult to simultaneously meet the requirements of realistic 3D rendering effects, complex control logic performance, and detailed dynamic models for interaction with scenes, especially when simulating ground robots in complex ground environments. At the same time, the processing of sensor data, transmission delays, and the complexity of multi-terminal transmission together lead to data interaction problems in simulation systems. In addition, the demand for hardware-in-the-loop increases the complexity and cost of simulation systems. These problems affect the development of ground robot simulation technology, and innovative solutions are urgently needed to improve the overall performance and reliability of simulation systems.

[0007] In summary, the existing technology has technical problems such as difficulty in balancing multiple control algorithms and sophisticated dynamic models, difficulty in data interaction in the simulation system, complexity of the simulation system, high cost and low reliability. Summary of the invention

[0008] The technical problem to be solved by the present invention is: how to solve the technical problems in the prior art that it is difficult to take into account both the multivariate control algorithm and the sophisticated dynamic model, the difficulty in data interaction of the simulation system, the complexity of the simulation system, the high cost and the low reliability.

[0009] The present invention adopts the following technical solutions to solve the above technical problems: a multi-engine collaborative robot simulation system comprises: a dynamics module, a three-dimensional simulation module and a universal controller module;

[0010] Dynamics module, which is used to implement the dynamics module using Vortex. The dynamics module is imported into the 3D simulation module as a plug-in to perform mechanical interaction with the static model in the preset 3D simulation environment.

[0011] The 3D simulation module is used to create patrol tasks and obstacle avoidance tasks using a preset graphical programming interface, jump to the corresponding level, load the Vortex dynamics model, and send task instructions corresponding to patrol tasks and obstacle avoidance tasks. The 3D simulation module is connected to the dynamics module;

[0012] A general controller module is implemented in ROS to receive task start commands through subscription, start the perception algorithm program and the planning algorithm program in sequence, obtain sensor information from the three-dimensional simulation environment and the Vortex dynamic model in the UE5 three-dimensional simulation engine, set the driving target point of the wheel-track robot, and publish it through the Topic of the FastDDS protocol; subscribe to sensors and form target points according to the Topic of the preset protocol, use the perception algorithm program and the perception planning program to obtain and publish the expected linear velocity and expected angular velocity of the wheel-track robot, and adjust the posture of the wheel-track robot in the three-dimensional simulation environment accordingly. The general controller module is connected to the three-dimensional simulation module.

[0013] The present invention provides a robot hardware-in-the-loop motion control simulation system based on the ROS2 control system of Fastdds communication middleware, UE5 three-dimensional simulation environment and Vortex dynamic modeling co-simulation. Based on the UE5 three-dimensional simulation engine, a simulation management system is developed to achieve effective management and display of various types of information and data in the simulation process, thereby improving the overall efficiency and visualization effect of the simulation system.

[0014] In a more specific technical solution, the UE5 3D simulation engine is used to build a 3D simulation module. The 3D simulation module and the general controller module in ROS2 establish data communication between UE5 and ROS2 through the preset communication middleware, and interact with the environmental information, model information and control information. It is compatible with the message format of ROS2 and performs simulation testing of the controller hardware in the loop.

[0015] The ROS used in the present invention has powerful algorithms and function libraries, is convenient for using the DDS communication protocol, and can be combined with a physical robot in the later stage. Based on the FastDDS communication middleware, the present invention establishes data communication between UE5 and ROS2, is compatible with the ROS2 message format, realizes the simulation test of the controller hardware in the loop, and provides reliable support and verification for the development of the actual control system. These technical combinations provide an important technical foundation and support for the comprehensive development of the simulation system, and will show great potential and value in various practical application scenarios.

[0016] In a more specific technical solution, the three-dimensional simulation module includes: an interactive display component, a sensor component, a three-dimensional model component, a dynamic interface component, and a communication component;

[0017] Interactive display components for user interaction;

[0018] The sensor component is used to load component data from a preset sensor component library to simulate the sensor function during the operation of the robot. The sensor component is connected to the interactive display component;

[0019] A three-dimensional model component is used to import a three-dimensional model library from outside, set a three-dimensional model of a three-dimensional simulation system, and connect the three-dimensional model component with the sensor component;

[0020] The dynamics interface component is used to exchange plug-in data between the 3D simulation module and the dynamics module. During the expansion of the external dynamics modeling library, the interface is added and set in the preset dynamics modeling software, and the interface is expanded using the dynamics interface component. The dynamics interface component is connected with the 3D model component.

[0021] The communication component is used for information exchange between the three-dimensional simulation module and the external module of the wheeled robot. The information sent by the communication component also includes: multi-dimensional sensor information, simulation instructions and receiving control information. The communication component is connected to the dynamic interface component, the three-dimensional model component and the dynamic interface component.

[0022] In a more specific technical solution, the interactive display component also includes:

[0023] Graphical programming components are used to obtain 3D selection models, sensor selection types and corresponding algorithms from 3D model components and sensor components; automatically generate simulation tasks and task instructions through task selection, dragging robot parts and algorithm module operations, and send task instructions to sensor components, 3D model components and dynamic interface components;

[0024] The data display component is used to generate and display the robot control information received from the communication component according to the task instructions. The data display component is connected with the graphical programming component.

[0025] In a more specific technical solution, the sensor assembly includes:

[0026] The laser radar sensor component is installed at the laser radar installation position of the wheeled robot to simulate the laser radar in a preset three-dimensional simulation environment, generate 3D point cloud data, encapsulate the 3D point cloud data into a preset protocol format and send it to the communication component;

[0027] The IMU sensor component obtains the linear acceleration and angular velocity of the wheel-track robot from the dynamics module, encapsulates the linear acceleration and angular acceleration into a preset protocol format and sends them to the communication component;

[0028] The magnetometer sensor component is used to add the body position of the wheeled robot to the three-dimensional model component to process and obtain the body angle quaternion.

[0029] The Fastdds communication middleware adopted by the present invention supports various platforms and is convenient for transmitting large amounts of data with fast transmission speed and stable transmission effect. When these four components are jointly simulated, a good three-dimensional rendering effect can be obtained, and multiple control algorithms and sophisticated dynamic models can be taken into account. The simulation system framework designed by the present invention has good flexibility and scalability, and can be used for simulation tasks of various types of robots in various scenarios of sea, land, air and space.

[0030] In a more specific technical solution, the three-dimensional model component also includes:

[0031] Static object model components, including: Figure 3 3D models, 3D scene models, 3D models of static objects in the scene, and natural special effects; among them, natural special effects also include: level lighting data, level wind data;

[0032] The dynamic object model component includes: a wheeled robot model, which performs data exchange operations with the dynamics interface component 233.

[0033] The present invention develops high-precision maps, simulates sensors such as laser radar to realize scene data collection and interaction, enhances the authenticity and credibility of the simulation environment, and provides a more accurate basis for simulation of practical applications. The framework of the simulation system of the present invention is also applicable to the simulation needs of other types of robots such as drones and mechanical arms, and has good flexibility and compatibility.

[0034] In a more specific technical solution, the universal controller module includes:

[0035] The algorithm control module is used to subscribe to the task instructions in the UE5 3D simulation engine. The task instructions include: task start instructions and task end instructions;

[0036] The dynamics module is built using the dynamics modeling software Vortex, and the dynamics modeling software Vortex is used as a plug-in for the UE5 3D simulation engine. The robot and scene interaction dynamics modeling is performed through the VHL_Interface data interaction method, and the Vortex dynamics model of the wheeled robot is constructed. The Vortex dynamics model is imported into the UE5 3D simulation engine in the preset import mode, and the input and output interfaces in the Vortex dynamics model are called using the blueprint of the UE 3D simulation engine to realize model data exchange. Among them, the preset import methods include: .Mechanism resource method.

[0037] The present invention is based on Vortex dynamic modeling software, and through the VHL_Interface data interaction method, it realizes high-precision robot and scene interactive dynamic modeling, and obtains more accurate and realistic simulation results. Vortex provides sophisticated and highly customizable dynamic modeling.

[0038] The present invention uses the rich algorithms and function libraries in ROS2 and builds a real high-definition three-dimensional simulation scene in UE5. The integration of Vortex into UE5 allows the high-precision dynamic model to interact well with the three-dimensional simulation scene, and simulates the undulating effect of the road surface during the driving process of the wheeled robot.

[0039] In a more specific technical solution, when a task start instruction is received, the perception algorithm program and the planning algorithm program in the algorithm module are started in sequence to subscribe to the radar point cloud data and IMU and magnetometer data through the Topic of the preset protocol; when the algorithm control module receives a task end instruction, the perception algorithm program and the planning algorithm program are closed.

[0040] In a more specific technical solution, the UE5 three-dimensional simulation engine is used to receive the expected linear velocity and expected angular velocity through Topic subscription, and the expected linear velocity and expected angular velocity are transmitted to the dynamic model through the input and output interface in the Vortex dynamic model. The dynamic model solves the expected linear velocity and expected angular velocity to adjust the posture of the wheeled robot in the three-dimensional simulation environment.

[0041] The Vortex dynamics model and UE5 3D simulation engine used in the present invention provide high-precision sensor simulation data for the ROS2 algorithm, allowing the algorithm in ROS2 to play a role similar to that of a real robot. UE5 provides powerful 3D rendering effects and is highly compatible with the Vortex plug-in.

[0042] In a more specific technical solution, the multi-engine collaborative robot simulation method includes:

[0043] S1. Use Vortex to implement the dynamics module, import the dynamics module as a plug-in into the 3D simulation module, and perform mechanical interaction with the static model in the preset 3D simulation environment;

[0044] S2. Use the preset graphical programming interface to create patrol tasks and obstacle avoidance tasks, jump to the corresponding level, load the Vortex dynamics model, and send task instructions corresponding to patrol tasks and obstacle avoidance tasks;

[0045] S3. Receive the task start command through subscription, start the perception algorithm program and the planning algorithm program in sequence, obtain sensor information from the three-dimensional simulation environment and the Vortex dynamic model in the UE5 three-dimensional simulation engine, set the driving target point of the wheel-track robot, and publish it through the Topic of the FastDDS protocol; subscribe to the sensors and the form target points according to the Topic of the preset protocol, use the perception algorithm program and the perception planning program to obtain and publish the expected linear velocity and expected angular velocity of the wheel-track robot, and adjust the posture of the wheel-track robot in the three-dimensional simulation environment.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] The present invention provides a robot hardware-in-the-loop motion control simulation system based on the ROS2 control system of Fastdds communication middleware, UE5 three-dimensional simulation environment and Vortex dynamic modeling joint simulation. Based on the UE5 three-dimensional simulation engine, a simulation management system is developed to achieve effective management and display of various types of information and data in the simulation process, thereby improving the overall efficiency and visualization of the simulation system. High-precision maps are developed, and sensors such as laser radars are simulated to achieve scene data collection and interaction, enhance the authenticity and credibility of the simulation environment, and provide a more accurate basis for simulation of practical applications. The framework of the simulation system of the present invention is also applicable to the simulation needs of other types of robots such as drones and robotic arms, and has good flexibility and compatibility.

[0048] The ROS used in the present invention has powerful algorithms and function libraries, is convenient for using the DDS communication protocol, and can be combined with a physical robot in the later stage. Based on the FastDDS communication middleware, the present invention establishes data communication between UE5 and ROS2, is compatible with the ROS2 message format, realizes the simulation test of the controller hardware in the loop, and provides reliable support and verification for the development of the actual control system. These technical combinations provide an important technical foundation and support for the comprehensive development of the simulation system, and will show great potential and value in various practical application scenarios.

[0049] The Fastdds communication middleware adopted by the present invention supports various platforms and is convenient for transmitting large amounts of data with fast transmission speed and stable transmission effect. When these four components are jointly simulated, a good three-dimensional rendering effect can be obtained, and multiple control algorithms and sophisticated dynamic models can be taken into account. The simulation system framework designed by the present invention has good flexibility and scalability, and can be used for simulation tasks of various types of robots in various scenarios of sea, land, air and space.

[0050] The present invention is based on Vortex dynamic modeling software, and through the VHL_Interface data interaction method, it realizes high-precision robot and scene interactive dynamic modeling, and obtains more accurate and realistic simulation results. Vortex provides sophisticated and highly customizable dynamic modeling.

[0051] The present invention uses the rich algorithms and function libraries in ROS2 and builds a real high-definition three-dimensional simulation scene in UE5. The integration of Vortex into UE5 allows the high-precision dynamic model to interact well with the three-dimensional simulation scene, and simulates the undulating effect of the road surface during the driving process of the wheeled robot.

[0052] The Vortex dynamics model and UE5 3D simulation engine used in the present invention provide high-precision sensor simulation data for the ROS2 algorithm, allowing the algorithm in ROS2 to play a role similar to that of a real robot. UE5 provides powerful 3D rendering effects and is highly compatible with the Vortex plug-in.

[0053] The present invention solves the technical problems existing in the prior art, namely, difficulty in balancing multivariate control algorithms and sophisticated dynamic models, difficulty in data interaction of simulation systems, complexity of simulation systems, high costs and low reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a schematic diagram of the connection of modules of a multi-engine collaborative robot simulation system according to Embodiment 1 of the present invention;

[0055] Figure 2 This is a schematic diagram of information flow between three-dimensional simulation modules according to Embodiment 1 of the present invention;

[0056] Figure 3 This is a schematic diagram of the collaborative data flow processing of each module when implementing the patrol / obstacle avoidance task of the wheeled robot according to Example 2 of the present invention;

[0057] Figure 4 This is a schematic diagram of specific steps of cooperation among modules when implementing the patrol / obstacle avoidance task of the wheeled robot according to Embodiment 2 of the present invention;

[0058] Figure 5 This is a schematic diagram of data flow processing of a wheeled robot patrol obstacle avoidance task component according to Embodiment 2 of the present invention;

[0059] Figure 6 This is a schematic diagram of specific steps of the process of UE5 of the patrol obstacle avoidance task of the wheeled robot in Embodiment 2 of the present invention;

[0060] Figure 7 This is a schematic diagram of a wheeled robot patrol scene according to Embodiment 2 of the present invention;

[0061] Figure 8 This is a schematic diagram of an obstacle avoidance scenario for a wheeled robot according to Embodiment 2 of the present invention;

[0062] Fig. 9 This is a schematic diagram of the specific steps of the ROS process of the wheeled robot patrol obstacle avoidance task in Example 2 of the present invention;

[0063] Fig.10 This is a flow chart of Vortex multi-body dynamics modeling according to Example 2 of the present invention;

[0064] Fig.11 This is a rendering effect diagram of the laser radar sensor according to Embodiment 2 of the present invention;

[0065] Fig.12 This is a 3D model diagram of a two-wheeled robot according to embodiment 2 of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Example 1

[0068] like Figure 1 As shown, the basic modules of the multi-engine collaborative robot simulation system include: a dynamics module 1, a three-dimensional simulation module 2 and a general controller module 3.

[0069] In this embodiment, the dynamics module 1 includes but is not limited to: a tracked vehicle dynamics modeling component 11 and other dynamics modeling components 12; it can be implemented by, for example, Vortex, imported into the three-dimensional simulation module 2 as a plug-in, and mechanically interacts with the static model in the three-dimensional simulation environment;

[0070] The 3D simulation module 2 is built on the UE5 platform. The 3D simulation module 2 is connected to the general controller module 3 in ROS2, and the environment information, model information and control information are interacted through the communication middleware. For the specific interaction principle, see Figure 1 .

[0071] like Figure 2 As shown, in this embodiment, the three-dimensional simulation module 2 includes but is not limited to: an interactive display component 21, a sensor component 22, a three-dimensional model component 23, a dynamic interface component 24 and a communication component 25.

[0072] In this embodiment, the interactive display component 21 performs user interaction; in this embodiment, the interactive display component 21 includes but is not limited to the following subcomponents:

[0073] The graphical programming component 211 is used to obtain the three-dimensional model, sensor type and corresponding algorithm that can be selected from the three-dimensional model component 23 and the sensor component 22; automatically generate simulation tasks and issue corresponding task instructions to other components through task selection and dragging robot parts and algorithm module operations;

[0074] The data display component 212 is used to display the data received through the communication component 25, including but not limited to: robot control information.

[0075] In this embodiment, the sensor component 22 is used to simulate the sensor function during the operation of the robot, and a new component can be loaded from the sensor component library. The sensor component 22 is connected to the interactive display component 21;

[0076] In this embodiment, the sensor assembly 22 includes but is not limited to the following subassemblies;

[0077] The laser radar sensor component 221 can be placed at any position of the wheeled robot, simulates the laser radar in a three-dimensional environment, generates 3D point cloud data and encapsulates it into a protocol format for transmission;

[0078] The IMU sensor component 222 obtains the linear acceleration and angular velocity information of the wheel-track robot body from the dynamics module 1, and encapsulates it into a protocol format for transmission;

[0079] The magnetometer sensor component 223 is used to add the wheeled robot body position to the three-dimensional model component 23 and obtain the quaternion of the body angle.

[0080] In this embodiment, all three-dimensional models of the three-dimensional simulation system are set in the three-dimensional model component 23, and a new three-dimensional model library can be imported from the outside. The three-dimensional model component 23 is connected to the sensor component 22;

[0081] In this embodiment, the three-dimensional model component 23 includes but is not limited to the following subcomponents:

[0082] Static object model component 231, including: Figure 3 3D model, 3D scene model, 3D model of static objects in the scene, and natural special effects; in this embodiment, natural special effects include but are not limited to: lighting and wind in the level;

[0083] The dynamic object model component 232 includes: a wheel-track robot model, which exchanges data with the dynamic interface component 233;

[0084] In this embodiment, the dynamic interface component 24 is used for data exchange between the plug-in of the three-dimensional simulation module 2 and the dynamic module 1. When the external dynamic modeling library is expanded, the interface is added and set in the dynamic modeling software, and the aforementioned dynamic interface component 24 performs the corresponding interface expansion, and the dynamic interface component 24 is connected to the three-dimensional model component 23.

[0085] In this embodiment, the communication component 25 is responsible for the information exchange between the three-dimensional simulation module 2 and the external module. The information sent by the communication component 25 includes but is not limited to: multi-dimensional sensor information, simulation instructions and receiving control information. The communication component 24 is connected with the interactive display component 21, the sensor component 22 and the three-dimensional model component 23;

[0086] In this embodiment, the universal controller module 3 may be implemented in, for example: ROS;

[0087] In this embodiment, the general controller module 3 includes but is not limited to: an algorithm control module 31 and an algorithm module 32. Taking the perception and planning algorithms required for the patrol and obstacle avoidance tasks of the wheeled robot in the embodiment of the present invention as an example, the algorithm control module 31 subscribes to the task instructions in UE5 through Fastdds. If a task start instruction is received, the perception and planning algorithm programs in the algorithm module 32 are started in sequence. These two programs then subscribe to the radar point cloud data and IMU and magnetometer data through the Topic of the protocol respectively. When the algorithm control module 31 receives the end instruction, it will close the perception and planning algorithm programs.

[0088] In this embodiment, the dynamics module 32 can be constructed using, for example, the dynamics modeling software Vortex, and Vortex can be used as a plug-in for the UE5 engine. The dynamics modeling of the wheeled robot can be imported into UE5 in the form of .Mechanism resources. The UE can call the input and output interfaces in the dynamics model through the blueprint to realize data exchange.

[0089] Example 2

[0090] like Figure 3 and Figure 4 As shown, in this embodiment, in the multi-engine collaborative robot simulation method, the process of collaboration of each module when implementing the wheeled robot patrol / obstacle avoidance task also includes the following specific implementation steps:

[0091] S1, enter the 3D simulation module 2 on the UE side, use the graphical programming interface to create a patrol / obstacle avoidance task, after successful creation, jump to the corresponding level, load the corresponding Vortex dynamics model, and send the corresponding task instructions;

[0092] S2, the ROS-side algorithm control module, starts the perception and planning algorithm programs in sequence after receiving the start command through subscription;

[0093] S3, the UE obtains sensor information from the environment or the Vortex dynamics model, sets the target point for the wheeled robot to travel, and publishes it through the Topic of the FastDDS protocol;

[0094] S4, the algorithm control module 31 in the general controller module 3 in ROS2 subscribes to the sensor and target point information according to the Topic of the protocol, and the perception planning algorithm calculates based on this information, outputs the expected linear velocity and angular velocity of the wheeled robot and publishes it;

[0095] S5. The UE receives the expected linear velocity and angular velocity through Topic subscription, and passes this information to the dynamic model through the input interface provided by the Vortex plug-in;

[0096] S6, the dynamic model is solved to directly change the position and posture of the wheeled robot in the three-dimensional environment;

[0097] S7, repeat the above steps S3 to S5 until the UE determines that the task is completed and issues a task completion command;

[0098] S8. After receiving the task end command, the ROS end closes the perception planning algorithm program, and the algorithm control program is retained.

[0099] Functional components in the system Figure 5 , the UE end includes four parts: magnetometer sensor component, tracked vehicle model component, lidar sensor component and Vortex dynamics component. The tracked vehicle model component is the core component. It obtains quaternion information from the magnetometer sensor component through the interface, obtains radar point cloud information from the lidar sensor component, and obtains IMU information from the Vortex dynamics component. At the same time, it feeds back the expected linear velocity and angular velocity of the tracked vehicle to drive the Vortex dynamics component to run. In this embodiment, the UE end obtains the above information, sends it to the communication component through the interface, and obtains the expected linear velocity and angular velocity of the tracked vehicle from the communication component. The communication component and the ROS end also exchange data through the interface, sending IMU, quaternion and radar point cloud information to the perception algorithm component of the ROS end. The perception algorithm component generates target points and global radar point clouds based on these information data and sends them to the planning algorithm component. The planning algorithm component feeds back the communication component the expected speed and angular velocity of the tracked vehicle. The UE end includes the wheeled robot patrol obstacle avoidance task in this embodiment, and completes the ROS-UE-Vortex closed loop based on Fastdds.

[0100] like Figure 6 As shown, the UE5 process of the three-wheeled robot patrol obstacle avoidance task in this embodiment also includes the following specific steps:

[0101] S11, determining whether to enter the patrol mission level;

[0102] S12: If yes, then enter the training mission level;

[0103] S13, if not, determine whether to enter the obstacle avoidance task level;

[0104] S14: If yes, enter the obstacle avoidance task level;

[0105] S15, if not, then determine to enter other task levels;

[0106] S16. Create publisher and subscriber for communication;

[0107] S17, judging whether the communication is initialized successfully;

[0108] S18. When it is determined that the communication initialization is successful, use the publisher to issue a crawling task start command;

[0109] S19, publishing the current target point;

[0110] S110, using a subscriber to receive the expected linear velocity and angular velocity of the tracked vehicle;

[0111] S111, setting the tracked vehicle to a desired linear velocity and angular velocity;

[0112] S112, obtaining quaternion information of the tracked vehicle;

[0113] S113, obtaining IMU information of the tracked vehicle;

[0114] S114. Use publisher to publish IMU and quaternion information;

[0115] S115, obtaining radar point cloud information;

[0116] S116. Use publisher to publish radar point cloud information;

[0117] S117, using subscriber to receive ROS target point arrival instructions;

[0118] S118, determining whether the crawler vehicle has reached the current target position;

[0119] S119: If yes, determine whether the current target point is the last target point;

[0120] S120: If yes, the task is determined to be successful;

[0121] S121, when the crawler vehicle has not reached the current target position, determine whether the crawler vehicle hits an obstacle; if so, determine that the task has failed; if not, jump to the aforementioned steps S110, S112, S115, S117;

[0122] S122, when it is determined that the communication initialization is unsuccessful, determining that the task has failed;

[0123] S123, determine whether the publisher is initialized successfully; if not, end the task;

[0124] S124. If yes, use publisher to publish the task end command.

[0125] The patrol mission scenario in this embodiment can be found in Figure 7 , obstacle avoidance task scenario see Figure 8 .

[0126] like Fig. 9 As shown, in this embodiment, the ROS process of the wheeled robot patrol obstacle avoidance task also includes the following basic steps:

[0127] S41, receiving the target point;

[0128] S42. Use subscriber to receive radar point cloud data;

[0129] S43, judging whether the point cloud data is valid; if not, jumping to the aforementioned step S42;

[0130] S44, if yes, draw a radar point cloud map to identify obstacles;

[0131] S45. Use subscriber to receive IMU and quaternion information;

[0132] S46, performing path planning according to target points, quaternions, IMU, and obstacles;

[0133] S47, outputting the expected linear velocity and angular velocity according to the planned target point;

[0134] S48. Use publisher to publish the expected linear velocity and angular velocity of the tracked vehicle;

[0135] S49, using subscriber to receive the end instruction;

[0136] S410, determine whether an end instruction is received; if so, end the ROS process of the wheeled robot patrol obstacle avoidance task;

[0137] S411, if not, determine whether the crawler vehicle has reached the target point; if not, jump to execute the aforementioned steps S42, S45 and S49;

[0138] S412: If yes, use publisher to publish the arrival instruction and jump to the aforementioned step S41.

[0139] like Fig.10 As shown, in this embodiment, the Vortex multi-body dynamics modeling operation also includes the following specific steps:

[0140] S31. Create a Mechanism project, name it, and save the .vxmechanism file.

[0141] S32. Import the .fbx 3D model, create a Graphics Gallery file and save it as .vxgraphicgallery;

[0142] In this embodiment, a project is created and the .fbx 3D model of the robot to be modeled is imported;

[0143] S33. Open the gallery model file and check each node and axis of the model;

[0144] S34, judging whether the nodes and axes of the model are correct;

[0145] S35, if not, adjust the nodes of the model and align the axis to the correct node;

[0146] S36. If yes, select the model node for which you want to create dynamics, Create partin to create an Assembly dynamics module, and save it in a .vxassembly file;

[0147] S37. Enter the Assembly file and add appropriate collision bodies to each node of the model;

[0148] S38, adding collision rules to the collision body, for example, in this model, no collision occurs between component nodes;

[0149] S39. Whether the materials in the existing material list meet the requirements;

[0150] S310, if not, add a new material to the material list and set basic parameters such as friction and stiffness;

[0151] S311. Set material, contact type and other parameters for the collision body; set basic parameters such as quality and confidence for each component;

[0152] In this embodiment, a dynamics file is created, appropriate collision bodies are added to each component, and parameters such as collision rules, quality, and material are set;

[0153] S312, adding appropriate connection constraints between the collision bodies, and adjusting the coordinate system of the constraints according to the movement mode between the collision bodies;

[0154] S313, adjusting the connection constraints to add parameters such as motion restrictions, friction, and control methods;

[0155] S314. Adding logical control methods such as connection containers and control scripts according to model control requirements;

[0156] S315. Add an interface according to the requirements of the dynamic model for external input and output parameters.

[0157] In this embodiment, complex control logic such as control scripts is added to the dynamics modeling, and input and output parameters are added to the interface.

[0158] IMU sensor simulation: In Vortex, add mounting points to the wheeled robot body, add kinematics sensors to the mounting points, connect the relative linear acceleration and relative angular velocity in the sensor to the output parameters of the interface, and transmit them to the UE in real time. The UE encapsulates these parameters according to the protocol format, thus completing the simulation of the IMU sensor.

[0159] The process of acquiring lidar data is as follows: Fig.11 As shown:

[0160] S31', initialize the horizontal scanning times technical variable xn to 0, obtain the initial position and initial angle of the radar component, and initialize the point cloud data storage array SendList;

[0161] S32', determine whether xn is less than the total number of horizontal scans, if yes, jump to S33', otherwise jump to S34';

[0162] S33', initialize the storage array of the distance from the point to the radar, the length of which is the number of radar lines * 2, initialize the vertical direction scanning count variable yn to 0, and add 1 to the xn count;

[0163] S34', send the SendList array, and then end this process;

[0164] S35', determine whether yn is less than the number of radar lines 32, if yes, go to S36', otherwise go to S37';

[0165] S36', obtain the distance value obtained by the radar on the ynth vertical line on the xnth horizontal line and put it into an array, add 1 to the yn count, and go to S320;

[0166] S37', put the timestamp and radar scanning distance array into SendList, and jump to S32';

[0167] Laser radar effect Fig.12As shown, in the laser radar sensor simulation process of this embodiment, the laser radar used is a remote sensing technology that detects targets by sending laser pulses and measuring their return time. The laser radar system generally includes a laser, a receiver, a positioning / scanning system, and a data processing unit. The general working principle of laser radar is: emit laser pulses to the target surface, and by measuring the return time of the laser pulse and its reflection intensity, the distance and shape between the targets can be calculated. Laser radar is usually able to achieve high-precision measurement and is widely used in unmanned vehicles, robots, map making, environmental monitoring and other fields. In the UE-side three-dimensional simulation system, rays are used to simulate laser pulses. When the rays are emitted to the surface of an object, a collision will occur. The straight-line distance between the ray emission source and the collision surface can be directly obtained, thereby achieving the effect of simulating laser pulses.

[0168] In this embodiment, a 32-line laser radar is used as an example, and the simulation parameters are: range: 200m, horizontal field of view: 360°, horizontal angle resolution: 0.2°, vertical field of view: 40° (-25°-15°), and transmission frequency 10HZ. The process of simulating a circle of laser radar is as follows: Fig.10 , a circle of point cloud data is encapsulated and published every 100ms. The data contains timestamp information. The size of the data sent at one time is about 133200 bytes. Fastdds can be used to undertake the task of sending such large data.

[0169] In the magnetometer sensor simulation process of this embodiment, you can use, for example: the method in the UE blueprint to obtain the angle information of the wheeled robot in real time, convert it into quaternion, package it according to the format of the protocol and publish it, thus completing the simulation of the magnetometer sensor.

[0170] In this embodiment, Fastdds is integrated into UE5 as a plug-in, and the data to be sent to ROS is encapsulated into the data format of ROS itself. Finally, the subscription / publishing function is created through the method called by the blueprint. The ROS side uses its own publish / subscribe mode function to directly send, receive and parse the data received from UE5.

[0171] In summary, the key point of the present invention is to use Fastdd communication middleware, only using data coupling to connect the design ideas of Vortex, UE5 and ROS modules; the existing joint simulation technology uses simulation tools such as gazebo and three-dimensional engines, or uses Modelica and three-dimensional engines. The former cannot ensure high-precision dynamic models, and the latter cannot use a powerful control algorithm library, and the dynamic model cannot interact with the three-dimensional environment, which is not suitable for road robots. The present invention takes into account the use of control algorithms, high-precision multi-body dynamics and good three-dimensional rendering effects, and the Fastdds communication middleware is open source and has excellent performance, which can ensure the running speed of the simulation system.

[0172] The present invention provides a robot hardware-in-the-loop motion control simulation system based on the ROS2 control system of Fastdds communication middleware, UE5 three-dimensional simulation environment and Vortex dynamic modeling joint simulation. Based on the UE5 three-dimensional simulation engine, a simulation management system is developed to achieve effective management and display of various types of information and data in the simulation process, thereby improving the overall efficiency and visualization of the simulation system. High-precision maps are developed, and sensors such as laser radars are simulated to achieve scene data collection and interaction, enhance the authenticity and credibility of the simulation environment, and provide a more accurate basis for simulation of practical applications. The framework of the simulation system of the present invention is also applicable to the simulation needs of other types of robots such as drones and robotic arms, and has good flexibility and compatibility.

[0173] The ROS used in the present invention has powerful algorithms and function libraries, is convenient for using the DDS communication protocol, and can be combined with a physical robot in the later stage. Based on the FastDDS communication middleware, the present invention establishes data communication between UE5 and ROS2, is compatible with the ROS2 message format, realizes the simulation test of the controller hardware in the loop, and provides reliable support and verification for the development of the actual control system. These technical combinations provide an important technical foundation and support for the comprehensive development of the simulation system, and will show great potential and value in various practical application scenarios.

[0174] The Fastdds communication middleware adopted by the present invention supports various platforms and is convenient for transmitting large amounts of data with fast transmission speed and stable transmission effect. When these four components are jointly simulated, a good three-dimensional rendering effect can be obtained, and multiple control algorithms and sophisticated dynamic models can be taken into account. The simulation system framework designed by the present invention has good flexibility and scalability, and can be used for simulation tasks of various types of robots in various scenarios of sea, land, air and space.

[0175] The present invention is based on Vortex dynamic modeling software, and through the VHL_Interface data interaction method, it realizes high-precision robot and scene interactive dynamic modeling, and obtains more accurate and realistic simulation results. Vortex provides sophisticated and highly customizable dynamic modeling.

[0176] The present invention uses the rich algorithms and function libraries in ROS2 and builds a real high-definition three-dimensional simulation scene in UE5. The integration of Vortex into UE5 allows the high-precision dynamic model to interact well with the three-dimensional simulation scene, and simulates the undulating effect of the road surface during the driving process of the wheeled robot.

[0177] The Vortex dynamics model and UE5 3D simulation engine used in the present invention provide high-precision sensor simulation data for the ROS2 algorithm, allowing the algorithm in ROS2 to play a role similar to that of a real robot. UE5 provides powerful 3D rendering effects and is highly compatible with the Vortex plug-in.

[0178] The present invention solves the technical problems existing in the prior art, namely, difficulty in balancing multivariate control algorithms and sophisticated dynamic models, difficulty in data interaction of simulation systems, complexity of simulation systems, high costs and low reliability.

[0179] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-engine collaborative robot simulation system, characterized in that: include: Dynamics module, 3D simulation module and general controller module; The dynamics module is used to implement the dynamics module using Vortex, and the dynamics module is imported into the three-dimensional simulation module as a plug-in to perform mechanical interaction operations with a static model in a preset three-dimensional simulation environment; The three-dimensional simulation module is used to create patrol tasks and obstacle avoidance tasks using a preset graphical programming interface, jump to the corresponding level, load the Vortex dynamics model, and send task instructions corresponding to the patrol tasks and the obstacle avoidance tasks. The three-dimensional simulation module is connected to the dynamics module; The general controller module is implemented in ROS to receive task start commands through subscription, start the perception algorithm program and the planning algorithm program in sequence, obtain sensor information from the three-dimensional simulation environment and the Vortex dynamic model in the UE5 three-dimensional simulation engine, set the driving target point of the wheel-crawler robot, and publish it through the Topic of the FastDDS protocol; subscribe to sensors and the form target point according to the Topic of the preset protocol, use the perception algorithm program and the perception planning program to obtain and publish the expected linear velocity and the expected angular velocity of the wheel-crawler robot, and adjust the posture of the wheel-crawler robot in the three-dimensional simulation environment accordingly. The general controller module is connected to the three-dimensional simulation module.

2. The multi-engine collaborative robot simulation system according to claim 1 is characterized in that: The method includes: using the UE5 3D simulation engine to build the 3D simulation module, the 3D simulation module and the general controller module in ROS2, establishing data communication between UE5 and ROS2 through preset communication middleware, performing interactive operations on environmental information, model information and control information, being compatible with the message format of ROS2, and performing simulation testing of controller hardware in the loop.

3. The multi-engine collaborative robot simulation system according to claim 1 is characterized in that: The three-dimensional simulation module includes: an interactive display component, a sensor component, a three-dimensional model component, a dynamic interface component and a communication component; The interactive display component is used to perform user interactive operations; The sensor component is used to load component data from a preset sensor component library to simulate the sensor function during the operation of the robot, and the sensor component is connected to the interactive display component; The three-dimensional model component is used to import a three-dimensional model library from the outside to set a three-dimensional model of a three-dimensional simulation system, and the three-dimensional model component is connected to the sensor component; The dynamics interface component is used to exchange plug-in data between the three-dimensional simulation module and the dynamics module. During the expansion of the external dynamics modeling library, the interface is added and set in the preset dynamics modeling software, and the interface is expanded using the dynamics interface component. The dynamics interface component is connected to the three-dimensional model component. The communication component is used to perform information exchange operations between the three-dimensional simulation module and the external module of the wheeled robot, wherein the information sent by the communication component also includes: multi-dimensional sensor information, simulation instructions and receiving control information. The communication component is connected to the dynamic interface component, the three-dimensional model component and the dynamic interface component.

4. The multi-engine collaborative robot simulation system according to claim 3 is characterized in that: The interactive display component also includes: A graphical programming component is used to obtain a three-dimensional selection model, a sensor selection type and a corresponding algorithm from the three-dimensional model component and the sensor component; automatically generate simulation tasks and task instructions through task selection, dragging robot parts and algorithm module operations, and send the task instructions to the sensor component, the three-dimensional model component and the dynamic interface component; The data display component is used to generate and display the robot control information received from the communication component according to the task instruction, and the data display component is connected to the graphical programming component.

5. The multi-engine collaborative robot simulation system according to claim 3 is characterized in that: The sensor assembly comprises: A laser radar sensor component is installed at the laser radar installation position of the wheeled robot to simulate the laser radar in a preset three-dimensional simulation environment, generate 3D point cloud data, encapsulate the 3D point cloud data into a preset protocol format and send it to the communication component; An IMU sensor component obtains the linear acceleration and angular velocity of the wheel-track robot body from the dynamics module, encapsulates the linear acceleration and the angular acceleration into the preset protocol format and sends them to the communication component; The magnetometer sensor component is used to add the body position of the wheeled robot to the three-dimensional model component to obtain the body angle quaternion.

6. The multi-engine collaborative robot simulation system according to claim 3 is characterized in that: The three-dimensional model component also includes: Static object model components, including: a 3D map model, a 3D scene model, a 3D model of static objects in the scene, and natural special effects; wherein the natural special effects also include: level lighting data and level wind data; The dynamic object model component includes: a wheel-track robot model, which performs data exchange operations with the dynamic interface component 233.

7. The multi-engine collaborative robot simulation system according to claim 1 is characterized in that: The universal controller module comprises: An algorithm control module, used to subscribe to the task instructions in the UE5 three-dimensional simulation engine, wherein the task instructions include: a task start instruction and a task end instruction; The dynamics module is built using the dynamics modeling software Vortex, and the dynamics modeling software Vortex is used as a plug-in of the UE5 three-dimensional simulation engine. The robot and scene interaction dynamics modeling is performed through the VHL_Interface data interaction method to construct a Vortex dynamics model of the wheeled robot. The Vortex dynamics model is imported into the UE5 three-dimensional simulation engine in a preset import method, and the input and output interfaces in the Vortex dynamics model are called using the blueprint of the UE three-dimensional simulation engine to realize model data exchange; wherein, the preset import method includes: .Mechanism resource method.

8. The multi-engine collaborative robot simulation system according to claim 7 is characterized in that: Said include: When receiving the task start instruction, the perception algorithm program and the planning algorithm program in the algorithm module are started in sequence to subscribe to the radar point cloud data and the IMU and magnetometer data through the Topic of the preset protocol; When the algorithm control module receives the task end instruction, it closes the perception algorithm program and the planning algorithm program.

9. The multi-engine collaborative robot simulation system according to claim 7, characterized in that: Utilizing the UE5 three-dimensional simulation engine, the expected linear velocity and the expected angular velocity are received through the Topic subscription, and the expected linear velocity and the expected angular velocity are transmitted to the dynamic model through the input and output interface in the Vortex dynamic model. The dynamic model solves the expected linear velocity and the expected angular velocity to adjust the posture of the wheeled robot in the three-dimensional simulation environment.

10. A multi-engine collaborative robot simulation method, characterized in that: The method comprises: S1. Implement the dynamics module using Vortex, and import the dynamics module as a plug-in into the three-dimensional simulation module to perform mechanical interaction operations with a static model in a preset three-dimensional simulation environment; S2. Use the preset graphical programming interface to create patrol tasks and obstacle avoidance tasks, jump to the corresponding level, load the Vortex dynamics model, and send task instructions corresponding to the patrol tasks and obstacle avoidance tasks; S3. Receive the task start command by subscription, start the perception algorithm program and the planning algorithm program in sequence, obtain sensor information from the three-dimensional simulation environment and the Vortex dynamic model in the UE5 three-dimensional simulation engine, set the driving target point of the wheel-crawler robot, and publish it through the Topic of the FastDDS protocol; subscribe to the sensor and the form target point according to the Topic of the preset protocol, use the perception algorithm program and the perception planning program to obtain and publish the expected linear velocity and the expected angular velocity of the wheel-crawler robot, and adjust the posture of the wheel-crawler robot in the three-dimensional simulation environment.

Images