Digital twin system construction method based on Isaac Sim and UDP protocol

Abstract

The invention discloses a digital twin system construction method based on Isaac Sim and UDP communication protocols, and belongs to the technical field of robot intelligent control. The system adopts a virtual-real collaborative framework to construct three core modules: a virtual robot model based on a URDF format is established through an Isaac Sim platform, and geometric structure parameters and kinematic characteristics of a real robot are accurately matched; a dynamic control strategy module is constructed by applying an RMPflow algorithm, an optimal motion path is generated through a synergistic effect of a target RMP and a collision avoidance RMP, and accurate target tracking and active obstacle avoidance of an end effector are realized; a bidirectional communication module supporting a UDP protocol is designed, a Socket communication link is constructed based on a local area network, a lightweight data packet is adopted to transmit a control signal and a robot state, and an ROS is used to realize control and state acquisition of a physical robot. The system comprises a virtual control real mode and a real control virtual mode, and supports feedback and synchronization of a virtual environment and a physical entity state. In conclusion, the invention provides the robot digital twin system construction method based on the Isaac Sim and the UDP communication protocol.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot intelligent control and relates to a method for constructing a digital twin system including virtual world modeling, data communication and robot control. Background Art

[0002] A digital twin is a virtual, digitized entity that corresponds to a real-world entity. Digital twins model and simulate real-world physical entities to enable real-time monitoring, prediction, and optimization. A digital twin typically consists of three components: a physical entity, a digital model, and data connectivity. Physical entities are real-world entities such as machines, equipment, and factories. The digital model, at the core of the digital twin, is a virtual model that corresponds to the physical entity. Data connectivity is the technology that connects the physical entity and the digital model, encompassing sensors, data acquisition, data storage, and data analysis.

[0003] UDP (User Datagram Protocol) is a datagram protocol that provides packet-switched computer communications across a network of interconnected computers. It assumes IP as the underlying protocol and operates at the transport layer according to the OSI model. UDP provides a process for applications to send messages to other programs with minimal protocol mechanisms. The protocol is transaction-oriented and does not guarantee delivery or duplicate protection. Applications that require ordered, reliable data streams should use the Transmission Control Protocol (TCP).

[0004] ROS (Robot Operating System) is an open-source meta-operating system for robots. It provides the services expected of an operating system, including hardware abstraction, low-level device control, implementation of common functions, inter-process messaging, and package management. ROS is a "robot framework" and is widely used in physical robot control.

[0005] Isaac Sim, a simulation tool developed by Nvidia for building digital twin scenarios, uses PhysX as its simulation engine and GPU rendering to create highly accurate and realistic virtual scenes. Isaac Sim provides a series of APIs for manipulating simulation scenes, allowing for the definition and modification of scenes and robots, making it suitable for building virtual simulation scenes within digital twins.

[0006] With the rapid development of technologies in the field of robots, digital twin technology has received more and more attention. It provides a new way to control robots. Through real-time interaction between the physical space and the virtual space, it can realize the status detection and real-time warning of the robot. It can be planned and executed in advance in the virtual space, which is also safer than in the physical space. In the existing digital twin construction technology, engines such as Unity 3D are mostly used to build virtual robot models. The necessary dynamic characteristics are lacking in the virtual world simulation, and it is difficult to reflect the complex dynamic characteristics in the physical space. The present invention proposes a digital twin system construction method based on Isaac Sim and UDP communication protocol, which involves virtual world modeling, data communication and robot control. Isaac Sim with a physical engine is used as a virtual world construction tool. The constructed system is more realistic and closer to the physical world. Summary of the Invention

[0007] To achieve the above functions, the technical solution adopted by the present invention is: a method for constructing a robot digital twin system based on Isaac Sim and UDP protocol. The invention process is as follows:

[0008] During the construction of the robot control system, a virtual-reality collaborative framework was used to achieve a closed-loop control system for the entire robot lifecycle. This system construction process consisted of four parts: virtual robot modeling based on Isaac Sim, reactive control strategy design based on RMPflow (Riemannian Motion Policy flow), virtual-reality communication link construction using the UDP protocol and ROS, and digital twin scenario construction.

[0009] Furthermore, the virtual robot modeling based on Isaac Sim is specifically as follows:

[0010] Use Isaac Sim to build a virtual world model, model the robot's geometric model and physical property location based on the collected and organized actual robot data, simulate the physical engine configuration in the virtual environment, and generate a digital twin that maps virtuality to reality;

[0011] Using Isaac Sim as the virtual world simulation engine, the URDF (Unified Robot Description Format) format is used to import the 3D model of the real robot, accurately restoring geometric characteristics such as the rotation range of each joint and the length of the connecting rod. The robot's physical characteristics such as the connecting rod weight and collision geometry are then set to ensure a unified description of the virtual model and the real robot, thus constructing a virtual world model of the robot.

[0012] Furthermore, in the virtual world model constructed based on Isaac Sim, the Isaac Sim simulation software is used as the virtual world simulation engine, and a robot virtual world model is constructed in Isaac Sim. The parameters are adjusted so that the geometric characteristics of the robot in the virtual world are the same as those of the real robot and the dynamic characteristics are similar.

[0013] Furthermore, the RMPflow-based reactive control strategy design specifically includes: adjusting the RMP strategy weight according to the robot characteristics, and generating the robot motion control instructions in the virtual world through the RMPflow algorithm;

[0014] The robot motion control strategy is generated through the RMPflow algorithm, the real-time motion trajectory is planned based on the model's geometric constraints, and the joint acceleration of the robot in the virtual environment is adjusted.

[0015] Furthermore, the RMPflow reactive control strategy is based on the Riemannian manifold geometry theory. According to the geometric and dynamic characteristics of the robot, the RMPflow strategy weight is adjusted to achieve dynamic obstacle avoidance and real-time control.

[0016] Furthermore, the virtual-real communication link construction combining the UDP protocol and ROS is specifically as follows:

[0017] The UDP protocol is used to realize data interaction between the virtual world and the physical world. The virtual world transmits control information to the physical robot with low latency. The physical robot status information is fed back to the simulation environment in real time. ROS is used to control and obtain the status of the physical robot.

[0018] The ROS topic and service mechanism is used to realize the status acquisition and control of the physical robot, WiFi is used to build a local area network communication between the physical and virtual worlds, and the UDP protocol is used for two-way transmission. The joint status of the physical robot, the robot control instructions in the virtual space and other data are packaged into lightweight data packets for transmission to realize virtual-real communication.

[0019] Furthermore, in the UDP protocol data transmission, bidirectional communication transmits the joint status of the physical robot and the joint control data of the virtual space robot, ensuring that the simulation is aligned with the physical world status.

[0020] Furthermore, the digital twin scene construction is specifically as follows:

[0021] Digital twin scenarios are divided into two modes: one is the virtual-control-real mode, where the robot path is planned in the virtual environment and the control flow is transmitted to the physical robot for execution after it is correct; the other is the real-control-virtual mode, where the physical robot sends status data packets at a fixed frequency, transmits the robot joint status to the virtual robot, and drives the virtual model to update synchronously.

[0022] The beneficial effects of the present invention compared with the prior art are:

[0023] The present invention uses the Isaac Sim platform as the virtual world simulation engine, making the simulation more realistic and closer to the physical characteristics of the real robot; it adopts a virtual-real communication method combining the UDP protocol with ROS to improve the efficiency of virtual-real communication while taking into account the simplicity of robot control.

[0024] This invention uses a virtual-real collaborative framework to construct three core modules: A virtual robot model based on the URDF format is established through the Isaac Sim platform, accurately matching the geometric structure parameters and kinematic characteristics of the real robot; a dynamic control strategy module is constructed using the RMPflow algorithm, which generates an optimal motion path through the synergy of the target Restricted Physical Model (RMP) and the collision avoidance Restricted Physical Model (RMP), enabling precise target tracking and active obstacle avoidance for the end effector; a bidirectional communication module supporting the UDP protocol is designed, a Socket communication link is constructed based on a local area network, and lightweight data packets are used to transmit control signals and robot status. ROS is used to realize the control and status acquisition of the physical robot. The system includes dual modes: virtual-real and real-real control of virtual, supporting feedback and synchronization between the virtual environment and the physical entity state. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of the robot control system of the present invention.

[0026] Figure 2 This is the collision volume envelope diagram in the robot virtual space of the present invention.

[0027] Figure 3 This is the digital twin system control UI interface of the present invention.

[0028] Figure 4 This is a demonstration effect diagram of the digital twin system of the present invention in virtual control real mode.

[0029] Figure 5 This is a demonstration effect diagram of the digital twin system of the present invention in real-control virtual mode. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the following examples are given to further illustrate the present invention in detail. It should be noted that the specific implementation described here is only used to explain the present invention and is not intended to limit the present invention.

[0031] The specific implementation process is divided into four steps, as follows: Figure 1 As shown:

[0032] The first step is to model the virtual robot.

[0033] 1. First, write the robot description file in URDF format according to the actual robot, including the robot geometric shape description Stl file;

[0034] 2. In Isaac Sim, import the 3D model of the physical robot using the URDF import tool;

[0035] 3. Use the material management tool in Isaac Sim to define the material properties of the robot link and environment to simulate the material of the physical robot, including material type, color, friction coefficient, stiffness and damping, etc.

[0036] 4. Adjust the mass and inertia of the robot's connecting rods to keep them consistent with the actual robot, and then adjust the stiffness and damping of the joints, as well as the angle range, speed range, and force limit of the joints.

[0037] The second step is to design the robot control strategy so that the robot end can move to the target position while avoiding obstacles in the space.

[0038] 1. Based on the robot joint information set in the first step, set whether the joints in the RMP are controllable, the upper limit of the joint speed, and the weight of each joint in the control;

[0039] 2. Design the robot envelope sphere. Take the coordinates of each joint in the URDF file as the origin, and use spheres with different radii and different coordinates relative to the origin to envelop each link, such as Figure 2 As shown;

[0040] 3. Adjust the weights of each RMP strategy and set the weight value of each strategy to 0. First, adjust the c-space_target_rmp / metric scalar in the joint space so that the robot can reach the specified joint angle position;

[0041] 4. Adjust the RMP policy weights in the task space by setting target_rmp / min_metric_alpha to 0 and target_rmp / metric_alpha_length_scale to a large value relative to the robot size (in meters); set target_rmp / max_metric_scalar to a large value relative to c-space_target_rmp / metric_scalar so that it dominates.

[0042] 5. Adjust target_rmp / accel_p_gain, target_rmp / accel_d_gain, and target_rmp / accel_norm_eps until the robot end can reach the target position well;

[0043] 6. Adjust the obstacle avoidance RMP strategy weights and enable obstacle avoidance RMP by setting collision rmp / metric scalar to a value equivalent to target_rmp / max_metric_scalar.

[0044] The third step is to design the communication strategy between the virtual and real worlds. The UDP protocol is used to achieve low-latency data transmission between the physical robot and the virtual space, and the ROS message mechanism is used to realize the transmission of robot control instructions and the acquisition of joint status.

[0045] 1. Use Wi-Fi to build a local area network. Connect the machine where the simulation engine Isaac Sim is located to the Wi-Fi where the physical robot is located. Use the ping tool to check the connectivity of the link.

[0046] 2. Use Socket to build persistent communication between the physical robot and the virtual space, where the physical robot is the server and the virtual space is the client. The IP address is configured according to the local area network (such as 192.168.1.x) and the port number is set to 5500.

[0047] 3. Use JSON format to encapsulate the message, including the control mode. If the virtual control real mode is set to 1, the transmitted message is the angle, velocity, and acceleration information of each joint in the virtual space, and no response is required. If the real control virtual mode is set to 2, the transmitted message is the "action" string, and the returned response includes the angle and velocity status information of each joint of the physical robot;

[0048] 4. Use the ROS publish-subscribe mechanism to build a node for the physical robot's joint status information, publish the robot status information at a fixed frequency, and use the ROS service call mechanism to build a physical robot control node. After receiving the control information from the virtual space, call the control service to control the movement of the robot's joints.

[0049] The fourth step is to build a digital twin scenario, including controlling the real with the virtual and controlling the virtual with the real.

[0050] 1. Use the Isaac Sim plug-in mechanism to build the digital twin system control UI, such as Figure 3 As shown, it includes loading the virtual world model, resetting the virtual world model, controlling the movement of the virtual robot, switching the control mode, adding obstacles and recording data;

[0051] 2. The process of using virtual control real mode is as follows: switch the control mode to Sim2Real, set the target point in Isaac Sim, use the RMP flow reactive control strategy to calculate the angle and speed of each joint, encapsulate it into a data packet, and send a request to the server where the physical robot is located in each analysis step of the simulation. No response is required to avoid reducing the simulation rendering frequency while waiting for the response; after receiving the request, the physical robot calls the robot's ROS control service to modify the physical robot joint status, the effect is as follows Figure 4 As shown;

[0052] The process of real-to-virtual control mode is as follows: switch the control mode to Real2Sim, the virtual robot sends a request to the server at the physical robot at a fixed frequency, and updates the status of the robot in the virtual space through the joint status information in the response; at the same time, the physical robot's ROS control service modifies the joint status of the physical robot, and publishes the robot status through the ROS / joint_states topic. After receiving the request from the virtual robot, subscribe to the robot status topic, obtain real-time status information, encapsulate it into a data packet, and transmit it to the virtual space in real time through the response. The effect is as follows Figure 5 shown.

[0053] There are many specific implementation methods of the present invention. The above is only a preferred implementation method of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principles of the present invention. These improvements should also be considered as the scope of protection of the present invention.

Claims

1. A method for constructing a digital twin system based on Isaac Sim and UDP protocol, characterized in that: The method includes: Virtual robot modeling based on Isaac Sim; Reactive control strategy design based on RMPflow (Riemannian Motion Policy flow); Combining UDP protocol with ROS to build a virtual-real communication link; Digital twin scenario construction.

2. A method for constructing a digital twin system based on Isaac Sim and UDP protocol according to claim 1, characterized in that: The virtual robot modeling based on Isaac Sim is specifically as follows: Use Isaac Sim to build a virtual world model, model the robot's geometric model and physical property location based on the collected and organized actual robot data, simulate the physical engine configuration in the virtual environment, and generate a digital twin that maps virtuality to reality; Using Isaac Sim as the virtual world simulation engine, the URDF (Unified Robot Description Format) format is used to import the 3D model of the real robot, accurately restore the rotation range of each joint, the length of the connecting rod and other geometric characteristics, and then set the physical characteristics of the robot; ensure that the virtual model and the real robot are described in a unified manner, and construct a virtual world model of the robot.

3. The method for constructing a digital twin system based on Isaac Sim and UDP protocol according to claim 2, characterized in that: In the virtual world model built based on Isaac Sim, Isaac Sim simulation software is used as the virtual world simulation engine, and a robot virtual world model is built in Isaac Sim. The parameters are adjusted to make the geometric characteristics of the robot in the virtual world the same as the real robot and the dynamic characteristics similar.

4. The method for constructing a digital twin system based on Isaac Sim and UDP protocol according to claim 1, characterized in that: The RMPflow-based reactive control strategy design specifically includes: adjusting the RMP strategy weight according to the robot characteristics, and generating the robot motion control instructions in the virtual world through the RMPflow algorithm; The robot motion control strategy is generated through the RMPflow algorithm, the real-time motion trajectory is planned based on the model's geometric constraints, and the joint acceleration of the robot in the virtual environment is adjusted.

5. The method for constructing a digital twin system based on Isaac Sim and UDP protocol according to claim 4, characterized in that: The RMPflow reactive control strategy is based on the Riemannian manifold geometry theory. According to the geometric and dynamic characteristics of the robot, the RMPflow strategy weight is adjusted to achieve dynamic obstacle avoidance and real-time control.

6. The method for constructing a digital twin system based on Isaac Sim and UDP protocol according to claim 1, characterized in that: The virtual-real communication link construction combining UDP protocol and ROS is specifically as follows: The UDP protocol is used to realize data interaction between the virtual world and the physical world. The virtual world transmits control information to the physical robot with low latency. The physical robot status information is fed back to the simulation environment in real time. ROS is used to control and obtain the status of the physical robot. The ROS topic and service mechanism is used to realize the status acquisition and control of the physical robot, WiFi is used to build a local area network communication between the physical and virtual worlds, and the UDP protocol is used for two-way transmission. The joint status of the physical robot, the robot control instructions in the virtual space, and the control mode data are packaged into lightweight data packets for transmission to realize virtual-real communication.

7. The method for constructing a digital twin system based on Isaac Sim and UDP protocol according to claim 6, characterized in that: In the UDP protocol data transmission, bidirectional communication transmits the joint status of the real robot and the joint control data of the virtual space robot, ensuring that the simulation is aligned with the physical world state.

8. The method for constructing a digital twin system based on Isaac Sim and UDP protocol according to claim 1, characterized in that: The digital twin scene construction is specifically as follows: Digital twin scenarios are divided into two modes. One is the virtual-control-real mode, in which the robot path is planned in a virtual environment, and after confirmation, the control flow is transmitted to the physical robot for execution. The other is the real-control-virtual mode, in which the physical robot sends status data packets at a fixed frequency, transmits the robot joint status to the virtual robot, and drives the virtual model to update synchronously.

Images