A mechanical arm virtual synchronous control system and method based on digital twinning

Abstract

The application discloses a mechanical arm virtual synchronous control system and method based on digital twinning, which comprises a physical mechanical arm, a relay control unit and a virtual mechanical arm, the virtual mechanical arm being a virtual mapping of the physical mechanical arm; the physical mechanical arm and the relay control unit perform physical communication, and the virtual mechanical arm and the relay control unit perform virtual communication; the relay control unit comprises a virtual mechanical arm class module, a virtual environment class module, a control algorithm module and a man-machine interaction control interface; the virtual mechanical arm class module is used for defining kinematics and dynamics models of the virtual mechanical arm; the virtual environment class module is used for providing function and algorithm libraries for function implementation; the virtual mechanical arm class module provides parameter sharing for the virtual environment class module, so as to realize visualization of motion mode data of the virtual mechanical arm; in the application, the virtual mechanical arm can be controlled through a user control interface or an external interface, and motion changes of the virtual mechanical arm are synchronized to the physical mechanical arm.

Description

Technical Field

[0001] The present invention relates to the field of digital twin technology, and more specifically to a digital twin-based robotic arm virtual synchronization control system and method. Background Art

[0002] Digital twins map virtual objects of physical entities in a digital way, and by representing and simulating the real behavior of physical entities in virtual bodies, they aim to enhance and expand the capabilities of physical entities through virtual-reality interactive feedback, data fusion analysis, and iterative decision optimization.

[0003] Currently, digital twins are widely used in simulation and condition monitoring. The data flow is mostly from the physical end to the virtual end, while there is little research on the reverse control of the virtual end to the physical end, and the two-line feature mapping capability is still insufficient.

[0004] Different from condition monitoring, the data source of virtual control is transformed from physical equipment to virtual body, and the nature of data is transformed from real working condition data to simulated data of the virtual body under virtual working conditions, that is, the data of the virtual body self-evolving after physical empowerment. The control from virtual end to physical end provides the possibility for the virtual body to remotely guide the work of physical equipment. It can perform virtual maintenance when the physical equipment fails and assign the local working mode to the entity, such as deploying the trained working trajectory of the virtual robotic arm to the physical robotic arm. Summary of the Invention

[0005] In view of this, the present invention provides a virtual synchronous control system and method of a robotic arm based on digital twins to solve the problems of insufficient research on the reverse control of the virtual robotic arm to the physical robotic arm and insufficient dual-line feature mapping capabilities.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention provides a virtual synchronous control system for a robotic arm based on digital twins, comprising:

[0008] A physical robotic arm, a relay control unit, and a virtual robotic arm, wherein the virtual robotic arm is a virtual mapping of the physical robotic arm; the physical robotic arm performs physical communication with the relay control unit, and the virtual robotic arm performs virtual communication with the relay control unit;

[0009] The relay control unit includes a virtual robotic arm module, a virtual environment module, a control algorithm module and a human-computer interaction control interface;

[0010] The virtual robotic arm module is used to define the kinematic and dynamic models of the virtual robotic arm;

[0011] The virtual environment class module is used to provide a function and algorithm library for function implementation;

[0012] The virtual mechanical arm type module provides parameter sharing for the virtual environment type module to realize visualization of virtual mechanical arm movement mode data;

[0013] The virtual environment type module provides virtual mechanical arm control data for the control algorithm module to the physical mechanical arm;

[0014] The control algorithm module adopts a sliding mode variable structure control algorithm to optimize the control data from the virtual mechanical arm to the physical mechanical arm in real time, and is used to solve the problem of tracking the movement trajectory of the virtual mechanical arm by the movement trajectory of the physical mechanical arm;

[0015] The human-computer interaction control interface is used for twin data integration, converts and escapes the format of the optimized control data, and generates control instructions for the physical mechanical arm.

[0016] Further, the kinematics and dynamics model of the virtual mechanical arm includes the current speed, current position, current torque of each joint of the mechanical arm, and target speed, target position and target torque;

[0017] The function and algorithm library of the function implementation includes the functions and algorithms of the client establishment of the virtual environment, the environment start, the parameter update, the virtual communication establishment and the object iteration.

[0018] Further, the physical mechanical arm is provided with a CAN, serial or network communication interface;

[0019] The physical mechanical arm and the relay control unit adopt TCP / IP communication.

[0020] Further, the relay control unit is created based on Qt, the virtual mechanical arm is constructed based on CoppeliaSim, CoppeliaSim provides a BlueZero or B0-based communication interface, and the communication between the virtual mechanical arm and the relay control unit established by Qt is realized.

[0021] In order to further optimize the above technical solution: the digital twin virtual environment is established by CoppeliaSim, and the movement trajectory of the virtual mechanical arm is planned through the virtual environment itself or an external interface of CoppeliaSim, for example, through a thread script of CoppeliaSim or a deep reinforcement learning platform to complete this work.

[0022] The application also provides a mechanical arm virtual synchronous control method based on digital twinning, applied to the mechanical arm virtual synchronous control system based on digital twinning, and comprising the following steps:

[0023] S1. Create a twin space, build a virtual environment and virtual robotic arm based on CoppeliaSim, and establish a relay control unit based on Qt;

[0024] S2. Initialize the configuration, initialize the physical robotic arm and the virtual robotic arm, and initialize the physical robotic arm, the virtual robotic arm, and each joint of the virtual robotic arm based on the virtual robotic arm class module constructed in the relay control unit;

[0025] S3, establishing a physical communication connection between the physical robotic arm and the relay control unit, and establishing a virtual communication connection between the virtual robotic arm and the relay control unit. When the physical robotic arm and the virtual robotic arm are respectively connected to the relay control unit, executing step S4, otherwise executing step S3 again;

[0026] S4, controlling each joint of the virtual robotic arm through kinematic forward solution, sampling virtual motion data of each joint of the virtual robotic arm, and obtaining real-time position data of each joint;

[0027] Solve the inverse kinematics of the virtual robotic arm, control the end of the robotic arm, perform motion planning for the virtual robotic arm, sample the virtual motion data, obtain the spatial pose data of the end of the robotic arm, add a thread script to the parent object of the virtual robotic arm, perform inverse kinematics solution, and convert the spatial pose data of the end of the robotic arm into real-time position data of each joint;

[0028] S5. Based on the robot sliding mode variable structure control algorithm in the relay control unit, the collected real-time position data of each joint of the virtual robotic arm are optimized in real time to generate corresponding control data of the real-time position of each joint;

[0029] S6. Based on the human-computer interaction interface constructed in the relay control unit, convert the real-time optimized control data of the real-time position of each joint into control instructions for the physical robotic arm;

[0030] S7. The relay control unit controls the motion of the physical robotic arm through control instructions.

[0031] Furthermore, the initialization configuration in step S2 specifically includes: when the system is running, unifying the coordinate origin to the base coordinates of the physical robotic arm and the virtual robotic arm, resetting the joints of the physical robotic arm and the virtual robotic arm, and keeping the origin of each joint of the virtual robotic arm consistent with the origin of each joint of the physical robotic arm, so that the initial posture of the virtual robotic arm is consistent with the initial posture of the physical robotic arm; and also performing joint initialization on the virtual robotic arm class in the relay control unit.

[0032] Furthermore, in step S4, the real-time position data of each joint and the spatial posture data of the end of the robotic arm are acquired and collected every 10 ms and recorded.

[0033] Furthermore, the relay control unit controls the motion of the physical robotic arm through control instructions, specifically including: the relay control unit sends a control instruction to the physical robotic arm every 20ms based on a timer, and exits a communication cycle each time a control instruction is sent.

[0034] It can be seen from the above technical solutions that, compared with the prior art, the technical effects of the present invention are:

[0035] 1. The CoppeliaSim used in the present invention has the conditions for building a multi-physics-driven digital twin virtual environment. Since the virtual robotic arm cannot communicate directly with the physical robotic arm; therefore, the present invention constructs a relay control unit based on Qt to realize communication between the physical robotic arm and the virtual robotic arm. The physically empowered virtual robotic arm can perform motion planning according to the dynamic model of the physical robotic arm with high fidelity, has high reliability, and is more conducive to the integration and packaging of control algorithms.

[0036] 2. The virtual robot arm can be controlled by directly dragging the virtual robot arm with the mouse or by program instructions. The drag control method is quick and convenient. Users can operate virtual objects with just the mouse, and can visually judge the operation effect and safety. The program control method fully utilizes the advantages of CoppeliaSim script driver. By building a user control interface or external interface to control the virtual robot arm, the motion state of the virtual robot arm is synchronized with the physical robot arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] The following is a further description of the virtual synchronization control system and method of the robotic arm based on digital twins of the present invention with reference to the accompanying drawings;

[0039] Figure 1 This is a structural diagram of the virtual synchronous control system of the robotic arm based on digital twins of the present invention;

[0040] Figure 2 It is a flow chart of the virtual synchronous control method of the robotic arm based on digital twin of the present invention. DETAILED DESCRIPTION

[0041] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0042] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the embodiment of the present invention discloses a virtual synchronous control system of a robotic arm based on digital twin, including:

[0044] A physical robotic arm, a relay control unit, and a virtual robotic arm, wherein the virtual robotic arm is a virtual mapping of the physical robotic arm; the physical robotic arm performs physical communication with the relay control unit, and the virtual robotic arm performs virtual communication with the relay control unit;

[0045] The relay control unit includes a virtual robotic arm module, a virtual environment module, a control algorithm module and a human-computer interaction control interface;

[0046] The virtual robotic arm module is used to define the kinematic and dynamic models of the virtual robotic arm;

[0047] The virtual environment class module is used to provide a function and algorithm library for function implementation;

[0048] The virtual robotic arm module provides parameter sharing for the virtual environment module to realize visualization of motion data of the virtual robotic arm;

[0049] The virtual environment module provides the control algorithm module with control data of the virtual robotic arm on the physical robotic arm;

[0050] The control algorithm module adopts a sliding mode variable structure control algorithm to optimize the control data from the virtual manipulator to the physical manipulator in real time, so as to solve the problem of tracking the motion trajectory of the physical manipulator to the motion trajectory of the virtual manipulator;

[0051] The human-computer interaction control interface is used for twin data integration, converting and translating the format of motion-optimized control data to generate control instructions for the physical robotic arm.

[0052] In order to optimize the above technical solution, since the communication protocols required by the physical robotic arms themselves are different, data that has not been converted and escaped cannot be directly used by the physical robotic arms. Therefore, the data conversion and escape are completed in this step.

[0053] The kinematic and dynamic models of the virtual robotic arm include the current speed, current position, current torque, target speed, target position and target torque of each joint of the robotic arm;

[0054] The function and algorithm library for implementing the functions include functions and algorithms for establishing a virtual environment client, starting the environment, updating parameters, establishing virtual communication, and iterating objects.

[0055] The physical robotic arm is provided with a CAN, serial port or network communication interface;

[0056] TCP / IP communication is adopted between the physical robotic arm and the relay control unit.

[0057] The relay control unit is created based on Qt, and the virtual robotic arm is constructed based on CoppeliaSim. CoppeliaSim provides a BlueZero or B0-based communication interface for realizing communication between the virtual robotic arm and the relay control unit established by Qt.

[0058] To further optimize the above technical solution, twin communication is primarily established around the physical end, relay control unit, and virtual end. The physical end communicates with the relay control unit physically, using a network, serial port, or CAN communication method, with the physical end acting as a server and the relay control unit acting as a client. The virtual end communicates virtually with the relay control unit. The virtual end is built using CoppeliaSim, which provides BlueZero and B0-based communication interfaces for communication between the virtual end and the relay control unit established by Qt.

[0059] It should be noted that Qt is a cross-platform C++ development library, mainly used to develop graphical user interface (GUI) programs; CoppeliaSim is a lightweight robot simulation simulator; BlueZero is a cross-platform middleware that interconnects multiple processes or multiple threads and transmits messages according to loosely coupled distributed communication similar to ROS (Robot Operating System); B0-Based is based on the BlueZero middleware and its interface plug-in with CoppeliaSim.

[0060] like Figure 2 As shown, the present invention also provides a digital twin-based virtual synchronous control method for a robotic arm, which is applied to the above-mentioned digital twin-based virtual synchronous control system for a robotic arm, comprising the following steps:

[0061] S1. Create a twin space, build a virtual environment and virtual robotic arm based on CoppeliaSim, and establish a relay control unit based on Qt;

[0062] S2. Initialize the configuration, initialize the physical robotic arm and the virtual robotic arm, and initialize the physical robotic arm, the virtual robotic arm, and each joint of the virtual robotic arm based on the virtual robotic arm class module constructed in the relay control unit;

[0063] S3, establishing a physical communication connection between the physical robotic arm and the relay control unit, and establishing a virtual communication connection between the virtual robotic arm and the relay control unit. When the physical robotic arm and the virtual robotic arm are respectively connected to the relay control unit, executing step S4, otherwise executing step S3 again;

[0064] S4, controlling each joint of the virtual robotic arm through kinematic forward solution, sampling virtual motion data of each joint of the virtual robotic arm, and obtaining real-time position data of each joint;

[0065] Solve the inverse kinematics of the virtual robotic arm, control the end of the robotic arm, perform motion planning for the virtual robotic arm, sample the virtual motion data, obtain the spatial pose data of the end of the robotic arm, add a thread script to the parent object of the virtual robotic arm, perform inverse kinematics solution, and convert the spatial pose data of the end of the robotic arm into real-time position data of each joint;

[0066] S5. Based on the robot sliding mode variable structure control algorithm in the relay control unit, the collected real-time position data of each joint of the virtual robotic arm are optimized in real time to generate corresponding control data of the real-time position of each joint;

[0067] S6. Based on the human-computer interaction interface constructed in the relay control unit, convert the real-time optimized control data of the real-time position of each joint into control instructions for the physical robotic arm;

[0068] S7. The relay control unit controls the motion of the physical robotic arm through control instructions.

[0069] In order to optimize the above technical solution, after the data of each joint of the virtual robotic arm and the end of the virtual robotic arm are converted and translated into control instructions for the physical robotic arm, in order to prevent packet loss due to blockage in data transmission, the relay control unit sends a control instruction to the physical robotic arm every 20ms based on a timer. Each time a control instruction is sent, a communication loop is exited, effectively avoiding the situation where the robotic arm only executes the first instruction.

[0070] The initialization configuration in step S2 specifically includes: when the system is running, unifying the coordinate origin to the base coordinates of the physical robotic arm and the virtual robotic arm, resetting the joints of the physical robotic arm and the virtual robotic arm, keeping the origin of each joint of the virtual robotic arm consistent with the origin of each joint of the physical robotic arm, so that the initial posture of the virtual robotic arm is consistent with the initial posture of the physical robotic arm; and also performing joint initialization on the virtual robotic arm class in the relay control unit.

[0071] In step S4, the real-time position data of each joint and the spatial posture data of the end of the robotic arm are acquired and recorded every 10 ms.

[0072] The relay control unit controls the motion of the physical robotic arm through control instructions, specifically including: the relay control unit sends a control instruction to the physical robotic arm every 20ms based on a timer, and jumps out of a communication cycle each time a control instruction is sent.

[0073] In order to optimize the above technical solutions, the virtual environment module includes functions and algorithms for virtual environment client establishment, environment startup, parameter update, virtual communication establishment, and object iteration. These iterative algorithms are independent of each other and are all implemented under the virtual environment module. The virtual environment module can be regarded as the underlying technical support, which requires the help of parameter sharing from the virtual robotic arm module to complete the conversion from digital to virtual robotic arm and physical robotic arm.

[0074] In order to optimize the above technical solution, the control algorithm adopts the robot sliding mode variable structure control algorithm, which does not require an accurate mathematical model of the controlled object, but only requires the parameter variation range in the model.

[0075] To optimize this technical solution, the primary output of the virtual robot class in the relay control unit is the target joint velocity: joint.out[i] = joint.tar_vel[i], where joint.out[i] is the actual joint velocity and joint.tar_vel[i] is the target joint velocity. The virtual robot's joint velocity direction is a Boolean variable; 1 and 0 represent forward and reverse directions, respectively. This virtual robot's target velocity data is monitored repeatedly to provide real-time virtual data when the interface connects to a physical device.

[0076] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A virtual synchronous control system for a robotic arm based on digital twins, characterized in that: include: A physical robotic arm, a relay control unit, and a virtual robotic arm, wherein the virtual robotic arm is a virtual mapping of the physical robotic arm; the physical robotic arm performs physical communication with the relay control unit, and the virtual robotic arm performs virtual communication with the relay control unit; The relay control unit includes a virtual robotic arm module, a virtual environment module, a control algorithm module and a human-computer interaction control interface; The virtual robotic arm module is used to define the kinematic and dynamic models of the virtual robotic arm; The virtual environment class module is used to provide a function and algorithm library for function implementation; The virtual robotic arm module provides parameter sharing for the virtual environment module to realize visualization of motion data of the virtual robotic arm; The virtual environment module provides the control algorithm module with control data of the virtual robotic arm on the physical robotic arm; The control algorithm module adopts a sliding mode variable structure control algorithm to optimize the control data from the virtual manipulator to the physical manipulator in real time, so as to solve the problem of tracking the motion trajectory of the physical manipulator to the motion trajectory of the virtual manipulator; The human-computer interaction control interface is used for twin data integration, converting and translating the format of motion-optimized control data to generate control instructions for the physical robotic arm.

2. The digital twin-based robotic arm virtual synchronization control system according to claim 1, characterized in that: The kinematic and dynamic models of the virtual robotic arm include the current speed, current position, current torque, target speed, target position and target torque of each joint of the robotic arm; The function and algorithm library for implementing the functions include functions and algorithms for establishing a virtual environment client, starting the environment, updating parameters, establishing virtual communication, and iterating objects.

3. The digital twin-based robotic arm virtual synchronization control system according to claim 1, characterized in that: The physical robotic arm is provided with a CAN, serial port or network communication interface; TCP / IP communication is adopted between the physical robotic arm and the relay control unit.

4. The digital twin-based robotic arm virtual synchronization control system according to claim 1, characterized in that: The relay control unit is created based on Qt, and the virtual robotic arm is constructed based on CoppeliaSim. CoppeliaSim provides a BlueZero or B0-based communication interface for realizing communication between the virtual robotic arm and the relay control unit established by Qt.

5. A method for virtual synchronous control of a robotic arm based on digital twins, applied to the virtual synchronous control system of a robotic arm based on digital twins according to any one of claims 1 to 4, characterized in that: The following steps are involved: S1. Create a twin space, build a virtual environment and virtual robotic arm based on CoppeliaSim, and establish a relay control unit based on Qt; S2. Initialize the configuration, initialize the physical robotic arm and the virtual robotic arm, and initialize the joints of the physical robotic arm and the virtual robotic arm based on the virtual robotic arm module constructed in the relay control unit; S3, establishing a physical communication connection between the physical robotic arm and the relay control unit, and establishing a virtual communication connection between the virtual robotic arm and the relay control unit. When the physical robotic arm and the virtual robotic arm are respectively connected to the relay control unit, executing step S4; Otherwise, re-execute step S3; S4, controlling each joint of the virtual robotic arm through kinematic forward solution, sampling virtual motion data of each joint of the virtual robotic arm, and obtaining real-time position data of each joint; Solve the inverse kinematics of the virtual robotic arm, control the end of the robotic arm, perform motion planning for the virtual robotic arm, sample the virtual motion data, obtain the spatial pose data of the end of the robotic arm, add a thread script to the parent object of the virtual robotic arm, perform inverse kinematics solution, and convert the spatial pose data of the end of the robotic arm into real-time position data of each joint; S5. Based on the robot sliding mode variable structure control algorithm in the relay control unit, the collected real-time position data of each joint of the virtual robotic arm are optimized in real time to generate corresponding control data of the real-time position of each joint; S6. Based on the human-computer interaction interface constructed in the relay control unit, convert the real-time optimized control data of the real-time position of each joint into control instructions for the physical robotic arm; S7. The relay control unit controls the motion of the physical robotic arm through control instructions.

6. The method for virtual synchronous control of a robotic arm based on digital twinning according to claim 5, characterized in that: The initialization configuration in step S2 specifically includes: when the system is running, unifying the coordinate origin to the base coordinates of the physical robotic arm and the virtual robotic arm, resetting the joints of the physical robotic arm and the virtual robotic arm, keeping the origin of each joint of the virtual robotic arm consistent with the origin of each joint of the physical robotic arm, so that the initial posture of the virtual robotic arm is consistent with the initial posture of the physical robotic arm; and also performing joint initialization on the virtual robotic arm class in the relay control unit.

7. The method for virtual synchronous control of a robotic arm based on digital twin according to claim 5, characterized in that: In step S4, the real-time position data of each joint and the spatial posture data of the end of the robotic arm are acquired and recorded every 10 ms.

8. The method for virtual synchronous control of a robotic arm based on digital twin according to claim 5, characterized in that: The relay control unit controls the motion of the physical robotic arm through control instructions, specifically including: the relay control unit sends a control instruction to the physical robotic arm every 20ms based on a timer, and jumps out of a communication cycle each time a control instruction is sent.

Images