A robot teleoperation method and system based on digital twin

Abstract

The present invention discloses a robot remote operation method and system based on digital twin. By establishing a digital twin model, a robot digital operation scene is constructed according to the actual working environment of the robot, and the movement of the robot and the robot geometric model is controlled simultaneously. The movement trajectory of the robot is predicted according to the joint kinematic model, and the robot is visually monitored through the real-time dynamics of the robot entity and the real-time characteristic parameters of the digital twin model to realize remote operation of the robot. More intuitive and rapid visual feedback can be given to the remote operator. The digital twin model and the physical robot simultaneously receive control instructions from the operator, and the operator can obtain the effect of the control instruction by observing the digital twin model.

Description

Technical Field

[0001] The present invention relates to the technical field of remote operation, and in particular to a robot remote operation method and system based on digital twins. Background Art

[0002] With the progress of society and the development of science and technology, the application field of robots is constantly expanding. People require robots to have higher intelligence, larger operating space, stronger operating flexibility, and the ability to operate in extreme environments. Limited by the current scientific level, robots cannot complete all complex tasks autonomously in harsh environments and still require human operation. Teleoperation technology has emerged. Dangerous environments or the execution of difficult and complex tasks no longer rely entirely on manual field operations. Operators only need to issue control instructions at the remote end, combined with the powerful functions of robots, to successfully complete various difficult tasks. Telepresence technology is human-centered. Through various sensor devices of remote robots, the interactive information (including visual, force telepresence, etc.) between the remote robot and the environment is fed back to the operator in real time, allowing the operator to make reasonable operations based on the feedback. The core concept of telepresence is to provide human operators with immersive sensory assistance on site, thereby significantly improving the efficiency of teleoperation systems in completing complex tasks.

[0003] In teleoperation technology, improving telepresence has always been a research focus. Only when the operator has a clear perception of the status of the slave robot and the surrounding environment can he output correct operating instructions to the slave robot. Force and visual telepresence are the focus of telepresence technology research. Force telepresence monitors the force of interaction with the environment by placing a force sensor at the end of the slave robot, and then transmits it to the master robot through the communication channel. The operator senses the force and performs fine control of the slave robot. Visual telepresence refers to the environmental information of the slave robot's work, which is monitored and collected by the slave robot's camera, lidar and other environmental perception devices, and transmitted to the master through the communication channel. The operator can construct the slave robot's working environment through the returned images and point cloud information, clearly understand the surrounding information, and thus control the slave robot to complete complex tasks.

[0004] Digital twin refers to the process and method of using digital technology to describe and model the characteristics, behavior, formation process and performance of physical objects, also known as digital twin technology. Digital twin refers to a virtual model that is completely corresponding and consistent with the physical entity in the real world, and can simulate its own behavior and performance in the real environment in real time, also known as a digital twin model.

[0005] Existing remote operation technology provides the operator with a sense of visual presence through remote real-time images, but it has a certain communication delay that cannot be eliminated. At the same time, if the status of the remote robot itself needs to be monitored, a corresponding camera needs to be placed on it, which increases the amount of information transmitted and the communication delay, which is not conducive to real-time control of the robot. Summary of the invention

[0006] In view of this, the present invention provides a robot remote operation method and system based on digital twins, which can monitor and control the status of the robot more intuitively and quickly.

[0007] The specific technical solution adopted in the present invention is as follows:

[0008] A robot teleoperation method based on digital twins, comprising:

[0009] Establish a digital twin model, including robot geometry model, joint kinematics model and physical model;

[0010] Constructing a digital operation scene of the robot according to the actual operation scene of the robot, and controlling the movement of the robot and the geometric model of the robot at the same time;

[0011] The motion trajectory of the robot is predicted according to the joint kinematic model, and the robot is visually monitored through the real-time dynamics of the robot and the real-time characteristic parameters of the digital twin model to achieve remote operation of the robot.

[0012] Furthermore, it also includes: the robot feeds back characteristic parameters to the digital twin model, determines the error between the characteristic parameters and the real-time characteristic parameters, and if the error is less than a preset error threshold, continues to remotely operate the robot; if the error is greater than the preset error threshold, performs error detection on the digital twin model, optimizes the error, and then remotely operates the robot.

[0013] Furthermore, the error detection on the digital twin model and the error optimization include:

[0014] Performing prediction error detection on the joint kinematics model, and optimizing and updating parameters of the joint kinematics model;

[0015] Fault detection is performed on the robot, the robot geometric model, and the physical model, and corresponding fault repair is performed according to the fault detection result.

[0016] Furthermore, the characteristic parameters and / or the real-time characteristic parameters at least include motion characteristic parameters and structural characteristic parameters.

[0017] Furthermore, after predicting the motion trajectory of the robot according to the joint kinematic model, the method further includes:

[0018] Motion simulation is performed in the digital operation scene according to the motion trajectory, and control instructions for adjusting the motion of the robot are formulated according to the motion simulation situation.

[0019] Furthermore, the robot geometric model is a pure shape model, and joints are set at corresponding positions, wherein the pure shape includes a cube, a cylinder and a sphere.

[0020] Furthermore, the joint kinematics model is an autoregressive state experience ARX model.

[0021] A robot teleoperation system based on digital twin, comprising:

[0022] Digital twin module, used to build digital twin model, including robot geometry model, joint kinematic model and physical model;

[0023] A motion control module, used to construct a digital operation scene of the robot according to the actual operation scene of the robot, and to control the movement of the robot and the geometric model of the robot;

[0024] A visual monitoring module is used to predict the motion trajectory of the robot based on the joint kinematic model, and to visually monitor the robot through the real-time dynamics of the robot and the real-time characteristic parameters of the digital twin model to achieve remote operation of the robot.

[0025] Furthermore, it also includes: an error optimization module, which is used to receive feedback characteristic parameters of the robot, determine the error between the characteristic parameters and the real-time characteristic parameters, and if the error is less than a preset error threshold, continue to remotely operate the robot; if the error is greater than the preset error threshold, perform error detection on the digital twin model, optimize the error, and then remotely operate the robot.

[0026] Furthermore, it also includes: a motion simulation module, which is used to perform motion simulation in the digital operation scene according to the motion trajectory, and formulate control instructions for adjusting the robot movement according to the motion simulation situation.

[0027] Beneficial effects:

[0028] (1) The present invention provides a robot remote operation method based on digital twins. By establishing a digital twin model, a digital operation scene of the robot is constructed according to the actual environment, and the movement of the robot and the robot geometric model are controlled simultaneously. The movement trajectory of the robot is predicted according to the joint kinematic model, and the robot is visually monitored through the real-time dynamics of the robot entity and the real-time characteristic parameters of the digital twin model to realize remote operation of the robot. More intuitive and rapid visual feedback can be given to the remote operator. The digital twin model and the physical robot simultaneously receive control instructions from the operator, and the operator can obtain the effect of the control instruction by observing the digital twin model.

[0029] (2) Based on the data fed back by the actual robot, the error of the digital twin model is compared and processed according to different situations. It is determined whether it is a fault problem or an error in the joint kinematic model, and corresponding adjustments and optimizations are made to ensure that the digital twin model is close to the physical entity of the robot. The joint kinematic model can be updated in real time. When receiving the same control instructions, the error between the motion trajectory of the digital twin model and the actual robot is within a certain range.

[0030] (3) Perform motion simulation in a digital operation scene based on the motion trajectory, and formulate control instructions for adjusting the robot's motion based on the motion simulation. The model control instructions can be pre-rehearsed and simulated and verified in the digital scene in advance to determine the rationality of the control instructions and decide whether to issue them to the remote robot, so as to better remotely operate the robot.

[0031] (4) The robot's geometric model is a pure shape model with stable shape and fast response.

[0032] (5) The joint kinematics model is an autoregressive state experience ARX model. The model construction is simpler and more efficient. It does not require the understanding of the internal structure of the joint. Each joint is regarded as a black box system. The joint kinematics can be identified by collecting enough sample data. The running status of the physical robot can be analyzed in a timely manner. The joint torque and the stress of the connecting rod can be analyzed without additional sensors, which is convenient for understanding and adjusting the mechanical structure of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of a robot teleoperation method based on digital twin according to an embodiment of the present invention;

[0034] Figure 2 is a structural block diagram of a robot teleoperation system based on digital twin according to an embodiment of the present invention;

[0035] Figure 3 It is a principle framework diagram of robot teleoperation according to the real-time example of the scene of the present invention;

[0036] Figure 4 is a flow chart of a robot teleoperation method according to a scenario embodiment of the present invention;

[0037] Figure 5 2 is a schematic diagram of the principle of a robot teleoperation method according to a scenario embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention provides a robot remote operation method and system based on digital twin. By establishing a digital twin model, a robot digital operation scene is constructed according to the actual environment of the robot, and the movement of the robot geometric model in the robot digital twin model is controlled simultaneously. The movement trajectory of the robot is predicted according to the joint kinematic model, and the robot is visually monitored through the real-time dynamics of the robot entity and the real-time characteristic parameters of the digital twin model to realize remote operation of the robot. More intuitive and rapid visual feedback can be given to the remote operator. The digital twin model and the physical robot simultaneously receive control instructions from the operator, and the operator can obtain the effect of the control instruction by observing the digital twin model.

[0039] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0040] The embodiment of the present invention provides a robot teleoperation method based on digital twin, Figure 1 is a flow chart of a robot teleoperation method based on digital twins according to an embodiment of the present invention, such as Figure 1 As shown, the process mainly includes the following steps:

[0041] Step 1: Establish a digital twin model, including robot geometry model, joint kinematics model and physical model;

[0042] In a specific embodiment, the robot geometric model is a pure shape model, and joints are set at corresponding positions, wherein the pure shape includes a cube, a cylinder and a sphere.

[0043] In one embodiment, the joint kinematics model is an autoregressive eXogenous (ARX) model.

[0044] Step 2: Build a digital operation scene of the robot according to the actual working environment of the robot, and control the movement of the robot and the robot geometric model at the same time;

[0045] Step 3: Predict the robot's motion trajectory based on the joint kinematic model, and visually monitor the robot through the robot's real-time dynamics and the real-time characteristic parameters of the digital twin model to achieve remote operation of the robot;

[0046] Step 4: The robot feeds back characteristic parameters to the digital twin model, determines the error between the characteristic parameters and the real-time characteristic parameters, and if the error is less than the preset error threshold, continues to remotely operate the robot; if the error is greater than the preset error threshold, performs error detection on the digital twin model, optimizes the error, and then remotely operates the robot.

[0047] In a specific embodiment, error detection is performed on the digital twin model and the error is optimized, including: predictive error detection is performed on the joint kinematics model, and the parameters of the joint kinematics model are optimized and updated; fault detection is performed on the robot and the robot geometric model and physical model, and corresponding fault repair is performed according to the fault detection results.

[0048] In a specific embodiment, the characteristic parameters and / or real-time characteristic parameters include at least motion characteristic parameters and structural characteristic parameters. The motion characteristic parameters and structural characteristic parameters are used to characterize the motion state and structural information of the robot, respectively. The motion characteristic parameters may include speed, distance, time, angle, etc., which are not specifically limited here. The structural characteristic parameters may include the number of joints, joint angles, distances between joints, etc., which are not specifically limited here.

[0049] In a specific embodiment, motion simulation can also be performed in a digital operation scene according to the motion trajectory, and control instructions for adjusting the robot motion can be formulated according to the motion simulation situation.

[0050] In an embodiment of the present invention, a robot teleoperation system based on digital twin is also provided. Figure 2 is a structural block diagram of a robot teleoperation system based on digital twin according to an embodiment of the present invention, such as Figure 2 As shown, the teleoperation system includes:

[0051] The digital twin module is used to establish a digital twin model, including the robot geometry model, joint kinematic model and physical model.

[0052] The motion control module is used to build a digital operation scene of the robot according to the robot's working environment, and to control the movement of the robot and the robot's geometric model.

[0053] The visual monitoring module is used to predict the robot's motion trajectory based on the joint kinematic model, and to perform visual monitoring of the robot through the robot's real-time dynamics and real-time characteristic parameters of the digital twin model to achieve remote operation of the robot.

[0054] The error optimization module is used to receive the feedback characteristic parameters of the robot, determine the error between the characteristic parameters and the real-time characteristic parameters, and continue to remotely operate the robot if the error is less than the preset error threshold; if the error is greater than the preset error threshold, perform error detection on the digital twin model, and remotely operate the robot after optimizing the error.

[0055] The motion simulation module is used to perform motion simulation in a digital operation scene according to the motion trajectory, and formulate control instructions for adjusting the robot motion according to the motion simulation situation.

[0056] The specific operation process in the above-mentioned specific modules can refer to the embodiment of the above-mentioned remote operation method, which will not be repeated here.

[0057] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is described below in conjunction with specific scenario embodiments.

[0058] Scenario Example 1

[0059] Figure 3 : is a principle framework diagram of robot teleoperation according to the real-time example of the scene of the present invention, such as Figure 3 As shown, where f h is the control force given by the operator to the hand controller, V m The hand controller converts the thrust into the corresponding speed control signal, V sd is the expected speed of the slave end, V s is the actual speed, f e is the environmental feedback force, f s To provide feedback force from the end (which may include environmental force and its own output torque), a digital twin is pre-built. The operator sends instructions to the digital twin and the mobile robot through the hand controller (such as the movement speed V m ), where the command is transmitted to the mobile robot through the communication channel. The mobile robot moves according to the environment and returns force information to the digital twin. The digital twin can reproduce the actual state of the mobile robot based on the force information and other data returned by the mobile robot, and can also save the data information as data support for the historical database and return the actual remote image to the operator.

[0060] Only by building an accurate digital twin can we ensure the feasibility of controlling the actual robot. The digital twin should include three aspects: 1. A physical model built with the help of simulation software based on the three-dimensional model, with its corresponding physical properties, including appearance, mass, center of mass, inertia matrix, collision volume, etc. 2. A joint kinematic model that conforms to reality and can be updated in real time, ensuring that the motion trajectory of the digital twin robot is within a certain range of error with the actual robot when receiving the same control instructions. 3. Based on the environmental information fed back by the environmental perception device, the corresponding virtual scene is built in the simulation software.

[0061] The more mainstream simulations for building physical models include V-rep, Webots, Gazebo, etc. Taking V-rep as an example, the physical model building process is briefly explained. 1. Get the geometric model of the physical robot in the solidworks software. Generally, there will be a corresponding geometric model in the early stage of robot design. 2. Simplify the model to a certain extent and delete unimportant small parts such as nuts and screws. 3. Import the three-dimensional model into V-rep in STL file format to get the visual model. 4. Build dynamic shapes for the imported model, and build the model into a dynamic shape with pure shapes (cubes, cylinders and spheres) as the visual model. Pure shapes are stable and respond quickly. 5. Set the joints in the corresponding positions (rotation, translation, passive, etc.). 6. Arrange the joints, visual shapes, and dynamic shapes reasonably into a relationship tree from top to bottom. The visual shapes and dynamic shapes are bound, and the parts are connected through joints.

[0062] For the robot joint kinematics, an ARX model is constructed to predict the motion trajectory. The construction of the ARX model is mainly to ensure that the motion effect of the digital twin in response to the control command can be close to the physical entity. Since the ARX model does not need to understand the internal structure of the joint, each joint is regarded as a black box system, and sufficient sample data can be collected to realize the identification of joint kinematics. Each joint is analyzed, and its ARX model is as follows: where u(k) is the input, y(k) is the output, and n a and n b is the polynomial order, e(k) is the error, a is the coefficient corresponding to the output term, b is the coefficient corresponding to the input term, and k is the time series of the discrete system.

[0063]

[0064] Solve for model parameters using the least squares method:

[0065] y(k)=φ T (k)θ+e(k)

[0066] Among them, φ T (k) is the sample set, and θ is the coefficient set to be solved.

[0067]

[0068] The least squares estimate of the coefficients is obtained as follows:

[0069]

[0070] After obtaining the motion model of the joint, it is necessary to use physical simulation software to establish the geometric model and physical model of the digital twin. It is more common to build the geometric model and physical model in the simulation software according to the appearance and physical characteristics of the actual robot, and combine the ARX model to predict the motion trajectory of the slave robot. The operator can directly observe the control effect of the digital twin to control the slave robot. At the same time, the surrounding environment at the remote end can also be constructed as a digital scene in the digital twin through equipment such as lidar and infrared cameras.

[0071] Figure 4 is a flow chart of a robot teleoperation method according to a scenario embodiment of the present invention, Figure 5 Schematic diagram of the principle of the robot teleoperation method according to the scenario embodiment of the present invention. Figure 4 and Figure 5 As shown in the figure, the operator can operate the remote robot based on the movement of the digital twin model and the real scene fed back, bringing a good operating experience. At the same time, the data comparison between the digital twin and the physical entity can also perform certain fault diagnosis on the physical entity. The identified ARX model is continuously updated during the control process to ensure the same control effect as the physical entity operation.

[0072] Among them, the slave robot, that is, the actual mobile robot and the digital twin, receive the control instructions issued by the operator through communication delay and perform synchronous movement. If it is judged that the operating status is normal, it will continue to start the movement of the next control cycle according to the control instructions. If the operating status is abnormal, it is necessary to judge the cause of the abnormality, including whether the parameters of the ARX model need to be updated or whether there is a fault.

[0073] If it is confirmed that the parameters of the ARX model need to be updated, the Particle Swarm Optimization (PSO) algorithm can be used to optimize the parameters of the ARX model, with the error between the model and the historical data as the cost function. If the error detection finds that it is a fault, such as a fault of the robot itself, it is necessary to combine the system knowledge base of the robot operation system, that is, the historical database, and the current operation data to perform fault diagnosis, determine the fault location, and stop the operation.

[0074] In this scenario embodiment, each joint of the remotely operated robotic arm is regarded as a black box model, and the ARX model is used to perform system identification on each joint. At the same time, a digital twin of the remote robot is constructed and visualized in the software, and the control signal is sent to the digital twin after a communication delay. The operator can control the remote robot according to the virtual scene, which enhances the visual presence of the remote operation system and achieves synchronization between the digital twin and the physical robot.

[0075] In this scenario, the data fed back by the actual robot is compared with the error of the digital twin operation to process the situation separately, and determine whether it is a fault problem or an ARX model estimation error. If it is a fault, fault diagnosis is performed based on the knowledge base. If it is a prediction error, the ARX model parameters are updated using an optimization algorithm based on historical operation data to ensure that the digital twin is close to the physical entity.

[0076] The present invention has the following advantages:

[0077] 1. Use digital twin technology to simulate and predict the physical robot, giving the remote operator more intuitive and rapid visual feedback. The digital twin and the physical entity receive control instructions from the operator at the same time, and the operator can obtain the effect of the control instruction by observing the digital twin.

[0078] 2. The constructed digital twin can timely analyze the operating status of the physical entity. It does not require additional sensors to analyze the joint torque, the stress on the connecting rod, etc., and the digital twin will save the analysis data for subsequent updates to the mechanical structure.

[0079] 3. For model control instructions, a pre-rehearsal method can be adopted to conduct simulation verification in a digital scene in advance to determine the rationality of the control instructions and decide whether to issue them to the remote robot.

[0080] In summary, the present invention provides a robot remote operation method and system based on digital twins, which utilizes digital twin technology to construct a digital twin of a remote robot, and realizes synchronous control of the digital twin and the slave robot by giving the same control instructions. At the same time, the established digital twin model is timely updated with parameters to ensure that the motion trajectory error with the physical entity is controlled within an acceptable range.

[0081] The above specific embodiments only describe the design principle of the present invention. The shapes and names of the components in the description may be different and are not limited. Therefore, those skilled in the art in the field of the present invention may modify or replace the technical solutions recorded in the above embodiments; and these modifications and replacements do not deviate from the creative purpose and technical solutions of the present invention and should all fall within the protection scope of the present invention.

Claims

1. A robot teleoperation method based on digital twins, characterized in that: include: Establish a digital twin model, including robot geometry model, joint kinematics model and physical model; Constructing a digital operation scene of the robot according to the actual operation scene of the robot, and controlling the movement of the robot and the geometric model of the robot at the same time; The motion trajectory of the robot is predicted according to the joint kinematic model, and the robot is visually monitored through the real-time dynamics of the robot and the real-time characteristic parameters of the digital twin model to realize remote operation of the robot; the joint kinematic model is an autoregressive state experience ARX model; Each joint is analyzed, and its ARX model is as follows: where u(k) is the input, y(k) is the output, and n a and n b is the polynomial order, e(k) is the error, a is the coefficient corresponding to the output term, b is the coefficient corresponding to the input term, and k is the time series of the discrete system; Solve for model parameters using the least squares method: y(k)=φ T (k)θ+e(k) Among them, φ T (k) is the sample set, θ is the coefficient set to be solved; The least squares estimate of the coefficients is obtained as follows:

2. The method according to claim 1, characterized in that Also includes: The robot feeds back characteristic parameters to the digital twin model, determines the error between the characteristic parameters and the real-time characteristic parameters, and if the error is less than a preset error threshold, continues to remotely operate the robot; if the error is greater than the preset error threshold, performs error detection on the digital twin model, optimizes the error, and then remotely operates the robot.

3. The method according to claim 2, characterized in that Performing error detection on the digital twin model and optimizing the error includes: Performing prediction error detection on the joint kinematics model, and optimizing and updating parameters of the joint kinematics model; Fault detection is performed on the robot, the robot geometric model, and the physical model, and corresponding fault repair is performed according to the fault detection result.

4. The method according to claim 2, characterized in that The characteristic parameters and / or the real-time characteristic parameters at least include motion characteristic parameters and structural characteristic parameters.

5. The method according to claim 1, characterized in that After predicting the motion trajectory of the robot according to the joint kinematics model, the method further includes: A motion simulation is performed in the digital operation scene according to the motion trajectory, and a control instruction for adjusting the motion of the robot is formulated according to the motion simulation situation.

6. The method according to claim 1, characterized in that The robot geometric model is a pure shape model, and joints are set at corresponding positions, wherein the pure shape includes a cube, a cylinder and a sphere.

7. A robot teleoperation system based on digital twins, characterized in that: include: Digital twin module, used to build digital twin model, including robot geometry model, joint kinematic model and physical model; A motion control module, used to construct a digital operation scene of the robot according to the actual operation scene of the robot, and to control the movement of the robot and the geometric model of the robot; A visual monitoring module is used to predict the motion trajectory of the robot according to the joint kinematic model, and to visually monitor the robot through the real-time dynamics of the robot and the real-time characteristic parameters of the digital twin model, so as to realize the remote operation of the robot; the joint kinematic model is an autoregressive state experience ARX model; each joint is analyzed, and its ARX model is as follows: where u(k) is the input, y(k) is the output, and n a and n b is the polynomial order, e(k) is the error, a is the coefficient corresponding to the output term, b is the coefficient corresponding to the input term, and k is the time series of the discrete system; Solve for model parameters using the least squares method: y(k)=φ T (k)θ+e(k) Among them, φ T (k) is the sample set, θ is the coefficient set to be solved; The least squares estimate of the coefficients is obtained as follows:

8. The system according to claim 7, characterized in that Also includes: The error optimization module is used to receive the feedback characteristic parameters of the robot, determine the error between the characteristic parameters and the real-time characteristic parameters, and continue to remotely operate the robot if the error is less than a preset error threshold; if the error is greater than the preset error threshold, perform error detection on the digital twin model, optimize the error, and then remotely operate the robot.

9. The system according to claim 7, characterized in that Also includes: The motion simulation module is used to perform motion simulation in the digital operation scene according to the motion trajectory, and formulate control instructions for adjusting the robot motion according to the motion simulation situation.

Images