Robot online control system and method based on augmented reality technology

Abstract

The invention relates to a robot online control system and method based on an augmented reality technology. The system comprises an augmented reality interaction terminal, a robot system and a data server. The augmented reality interaction end comprises a virtual-real fusion registration module, a gesture recognition module, a visualization module, a network communication module, an inverse kinematics calculation module and a collision detection module; the robot system comprises an entity robot, a robot controller, a robot upper computer and a network communication module. According to the robot control method for augmented reality interaction and digital twinning, the digital twinning robot is presented on the basis of augmented reality glasses, the gesture posture of an operator is obtained in real time, and a user can control the robot in the mode of grabbing the joint axis position of the digital twinning robot through gestures or clicking virtual keys. According to the invention, online control of the robot can be rapidly and visually realized, the robot can be rapidly and flexibly moved to a target pose, and the efficiency and safety of online control of the robot are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of augmented reality and robot online control, and relates to a robot online control system and method based on augmented reality technology. Background Art

[0002] Augmented reality technology is a comprehensive use of stereoscopic display, intelligent interaction, scene positioning, real-time registration and tracking technologies to integrate the virtual world with the real world, providing users with a visual interactive environment that integrates the virtual and the real, allowing users to perceive and interact with the virtual and real worlds at the same time. Industrial robots are multi-joint robotic arms or multi-degree-of-freedom machine devices widely used in the industrial field. They are usually composed of mechanical parts, sensor parts and control parts. They have the advantages of being simple to use, programmable, highly safe and easy to manage. At present, industrial robots have made great progress in manufacturing, assembly, testing, logistics and other fields, and have produced significant economic benefits.

[0003] At present, the control of industrial robots usually adopts offline programming and robot teaching to predefine the robot's motion trajectory, and then make the robot perform repetitive movements according to certain rules. For some scenarios that rely on human-machine collaborative operation, operators are required to control the robot. Traditional robot online control is usually achieved by operators directly operating the robot controller, which lacks flexibility and is difficult to operate. Summary of the invention

[0004] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, propose a robot online control system and method based on augmented reality technology, realize a new mode of robot online control with natural interaction and flexibility, and perform real-time early warning during the robot control process through collision detection technology, which is conducive to the robot being able to move quickly and flexibly to the target posture position, thereby improving the efficiency and safety of the robot online control.

[0005] The solution of the present invention is: a robot online control system based on augmented reality technology, including an augmented reality interaction terminal, a robot system, and a data service terminal;

[0006] The augmented reality interactive terminal is used to present the digital twin robot and provide a human-computer interaction interface for the operator, including a visualization module, a virtual-reality fusion registration module, a gesture control module, a first network communication module, an inverse kinematics solution module, and a collision detection module;

[0007] Visualization module: presents the digital twin robot in a virtual-real fusion manner, and displays the posture of the physical robot in real time through virtual-real posture mapping;

[0008] Virtual-reality fusion registration module: used to integrate the digital twin robot with the real scene, build a grid model of the real scene through robot vision and SLAM technology, and place the digital twin robot in a fixed plane position in the real scene through the plane detection algorithm;

[0009] Gesture control module: acquires the operator's gestures and postures in real time, and executes corresponding operation instructions according to the recognized key operation gestures;

[0010] The first network communication module: establishes data communication with the data server through a wireless network, is used to exchange data with the physical robot, updates the posture of the digital twin robot model according to the received posture data of the physical robot, and can send the user's control instructions to the physical robot to achieve reverse control of the physical robot device;

[0011] Inverse kinematics solution module: used to implement inverse kinematics solution of the digital twin robot model. The user grabs a certain axis of the digital twin robot model through gesture control and drags it to a certain position. The module calculates the robot's posture in real time according to the position of the grabbed axis.

[0012] Collision detection module: performs real-time collision detection on the digital twin robot model and the real scene mesh model. When the digital twin robot model collides with the real scene, a collision warning prompt is triggered and the operator is reminded to adjust the robot posture. When the operator corrects the posture of the digital twin robot so that it no longer triggers the collision warning, the physical robot is synchronized to the posture of the digital twin robot.

[0013] A robot system, including a physical robot, a robot controller, a robot host computer, and a second network communication module;

[0014] Physical robot: an N-axis robotic arm, where N is an integer greater than 1;

[0015] Robot controller: It is the data collection and control unit of the physical robot;

[0016] Robot host computer: built-in robot data acquisition and control program, obtains the posture and status of the physical robot, and sends control instructions to the robot controller;

[0017] The second network communication module: establishes data communication with the data server through a wireless network, is used to exchange data with the robot controller, sends the collected robot posture data to the augmented reality interaction terminal, and receives user control instructions from the augmented reality interaction terminal;

[0018] The data server is used to realize data interaction between the augmented reality interaction terminal and the robot system. It can distinguish the types of connected clients, forward the posture data and status data of the physical robot to the augmented reality interaction terminal, and forward the control instructions issued by the augmented reality interaction terminal to the robot system. The data server supports the access of multiple robot systems and augmented reality interaction terminals, and can coordinate the exclusivity of the augmented reality interaction terminal to the robot system, ensuring that the robot system can only be controlled by one augmented reality interaction terminal at the same time.

[0019] Furthermore, in the gesture control module, the gesture operation modes include clicking and grabbing, wherein the clicking gesture is used to interact with the virtual control buttons, and the grabbing gesture is used to drag an axis of the digital twin robot to manipulate the movement of the digital twin robot, thereby realizing the control of the physical robot.

[0020] Furthermore, the data service end includes a third network communication module and a data synchronization module;

[0021] The third network communication module: establishes a network communication bridge between the robot system and the augmented reality interaction terminal through the wireless communication network, realizes data exchange between the robot system and the augmented reality interaction terminal, can send the state data and posture data of the physical robot to the augmented reality interaction terminal, and forwards the control instructions issued by the augmented reality interaction terminal to the robot system;

[0022] Data synchronization module: used to achieve data synchronization between multiple augmented reality interaction terminals. This module can synchronize the interaction status data of the augmented reality interaction terminal to other clients, and cache the latest status data of each client to ensure that the status data of other clients can be quickly synchronized when a new client is connected.

[0023] A robot online control method based on augmented reality technology includes the following three interactive operation modes:

[0024] Free mode: The operator drags the joint axis of the digital twin robot through gestures. The operator grabs a joint axis of the digital twin robot and drags it. The inverse kinematics solution module performs inverse kinematics solution on the robot and updates the posture of the digital twin robot in real time. At the same time, the posture data of the digital twin robot is sent to the physical robot at a fixed frequency, and the physical robot is controlled to move synchronously with the virtual robot to the target posture position.

[0025] Single-axis mode: By clicking the virtual control button with a gesture, a certain axis of the physical robot is controlled to perform forward or reverse rotation. When the button is pressed, the collision detection module calculates the minimum distance between the digital twin robot and the surrounding environment. If the minimum distance is greater than the set collision warning threshold, a motion command is sent to the physical robot to control the movement of the physical robot. During the movement of the physical robot, the posture data of the physical robot is obtained in real time, and the posture of the digital twin robot is updated synchronously. If the minimum distance is less than or equal to the collision warning threshold, the physical robot is controlled to stop and a collision warning prompt is issued;

[0026] Translation mode: By clicking the virtual control button through gestures, the robot terminal is controlled to perform up, down, left, right, forward and backward movements in space. When the button is pressed, the collision detection module calculates the minimum distance between the digital twin model and the surrounding environment. If the minimum distance is greater than the set collision warning threshold, a motion command is sent to the physical robot to control the movement of the physical robot. During the movement of the physical robot, the posture data of the physical robot is obtained in real time, and the posture of the digital twin robot is updated synchronously. If the minimum distance is less than or equal to the collision warning threshold, the physical robot is controlled to stop and a collision warning prompt is issued.

[0027] Furthermore, the free mode is operated by dragging the joint axes of the digital twin robot through gestures, including two control modes: one-hand operation and two-hand operation.

[0028] Furthermore, the one-handed operation control mode, in which the user grasps a joint axis of the digital twin robot with one hand to perform an operation, specifically includes the following steps:

[0029] S1. The operator's gesture is recognized through the augmented reality interaction terminal, and a virtual ray is emitted based on the position and direction of the operator's index finger;

[0030] S2. When the ray collides with the digital twin robot, the collision position is highlighted;

[0031] S3. When the ray selects the i∈(1,N]th joint axis, a grab gesture operation can be performed to drag the robot joint axis to the target position;

[0032] S4. When the user drags the robot through gestures, the inverse kinematics solution module is used to solve the joint variables of the 1st to i-1th joint axes of the digital twin robot according to the target position and posture of the joint axis dragged by the user, and the posture of the digital twin robot is updated at the same time.

[0033] S5. The digital twin robot performs collision detection on the digital twin robot through the collision detection module during the movement. If a collision warning is triggered, the physical robot does not move synchronously, and a collision warning prompt is given to the operator; if no collision warning is triggered, an update posture instruction is sent to the physical robot at a fixed frequency;

[0034] S6. After receiving the command, the physical robot is placed in the posture queue, and the motion trajectory of the physical robot is updated in real time according to the posture data in the queue, so that the physical robot moves smoothly to the target position;

[0035] Furthermore, the two-handed operation control mode, in which the user grasps a joint axis of the digital twin robot with both hands to operate, specifically includes the following steps:

[0036] S1. Recognize the operator's gesture through the augmented reality interaction terminal, detect the position and direction of the operator's index finger and emit virtual rays;

[0037] S2. When the ray collides with the digital twin robot, the collision position is highlighted;

[0038] S3, one hand selects the i-th joint axis through the ray, i∈(1,N-1], and the other hand selects the j-th joint axis through the ray, j∈(i,N], and performs a grabbing gesture operation, and each drags the grabbed joint axis to the target position;

[0039] S4. When the user drags the robot with a gesture, the inverse kinematics solution module calculates the joint variables of the 1st to i-1th and i-j-1th joint axes, and updates the posture of the digital twin robot at the same time.

[0040] S5, releasing the grasping gesture. If a collision warning is triggered, a collision warning prompt is given, and the physical robot does not perform synchronous motion operations; if no collision warning is triggered, an update posture instruction is sent to the physical robot at a fixed frequency;

[0041] S6. After receiving the command, the physical robot is placed in the posture queue, and the motion trajectory of the physical robot is updated in real time according to the posture data in the queue, so that the physical robot moves smoothly to the target position.

[0042] Furthermore, the solution process of the inverse kinematics solution module is as follows:

[0043] (1) Define the position vector and posture matrix of the grasped joint axis relative to the base coordinate system O in the current configuration of the digital twin robot as P c and R c =[r1,r2,r3],r j, j = 1, 2, 3 are the unit vectors of the three coordinate axes of the local coordinate system of the grasped joint axis in the current configuration; in the one-handed operation mode, the base coordinate system is the base coordinate system O of the digital twin robot. o In the two-hand operation mode, the base coordinate system of the first grasp is the base coordinate system O of the digital twin robot. o The base coordinate system for the second grasping is the fixed coordinate system O of the joint axis of the digital twin robot grasped by the gesture for the first time. i ; Define the target position vector and posture matrix of the gesture drag as P d and R d =[d1,d2,d3], d j ,j=1,2,3 are the unit vectors of the three coordinate axes of the local coordinate system of the grasped joint axis at the target position and target posture;

[0044] (2) Construct the position error function of the grasped joint axis of the digital twin robot from the current position to the target position, and define the error between the target position and the current position of the grasping axis as the Euclidean norm of the vector formed by the two position points:

[0045] ΔP=||P d -P c ||2

[0046] Assume o i is the coordinate origin of the i-th, i=1,...,N joint axis in the current configuration, y i is the unit vector along the rotation axis of the i-th joint, P ic Yes i Pointing to the position vector of the grasped joint axis in the current posture of the robot, P id Yes i Points to the position vector of the grasped joint axis in the robot's target posture. Since the rotation axis of each joint of the robot is fixed, P ic Around y i The axis rotates by θ degrees to obtain the new vector P it :

[0047] P it (θ) = R(y i ,θ)P ic

[0048] Among them, R(y i ,θ) is a 3×3 spatial rotation matrix, representing rotation around y i Axis rotation θ degrees;

[0049] The position error function of the grasped axis between the current posture and the target posture of the digital twin robot is expressed as:

[0050]

[0051] Convert minimizing ΔP(θ) to maximizing P id [R(y i ,θ)P ic ] question,

[0052] e1(θ)=P id [R(y i ,θ)P ic ]

[0053] (3) Construct the posture error function of the grasped joint axis of the digital twin robot from the current posture to the target posture:

[0054]

[0055] Among them, (d j ·r j -1)∈[-2,0];

[0056] r j Around y i The axis rotates by θ degrees to get the new vector r jt :

[0057] r jt (θ) = R(y i ,θ)r j

[0058] Definition φ j (θ) is the vector d j With r jt The angle between (θ) and φ j (θ)∈[0,π], vector d j and r jt (θ) are all unit vectors, and d j r jt (θ) = cosφ j (θ);

[0059] The posture error function between the current posture of the grasped axis and the target posture of the digital twin robot is expressed as:

[0060]

[0061] Transform minimization of ΔR(θ) into maximization Problem, order

[0062]

[0063] (4) Define the error function of the grasped joint axis of the digital twin robot from the current pose to the target pose:

[0064] e(θ)=ω1e1(θ)+ω2e2(θ)

[0065] Where ω1 and ω2 are the weights of the position error and attitude error respectively. Substituting the Rodriguez formula into the above formula, we get:

[0066] e(θ)=αcosθ+βsinθ+γ(1-cosθ)

[0067] in:

[0068]

[0069]

[0070]

[0071] Simplify to get: g(θ)=Acosθ+Bsinθ+C, where A=α-γ, B=β, C=γ;

[0072] Convert g(θ) into the following form:

[0073]

[0074] in δ∈[-π,π]; when When k∈Z, g(θ) reaches its maximum value, correspondingly e(θ) reaches its maximum value;

[0075] (5) According to the above formula, in each iteration, for each joint axis that needs to be adjusted, the rotation value θ that makes g(θ) take the maximum value is solved, and the joint variables of each joint axis are adjusted in turn until the iteration ends.

[0076] Furthermore, the iterative solution process of the inverse kinematics solution includes the following steps:

[0077] 1) Get the index of the joint axis that the user is currently grasping through gestures. For an N-axis robot arm, obtain the range of the joint axis that needs to be iteratively updated [l,u]∈[1,N] according to the current one-hand or two-hand operation mode and the position of the joint axis grasped by gestures;

[0078] 2) Initialize and define the maximum number of iterations H max , maximum position error ε p , maximum attitude error ε R , weights ω1, ω2, the current posture value θ of each joint axis to be updated i c. The joint limit range θ of each joint axis that needs to be updated i min ,θ i max , the current number of iterations H, where i∈[l,u];

[0079] 3) For each joint axis that needs to be updated, calculate the rotation value θ that maximizes the objective function g(θ) i , and update the current posture value of each joint axis in: like but like but

[0080] 4) Update the number of iterations H to H+1, and determine whether the number of iterations H is greater than the maximum number of iterations H max , if it is greater, exit;

[0081] 5) Calculate the position error ΔP and attitude error ΔR. If ΔP≤ε p And ΔR≤ε R If yes, exit; otherwise, jump to step 3) to continue iterating.

[0082] The beneficial effects of the present invention compared with the prior art are:

[0083] (1) The present invention combines augmented reality technology, body perception technology and digital twin technology to provide real-time visual feedback on the robot's posture and state in an augmented reality visualization manner, and interacts with the digital twin robot through gestures to achieve online control of the robot. The present invention provides an intuitive and flexible new mode of robot online control and robot posture feedback. Compared with the existing robot offline programming and teaching control method, while meeting the robot control function, it improves the flexibility of human-machine collaboration and control; in addition, based on network communication transmission, the present invention provides a new mode of robot remote collaborative control;

[0084] (2) The present invention obtains the operator's gesture posture in real time through the gesture control module, supports grabbing a joint axis with both hands to perform dragging movement, and uses the inverse kinematics solution module to "segmentally" solve the robot posture according to the grasping position. Compared with the existing control method of grabbing the end of the robot and performing inverse kinematics solution on all joint axes, the present invention solves the problem of uncontrollable posture of the intermediate joint axis on the premise of realizing the inverse kinematics solution of the robot. The operator can quickly adjust the robot to the desired posture, thereby improving the efficiency and flexibility of robot control;

[0085] (3) The present invention uses a collision detection module to perform real-time collision detection on the robot based on the digital twin robot model during the operation of the robot, thereby improving the safety of the robot's online control. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a structural diagram of the robot online control system according to an embodiment of the present invention;

[0087] Figure 2 This is a schematic diagram of a robot online control system according to an embodiment of the present invention;

[0088] Among them: 1-augmented reality interaction terminal, 2-digital twin robot, 3-physical robot, 4-robot controller, 5-robot host computer, 6-data server;

[0089] Figure 3 The figure is a flow chart of the free mode interactive operation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0090] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0091] The present invention provides a robot online control system and method based on augmented reality technology, which realizes online control of the robot through augmented reality interaction. The augmented reality robot system integrates augmented reality technology with the robot, uses augmented reality glasses and other devices as display and interaction terminals, and establishes real-time communication between the augmented reality device and the physical robot based on network communication technology, thereby establishing a new control and state feedback path for the robot. Through the augmented reality device, the user can transmit the operation instructions to the robot, and present the state parameter information of the physical robot to the user in real time.

[0092] Example 1

[0093] This embodiment provides a robot online control system based on augmented reality technology, including an augmented reality interaction terminal, a robot system, and a data service terminal. Figure 1 As shown. Among them:

[0094] The augmented reality interactive terminal is used to present the digital twin robot and provide a human-computer interaction interface for the operator. It includes a visualization module, a virtual-reality fusion registration module, a gesture control module, a first network communication module, an inverse kinematics solution module, and a collision detection module.

[0095] Visualization module: presents the digital twin robot in a virtual-real fusion way, and displays the posture of the physical robot in real time through virtual-real posture mapping.

[0096] Virtual-reality fusion registration module: used to integrate the digital twin robot with the real scene, build a grid model of the real scene through robot vision and SLAM technology, and place the digital twin robot in a fixed plane position in the real scene through a plane detection algorithm.

[0097] Gesture control module: acquires the operator's gestures and postures in real time, and executes corresponding operation instructions according to the identified key operation gestures; gesture operation methods include clicking and grabbing, where the click gesture is used to interact with the virtual control buttons, and the grab gesture is used to drag a certain axis of the digital twin robot to manipulate the movement of the digital twin robot, thereby realizing the control of the physical robot.

[0098] The first network communication module: establishes data communication with the data server through a wireless network, is used to exchange data with the physical robot, updates the posture of the digital twin robot model according to the received physical robot posture data, and can send the user's control instructions to the physical robot to achieve reverse control of the physical robot equipment.

[0099] Inverse kinematics solution module: used to realize the inverse kinematics solution of the digital twin robot model. The user grabs an axis of the digital twin robot model through gesture control and drags it to a certain position. The module solves the robot's posture in real time according to the position of the grabbed axis, thereby improving the interactive flexibility of the system.

[0100] Collision detection module: performs real-time collision detection on the digital twin robot model and the real-scene mesh model. When the digital twin robot model collides with the real scene, a collision warning prompt is triggered and the operator is reminded to adjust the robot's posture. When the operator corrects the posture of the digital twin robot so that it no longer triggers the collision warning, the physical robot is synchronized to the posture of the digital twin robot.

[0101] The robot system includes a physical robot, a robot controller, a robot host computer, and a second network communication module.

[0102] Physical robot: a six-axis robotic arm;

[0103] Robot controller: It is the data collection and control unit of the physical robot;

[0104] Robot host computer: built-in robot data acquisition and control program, obtains the posture and status of the physical robot, and sends control instructions to the robot controller;

[0105] The second network communication module: establishes data communication with the data server through a wireless network, is used to exchange data with the robot controller, sends the collected robot posture data to the augmented reality interaction terminal, and receives user control instructions from the augmented reality interaction terminal.

[0106] The data server is used to realize data interaction between the augmented reality interaction terminal and the robot system. It can distinguish the types of connected clients, forward the posture data and status data of the physical robot to the augmented reality interaction terminal, and forward the control instructions issued by the augmented reality interaction terminal to the robot system; the data server supports the access of multiple robot systems and augmented reality interaction terminals, and can coordinate the exclusivity of the augmented reality interaction terminal to the robot system, ensuring that the robot system can only be controlled by one augmented reality interaction terminal at the same time.

[0107] In this embodiment, the data service end includes a third network communication module and a data synchronization module;

[0108] The third network communication module: establishes a network communication bridge between the robot system and the augmented reality interaction terminal through the wireless communication network, realizes data exchange between the robot system and the augmented reality interaction terminal, can send the state data and posture data of the physical robot to the augmented reality interaction terminal, and forwards the control instructions issued by the augmented reality interaction terminal to the robot system;

[0109] Data synchronization module: used to achieve data synchronization between multiple augmented reality interaction terminals. This module can synchronize the interaction status data of the augmented reality interaction terminal to other clients, and cache the latest status data of each client to ensure that the status data of other clients can be quickly synchronized when a new client is connected.

[0110] Based on the robot online control system, this embodiment proposes a robot online control method based on augmented reality technology, including the following three interactive operation modes:

[0111] Free mode: Drag the joint axis of the digital twin robot by gestures, such as Figure 2 As shown, the operator grabs a joint axis of the digital twin robot and drags it. The inverse kinematics solution module performs inverse kinematics solution on the robot and updates the posture of the digital twin robot in real time. At the same time, the posture data of the digital twin robot is sent to the physical robot at a fixed frequency, controlling the physical robot and the virtual robot to move synchronously to the target posture position.

[0112] Single-axis mode: Click the virtual control button through gestures to control a certain axis of the physical robot to perform forward or reverse rotation. When the button is pressed, the collision detection module calculates the minimum distance between the digital twin robot and the surrounding environment. If the minimum distance is greater than the set collision warning threshold, a motion command is sent to the physical robot to control the movement of the physical robot. During the movement of the physical robot, the posture data of the physical robot is obtained in real time, and the posture of the digital twin robot is updated synchronously. If the minimum distance is less than or equal to the collision warning threshold, the physical robot is controlled to stop and a collision warning prompt is issued.

[0113] Translation mode: By clicking the virtual control button through gestures, the robot terminal is controlled to perform up, down, left, right, forward and backward movements in space. When the button is pressed, the collision detection module calculates the minimum distance between the digital twin model and the surrounding environment. If the minimum distance is greater than the set collision warning threshold, a motion command is sent to the physical robot to control the movement of the physical robot. During the movement of the physical robot, the posture data of the physical robot is obtained in real time, and the posture of the digital twin robot is updated synchronously. If the minimum distance is less than or equal to the collision warning threshold, the physical robot is controlled to stop and a collision warning prompt is issued.

[0114] In the free mode, the user can grab different joint axes of the robot digital twin to perform operations, including two control modes: one-hand operation and two-hand operation.

[0115] One-handed control mode, such as Figure 3 As shown, the user can grab a joint axis of the digital twin robot with one hand to perform operations, which include the following steps:

[0116] S1. The operator's gesture is recognized through the augmented reality interaction terminal, and a virtual ray is emitted based on the position and direction of the operator's index finger;

[0117] S2. When the ray collides with the digital twin robot, the collision position is highlighted;

[0118] S3. When the ray selects the i∈(1,6]th joint axis, a grab gesture operation can be performed to drag the robot joint axis to the target position;

[0119] S4. When the user drags the robot by gesture, the inverse kinematics solution module is used to solve the joint variables of the 1st to i-1th joint axes of the digital twin robot according to the target position and posture of the joint axis dragged by the user, and the posture of the digital twin robot is updated at the same time;

[0120] S5. The digital twin robot performs collision detection on the digital twin robot through the collision detection module during the movement. If a collision warning is triggered, the physical robot does not move synchronously, and a collision warning prompt is given to the operator; if no collision warning is triggered, an update posture instruction is sent to the physical robot at a fixed frequency;

[0121] S6. After receiving the command, the physical robot is placed in the posture queue, and the motion trajectory of the physical robot is updated in real time according to the posture data in the queue, so that the physical robot moves smoothly to the target position.

[0122] The two-handed control mode allows the user to grasp a joint axis of the digital twin robot with both hands to operate it. The specific steps include:

[0123] S1. Recognize the operator's gesture through the augmented reality interaction terminal, detect the position and direction of the operator's index finger and emit virtual rays;

[0124] S2. When the ray collides with the digital twin robot, the collision position is highlighted;

[0125] S3, one hand selects the i-th joint axis through the ray, i∈(1,5], and the other hand selects the j-th joint axis through the ray, j∈(i,6], and performs a grabbing gesture operation, and each drags the grabbed joint axis to the target position;

[0126] S4. When the user drags the robot with a gesture, the inverse kinematics solution module calculates the joint variables of the 1st to i-1th and i-j-1th joint axes, and updates the posture of the digital twin robot at the same time.

[0127] S5, releasing the grasping gesture. If a collision warning is triggered, a collision warning prompt is given, and the physical robot does not perform synchronous motion operations; if no collision warning is triggered, an update posture instruction is sent to the physical robot at a fixed frequency;

[0128] S6. After receiving the command, the physical robot is placed in the posture queue, and the motion trajectory of the physical robot is updated in real time according to the posture data in the queue, so that the physical robot moves smoothly to the target position.

[0129] In the above method, the inverse kinematics solution module performs the following inverse kinematics solution process:

[0130] (1) Define the position vector and posture matrix of the grasped joint axis relative to the base coordinate system O in the current configuration of the digital twin robot as P c and R c =[r1,r2,r3],r j , j = 1, 2, 3 are the unit vectors of the three coordinate axes of the local coordinate system of the grasped joint axis in the current configuration; in the one-handed operation mode, the base coordinate system is the base coordinate system O of the digital twin robot. o In the two-hand operation mode, the base coordinate system of the first grasp is the base coordinate system O of the digital twin robot. o The base coordinate system for the second grasping is the fixed coordinate system O of the joint axis of the digital twin robot grasped by the gesture for the first time. i ; Define the target position vector and posture matrix of the gesture drag as P d and R d =[d1,d2,d3], where d j ,j=1,2,3 are the unit vectors of the three coordinate axes of the local coordinate system of the grasped joint axis at the target position and target posture;

[0131] (2) Construct the position error function of the grasped joint axis of the digital twin robot from the current position to the target position, and define the error between the target position and the current position of the grasping axis as the Euclidean norm of the vector formed by the two position points:

[0132] ΔP=||P d -P c ||2

[0133] Assume o i is the coordinate origin of the i-th, i=1,...,N joint axis in the current configuration, y i is the unit vector along the rotation axis of the i-th joint, P ic Yes i Pointing to the position vector of the grasped joint axis in the current posture of the robot, P id Yes i Points to the position vector of the grasped joint axis in the robot's target posture. Since the rotation axis of each joint of the robot is fixed, P ic Around y i The axis rotates by θ degrees, and the new vector P is obtained. it :

[0134] P it (θ) = R(y i ,θ)P ic

[0135] Among them, R(y i ,θ) is a 3×3 spatial rotation matrix, representing rotation around y i Axis rotation θ degrees;

[0136] At this time, the position error function of the grasped axis between the current posture and the target posture of the digital twin robot is expressed as:

[0137]

[0138] Therefore, minimizing ΔP(θ) can be transformed into maximizing P id [R(y i ,θ)P ic ] question,

[0139] e1(θ)=P id [R(y i ,θ)P ic ]

[0140] (3) Construct the posture error function of the grasped joint axis of the digital twin robot from the current posture to the target posture. j With d j are all unit vectors, which satisfy Time j With dj The deviation is the smallest, so the error between the target posture of the grasped joint axis and the current posture can be expressed as:

[0141]

[0142] Among them, (d j ·r j -1)∈[-2,0];

[0143] Since the rotation axis of each joint of the robot is fixed, j Around y i The axis rotates by θ degrees to get the new vector r jt :

[0144] r jt (θ) = R(y i ,θ)r j

[0145] Definition φ j (θ) is the vector d j With r jt The angle between (θ) and φ j (θ)∈[0,π], vector d j and r jt (θ) are all unit vectors, so d j r jt (θ) = cosφ j (θ);

[0146] At this time, the posture error function between the current posture of the grasped axis and the target posture of the digital twin robot is expressed as:

[0147]

[0148] Since cosφ j (θ) is a bounded function, so minimizing ΔR(θ) can be transformed into maximizing Problem, order

[0149]

[0150] (4) Define the error function of the grasped joint axis of the digital twin robot from the current pose to the target pose:

[0151] e(θ)=ω1e1(θ)+ω2e2(θ)

[0152] Where ω1 and ω2 are the weights of the position error and attitude error respectively. Substituting the Rodriguez formula into the above formula, we get:

[0153] e(θ)=αcosθ+βsinθ+γ(1-cosθ)

[0154] in:

[0155]

[0156]

[0157]

[0158] Simplify to get: g(θ)=Acosθ+Bsinθ+C, where A=α-γ, B=β, C=γ;

[0159] Convert g(θ) into the following form:

[0160]

[0161] in δ∈[-π,π]; when When k∈Z, g(θ) reaches its maximum value, and correspondingly e(θ) reaches its maximum value.

[0162] (5) According to the above formula, in each iteration, for each joint axis that needs to be adjusted, the rotation value θ that makes g(θ) take the maximum value is solved, and the joint variables of each joint axis are adjusted in turn until the iteration ends.

[0163] Specifically, step (5) includes the following steps in the iterative solution process:

[0164] 1) Get the index of the joint axis that the user is currently grasping through gestures. For an N-axis robot arm, obtain the range of the joint axis that needs to be iteratively updated [l,u]∈[1,N] according to the current one-hand or two-hand operation mode and the position of the joint axis grasped by gestures;

[0165] 2) Initialize and define the maximum number of iterations H max , maximum position error ε p , maximum attitude error ε R , weights ω1, ω2, the current posture value of each joint axis that needs to be updated The joint limit range θ of each joint axis that needs to be updated i min ,θ i max , the current number of iterations H, where i∈[l,u];

[0166] 3) For each joint axis that needs to be updated, calculate the rotation value θ that maximizes the objective function g(θ) i , and update the current posture value of each joint axis in: like but like but

[0167] 4) Update the number of iterations H = H + 1, and determine whether the number of iterations H is greater than the maximum number of iterations H max , if it is greater, exit;

[0168] 5) Calculate the position error ΔP and attitude error ΔR. If ΔP≤ε p And ΔR≤ε R If yes, exit; otherwise, jump to step 3) to continue iterating.

[0169] In summary, this embodiment presents a digital twin robot based on augmented reality glasses, obtains the operator's gestures and postures in real time, and enables the user to control the robot online by grasping the joint axis position of the digital twin robot through gestures or clicking virtual buttons.

[0170] The present invention can quickly and intuitively realize online control of the robot, can quickly and flexibly move the robot to a target position, and improve the efficiency and safety of the robot's online control.

[0171] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

[0172] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A robot online control system based on augmented reality technology, characterized in that: Including augmented reality interaction terminal, robot system, and data service terminal; The augmented reality interactive terminal is used to present the digital twin robot and provide a human-computer interaction interface for the operator, including a visualization module, a virtual-reality fusion registration module, a gesture control module, a first network communication module, an inverse kinematics solution module, and a collision detection module; Visualization module: presents the digital twin robot in a virtual-real fusion manner, and displays the posture of the physical robot in real time through virtual-real posture mapping; Virtual-reality fusion registration module: used to integrate the digital twin robot with the real scene, build a grid model of the real scene through robot vision and SLAM technology, and place the digital twin robot in a fixed plane position in the real scene through the plane detection algorithm; Gesture control module: acquires the operator's gestures and postures in real time, and executes corresponding operation instructions according to the recognized key operation gestures; The first network communication module: establishes data communication with the data server through a wireless network, is used to exchange data with the physical robot, updates the posture of the digital twin robot model according to the received posture data of the physical robot, and can send the user's control instructions to the physical robot to achieve reverse control of the physical robot device; Inverse kinematics solution module: used to implement inverse kinematics solution of the digital twin robot model. The user grabs a certain axis of the digital twin robot model through gesture control and drags it to a certain position. The module calculates the robot's posture in real time according to the position of the grabbed axis. Collision detection module: performs real-time collision detection on the digital twin robot model and the real scene mesh model. When the digital twin robot model collides with the real scene, a collision warning prompt is triggered and the operator is reminded to adjust the robot posture. When the operator corrects the posture of the digital twin robot so that it no longer triggers the collision warning, the physical robot is synchronized to the posture of the digital twin robot. A robot system, including a physical robot, a robot controller, a robot host computer, and a second network communication module; Physical robot: an N-axis robotic arm, where N is an integer greater than 1; Robot controller: It is the data collection and control unit of the physical robot; Robot host computer: built-in robot data acquisition and control program, obtains the posture and status of the physical robot, and sends control instructions to the robot controller; The second network communication module: establishes data communication with the data server through a wireless network, is used to exchange data with the robot controller, sends the collected robot posture data to the augmented reality interaction terminal, and receives user control instructions from the augmented reality interaction terminal; The data server is used to realize data interaction between the augmented reality interaction terminal and the robot system. It can distinguish the types of connected clients, forward the posture data and status data of the physical robot to the augmented reality interaction terminal, and forward the control instructions issued by the augmented reality interaction terminal to the robot system. The data server supports the access of multiple robot systems and augmented reality interaction terminals, and can coordinate the exclusivity of the augmented reality interaction terminal to the robot system to ensure that the robot system can only be controlled by one augmented reality interaction terminal at the same time.

2. The robot online control system based on augmented reality technology according to claim 1, characterized in that: The gesture control module includes gesture operation modes such as clicking and grabbing, wherein the clicking gesture is used to interact with the virtual control buttons, and the grabbing gesture is used to drag an axis of the digital twin robot to manipulate the movement of the digital twin robot, thereby realizing control of the physical robot.

3. The robot online control system based on augmented reality technology according to claim 1, characterized in that: The data service end includes a third network communication module and a data synchronization module; The third network communication module: establishes a network communication bridge between the robot system and the augmented reality interaction terminal through the wireless communication network, realizes data exchange between the robot system and the augmented reality interaction terminal, can send the state data and posture data of the physical robot to the augmented reality interaction terminal, and forwards the control instructions issued by the augmented reality interaction terminal to the robot system; Data synchronization module: used to achieve data synchronization between multiple augmented reality interaction terminals. This module can synchronize the interaction status data of the augmented reality interaction terminal to other clients, and cache the latest status data of each client to ensure that the status data of other clients can be quickly synchronized when a new client is connected.

4. A robot online control method based on augmented reality technology based on the system according to any one of claims 1 to 3, characterized in that: There are three interactive operation modes: Free mode: The operator drags the joint axis of the digital twin robot through gestures. The operator grabs a joint axis of the digital twin robot and drags it. The inverse kinematics solution module performs inverse kinematics solution on the robot and updates the posture of the digital twin robot in real time. At the same time, the posture data of the digital twin robot is sent to the physical robot at a fixed frequency, and the physical robot is controlled to move synchronously with the virtual robot to the target posture position. Single-axis mode: By clicking the virtual control button with a gesture, a certain axis of the physical robot is controlled to perform forward or reverse rotation. When the button is pressed, the collision detection module calculates the minimum distance between the digital twin robot and the surrounding environment. If the minimum distance is greater than the set collision warning threshold, a motion command is sent to the physical robot to control the movement of the physical robot. During the movement of the physical robot, the posture data of the physical robot is obtained in real time, and the posture of the digital twin robot is updated synchronously. If the minimum distance is less than or equal to the collision warning threshold, the physical robot is controlled to stop and a collision warning prompt is issued; Translation mode: By clicking the virtual control button through gestures, the robot terminal is controlled to perform up, down, left, right, forward and backward movements in space. When the button is pressed, the collision detection module calculates the minimum distance between the digital twin model and the surrounding environment. If the minimum distance is greater than the set collision warning threshold, a motion command is sent to the physical robot to control the movement of the physical robot. During the movement of the physical robot, the posture data of the physical robot is obtained in real time, and the posture of the digital twin robot is updated synchronously. If the minimum distance is less than or equal to the collision warning threshold, the physical robot is controlled to stop and a collision warning prompt is issued.

5. The robot online control method based on augmented reality technology according to claim 4, characterized in that: The free mode is operated by dragging the joint axes of the digital twin robot through gestures, including two control modes: one-hand operation and two-hand operation.

6. The robot online control method based on augmented reality technology according to claim 5, characterized in that: The one-handed operation control mode, in which the user grasps a joint axis of the digital twin robot with one hand to perform an operation, specifically includes the following steps: S1. The operator's gesture is recognized through the augmented reality interaction terminal, and a virtual ray is emitted based on the position and direction of the operator's index finger; S2. When the ray collides with the digital twin robot, the collision position is highlighted; S3. When the ray selects the i∈(1,N]th joint axis, a grab gesture operation can be performed to drag the robot joint axis to the target position; S4. When the user drags the robot through gestures, the inverse kinematics solution module is used to solve the joint variables of the 1st to i-1th joint axes of the digital twin robot according to the target position and posture of the joint axis dragged by the user, and the posture of the digital twin robot is updated at the same time. S5. The digital twin robot performs collision detection on the digital twin robot through the collision detection module during the movement. If a collision warning is triggered, the physical robot does not move synchronously, and a collision warning prompt is given to the operator; if no collision warning is triggered, an update posture instruction is sent to the physical robot at a fixed frequency; S6. After receiving the command, the physical robot is placed in the posture queue, and the motion trajectory of the physical robot is updated in real time according to the posture data in the queue, so that the physical robot moves smoothly to the target position.

7. The robot online control method based on augmented reality technology according to claim 5, characterized in that: The two-handed operation control method, in which the user grasps a joint axis of the digital twin robot with both hands to operate, specifically includes the following steps: S1. Recognize the operator's gesture through the augmented reality interaction terminal, detect the position and direction of the operator's index finger and emit virtual rays; S2. When the ray collides with the digital twin robot, the collision position is highlighted; S3, one hand selects the i-th joint axis through the ray, i∈(1,N-1], and the other hand selects the j-th joint axis through the ray, j∈(i,N], and performs a grabbing gesture operation, and each drags the grabbed joint axis to the target position; S4. When the user drags the robot with a gesture, the inverse kinematics solution module calculates the joint variables of the 1st to i-1th and i-j-1th joint axes, and updates the posture of the digital twin robot at the same time. S5, releasing the grasping gesture. If a collision warning is triggered, a collision warning prompt is given, and the physical robot does not perform synchronous motion operations; if no collision warning is triggered, an update posture instruction is sent to the physical robot at a fixed frequency; S6. After receiving the command, the physical robot is placed in the posture queue, and the motion trajectory of the physical robot is updated in real time according to the posture data in the queue, so that the physical robot moves smoothly to the target position.

8. A robot online control method based on augmented reality technology according to any one of claims 6 or 7, characterized in that: The solution process of the inverse kinematics solution module is as follows: (1) Define the position vector and posture matrix of the grasped joint axis relative to the base coordinate system O in the current configuration of the digital twin robot as P c and R c =[r1,r2,r3],r j , j = 1, 2, 3 are the unit vectors of the three coordinate axes of the local coordinate system of the grasped joint axis in the current configuration; in the one-handed operation mode, the base coordinate system is the base coordinate system O of the digital twin robot. o In the two-hand operation mode, the base coordinate system of the first grasp is the base coordinate system O of the digital twin robot. o The base coordinate system for the second grasping is the fixed coordinate system O of the joint axis of the digital twin robot grasped by the gesture for the first time. i ; Define the target position vector and posture matrix of the gesture drag as P d and R d =[d1,d2,d3], d j ,j=1,2,3 are the unit vectors of the three coordinate axes of the local coordinate system of the grasped joint axis at the target position and target posture; (2) Construct the position error function of the grasped joint axis of the digital twin robot from the current position to the target position, and define the error between the target position and the current position of the grasping axis as the Euclidean norm of the vector formed by the two position points: ΔP=||P d -P c ||2 Assume o i is the coordinate origin of the i-th, i=1,...,N joint axis in the current configuration, y i is the unit vector along the rotation axis of the i-th joint, P ic Yes i Pointing to the position vector of the grasped joint axis in the current posture of the robot, P id Yes i Points to the position vector of the grasped joint axis in the robot's target posture. Since the rotation axis of each joint of the robot is fixed, P ic Around y i The axis rotates by θ degrees to obtain the new vector P it : P it (θ)=R(y i ,i)P ic Among them, R(y i ,θ) is a 3×3 spatial rotation matrix, representing rotation around y i Axis rotation θ degrees; The position error function of the grasped axis between the current posture and the target posture of the digital twin robot is expressed as: Convert minimizing ΔP(θ) to maximizing P id [R(y i ,θ)P ic ] question, e1(θ)=P id [R(y i ,θ)P ic ] (3) Construct the posture error function of the grasped joint axis of the digital twin robot from the current posture to the target posture: Among them, (d j ·r j -1)∈[-2,0]; r j Around y i The axis rotates by θ degrees to get the new vector r jt : r jt (θ)=R(y i ,i)r j Definition φ j (θ) is the vector d j With r jt The angle between (θ) and φ j (θ)∈[0,π], vector d j and r jt (θ) are all unit vectors, and d j r jt (θ) = cosφ j (θ); The posture error function between the current posture of the grasped axis and the target posture of the digital twin robot is expressed as: Transform minimization of ΔR(θ) into maximization Problem, order (4) Define the error function of the grasped joint axis of the digital twin robot from the current pose to the target pose: e(θ)=ω1e1(θ)+ω2e2(θ) Where ω1 and ω2 are the weights of the position error and attitude error respectively. Substituting the Rodriguez formula into the above formula, we get: e(θ)=α cosθ+βsinθ+γ(1-cosθ) in: Simplify to get: g(θ)=Acosθ+Bsinθ+C, where A=α-γ, B=β, C=γ; Convert g(θ) into the following form: in when When g(θ) reaches its maximum value, correspondingly e(θ) reaches its maximum value; (5) According to the above formula, in each iteration, for each joint axis that needs to be adjusted, the rotation value θ that makes g(θ) take the maximum value is solved, and the joint variables of each joint axis are adjusted in turn until the iteration ends.

9. The robot online control method based on augmented reality technology according to claim 8, characterized in that: The iterative solution process of inverse kinematics solution includes the following steps: 1) Get the index of the joint axis that the user is currently grasping through gestures. For an N-axis robot arm, obtain the range of the joint axis that needs to be iteratively updated [l,u]∈[1,N] according to the current one-hand or two-hand operation mode and the position of the joint axis grasped by gestures; 2) Initialize and define the maximum number of iterations H max , maximum position error ε p , maximum attitude error ε R , weights ω1, ω2, the current posture value of each joint axis that needs to be updated The joint limit range of each joint axis that needs to be updated The current number of iterations H, where i∈[l,u]; 3) For each joint axis that needs to be updated, calculate the rotation value θ that maximizes the objective function g(θ) i , and update the current posture value of each joint axis Among them, if but like but 4) Update the number of iterations H to H+1, and determine whether the number of iterations H is greater than the maximum number of iterations H max , if it is greater, exit; 5) Calculate the position error ΔP and attitude error ΔR. If ΔP≤ε p And ΔR≤ε R If yes, exit; otherwise, jump to step 3) to continue iterating.

Images