Method and system for teleoperation of humanoid robot through tracker and data glove

Abstract

The invention discloses a method and system for remotely operating a humanoid robot through a tracker and a data glove, and the method comprises the following steps: capturing the position and posture of a hand in a three-dimensional space based on the tracker, and taking the posture information of the tracker as the expected posture of the tail end of the humanoid robot; each joint angle of the mechanical arm is solved based on a Cartesian motion control inverse kinematics method, and the humanoid robot mechanical arm is controlled to move to the expected position and posture; the motion of the fingers is captured based on the data glove, the finger joint bending angle captured by the data glove is mapped to the manipulator joint of the humanoid robot, the joint angle of the manipulator is obtained, and the fingers of the manipulator of the humanoid robot are controlled to move to the specified angle position based on the joint angle of the manipulator. The posture of the hand and the movement of the fingers are mapped to the real humanoid robot on the basis of the tracker and the data glove, the flexible operation task of the two hands is completed, the intuition of remote operation is improved, and meanwhile the burden of a user is reduced.

Description

Technical Field

[0001] The present invention relates to the field of remote control technology, and in particular to a method and system for remotely controlling a humanoid robot through a tracker and a data glove. Background Art

[0002] Humanoid robots have a human-like structure, flexible movement ability, and can coordinate both hands to complete various tasks, so they have great potential in practical applications. However, humanoid robots face problems such as high degrees of freedom, complex kinematic characteristics, and control difficulties, and their remote operation is particularly difficult.

[0003] Most existing methods are based on visual recognition, such as a method for remotely controlling a humanoid robot by recognizing hand postures through virtual reality glasses, which is patented as CN117746494A. The operator wears VR glasses, and the inertial sensors on the glasses collect the operator's head and neck posture signals, which are then sent to the humanoid robot after being resolved to drive its head to follow the operator's synchronous movement. The camera on the VR glasses collects images of the operator's upper limbs and hands, identifies the operator's finger joint angles, and uses them to control the finger joints of the humanoid robot. However, the operator's wearing of VR glasses on his head increases the burden on the operator, and long-term operation can easily cause the operator to feel dizzy. Methods based on visual image recognition are easily affected by noise, resulting in inaccurate recognition results. In addition, they are limited by the camera's field of view, resulting in a limited range of operation. The movement of fingers will also cause mutual occlusion, which increases the difficulty of the image recognition algorithm. Summary of the invention

[0004] In order to overcome the defects and shortcomings of the prior art, the present invention provides a method and system for remotely operating a humanoid robot through a tracker and a data glove. Based on the tracker and the data glove, the hand posture and finger movement are mapped to the real humanoid robot, so as to realize real-time control of the multi-degree-of-freedom humanoid robot to complete the task of dexterous operation of both hands, and can improve the intuitiveness of remote operation, simplify the operation, give the user a sense of intuition and immersion while reducing the burden on the user. It can also be used for data collection combined with deep learning for robot skill training. The detection method based on the tracker and the data glove is more stable and not easily affected by noise. Multiple bending sensors are used in the data glove to detect the bending degree of the finger, and there will be no problem of visual occlusion detection.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a method for remotely operating a humanoid robot by using a tracker and a data glove, comprising the following steps:

[0007] Based on the tracker capturing the position and posture of the hand in three-dimensional space, the tracker's posture information is used as the desired posture of the humanoid robot end, and the Cartesian motion control kinematics inverse solution method is used to solve the angle of each joint of the manipulator arm, and the manipulator arm of the humanoid robot is controlled to move to the desired position and posture;

[0008] Based on the movement of fingers captured by data gloves, the bending angles of the finger joints captured by the data gloves are mapped to the joints of the manipulator of the humanoid robot to obtain the joint angles of the manipulator. Based on the joint angles of the manipulator, each finger of the manipulator of the humanoid robot is controlled to move to a specified angle position.

[0009] As a preferred technical solution, the position and posture information of the tracker is used as the desired position and posture of the end of the humanoid robot, which is specifically expressed as:

[0010] x d =[x,y,z,α,β,γ]

[0011] Among them, x d represents the desired position of the humanoid robot terminal, x, y, z are the x-axis, y-axis, and z-axis position coordinates of the robot terminal coordinate system relative to the base coordinate system, α, β, γ are the rotation angles of the robot terminal coordinate system around the x-axis, y-axis, and z-axis of the base coordinate system, which are the roll angle, pitch angle, and yaw angle, respectively.

[0012] As a preferred technical solution, the angle of each joint of the robot arm is solved based on the inverse solution method of Cartesian motion control kinematics, specifically including:

[0013] Calculate the error between the current robot end position and the desired end position, expressed as:

[0014] ∈=x d -g(q)

[0015] Among them, ∈ represents the error, q represents the angle of each joint of the robot arm, and g(·) is the rotation matrix;

[0016] The error is regarded as the external force applied to the end of the robot and expressed as:

[0017]

[0018] Where f represents the external force, is the differential of the error with respect to time, K p and K d are the proportional coefficient and the differential coefficient;

[0019] Calculate the joint angular acceleration of the robot under the action of external force, expressed as:

[0020]

[0021] in, represents the angular acceleration of each joint of the robot, H represents the inertia matrix, and J represents the Jacobian matrix;

[0022] The joint angular acceleration of the robot arm is integrated twice to obtain the corresponding joint angle.

[0023] As a preferred technical solution, the captured finger joint bending angles are mapped to the manipulator joints of the humanoid robot to obtain the joint angles of the manipulator, specifically including:

[0024] The captured bending angle data of each finger joint is divided by the maximum detection range and converted into a value between 0 and 1. The normalized value is then multiplied by the maximum motion range of the dexterous hand joint and mapped to the joint angle of the robot.

[0025] The present invention also provides a system for remotely controlling a humanoid robot through a tracker and a data glove, comprising: a host computer, a tracker, a data glove, a main control computer, a mechanical arm motor controller, a mechanical hand motor controller, a humanoid robot mechanical arm, and a humanoid robot mechanical hand;

[0026] The host computer is used to obtain data from the tracker and the data gloves, and transmit the data to the main control computer;

[0027] The tracker is used to capture the position and posture of the hand in three-dimensional space;

[0028] The main control computer is used to use the position information of the tracker as the desired position of the end of the humanoid robot, solve each joint angle of the robot arm based on the Cartesian motion control kinematics inverse solution method, and transmit it to the motor controller of the robot arm;

[0029] The robot arm motor controller is used to control the robot arm motor, and control the humanoid robot robot arm to move to a desired position and posture based on each joint angle of the robot arm;

[0030] The data glove is used to capture the movement of fingers;

[0031] The main control computer is used to map the bending angles of the finger joints captured by the data gloves to the manipulator joints of the humanoid robot, obtain the joint angles of the manipulator, and transmit them to the manipulator motor controller;

[0032] The manipulator motor controller is used to control the manipulator motor, and controls each finger of the manipulator of the humanoid robot to move to a specified angle position based on the joint angle of the manipulator.

[0033] As a preferred technical solution, the main control computer is used to use the position and posture information of the tracker as the desired position and posture of the end of the humanoid robot, which is specifically expressed as:

[0034] x d=[x,y,z,α,β,γ]

[0035] Among them, x d represents the desired position of the humanoid robot terminal, x, y, z are the x-axis, y-axis, and z-axis position coordinates of the robot terminal coordinate system relative to the base coordinate system, α, β, γ are the rotation angles of the robot terminal coordinate system around the x-axis, y-axis, and z-axis of the base coordinate system, which are the roll angle, pitch angle, and yaw angle, respectively.

[0036] As a preferred technical solution, the Cartesian motion control kinematics inverse solution method is used to solve each joint angle of the robot arm, specifically including:

[0037] Calculate the error between the current robot end position and the desired end position, expressed as:

[0038] ∈=x d -g(q)

[0039] Among them, ∈ represents the error, q represents the angle of each joint of the robot arm, and g(·) is the rotation matrix;

[0040] The error is regarded as the external force applied to the end of the robot and expressed as:

[0041]

[0042] Where f represents the external force, is the differential of the error with respect to time, K p and K d are the proportional coefficient and the differential coefficient;

[0043] Calculate the joint angular acceleration of the robot under the action of external force, expressed as:

[0044]

[0045] in, represents the angular acceleration of each joint of the robot, H represents the inertia matrix, and J represents the Jacobian matrix;

[0046] The joint angular acceleration of the robot arm is integrated twice to obtain the corresponding joint angle.

[0047] As a preferred technical solution, the main control computer is used to map the bending angles of the finger joints captured by the data gloves to the manipulator joints of the humanoid robot to obtain the joint angles of the manipulator, specifically including:

[0048] The captured bending angle data of each finger joint is divided by the maximum detection range and converted into a value between 0 and 1. The normalized value is then multiplied by the maximum motion range of the dexterous hand joint and mapped to the joint angle of the robot.

[0049] As a preferred technical solution, the main control computer is connected to the robot arm motor controller via a CAN bus, and the main control computer is connected to the robot arm motor controller via a serial port.

[0050] As a preferred technical solution, the humanoid robot is also provided with a camera for collecting image information of the robot's working environment.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] (1) The present invention detects wrist posture based on a tracker, which solves the technical problem of obtaining the operator's wrist posture during remote operation. Compared with the existing vision-based (camera or VR glasses) method, the tracker-based method adopted by the present invention is not easily affected by noise and is not limited by the camera's field of view. Compared with the existing exoskeleton-based method, the tracker-based method adopted by the present invention brings less burden to the operator and achieves a more intuitive and accurate wrist posture detection effect.

[0053] (2) The present invention detects the finger bending angle based on data gloves, which solves the technical problem of obtaining the operator's finger bending angle during remote operation. Compared with the existing vision-based method, the data glove-based method of the present invention is not easily affected by noise and does not have the problem of finger occlusion, thereby achieving a more stable and accurate wrist joint angle detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of a method for remotely controlling a humanoid robot through a tracker and a data glove according to the present invention;

[0055] Figure 2 Schematic diagram of the overall architecture of the system for remotely controlling a humanoid robot through a tracker and a data glove according to the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] Example 1

[0058] like Figure 1 As shown, this embodiment provides a method for remotely operating a humanoid robot by using a tracker and a data glove, comprising the following steps:

[0059] The position and posture of the hand in three-dimensional space are captured by the Vive Tracker tracker, and mapped to the desired end position and posture of the end of the humanoid robot arm. The Cartesian motion control kinematic inverse solution method is used to solve the angle of each joint of the robot arm, and the robot arm is controlled to move to the desired position and posture;

[0060] In this embodiment, the base station emits an infrared scanning light, which is reflected by the Tracker worn on the user's wrist, and then the spatial position and rotation angle of the Tracker are determined. A receiver is connected to the host computer to receive the position information sent by the Vive Tracker. At the beginning, the position and angle of the Vive Tracker worn on the operator's wrist are mapped to the initial position of the end of the humanoid robot, which can bring a more intuitive operation experience; then, the posture information of the Vive Tracker is used as the expected posture of the robot end, expressed as x d =[x,y,z,α,β,γ], where x, y, z are the x-axis, y-axis, and z-axis position coordinates of the robot’s end coordinate system relative to the base coordinate system, α, β, and γ are the rotation angles of the robot’s end coordinate system around the x-axis, y-axis, and z-axis of the base coordinate system, which are the roll angle, pitch angle, and yaw angle, respectively. The joint angles of the humanoid robot can be expressed as q=[q1,q2,…,q n ], generally humanoid robots have 6 to 7 degrees of freedom, then n = 6 or n = 7, the corresponding end position and posture can be calculated according to the joint angle, expressed as x = g(q), where g(·) is the rotation matrix. The error ∈ = x between the current robot end posture and the desired end posture can be calculated d -g(q), taking this error as the external force applied to the end of the robot, it can be expressed as in is the differential of the error with respect to time, K p and K d are the proportional coefficient and differential coefficient, substituted into the calculation formula of joint angular acceleration: The angular acceleration of each joint under the action of f can be calculated Where H represents the inertia matrix, J represents the Jacobian matrix, and the angular acceleration of each joint After two more integrations, the corresponding joint angles can be obtained, which are sent to the robot for execution, thereby controlling the robot to track the movement of the operator's wrist to achieve the purpose of remote operation. Since this method does not involve the inversion of the rotation matrix, the solution speed is fast, which can ensure the intuitiveness and smoothness of remote operation.

[0061] In this embodiment, the data glove integrates multiple bending sensors. By wearing the data glove on the hand to capture the movement of the fingers, the captured bending angle of the finger joints is mapped to the dexterous hand of the humanoid robot. Since the bending angle measurement range of the data glove is inconsistent with the joint motion range of the dexterous hand, it is necessary to map the angle. First, the finger joint bending angle data is obtained from the supporting software of the data glove, expressed as p = [p1, p2, ..., p5]. Each joint data is divided by the maximum detection range to convert it into a value between 0 and 1. Then the normalized value is multiplied by the maximum motion range of the dexterous hand joint and mapped to the joint angle of the robot hand, expressed as p' = [p ′ 1,p ′ 2,…,p ′ 5], which is used to control the joints of the dexterous hand.

[0062] Example 2

[0063] like Figure 2 As shown, this embodiment provides a system for remotely controlling a humanoid robot through a tracker and a data glove, which is used to implement the method for remotely controlling a humanoid robot through a tracker and a data glove in the above embodiment 1. The system specifically includes: a host computer, a tracker, a data glove, a main control computer, a mechanical arm motor controller, a mechanical hand motor controller, a humanoid robot mechanical arm, and a humanoid robot mechanical hand;

[0064] In this embodiment, the host computer is used to obtain data from the tracker and the data glove and transmit it to the main control computer;

[0065] In this embodiment, the tracker is used to capture the position and posture of the hand in three-dimensional space;

[0066] In this embodiment, the main control computer is used to use the position information of the tracker as the desired position of the end of the humanoid robot, solve each joint angle of the robot arm based on the Cartesian motion control kinematics inverse solution method, and transmit it to the motor controller of the robot arm;

[0067] In this embodiment, the robot arm motor controller is used to control the robot arm motor, and control the humanoid robot robot arm to move to a desired position and posture based on each joint angle of the robot arm;

[0068] In this embodiment, a data glove is used to capture the movement of the finger;

[0069] In this embodiment, the main control computer is used to map the bending angles of the finger joints captured by the data gloves to the manipulator joints of the humanoid robot, obtain the joint angles of the manipulator, and transmit them to the manipulator motor controller;

[0070] In this embodiment, the manipulator motor controller is used to control the manipulator motor, and controls each finger of the manipulator of the humanoid robot to move to a specified angle position based on the joint angle of the manipulator.

[0071] In this embodiment, the main control computer motion motor driver solves the angle of each joint of the robot according to the wrist posture obtained from the tracker and sends it to the manipulator motor controller through the CAN bus, and sends the finger joint angle obtained from the data glove through the serial port to the manipulator motor controller. The manipulator motor controller and the manipulator motor controller control each motor according to the received control instructions to move the manipulator to the specified position and posture, and the manipulator fingers to the specified angle position. The data gloves can detect the user's finger bending degree in real time, avoid the occlusion problem of visual recognition, improve the recognition accuracy, and avoid safety problems caused by occlusion.

[0072] Specifically, the tracker is worn on the operator's wrist to capture the position and posture of the operator's wrist. The tracker's detection method based on optical positioning is more stable and less susceptible to noise. The data gloves are worn on the operator's hands. Multiple bending sensors are used in the data gloves to detect the degree of bending of the fingers. There will be no problem of visual occlusion detection. Overall, it brings less burden to the user while ensuring the accuracy of the detection results. In this embodiment, other optical positioning sensors can also be used to replace the tracker in this application, such as OptiTrack's motion capture system.

[0073] In this embodiment, the upper limbs of the humanoid robot include arms and hands, so the focus is on achieving control of the arms and hands. The robot arm generally has 6-7 degrees of freedom. In order to achieve the purpose of intuitive operation, the operator controls the end position and posture of the robot arm, uses a tracker to obtain the position and posture of the operator's wrist, and then uses the Cartesian motion control method to solve the angles of each motor, and then controls the robot to follow the operator's wrist movement. The humanoid robot of this embodiment includes two arms with seven degrees of freedom and two manipulators with five degrees of freedom. Each joint is driven by a motor. The bending degree of the operator's fingers is obtained using data gloves, and then the motor of the robot hand is controlled to move to the corresponding angle. The head of the humanoid robot is equipped with a camera for transmitting the collected image information back to the operator so that the operator can observe the working environment of the robot.

[0074] In this embodiment, the tracker is used to identify the position and posture of the operator's wrist and the kinematic inverse solution process, thereby ensuring the intuitiveness and smoothness of the robot operation. The data gloves are used to identify the bending angle of the operator's fingers, thereby avoiding the occlusion and instability problems caused by the visual recognition method, and ensuring the accurate recognition and tracking of finger movements.

[0075] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for remotely operating a humanoid robot by means of a tracker and a data glove, characterized in that: The steps include: Based on the tracker capturing the position and posture of the hand in three-dimensional space, the tracker's posture information is used as the desired posture of the humanoid robot end, and the Cartesian motion control kinematics inverse solution method is used to solve the angle of each joint of the manipulator arm, and the manipulator arm of the humanoid robot is controlled to move to the desired position and posture; Based on the movement of fingers captured by data gloves, the bending angles of the finger joints captured by the data gloves are mapped to the joints of the manipulator of the humanoid robot to obtain the joint angles of the manipulator. Based on the joint angles of the manipulator, each finger of the manipulator of the humanoid robot is controlled to move to a specified angle position.

2. The method for remotely controlling a humanoid robot by a tracker and a data glove according to claim 1, characterized in that: The tracker's posture information is used as the desired posture of the humanoid robot terminal, which is specifically expressed as: x d =[x, y, z, α, β, γ] Among them, x d represents the desired position of the humanoid robot terminal, x, y, z are the x-axis, y-axis, and z-axis position coordinates of the robot terminal coordinate system relative to the base coordinate system, α, β, γ are the rotation angles of the robot terminal coordinate system around the x-axis, y-axis, and z-axis of the base coordinate system, which are the roll angle, pitch angle, and yaw angle, respectively.

3. The method for remotely controlling a humanoid robot by a tracker and a data glove according to claim 2, characterized in that: Based on the Cartesian motion control kinematics inverse solution method, the angle of each joint of the robot arm is solved, including: Calculate the error between the current robot end position and the desired end position, expressed as: ∈=x d -g(q) Among them, ∈ represents the error, q represents the angle of each joint of the robot arm, and g(·) is the rotation matrix; The error is regarded as the external force applied to the end of the robot and expressed as: Where f represents the external force, is the differential of the error with respect to time, K p and K d are the proportional coefficient and the differential coefficient; Calculate the joint angular acceleration of the robot under the action of external force, expressed as: in, represents the angular acceleration of each joint of the robot, H represents the inertia matrix, and J represents the Jacobian matrix; The joint angular acceleration of the robot arm is integrated twice to obtain the corresponding joint angle.

4. The method of remotely operating a humanoid robot by a tracker and a data glove according to claim 1, characterized in that: The captured finger joint bending angles are mapped to the manipulator joints of the humanoid robot to obtain the joint angles of the manipulator, including: The captured bending angle data of each finger joint is divided by the maximum detection range and converted into a value between 0 and 1. The normalized value is then multiplied by the maximum motion range of the dexterous hand joint and mapped to the joint angle of the robot.

5. A system for remotely controlling a humanoid robot through a tracker and a data glove, characterized in that: include: Host computer, tracker, data gloves, main control computer, robotic arm motor controller, robotic hand motor controller, humanoid robot robotic arm, humanoid robot manipulator; The host computer is used to obtain data from the tracker and the data gloves, and transmit the data to the main control computer; The tracker is used to capture the position and posture of the hand in three-dimensional space; The main control computer is used to use the position information of the tracker as the desired position of the end of the humanoid robot, solve each joint angle of the robot arm based on the Cartesian motion control kinematics inverse solution method, and transmit it to the motor controller of the robot arm; The robot arm motor controller is used to control the robot arm motor, and control the humanoid robot robot arm to move to a desired position and posture based on each joint angle of the robot arm; The data glove is used to capture the movement of fingers; The main control computer is used to map the bending angles of the finger joints captured by the data gloves to the manipulator joints of the humanoid robot, obtain the joint angles of the manipulator, and transmit them to the manipulator motor controller; The manipulator motor controller is used to control the manipulator motor, and controls each finger of the manipulator of the humanoid robot to move to a specified angle position based on the joint angle of the manipulator.

6. The system for remotely controlling a humanoid robot through a tracker and a data glove according to claim 5, characterized in that: The main control computer is used to use the tracker's position information as the desired position of the humanoid robot terminal, which is specifically expressed as: x d =[x, y, z, α, β, γ] Among them, x d represents the desired position of the humanoid robot terminal, x, y, z are the x-axis, y-axis, and z-axis position coordinates of the robot terminal coordinate system relative to the base coordinate system, α, β, γ are the rotation angles of the robot terminal coordinate system around the x-axis, y-axis, and z-axis of the base coordinate system, which are the roll angle, pitch angle, and yaw angle, respectively.

7. The system for remotely controlling a humanoid robot through a tracker and a data glove according to claim 6, characterized in that: The method for solving each joint angle of the robot arm based on the Cartesian motion control kinematics inverse solution specifically includes: Calculate the error between the current robot end position and the desired end position, expressed as: ∈=x d -g(q) Among them, ∈ represents the error, q represents the angle of each joint of the robot arm, and g(·) is the rotation matrix; The error is regarded as the external force applied to the end of the robot and expressed as: Where f represents the external force, is the differential of the error with respect to time, K p and K d are the proportional coefficient and the differential coefficient; Calculate the joint angular acceleration of the robot under the action of external force, expressed as: in, represents the angular acceleration of each joint of the robot, H represents the inertia matrix, and J represents the Jacobian matrix; The joint angular acceleration of the robot arm is integrated twice to obtain the corresponding joint angle.

8. The system for remotely controlling a humanoid robot through a tracker and a data glove according to claim 5, characterized in that: The main control computer is used to map the bending angles of the finger joints captured by the data gloves to the manipulator joints of the humanoid robot to obtain the joint angles of the manipulator, specifically including: The captured bending angle data of each finger joint is divided by the maximum detection range and converted into a value between 0 and 1. The normalized value is then multiplied by the maximum motion range of the dexterous hand joint and mapped to the joint angle of the robot.

9. The system for remotely controlling a humanoid robot through a tracker and a data glove according to claim 5, characterized in that: The main control computer is connected to the mechanical arm motor controller via a CAN bus, and the main control computer is connected to the mechanical hand motor controller via a serial port.

10. The system for remotely controlling a humanoid robot through a tracker and a data glove according to claim 5, characterized in that: The humanoid robot is also equipped with a camera for collecting image information of the robot's working environment.

Images