A data glove for a teleoperation system and a control method thereof

Abstract

This invention discloses a data glove for a teleoperation system and its control method. The data glove body is worn on the user's hand, and a signal transmission network is constructed between hand, wrist, and arm data units mounted on the glove body. Complete kinematic chain data from the shoulder joint to the finger joints is collected by each data unit and transmitted through the signal transmission network, including quaternion pose data and gesture data. The quaternion pose data is processed to obtain a spatial pose matrix, and the gesture data is processed using a gesture mapping algorithm to obtain a gesture sequence number. A master-slave mapping algorithm maps the workspace of the glove to the workspace of the slave device, thereby enabling the master device to freely control the slave robotic arm and execute specific complex task processes. This invention, through modular sensor layout and algorithm optimization, achieves precise, intuitive, and natural control of the slave robotic arm by the operator, reducing operational complexity and improving the user experience.

Description

Technical Field

[0001] This invention relates to an operating glove, and to the fields of robot teleoperation, human-computer interaction and intelligent control technology, specifically to a data glove for a teleoperation system and its control method. Background Technology

[0002] With the continuous development of robotics technology, teleoperation systems have been widely used in industries such as manufacturing, medicine, military, aerospace, and deep-sea exploration. Teleoperation systems allow operators to control robots remotely, enabling remote operations in hazardous environments or scenarios where direct contact is difficult. However, traditional teleoperation methods typically rely on joysticks, buttons, or other mechanical control devices. These devices are complex to operate, lack intuitiveness, and have limited control precision, making them unsuitable for complex tasks. Furthermore, large systems often contain numerous slave devices, and simultaneously considering teleoperation operations for multiple devices leads to system redundancy and increased costs. In recent years, with the development of sensor technology and wearable devices, data gloves have gradually gained attention as a new type of teleoperation input device. Data gloves integrate sensors to capture real-time hand movement information and convert it into control commands, enabling natural and intuitive control of robotic equipment. However, existing data glove technology still suffers from problems such as unreasonable sensor layout, imperfect posture representation methods, and insufficient communication and algorithm design, limiting its application in high-precision teleoperation tasks. Traditional remote-controlled devices mostly use mechanical control tools such as joysticks and buttons, which are relatively complex to operate and lack intuitiveness, and the control precision is also difficult to meet the needs of complex tasks. Summary of the Invention

[0003] To address the problems existing in the background art, this invention provides a data glove for a teleoperation system and its control method. Through a reasonable sensor layout, optimized attitude representation method, and a special teleoperation mode design, this invention achieves precise, intuitive, and natural control of multiple slave devices from a single master data glove, reducing operational complexity and improving the user experience.

[0004] The technical solution adopted in this invention is:

[0005] I. A data glove for a teleoperation system, comprising:

[0006] The glove itself.

[0007] A hand data unit, mounted on the glove body and at the hand position on the human body, is used to transmit quaternion attitude data of the hand to one or more slave devices for remote operation control at the end of the slave device.

[0008] A wrist data unit, mounted on the glove body and positioned at the wrist of the human body, is used to transmit wrist quaternion attitude data to one or more slave devices for remote operation control in the slave device.

[0009] An arm data unit, mounted on the glove body and positioned at the arm of the human body, is used to transmit quaternion attitude data of the arm to one or more slave devices for remote operation control at the root of the slave devices.

[0010] The glove body includes a fingerless base, a first elastic strap base, and a second elastic strap base. The fingerless base is worn on the hand during remote operation, the first elastic strap base is worn on the wrist, and the second elastic strap base is worn on the upper arm. The hand data unit includes a first integrated printed circuit board (PCB) and a Flex Sensor bending sensor group. The first PCB is mounted on the back of the hand of the fingerless base. The first PCB houses a first core control chip, a first power management module for power supply, and a second inertial sensor (MPU). The bending sensor group includes five Flex Sensors. Five bending sensors are installed on the back of the five fingers of the finger-split base. All five bending sensors and the second inertial sensor are electrically connected to the first core control chip. The wrist data unit is the first inertial sensor, installed on the outer side of the first elastic band base and electrically connected to the first core control chip. The first core control chip is wirelessly connected to an external master PC via Wi-Fi. The arm data unit is the second integrated printed circuit board (PCB), installed on the outer side of the second elastic band base. The second PCB houses a second power management module, a second core control chip, and a third inertial sensor. The second power management module and the third inertial sensor are electrically connected to the second core control chip. The second core control chip is wirelessly connected to an external master PC via Wi-Fi. The master PC is wirelessly connected to the slave device.

[0011] II. A teleoperation control method for a data glove used in a teleoperation system, comprising:

[0012] First, the finger-type base, the first elastic band base, and the second elastic band base of the glove body are respectively placed on the human hand, wrist, and upper arm. Then, the main PC is used as the master node, and the first and second integrated printed circuit boards (PCBs) are used as child nodes. A wireless communication network is established between the master node and the two child nodes via Wi-Fi using the TCP / IP protocol. When the human hand, wrist, and upper arm perform preset movements, quaternion pose data of the human hand, wrist, and upper arm are collected by various inertial sensors and wirelessly transmitted to the master PC via the core main control chip. At the same time, gesture data of each finger is collected by various bending sensors and wirelessly transmitted to the master PC via the first core main control chip. The master PC performs preliminary processing on the quaternion pose data of the human hand, wrist, and upper arm to obtain a spatial pose matrix. The system performs kinematic modeling of the glove and obtains gesture numbers by initially processing the gesture data of each finger using a gesture mapping algorithm. When multiple slave devices need to be controlled, the master PC performs a master-slave heterogeneous mapping of the spatial pose matrix to obtain the target pose of the core focus point of each slave device as the control input. This is transmitted to the data recording unit for recording via the host computer program on the master PC. The core focus point can be the end effector of the robotic arm, etc. The system then obtains the control data of each slave device through an inverse kinematics algorithm. Finally, the control data and gesture numbers are transmitted to each slave device for remote operation control. This data is also transmitted to the data recording unit for recording via the host computer program on the master PC. The slave device can be a five-degree-of-freedom hydraulic robotic arm. The core focus point is the end effector of the five-degree-of-freedom hydraulic robotic arm, and the control data is the angle data of each joint.

[0013] The main PC terminal uses the quaternion pose data of the human hand, wrist, and upper arm collected by various inertial sensors to obtain the spatial pose matrix of the hand relative to the shoulder joint through a human hand kinematics model. as follows:

[0014]

[0015] in, Represents a kinematic model of the human hand joints; , and These represent the quaternion attitude data read by each inertial sensor; , and These represent the length of the upper arm, the length of the forearm, and the distance from the wrist to the palm, respectively.

[0016] The main PC terminal performs preliminary processing on the gesture data of each finger using a gesture mapping algorithm to obtain a gesture sequence number. The finger gesture data is finger flexion and extension data, which is obtained by the first core main control chip through analog-to-digital converter (ADC) acquiring the voltage of each flexion sensor. In specific processing, for the finger flexion and extension data of each finger, the finger flexion and extension data that is greater than a preset flexion and extension threshold is selected. The binary bits of the finger are set to 0, and the finger flexion-extension data is less than or equal to a preset flexion-extension threshold. The binary bits of each finger are set to 1, ultimately constructing a five-bit binary number for each hand as the gesture sequence. The details are as follows:

[0017]

[0018]

[0019] in, Indicates the first The binary bits of one finger; Indicates the first Data on the flexion and extension of each finger;

[0020] Each gesture number corresponds to a preset motion task on the slave device.

[0021] The master-slave heterogeneous mapping is described in detail below:

[0022]

[0023]

[0024]

[0025] in, This indicates the target pose of the core concern of the slave device; This represents the spatial pose matrix of the master-end reconstructed object, which represents the core focus of the slave device. Specifically, the master-end reconstructed object is the reconstructed object of the core focus of the slave device on the master end, which is reconstructed into a unified object with pose information (which can be represented as a small cube). It does not focus on the specific form of the slave device, but only on the pose information attributes of the core focus of the slave device. This represents pose data representing the core concerns of the slave device; This represents the workspace representing the core concerns of the end device; This indicates that the emission originates from the coordinate origin and the direction of emission is... Rays and workspace surface The intersection point is determined with the zero point of the world coordinate system calibrated by the slave device as the coordinate zero point, and the scaling scale is consistent with the master device's world coordinate system. It is determined by the characteristics of the slave device itself.

[0026] This allows the pose information of the object to be reconstructed from the core focus of the slave device by the master device. The mapping is used to define the target pose of the core concern of the slave device.

[0027] The control conditions for the data gloves during remote operation are as follows:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] in, The spatial pose matrix representing the hand of the data glove. and These represent the hand pose rotation matrix and spatial position vector of the data glove, respectively; This represents the spatial pose matrix of the core concern reconstructed object of the i-th slave device. and Let represent the pose rotation matrix and spatial position vector of the i-th slave device, respectively; This represents the relative pose of the hand of the i-th slave device with respect to the object. and Let represent the orientation rotation matrix and spatial position vector of the i-th slave device's hand relative to the object, respectively; This represents the logic function for fetching data. A value of 1 indicates that the core concerns of the slave device, the reconfigurable objects of the master device, can be captured. A value of 0 indicates that the core concern of the slave device, the reconstructed object of the master device, cannot be captured; t represents time; A five-bit binary number representing a single hand gesture is used as the gesture number; This indicates the maximum distance between the core focus main-end reconstructed object and the data glove when it can be crawled; The direction vector directly in front of the palm of the data glove.

[0035] III. A teleoperation system based on data gloves:

[0036] The system includes a master unit, a master-slave communication unit, and slave units. The master unit includes a data glove and a master PC. The master-slave communication unit is the Robot Operating System (ROS). The slave units include multiple slave devices. The data glove collects quaternion pose data of the human hand, wrist, and upper arm in real time, as well as gesture data of each finger, and wirelessly transmits it to the master PC for preliminary processing. After processing the quaternion pose data, a spatial pose matrix is ​​obtained. The gesture data of each finger is processed by a gesture mapping algorithm to obtain a gesture number. Then, the spatial pose matrix is ​​subjected to a master-slave heterogeneous mapping to obtain the target pose of the core focus point of each slave device as the control input. The control data of each slave device is then obtained through an inverse kinematics algorithm. Finally, the control data and gesture numbers are transmitted to each slave device through the Robot Operating System (ROS) for remote operation control.

[0037] This invention, based on inertial sensors, bending sensors, and a fully programmable system-on-a-chip, designs a hardware system capable of acquiring complete kinematic chain data from the shoulder joint to the finger joints. By wearing a glove on the operator's hand, posture measurement devices distributed across the upper arm, forearm, and back of the hand capture the spatial pose data of each joint; simultaneously, finger flexion-extension measurement devices distributed on each finger accurately measure the bending angle of the finger joints. The spatial pose information of the palm is calculated using a human hand joint kinematic model, and a master-slave mapping algorithm is used to map the workspace of the glove to the workspace of the slave robotic arm's end effector, thereby achieving free control of the slave robotic arm from the master end. Furthermore, through a gesture mapping algorithm, the slave robotic arm can execute specific complex task flows.

[0038] The beneficial effects of this invention are:

[0039] This invention, through the design of a data glove, utilizes posture measurement devices and finger flexion / extension measurement devices distributed across the upper arm, forearm, and back of the hand to accurately capture complete kinematic chain data from the shoulder joint to the finger joints. This comprehensive motion capture method more accurately reflects the operator's natural hand movements. Furthermore, through master-slave heterogeneous mapping algorithms and gesture mapping algorithms, the operator's actions are translated into control commands for the robotic arm, thereby simplifying the operation process and improving control precision and efficiency. Simultaneously, the use of quaternions as the posture representation method effectively avoids the gimbal lock problem that may occur during human hand movements, ensuring the stability and reliability of the system.

[0040] In terms of hardware design and communication protocol, this invention employs a fingerless flexible fabric glove base and an elastic strap base, combined with a modular layout integrating a PCB board and sensors, facilitating assembly and maintenance while also supporting functional expansion according to actual needs. A master-slave signal transmission network built via Wi-Fi wireless communication and TCP / IP protocol enables rapid and stable transmission of hand posture and gesture information to the master PC, and further to the slave robotic arm, ensuring the system can reflect the operator's intentions in real time. Furthermore, the entire hardware system is powered by a lithium battery or an external USB port, featuring low power consumption and high portability, making it suitable for extended wear and various application scenarios.

[0041] Furthermore, this invention is applicable to various remote operation scenarios, including industrial automation, medical robots, military equipment, aerospace, and deep-sea exploration. By setting the core focus of the slave device on the reconstructed master device, it is possible to easily and quickly match various different slave devices, including robotic arms or various mobile robots. It allows for the same master device to control different slave devices within the same framework, demonstrating good applicability. Simultaneously, through a gesture mapping algorithm, the data glove can recognize various preset gestures and map them to specific complex task flows of the slave devices, providing a convenient solution for executing complex tasks.

[0042] In summary, this invention, through modular sensor layout and algorithm optimization, enables operators to achieve precise, intuitive, and natural control of the slave robotic arm, reducing operational complexity and improving the user experience. Attached Figure Description

[0043] Figure 1 This is a design assembly drawing of the data gloves of the present invention;

[0044] Figure 2 This is a first integrated printed circuit board (PCB) design diagram of the data glove of the present invention;

[0045] Figure 3 This is a PCB design diagram of the second integrated printed circuit board for the data glove of the present invention;

[0046] Figure 4 This is a diagram of the overall system control framework of the present invention;

[0047] Figure 5 This is a diagram showing the calibration results of the MPU6050 six-axis motion sensor of the present invention;

[0048] Figure 6 This is a schematic diagram of a five-degree-of-freedom hydraulic robotic arm used in a specific embodiment of the present invention.

[0049] In the figure: 1. Glove body; 1.1. Finger-separated base; 1.2. First elastic strap base; 1.3. Second elastic strap base; 2. First integrated printed circuit board (PCB); 3. Second integrated printed circuit board (PCB); 4. First inertial sensor; 5. Bending sensor group; 6. Bending sensor; 7. First core main control chip; 8. First power management module; 9. Second inertial sensor; 10. Second power management module; 11. Second core main control chip; 12. Third inertial sensor. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0051] In a specific embodiment of the present invention, a five-degree-of-freedom hydraulic robotic arm is used as the slave device, and the end gripper of the hydraulic robotic arm is used as the core focus of the slave device. That is, by designing a remote operation control method for the core focus of the slave device, a master-slave control effect is achieved.

[0052] The data glove for teleoperation system and its control method of the present invention include a hand posture capture part and a teleoperation master-slave control algorithm part; the hand posture capture part includes a glove body, an arm posture measurement device, a finger flexion and extension measurement device, and a signal processing and forwarding module; the teleoperation master-slave control algorithm part includes a master-slave end signal transmission network design and a master-slave heterogeneous mapping algorithm design.

[0053] like Figure 1 As shown, the data glove for a teleoperation system of the present invention includes a glove body 1, a hand data unit, a wrist data unit, and an arm data unit. The hand data unit is mounted on the glove body 1 and at the hand position of the human body, and is used to transmit quaternion posture data of the hand to one or more slave devices for remote operation control at the end of the slave device. The wrist data unit is mounted on the glove body 1 and at the wrist position of the human body, and is used to transmit quaternion posture data of the wrist to one or more slave devices for remote operation control at the middle of the slave device. The arm data unit is mounted on the glove body 1 and at the arm position of the human body, and is used to transmit quaternion posture data of the arm to one or more slave devices for remote operation control at the root of the slave device.

[0054] The glove body 1 includes a fingerless base 1.1, a first elastic strap base 1.2, and a second elastic strap base 1.3. The fingerless base 1.1 is worn on the hand during remote operation, the first elastic strap base 1.2 is worn on the wrist during remote operation, and the second elastic strap base 1.3 is worn on the upper arm during remote operation. The hand data unit includes a first integrated printed circuit board PCB2 and a Flex Sensor bending sensor group 5. The first integrated printed circuit board PCB2 is installed on the back of the hand of the fingerless base 1.1, such as... Figure 2 As shown, the first integrated printed circuit board PCB2 is equipped with a first core main control chip 7, a first power management module 8 for power supply, and a second inertial sensor MPU 9. The bending sensor group 5 includes five Flex Sensors 6, which are respectively installed on the back of the five fingers of the finger-split base 1.1. The five bending sensors 6 and the second inertial sensor 9 are all electrically connected to the first core main control chip 7. The wrist data unit is the first inertial sensor 4, which is installed on the outer side of the first elastic band base 1.2 and electrically connected to the first core main control chip 7. The first core main control chip 7 is wirelessly connected to an external master PC via WIFI. The arm data unit is the second integrated printed circuit board PCB3, which is installed on the outer side of the second elastic band base 1.3, as shown in the figure. Figure 3 As shown, a second power management module 10, a second core main control chip 11, and a third inertial sensor 12 are installed on the second integrated printed circuit board PCB3. The second power management module 10 and the third inertial sensor 12 are both electrically connected to the second core main control chip 11. The second core main control chip 11 is wirelessly connected to an external master PC via WIFI. The master PC is wirelessly connected to the slave device.

[0055] In specific implementation, the finger-type substrate 1.1 is made of flexible polyester fabric, and the first elastic band-type substrate 1.2 and the second elastic band-type substrate 1.3 are made of elastic Velcro. Inertial sensors 4, 9, and 12 all use MPU6050 six-axis motion sensor modules, and each bending sensor 6 is a flex2.2 model. Core control chips 7 and 11 both use the ESP32-C3SuperMini development board, and power management modules 8 and 10 both use 5V charging and discharging integrated modules. The effective input voltage of power management modules 8 and 10 is 5V, the effective voltage input from power management modules 8 and 10 to core control chips 7 and 11 is 5V, and the effective output voltage of core control chips 7 and 11 is 3.3V. The signals from each bending sensor 6 are read through the general purpose input / output (GPIO) port of the first core control chip 7 using analog-to-digital (ADC) conversion. The signals from each inertial sensor 4, 9, and 12 are transmitted through the integrated circuit bus IIC (Inter-Integrated Circuit). The circuit is read by the core main control chips 7 and 11, and the power management modules 8 and 10 are powered by lithium batteries or external USB.

[0056] In the specific design of the glove body 1, a first integrated printed circuit board (PCB) 2 is fixed to the back of the palm using Velcro. The loop side of the Velcro is sewn onto the finger-type base 1.1 of the glove body 1, and the hook side is glued to the back of the first integrated printed circuit board PCB 2 with hot melt adhesive. The size of the hook side is the same as that of the first integrated printed circuit board PCB 2. A first core main control chip 7, a first power management module 8, and a second inertial sensor 9 are soldered onto the first integrated printed circuit board PCB 2, and a connection interface for connecting to each bending sensor 6 of the bending sensor group 5 is reserved. A narrow pocket is sewn onto the back of each finger to hold each bending sensor 6, and each bending sensor 6 is connected to the reserved interface on the first integrated printed circuit board PCB 2. A first inertial sensor 4 is fixed to the first elastic strap base 1.2 of the forearm using Velcro, and the first inertial sensor 4 is connected to the first integrated printed circuit board PCB 2 via a wire. A second integrated printed circuit board (PCB) 3 is fixed on the second elastic strap-type base 1.3 located in the upper arm. A second power management module 10, a second core control chip 11, and a third inertial sensor 12 are soldered onto the second PCB 3. The core control chips 7 and 11 on the first and second PCBs have Wi-Fi functionality for wireless signal transmission. The entire hardware system is powered by a lithium battery or an external USB port. The first and second PCBs of this invention are self-designed FR-4 circuit boards, smaller than the size of an adult's hand, reducing the feeling of foreign objects when wearing them.

[0057] Inertial sensors 4, 9, and 12 are MPU6050 six-axis motion sensor modules, model GY-521; bending sensor 6 is a flex sensor, model flex2.2; core main control chips 7 and 11 are ESP32-C3. The ESP32-C3 is a high-performance, low-power system-on-a-chip (SoC) equipped with a RISC-V 32-bit single-core processor with a working frequency of up to 160 MHz, possessing powerful computing capabilities. It has 400 KB of built-in SRAM and 384 KB of ROM, and also supports multiple external Flash memory connections via SPI, DualSPI, Quad SPI, and QPI interfaces to meet the storage needs of different devices. In addition, the ESP32-C3 integrates 2.4 GHz Wi-Fi and Bluetooth 5 (LE) functions, supports IEEE 802.11 b / g / n protocols, has multiple operating modes and advanced security features, and can achieve efficient and secure wireless communication. Its module circuit board is a SuperMini development board with a size of 18mm*22.52mm, which meets the size requirements of this invention.

[0058] The calibration methods for the MPU6050 six-axis motion sensor module and the Flex Sensor bending sensor are as follows:

[0059] 1) Calibration method for the MPU6050 six-axis motion sensor module: First, place the MPU6050 module on a horizontal and stable surface, keeping it stationary to obtain an initial offset. Then, sample the output angle data multiple times, calculate the average value for each axis, and use this as the offset. Subtract these offsets from subsequent measurements. After calibration, verify that the sensor output meets expectations, ensuring the error is within acceptable limits. Figure 5 The image shows the calibration results of the MPU6050 six-axis motion sensor. After calibration, data was recorded for 10 minutes. It can be seen that the data output as quaternions is only... and There is a deviation of 0.01 in the direction, which meets the actual use requirements.

[0060] 2) Calibration method for Flex Sensor bending sensor: Record the sensor's output voltage value with the sensor in an unbent state. As a reference value; mount the sensor on a glove, put on the glove and clench your fist, then measure the output voltage value at this time. ;Pick and average As a reference value, a curvature greater than the set value represents 0, and less than or equal to the set value represents 1. In practice, to calibrate the Flex Sensor bending sensor, its output voltage is first measured when the sensor is not bent. Then, attach it to the glove and clench your fist, measuring the output voltage at this point. .calculate and average As a reference value, it is used for subsequent bending state judgment. After calibration, a static test was conducted to keep the sensor stationary and measure the stability of the output voltage. A dynamic test was also performed, involving continuous bending and stretching of the sensor to observe whether the output voltage changed with the bending angle, thus verifying the sensor's performance. Test results show that the sensor exhibits good stability and responsiveness under both static and dynamic conditions, and can accurately distinguish bending states greater than [a certain value]. The output is 0, less than or equal to The output is 1, and there is no significant drift during long-term operation, which verifies its reliability and stability.

[0061] The data glove uses quaternions as the posture representation method for each joint of the glove, which can avoid the gimbal lock problem that may occur during measurement in the process of human hand movement.

[0062] like Figure 4As shown, the teleoperation control method for the data glove used in the teleoperation system of the present invention is as follows:

[0063] Assemble the data gloves: Wear the fingerless base 1.1 on your hand. Secure the first elastic strap base 1.2 to the forearm near the wrist, and the second elastic strap base 1.3 to the middle of the upper arm. Secure the first integrated printed circuit board PCB 2 to the back of your hand with Velcro. Secure the first inertial sensor 4 in the forearm to the first elastic strap base 1.2 with Velcro, and the second integrated printed circuit board PCB 3 to the second elastic strap base 1.3 with Velcro. Insert five bending sensors 6 into the narrow pockets on the back of each finger of the fingerless base 1.1. Connect the bending sensors 6 and the inertial sensors 4 in the forearm to the first integrated printed circuit board PCB 2 with wires.

[0064] Signal processing and transmission flow: The master PC acts as the master node, and the first integrated printed circuit board (PCB2) and the second integrated printed circuit board (PCB3) act as child nodes. A wireless communication network is constructed using Wi-Fi and TCP / IP protocols. The first and second integrated printed circuit boards (PCB2 and PCB3) directly transmit the acquired pose quaternion signals of the palm, forearm, and upper arm to the master PC. The master PC then calculates the palm's pose information relative to the shoulder joint using a human hand kinematics model. The finger flexion and extension information acquired by the first integrated printed circuit board (PCB2) is preprocessed locally; if the flexion exceeds a set value... A value of 0 is used to represent a gesture, and a value less than or equal to a set value is used to represent a gesture of 1. A five-bit binary number is constructed for each hand gesture, which represents the gesture sequence number. This gesture sequence number is then sent to the host PC. This method can represent up to 32 specific tasks while reducing data transmission volume.

[0065] In specific control, firstly, the finger-type base 1.1, the first elastic band base 1.2, and the second elastic band base 1.3 of the glove body 1 are respectively placed on the human hand, wrist, and upper arm. Then, the main PC is used as the master node, and the first integrated printed circuit board PCB2 and the second integrated printed circuit board PCB3 are used as child nodes. A wireless communication network is established between the master node and the two child nodes via Wi-Fi network and TCP / IP protocol. When the human hand, wrist, and upper arm perform preset movements, the quaternion posture data of the human hand, wrist, and upper arm are collected by each inertial sensor 4, 9, and 12 and wirelessly transmitted to the master PC via the core main control chips 7 and 11. At the same time, the gesture data of each finger is collected by each bending sensor 6 and wirelessly transmitted to the master PC via the first core main control chip 7. The master PC processes the quaternion posture data of the human hand, wrist, and upper arm. After initial processing, a spatial pose matrix is ​​obtained for glove kinematic modeling. A gesture mapping algorithm is then used to preliminarily process the gesture data of each finger to obtain gesture numbers. When multiple slave devices need to be controlled, the master PC performs a master-slave heterogeneous mapping of the spatial pose matrix to obtain the target pose of the core focus point of each slave device as control input. This is simultaneously transmitted to the data recording unit via the host computer program on the master PC for recording. The core focus point can be the end effector of the robotic arm, etc. Then, an inverse kinematics algorithm is used to obtain the control data of each slave device. Finally, the control data and gesture numbers are transmitted to each slave device for remote operation control. This data is also transmitted to the data recording unit via the host computer program on the master PC for recording. Specifically, a five-degree-of-freedom hydraulic robotic arm can be used as the slave device, with the core focus point being the end effector of the five-degree-of-freedom hydraulic robotic arm, and the control data specifically being the angle data of each joint.

[0066] The main PC uses the quaternion pose data of the human hand, wrist, and upper arm collected by inertial sensors 4, 9, and 12 to obtain the spatial pose matrix of the hand relative to the shoulder joint through a human hand kinematics model. as follows:

[0067]

[0068] in, Represents a kinematic model of the human hand joints; , and These represent the quaternion attitude data read by each inertial sensor (4, 9, 12). , and These represent the length of the upper arm, the length of the forearm, and the distance from the wrist to the palm, respectively.

[0069] The main PC terminal performs preliminary processing on the gesture data of each finger using a gesture mapping algorithm to obtain the gesture sequence number. The finger gesture data is finger flexion and extension data, which is obtained by the first core main control chip 7 through analog-to-digital converter (ADC) to collect the voltage of each flexion sensor 6. In specific processing, for the finger flexion and extension data of each finger, the finger flexion and extension data that is greater than a preset flexion and extension threshold is selected. The binary bits of the finger are set to 0, and the finger flexion-extension data is less than or equal to a preset flexion-extension threshold. The binary bits of each finger are set to 1, ultimately constructing a five-bit binary number for each hand as the gesture sequence. The details are as follows:

[0070]

[0071]

[0072] in, Indicates the first The binary bits of each finger These represent the thumb, index finger, middle finger, ring finger, and little finger, respectively. Indicates the first Data on the flexion and extension of each finger.

[0073] Each gesture number corresponds to a preset motion task of the slave device. By defining a five-bit binary number for each hand, a specific task of the slave device can be clearly defined. This task is determined by the slave device's operation and can represent up to 32 specific tasks while reducing data transmission volume.

[0074] The master-slave heterogeneous algorithm in the control method is as follows:

[0075]

[0076] in, This represents the expected trajectory of the mapped slave end. , , These represent the three-axis positional quantities of the palm in the Cartesian coordinate system with the shoulder joint as the origin; This represents the expected trajectory of the master end before mapping. , , These represent the three-axis position quantities of the end effector of the slave robot arm with the robot arm base as the origin in the Cartesian coordinate system; and These are the first and second adjustment factors, respectively; The rotation matrix for the attitude is expressed as follows:

[0077]

[0078] The master-end working space revolves around the coordinate system of the slave-end hydraulic robotic arm. Axis rotation angle.

[0079] The final master-slave heterogeneous mapping is constructed as follows:

[0080]

[0081]

[0082]

[0083] in, This indicates the target pose of the core concern of the slave device; This represents the spatial pose matrix of the master-end reconstructed object, which represents the core focus of the slave device. Specifically, the master-end reconstructed object is the reconstructed object of the core focus of the slave device on the master end, which is reconstructed into a unified object with pose information (which can be represented as a small cube). It does not focus on the specific form of the slave device, but only on the pose information attributes of the core focus of the slave device. This represents pose data representing the core concerns of the slave device; This represents the workspace representing the core concerns of the end device; This indicates that the emission originates from the coordinate origin and the direction of emission is... Rays and workspace surface The intersection point is determined with the zero point of the world coordinate system calibrated by the slave device as the coordinate zero point, and the scaling scale is consistent with the master device's world coordinate system. It is determined by the characteristics of the slave device itself.

[0084] This allows the pose information of the object to be reconstructed from the core focus of the slave device by the master device. The mapping is used to define the target pose of the core concern of the slave device.

[0085] The control conditions for data gloves during remote operation are as follows:

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] in, The spatial pose matrix representing the hand of the data glove. and These represent the hand pose rotation matrix and spatial position vector of the data glove, respectively. , ; This represents the spatial pose matrix of the core concern reconstructed object of the i-th slave device. and Let the rotation matrix and spatial position vector of the hand of the i-th slave device be represented respectively. , , ; This represents the relative pose of the hand of the i-th slave device with respect to the object. and Let represent the rotation matrix and spatial position vector of the hand of the i-th slave device relative to the object, respectively. , ; This represents the logic function for fetching data. A value of 1 indicates that the core concerns of the slave device, the reconfigurable objects of the master device, can be captured. A value of 0 indicates that the core concern of the slave device, the reconstructed object of the master device, cannot be captured; t represents time; A five-bit binary number representing a single hand gesture is used as the gesture number; This indicates the maximum distance between the core focus main-end reconstructed object and the data glove when it can be crawled; This represents the direction vector directly in front of the palm of the data glove. .

[0093] The teleoperation system based on a data glove of the present invention includes a master unit, a master-slave communication unit, and a slave unit. The master unit includes a data glove and a master PC. The master-slave communication unit is a robot operating system (ROS). The slave unit includes multiple slave devices. The data glove collects quaternion pose data of the human hand, wrist, and upper arm, as well as gesture data of each finger, in real time and wirelessly transmits it to the master PC for preliminary processing. In specific implementation, it can collect complete kinematic chain data from the shoulder joint to the finger joint in real time, covering information such as the angle, position, and movement speed of each joint, and then transmits it to the master PC wirelessly via Wi-Fi. After processing the quaternion pose data, a spatial pose matrix is ​​obtained. The gesture data of each finger is processed by a gesture mapping algorithm to obtain a gesture number. Then, the spatial pose matrix is ​​subjected to a master-slave heterogeneous mapping to obtain the target pose of the core focus point of each slave device as the control input. Then, the control data of each slave device is obtained through an inverse kinematics algorithm. Finally, the control data and gesture number are transmitted to each slave device through the robot operating system (ROS) for teleoperation control.

[0094] The remote operating system enables the control of multiple slave devices by setting up a master-side reconfiguration object based on the core concerns of multiple slave devices. This allows the same master-side data glove to control multiple slave devices, while achieving more precise, intuitive, and natural control, reducing operational complexity, and improving the user experience.

[0095] For example Figure 6 The five-degree-of-freedom hydraulic robotic arm shown is used as a slave device to conduct a remote grasping experiment, verifying the accuracy, intuitiveness, and naturalness of the data glove for the remote operating system and its control method of the present invention, as well as its effectiveness in reducing operational complexity and improving the control experience.

[0096] Specifically, the master PC in the master device is equipped with teleoperation control software developed based on Unity. It communicates wirelessly with the data glove via Wi-Fi to achieve real-time data transmission and ensure timely delivery of operational commands. The software integrates a calibration program to calibrate the data glove's sensors, ensuring the accuracy of the collected hand movement data. Based on the collected sensor data, the software can calculate the posture information of each joint of the hand in real time, providing data support for the generation of control commands. Through the ROS communication protocol, the master software communicates with the slave device, sending processed control commands to the slave hydraulic robotic arm and receiving feedback information from the slave, thus completing the control and monitoring of the slave device.

[0097] The core components of the slave device include a five-degree-of-freedom (DOF) hydraulic manipulator, a matching hydraulic drive system, and a high-precision signal acquisition system. The five-DOF hydraulic manipulator mainly consists of five parts: the elbow, upper arm, forearm, wrist, and end effector. The elbow is hinged to the manipulator support structure mounted on the base. Except for the support frame and base, the rest of the manipulator has rotational capabilities. Specific structural parameters are shown in Table 1. The entire system runs in the MATLAB / Simulink environment, using ROS to build a master-slave communication network for transmitting master control commands and providing feedback from the slave. Furthermore, an external camera is mounted on the slave hydraulic manipulator base and connected to the master PC via USB to simulate video signal feedback in actual teleoperation scenarios.

[0098] Table 1 Structural parameters of the hydraulic robotic arm

[0099] structure length Range of motion elbow 230mm 120° upper arm 260mm 120° forearm 245mm 110° wrist 195mm 100°

[0100] Before the experiment begins, all experimental equipment must be checked and prepared to ensure it is functioning properly. The specific steps are as follows:

[0101] 1) Wearing and calibrating the data gloves: Correctly wear the data gloves on the operator's hands, ensuring all sensors are securely fixed and properly connected. Calibrate inertial sensors 4, 9, and 12 and bending sensor 6 using the preset calibration procedure to ensure the accuracy of their measurement data.

[0102] 2) Inspection of the hydraulic robotic arm: Start the hydraulic robotic arm and check the range of motion and flexibility of each joint to ensure it is in good working order. Simultaneously, check the connections of the hydraulic drive system and signal acquisition system to ensure they are functioning correctly.

[0103] 3) Configuration of the master PC: Start the master PC, run the remote control software, and ensure that it can receive signals transmitted by the data glove and correctly process them before sending them to the slave hydraulic robotic arm. Check the connection status of the communication network to ensure stable communication between the master and slave ends.

[0104] 4) External camera setup: Ensure the external camera is working properly and adjust its position and angle for clear observation of the end effector of the hydraulic robotic arm and the test item. Check the USB connection between the camera and the host PC to ensure stable visual feedback in real time.

[0105] To fully verify the performance of the data gloves, the following experimental tasks were designed:

[0106] Task 1: Simple Grasping Task: The operator uses a data glove to control a hydraulic robotic arm to sequentially grasp and move several simple-shaped objects (such as cubes and cylinders) to designated locations. The completion time is recorded after each object is grasped and placed in its position.

[0107] Task 2: Complex Grasping Task: The operator controls a hydraulic robotic arm to grasp and manipulate irregularly shaped objects (such as plastic models). Record the time taken to complete the task and observe whether any errors occur during the operation.

[0108] Task 3: Path Tracking Task: Set up a predetermined path (circular path) in the experimental site. The operator controls the end effector of the hydraulic robotic arm to move along the path and records the time and path deviation for completing the path tracking.

[0109] Task 4: Gesture Recognition Task: The operator uses preset gestures (such as "grab," "release," "rotate," etc.) to control the hydraulic robotic arm to complete specific actions through a data glove. Record the accuracy and response time of gesture recognition.

[0110] In this experiment, the performance of Task 1 (simple grasping task) demonstrates that the data glove can efficiently and accurately control the hydraulic robotic arm to grasp and move objects of simple shapes, with an average completion time of 25 seconds and a grasping success rate of 90%. This result verifies the high accuracy and reliability of the data glove in basic operational tasks. Task 2 (complex grasping task) further tested the performance of the data glove. Although the average completion time increased to 35 seconds when dealing with irregularly shaped objects, the grasping success rate still reached 80%. Although a few errors occurred under complex conditions, such as unstable grasping, the overall performance still showed the adaptability and accuracy of the data glove in handling complex objects. Task 3 (path tracking task) measured the deviation of the hydraulic robotic arm's end effector along a predetermined path. The results showed an average deviation of 5.2 mm and a standard deviation of 1.3 mm. This result indicates that the data glove can achieve complex path tracking operations with high accuracy, which is of great significance for applications requiring precise trajectory control. The test results of Task 4 (gesture recognition task) showed that the gesture recognition accuracy was 100% and the response time was 0.2 seconds. This result demonstrates that data gloves excel in gesture recognition, rapidly responding to operator intentions and significantly enhancing the naturalness and intuitiveness of teleoperation. In summary, the data gloves demonstrate outstanding performance in accuracy, intuitiveness, naturalness, reduced operational complexity, and improved user experience, effectively validating their application potential and advantages in teleoperation systems.

[0111] The above content is merely a technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A data glove for a teleoperation system, characterized in that, include: Glove body (1); A hand data unit is installed on the glove body (1) and at the hand position of the human body for transmitting quaternion attitude data of the hand to one or more slave devices for remote operation control of the slave device end. A wrist data unit, mounted on the glove body (1) and at the wrist position of the human body, is used to transmit wrist quaternion attitude data to one or more slave devices for remote operation control in the middle of the slave devices. An arm data unit, mounted on the glove body (1) and at the arm position of the human body, is used to transmit quaternion attitude data of the arm to one or more slave devices for remote operation control at the root of the slave device. The glove body (1) includes a fingerless base (1.1), a first elastic strap base (1.2), and a second elastic strap base (1.3). The fingerless base (1.1) is worn on the hand during remote operation, the first elastic strap base (1.2) is worn on the wrist during remote operation, and the second elastic strap base (1.3) is worn on the upper arm during remote operation. The hand data unit includes a first integrated printed circuit board (PCB) (2) and a bending sensor group (5). The first integrated printed circuit board (PCB) (2) is installed on the back of the hand of the fingerless base (1.1). The first integrated printed circuit board (PCB) (2) is equipped with a first core main control chip (7), a first power management module (8) for power supply, and a second inertial sensor (9). The bending sensor group (5) includes five bending sensors (6). The five bending sensors (6) are respectively installed on the back of the five fingers of the fingerless base (1.1). The first core control chip (7) is electrically connected to both the first inertial sensor (6) and the second inertial sensor (9). The wrist data unit is the first inertial sensor (4), which is installed on the outer side of the first elastic band base (1.2) and electrically connected to the first core control chip (7). The first core control chip (7) is wirelessly connected to the external master PC. The arm data unit is the second integrated printed circuit board (PCB) (3), which is installed on the outer side of the second elastic band base (1.3). The second integrated printed circuit board (PCB) (3) is equipped with a second power management module (10), a second core control chip (11), and a third inertial sensor (12). The second power management module (10) and the third inertial sensor (12) are electrically connected to the second core control chip (11). The second core control chip (11) is wirelessly connected to the external master PC. The master PC is wirelessly connected to the slave device.

2. The teleoperation control method for a data glove used in a teleoperation system according to claim 1, characterized in that, include: First, the finger-type base (1.1), the first elastic band base (1.2), and the second elastic band base (1.3) of the glove body (1) are respectively put on the human hand, wrist, and upper arm. Then, the main PC is used as the main node, and the first integrated printed circuit board (PCB) (2) and the second integrated printed circuit board (PCB) (3) are used as child nodes. A wireless communication network is constructed between the main node and the two child nodes. When the human hand, wrist, and upper arm perform preset actions, the quaternion posture data of the human hand, wrist, and upper arm are collected by each inertial sensor (4, 9, 12) and wirelessly transmitted to the main PC through the core main control chip (7, 11). At the same time, the gesture data of each finger is collected by each bending sensor (6) and wirelessly transmitted to the main PC through the first core main control chip (7). The main PC performs preliminary processing on the quaternion posture data of the human hand, wrist, and upper arm to obtain the spatial pose matrix, and performs preliminary processing on the gesture data of each finger through the gesture mapping algorithm to obtain the gesture number. When multiple slave devices need to be controlled, the master PC performs a master-slave heterogeneous mapping of the spatial pose matrix to obtain the target pose of the core focus of each slave device as the control input. Then, the control data of each slave device is obtained through the inverse kinematics algorithm. Finally, the control data and gesture sequence number are transmitted to each slave device for remote operation control.

3. The teleoperation control method for a data glove used in a teleoperation system according to claim 2, characterized in that: The main PC terminal uses the quaternion pose data of the human hand, wrist, and upper arm collected by each inertial sensor (4, 9, 12) to obtain the spatial pose matrix of the hand relative to the shoulder joint through a human hand kinematics model. as follows: in, Represents a kinematic model of the human hand joint; , and These represent the quaternion attitude data read by each inertial sensor (4, 9, 12); , and These represent the length of the upper arm, the length of the forearm, and the distance from the wrist to the palm, respectively.

4. The teleoperation control method for a data glove used in a teleoperation system according to claim 2, characterized in that: The main PC terminal uses a gesture mapping algorithm to perform preliminary processing on the gesture data of each finger to obtain a gesture sequence number. The gesture data of each finger is finger flexion and extension data. In specific processing, for each finger's finger flexion and extension data, the finger flexion and extension data that is greater than a preset flexion and extension threshold is selected. The binary bits of the finger are set to 0, and the finger flexion-extension data is less than or equal to a preset flexion-extension threshold. The binary bits of each finger are set to 1, ultimately constructing a five-bit binary number for each hand as the gesture sequence. The details are as follows: in, Indicates the first The binary bits of one finger; Indicates the first Data on the flexion and extension of each finger; Each gesture number corresponds to a preset motion task on the slave device.

5. The teleoperation control method for a data glove for a teleoperation system according to claim 2, characterized in that: The master-slave heterogeneous mapping is described in detail below: in, This indicates the target pose of the core concern of the slave device; This represents the spatial pose matrix of the master-end reconstructed object, which is the core concern of the slave device. This represents pose data representing the core concerns of the slave device; This represents the workspace representing the core concerns of the end device; This indicates that the emission originates from the coordinate origin and the direction of emission is... Rays and workspace surface The intersection; This allows the pose information of the object to be reconstructed from the core focus of the slave device by the master device. The mapping is used to define the target pose of the core concern of the slave device.

6. The teleoperation control method for a data glove used in a teleoperation system according to claim 2, characterized in that: The control conditions for the data gloves during remote operation are as follows: in, The spatial pose matrix representing the hand of the data glove. and These represent the hand pose rotation matrix and spatial position vector of the data glove, respectively; This represents the spatial pose matrix of the core concern reconstructed object of the i-th slave device. and Let represent the pose rotation matrix and spatial position vector of the i-th slave device, respectively; This represents the relative pose of the hand of the i-th slave device with respect to the object. and Let represent the orientation rotation matrix and spatial position vector of the i-th slave device's hand relative to the object, respectively; This represents the logic function for fetching data. A value of 1 indicates that the core concerns of the slave device, the reconfigurable objects of the master device, can be captured. A value of 0 indicates that the core concern of the slave device, the reconstructed object of the master device, cannot be captured; t represents time; A five-bit binary number representing a single hand gesture is used as the gesture number; This indicates the maximum distance between the core focus main-end reconstructed object and the data glove when it can be crawled; The direction vector directly in front of the palm of the data glove.

7. The teleoperation system for a data glove used in a teleoperation system according to claim 1, characterized in that: The system comprises a master unit, a master-slave communication unit, and slave units. The master unit includes a data glove and a master PC. The master-slave communication unit is the Robot Operating System (ROS). The slave units include multiple slave devices. The data glove collects quaternion pose data of the human hand, wrist, and upper arm in real time, as well as gesture data of each finger, and wirelessly transmits it to the master PC for preliminary processing. After processing the quaternion pose data, a spatial pose matrix is ​​obtained. The gesture data of each finger is processed by a gesture mapping algorithm to obtain a gesture number. Then, the spatial pose matrix is ​​subjected to a master-slave heterogeneous mapping to obtain the target pose of the core focus point of each slave device as the control input. The control data of each slave device is then obtained through an inverse kinematics algorithm. Finally, the control data and gesture numbers are transmitted to each slave device through the Robot Operating System (ROS) for remote operation control.

Images