Data glove-based human-robot interaction teleoperation control system and method

Abstract

The application provides a kind of human-computer interaction robot remote control system and method based on data glove, comprising: data acquisition module, data processing module and robot execution module;Data acquisition module collects and sends finger bending degree and palm posture information;Data processing module pre-processes the information collected, gesture recognition and pose mapping are carried out based on the information after pre-processing, control instructions are generated and sent;Robot execution module receives control instructions and controls the robot to execute corresponding actions.The application solves the technical problems of the existing robot control system based on data glove in the aspects of recognition accuracy, real-time performance and system compatibility.

Description

Technical Field

[0001] This invention relates to the field of intelligent robots, and more specifically, to a human-computer interaction robot teleoperation control system and method based on data gloves. Background Technology

[0002] With the rapid development of robotics technology, intelligent robots have shown broad application prospects in various fields such as industrial manufacturing, disaster relief, medical rehabilitation, and service companionship. To improve the efficiency and safety of robots operating in complex environments, there is an urgent need for an efficient, intuitive, and real-time human-computer interaction method. However, the control of current robot systems still heavily relies on manual operation and semi-autonomous control mechanisms, and existing interaction methods still have significant shortcomings in terms of operational freedom, feedback response, and adaptability to complex scenarios.

[0003] Currently, common robot control methods include handheld control, PC-based graphical user interface operation, voice interaction, and motion capture systems. Traditional button-type remote controls (such as CN222465063U) have the advantages of being easy to use and having a mature structure, but they struggle to support high-degree-of-freedom, continuous motion control, limiting their application in complex tasks. Host computer graphical interfaces (such as CN216697069U) rely on a fixed operating environment and a stable network, exhibiting significant limitations in remote mobile or field operations. While voice recognition-based systems (such as CN214622742U) offer some ease of interaction, their recognition rate and real-time performance are limited in noisy or dynamic environments, failing to support high-precision, continuous control requirements. IMU-based motion capture systems (such as CN120985651A), although avoiding the environmental dependence of traditional visual motion capture, are sensitive to issues such as wearable synchronization and magnetic interference, limiting their applicability.

[0004] In contrast, data gloves, as lightweight and wearable interactive devices, can collect hand movement information in real time, enabling more natural control of robots and showing great potential in teleoperation and immersive interaction. However, current data glove systems are mostly laboratory prototypes or customized solutions for specific robots, and still have many shortcomings. For example, although CN106354262B adopts a gesture recognition scheme based on neural networks, it lacks efficient filtering algorithms, has weak noise resistance, and is prone to recognition jitter; CN111645093A integrates force feedback function, but the communication link latency is large, making it difficult to achieve real-time feedback control; CN118322199A enhances the intuitiveness of operation through virtual simulation, but has poor compatibility with ROS systems and is difficult to integrate into general robot platforms.

[0005] Furthermore, existing systems generally lack comprehensive safety mechanisms, such as communication interruption protection, accidental touch protection, and emergency braking design. At the same time, the widespread use of low-speed wireless communication links results in significant control delays and poor system robustness, making it difficult to meet the high-reliability application requirements of industrial sites, outdoor remote control, and other similar scenarios.

[0006] In summary, existing data glove-based robot control technologies still have significant room for improvement in terms of recognition accuracy, naturalness of interaction, real-time performance, safety, and system compatibility. There is an urgent need to provide a data glove control method and system that features optimized structure, rapid response, precise control, high robustness, and wide adaptability to enhance robot maneuverability and human-robot collaboration in complex environments. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a human-computer interaction robot teleoperation control system and method based on data gloves.

[0008] According to one aspect of the present invention, a human-computer interaction robot teleoperation control system based on a data glove includes:

[0009] Data acquisition module: Collects and sends information on finger flexion and hand posture;

[0010] Data processing module: preprocesses the collected information, performs gesture recognition and pose mapping based on the preprocessed information, and generates and sends control commands;

[0011] Robot execution module: Receives control commands and controls the robot to perform corresponding actions.

[0012] Preferably, the data acquisition module includes:

[0013] Submodule M1.1: Acquires finger joint bending angle information through a resistive bending sensor;

[0014] Submodule M1.2: Acquires the acceleration and angular velocity information of the palm through an inertial measurement unit, wherein the inertial measurement unit includes one or more of a triaxial accelerometer, a triaxial gyroscope, and a triaxial magnetometer;

[0015] Submodule M1.3: Filters and encapsulates the information acquired by submodules M1.1 and M1.2;

[0016] Submodule M1.4: Sends filtered and encapsulated information.

[0017] Preferably, the data processing module includes:

[0018] Submodule M2.1: Receives information and preprocesses the received information, wherein the preprocessing includes normalization, low-pass filtering, attitude calculation and jitter suppression;

[0019] Submodule M2.2: Based on the preprocessed bending angle information of finger joints, gesture recognition is performed through threshold judgment and template matching;

[0020] Submodule M2.3: Maps the preprocessed acceleration and angular velocity information of the palm into the robot's motion speed through a nonlinear function;

[0021] Submodule M2.4: Based on the results of gesture recognition, determine the control mode using a hierarchical state machine.

[0022] Submodule M2.5: Based on gesture recognition and mapping results, it generates control commands and sends them via ROS2.

[0023] Preferably, the normalization process in submodule M2.1 includes:

[0024] Zero-point and full-scale calibrations are performed on each bending sensor, and the normalized curvature value is calculated based on the calibration results; the normalized curvature value is obtained according to the following formula:

[0025]

[0026] Where norm_i is the normalized curvature value in the range [0,1], raw_i is the raw data collected by the bending sensor, offset_i is the calibrated zero point of the sensor curvature, and scale_i is the calibrated range of sensor curvature variation.

[0027] The low-pass filter in submodule M2.1 includes:

[0028] A second-order Butterworth low-pass filter was used, with the cutoff frequency set to 10Hz, to suppress high-frequency noise in the raw data acquired by the inertial measurement unit.

[0029] The attitude calculation in submodule M2.1 includes:

[0030] The acceleration and angular velocity data collected by the inertial measurement unit are fused based on the Mahony attitude calculation algorithm or the Madgwick attitude calculation algorithm to obtain a quaternion representing the palm attitude; and the quaternion is converted into the corresponding Euler angles as needed. The Euler angles are attitude angle data, including pitch angle, roll angle or yaw angle.

[0031] The jitter suppression in submodule M2.1 includes:

[0032] An exponential moving average filtering algorithm is used to smooth the attitude angle data, and an attitude dead zone threshold is set. When the absolute value of the pitch angle, roll angle, or yaw angle is lower than the preset threshold, the corresponding output is set to zero.

[0033] Preferably, the submodule M2.3 includes:

[0034] Submodule M2.3.1: Maps the palm pitch angle to a forward or backward velocity using a nonlinear function, wherein the forward or backward velocity is:

[0035]

[0036] Submodule M2.3.2: Maps the hand roll angle to a lateral velocity using a nonlinear function, where the lateral velocity is:

[0037]

[0038] Submodule M2.3.3: Maps the hand yaw angle to angular velocity using a nonlinear function; the angular velocity is:

[0039]

[0040] Where pitch is the palm pitch angle, roll is the roll angle, and yaw is the yaw angle. The maximum allowable linear velocity for the robot. Let α be the maximum permissible angular velocity of the robot, β be the attitude control sensitivity coefficients, and tanh(·) be the hyperbolic tangent function.

[0041] Preferably, the control modes in the submodule M2.4 include:

[0042] IDLE: Standby mode, only collects data and does not send commands;

[0043] TELEOP: Speed ​​control mode, which maps robot speed according to hand posture;

[0044] GAIT_SELECT: Gait switching mode, allowing you to switch between walking, running, crawling, and other modes using gestures;

[0045] BODY_POSE: Attitude control mode, used to adjust the robot's height and pitch angle;

[0046] ACTION: Action mode, supports complex actions such as sitting down, standing up, and crossing obstacles;

[0047] SAFE_STOP: Safe stop state, automatically entered by emergency stop or loss of connection protection.

[0048] Preferably, the submodule M2.2 also triggers the following when performing gesture recognition:

[0049] Submodule M2.2.1: Adopts a "three-stage confirmation" mechanism, in which gesture recognition needs to go through three stages in sequence: "press - hold - release", and the duration of the hold stage is not less than a preset threshold.

[0050] Submodule M2.2.1: Set the hysteresis interval so that the entry threshold and exit threshold are different.

[0051] Preferably, the submodule M2.5 also triggers the following when generating control commands:

[0052] Submodule M2.5.1: Introduces a speed variation limiter to limit the desired speed v. des Compared with the output speed v at the previous moment cmd [t The difference in [1] is limited to ensure that the speed change does not exceed a preset range, thereby preventing the robot from experiencing sudden attitude changes due to instantaneous command jumps. The limiting process can be expressed as:

[0053]

[0054] Among them, v cmd [t] represents the output speed command at the current moment, v des Δv is the desired velocity obtained from the gesture or posture mapping. max The maximum allowable speed variation range; the function clamp(x,a,b) restricts the variable x to the interval [a,b].

[0055] Preferably, the system further includes a security mechanism module, the security mechanism module comprising:

[0056] The dual emergency stop mechanism module is used to immediately stop the robot's movement and switch to a safe stop state when a preset emergency stop gesture is detected or an emergency stop button trigger signal is received.

[0057] The communication loss protection mechanism is used to automatically switch the robot to a safe stop state when no control data is received from the data glove within a preset time threshold.

[0058] The speed limiting and virtual fence protection mechanism is used to limit the linear and angular velocities of robot control commands, and to trigger speed limiting or stop when the robot moves beyond the preset safe area.

[0059] According to another aspect of the present invention, a teleoperation control method for a human-computer interaction robot based on a data glove includes:

[0060] Step S1: Hand data acquisition. The bending angle of the fingers and the hand posture information are acquired by bending sensors and IMUs placed on the hand, and high-speed sampling, filtering and fusion processing are completed by the microcontroller.

[0061] Step S2: Action information preprocessing, including noise suppression, normalization, and feature extraction of the collected data;

[0062] Step S3: Gesture recognition, determining the operator's current gesture based on feature data;

[0063] Step S4: Pose mapping and command generation, mapping gesture and posture information into robot control commands, and selecting the control mode according to the state machine;

[0064] Step S5: Command transmission, sending control commands to the robot via ROS2 or other communication interfaces;

[0065] Step S6: Action execution, the robot executes the corresponding motion, navigation, grasping or manipulation actions according to the control commands;

[0066] Step S7: Safety mechanism processing. In the event of misidentification, communication interruption, or out-of-bounds operation, an emergency stop, amplitude limit, or loss of connection protection mechanism is triggered.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] (1) The operation is natural and intuitive. This technology uses five resistive bending sensors and a six-axis IMU arranged in the data glove. Combined with the Mahony / Madgwick attitude calculation algorithm, it obtains the operator's finger bending degree and palm pitch, roll and yaw angles in real time. Then, through gesture recognition with threshold judgment and nonlinear mapping function (tanh), the hand posture is continuously mapped to robot speed commands, realizing the interaction mode of "attitude control speed and gesture control mode". Since the control method is consistent with the natural human movement, the operator does not need to learn complex commands to realize intuitive operation of the robot to move forward, backward, lateral, turn and trigger actions, which significantly reduces the threshold of use.

[0069] (2) High real-time performance: The system uses an STM32G431 microcontroller on the data acquisition side to read sensor data at a sampling rate of 200Hz. It also performs real-time preprocessing through lightweight filtering methods such as second-order Butterworth low-pass filtering and exponential moving average algorithm to minimize signal stabilization overhead. On the communication side, an ESP8266 Wi-Fi module is used to provide high-speed wireless transmission. On the processing side, an efficient attitude fusion algorithm and a lightweight gesture recognition strategy are adopted to ensure low latency in both transmission and computation. After end-to-end link optimization, the overall control latency can be kept within the range of 20–58ms, which can meet the real-time teleoperation requirements of mobile robots, quadruped robots, service robots and other platforms.

[0070] (3) Accurate recognition: The system adopts a dual gesture recognition mechanism of "bending threshold judgment + dynamic time warping (DTW) template matching". Threshold judgment is used to quickly recognize basic gestures with obvious shapes such as clenched fist and open fist, while template matching is used to recognize complex gestures composed of bending degree changes and posture changes. At the same time, IMU posture calculation uses the Mahony / Madgwick algorithm to achieve noise suppression and posture drift compensation. In addition, micro-jitter is effectively suppressed by EMA smoothing and posture dead zone processing, so that the overall recognition accuracy can still reach more than 97% in dynamic environment, which is significantly higher than the traditional single-modal recognition method.

[0071] (4) Safety and reliability: This system is designed with multi-level safety mechanisms, including: ① a dual emergency stop link consisting of a hardware emergency stop button and a software emergency stop gesture (such as a fist clenching ≥150ms), which can immediately terminate the control output before a dangerous action occurs; ② a communication watchdog mechanism, which automatically switches to a safe stop state when no glove data is received for more than 50ms; ③ a speed limiter combined with a virtual fence strategy, which avoids sudden speed jumps and boundary crossings by limiting the increment of control commands and the robot's movable area. The above safety mechanisms ensure that the system maintains high robustness and safety in highly dynamic scenarios.

[0072] (5) Strong compatibility: The data processing end is based on the ROS2 communication middleware, which publishes the posture, speed, gesture and action commands parsed by the glove in a standardized topic format, including interfaces such as / cmd_vel, / gait_mode, / body_pose and / action, which can be directly connected to the existing ROS2 robot ecosystem. In addition, the system adopts a modular architecture, decoupling the acquisition, recognition, mapping and control, so that it can be adapted to various platforms such as mobile chassis, robotic arms, exoskeletons and special operation robots, and has good scalability and cross-platform capability. Attached Figure Description

[0073] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0074] Figure 1 This is a diagram of the overall system architecture, showing the relationship between the data acquisition module, the data processing module, and the robot execution module.

[0075] Figure 2 This is a schematic diagram of the sensor arrangement for a data glove, including a bending sensor and an IMU.

[0076] Figure 3 The algorithm's pipeline diagram includes normalization, low-pass filtering, attitude calculation, and jitter suppression.

[0077] Figure 4 This is a schematic diagram of the ROS2 communication interface, showing the input, output, and feedback channels.

[0078] Figure 5 This is the system state machine transition diagram, including the IDLE, TELEOP, GAIT_SELECT, BODY_POSE, ACTION, and SAFE_STOP states.

[0079] Figure 6 : Schematic diagram of emergency stop and loss of contact protection mechanism. Detailed Implementation

[0080] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0081] For ease of understanding, the terms or concepts involved in this application are explained below:

[0082] (1) Bending sensor: used to detect the degree of finger bending, its resistance changes with the bending angle.

[0083] (2) IMU (Inertial Measurement Unit): Composed of an accelerometer and a gyroscope, used to collect information on the acceleration and angular velocity of the palm.

[0084] (3) Accelerometer: measures the linear acceleration of the palm in three directions, which can be used to calculate the direction of gravity.

[0085] (4) Gyroscope: measures the angular velocity of the palm around three axes, used for attitude change calculation.

[0086] (5) Attitude calculation: The process of fusing IMU data to calculate hand posture, outputting quaternions or Euler angles.

[0087] (6) Euler angles (pitch / roll / yaw): The pitch, roll and yaw angles describe the spatial attitude of the hand.

[0088] (7) Quaternion: A rotational representation without singularity, used for attitude fusion calculation.

[0089] (8) Mahony / Madgwick algorithm: a lightweight fusion algorithm that uses raw IMU data to estimate attitude in real time.

[0090] (9) Normalization: Map the original values ​​of the bending sensor to the 0–1 range according to the zero point and full scale.

[0091] (10) Low-pass filter (Butterworth filter): Used to remove high-frequency noise from IMU data.

[0092] (11) Exponential Moving Average (EMA): A jitter suppression algorithm that smooths the attitude angle.

[0093] (12) Dead zone handling: Set a threshold, and when the attitude angle change is lower than the threshold, the output is set to zero to eliminate micro-motion.

[0094] (13) Gesture recognition: The process of determining the type of operator's gesture based on curvature and posture characteristics.

[0095] (14) Threshold determination: A method for identifying simple gestures by comparing the curvature with a preset threshold.

[0096] (15) Template matching (DTW): The real-time sequence is compared with the template using the dynamic time warping algorithm for complex gesture recognition.

[0097] (16) Pose mapping: Generate robot speed or motion control commands based on the palm's pose angle.

[0098] (17) tanh Nonlinear mapping function: a function used to smoothly convert attitude angles into velocity and limit the output range.

[0099] (18) State machine: A logic unit that manages different control modes (such as teleoperation, gait selection, and action execution).

[0100] (19) Emergency stop mechanism: a safety function that allows the robot to stop immediately via a button or emergency stop gesture.

[0101] (20) Communication loss protection: a mechanism that forces the robot to switch to a safe state when no glove data is received after a communication timeout.

[0102] Example 1:

[0103] This embodiment provides a human-machine interface robot teleoperation control system based on a data glove. The system consists of a data acquisition module, a data processing module, a robot execution module, and a safety mechanism module. The data acquisition end uses bending sensors on the fingers and an IMU on the back of the hand. A microcontroller performs data acquisition, filtering, and fusion at frequencies above 200Hz to obtain finger bending and hand posture. The data processing end obtains stable signals based on normalization, low-pass filtering, exponential moving average, and dead-zone processing. Euler angles are calculated using Mahony / Madgwick posture calculation, and gesture recognition is achieved by combining threshold judgment and template matching. Subsequently, a nonlinear mapping function is used to convert the posture angles into robot speed and control commands, and a state machine manages different control modes. The robot execution end receives speed, gait, posture, and motion commands via ROS2 to achieve navigation, movement, and task execution. The system incorporates safety mechanisms such as emergency stop, disconnection protection, and virtual fences to ensure the smoothness, real-time performance, and safety of the control process. This system features high real-time performance, high accuracy, and natural and intuitive human-machine interaction characteristics, making it suitable for remote operation of various robot platforms.

[0104] The following is a description of each module of the system:

[0105] Data acquisition module: Collects and sends information on finger bending degree and palm posture.

[0106] It is understood that the data acquisition module includes:

[0107] Submodule M1.1: Acquires finger joint bending angle information through a resistive bending sensor.

[0108] Submodule M1.2: Acquires the acceleration and angular velocity information of the palm through an inertial measurement unit, wherein the inertial measurement unit includes one or more of a triaxial accelerometer, a triaxial gyroscope, and a triaxial magnetometer.

[0109] Submodule M1.3: Filters and encapsulates the information obtained from submodules M1.1 and M1.2.

[0110] Submodule M1.4: Sends filtered and encapsulated information.

[0111] Data processing module: preprocesses the collected information, performs gesture recognition and pose mapping based on the preprocessed information, and generates and sends control commands.

[0112] It is understood that the data processing module includes:

[0113] Submodule M2.1: Receives information and preprocesses the received information, wherein the preprocessing includes normalization, low-pass filtering, attitude calculation and jitter suppression.

[0114] Submodule M2.2: Based on the preprocessed bending angle information of finger joints, gesture recognition is performed through threshold judgment and template matching.

[0115] Based on the above scheme, it can be understood that the threshold determination module is used to identify gestures by comparing the normalized curvature of the bending sensor with a preset threshold. Specifically, when the curvature of all five fingers is greater than a first threshold, it is determined to be a clenched fist gesture; when the curvature of all five fingers is less than a second threshold, it is determined to be an open palm gesture. The template matching module is used to perform similarity matching between real-time gestures and preset gesture templates based on the curvature sequence and posture sequence. The similarity matching employs the Dynamic Time Warping (DTW) algorithm to improve the robustness of complex gesture recognition.

[0116] Submodule M2.3: Maps the preprocessed acceleration and angular velocity information of the palm into the robot's motion speed through a nonlinear function.

[0117] Submodule M2.4: Based on the results of gesture recognition, determine the control mode using a hierarchical state machine.

[0118] Submodule M2.5: Based on gesture recognition and mapping results, it generates control commands and sends them via ROS2.

[0119] It is understood that the normalization process in the submodule M2.1 includes:

[0120] Zero-point and full-scale calibrations are performed on each bending sensor, and the normalized curvature value is calculated based on the calibration results. The normalized curvature value is obtained according to the following formula:

[0121]

[0122] Where norm_i is the normalized curvature value in the range [0,1], raw_i is the raw data collected by the bending sensor, offset_i is the calibrated zero point of the sensor curvature, and scale_i is the calibrated range of sensor curvature variation.

[0123] The low-pass filter in submodule M2.1 includes:

[0124] A second-order Butterworth low-pass filter was used, with the cutoff frequency set to 10Hz, to suppress high-frequency noise in the raw data acquired by the inertial measurement unit.

[0125] The attitude calculation in submodule M2.1 includes:

[0126] The acceleration and angular velocity data collected by the inertial measurement unit are fused using the Mahony or Madgwick attitude calculation algorithms to obtain a quaternion representing the hand's attitude. The quaternion is then converted into corresponding Euler angles as needed. These Euler angles are attitude angle data, including pitch, roll, or yaw angles.

[0127] The jitter suppression in submodule M2.1 includes:

[0128] An exponential moving average filtering algorithm is used to smooth the attitude angle data, and an attitude dead zone threshold is set. When the absolute value of the pitch angle, roll angle, or yaw angle is lower than the preset threshold, the corresponding output is set to zero.

[0129] It is understood that the submodule M2.3 includes:

[0130] Submodule M2.3.1: Maps the palm pitch angle to a forward or backward velocity using a nonlinear function, wherein the forward or backward velocity is:

[0131]

[0132] Submodule M2.3.2: Maps the hand roll angle to a lateral velocity using a nonlinear function, where the lateral velocity is:

[0133]

[0134] Submodule M2.3.3: Maps the hand yaw angle to angular velocity using a nonlinear function. The angular velocity is:

[0135]

[0136] Where pitch is the hand's pitch angle, roll is the roll angle, and yaw is the yaw angle. The maximum allowable linear velocity for the robot. Let α, β, and γ be the maximum permissible angular velocities of the robot. Let tanh(·) be the attitude control sensitivity coefficient. Let be the hyperbolic tangent function.

[0137] It is understood that the control modes in the submodule M2.4 include:

[0138] IDLE: Standby mode, only collects data and does not send commands.

[0139] TELEOP: Speed ​​control mode, which maps robot speed according to hand posture.

[0140] GAIT_SELECT: Gait switching mode, allowing you to switch between walking, running, crawling, and other modes using gestures.

[0141] BODY_POSE: Attitude control mode, used to adjust the robot's height and pitch angle.

[0142] ACTION: Action mode, supports complex actions such as sitting, standing, and crossing obstacles.

[0143] SAFE_STOP: Safe stop state, automatically entered by emergency stop or loss of connection protection.

[0144] It is understandable that the submodule M2.2 also triggers the following when performing gesture recognition:

[0145] Submodule M2.2.1: It adopts a "three-stage confirmation" mechanism, in which gesture recognition needs to go through three stages in sequence: "press-hold-release", and the duration of the hold stage is not less than a preset threshold.

[0146] Submodule M2.2.1: Set the hysteresis interval so that the entry threshold and exit threshold are different.

[0147] It is understandable that the submodule M2.5 also triggers the following when generating control commands:

[0148] Submodule M2.5.1: Introduces a speed variation limiter to limit the desired speed v. des Compared with the output speed v at the previous moment cmd [t The difference in 1] is limited to ensure that the speed change does not exceed a preset range. The limiting process can be expressed as:

[0149]

[0150] Among them, v cmd [t] represents the output speed command at the current moment, v des Δv is the desired velocity obtained from the gesture or posture mapping. max This represents the maximum permissible speed variation. The function `clamp(x,a,b)` restricts the variable `x` to the interval [a,b].

[0151] Robot execution module: Receives control commands and controls mobile robots, robotic arms, exoskeletons, service robots, or special-purpose robots to perform tasks.

[0152] The security mechanism module is used to ensure the security of the remote operation process.

[0153] It is understood that the security mechanism module includes:

[0154] The dual emergency stop mechanism module is used to immediately stop the robot's movement and switch to a safe stop state when a preset emergency stop gesture is detected or an emergency stop button trigger signal is received.

[0155] The communication loss protection mechanism is used to automatically switch the robot to a safe stop state when no control data is received from the data glove within a preset time threshold.

[0156] The speed limiting and virtual fence protection mechanism is used to limit the linear and angular velocities of robot control commands, and to trigger speed limiting or stop when the robot moves beyond the preset safe area.

[0157] The system also includes a finger bending calibration module, which is used to collect the output of the bending sensor and hold it for a preset time when the operator's palm is open and fist is clenched, respectively, to obtain the zero-point offset value and full-scale range value of the bending sensor, and to normalize the raw data of the bending sensor based on the offset and scale.

[0158] The system includes an attitude zero-drift calibration module, which is used to continuously collect IMU data for a preset time when the glove is horizontally placed, so as to reset the bias parameters of the attitude calculation algorithm and thus correct the initial reference value of the palm attitude angle.

[0159] The system includes a calibration module, which is used to perform finger bending zero-point calibration, posture zero-drift calibration, and gesture template recording to establish bending normalization parameters, posture initial bias, and gesture recognition templates before the system is used.

[0160] Example 2:

[0161] like Figure 1 As shown, a robot teleoperation control method and system based on a data glove is presented. The system consists of three parts: a data acquisition module, a data processing module, and a robot execution module. The data acquisition module, also known as the data glove, is worn on the operator's hand and collects hand movements and posture information through a bend sensor and an inertial measurement unit. This information is then processed by a microcontroller and transmitted via a wireless communication unit. The data processing module runs on a relay computing device (laptop) and is mainly responsible for data parsing, signal preprocessing, gesture recognition, pose mapping, and command generation. The robot execution module runs on the robot body, receives control commands, and executes motion through its own controller. This system adopts an end-to-end closed-loop architecture, supports real-time feedback, and ensures low latency and high reliability during the control process.

[0162] like Figure 2As shown, each finger is equipped with a resistive bending sensor to detect the bending angle of the finger joint; a six-axis IMU is installed on the back of the hand to collect acceleration and angular velocity information of the palm. The microcontroller uses an STM32G431 chip, which has a high main frequency and high ADC accuracy, and can meet the real-time sampling requirements above 200Hz. Its function is to handle sensor data acquisition, filtering preprocessing, and data encapsulation. The wireless communication module is a Wi-Fi module (ESP8266), which utilizes its single-chip integrated TCP / IP protocol stack to eliminate the need for an external network processor, thereby ensuring low latency and stability of communication. An emergency stop button (E-stop) is also located on the back of the glove, and touch pads can be attached to the fingertips for triggering special commands.

[0163] like Figure 3 As shown, the sensor data undergoes preprocessing after acquisition, including normalization, low-pass filtering, attitude calculation, and jitter suppression. Normalization involves zero-point and full-scale calibration of the bending sensor resistance value. The formula for calculating the normalized bending value is:

[0164]

[0165] Where norm_i is the calibrated curvature value in the range [0,1], raw_i is the raw data collected by the sensor, offset_i is the calibrated zero point of the sensor curvature, and scale_i is the calibrated range of sensor curvature variation.

[0166] Low-pass filtering uses a second-order Butterworth low-pass filter on the IMU data, with a cutoff frequency set to 10Hz to filter out high-frequency noise; attitude calculation uses the Mahony / Madgwick algorithm to convert the raw IMU data into hand posture quaternions or Euler angles; jitter suppression introduces the exponential moving average (EMA) algorithm and sets a dead zone range, for example, the output is set to zero when pitch, roll, and yaw are less than 3°, in order to eliminate small jitter.

[0167] After the above processing, smooth and stable finger curvature values ​​and hand pose data are obtained and transmitted to the data processing module via a wireless link at a frequency of 200Hz. The data processing module receives sensor data packets from the data glove through the communication interface, parses them, stores them in a buffer queue, and publishes them in the form of ROS2 messages. Typical interfaces include: / glove / pose represents hand pose (quaternion or Euler angle); / glove / fingers represents normalized finger curvature values; / glove / gestures represents the identified discrete gesture labels and confidence levels. In terms of gesture recognition, this invention adopts a gesture recognition method that combines threshold determination and template matching. Threshold determination is enabled when finger curvature is the main feature. For example, when recognizing a clenched fist gesture, if the curvature values ​​of all five fingers are greater than 0.8, it is determined as a "fist"; if they are all less than 0.2, it is determined as an "open palm". Template matching is enabled when curvature and pose are used together as gesture features. For example, the Dynamic Time Warping (DTW) algorithm is used to compare real-time gestures with pre-recorded templates to improve robustness. In terms of pose and velocity mapping, continuous pose information is converted into robot motion velocity through a nonlinear mapping function:

[0168]

[0169]

[0170]

[0171] Where pitch is the palm pitch angle, roll is the roll angle, and yaw is the yaw angle. The nonlinear function (tanh) can ensure control sensitivity at small angles and limit output at large angles, improving operation smoothness and safety.

[0172] like Figure 4 As shown, the robot's execution end subscribes to command topics from the data glove via the ROS2 interface: / cmd_vel represents linear velocity and angular velocity; / gait_mode represents gait mode; / body_pose represents body height, pitch angle, and roll angle; and / action represents complex motion commands. The controller inside the robot body is responsible for closed-loop execution, including foot force control, gait generator, and state estimation, ensuring that the actual movement conforms to the expected commands.

[0173] This invention designs a hierarchical state machine to manage different control modes, and the gestures for switching between states are as follows: Figure 5 As shown:

[0174] IDLE: Standby mode, only collects data and does not send commands;

[0175] TELEOP: Speed ​​control mode, which maps robot speed according to hand posture;

[0176] GAIT_SELECT: Gait switching mode, which allows you to switch between walking, running, crawling, and other modes using gestures (such as "V");

[0177] BODY_POSE: Attitude control mode, used to adjust the robot's height and pitch angle;

[0178] ACTION: Action mode, supports complex actions such as sitting down, standing up, and crossing obstacles;

[0179] SAFE_STOP: Safe stop state, automatically entered by emergency stop or loss of connection protection.

[0180] In terms of preventing accidental touches and robust design, gesture recognition adopts a "three-stage confirmation," meaning the gesture must go through three stages: "press—hold for ≥150ms—release," to avoid instantaneous accidental touches. Simultaneously, a hysteresis interval is set, with different entry and exit thresholds, to further reduce jitter-induced misidentification. Finally, speed variation limiting is introduced in the output command to prevent sudden and drastic changes in the robot's posture.

[0181]

[0182] like Figure 6 As shown, the system features a dual emergency stop safety design, including hardware and software emergency stops. The hardware emergency stop directly cuts off power or the signal channel via a button; the software emergency stop immediately resets the speed to zero and switches to the SAFE_STOP state upon detecting a "fist" gesture. A disconnection protection mechanism is also included, with a watchdog timer that automatically enters the SAFE_STOP state if no signal from the data glove is received within 50ms. Furthermore, the system can be configured with geofences or safe zones, automatically reducing speed or stopping the robot when it approaches the operator or the boundary.

[0183] Before using this system, it needs to be calibrated, including finger flexion zero-point calibration, posture zero-drift calibration, and gesture template recording. The finger flexion zero-point calibration process involves the operator opening their palm and then clenching their fist, holding each position for 2 seconds. The system automatically records the offset and scale, where offset is the calibrated zero point of sensor flexion, and scale is the calibrated range of sensor flexion variation. The posture zero-drift calibration process involves horizontalizing the glove for 3 seconds and resetting the filter bias. The gesture template recording process involves recording each gesture three times and using DTW alignment to generate a recognition threshold.

[0184] Example 3:

[0185] A robot teleoperation control system based on a data glove includes: a data acquisition module, a data processing module, a robot execution module, and a safety mechanism module. The system includes a bending sensor, an inertial measurement unit (IMU), a microcontroller, and a wireless communication module to acquire information on the operator's finger bending degree and hand posture. The data processing module includes a preprocessing unit, a gesture recognition unit, a pose mapping unit, a state machine management unit, and a communication interface unit, used for preprocessing the information, gesture recognition, motion mapping, and generating robot teleoperation control commands. The robot execution module receives the control commands and drives the robot to perform corresponding actions. The safety mechanism module includes: a dual emergency stop mechanism, a communication failure protection mechanism, and a virtual fence limiting or speed limiting protection mechanism to ensure the safety of the teleoperation process.

[0186] The bending sensor is a resistive flexible sensor. The IMU includes at least one of a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. The microcontroller is an STM32G431 with a sampling rate of at least 200 Hz, employing at least one algorithm of Kalman filtering, low-pass filtering, or complementary filtering to improve the stability of the motion data.

[0187] The preprocessing unit includes normalization, low-pass filtering, jitter suppression, and dead-zone processing. The gesture recognition unit employs gesture classification methods based on thresholding, machine learning algorithms, or deep learning models. The pose mapping unit includes mapping the hand pitch angle to forward / backward speed; mapping the hand roll angle to lateral speed; and mapping the hand yaw angle to angular velocity. The state machine management unit includes navigation control mode, operation action mode, attitude adjustment mode, and safety protection mode. The communication interface unit communicates with the robot execution module based on ROS2.

[0188] The robot execution module is used to control mobile robots, robotic arms, exoskeletons, service robots, or special-purpose robots to perform tasks. The end-to-end control latency of the system is 20-58 ms.

[0189] The dual emergency stop mechanism includes both physical and gesture-based emergency stop triggers. The system includes a disconnection protection mechanism; if no gesture signal is received for more than 50 ms, the robot automatically enters a safe stop state. The system also includes speed limiting and virtual fence mechanisms to prevent the robot from entering dangerous situations.

[0190] Example 4:

[0191] This embodiment provides a robot teleoperation control method based on a data glove, which includes the following steps:

[0192] Step S1: Hand data acquisition. The bending angle of the fingers and the hand posture information are collected by bending sensors and IMUs placed on the hand, and high-speed sampling, filtering and fusion processing are completed by the microcontroller.

[0193] Step S2: Action information preprocessing, including noise suppression, normalization and feature extraction of the collected data.

[0194] Step S3: Gesture recognition, determine the operator's current gesture based on feature data.

[0195] Step S4: Pose mapping and command generation, mapping gesture and posture information into robot control commands, and selecting the control mode according to the state machine.

[0196] Step S5: Command transmission, send control commands to the robot via ROS2 or other communication interfaces.

[0197] Step S6: Action execution. The robot executes the corresponding motion, navigation, grasping, or manipulation actions according to the control commands.

[0198] Step S7: Safety mechanism processing. In the event of misidentification, communication interruption, or out-of-bounds operation, an emergency stop, amplitude limit, or loss of connection protection mechanism is triggered.

[0199] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0200] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A human-computer interaction robot teleoperation control system based on a data glove, characterized in that, include: Data acquisition module: Collects and sends information on finger flexion and hand posture; Data processing module: preprocesses the collected information, performs gesture recognition and pose mapping based on the preprocessed information, and generates and sends control commands; Robot execution module: Receives control commands and controls the robot to perform corresponding actions; The data processing module includes: Submodule M2.1: Receives information and preprocesses the received information, wherein the preprocessing includes normalization, low-pass filtering, attitude calculation and jitter suppression; Submodule M2.2: Based on the preprocessed bending angle information of finger joints, gesture recognition is performed through threshold judgment and template matching; Submodule M2.3: Maps the preprocessed acceleration and angular velocity information of the palm into the robot's motion speed through a nonlinear function; Submodule M2.4: Based on the results of gesture recognition, determine the control mode using a hierarchical state machine. Submodule M2.5: Based on gesture recognition and mapping results, it generates control commands and sends them via ROS2.

2. The system according to claim 1, characterized in that, The data acquisition module includes: Submodule M1.1: Acquires finger joint bending angle information through a resistive bending sensor; Submodule M1.2: Acquires the acceleration and angular velocity information of the palm through an inertial measurement unit, wherein the inertial measurement unit includes one or more of a triaxial accelerometer, a triaxial gyroscope, and a triaxial magnetometer; Submodule M1.3: Filters and encapsulates the information acquired by submodules M1.1 and M1.2; Submodule M1.4: Sends filtered and encapsulated information.

3. The system according to claim 1, characterized in that, The normalization process in submodule M2.1 includes: Zero-point and full-scale calibrations are performed on each bending sensor, and the normalized curvature value is calculated based on the calibration results; the normalized curvature value is obtained according to the following formula: Where norm_i is the normalized curvature value, in the range [0,1], raw_i is the raw data collected by the bending sensor, offset_i is the calibrated zero point of the sensor curvature, scale_i is the calibrated range of sensor curvature variation, and i represents the i-th bending sensor, i=1,2,3,4,5; The low-pass filter in submodule M2.1 includes: A second-order Butterworth low-pass filter was used, with the cutoff frequency set to 10Hz, to suppress high-frequency noise in the raw data acquired by the inertial measurement unit. The attitude calculation in submodule M2.1 includes: The acceleration and angular velocity data collected by the inertial measurement unit are fused based on the Mahony attitude calculation algorithm or the Madgwick attitude calculation algorithm to obtain a quaternion representing the palm attitude; and the quaternion is converted into the corresponding Euler angles as needed. The Euler angles are attitude angle data, including pitch angle, roll angle or yaw angle. The jitter suppression in submodule M2.1 includes: An exponential moving average filtering algorithm is used to smooth the attitude angle data, and an attitude dead zone threshold is set. When the absolute value of the pitch angle, roll angle, or yaw angle is lower than the preset threshold, the corresponding output is set to zero.

4. The system according to claim 1, characterized in that, The submodule M2.3 includes: Submodule M2.3.1: Maps the palm pitch angle to a forward or backward velocity using a nonlinear function, wherein the forward or backward velocity is: Submodule M2.3.2: Maps the hand roll angle to a lateral velocity using a nonlinear function, where the lateral velocity is: Submodule M2.3.3: Maps the hand yaw angle to angular velocity using a nonlinear function; the angular velocity is: Where pitch is the palm pitch angle, roll is the roll angle, and yaw is the yaw angle. The maximum allowable linear velocity for the robot. Let α be the maximum permissible angular velocity of the robot, β be the attitude control sensitivity coefficients, and tanh(·) be the hyperbolic tangent function.

5. The system according to claim 1, characterized in that, The control modes in submodule M2.4 include: IDLE: Standby mode, only collects data and does not send commands; TELEOP: Speed ​​control mode, which maps robot speed according to hand posture; GAIT_SELECT: Gait switching mode, allowing you to switch between walking, running, crawling, and other modes using gestures; BODY_POSE: Attitude control mode, used to adjust the robot's height and pitch angle; ACTION: Action mode, supports complex actions such as sitting down, standing up, and crossing obstacles; SAFE_STOP: Safe stop state, automatically entered by emergency stop or loss of connection protection.

6. The system according to claim 1, characterized in that, The submodule M2.2 also triggers the following when performing gesture recognition: Submodule M2.2.1: Adopts a "three-stage confirmation" mechanism, in which gesture recognition needs to go through three stages in sequence: "press - hold - release", and the duration of the hold stage is not less than a preset threshold. Submodule M2.2.1: Set the hysteresis interval so that the entry threshold and exit threshold are different.

7. The system according to claim 1, characterized in that, The submodule M2.5 also triggers the following when generating control commands: Submodule M2.5.1: Introduces a speed variation limiter to limit the desired speed v. des Compared with the output speed v at the previous moment cmd [t The difference in 1] is limited to ensure that the speed change does not exceed a preset range. The limiting process can be expressed as: Among them, v cmd [t] represents the output speed command at the current moment, v des Δv is the desired velocity obtained from the gesture or posture mapping. max The maximum allowable speed variation range; the function clamp(x,a,b) restricts the variable x to the interval [a,b].

8. The system according to claim 1, characterized in that, The system also includes a security mechanism module, which includes: The dual emergency stop mechanism module is used to immediately stop the robot's movement and switch to a safe stop state when a preset emergency stop gesture is detected or an emergency stop button trigger signal is received. The communication loss protection mechanism is used to automatically switch the robot to a safe stop state when no control data is received from the data glove within a preset time threshold. The speed limiting and virtual fence protection mechanism is used to limit the linear and angular velocities of robot control commands, and to trigger speed limiting or stop when the robot moves beyond the preset safe area.

9. A method for remotely controlling a human-computer interaction robot based on a data glove, characterized in that, include: Step S1: Hand data acquisition. The bending angle of the fingers and the hand posture information are acquired by bending sensors and IMUs placed on the hand, and high-speed sampling, filtering and fusion processing are completed by the microcontroller. Step S2: Action information preprocessing, including noise suppression, normalization, and feature extraction of the collected data; Step S3: Gesture recognition, determining the operator's current gesture based on feature data; Step S4: Pose mapping and command generation, mapping gesture and posture information into robot control commands, and selecting the control mode according to the state machine; Step S5: Command transmission, sending control commands to the robot via ROS2 or other communication interfaces; Step S6: Action execution, the robot executes the corresponding motion, navigation, grasping or manipulation actions according to the control commands; Step S7: Safety mechanism processing. In the event of misidentification, communication interruption, or out-of-bounds operation, an emergency stop, amplitude limit, or loss of connection protection mechanism is triggered.

Images