Teleoperated tactile feedback manipulator system

Abstract

The invention discloses a teleoperation tactile feedback manipulator system which comprises a bionic manipulator, a somatosensory glove and a hand posture recognition module, the bionic manipulator is in wireless communication with the somatosensory glove, and the hand posture recognition module is in wireless communication with the bionic manipulator and the somatosensory glove. According to the teleoperated tactile feedback manipulator system, through cooperative work of the somatosensory glove and the bionic manipulator, an operator can visually and flexibly control the manipulator, the operation precision and efficiency are improved, the flexible pressure sensor is used for detecting the grabbing pressure in real time, and a pressure signal is fed back to the somatosensory glove; the motion sensing glove generates corresponding tactile feedback according to the pressure signal, so that an operator can sense the grabbing condition of the bionic manipulator, the object is prevented from being damaged or sliding off, the operation accuracy and reliability of the bionic manipulator are further improved, support is provided for fine operation in a complex environment, and the motion sensing glove is convenient to install and has good applicability.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial machinery, and specifically to a remotely operated haptic feedback robotic hand system. Background Art

[0002] In recent years, in the fields of industrial production, operation in dangerous environments, and medical treatment, remote control is often required to complete specific tasks. Traditional remotely controlled robotic hands are usually operated through devices such as buttons and joysticks, lacking intuitiveness and flexibility, and unable to provide haptic feedback to the operator, resulting in limited operation accuracy and efficiency.

[0003] With the development of robot technology and sensor technology, remotely controlled robotic hands based on somatosensory and visual recognition have emerged. However, these systems still have deficiencies in haptic feedback and cannot meet the requirements of complex operations. Therefore, a remotely operated haptic feedback robotic hand system is specifically proposed to achieve intuitive and flexible control of the robotic hand by the operator, provide real-time haptic feedback, and improve the operation accuracy and efficiency. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the present invention provides a remotely operated haptic feedback robotic hand system, which solves the problems raised in the above background art.

[0005] To achieve the above objectives, the present invention is realized through the following technical solutions: A remotely operated haptic feedback robotic hand system, comprising:

[0006] A bionic robotic hand, which is used to simulate the joint movement of a human hand and continuously monitors the grasping pressure through a flexible pressure sensor.

[0007] A somatosensory glove, the bionic robotic hand communicates wirelessly with the somatosensory glove, and the somatosensory glove is used for haptic feedback according to the grasping pressure.

[0008] A potentiometer, the potentiometer is installed on the disc of the somatosensory glove, and there is a rotary potentiometer for each finger connecting a wire to the finger tip to detect the voltage change caused by the resistance change of the potentiometer and judge the finger bending angle.

[0009] A hand gesture recognition module, the hand gesture recognition module communicates wirelessly with the bionic robotic hand and the somatosensory glove respectively, and the hand gesture recognition module is used to obtain the finger bending angle after the operator wears the somatosensory glove and drive the mechanical fingers of the bionic robotic hand to perform joint movement.

[0010] The present invention is further configured as: Five brushless motors and five flexible pressure sensors are arranged on the bionic robotic hand.

[0011] The five brushless motors are respectively used to control the joint movements of the five mechanical fingers of the bionic manipulator;

[0012] The five flexible pressure sensors are respectively used to detect the grasping pressures of the five mechanical fingers of the bionic manipulator.

[0013] The present invention is further configured such that: five servos are provided on the somatosensory glove, and the five servos are used to limit the hand movements of the operator through a self-locking mechanism.

[0014] The present invention is further configured such that: the hand gesture recognition module includes a camera and a main control module;

[0015] The camera is used to collect the hand video stream of the operator in real time;

[0016] The main control module is used to estimate the hand gesture according to the hand video stream, and control the brushless motor to perform the bending control of the mechanical fingers of the bionic manipulator according to the estimated hand gesture.

[0017] The present invention is further configured such that: the method for calculating the finger bending angle includes:

[0018] Read the voltage at both ends of the potentiometer through an ADC analog-to-digital converter, and the read value is a 12-bit digital quantization value. After preprocessing through a low-pass filtering algorithm, map its range to the range of the finger bending angle, and then the finger bending angle can be calculated according to this digital quantization value.

[0019] The formula of the low-pass filtering algorithm is as follows:

[0020] result = α × pre_value + (1 - α) × value

[0021] In the formula, result is the filtered result, α is the low-pass filtering coefficient, pre_value is the value read by the ADC last time, and value is the value read by the ADC this time;

[0022] The calculation method for mapping the digital quantization value range to the finger bending angle is:

[0023]

[0024] In the formula, θ * is the finger bending angle, value ADC is the digital quantization value read by the ADC, range angle is the finger bending angle range, range ADC is the digital quantization value range;

[0025] The method for estimating the hand gesture according to the hand video stream includes:

[0026] Preprocess the hand video stream, and determine the bending angle of the operator's finger by the method of determining the included angle with three points. Among them, the methods for preprocessing the hand video stream include: standardizing the video frame rate, resolution, horizontal image flipping, and format conversion.

[0027] The present invention is further configured as: the method of determining the included angle with three points includes:

[0028] Obtain three nodes in the same finger joint. After defining the vectors, calculate the bending angle. The calculation formula is:

[0029]

[0030] In the formula, θ is the bending angle, A is the base joint point of the finger, B is the middle joint point of the finger, C is the end joint point of the finger, is the vector from the base joint point to the middle joint point, is the vector from the middle joint point to the end joint point.

[0031] The present invention is further configured as: the method of controlling the bending of the mechanical finger of the bionic manipulator by controlling the brushless motor according to the estimated hand posture includes:

[0032] Perform Kalman filtering on the measured value of the operator's finger bending angle to obtain the angle prediction value. Construct a linear relationship between the absolute position of the brushless motor of the bionic manipulator and the angle prediction value. Control the bending of the mechanical finger of the bionic manipulator by controlling the absolute position of the brushless motor, so as to realize the bending control of the mechanical finger of the bionic manipulator.

[0033] The present invention is further configured as: during the process of controlling the bending of the mechanical finger of the bionic manipulator by controlling the absolute position of the brushless motor, the flexible pressure sensor real-time detects the grasping pressure and feeds back the grasping pressure to the somatosensory glove. After the grasping is processed by Kalman filtering, the pressure prediction value is obtained. When the pressure prediction value approaches the preset threshold, adjust the feedback force of the somatosensory glove to the operator. When the pressure prediction value reaches the preset threshold, the servo starts the self-locking mechanism to limit the hand movement of the operator.

[0034] The present invention provides a remotely operated tactile feedback manipulator system. It has the following beneficial effects:

[0035] Through the collaborative work of the somatosensory glove and the bionic manipulator, the present invention realizes the intuitive and flexible control of the manipulator by the operator, improves the operation accuracy and efficiency, and can use the flexible pressure sensor to detect the grasping pressure in real time and feedback the pressure signal to the somatosensory glove. The somatosensory glove generates corresponding tactile feedback according to the pressure signal, enabling the operator to perceive the grasping situation of the bionic manipulator, avoiding object damage or slipping, further improving the accuracy and reliability of the bionic manipulator operation, providing support for fine operations in complex environments, being easy to install, being able to well simulate human motion behaviors, and having good applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the system architecture of the present invention;

[0037] Figure 2 It is a bottom view schematic diagram of the structure of the somatosensory glove of the present invention;

[0038] In the figure: 1. Potentiometer; 2. Somatosensory glove;

[0039] Figure 3 It is a side view of the structure of the somatosensory glove of the present invention;

[0040] In the figure, 101. Bracket; 102. Servo bracket; 103. Disc; 104. Servo;

[0041] Figure 4 It is a schematic diagram of the structure of the simulation manipulator of the present invention;

[0042] In the figure, 3. Manipulator; 4. Brushless motor; 5. Mechanical finger with shell; 6. Flexible pressure sensor; 7. Mechanical finger without shell; 8. Mechanical thumb;

[0043] Figure 5 It is a waveform diagram of the angle measurement value in the embodiment of the present invention;

[0044] Figure 6 It is a waveform diagram of the angle prediction value after Kalman filtering processing in the embodiment of the present invention;

[0045] Figure 7 It is a waveform diagram of the pressure measurement value in the embodiment of the present invention;

[0046] Figure 8 It is a waveform diagram of the pressure prediction value after Kalman filtering processing in the embodiment of the present invention;

[0047] Figure 9 It is a schematic diagram of 21 key hand nodes in the embodiment of the present invention;

[0048] Figure 10 It is a self-locking schematic diagram of the limit screw on the disc and the servo steering disc in the embodiment of the present invention

[0049] In the figure: 201, servo steering wheel; 202, self-locking screw. Specific implementation mode

[0050] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention.

[0051] Please refer to Figures 1-10 , the embodiments of the present invention provide the following two technical solutions:

[0052] Embodiment 1. A remotely operated tactile feedback manipulator system includes a bionic manipulator, a somatosensory glove, and a hand gesture recognition module. The bionic manipulator is used to simulate the joint movement of the human hand. The finger joints of the bionic manipulator adopt a multi-degree-of-freedom design, which can simulate the complex movements of the human hand, improve the flexibility and precision of operation. As shown in the appendix Figure 4 , five brushless motors are arranged on the bionic manipulator. The five brushless motors are respectively used to control the joint movements of the five mechanical fingers of the bionic manipulator. Specifically, the brushless motors use a field-oriented control algorithm for position control, and control the mechanical finger joints through mechanical structure transmission. The outer rotor of the brushless motor rotates in the rotation range [0, 2π] and controls the finger bending angle through a mechanical link mechanism;

[0053] As a preferred solution, the hand gesture recognition module communicates wirelessly with the bionic manipulator and the somatosensory glove respectively. The wireless communication methods include but are not limited to Bluetooth, WIFI, and ZigBee. The hand gesture recognition module is used to obtain the finger bending angle of the operator wearing the somatosensory glove and drive the mechanical fingers of the bionic manipulator to perform joint movements. Specifically, the hand gesture recognition module includes a camera and a main control module;

[0054] The camera is used to collect the hand video stream of the operator in real time;

[0055] The main control module uses an STM32 series single-chip microcomputer as the core processor, which is used to estimate the hand gesture according to the hand video stream and control the brushless motor to perform the bending control of the mechanical fingers of the bionic manipulator according to the estimated hand gesture. The methods for estimating the hand gesture according to the hand video stream include:

[0056] Preprocessing of the potentiometer voltage ADC value, including a low-pass filtering algorithm, and the calculation method is as follows:

[0057] result = α × pre_value + (1 - α) × value

[0058] In the formula, result is the filtered result, α is the low-pass filtering coefficient, pre_value is the previous value read by the ADC, and value is the current value read by the ADC;

[0059] The calculation method for mapping the digital quantization value range to the finger bending angle is as follows:

[0060]

[0061] In the formula, θ * is the finger bending angle, value ADC is the digital quantization value read by the ADC, range angle is the finger bending angle range, range ADC is the digital quantization value range;

[0062] Preprocess the hand video stream. The preprocessing methods include: standardizing the video frame rate, resolution, horizontal image flipping, and format conversion. Then, determine the operator's finger bending angle by the method of determining the included angle with three points. The method of determining the included angle with three points includes:

[0063] Identify 21 main key nodes through the hand detection algorithm, as shown in the appendix Figure 9 Each key node corresponds to a specific part of the hand. Each key node has three coordinate values (x, y, z), where x and y respectively represent the positions in the two-dimensional image, and z represents the depth information. Obtain three nodes in the same finger joint, define a vector, and then calculate the bending angle. The calculation formula is:

[0064]

[0065] In the formula, θ is the bending angle, A is the base joint point of the finger, B is the middle joint point of the finger, C is the end joint point of the finger, is the vector from the base joint point to the middle joint point, is the vector from the middle joint point to the end joint point.

[0066] The method for controlling the bending of the mechanical finger of the bionic mechanical hand according to the estimated hand posture to control the brushless motor includes:

[0067] Perform Kalman filtering on the measured value of the operator's finger bending angle to obtain the angle prediction value. Construct a linear relationship between the absolute position of the brushless motor of the bionic mechanical hand and the angle prediction value, and control the bending of the mechanical finger of the bionic mechanical hand by controlling the absolute position of the brushless motor to achieve the bending control of the mechanical finger of the bionic mechanical hand.

[0068] Embodiment 2. As an improvement over the previous embodiment, a remotely operated haptic feedback robotic hand system further includes five flexible pressure sensors provided on the bionic robotic hand. The model of the flexible pressure sensor is RP-L-110. When the surface pressure changes, its resistance value changes. It is used to detect in real time the pressure value of the contact between the robotic fingers and the object during the grasping process and convert the pressure signal into an electrical signal. The five flexible pressure sensors are respectively used to detect the grasping pressures of the five robotic fingers of the bionic robotic hand. There are five servos provided on the somatosensory glove. The servo model is SG90. The five servos are used to limit the hand movements of the operator through a self-locking mechanism. Among them, the somatosensory glove is for the operator to wear. The surface of the somatosensory glove is made of a soft and breathable material, with a padding inside and a limiting device outside, such as Figure 10 as shown. The hand movement is restricted by locking the limiting screw on a servo disc connected to the finger by a servo, without large-area rigid instruments for restriction, improving the wearing comfort of the operator. The material color of the somatosensory glove is close to the skin color, which does not affect the detection of key hand nodes. And the somatosensory glove is used for haptic feedback according to the grasping pressure. The specific haptic feedback methods include:

[0069] During the process of controlling the bending of the robotic fingers of the bionic robotic hand by controlling the absolute position of the brushless motor, the flexible pressure sensor detects the grasping pressure in real time and feeds back the grasping pressure to the somatosensory glove. After the grasping is processed by Kalman filtering, the pressure prediction value is obtained. When the pressure prediction value approaches the preset threshold, the feedback strength of the somatosensory glove to the operator is adjusted. When the pressure prediction value reaches the preset threshold, the servo starts the self-locking mechanism to limit the hand movement of the operator

[0070] The strength feedback adjusts the time of self-locking and unlocking by the servo within a short period of milliseconds. During the self-locking time, the servo locks the limiting screw to restrict the finger bending, and during the unlocking time, the finger can bend freely. Thus, on the macroscopic time accumulated by multiple short periods, it is reflected as generating different torques, thereby adjusting the feedback strength.

[0071] The advantages of Embodiment 2 over Embodiment 1 are that the grasping accuracy and reliability of the bionic robotic hand can also be ensured in the way of haptic feedback.

[0072] Further explanation, the processing process and calculation expression of the angle prediction value include:

[0073] State prediction:

[0074] X k = X k-1 + U k

[0075] In the formula, X k is the predicted hand angle value at the kth time, X k-1is the predicted value of the angle at the (k-1)th time, U k Peripheral control input;

[0076] Error covariance prediction:

[0077] P k = P k-1 + Q

[0078] where P k is the error covariance predicted at the kth time, P k-1 is the error covariance predicted at the (k-1)th time, and Q is the process noise covariance;

[0079] Kalman gain calculation:

[0080]

[0081] where R is the covariance of the measurement noise, and K k is the Kalman gain;

[0082] Update state estimate:

[0083] X k = X k-1 + K k1 ·(Z k1 - X k )

[0084] where X k is the updated angle estimate, and Z k1 is the currently measured angle;

[0085] Update error covariance:

[0086] P k - = (1 - K k1 )·P k

[0087] where P k - is the estimated error covariance;

[0088] The conversion from the angle to the absolute position control signal is:

[0089]

[0090] where Position is the absolute position of the brushless motor, X k is the updated angle estimate, and θ max is the maximum value of the bendable angle of the finger.

[0091] As shown in Appendix Figure 5 and Appendix Figure 6As shown, it can be found that after Kalman filtering, the waveform of the predicted value becomes stable. Such a stable predicted value of the analog quantity of the angle is convenient for constructing a linear relationship between the absolute position and the predicted value. The expression of the linear relationship is as follows:

[0092] Position=AX k +B

[0093] Where, Position represents the absolute position of the brushless motor, X k represents the predicted value of the angle, and both A and B represent constant parameters. By presetting several groups of angle predicted values X k generated when the bending degree of the finger is consistent with the bending degree of the bionic manipulator and the required absolute position of the brushless motor, substituting them into the linear relationship, the constant parameters represented by A and B can be obtained.

[0094] As an alternative solution, the system performance can also be optimized by adjusting the process noise covariance Q and measurement noise covariance R of the Kalman filter.

[0095] The processing process and calculation expression of the pressure predicted value include:

[0096] The somatosensory glove receives the pressure signal from the flexible pressure sensor through wireless communication. Through program coding, the pressure analog quantity is set as Z k2 and performs first-order Kalman filtering on the pressure analog quantity to obtain the predicted value Y k , and its calculation expression is as follows:

[0097] Y k =Y k-1 +K k2 ×(Z k2 -Y k-1 )

[0098]

[0099]

[0100] Where, Y k represents the pressure predicted value at the k-th time, Y k-1 represents the pressure predicted value at the (k - 1)-th time, K k2 represents the Kalman filter coefficient, Z k2 represents the grasping pressure at the k-th time, represents the prediction error value at the k-th time, represents the prediction error value at the (k - 1)-th time, represents the measurement error value at the k-th time.

[0101] As shown in the appendix Figure 7 and appendix Figure 8As shown, it can be found that after Kalman filtering, the waveform of the pressure prediction value becomes stable. Only such a stable predicted value of the pressure analog quantity is convenient for constructing a linear relationship.

[0102] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A teleoperated tactile feedback manipulator system, characterized in that: include: A bionic manipulator, which is used to simulate the joint movement of a human hand and monitor the grasping pressure in real time through a flexible pressure sensor; A somatosensory glove, wherein the bionic manipulator communicates wirelessly with the somatosensory glove, and the somatosensory glove is used to provide tactile feedback according to grasping pressure; Potentiometer, the potentiometer is installed on the somatosensory glove, and each finger has a rotary potentiometer connected to a line to the finger end, which is used to calculate the bending angle of the finger; A hand gesture recognition module, which wirelessly communicates with the bionic manipulator and the somatosensory glove respectively, and is used to obtain the bending angle of the operator's fingers after wearing the somatosensory glove, and drive the mechanical fingers of the bionic manipulator to perform joint movements.

2. A teleoperated tactile feedback manipulator system according to claim 1, characterized in that: The bionic manipulator is provided with five brushless motors and five flexible pressure sensors; The five brushless motors are respectively used to control the joint movements of the five mechanical fingers of the bionic manipulator; The five flexible pressure sensors are respectively used to detect the grasping pressure of the five mechanical fingers of the bionic manipulator.

3. A teleoperated tactile feedback manipulator system according to claim 2, characterized in that: The somatosensory gloves are provided with five servos, which are used to limit the hand movements of the operator through a self-locking mechanism.

4. A teleoperated tactile feedback manipulator system according to claim 3, characterized in that: The hand gesture recognition module includes a camera and a main control module; The camera is used to collect the operator's hand video stream in real time; The main control module is used to estimate the hand posture according to the hand video stream, and control the brushless motor to perform bending control of the mechanical fingers of the bionic manipulator according to the estimated hand posture.

5. A teleoperated tactile feedback manipulator system according to claim 4, characterized in that: The method of estimating the hand posture according to the hand video stream includes: The hand video stream is preprocessed, and the bending angle of the operator's finger is determined by determining the angle by three points, wherein the preprocessing method of the hand video stream includes: standardizing the video frame rate, resolution, horizontal image flipping and format conversion.

6. A teleoperated tactile feedback manipulator system according to claim 5, characterized in that: The method for calculating the finger bending angle includes: The voltage across the potentiometer is read through the ADC analog-to-digital converter. The read value is a 12-bit digital quantization value. After pre-processing by a low-pass filtering algorithm, its range is mapped to the range of the finger bending angle. After that, the finger bending angle can be solved based on the digital quantization value. The low-pass filtering algorithm formula is as follows: result=α×pre_value+(1-α)×value In the formula, result is the result after filtering, α is the low-pass filter coefficient, pre_value is the value read by ADC last time, and value is the current value read by ADC; The calculation method of mapping the digital quantization value range to the finger bending angle is: In the formula, θ * is the finger bending angle, value ADC The digital quantization value read by ADC, range angle Finger bending angle range, range ADC It is the range of digital quantization value.

7. A teleoperated tactile feedback manipulator system according to claim 6, characterized in that: The three-point angle determination method includes: Get the three nodes in the same finger joint, define the vector, and calculate the bending angle. The calculation formula is: Where θ is the bending angle, A is the base joint of the finger, B is the middle joint of the finger, and C is the end joint of the finger. is the vector from the base joint point to the middle joint point, is the vector from the middle joint point to the end joint point.

8. A teleoperated tactile feedback manipulator system according to claim 7, characterized in that: The method of controlling the brushless motor to perform bending control of the mechanical fingers of the bionic manipulator according to the estimated hand posture includes: The measured values ​​of the operator's finger bending angle are processed by Kalman filtering to obtain the angle prediction value, and a linear relationship between the absolute position of the brushless motor of the bionic manipulator and the angle prediction value is constructed. The bending control of the bionic manipulator's mechanical fingers is carried out by controlling the absolute position of the brushless motor, thereby realizing the bending control of the bionic manipulator's mechanical fingers. The absolute position of the brushless motor refers to the actual position of the outer rotor of the brushless motor within the rotation range [0, 2π].

9. A teleoperated tactile feedback manipulator system according to claim 7, characterized in that: In the process of controlling the bending of the mechanical fingers of the bionic manipulator by controlling the absolute position of the brushless motor, the flexible pressure sensor detects the grasping pressure in real time and feeds back the grasping pressure to the somatosensory glove. After the grasping is processed by the Kalman filter, the pressure prediction value is obtained. When the pressure prediction value is close to the preset threshold, the feedback strength of the somatosensory glove to the operator is adjusted. When the pressure prediction value reaches the preset threshold, the servo starts the self-locking mechanism to limit the operator's hand movement.

Images