Multi-modal data glove

Abstract

The invention discloses a multi-modal data glove. The multi-modal data glove comprises a glove body; the power supply circuit, the sampling circuit and the wireless transmission circuit are electrically connected with the main controller; the finger sensor is electrically connected with the sampling circuit and used for collecting finger data and sending the finger data to the sampling circuit; the palm pose sensing circuit is electrically connected with the main controller and is used for acquiring pose data of a palm and sending the pose data to the main controller; the sampling circuit is used for filtering and amplifying opponent data and then sending the opponent data to the main controller; the wireless transmission circuit is used for sending the data received by the main controller to an upper computer for processing; and the power supply circuit is used for supplying power to the main controller, the wireless transmission circuit, the finger sensor, the palm pose sensing circuit and the sampling circuit. According to the multi-modal data glove, data acquisition of touch, finger postures and palm postures can be achieved, and data transmission is achieved in a wireless mode.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition equipment, and in particular to a multimodal data glove. Background Art

[0002] Data gloves can collect a variety of hand data, but existing data gloves have problems in motion capture systems such as insufficient degrees of freedom, small range of posture information, low sensitivity, and no tactile feedback. Summary of the invention

[0003] The present invention provides a multimodal data glove, which can realize data collection of touch, finger posture and palm posture, and realize data transmission through wireless.

[0004] An embodiment of the present invention provides a multimodal data glove, comprising:

[0005] Glove body;

[0006] A main controller and a power supply circuit, a sampling circuit, and a wireless transmission circuit electrically connected to the main controller;

[0007] A finger sensor, electrically connected to the sampling circuit, for collecting finger data and sending the finger data to the sampling circuit;

[0008] A palm posture sensing circuit is electrically connected to the main controller and is used to collect palm posture data and send it to the main controller;

[0009] The sampling circuit is used to filter and amplify the finger data and then send it to the main controller;

[0010] The wireless transmission circuit is used to send the data received by the main controller to the host computer for processing;

[0011] The power supply circuit is used to supply power to the main controller, the wireless transmission circuit, the finger sensor, the palm posture sensing circuit and the sampling circuit.

[0012] Optionally, in one embodiment of the present invention, the finger sensor includes five tactile sensors at the thumb end, index finger end, middle finger end, ring finger end and little finger end, and six curvature sensors at the thumb, index finger, middle finger, ring finger and little finger end.

[0013] Optionally, in one embodiment of the present invention, the data glove further comprises:

[0014] The calibration module connected to the main controller is used to complete the mapping of the angle and pressure of the fingers, palms and fingertips through the curvature calibration algorithm and the fingertip pressure algorithm.

[0015] Optionally, in one embodiment of the present invention, the calibration module is specifically used to complete the calibration of the bending sensor through three basic gestures, the three basic gestures are five fingers together, C-shaped virtual grip, and clenched fist. The calibration principle of the bending degree calibration algorithm is derived based on Ohm's voltage divider law, and the average resistance of five fingers together, the average resistance of C-shaped virtual grip, and the average resistance of clenched fist are mapped to angles of 0, 45, and 90, completing the mapping of resistance to finger angle bending.

[0016] Optionally, in one embodiment of the present invention, the calibration module is specifically used for: a fingertip pressure algorithm completes the mapping of resistance to fingertip pressure based on Ohm's voltage division law and a reference resistance-pressure curve of the tactile sensor.

[0017] Optionally, in one embodiment of the present invention, the data glove further comprises:

[0018] The visualization module connected to the main controller is used to receive the data processed by the main controller, visualize the received data, and display hand tactile information or hand posture information in real time.

[0019] Optionally, in one embodiment of the present invention, the data glove further comprises:

[0020] A charging circuit connected to the power circuit is used to charge the power circuit.

[0021] Optionally, in one embodiment of the present invention, the sampling circuit includes:

[0022] Low-pass filter and operational amplifier for filtering and amplifying finger data.

[0023] The multimodal data gloves of the embodiment of the present invention can realize the data collection of touch, finger posture, and palm posture, and realize data transmission through wireless.

[0024] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A schematic diagram of a circuit of a multimodal data glove provided according to an embodiment of the present invention;

[0027] Figure 2 A sampling circuit diagram of an embodiment of the present invention;

[0028] Figure 3A palm posture sensing circuit diagram of an embodiment of the present invention;

[0029] Figure 4 is a flowchart of a program according to an embodiment of the present invention;

[0030] Figure 5 A power supply circuit diagram of an embodiment of the present invention;

[0031] Figure 6 FIG. 4 is a charging circuit diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0033] like Figure 1 As shown, the multimodal data glove comprises:

[0034] Glove body;

[0035] A main controller and a power supply circuit, a sampling circuit, and a wireless transmission circuit electrically connected to the main controller;

[0036] The finger sensor is electrically connected to the sampling circuit and is used to collect finger data and send the finger data to the sampling circuit;

[0037] The palm posture sensing circuit is electrically connected to the main controller and is used to collect the palm posture data and send it to the main controller;

[0038] The sampling circuit is used to filter and amplify the finger data and then send it to the main controller;

[0039] The wireless transmission circuit is used to send the data received by the main controller to the host computer for processing;

[0040] The power supply circuit is used to supply power to the main controller, wireless transmission circuit, finger sensor, palm posture sensing circuit and sampling circuit.

[0041] like Figure 2 As shown in Figure , the circuit diagram of the sampling circuit is shown. Figure 3 As shown, the palm posture sensing circuit diagram is shown.

[0042] In an embodiment of the present invention, the main controller adopts STM32 series MCU, the semiconductor model is STM32F103RCT6, and the main controller is responsible for data collection, data integration processing, and data forwarding. After the main controller completes all data collection, it converts the data into hexadecimal and re-encodes it.

[0043] In an embodiment of the present invention, the finger sensor includes five tactile sensors at the thumb, index finger, middle finger, ring finger and little finger, and six curvature sensors at the thumb, index finger, middle finger, ring finger and little finger.

[0044] The finger front curvature sensor uses the imported Flex sensor flex 4.5" unidirectional bending sensor, which can achieve 180-degree bending measurement of the finger. The thumb inner flex 2.2" unidirectional bending sensor can detect the thumb swing angle. The analog data of the sensor is sent to the sampling circuit.

[0045] In an embodiment of the present invention, the palm posture sensing circuit is electrically connected to the main controller via an I2C interface. The palm posture sensor is implemented using a 6-axis IMU chip BM270, and the sampled data is input to the main controller via the I2C interface, and the main controller performs compensation, filtering and other algorithms to process the data.

[0046] In the embodiment of the present invention, the data connection between the wireless transmission circuit and the main control part adopts a serial port connection. The wireless transmission circuit mainly transmits the data processed by the main controller to the host computer software for analysis through the WIFI wireless transmission protocol. The wireless transmission circuit adopts a wireless communication module to transparently transmit data with the host computer. The Wi-Fi module supports the IEEE802.11b / g / n protocol, and the maximum transmission speed can reach 150Mbps.

[0047] The wireless transmission circuit uses Espressif's ESP32-C3 series module, model ESP32-C3-WROOM-02-N4, which supports general Wi-Fi and low-power Bluetooth (Bluetooth LE) modules and IEEE 802.11b / g / n protocols, with a maximum transmission speed of up to 150Mbps. The module selectively sends UDP data synchronously, including pressure data, palm posture data, and finger joint movement data, with a delay of less than 50ms.

[0048] In an embodiment of the present invention, the embedded software system part is as follows Figure 4As shown, it is roughly divided into two parts. One part is the MCU data acquisition: after initializing BMI270, ADC, IIC, UART and ESP32, read the ADC data (bending and pressure) and gyroscope raw data from the MCU peripherals, then calculate the Euler angle according to the gyroscope raw data, and transmit the ADC data, gyroscope raw data and converted Euler angle to ESP32 in modbus format; the other part is ESP32 configuration and data reception: after initializing the NVS and WiFi of ESP32, establish a serial port receiving task to receive the glove data transmitted by the MCU, then establish a UDP server connection and sending task, and forward the received data to the connected client, that is, the host computer.

[0049] The upper computer software part includes data receiving module, algorithm calibration module, data sending module and data visualization module.

[0050] The data receiving module receives the raw data from the data glove, then decrypts and presents it to determine the accuracy of the data obtained. After receiving and saving the raw data, the calibration module completes the mapping of the finger, palm, and fingertip pressure through the curvature calibration algorithm and fingertip pressure algorithm.

[0051] In an embodiment of the present invention, the data glove further comprises:

[0052] The calibration module connected to the main controller is used to complete the mapping of finger, palm, fingertip angle and pressure through the curvature calibration algorithm and fingertip pressure algorithm.

[0053] In an embodiment of the present invention, the calibration module is specifically used to complete the calibration of the bending sensor through three basic gestures, the three basic gestures are five fingers together, C-shaped virtual grip, and clenched fist. The calibration principle of the bending degree calibration algorithm is derived based on Ohm's voltage divider law. The average resistance of five fingers together, the average resistance of C-shaped virtual grip, and the average resistance of clenched fist are mapped to angles of 0, 45, and 90, completing the mapping of resistance to finger angle bending.

[0054] Specifically, the calibration principle of the thin film bending sensor is based on Ohm's voltage divider law. Derived Among them, V out is the measured output voltage, V cc is the input reference voltage, R flex is the derived bending resistance data, and R0 is the known voltage-dividing resistance data. Based on formula (2), the average resistance R of the five fingers together is calculated respectively. f0 , C-type virtual grip average resistance R f1 , average resistance R f2 , and then R f0 , Rf1 and R f2 Mapping to angles 0, 45, and 90 completes the mapping of resistance to finger bending angle.

[0055] In an embodiment of the present invention, the calibration module is specifically used for: the fingertip pressure algorithm completes the mapping from resistance to fingertip pressure based on Ohm's voltage division law and the reference resistance-pressure curve of the tactile sensor.

[0056] Specifically, based on Ohm's law of voltage division, And tactile sensor reference resistance-pressure curve The mapping formula from resistance to fingertip pressure is derived as F = -113.7ln(R press )+990.4(5), where V out is the measured output voltage, V cc is the input reference voltage, R press is the derived pressure resistance data, R0 is the known voltage divider resistance data, and F is the measured fingertip pressure data, completing the mapping from resistance to fingertip pressure. Formula (4) comes from the data sheet of the fingertip pressure sensor.

[0057] In an embodiment of the present invention, the acceleration and angular velocity of the inertial measurement unit in the x, y, and z directions are directly read out, and the Euler angle is calculated through quaternion. There are three Euler angles, namely yaw (yaw angle), pitch (pitch angle), and roll (roll angle), which represent the angle of rotation around the Z / Y / X axis. Quaternion is only a way of expressing 3D posture, which consists of a real part and three imaginary parts, which can be expressed as q=[s,v], s=q0, v=[q1,q2,q3], where s is the real part and v is the imaginary part.

[0058] First, the original values ​​of acceleration and angular velocity of the inertial measurement unit in the x, y, and z directions are collected, and then the quaternion differential equation is solved by the first-order Runge-Kutta method. The calculation formula is as follows:

[0059] q0=q0+(-q1*gyro_x-q2*gyro_y-q3*gyr0_z)*T / 2

[0060] q1=q1+(q0*gyro_x+q2*gyro_z-q3*gyr0_y)*T / 2

[0061] q2=q2+(q0-gyro_y-q1*gyro_z+q3-gyr0_x)-T / 2

[0062] q3=q3+(q0*gyro_z+q1*gyro_y-q2*gyr0_x)*T / 2

[0063] Where T is the measurement period, gyro_x, gyro_y, gyro_z are the values ​​of the gyroscope angular velocity after error elimination on the three components of x, y, and z. Then the quaternion is normalized to calculate yaw, pitch, and roll. The calculation formulas are expressed in quaternions as follows:

[0064] pitch=asin(2*q0*q2-2*q1*q3)*180 / π

[0065] roll=atan2(2*q2*q3+2*q0*q1,-2*q1*q1-2*q2*q2+1)*180 / π

[0066] yaw=atan2(2*q1*q2+2*q0*q3,-2*q2*q2-2*q3*q3+1)*180 / π

[0067] Among them, q0, q1, q2, q3 are the components of the quaternion, representing the rotation of the quaternion, q0 is the real part of the quaternion, representing the angle of rotation, q1, q2, q3 are the imaginary part of the quaternion, representing the direction of the rotation axis, and π is pi, which is used in the formula to convert radians to degrees (1 radian is approximately equal to 180 / π degrees).

[0068] The data transmission module converts the data calibrated by the calibration module and sends pressure data, palm posture data and finger joint movement data with a delay of less than 5ms. The data visualization module uses the original data and the converted data to display the hand tactile information in real time in the form of icons through GUI tools, or uses 3D software to reproduce the hand posture.

[0069] In an embodiment of the present invention, the data glove further comprises:

[0070] The visualization module connected to the main controller is used to receive the data processed by the main controller, visualize the received data, and display the hand tactile information or hand posture information in real time.

[0071] In the embodiment of the present invention, the power circuit adopts a 3.7V high-capacity lithium battery with a battery life of 1-2 hours. The boost chip in the power circuit boosts the battery 3.7V to 5V, and then stabilizes the voltage to 3.3V through the voltage regulator chip to power various circuits. Figure 5 shown.

[0072] In an embodiment of the present invention, the data glove further includes: a charging circuit connected to the power circuit, for charging the power circuit.

[0073] The charging circuit uses the commonly used Micro USB and USB Type C data ports for charging, and the lithium battery charging circuit uses a dedicated lithium battery charging management chip for lithium battery charging management. The lithium battery charging circuit uses TP4054 for lithium battery charging management. This IC supports the chip to automatically enter the protection state when the battery is connected reversely, charging current monitoring, input low voltage lockout, automatic recharging, and charging full and charging start signs, such as Figure 6 shown.

[0074] In an embodiment of the present invention, the sampling circuit comprises:

[0075] Low-pass filter and operational amplifier for filtering and amplifying finger data.

[0076] In the embodiments of the present invention, the tactile sensor is not limited to the piezoresistive sensor, and can be replaced by a capacitive proximity sensor, a visual tactile sensor, a dot matrix sensor, or other tactile sensor. The palm posture sensing circuit is not limited to the bending sensor, and can be replaced by other posture sensors such as electronic skin and flexible stretch sensor.

[0077] The multimodal data gloves proposed in the embodiment of the present invention can realize the data collection of touch, finger posture, and palm posture, and realize data transmission through wireless.

[0078] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0079] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A multimodal data glove, characterized in that: include: Glove body; A main controller and a power supply circuit, a sampling circuit, and a wireless transmission circuit electrically connected to the main controller; A finger sensor, electrically connected to the sampling circuit, for collecting finger data and sending the finger data to the sampling circuit; A palm posture sensing circuit is electrically connected to the main controller and is used to collect palm posture data and send it to the main controller; The sampling circuit is used to filter and amplify the finger data and then send it to the main controller; The wireless transmission circuit is used to send the data received by the main controller to the host computer for processing; The power supply circuit is used to supply power to the main controller, the wireless transmission circuit, the finger sensor, the palm posture sensing circuit and the sampling circuit.

2. The multimodal data glove according to claim 1, characterized in that: The finger sensor includes five tactile sensors at the thumb end, index finger end, middle finger end, ring finger end and little finger end, and six curvature sensors at the thumb, index finger, middle finger, ring finger and little finger end.

3. The multimodal data glove according to claim 1, characterized in that: The data glove further comprises: The calibration module connected to the main controller is used to complete the mapping of the angle and pressure of the fingers, palms and fingertips through the curvature calibration algorithm and the fingertip pressure algorithm.

4. The multimodal data glove according to claim 3, characterized in that: The calibration module is specifically used to complete the calibration of the bending sensor through three basic gestures, which are five fingers together, C-shaped virtual grip, and clenched fist. The calibration principle of the bending degree calibration algorithm is derived based on Ohm's voltage divider law. The average resistance of five fingers together, the average resistance of C-shaped virtual grip, and the average resistance of clenched fist are mapped to angles of 0, 45, and 90, completing the mapping of resistance to finger angle bending.

5. The multimodal data glove according to claim 4, characterized in that: The calibration module is specifically used for: the fingertip pressure algorithm is based on Ohm's voltage division law and the reference resistance-pressure curve of the tactile sensor to complete the mapping from resistance to fingertip pressure.

6. The multimodal data glove according to claim 1, characterized in that: The data glove further comprises: The visualization module connected to the main controller is used to receive the data processed by the main controller, visualize the received data, and display hand tactile information or hand posture information in real time.

7. The multimodal data glove according to claim 1, characterized in that: The data glove further comprises: A charging circuit connected to the power circuit is used to charge the power circuit.

8. The multimodal data glove according to claim 1, characterized in that: The sampling circuit comprises: Low-pass filter and operational amplifier for filtering and amplifying finger data.

Images