Somatosensory glove and system based on flexible photoelectric bimodal sensor

Abstract

The invention relates to a somatosensory glove and system based on a flexible photoelectric dual-mode sensor. The somatosensory glove based on the flexible photoelectric bimodal sensor comprises a glove body, the flexible photoelectric bimodal sensor and a control circuit, the flexible photoelectric bimodal sensor is arranged on the glove body and can deform along with the hand action of a user so as to collect an original bimodal signal; the control circuit is electrically connected with the flexible photoelectric dual-mode sensor, and is used for preprocessing an original dual-mode signal acquired by the flexible photoelectric dual-mode sensor to obtain a preprocessed dual-mode signal, and then transmitting the preprocessed dual-mode signal to computing equipment. According to the flexible photoelectric bimodal sensor, bimodal signals caused by hand actions can be respectively collected from two angles through the optical sensing layer and the electric sensing layer, so that richer and more comprehensive action detail information is captured, and the accuracy and sensitivity of action capture are improved.

Description

Technical Field

[0001] The present invention relates to the field of sensors, and in particular to a somatosensory glove and system based on a flexible photoelectric dual-mode sensor. Background Art

[0002] In the prior art, there is a robot that can grasp objects with a manipulator. This robot has shown great application value in application scenarios such as object classification, bomb disposal, and working in dangerous environments such as high-energy rays. How to achieve remote control of the robot's hand movements to achieve the effect of the robot's remote and precise manipulation of objects has become a key issue in whether this robot can be widely used. Although the traditional manipulator control algorithm based on robot kinematics is mature, stable and effective, it is still insufficient in achieving accurate, fast and intuitive object manipulation.

[0003] In this context, somatosensory gloves, as an innovative interactive device, have received widespread attention. They can not only capture and recognize the subtle movements and gestures of the wearer's hands, providing users with a natural and intuitive interactive experience, but also provide accurate data support for the remote control of robot hand movements.

[0004] Existing somatosensory gloves are mainly based on ultrasonic sensors or inertial sensors. For example, the energy gloves developed by Nintendo are a type of somatosensory gloves based on ultrasonic sensors. Its working principle is as follows: an ultrasonic transmitter is provided inside the somatosensory gloves, and a plurality of ultrasonic receivers are provided on the interactive device; the ultrasonic signal emitted by the somatosensory gloves is received by a plurality of ultrasonic receivers. When the user wears the somatosensory gloves and makes a movement, the ultrasonic signal received by the interactive device changes. By analyzing the changed ultrasonic signal, the glove position and hand movement information can be obtained. The working principle of the somatosensory gloves based on inertial sensors (such as the VRTRIX data gloves developed by Wuyuanfujie) is as follows: the acceleration, angular velocity and magnetic field direction of the hand movement are collected by sensors such as accelerometers, gyroscopes and magnetometers provided on the somatosensory gloves, and the hand posture information is obtained by analyzing the above parameter data, thereby realizing hand movement capture.

[0005] However, existing somatosensory gloves have the following defects: 1. The sensor module installed on the gloves is not only large in size, but also cannot be deformed, and the wearing comfort is poor. 2. The motion capture accuracy is low, and the user's hand movements cannot be accurately captured, and misjudgment often occurs. Summary of the invention

[0006] Based on this, the purpose of the present invention is to provide a somatosensory glove and system based on a flexible photoelectric dual-modal sensor, which has the advantages of small size, good wearing comfort, high precision and high sensitivity.

[0007] The present invention provides a somatosensory glove based on a flexible photoelectric dual-mode sensor, comprising: a glove body; a flexible photoelectric dual-mode sensor, which is embedded in the glove body and comprises a stacked electrical sensing layer, an insulating layer and a light sensing layer.

[0008] The present invention uses a flexible photoelectric dual-modal sensor to collect the first modal signal and the second modal signal caused by hand movements from two angles. Compared with single-modal signals, dual-modal signals can capture richer and more comprehensive action detail information, and can correct each other, so that the computing device can more accurately reconstruct the hand movement details through a dual-modal data fusion algorithm, thereby improving the accuracy and sensitivity of motion capture. The invented flexible photoelectric dual-modal sensor can deform with the hand movement and has a small size. Therefore, the somatosensory gloves of the present invention also have the advantages of small size, high integration and more comfortable wearing.

[0009] Furthermore, the light sensing layer includes: a backlight layer, which is a diffuse transmission film, used to convert external light into uniform transmitted light; a pressure-sensitive layer, which is a translucent porous rubber film; and a photosensitive layer, which converts a light spot formed by the transmitted light passing through the pressure-sensitive layer into a first modal signal.

[0010] Furthermore, the electric sensing layer is a silver nanowire-PVDF electrospinning composite layer, which generates a second modal signal.

[0011] Furthermore, the insulating layer is a PDMS gel layer.

[0012] Furthermore, the first modal signal collected by the optical sensing layer and the second modal signal collected by the electrical sensing layer constitute a bimodal signal; the somatosensory glove also includes a control circuit, which includes: a signal conditioning module, which obtains the original bimodal signal collected by the flexible photoelectric bimodal sensor, and performs signal conditioning on the original bimodal signal to obtain a conditioned bimodal signal; a digital-to-analog conversion module, which performs digital-to-analog conversion on the conditioned bimodal signal to obtain a digitized bimodal signal; an output module, which encodes the digitized bimodal signal according to a preset protocol to obtain a preprocessed bimodal signal, and finally transmits the preprocessed bimodal signal to a computing device.

[0013] Furthermore, the somatosensory glove also includes an IMU inertial sensing module arranged on the glove body and electrically connected to the control circuit; the IMU inertial sensing module includes an accelerometer, a gyroscope and a magnetometer, which respectively collect acceleration signals, angular velocity signals and magnetic field direction signals generated by the user's hand movements, and transmit the collected acceleration signals, angular velocity signals and magnetic field direction signals to the control circuit; the acceleration signals, angular velocity signals and magnetic field direction signals collected by the IMU inertial sensing module and the bimodal signals collected by the flexible photoelectric bimodal sensor together constitute a multimodal signal; the control circuit preprocesses each multimodal signal respectively to obtain a preprocessed multimodal signal, and then transmits the preprocessed multimodal signal to the computing device.

[0014] Based on the same inventive concept, the present invention also provides a hand motion capture system, which includes: a computing device and any one of the above-mentioned somatosensory gloves based on flexible photoelectric dual-modal sensors; the somatosensory glove is signal-connected to the computing device, for collecting bimodal signals or multimodal signals generated by the user's hand motions, and transmitting the bimodal signals or multimodal signals to the computing device; the computing device reconstructs the user's hand motion data based on the bimodal signals or multimodal signals.

[0015] Furthermore, the computing device reconstructs hand motion data based on the dual-modal signal, specifically including the following steps: inputting the first modal signal and the second modal signal continuously collected over a period of time into the Kalman filter, the Kalman filter performing hand state estimation and error correction based on the first modal signal and the second modal signal, outputting the corrected hand state, and outputting the final corrected value of the hand state as the hand motion data.

[0016] Furthermore, the computing device reconstructs hand motion data based on the multimodal signal, specifically including the following steps: inputting the multimodal signal into a pre-trained machine learning model, and the machine learning model performing feature extraction, feature fusion and recognition on the multimodal signal, thereby reconstructing the user's hand motion data.

[0017] Furthermore, the machine learning model is trained by the following training method: ST1, allowing the user to wear the somatosensory glove to perform a preset action, and record the standard hand motion data corresponding to the preset action; then the multimodal signal corresponding to the preset action is collected through the flexible photoelectric dual-modal sensor and the IMU inertial sensing module; ST2, the collected multimodal signal is input into the machine learning model to be trained, and the machine learning model is allowed to reconstruct the hand motion data according to the input multimodal signal; ST3, comparing the hand motion data reconstructed by the machine learning model with the difference between the standard hand motion data recorded in step ST1, and calculating the loss function; then according to the loss function, optimizing the parameters of the machine learning model; ST4, repeating steps ST2-ST3, thereby iteratively updating the parameters of the machine learning model, until the number of iterations reaches a preset threshold or the parameters of the machine learning model converge, then the training is terminated to obtain a trained machine learning model.

[0018] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a module schematic diagram of a hand motion capture system according to Embodiment 1 of the present invention; Figure 2 This is a schematic structural diagram of a somatosensory glove based on a flexible photoelectric dual-modal sensor according to Example 1 of the present invention; Figure 3 Schematic diagram of the structure of a flexible photoelectric dual-mode sensor according to Example 1 of the present invention; Figure 4 This is a module schematic diagram of the control circuit of Example 1 of the present invention. DETAILED DESCRIPTION

[0020] Flexible somatosensory gloves based on optical sensors have emerged in the prior art, which are expected to solve the problem of poor wearing comfort of traditional somatosensory gloves based on ultrasonic sensors or inertial sensors. However, the existing flexible somatosensory gloves based on optical sensors are still in the early stages of research, with low integration, and the potential impact of factors such as light penetration, scattering, and changes in ambient light on detection accuracy and stability has not been fully resolved. For example, Jingyi Zhou and Qi Shao proposed to apply flexible optical waveguide multi-axis tactile sensors to the field of sensor interaction, which can collect interactive multi-axis information generated by user hand movements, but still have the defects of low integration, large size, difficulty in decoding sensor data, and poor motion capture accuracy and sensitivity.

[0021] In view of the defects of existing somatosensory gloves, the inventors have developed a new type of flexible photoelectric dual-modal sensor, which can collect the first modal signal and the second modal signal caused by hand movements from two angles respectively through the optical sensing layer and the electrical sensing layer. Compared with single-modal signals, dual-modal signals can capture richer and more comprehensive action detail information, and can correct each other, so that the computing device can more accurately reconstruct the hand movement details through the dual-modal data fusion algorithm, thereby improving the accuracy and sensitivity of motion capture. Since the flexible photoelectric dual-modal sensor can deform with the hand movement and has a small size, the somatosensory gloves of the present invention also have the advantages of small size, high integration and more comfortable wearing.

[0022] Example 1 See also Figure 1 , Figure 1 It is a module schematic diagram of the hand motion capture system of Example 1 of the present invention. The hand motion capture system of Example 1 of the present invention includes a somatosensory glove 1 and a computing device 2. The somatosensory glove 1 is the somatosensory glove based on a flexible photoelectric bimodal sensor of the present invention. The somatosensory glove 1 is signal-connected to the computing device 2, and is used to collect the bimodal signal generated by the user's hand motion, and transmit the bimodal signal to the computing device 2. The computing device 2 parses the bimodal signal and reconstructs the user's hand motion data. The hand motion data can be used for human-computer interaction or robot control, etc. The present invention does not limit the subsequent use of the hand motion data.

[0023] See also Figure 2 , Figure 2 The structure diagram of the somatosensory glove based on the flexible photoelectric dual-modal sensor according to the first embodiment of the present invention is as follows. The somatosensory glove based on the flexible photoelectric dual-modal sensor according to the first embodiment of the present invention comprises: a glove body 11 , a flexible photoelectric dual-modal sensor 12 and a control circuit 13 .

[0024] The flexible photoelectric bimodal sensor 12 is fixedly arranged on the glove body 11. Specifically, the flexible photoelectric bimodal sensor 12 covers the finger part of the glove body 11 and can be deformed following the user's finger movements. When the user wears the glove body 11, the user's hand movements can cause the flexible photoelectric bimodal sensor 12 to deform, so that the flexible photoelectric bimodal sensor 12 collects the original bimodal signal.

[0025] The flexible photoelectric bimodal sensor 12 can be embedded in the glove body 11 or fixed on the outer surface of the glove body 11. In this embodiment, the flexible photoelectric bimodal sensor 12 is fixed on the outer surface of the glove body 11.

[0026] The control circuit 13 is electrically connected to the flexible photoelectric bimodal sensor 12, and is used to perform signal conditioning, analog-to-digital conversion and other preprocessing on the original bimodal signal collected by the flexible photoelectric bimodal sensor 12 to obtain a preprocessed bimodal signal, and then transmit the preprocessed bimodal signal to the computing device 2 for analysis and modeling by the computing device 2.

[0027] The control circuit 13 can be fixed on the glove body 11 or separated from the glove body 11 and electrically connected to the flexible photoelectric dual-mode sensor 12 via a wire. In this embodiment, the control circuit 13 is fixedly arranged on the back of the glove body 13.

[0028] See also Figure 3 , Figure 3 Schematic diagram of the structure of the flexible photoelectric dual-modal sensor of Example 1 of the present invention.

[0029] Specifically, the flexible photoelectric bimodal sensor 12 includes: a stacked optical sensing layer 121, an insulating layer 122, and an electric sensing layer 123. The original bimodal signal collected by the flexible photoelectric bimodal sensor 12 specifically includes a first modal signal collected by the optical sensing layer 121 and a second modal signal collected by the electric sensing layer 123. The optical sensing layer 121 is provided with a first electrode connected to the control circuit 13, which transmits the first modal signal to the control circuit 13 through the first electrode. The electric sensing layer 123 is provided with a second electrode connected to the control circuit 13, which transmits the second modal signal to the control circuit 13 through the second electrode. The insulating layer 122 is provided between the optical sensing layer 121 and the electric sensing layer 123, and is used to bond the optical sensing layer 121 and the electric sensing layer 123.

[0030] Specifically, the light sensing layer 121 includes: a backlight layer, a pressure-sensitive layer and a photosensitive layer which are stacked. The backlight layer is located at the outermost layer and is a diffuse transmission film, which is used to convert external light into uniform transmission light. The pressure-sensitive layer is a translucent porous rubber film. The photosensitive layer is located at the innermost layer, which converts the light spot formed by the transmission light passing through the translucent porous rubber film into a first modal signal, and then transmits the first modal signal to the control circuit 13.

[0031] The working principle of the light sensing layer 121 collecting the first modal signal is as follows: when the user's finger moves, the finger part of the somatosensory glove bends, causing the light sensing layer 121 to deform, causing the porous rubber in the pressure-sensitive layer to deform as well. The gap deformation of the porous rubber causes the shape and brightness of the light spot on the photosensitive layer to change. Therefore, the first modal signal obtained by the photosensitive layer based on the light spot conversion can reflect the bending of the finger to a certain extent.

[0032] Wherein, the method for preparing the backlight layer comprises the following steps: S101, spin-coating PDMS-TiO2 on a glass substrate and curing to form a PDMS-TiO2 layer; inserting an optical fiber into the PDMS-TiO2 layer; laying a glass plate on the PDMS-TiO2 layer and laminating and shaping it; finally, removing the glass substrate and the glass plate to obtain a shaped PDMS-TiO2 layer.

[0033] S102, uniformly depositing silver nanowires on a PI (polyimide) film to obtain a silver nanowire-PI film, and then using a laser cutter to print a lattice structure on the silver nanowire-PI film to obtain a silver nanowire-PI layer.

[0034] S103, laying the silver nanowire-PI layer on the shaped PDMS-TiO2 layer, performing lamination and shaping, and obtaining a prepared backlight layer.

[0035] Wherein, the method for preparing the pressure-sensitive layer comprises the following steps: S201, mixing sugar particles and PDMS (polydimethylsiloxane) to obtain a sugar-PDMS mixture.

[0036] S202, sandwiching the sugar-PDMS mixture between two pieces of glass, and then heating and curing the mixture to obtain a sugar-PDMS layer.

[0037] S203, soaking the sugar-PDMS layer in water to dissolve the sugar particles, thereby obtaining a prepared translucent porous rubber membrane.

[0038] The method for preparing the photosensitive layer is as follows: S301, printing a flexible imaging array composed of an organic photodiode (OPD) and a thin-film transistor (TFT) based on low-temperature polycrystalline silicon (LTPS) on a glass substrate covered with a PI (polyimide) film; removing the glass substrate to obtain a flexible imaging array-PI film, that is, obtaining the prepared photosensitive layer.

[0039] The principle of the flexible imaging array converting the light spot into the first modal signal is as follows: each imaging unit of the flexible imaging array includes an OPD and a capacitor connected in parallel with the OPD. During power-on, the cathodes of all OPDs are connected to a common voltage, and the anodes and capacitors are periodically precharged to a reference voltage through a reset switch. When light is irradiated on the imaging unit, the OPD will generate a photocurrent, and the charge will accumulate on the capacitor to generate a photovoltage. When reading data, a row of imaging units of the flexible imaging array is connected to the analog front end through an analog switch, so as to read the accumulated photovoltage on the capacitor of a row of imaging units, thereby generating a first modal signal. After the data is read, the reset switch is triggered, and the capacitor is restored to the reference voltage to generate a new photovoltage according to the light at the next moment.

[0040] Specifically, the insulating layer 122 is a PDMS (polydimethylsiloxane) gel layer.

[0041] Specifically, the electric sensing layer 123 is: a silver nanowire-PVDF (polyvinylidene fluoride) electrospinning composite layer.

[0042] The working principle of the silver nanowire-PVDF electrospinning composite layer to collect the second modal signal is: each silver nanowire is regarded as a resistor with a very small resistance. When the electric sensing layer 123 is not deformed, each silver nanowire is closely connected to each other and overlaps irregularly. At this time, multiple silver nanowires can be regarded as multiple parallel resistors, and the overall resistance of the electric sensing layer 123 is small. When the user's finger moves, the finger part of the somatosensory glove bends, driving the electric sensing layer 123 to deform. At this time, the PVDF electrospinning membrane carrying the silver nanowires is stretched, and the fibers slip against each other, causing the silver nanowires to move with the slip of the fibers; at this time, the multiple silver nanowires that were originally irregularly overlapped are separated from each other due to displacement, which can be regarded as a reduction in parallel resistance and an increase in series resistance, which is manifested as an increase in the overall resistance of the electric sensing layer 123 on a macro scale. Therefore, the resistance of the electric sensing layer 123 can reflect the bending of the finger to a certain extent. The resistance change of the electric sensing layer 123 is converted into a voltage change through the circuit, that is, a second modal signal is obtained. Therefore, the second modal signal transmitted by the electric sensing layer 123 to the control circuit 13 can also reflect the bending condition of the finger to a certain extent.

[0043] The method for preparing the silver nanowire-PVDF electrospinning composite layer 123 comprises the following steps: S401, DMF (N,N-dimethylformamide) and acetone are evenly mixed in a mass ratio of 1:1, and 3 g of PVDF (polyvinylidene fluoride) is added to obtain a DMF-acetone-PVDF mixed solution.

[0044] S402, heating the DMF-acetone-PVDF mixed solution to 50°C and placing it in an ultrasonic oscillator for uniform stirring to obtain a PVDF electrospinning solution.

[0045] S403, spreading the aluminum foil onto the product collecting drum of the electrospinning machine, and coating a layer of release agent with a moderate thickness on the surface of the aluminum foil, loading the PVDF electrospinning solution into the syringe of the electrospinning machine, and starting the electrospinning machine to perform electrospinning for 5 hours to obtain a PVDF electrospinning membrane.

[0046] S404, 0.020 g of AgNWs is filtered onto the surface of the PVDF electrospinning membrane to obtain a silver nanowire layer, and then conductive silver glue is coated on both ends of the silver nanowire layer and wires are attached to obtain a preliminary silver nanowire-PVDF electrospinning composite layer.

[0047] S405, coating PDMS (polydimethylsiloxane) on both sides of the preliminary silver nanowire-PVDF electrospinning composite layer, and curing the PDMS to obtain a prepared silver nanowire-PVDF electrospinning composite layer.

[0048] See also Figure 4 , Figure 4 This is a module schematic diagram of the control circuit of Example 1 of the present invention.

[0049] Specifically, the control circuit 13 includes a power supply module 131, a signal conditioning module 132, a digital-to-analog conversion module 133 and an output module 134. The power supply module 131 is used to convert the external power supply voltage into a preset voltage to power other modules. The signal conditioning module 132 is used to obtain the original bimodal signal collected by the flexible photoelectric bimodal sensor 12, and perform signal conditioning processing such as amplification, filtering, and linearization on the original bimodal signal to obtain a conditioned bimodal signal. The digital-to-analog conversion module 133 is used to perform digital-to-analog conversion processing on the conditioned bimodal signal to obtain a digitized bimodal signal. The output module 134 is used to encode the digitized bimodal signal according to a preset protocol to obtain a preprocessed bimodal signal, and finally transmit the preprocessed bimodal signal to the computing device 2.

[0050] The output module 134 can be electrically connected to the computing device 2 through a wire (including a printed wire), or can be connected to the computing device signal through a wireless transmission method such as Wi-Fi, Bluetooth, etc. In this embodiment, the computing device 2 is an embedded single-chip microcomputer (model STM32H723VGT6), which is arranged on the same PCB board as the control circuit 13 and is electrically connected to the output module 134 through a printed wire. In other embodiments, the computing device 2 can be other smart devices with computing capabilities such as mobile phones and computers. Those skilled in the art can adaptively adjust the connection method between the output module 134 and the computing device 2 according to the type of the computing device.

[0051] Specifically, the computing device 2 analyzes the bimodal signal and reconstructs the user's hand motion data using a bimodal data fusion algorithm based on Kalman filtering, which inputs the first modal signal (i.e., the change in light spot of the photosensitive layer caused by finger bending) and the second modal signal (i.e., the change in resistance of the electric sensing layer caused by finger bending) continuously collected over a period of time into the Kalman filter. The Kalman filter performs hand state estimation and error correction based on the first modal signal and the second modal signal, outputs the corrected hand state, and outputs the final corrected value of the hand state as the hand motion data.

[0052] The hand state can be the position and / or velocity of a certain joint point of the hand. The hand state is used as the state vector , the equation of the Kalman filter is as follows:

[0053] In the formula, is the system state transfer matrix, is the system measurement matrix, is the system estimation error covariance matrix, is the Kalman gain, is the system measurement error covariance matrix, is the first modal signal of the input, is the second modal signal of the input, is the sampling time, is the system process noise covariance matrix, , is the error of the first modal signal, is the error of the second modal signal.

[0054] By repeatedly iterating the calculations of equations (1) to (5) of the five Kalman filters, the information provided by the first modal signal and the second modal signal can be integrated to obtain a hand state prediction value with the highest prediction accuracy.

[0055] Example 2 The difference between the somatosensory glove based on the flexible photoelectric dual-modal sensor of this embodiment and the embodiment 1 is that it also includes an IMU (Inertial Measurement Unit) inertial sensing module arranged on the glove body 11 and electrically connected to the control circuit 13. The IMU inertial sensing module includes an accelerometer, a gyroscope and a magnetometer, which are respectively used to collect acceleration signals, angular velocity signals and magnetic field direction signals generated by the user's hand movements, and transmit the collected acceleration signals, angular velocity signals and magnetic field direction signals to the control circuit 13. The acceleration signals, angular velocity signals and magnetic field direction signals collected by the IMU inertial sensing module and the dual-modal signals collected by the flexible photoelectric dual-modal sensor 12 together constitute a multi-modal signal. The control circuit 13 performs signal conditioning, analog-to-digital conversion and other pre-processing on each multi-modal signal respectively to obtain the pre-processed multi-modal signal, and then transmits the pre-processed multi-modal signal to the computing device 2. The computing device 2 analyzes the pre-processed multi-modal signal and reconstructs the user's hand movement data.

[0056] Specifically, the method by which the computing device 2 analyzes the multimodal signal and reconstructs the user's hand motion data is a multimodal data fusion algorithm based on machine learning, which includes the steps of: inputting the multimodal signal into a pre-trained machine learning model, and the machine learning model performing feature extraction, feature fusion and recognition on the multimodal signal, thereby reconstructing the user's hand motion data.

[0057] The machine learning model is trained using the following training method: ST1, let the user wear the somatosensory gloves of the present invention to perform preset actions, and record the standard hand movement data corresponding to the preset actions; then collect the multimodal signals corresponding to the preset actions through the flexible photoelectric dual-modal sensor and the IMU inertial sensing module.

[0058] ST2, input the collected multimodal signal into the machine learning model to be trained, so that the machine learning model reconstructs the hand motion data according to the input multimodal signal.

[0059] ST3, compare the hand motion data reconstructed by the machine learning model with the standard hand motion data recorded in step ST1, and calculate the loss function; then optimize the parameters of the machine learning model based on the loss function.

[0060] ST4, repeat steps ST2-ST3 to iteratively update the parameters of the machine learning model until the number of iterations reaches a preset threshold or the parameters of the machine learning model converge, then end the training and obtain a trained machine learning model.

[0061] In one embodiment, the machine learning model is a BP neural network.

[0062] Furthermore, the glove body 11 is made by 3D printing. The shape of the glove body can be customized according to the shape of the user's hand, thereby improving the wearing comfort of the somatosensory glove.

[0063] The present invention has the following technical effects: 1. The present invention collects the first modal signal and the second modal signal caused by hand movements from two angles respectively through a flexible photoelectric dual-modal sensor. Compared with a single-modal signal, a dual-modal signal can capture richer and more comprehensive action detail information, and can correct each other, so that the computing device can more accurately reconstruct the hand movement details through a dual-modal data fusion algorithm, thereby improving the accuracy and sensitivity of motion capture. 2. The flexible photoelectric dual-modal sensor of the present invention can deform with the hand movement and has a small size. Therefore, the somatosensory gloves of the present invention also have the advantages of small size, high integration and more comfortable wearing. 3. After the somatosensory gloves of the present invention are installed with an IMU inertial sensor module, more action detail information can be obtained, and combined with a multi-modal data fusion algorithm based on machine learning, the accuracy of motion modeling is further improved.

[0064] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A somatosensory glove based on a flexible photoelectric dual-modal sensor, characterized in that: include: Glove body; The flexible photoelectric dual-mode sensor is embedded in the glove body and comprises an electric sensing layer, an insulating layer and a light sensing layer which are stacked.

2. The somatosensory glove based on the flexible photoelectric dual-modal sensor according to claim 1, characterized in that: The light sensing layer comprises: A backlight layer, which is a diffuse transmission film, used to convert external light into uniform transmission light; A pressure-sensitive layer, which is a translucent porous rubber film; The photosensitive layer converts a light spot formed by the transmitted light passing through the pressure-sensitive layer into a first modal signal.

3. The somatosensory glove based on the flexible photoelectric dual-modal sensor according to claim 2, characterized in that: The electric sensing layer is a silver nanowire-PVDF electrostatic spinning composite layer, which generates a second modal signal.

4. The somatosensory glove based on the flexible photoelectric dual-modal sensor according to claim 3 is characterized in that: The insulating layer is a PDMS gel layer.

5. The somatosensory glove based on the flexible photoelectric dual-modal sensor according to claim 1, characterized in that: The first modal signal collected by the optical sensing layer and the second modal signal collected by the electrical sensing layer constitute a dual-modal signal; Also included is a control circuit comprising: A signal conditioning module, which acquires the original bimodal signal collected by the flexible photoelectric bimodal sensor, and performs signal conditioning on the original bimodal signal to obtain a conditioned bimodal signal; A digital-to-analog conversion module performs digital-to-analog conversion processing on the conditioned dual-modal signal to obtain a digitized dual-modal signal; The output module encodes the digitized bimodal signal according to a preset protocol to obtain a preprocessed bimodal signal, and finally transmits the preprocessed bimodal signal to a computing device.

6. The somatosensory glove based on the flexible photoelectric dual-modal sensor according to claim 5, characterized in that: It also includes an IMU inertial sensor module disposed on the glove body and electrically connected to the control circuit; The IMU inertial sensor module includes an accelerometer, a gyroscope and a magnetometer, which respectively collect acceleration signals, angular velocity signals and magnetic field direction signals generated by the user's hand movements, and transmit the collected acceleration signals, angular velocity signals and magnetic field direction signals to the control circuit; The acceleration signal, angular velocity signal and magnetic field direction signal collected by the IMU inertial sensing module and the dual-modal signal collected by the flexible photoelectric dual-modal sensor together form a multi-modal signal; The control circuit preprocesses each multimodal signal respectively to obtain a preprocessed multimodal signal, and then transmits the preprocessed multimodal signal to a computing device.

7. A hand motion capture system, characterized in that: include: A computing device and a somatosensory glove based on a flexible photoelectric dual-modal sensor as described in any one of claims 1 to 6; The somatosensory glove is signal-connected to the computing device for collecting bimodal signals or multimodal signals generated by the user's hand movements, and transmitting the bimodal signals or multimodal signals to the computing device; the computing device reconstructs the user's hand movement data based on the bimodal signals or multimodal signals.

8. The hand motion capture system according to claim 7, characterized in that: The computing device reconstructs hand motion data based on the dual-modal signal, specifically including the following steps: inputting the first modal signal and the second modal signal continuously collected over a period of time into the Kalman filter, the Kalman filter performs hand state estimation and error correction based on the first modal signal and the second modal signal, outputs the corrected hand state, and outputs the final corrected value of the hand state as the hand motion data.

9. The hand motion capture system according to claim 7, characterized in that: The computing device reconstructs hand motion data based on the multimodal signal, specifically including the following steps: inputting the multimodal signal into a pre-trained machine learning model, and the machine learning model extracting, fusing and identifying features of the multimodal signal to reconstruct the user's hand motion data.

10. The hand motion capture system according to claim 9, characterized in that: The machine learning model is trained by the following training method: ST1, allowing the user to wear the somatosensory glove to perform a preset action, and record the standard hand motion data corresponding to the preset action; then collecting the multimodal signal corresponding to the preset action through the flexible photoelectric dual-modal sensor and the IMU inertial sensing module; ST2, inputting the collected multimodal signals into the machine learning model to be trained, so that the machine learning model reconstructs the hand motion data according to the input multimodal signals; ST3, comparing the hand motion data reconstructed by the machine learning model with the standard hand motion data recorded in step ST1, and calculating a loss function; and then optimizing the parameters of the machine learning model according to the loss function; ST4, repeat steps ST2-ST3 to iteratively update the parameters of the machine learning model until the number of iterations reaches a preset threshold or the parameters of the machine learning model converge, then end the training and obtain a trained machine learning model.

Images