Intelligent glasses interaction assembly based on gesture recognition

Abstract

The utility model relates to an intelligent glasses interaction assembly based on gesture recognition, which comprises a hand wearing device detachably arranged on a hand and an intelligent wearing device detachably arranged on a head, the hand wearing device is provided with a flexible sensor and an inertial sensor, the flexible sensor is used for detecting the bending degree of fingers, and the inertial sensor is used for detecting the bending degree of the fingers. The inertial sensor is used for detecting the displacement change of the hand wearing device, and the hand wearing device can transmit data of the flexible sensor and the inertial sensor to the intelligent wearing device for processing in a wireless or wired mode; according to the utility model, the inertial sensor and the flexible sensor are combined, so that the overall movement of the hand and the fine finger bending action can be captured at the same time, and a richer and more accurate data basis is provided for gesture recognition. Compared with a single sensor, the multi-modal data fusion mode has the advantage that the accuracy of gesture recognition is remarkably improved.

Description

TECHNICAL FIELD

[0001] The utility model relates to intelligent glove technical field, concretely relates to a kind of intelligent eyeglasses interactive components based on gesture recognition. BACKGROUND

[0002] Intelligent eyeglasses are a kind of head-mounted device combining augmented reality technology and wearable computing, which realizes information superimposed display and environment interaction through miniature display screen, camera and sensor. Its core functions include real-time navigation, information prompt, image recording and virtual interface operation, and it has wide application in industrial maintenance, medical assistance and consumer entertainment. Gesture control technology recognizes user's hand action through camera or electromyographic sensor, replacing traditional touch or voice input. Since there is still no good solution for the convenience control of intelligent eyeglasses at present, gesture control technology is expected to become a key way for intelligent eyeglasses control interaction in the future.

[0003] Its importance mainly reflects in three aspects: first, gesture operation conforms to human natural interaction habit, can realize non-contact quick response, and avoid the limitation of voice control in noisy environment; second, in complex task, both hands can keep working state, and drawing or tool menu can be called or switched through simple gesture, which significantly improves work efficiency; finally, in privacy protection scene, concealed gesture instruction is safer than voice. The current technical challenge is to improve the accuracy of gesture recognition and reduce the probability of false triggering, and in the future, eye movement tracking and tactile feedback will further improve the interaction experience. SUMMARY

[0004] The utility model mainly aims at the problem existing in the interaction between intelligent eyeglasses and gesture, and invents a kind of intelligent eyeglasses interactive components based on gesture recognition, which can capture the overall movement of hand and fine finger bending action by combining inertial sensor and flexible sensor, and provides more abundant and accurate data basis for gesture recognition.

[0005] The utility model discloses a kind of intelligent eyeglasses interactive components based on gesture recognition, including detachable setting on hand hand wearing device, detachable setting in head intelligent eyeglasses, flexible sensor and inertial sensor are equipped on the hand wearing device, flexible sensor is used to detect the bending degree of finger, inertial sensor is used to detect the displacement change of the hand wearing device, the hand wearing device can transmit the data of flexible sensor and inertial sensor to the intelligent eyeglasses for processing by wireless or wired mode.

[0006] As preferred, the hand wearing device includes a glove that can be worn on the hand and a wrist locking ring disposed on one side of the glove, and the wrist locking ring and the glove body are connected by a plurality of elastic bandages.

[0007] As preferred, the surface of the wrist locking ring is provided with a display screen for displaying information and several buttons for positioning the initial position, which is to realize the reset function of the hand wearing device.

[0008] As preferred, the flexible sensor is a resistance strain sensor, which is arranged at each finger bending position of the lower surface of the glove, which is to accurately collect the change of the user's fingers.

[0009] As preferred, the upper surface of the glove is provided with several support housings, and each support housing is internally provided with an inertial sensor, and the inertial sensors are connected through data transmission lines.

[0010] As preferred, the material of the glove is nano-silver fiber or metal blended yarn, which is to enhance the mechanical strength by mixing nylon / cotton fiber, significantly improve the wear resistance compared with pure cotton gloves, and prolong the service life. The activity of silver-plated nylon fiber is enhanced in high temperature and high humidity environment, and the antibacterial performance is further improved.

[0011] As preferred, the smart glasses include optical lenses, a frame and a camera, the side wall of the frame is provided with several physical buttons for controlling whether the camera identifies, the inside of the frame is integrated with a battery, and the side wall of the frame is provided with an interface for charging.

[0012] As preferred, the optical lenses are electrochromic lenses, and the frame is further provided with a processor chip and an audio playing and receiving element. The electrochromic lenses can not only automatically adjust the transmittance of the lenses according to the ambient light to provide comfortable visual experience, but also can display information on the lenses through specific technology.

[0013] Compared with the prior art, the utility model has the following beneficial effects:

[0014] 1. By combining the inertial sensor and the flexible sensor, the overall movement of the hand and the fine finger bending action can be captured at the same time, which provides more abundant and accurate data basis for gesture recognition. Compared with a single sensor, this multi-modal data fusion method significantly improves the accuracy of gesture recognition.

[0015] 2. The data glove adopts lightweight material and flexible circuit design, which reduces the weight of the device and improves the wearing comfort. The smart glasses are convenient to carry, and have a display screen and tactile feedback function, which provides good interactive experience for users.

[0016] 3、Compared with the smart glasses control mode by means of mobile phone instruction control or lifting hand to operate on the smart glasses, the gesture control is more convenient and fast, greatly expands the application scene of the smart glasses, and meanwhile, through gesture recognition, the deaf-mute or normal person can realize fast photographing, video recording, audio recording and various operations in the occasion inconvenient to talk, the scheme provides a new barrier-free communication mode, and improves the life portability. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 It is a perspective view of the hand wearing device of the utility model;

[0018] Figure 2 It is a perspective view of the hand wearing device of the utility model;

[0019] Figure 3 It is a perspective view of the hand wearing device of the utility model;

[0020] Figure 4 It is a schematic view of the inertial sensor installation position of the utility model;

[0021] Figure 5 It is a perspective view of the smart glasses of the utility model;

[0022] Figure 6 It is a perspective view of the smart glasses of the utility model;

[0023] Figure 7 It is a perspective view of the smart glasses and the hand wearing device of the utility model.

[0024] Marked in the figure: 1, hand wearing device; 11, glove; 12, wrist locking ring; 13, elastic bandage; 14, display screen; 15, button; 16, support shell; 17, data transmission line; 2, flexible sensor; 3, inertial sensor; 4, smart glasses; 41, optical lens; 42, frame; 43, camera. DETAILED DESCRIPTION

[0025] The utility model will be further described in connection with the embodiment shown in the accompanying drawings:

[0026] As Figure 1 And Figure 2 Shown, a kind of smart glasses interactive assembly based on gesture recognition, including detachably set on hand hand wearing device 1, the material of glove 11 is nano silver fiber or metal blended yarn.The hand wearing device 1 including the glove 11 that can be worn in hand and the wrist locking ring 12 being arranged in one side of glove 11, the wrist locking ring 12 is connected between glove 11 body by several elastic bandages 13.

[0027] In this embodiment, the surface of the wrist locking ring 12 is provided with a display screen 14 for displaying information and several buttons 15 for positioning the initial position. The flexible sensor 2 is a resistance strain sensor, which is arranged at each finger bending position on the lower surface of the glove 11. The hand wearing device 1 can transmit the data of the flexible sensor 2 and the inertial sensor 3 to the smart glasses 4 for processing through wireless or wired mode.

[0028] The manufacturing process of the flexible sensor 2 mainly includes the following steps: base layer and packaging layer material: soft silicone material with good ductility and durability (such as Ecoflex-30) is selected. Electrode material: carbon black (CB) and carbon nanotubes (CNTs) are selected and mixed in a ratio of 1:5. Electrode preparation: CB (0.25g) and CNTs (0.05g) are mixed and then poured into an appropriate amount of high-purity alcohol. A magnetic stirrer is used to stir for 20 minutes to form a uniform CB / CNTs suspension. The suspension is coated on a polyimide film (PI film), and the alcohol is evaporated by heating to 50°C to form a dry CB / CNTs composite electrode.

[0029] Sensor assembly: Mix silicone A and B in a ratio of 1:1 and evenly coat on the surface of the CB / CNTs composite electrode. After the silicone is cured, the electrode layer and the silicone layer are pasted together. The silicone is cut to the required size, placed in a mold, and the two ends are connected to the electrode leads. Pour in the silicone for packaging, and stand at room temperature for 4 hours to obtain a flexible resistance sensor.

[0030] When the button 15 is pressed to reset, the flexible sensor 2 continuously judges whether the user stretches out the palm. When the palm is stretched out, it means that the reset is completed, and then the user makes a specified gesture. The flexible sensor 2 can capture the change information of the user's fingers.

[0031] The principle of the flexible sensor 2 for detecting gestures: The flexible sensor 2 recognizes gestures by detecting the change of resistance when the finger joints are bent: when the finger joints below the glove 11 are bent, the conductive network inside the flexible sensor 2 changes due to deformation, causing the resistance value to change. There is a linear relationship between this resistance change and the bending angle of the finger joint, and a specific mathematical model (such as a first-order function fitting) can be established through calibration experiments. By monitoring the change of resistance value in real time, the bending angle of the finger joint can be calculated, and the corresponding gesture can be recognized.

[0032] Signal acquisition and processing of flexible sensor 2: In this scheme, multiple flexible sensors 2 are arranged at the knuckles of the glove, and the resistance data of each sensor is collected in real time by a signal acquisition system (such as an STM32F103 microcontroller). Convert resistance data to bending angle, calculate using mathematical model established in calibration experiment. Encode the bending angle of each finger joint to form a sequence of feature information of the gesture.

[0033] The processing chip inside the wrist locking ring 12 can calculate the gesture information obtained by the flexible sensor 2: template matching method: calculate the similarity (such as Euclidean distance) between the gesture to be recognized and the pre-stored template gesture, and determine the gesture category by comparing the similarity. In this scheme, mainly using BP neural network algorithm: through training model to learn the mapping relationship between resistance data and gesture category, improve the recognition accuracy of similar gestures. BP neural network takes resistance data as input layer and gesture category as output layer, and performs feature extraction and classification through multiple neurons.

[0034] The processing chip inside the wrist locking ring 12 encodes the recognized gesture into a format (such as ASCII code, digital code or specific instructions) that can be recognized by devices inside the smart glasses 4. Through wireless or wired communication (such as Bluetooth, Wi-Fi, USB, etc.), the encoded gesture information is transmitted to the smart glasses 4 for processing. After the smart glasses 4 receive the gesture information, they process it according to the pre-set application logic, such as executing functions such as taking pictures, recording videos or sounds.

[0035] Please continue to refer to Figure 4 , the hand-wearing device 1 is provided with flexible sensors 2 and inertial sensors 3, the flexible sensors 2 are used for detecting the bending degree of fingers, and the inertial sensors 3 are used for detecting the displacement change of the hand-wearing device 1. The upper surface of the glove 11 is provided with a plurality of support housings 16, the inside of each support housing 16 is provided with an inertial sensor 3, a plurality of inertial sensors 3 are connected through data transmission lines 17, and the inertial sensor 3 is a nine-axis inertial sensor.

[0036] In this embodiment, it is realized based on a nine-axis inertial sensor, and the model of the inertial sensor 3 selected in this embodiment is MPU9250. It integrates a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer, and is very suitable for gesture recognition and other applications.

[0037] The working principle of the nine-axis inertial sensor for detecting gestures: accelerometer: measures the acceleration change of the hand in three-dimensional space, used to detect the motion direction and speed of the hand. Gyroscope: measures the angular velocity change of the hand around three axes, used to determine the rotation direction and angle of the hand. Magnetometer: measures the direction and strength of the geomagnetic field, used to correct the hand posture and ensure the accuracy of the attitude solution.

[0038] In the detection process, the nine-axis inertial sensor can calculate the spatial pose of the hand (such as Euler angles or quaternions) in real time by fusing the data of the accelerometer, gyroscope, and magnetometer. Combined with the data of multiple inertial sensors 3 on the fingers and palm, the motion trajectory and pose change of each joint of the hand can be accurately captured, thereby identifying different gesture actions.

[0039] The calculation and recognition data processing of the processing chip inside the wrist locking ring 12: First, calibrate the data of the accelerometer, gyroscope, and magnetometer to eliminate the bias of the sensor itself and the influence of the external environment. Then, solve the pose, fuse the multi-sensor data using algorithms such as Kalman filtering and gradient descent to calculate the real-time pose of the hand. Finally, extract features from the pose data to extract features that can represent gesture actions, such as joint angles, angular velocities, accelerations, etc.

[0040] As for the recognition algorithm, machine learning algorithms can be selected: such as support vector machine (SVM), random forest (RF), etc., for static gesture recognition. Deep learning algorithms: such as long short-term memory neural network (LSTM), bidirectional long short-term memory neural network (Bi-LSTM) combined with attention mechanism (Attention-BiLSTM), etc., for dynamic gesture recognition: convert the calculated hand pose and feature data into a format that the internal chip of the smart glasses 4 can recognize, such as sending the data to the internal chip of the smart glasses 4 or embedded processing device through a wireless transmission module (such as WiFi, Bluetooth).

[0041] In this embodiment, the information between the flexible sensor 2 and the inertial sensor 3 needs to be combined: the flexible sensor is arranged at the finger joints of the glove to detect the bending degree of the fingers; at the same time, the inertial sensor 3 is arranged on the palm to detect the overall motion and pose change of the hand.

[0042] A fusion recognition algorithm can be used: fuse the data of the flexible sensor and the inertial sensor 3, and use a deep learning model to improve the accuracy and robustness of gesture recognition. The bending angle data of the flexible sensor and the pose data of the inertial sensor 3 are combined as input features of the deep learning model for accurate gesture recognition.

[0043] Please continue to refer to Figure 5 and Figure 6As shown, in the present embodiment, it also includes a smart glasses 4 detachably arranged on the head, the smart glasses 4 includes optical lenses 41, a frame 42 and a camera 43, the sidewall of the frame 42 is provided with a plurality of physical buttons for controlling whether the camera 43 identifies or not, the inside of the frame 42 is integrated with a battery and the sidewall of the frame 42 is provided with an interface for charging.

[0044] The inertial sensor 3 and the flexible sensor 2 are combined to perform gesture recognition, the inertial sensor 3 is arranged on the glove for detecting the overall motion (such as rotation, translation) and posture change of the hand. The flexible sensor 2 is arranged at the knuckles of the glove for accurately detecting the bending degree of the fingers.

[0045] Firstly, ensure that the data of the inertial sensor 3 and the flexible sensor 2 are synchronized in time, so as to subsequent data processing and analysis. Then extract the features such as posture, angular velocity and acceleration of the hand from the data of the inertial sensor 3; extract the features such as bending angle of the fingers from the data of the flexible sensor 2. Then fuse the data of the two sensors, and use machine learning or deep learning algorithm to recognize the gesture. In this scheme, the data of the two sensors need to be used as input features of the deep learning model, and the model is trained to learn the feature representation of different gestures.

[0046] The flexible sensor 2 and the inertial sensor 3 transmit information to the processing chip in the frame 42: wireless transmission can be selected: the data glove transmits the recognized gesture information to the smart glasses 4 or the mobile phone through Bluetooth, WiFi and other wireless communication technologies. It should be noted that: the gesture information should be encoded into a data format that the smart glasses 4 can understand, such as JSON, XML, etc.

[0047] After the processing chip of the frame 42 of the smart glasses 4 receives the data, it first parses the gesture information, and according to the preset gesture and operation mapping relationship, maps the gesture information to specific operation instructions, such as taking pictures, recording videos, recording sounds, etc. Taking pictures: when a specific gesture is recognized, the smart glasses 4 automatically start the camera 43 to take pictures. Recording video: when the video recording gesture is recognized, the camera 43 of the smart glasses 4 starts to record video. Recording sound: when the sound recording gesture is recognized, the audio playback element of the smart glasses 4 records sound.

[0048] Visual feedback: the smart glasses 4 displays the operation result through the optical lenses 41 or the display screen 14 of the wrist locking ring 12, such as photo preview after taking pictures, video and sound recording time, etc.

[0049] The working principle and use method of the utility model:

[0050] Sensor installation position, flexible sensor 2: CB / CNTs composite electrode (1:5 ratio) is installed on the palm side of each knuckle of the glove to detect the bending angle of the finger. Inertial sensor 3: MPU9250 module is deployed at the back of the hand, 3-axis acceleration + 3-axis gyroscope + 3-axis magnetic force.

[0051] Long press the button 15 can be set to close the data monitoring of the flexible sensor 2 and the inertial sensor 3, and when the button 15 is pressed once, the flexible sensor 2 continuously judges whether the user stretches the palm straight, and the stretching means that the reset is completed, and then the user makes a specified gesture, and the flexible sensor 2 and the inertial sensor 3 can capture the change information of the user's fingers. After the processing chip of the frame 42 of the smart glasses 4 receives the data, the gesture information is first analyzed.

[0052] According to the preset gesture and operation mapping relationship, the gesture information is mapped to specific operation instructions, such as photographing, video recording, and audio recording. Photographing: when a specific gesture is recognized, the smart glasses 4 automatically start the camera 43 to take a picture for the gesture information transmission to the smart glasses 4. Video recording: when the video recording gesture is recognized, the camera 43 of the smart glasses 4 starts to record the video. Audio recording: when the audio recording gesture is recognized, the audio receiving and playing element of the smart glasses 4 performs audio recording. For the gesture information transmission to the mobile phone, the gesture of double-clicking the little finger makes the mobile phone ring, the gesture of triple-clicking the little finger makes the mobile phone open the flash, different gestures open the mobile phone music or camera in the air, and the mobile phone camera can also be controlled by gestures to take a quick selfie.

[0053] The specific embodiments described in the present application are only illustrative of the spirit of the present application. Those skilled in the art to which the present application belongs can make various modifications or supplements to the described specific embodiments or use similar ways to replace, but will not deviate from the spirit of the present application or exceed the scope defined by the appended claims.

Claims

1. A gesture recognition based smart glasses interaction assembly comprising a hand worn device (1) detachably arranged on a hand, a smart worn device detachably arranged on a head, characterized in that, The hand wearing device (1) is provided with a flexible sensor (2) and an inertial sensor (3), the flexible sensor (2) is used for detecting the bending degree of fingers, the inertial sensor (3) is used for detecting the displacement change of the hand wearing device (1), and the hand wearing device (1) can transmit the data of the flexible sensor (2) and the inertial sensor (3) to the smart wearable device for processing in a wireless or wired mode.

2. The smart glasses interaction component based on gesture recognition as claimed in claim 1, wherein, The hand wearing device (1) includes a glove (11) capable of being worn on the hand and a wrist locking ring (12) arranged on one side of the glove (11), and the wrist locking ring (12) and the glove (11) body are connected through a plurality of elastic bandages (13).

3. The smart glasses interaction component based on gesture recognition as claimed in claim 2, wherein, The surface of the wrist locking ring (12) is provided with a display screen (14) for displaying information and a plurality of buttons (15) for positioning the initial position.

4. The smart glasses interaction component based on gesture recognition as claimed in claim 2, wherein, The flexible sensor (2) is a resistance strain sensor, and the flexible sensor (2) is arranged at each finger bending position on the lower surface of the glove (11).

5. The smart glasses interaction component based on gesture recognition as claimed in claim 4, wherein, The upper surface of the glove (11) is provided with a plurality of support housings (16), each of the support housings (16) is provided with an inertial sensor (3), and the plurality of inertial sensors (3) are connected through data transmission lines (17), and the inertial sensor (3) is a nine-axis inertial sensor (3).

6. The smart glasses interaction component based on gesture recognition as claimed in claim 5, wherein, The material of the glove (11) is nano-silver fiber or metal blended yarn.

7. The smart glasses interaction component based on gesture recognition as claimed in claim 1, wherein, The smart wearable device is smart glasses (4), and the smart wearable device includes an optical lens (41), a frame (42) and a camera (43), the sidewall of the frame (42) is provided with a plurality of physical buttons for controlling whether the camera (43) identifies, the inside of the frame (42) is integrated with a battery, and the sidewall of the frame (42) is provided with an interface for charging.

8. The smart glasses interaction component based on gesture recognition as claimed in claim 7, wherein, The optical lens (41) is an electrochromic lens, and the frame (42) is further provided with a processor chip and an audio playing and receiving element.

Images