Gesture recognition device and method
By using light source modules, plastic fiber sensor area and neural network model in gesture recognition device, combined with conical double C-shaped arrangement, the accuracy and application limitation of gesture recognition in the prior art are solved, and high integration and High accuracy recognition effect.
Patent Information
- Application Number
- CN202411243879.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-27
AI Technical Summary
The existing gesture recognition technology has problems such as marking point occlusion, complex algorithms, expensive detection equipment, accumulated measurement errors and electromagnetic interference, resulting in limited recognition accuracy and application.
Using a device including a light source module, a plastic fiber sensor area, a collection module and an identification module, a pre-trained neural network model is used to identify gestures through the light intensity changes generated by the bending of plastic fibers, and combined with a cone double C-shaped arrangement to improve sensitivity and accuracy. Spend.
A high-integration and miniaturized gesture recognition device is realized, which improves recognition accuracy and stability, reduces costs, is suitable for multiple application fields, and achieves high gesture recognition accuracy.
Smart Images
Figure CN119200837A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human gesture recognition and monitoring, and more specifically, to a gesture recognition device and method. Background Art
[0002] In recent years, with the development of virtual reality interaction technology, tracking of complex human movements has received more and more attention. Hands are one of the most important organs for the human body to communicate with the outside world and can transmit a large amount of information. As one of the main devices of human-computer interaction technology, data gloves are widely used in education, medical care, games, movies, engineering, military and other fields. Therefore, gesture recognition based on data gloves has become a research hotspot in recent years.
[0003] At present, data gloves are mainly based on inertial sensors, computer vision technology, point tracking and recognition technology, and fiber Bragg grating (FBG) sensor technology. However, in point tracking and recognition technology, the marker points are easily obscured during the movement of the hand. Data gloves based on computer vision technology are not feasible under different lighting and hand overlap due to their complex algorithms. FBG sensors have high sensitivity and accuracy, but the measurement system is relatively complex and the corresponding detection equipment is expensive, which limits their application in data gloves. Data gloves based on inertial sensors have good real-time gesture measurement performance, but due to the cumulative measurement errors of inertial sensors, they are not suitable for long-term work, and gesture recognition systems based on electronic sensors are easily affected by electromagnetic interference. Summary of the invention
[0004] Based on the problems existing in the prior art, the present invention provides a gesture recognition device and method.
[0005] According to an embodiment of the present invention, a gesture recognition device is disclosed. The gesture recognition device can be worn on a hand and may include: a light source module configured to generate light of predetermined intensity; a sensing module including a plastic optical fiber sensing area, and the plastic optical fiber sensing area is configured to generate light intensity changes based on finger movements; a collection module configured to collect data of light intensity changes; and a recognition module that uses a pre-trained neural network model to identify gestures based on the data.
[0006] Further, the plastic optical fiber sensing area may be arranged in a tapered double C-shape.
[0007] Further, the light source module may include an LED light source.
[0008] Furthermore, the plastic optical fiber sensing area can be arranged at the finger joints.
[0009] Furthermore, the plastic optical fiber sensing area can be set to be multiple.
[0010] According to an embodiment of the present invention, a gesture recognition method is disclosed, which may include: collecting data on light intensity changes; and using a pre-trained neural network model to identify gestures based on the data, wherein the data on light intensity changes are data on light intensity changes generated by light of a predetermined intensity based on finger movement.
[0011] Further, the light intensity variation generated based on the movement of the finger may be generated by the plastic optical fiber sensing area arranged in a tapered double C shape.
[0012] Further, the light of predetermined intensity may be generated by an LED light source.
[0013] Furthermore, the plastic optical fiber sensing area can be arranged at the finger joints.
[0014] Furthermore, the plastic optical fiber sensing area can be set to be multiple.
[0015] In the present invention, since plastic optical fiber (POF) has the advantages of good flexibility, strong plasticity, simple operation, easy bending, stretching, polishing, strong light guiding ability, large numerical aperture value, easy connection, high coupling efficiency, low price, etc., and its transmission loss in the visible light band is low, therefore, plastic optical fiber can be used for wearable gesture recognition devices, and cheap LEDs can be used as light source modules and photoresistors can be used as part of acquisition modules for signal detection.
[0016] Furthermore, by arranging the plastic optical fiber sensing area into a tapered double C shape, when the sensing area bends (for example, when a finger moves), it causes twice the transmission loss of the light intensity, thus greatly improving the sensitivity and thus the recognition accuracy.
[0017] In addition, by integrating the light source module (e.g., LED), the sensing module (e.g., plastic optical fiber sensor probe), the acquisition module (e.g., photoresistor, detector, etc.), and the recognition module (e.g., single-chip microcomputer, etc.), the gesture recognition device can be worn on the hand, realizing the advantages of miniaturization and high integration of the gesture recognition device. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 is a structural block diagram showing a gesture recognition device according to an embodiment of the present invention; Figure 2 is a structural diagram showing a gesture recognition device according to an embodiment of the present invention; Figure 3 is a schematic diagram showing a connection structure of a gesture recognition device according to an embodiment of the present invention; Figure 4is an optical microscope image showing a plastic optical fiber before and after tapering according to an embodiment of the present invention; Figure 5 is a graph showing 12 gestures and corresponding light intensity loss curves according to an embodiment of the present invention; Figure 6 is a flow chart showing a gesture recognition method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0019] Examples or embodiments of the present invention will be described below with reference to the accompanying drawings, in which specific examples or embodiments that can be implemented are shown by way of illustration, and in the accompanying drawings, even if the same reference numerals and symbols are shown in different drawings from each other, they can also be used to represent the same or similar parts. Further, in the following description of examples or embodiments of the present invention, when it is determined that the detailed description of well-known functions and components incorporated herein may obscure the subject matter in some embodiments of the present invention, these detailed descriptions will be omitted. Terms such as "including", "having", "comprising", "constituting", "composing", "forming", etc. used herein are generally intended to allow the addition of other components unless used with the term "only". As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise.
[0020] Terms such as "first", "second", "A", "B", "(A)" or "(B)" may be used herein to describe elements of the present invention. Each of these terms is not used to define an element, sequence, order, quantity, etc., but is only used to distinguish the corresponding element from other elements.
[0021] When it is mentioned that a first element is “connected or coupled”, “contacted or overlapped” with a second element, it should be understood that not only the first element can be “directly connected or coupled” or “directly contacted or overlapped” with the second element, but also a third element can be “inserted” between the first element and the second element, or the first element and the second element can be “connected or coupled”, “contacted or overlapped” with each other through a fourth element, etc. Here, the second element can be included in at least one of the two or more elements that are “connected or coupled”, “contacted or overlapped”, etc. with each other.
[0022] When time relative terms such as "after", "after...", "next", "before" and the like are used to describe a process or operation of an element or configuration, or a flow or step in an operation, process, or manufacturing method, unless these terms are used with the terms "directly" or "immediately", these terms may be used to describe non-continuous or non-sequential processes or operations.
[0023] In addition, when any dimension, relative dimension, etc. is mentioned, even if no relevant description is specified, it should be considered that the numerical value or corresponding information (e.g., level, range, etc.) of the element or feature includes a tolerance or error range that may be caused by various factors (e.g., process factors, internal or external influences, noise, etc.). Further, the term "may" fully encompasses all meanings of the term "can".
[0024] Figure 1 is a structural block diagram of a gesture recognition device 100 according to an embodiment of the present invention.
[0025] Reference Figure 1 The gesture recognition device 100 may include a light source module 110 , a sensing module 120 , a collection module 130 and a recognition module 140 .
[0026] In an embodiment, the light source module 110 may be configured to generate light of a predetermined intensity, and the generated light may be transmitted along a plastic optical fiber.
[0027] As mentioned above, since plastic optical fiber has low transmission loss in the visible light range, an LED capable of emitting visible light can be used as the light source module 110. For example, a 650nm laser diode can be used as the light source module 110. In addition, the present invention can also use other light sources, and there is no particular limitation on this.
[0028] In an embodiment, the sensing module 120 may include a plastic optical fiber sensing region. In an embodiment, a commercial plastic optical fiber with a diameter of 0.25 mm may be used, but the present invention is not limited thereto, and plastic optical fibers of other sizes may be used.
[0029] The plastic optical fiber sensing area can be configured to produce changes in light intensity based on the movement of the finger.
[0030] For example, when the gesture recognition device is worn properly, when the finger is not moving, the light transmitted along the plastic optical fiber has little loss, that is, the light intensity remains basically unchanged; when the finger moves, the plastic optical fiber sensing area bends, causing the light transmitted along the plastic optical fiber to have loss, that is, the light intensity changes, and when the degree of finger bending is different, the loss generated will also be different. Therefore, the present invention can recognize gestures by utilizing the principle of light loss caused by bending of plastic optical fiber.
[0031] In an embodiment, the acquisition module 130 may be configured to acquire data of light intensity variation.
[0032] For example, the acquisition module 130 may include a photoresistor, because a photoresistor (photoresistor) is a component that can change its resistance value according to the light intensity, and its electronic structure changes under light based on its own semiconductor material, thereby affecting the change of resistance value. In other words, when the light intensity changes (for example, loss) due to finger movement, the light intensity change can be obtained through the resistance value.
[0033] In an embodiment, the recognition module 140 may be configured to recognize gestures based on the collected data using a pre-trained neural network model.
[0034] In an embodiment, in the calibration stage of gesture recognition, a convolutional recurrent neural network model combining a convolutional neural network (CNN) and a long short-term memory artificial neural network (LSTM) is used, and the Pytorch library is called in Python to write the model for deep training of different gestures.
[0035] In order to make gesture recognition more specific, in the embodiment, 12 gesture categories are defined, namely "Finger0", "Finger1", "Finger2", "Finger3", "Finger4", "Finger5", "Hit", "OK", "Good", "Victory", "Rock" and "Cool".
[0036] In order to ensure the authenticity and accuracy of the data, five volunteers with basically the same hand size were found, and the 12 gestures were tested 10 times each. During the test, the total test time, bending degree and bending time of each test were kept basically the same.
[0037] The obtained light intensity loss curve is divided into a training set and a test set according to a certain ratio (for example, 9:1), and the two sets of data sets are filtered and normalized, and then imported into the convolutional recurrent neural network. The processed data is used as the input layer, processed by the convolution layer and the LSTM layer, and finally mapped to the predefined 12 gesture actions through the fully connected layer.
[0038] Finally, we get the recognition results of 12 gestures (see e.g. Figure 5 ), among which the lowest gesture recognition accuracy is 96.2%, corresponding to gesture "Finger3", and the average gesture recognition accuracy reaches 98.85%.
[0039] Based on the above description, it can be known that the recognition module 140 can use the above-mentioned convolutional recurrent neural network to obtain various gesture recognition results through the light intensity change related data collected by the collection module 130.
[0040] From the above, it can be seen that in the present invention, since plastic optical fiber has the advantages of good flexibility, strong plasticity, simple operation, easy bending, stretching, polishing, strong light guiding ability, large numerical aperture value, easy connection, high coupling efficiency, low price, etc., and its transmission loss in the visible light band is low, therefore, plastic optical fiber can be used for wearable gesture recognition devices, and cheap LEDs can be used as light source modules and photoresistors can be used as part of acquisition modules for signal detection.
[0041] Furthermore, by arranging the plastic optical fiber sensing area into a tapered double C shape, when the sensing area bends (for example, when a finger moves), it causes twice the transmission loss of the light intensity, thus greatly improving the sensitivity and thus the recognition accuracy.
[0042] The following combination Figure 2 The structure of the gesture recognition device according to the embodiment of the present invention is described in detail.
[0043] Reference Figure 2 The gesture recognition device 100 of the present invention comprises a fiber optic sensing glove ( Figure 2 The part covering the finger) and the integrated circuit board ( Figure 2 The wrist part in the figure). Figure 1 The sensing module 120 described above can be implemented on the optical fiber sensing glove. Figure 1 The light source module 110, the acquisition module 130, and the identification module 140 described above can be implemented on an integrated circuit board, and the optical fiber sensing glove and the integrated circuit board can be worn on the hand. In addition, for greater clarity, a signal receiving end ( Figure 2 laptop computer).
[0044] In an embodiment, the sensitivity of light intensity changes can be changed by changing the arrangement shape, arrangement position, arrangement number, etc. of the fiber optic sensing areas on the fiber optic sensing glove, thereby changing the accuracy of gesture recognition.
[0045] For example, in Figure 2 In order to improve the sensitivity of light transmission loss and thus improve the accuracy of gesture recognition, the plastic optical fiber can be tapered into a cone, the cone can be bent into a C shape, and the sensing area can be designed as two tapered C-shaped plastic optical fibers back-to-back structure (see, for example Figure 2 In this case, compared with the normal shape of the plastic optical fiber (not processed), when the finger movement causes the sensing area to bend, twice the transmission loss can be generated, that is, the sensitivity of the transmission loss is improved, thereby improving the gesture recognition accuracy. In addition, the present invention can also arrange the plastic optical fiber into other shapes, and there is no particular limitation on this.
[0046] exist Figure 2In order to improve the sensitivity of light transmission loss and thus improve the accuracy of gesture recognition, the plastic optical fiber sensing areas can be arranged at the joints of the fingers, and the number of sensing areas on the thumb is set to 2, and the number of sensing areas on the other fingers is set to 3. In addition, the present invention can also arrange the plastic optical fiber areas in other positions and numbers, and there is no particular limitation on this.
[0047] In an embodiment, in order to achieve miniaturization of the gesture recognition device and stable collection of gesture signals, the light source module 110 , the collection module 130 , the recognition module 140 , etc. may be integrated together.
[0048] For example, in Figure 2 In the case of a 650 nm laser diode (e.g. Figure 1 light source module 110), a photoresistor (eg, Figure 1 The acquisition module 130) and the single chip microcomputer (also called a microcontroller, for example, Figure 1 The identification module 140) can be integrated on a five-channel voltage signal acquisition circuit board (8cm×8cm).
[0049] The whole circuit board can be powered by a lithium battery. When the finger bends to different degrees, the tapered double C-shaped plastic optical fiber sensing area bends accordingly, causing the light intensity of the optical fiber sensor to change, and data is collected through a signal acquisition card (not shown). A 3D model hand is constructed using 3D modeling software on a computer, and the bending angle data collected by the acquisition card is sent to the 3D model hand through the serial port for real-time display (see, for example Figure 3 as shown).
[0050] exist Figure 2 As an example, the gesture recognition device 100 may include: a fiber optic sensing glove ( Figure 2 The optical fiber sensing probe includes 4 optical fiber sensing probes including 3 conical double C-shaped plastic optical fiber sensing areas (corresponding to the four fingers other than the thumb) and 1 optical fiber sensing probe including 2 conical double C-shaped plastic optical fiber sensing areas (corresponding to the thumb); and an integrated circuit board ( Figure 2 The wrist part in the figure) includes multiple (for example, 5) 650nm laser diodes, photoresistors, etc., a single-chip microcontroller, a serial port module, a lithium-ion battery, etc. In addition, for convenience, a computer can also be prepared for intuitive display.
[0051] As can be seen from the above, the various parts of the gesture recognition device (e.g., optical fiber sensing gloves, integrated circuit boards) can be integrated together and can be worn on the hands. Therefore, by integrating the light source module (e.g., LED), the sensing module (e.g., plastic optical fiber sensing probe), the acquisition module (e.g., photoresistor, detector, etc.), and the recognition module (e.g., single-chip microcomputer, etc.) into one, the advantages of miniaturization and high integration of the gesture recognition device are achieved.
[0052] exist Figure 2 Although it is shown that data is sent to an external device via a serial port, that is, wired communication ( Figure 2 The present invention is not limited to this, and the data can also be sent to an external device wirelessly. At this time, the components of the integrated circuit board can be appropriately adjusted or increased according to the wireless transmission method, and the present invention does not impose any special restrictions on this.
[0053] The following is an embodiment of the present invention. Figure 1 The formation process of the plastic optical fiber sensing area of the gesture recognition device is described.
[0054] The following description takes the tapered double C-shaped plastic optical fiber sensing area as an example: First, cut a section of plastic optical fiber, strip off the cladding to reserve 2cm of bare fiber, make a connector, grind, polish, and turn on the end face inspection instrument to ensure that the end face of the plastic optical fiber is flat; secondly, use a fusion taper machine to taper the plastic optical fiber to make a tapered plastic optical fiber with a taper diameter of 70.74μm ( Figure 4 The figure shows an optical microscope image of the plastic optical fiber before and after tapering). The second tapered area is stretched 5 cm away from the first tapered area, and a double C-shaped tapered POF probe is obtained using heat setting technology. Subsequently, according to the needs of the gloves, optical fiber sensing probes containing 4 plastic optical fiber sensing areas including 3 tapered double C-shapes (corresponding to the four fingers other than the thumb) and 1 optical fiber sensing probe containing 2 tapered double C-shapes (corresponding to the thumb) are formed.
[0055] The following reference Figure 6 A gesture recognition method according to an embodiment of the present invention is described.
[0056] In an embodiment, a gesture recognition method may include: S601, collecting data on light intensity changes; and S603, using a pre-trained neural network model to identify which type of gesture it is based on the data, wherein the data on light intensity changes are data on light intensity changes generated by light of a predetermined intensity based on the movement of a finger.
[0057] According to the above description, the light intensity variation generated based on the movement of the finger can be generated by the plastic optical fiber sensing area arranged in a tapered double C shape.
[0058] According to the above description, the light of predetermined intensity may be generated by an LED light source.
[0059] According to the above description, the plastic optical fiber sensing area can be set at the finger joint.
[0060] According to the above description, the plastic optical fiber sensing area can be set to be multiple. For example, the sensing area on the thumb is set to be two, and the sensing areas on the other fingers are set to be three.
[0061] Based on the above analysis, the present invention has the following advantages compared with the existing technology: (1) High integration and miniaturization: The present invention integrates multiple tapered double C-shaped plastic optical fiber sensing areas into a glove, and integrates the light source, detector, data processing module and power supply into an 8 cm × 8 cm PCB circuit board, thereby improving the system's integration and miniaturization.
[0062] (2) High measurement sensitivity: Gesture recognition is achieved by utilizing the bending loss of the tapered double C-shaped plastic optical fiber. This method has high sensitivity, with a sensitivity of 0.21dB / degree, a linear correlation coefficient of 0.9991, and a measurement range of 10~120°.
[0063] (3) Low cost: Plastic optical fiber is inexpensive and has low transmission loss in the visible light band. Inexpensive LEDs can be used as light sources and photoresistors can be used for signal detection, resulting in very low cost for this method.
[0064] (4) Stability and reliability: The sensor probe was bent repeatedly 10,000 times with a time interval of 100 ms using an automatic reciprocating telescopic testing machine. The bending angle was from 0° to 120°, and the change in light intensity did not exceed 0.2 dB, indicating that the sensor has good durability. At the same time, the inherent corrosion resistance and anti-electromagnetic interference advantages of optical fiber sensors also enhance the stability and reliability of this method.
[0065] (5) Wide range of application fields: This method is not only applicable to the fields of healthcare and sports science, but can also be applied to multiple fields such as human-computer interaction, and has broader market prospects and application potential.
[0066] (6) Simple operation process: Due to the high degree of system integration, users can complete gesture recognition without using multiple devices or performing complex operations. The operation process is simple and the human-computer interaction is friendly.
[0067] (7) High gesture recognition accuracy: By combining gesture recognition and deep learning, the recognition rate of six gestures reached 100%, the lowest accuracy rate was 96.2%, and the average accuracy rate of 12 gestures was 98.85%.
[0068] Although various embodiments of the present invention have been described with specific details and varying details for illustrative purposes, it will be understood by those skilled in the art that various modifications, additions and substitutions may be made based on the contents disclosed or described in the present invention without departing from the spirit and scope of the present invention as defined by the appended claims. In addition, embodiments may be combined to form additional embodiments.
Claims
1. A gesture recognition device, which can be worn on the hand, characterized in that: include: a light source module configured to generate light of a predetermined intensity; A sensing module, comprising a plastic optical fiber sensing area, wherein the plastic optical fiber sensing area is configured to generate light intensity changes based on finger movement; A collection module, configured to collect data of the light intensity change; as well as The recognition module uses a pre-trained neural network model to identify gestures based on the data.
2. The gesture recognition device according to claim 1, characterized in that: The plastic optical fiber sensing area is arranged in a tapered double C shape.
3. The gesture recognition device according to claim 1, characterized in that: The light source module includes an LED light source.
4. The gesture recognition device according to claim 2, characterized in that: The plastic optical fiber sensing area is arranged at the finger joint.
5. The gesture recognition device according to claim 2, characterized in that: The plastic optical fiber sensing area is arranged in a plurality.
6. A gesture recognition method, characterized in that: The method comprises: Collecting data on changes in light intensity; and Using a pre-trained neural network model to identify gestures based on the data, The data of light intensity change is data of light intensity change caused by light of predetermined intensity based on the movement of the finger.
7. The gesture recognition method according to claim 6, characterized in that: The light intensity variation based on the finger's movement is generated by the plastic optical fiber sensing area arranged in a tapered double C-shape.
8. The gesture recognition method according to claim 6, characterized in that: Light of predetermined intensity is generated by the LED light source.
9. The gesture recognition method according to claim 7, characterized in that: The plastic optical fiber sensing area is arranged at the finger joint.
10. The gesture recognition method according to claim 7, characterized in that: The plastic optical fiber sensing area is arranged in a plurality.