A sign language translation glove and a sign language recognition method

Abstract

This invention discloses a sign language translation glove and a sign language recognition method, including a glove body, a curvature sensor disposed on the fingers, an inertial sensor disposed on the back of the hand, a data processing module, and an output module. The data processing module extracts hand shape features based on finger bending data collected by the curvature sensor, extracts motion features based on hand motion data collected by the inertial sensor, and extracts coupling features based on the relative temporal relationship between hand shape features and motion features. Then, based on the hand shape prototype, motion prototype, and coupling prototype, a joint decision is made to output the sign language recognition result. This invention can improve the ability to distinguish similar gestures and supports the rapid expansion of new sign language words.

Description

Technical Field

[0001] This invention relates to the field of smart wearable devices and gesture recognition technology, specifically to a sign language translation glove and recognition method based on finger bending data and hand movement data for sign language recognition. Background Technology

[0002] Existing sign language translation gloves typically collect gesture data using flexion sensors on the fingers and motion sensors on the back of the hand, and then use a recognition model to output the corresponding translation. Current solutions often directly mix finger flexion data and hand motion data into the recognition model, failing to adequately distinguish the different roles of hand shape and motion information in sign language expression.

[0003] When hand shapes are similar but movements differ, or movements are similar but hand shapes differ, the above processing methods are prone to misidentification, affecting recognition accuracy and system scalability. Therefore, it is necessary to propose a sign language translation glove and recognition method that can separately characterize hand shape features, movement features, and the coupling relationship between the two. Summary of the Invention

[0004] The purpose of this invention is to provide a sign language translation glove and a sign language recognition method to solve the problems of mixed processing of hand shape information and motion information and insufficient ability to distinguish similar gestures in the prior art. Technical solution

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a sign language translation glove, comprising a glove body, a curvature sensor disposed on the fingers, an inertial sensor disposed on the back of the hand, a data processing module, and an output module. The curvature sensor is used to collect finger bending data, the inertial sensor is used to collect hand movement data, the data processing module is used to extract hand shape features based on the finger bending data, extract motion features based on the hand movement data, and extract coupling features based on the relative temporal relationship between the hand shape features and the motion features. Furthermore, based on the hand shape prototype, motion prototype, and coupling prototype, a joint decision is made on the gesture to be recognized to generate a sign language recognition result.

[0006] This invention also provides a sign language recognition method, comprising: collecting finger bending data and hand movement data; extracting hand shape features based on finger bending data; extracting movement features based on hand movement data; extracting coupling features based on the relative temporal relationship between hand shape features and movement features; constructing hand shape prototypes, movement prototypes, and coupling prototypes; performing joint decision-making based on the three prototypes; and outputting the sign language recognition result. Beneficial effects

[0007] Compared with the prior art, the present invention has at least the following beneficial effects: First, it can separate hand shape information and motion information for representation, reducing mutual interference between different types of features; second, it can use the coupling relationship between hand shape and motion for discrimination, improving the ability to distinguish similar gestures; third, it can generate corresponding prototypes of new sign language words through a small number of samples while keeping the main recognition parameters unchanged, thereby realizing the expansion of new words. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of the overall structure of the sign language translation glove of the present invention.

[0009] Figure 2 This is a hardware connection block diagram of the sign language translation glove of the present invention.

[0010] Figure 3 This is a schematic diagram of the training data format.

[0011] Figure 4 This is a flowchart of the sign language recognition method of the present invention.

[0012] Figure 5 This is a schematic diagram illustrating the joint decision-making of the hand-shaped prototype, motion prototype, and coupling prototype of the present invention.

[0013] Figure 6 A comparison chart showing the training time of the traditional LSTM after adding vocabulary and the training time of this invention.

[0014] Figure 7 This is a schematic diagram of the actual gloves. Detailed Implementation

[0015] like Figure 1 and Figure 2 As shown, the sign language translation glove in this embodiment includes a glove body 1, curvature sensors 21-25 disposed on the five fingers, a data processing module 3, and an inertial sensor 4 disposed on the back of the hand. The five curvature sensors are used to collect bending data of the five fingers, the inertial sensor is used to collect three-axis acceleration data and attitude angle data of the back of the hand, and the data processing module is used to process the collected data and output the sign language recognition result. Training data is as follows... Figure 3 As shown.

[0016] like Figure 4As shown, the sign language recognition method in this embodiment includes the following steps: S1, collecting bending data of five fingers and movement data of the back of the hand; S2, filtering, normalizing and temporally aligning the collected data; S3, extracting hand shape features based on bending data; S4, extracting movement features based on movement data; S5, extracting coupling features based on the relative temporal relationship between hand shape features and movement features; S6, constructing hand shape prototypes, movement prototypes and coupling prototypes respectively, and making joint decisions based on the three types of prototypes; S7, outputting the sign language recognition result, or outputting a rejection result or a pending confirmation result when the matching results are inconsistent.

[0017] like Figure 5 As shown, the gesture to be recognized is matched with the hand shape prototype, motion prototype, and coupling prototype respectively to obtain corresponding matching results. The data processing module performs a joint decision based on the three types of matching results to improve the recognition accuracy. In the joint decision process, the matching degree between the gesture to be recognized and the hand shape prototype, motion prototype, and coupling prototype is calculated respectively, and the final recognition result is determined based on the weighted result of the three types of matching degrees. Preferably, when the hand shape matching degree and motion matching degree both meet the preset threshold conditions, the corresponding sign language recognition result is output; when the hand shape matching degree meets the preset threshold but the motion matching degree does not meet the preset threshold, or the motion matching degree meets the preset threshold but the hand shape matching degree does not meet the preset threshold, a rejection result or a pending confirmation result is output.

[0018] When adding new sign language words, the corresponding hand shape prototype, motion prototype, and coupling prototype can be generated based on a small number of samples of the new sign language words, while keeping the main recognition parameters unchanged, thereby expanding the new sign language words.

[0019] To verify the application effect of the method of this invention in the scenario of adding new sign language words, a comparative experiment was conducted using the traditional whole-word classification LSTM method and the method of this invention. Experimental results show that, under the same test conditions, both methods can achieve the recognition of new sign language words; however, the method of this invention does not require retraining the backbone network, but instead achieves word expansion by generating prototypes of new categories. Figure 6 As shown, the traditional whole-word classification LSTM method takes 5.19 seconds to expand its vocabulary, while the method of this invention takes only 0.049 seconds. This invention achieves lower vocabulary expansion costs without compromising recognition performance.

Claims

1. A sign language translation glove, characterized in that... A sign language translation glove, characterized in that it comprises: a glove body; a curvature sensor disposed on the finger portion of the glove body for collecting finger bending data; an inertial sensor disposed on the back of the hand portion of the glove body for collecting hand motion data; a data processing module electrically connected to the curvature sensor and the inertial sensor respectively; and an output module electrically connected to the data processing module; wherein the data processing module is configured to: extract hand shape features based on the finger bending data, extract motion features based on the hand motion data, extract coupling features based on the relative temporal relationship between the hand shape features and the motion features, and perform a joint decision based on the hand shape prototype, motion prototype, and coupling prototype to generate a sign language recognition result, and control the output module to output the sign language recognition result.

2. A sign language recognition method, characterized in that, include: Collect finger bending data and hand movement data; extract hand shape features based on the finger bending data; extract movement features based on the hand movement data; extract coupling features based on the relative temporal relationship between the hand shape features and the movement features; construct hand shape prototype, movement prototype, and coupling prototype; perform joint decision based on the matching results between the gesture to be recognized and the hand shape prototype, movement prototype, and coupling prototype; output sign language recognition results.

3. The sign language translation glove according to claim 1, characterized in that: There are five curvature sensors, each corresponding to one of the five fingers; the inertial sensor includes a three-axis acceleration acquisition unit and an attitude angle acquisition unit.

4. The sign language translation glove or sign language recognition method according to claim 1 or 2, characterized in that: When the hand shape matching result meets the preset conditions but the motion matching result does not meet the preset conditions, or when the motion matching result meets the preset conditions but the hand shape matching result does not meet the preset conditions, the rejection result or the pending confirmation result is output.

5. The sign language translation glove or sign language recognition method according to claim 1 or 2, characterized in that: Without updating the main recognition parameters, corresponding hand shape prototypes, motion prototypes, and coupling prototypes are generated based on a small number of samples of newly added sign language words to achieve the expansion of newly added sign language words.

Images