Sign language recognition method based on intelligent gloves

Through smart gloves combining deep network and natural language processing technology, an end-to-end sign language recognition framework is built, which solves the problems of insufficient recognition accuracy and high equipment costs in existing gesture recognition technologies, and provides a more accurate and cheap gesture recognition solution. It has significantly improved the social communication ability of deaf and mute people.

CN119007294APending Publication Date: 2024-11-22THE ACAD OF TIANJIN UNIV HEFEI
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411140913.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing gesture recognition technology faces problems such as insufficient recognition accuracy, high equipment costs and privacy violations, which leads to the inability to promote it on a large scale, especially for social exchanges and quality of life improvement for deaf and mute people.

Method used

Using a smart glove-based sign language recognition method, by collecting open source sign language video data sets, using deep network to extract hand node information and imu simulation data, combining multi-dimensional Gaussian model and time series model for training, to build an end-to-end sign language recognition framework in real time Gather and output gesture data, and improve recognition accuracy through natural language processing.

Benefits of technology

A more accurate, natural and cheap gesture recognition solution is achieved, which significantly improves the social communication ability and quality of life of deaf and mute people, reduces the cost of equipment manufacturing, and improves data collection efficiency and model adaptability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007294A_ABST
    Figure CN119007294A_ABST
Patent Text Reader

Abstract

The invention discloses a sign language recognition method based on a smart glove, which is based on the smart glove, reduces the data acquisition difficulty by combining gesture sensor data and visual data through a multi-modal technology, trains a model by combining video data with a machine and a deep learning technology, and verifies the model by using sensor data. A sign language recognition model which is high in precision and easy to expand is completed, and the smoothness and accuracy of continuous sign language recognition are optimized in combination with a natural language processing method based on context calibration. Meanwhile, an end-to-end sign language recognition framework is constructed, and gesture data can be converted into texts or voices in real time, so that effective communication between deaf-mute people and people with normal hearing is supported.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent wearable devices and human-computer interaction, and in particular to a sign language recognition method based on intelligent gloves. Background Art

[0002] In contemporary society, with the rapid development of information technology, intelligent human-computer interaction technology has become one of the research hotspots, especially in the field of gesture recognition. Gesture recognition technology allows users to interact with devices non-contactly through simple gestures, greatly enhancing the naturalness and convenience of interaction. Existing gesture recognition technology can be mainly divided into three directions: vision-based gesture recognition, wireless perception-based gesture recognition, and sensor-based gesture recognition.

[0003] Vision-based gesture recognition technology relies on cameras to capture dynamic gesture data, combined with image processing and deep learning algorithms to recognize gestures. Although this method shows great potential in human-computer interaction, it is highly sensitive to ambient lighting and background, and relying only on two-dimensional information for recognition may lead to information loss, thus affecting the recognition accuracy and stability of the system. Stereoscopic vision-based systems provide higher recognition accuracy, but still face challenges in complex or changing backgrounds, and the difficulty and cost of technical implementation are high. In addition, this technology needs to deal with privacy and security issues, because continuous video surveillance may violate personal privacy.

[0004] Wireless sensing-based gesture recognition uses changes in wireless signals to identify gestures. It has the ability to penetrate occlusions and is suitable for environments where it is inconvenient to use visual sensors. However, these technologies still face challenges in environmental interference and recognition accuracy. For example, Wi-Fi-based gesture recognition and radar-based gesture recognition technologies, although they have demonstrated potential for application in human-computer interaction, their ability to recognize complex gestures still needs to be improved.

[0005] Sensor-based gesture recognition methods record gesture data in real time through various sensors and perform recognition in a model-driven or data-driven manner. These methods use a wide variety of sensors, such as capacitive sensors, electromyography sensors, accelerometers, and gyroscopes, which can provide more accurate gesture recognition. However, the application of sensor technology also needs to solve problems related to device portability and user comfort.

[0006] Therefore, although the existing technology has made certain progress in the field of gesture recognition, it still faces problems and challenges such as privacy infringement, high equipment cost and insufficient recognition accuracy, which prevents it from being widely promoted. Summary of the invention

[0007] The purpose of the present invention is to make up for the defects of the existing technology and provide a sign language recognition method based on smart gloves. The present invention provides a more accurate, natural and cheap gesture recognition solution for the deaf-mute people, significantly improving their social communication ability and quality of life; the present invention breaks the limitations of the existing technology and brings substantial life improvements to people who need special communication methods.

[0008] The present invention is achieved through the following technical solutions:

[0009] A sign language recognition method based on smart gloves specifically comprises the following steps:

[0010] (1) Make smart glove equipment;

[0011] (2) Collecting and preprocessing open source sign language video datasets: Extracting relevant sign language vocabulary videos from multiple open source sign language video datasets as datasets, and using the Real-ESRGAN deep network to preprocess the extracted videos;

[0012] (3) Hand deep model locates joint information: Use the MediaPine deep network to identify the skeleton information of the 21 nodes of the hand, then calculate the curvature data and IMU simulation data to reconstruct the three-dimensional coordinate axis of the hand; use the DBSCAN algorithm combined with the DTW distance to identify abnormal data caused by video quality;

[0013] (4) Model training and data verification: The hand data captured by the smart glove device is transmitted to the data processing center in real time, and the data is input into the multidimensional Gaussian model and time series model for training. The model and the smart glove device are combined to build a real-time gesture recognition method, and the recognized gestures are output;

[0014] (5) Build an end-to-end sign language recognition framework: Finally, complete the platform construction, build an end-to-end sign language recognition framework, collect glove data and output prediction results in real time, and convert them into voice or text.

[0015] The manufacturing process of the smart glove device described in step (1) is as follows: equip each finger of the glove with a Flex Sensor device, and equip the back of the glove with an IMU sensor, an Arduino board, and an STM board of a Bluetooth module.

[0016] The use of the mediapine deep network to identify the skeleton information of 21 nodes of the hand in step (3) is as follows: the joints of the hand are identified and located from the open source sign language video, wherein the joints include the joints of the wrist, thumb, index finger, middle finger, ring finger and little finger, and a total of 21 key points are marked.

[0017] The calculated curvature data described in step (3) is specifically as follows:

[0018] After using the Mediapipe model to process the video to obtain the hand joint data of each frame, the four key points of the finger, namely the metacarpophalangeal joint MCP, the proximal finger joint PIP, the distal finger joint DIP and the fingertip Tip, are located, and then two vectors are constructed to describe the main parts of the finger:

[0019] vector From MCP to PIP

[0020] vector From DIP to Tip

[0021] Then calculate the angle θ between the vectors as the curvature of the finger:

[0022]

[0023] According to this formula, the finger bending degree is calculated and saved by applying the joint point data of each frame.

[0024] The calculation of imu simulation data described in step (3) is as follows:

[0025] Based on the data of 21 nodes of the hand, a new three-dimensional coordinate system is constructed with the palm as the center, and the direction of the coordinate system conforms to the three-axis direction of the imu sensor; first, the y-axis passes through the wrist node and the palm node and points to the palm node, the x-axis is located in the palm plane and perpendicular to the y-axis, and the z-axis is calculated by the cross product of these two axes and points to the back of the palm. After obtaining the three axis vectors, in order to ensure that the X, Y, and Z axes are orthogonal, the three axis vectors are orthogonalized using the Gram-Schmidt method. First, the three axes are used as column vectors to form a matrix A=(a x ,a y ,a z ), and finally the normalized and orthogonalized axis vector e is obtained by the following calculation x 、e y 、e z ,in Represents a y in u x The projection on, and so on, and u x 、u y 、u z is the ununitized axis vector;

[0026]

[0027] Since the camera is perpendicular to the human body, and the coordinate system of MediaPipe Hands is based on the upper right corner of the image, the x-axis is from left to right, the y-axis is from top to bottom, and the z-axis points away from the camera, the vector G = (0, -1, 0) * local gravity acceleration is used to simulate the actual gravity, and then the acceleration is projected onto the reconstructed three-dimensional coordinate axis;

[0028] First, construct the matrix A of the new coordinate system = (a x ,a y ,a z ), and then find its inverse matrix A -1 , and finally calculate the projection of vector G on these three axes:

[0029] G′=A -1 G

[0030] In summary, the components of gravity acceleration actually measured by the IMU are simulated on the three axes, and used to determine the direction of the hand.

[0031] The step (3) of identifying abnormal data caused by video quality by using the DBSCAN algorithm combined with the DTW distance is as follows:

[0032] Given two sets of time series data X = (x 1 ,x 2 ,…,x n ) and Y=(y 1 ,y 2 ,…,y m ), first calculate the distance matrix D between them, where D(i,j)=d(x i ,y j ), d is the distance measurement method between vectors, using Euclidean distance; then calculate the cumulative distance matrix C, calculated by the distance matrix D, each element in the distance matrix represents the minimum cumulative distance from the starting point to point C(i,j), where i and j represent the i-th and j-th data in the time series X and Y, respectively, that is, x i ,y j , the calculation formula is:

[0033] C(i,j)=D(i,j)+min(C(i-1,j),C(i,j-1),C(i-1,j-1))

[0034] Among them, the boundary condition is C(1,1)=D(1,1);

[0035] Finally, trace back from the end point C(n,m) of the cumulative distance matrix C to the starting point C(1,1) to find a path that minimizes the cumulative distance on the path. This path is the best matching path between the two time series. The cumulative distance of the path is divided by the length of the alignment path to obtain the average distance of each alignment step, which is the similarity between time series pairs of different lengths. The similarity matrix between different files is obtained in turn, and then the dbscan clustering algorithm is used to find abnormal data, and an adaptive abnormal radius is used for each type of sign language.

[0036] In step (4), before outputting the recognized gesture, a natural language processing technique based on context calibration is applied for verification to obtain a more accurate result.

[0037] The advantages of the present invention are: (1) the equipment is simple to manufacture and has low cost.

[0038] The smart gloves used in the present invention are simple in design, low in cost, and easy to mass produce. Such lightweight gloves have little impact on the daily activities of users and are easy to wear for a long time, thereby improving user experience.

[0039] (2) Multimodal data acquisition

[0040] Using a public sign language video dataset as a training set, the deep hand model is used to extract the hand node data of the sign language video, and then converted into sensor data through an algorithm, which reduces the complexity of traditional data collection methods and eliminates the need to rely on volunteers for repeated data entry experiments. This method improves the convenience of data collection and the data collection efficiency of the training model, reducing costs.

[0041] (3) Innovative data conversion algorithm

[0042] The present invention implements an efficient algorithm for extracting similar sensor data from videos, which can generate a data set similar to real sensor data.

[0043] (4) Incremental learning method

[0044] The model design is based on the incremental learning method, so when the number of sign languages ​​in the data set is expanded, it is no longer necessary to retrain all the data. Moreover, as new data arrives, the incremental learning model can adapt to potential changes in data distribution, which makes the model more robust in the face of non-static environments. In short, the cost of model training and updating is reduced, and it can more conveniently learn and adapt to the user's unique sign language movement characteristics during use. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall flow chart of the present invention;

[0046] Figure 2 is a schematic diagram of the gloves of the present invention;

[0047] Figure 3 Schematic diagram of hand node recognition for the model;

[0048] Figure 4 Schematic diagram of reconstructed three-dimensional coordinate axes. DETAILED DESCRIPTION

[0049] like Figure 1 As shown, a sign language recognition method based on smart gloves comprises the following steps:

[0050] Step 1: Design and manufacture smart gloves, equip each finger with Flex Sensor, imu sensor and arduino board on the back of the glove. Use the stm board of the Bluetooth module to collect and transmit data in real time, so as to accurately capture the hand movement.

[0051] Step 2: Use the open source sign language video dataset as a training set, first pre-process the video to improve the recognition accuracy, then use the hand depth model to identify and obtain the hand node data. Convert the obtained data into a format corresponding to the smart glove sensor data for subsequent processing and analysis.

[0052] Step 3: Use multimodal data to train the sign language recognition model, and continuously optimize the model through machine and deep learning methods to improve the recognition accuracy and adaptability, so that it can adapt to multiple environments and user gesture characteristics and handle complex sign language expressions.

[0053] Step 4: The gesture data captured by the smart gloves is transmitted to the data processing center in real time for immediate processing and analysis. The data is used to verify the training model, and then a real-time gesture recognition method is built by combining the model and the hardware gloves to output the recognized gestures.

[0054] Step 5: Before outputting the recognition results, natural language processing technology based on contextual calibration is applied to enhance the contextual accuracy and semantic coherence of sign language recognition. This technology ensures that the system can accurately interpret continuous sign language and improve the natural fluency of communication.

[0055] Step 6: Build an end-to-end sign language recognition framework to convert gesture data into text or speech in real time to support effective communication between the deaf and hearing people.

[0056] Hand joint positioning

[0057] First, if Figure 2 As shown in Figure 1, the joints of the hand are identified and located from the sign language video. These joints include the joints of the wrist, thumb, index finger, middle finger, ring finger and little finger, with a total of 21 key points marked (such as Figure 3Points 0 to 20 shown).

[0058] Flexsensor data simulation

[0059] Taking the index finger as an example, after obtaining the hand joint data from the video, the four main key points of the index finger are located, namely the metacarpophalangeal joint (MCP), the proximal interphalangeal joint (PIP), the distal interphalangeal joint (DIP), and the tip. Then two vectors are constructed to describe the main parts of the index finger:

[0060] vector From MCP to PIP

[0061] vector From DIP to Tip

[0062] Then calculate the angle θ between the vectors as the curvature of the finger:

[0063]

[0064] According to this formula, the finger bending degree is calculated and saved by applying the joint point data of each frame.

[0065] imu data simulation

[0066] Based on the 21 node data of the hand, a new three-dimensional coordinate system is constructed with the palm as the center, and the orientation of the coordinate system is consistent with the three-axis orientation of the imu sensor. The three-dimensional coordinates of the hand are reconstructed as follows Figure 3 As shown; first, the y-axis passes through the wrist node and the palm node and points to the palm node, the x-axis is located in the palm plane and perpendicular to the y-axis, and the z-axis is calculated by the cross product of these two axes and points to the back of the palm. After obtaining these three axis vectors, in order to ensure that the X, Y, and Z axes are orthogonal, the Gram-Schmidt method is used to align them for orthogonality, as follows:

[0067]

[0068] Since the camera is perpendicular to the human body, and the coordinate system of MediaPipe Hands is based on the upper right corner of the image, the x-axis is from left to right, the y-axis is from top to bottom, and the z-axis points away from the camera, the vector G = (0, -1, 0) * local gravity acceleration is used to simulate the actual gravity, and then the acceleration is projected onto the reconstructed three-dimensional coordinate axis;

[0069] First, construct the matrix A of the new coordinate system = (a x ,a y,a z ), and then find its inverse matrix A -1 , and finally calculate the projection of vector G on these three axes:

[0070] G′=A -1 G

[0071] In summary, the components of gravity acceleration actually measured by the IMU are simulated on the three axes, and used to determine the direction of the hand.

[0072] Abnormal data extraction

[0073] Due to the problem of video quality, the extracted data is both good and bad. Based on the assumption that most of the data is good data, the dbscan algorithm with adaptive anomaly radius improved by this project is used to delete abnormal data. The experimental results show that basically all abnormal data that is different from other files can be found and deleted, thereby greatly improving the purity of the data.

[0074] The following uses a single sign language vocabulary to further illustrate the sign language recognition method based on smart gloves. The overall process is as follows: Figure 1 shown.

[0075] (1) Design and manufacture of smart glove equipment: The initial smart glove product successfully manufactured is shown in the figure below. Figure 2 As shown. And realize data synchronization and real-time data processing code between different devices.

[0076] (2) Collect and preprocess open source sign language video datasets: Combine multiple open source sign language datasets to extract relevant sign language vocabulary videos as datasets, and use the Real-ESRGAN deep network to preprocess the videos to improve clarity and enhance hand contours.

[0077] Hand depth model locates joint information: Use mediapine deep network to identify the skeleton information of 21 nodes of the hand (such as Figure 3 As shown), then calculate the curvature data and imu simulation data, such as reconstructing the three-dimensional coordinate axis of the hand (such as Figure 4 Then, the abnormal data caused by video quality is identified by using the improved DBSCAN algorithm combined with DTW distance.

[0078] Given two sets of time series data X = (x 1 ,x 2 ,…,x n ) and Y=(y 1 ,y 2 ,…,y m ), first calculate the distance matrix D between them, where D(i,j)=d(x i ,y j), d is the distance measurement method between vectors, using Euclidean distance; then calculate the cumulative distance matrix C, calculated by the distance matrix D, each element in the distance matrix represents the minimum cumulative distance from the starting point to point C (i, j), the calculation formula is:

[0079] C(i,j)=D(i,j)+min(C(i-1,j),C(i,j-1),C(i-1,j-1))

[0080] Among them, the boundary condition is C(1,1)=D(1,1);

[0081] Finally, trace back from the end point C(n,m) of the cumulative distance matrix C to the starting point C(1,1) to find a path that minimizes the cumulative distance on the path. This path is the best matching path between the two time series. The cumulative distance of the path is divided by the length of the alignment path to obtain the average distance of each alignment step, which is the similarity between time series pairs of different lengths. The similarity matrix between different files is obtained in turn, and then the dbscan clustering algorithm is used to find abnormal data, and an adaptive abnormal radius is used for each type of sign language.

[0082] (3) Model training and sensor data verification: The processed data is input into the multidimensional Gaussian model and time series model for training, and finally the data obtained by the sensor is used for verification.

[0083] (4) Apply context calibration to process the output results: Before the prediction results are output, they are verified in combination with the context to obtain more accurate results.

[0084] Build an end-to-end sign language recognition framework: Finally, complete the platform construction, collect glove data in real time, output prediction results in real time, and convert them into voice or text.

Claims

1. A sign language recognition method based on smart gloves, characterized by: The specific steps include: (1) Make smart glove equipment; (2) Collecting and preprocessing open source sign language video datasets: Extracting relevant sign language vocabulary videos from multiple open source sign language video datasets as datasets, and using the Real-ESRGAN deep network to preprocess the extracted videos; (3) Hand deep model locates joint information: Use the MediaPine deep network to identify the skeleton information of the 21 nodes of the hand, then calculate the curvature data and IMU simulation data to reconstruct the three-dimensional coordinate axis of the hand; use the DBSCAN algorithm combined with the DTW distance to identify abnormal data caused by video quality; (4) Model training and data verification: The hand data captured by the smart glove device is transmitted to the data processing center in real time, and the data is input into the multidimensional Gaussian model and time series model for training. The model and the smart glove device are combined to build a real-time gesture recognition method, and the recognized gestures are output; (5) Build an end-to-end sign language recognition framework: Finally, complete the platform construction, build an end-to-end sign language recognition framework, collect glove data and output prediction results in real time, and convert them into voice or text.

2. The sign language recognition method based on smart gloves according to claim 1, characterized in that: The manufacturing process of the smart glove device described in step (1) is as follows: equip each finger of the glove with a Flex Sensor device, and equip the back of the glove with an IMU sensor, an Arduino board, and an STM board of a Bluetooth module.

3. The sign language recognition method based on smart gloves according to claim 2, characterized in that: The use of the mediapine deep network to identify the skeleton information of 21 nodes of the hand in step (3) is as follows: the joints of the hand are identified and located from the open source sign language video, wherein the joints include the joints of the wrist, thumb, index finger, middle finger, ring finger and little finger, and a total of 21 key points are marked.

4. The sign language recognition method based on smart gloves according to claim 3 is characterized in that: The calculated curvature data described in step (3) is specifically as follows: After using the Mediapipe model to process the video to obtain the hand joint data of each frame, the four key points of the finger, namely the metacarpophalangeal joint MCP, the proximal finger joint PIP, the distal finger joint DIP and the fingertip Tip, are located, and then two vectors are constructed to describe the main parts of the finger: vector From MCP to PIP vector From DIP to Tip Then calculate the angle θ between the vectors as the curvature of the finger: According to this formula, the finger bending degree is calculated and saved by applying the joint point data of each frame.

5. The sign language recognition method based on smart gloves according to claim 4, characterized in that: The calculation of imu simulation data described in step (3) is as follows: Based on the data of 21 nodes of the hand, a new three-dimensional coordinate system is constructed with the palm as the center, and the direction of the coordinate system conforms to the three-axis direction of the imu sensor; first, the y-axis passes through the wrist node and the palm node and points to the palm node, the x-axis is located in the palm plane and perpendicular to the y-axis, and the z-axis is calculated by the cross product of these two axes and points to the back of the palm. After obtaining the three axis vectors, in order to ensure that the X, Y, and Z axes are orthogonal, the three axis vectors are orthogonalized using the Gram-Schmidt method. First, the three axes are used as column vectors to form a matrix A=(a x ,a y ,a z ), and finally the normalized and orthogonalized axis vector e is obtained by the following calculation x 、e y、 e z ,in Represents a y in u x The projection on, and so on, and u x 、u y 、u z is the ununitized axis vector; Since the camera is perpendicular to the human body, and the coordinate system of MediaPipe Hands is based on the upper right corner of the image, the x-axis is from left to right, the y-axis is from top to bottom, and the z-axis points away from the camera, the vector G = (0, -1, 0) is multiplied by the local gravity acceleration to simulate the actual gravity, and then the acceleration is projected onto the reconstructed three-dimensional coordinate axis; the process is to first find the inverse matrix A -1 , and finally calculate the projection of vector G on these three axes: G ′ =A -1 G In summary, the components of gravity acceleration actually measured by the IMU are simulated on the three axes, and used to determine the direction of the hand.

6. The sign language recognition method based on smart gloves according to claim 5, characterized in that: The step (3) of identifying abnormal data caused by video quality by using the DBSCAN algorithm combined with the DTW distance is as follows: Given two sets of time series data X = (x1, x2, ..., x n ) and Y=(y1,y2,…,y m ), first calculate the distance matrix D between them, where D(i,j)=d(x i ,y j ), d is the distance measurement method between vectors, using Euclidean distance; then calculate the cumulative distance matrix C, calculated by the distance matrix D, each element in the distance matrix represents the minimum cumulative distance from the starting point to point C(i,j), where i and j represent the i-th and j-th data in the time series X and Y, respectively, that is, x i ,y j , the calculation formula is: C(i,j)=D(i,j)+min(C(i-1,j),C(i,j-1),C(i-1,j-1)) Among them, the boundary condition is C(1,1)=D(1,1); Finally, trace back from the end point C(n,m) of the cumulative distance matrix C to the starting point C(1,1) to find a path that minimizes the cumulative distance on the path. This path is the best matching path between the two time series. The cumulative distance of the path is divided by the length of the alignment path to obtain the average distance of each alignment step, which is the similarity between time series pairs of different lengths. The similarity matrix between different files is obtained in turn, and then the dbscan clustering algorithm is used to find abnormal data, and an adaptive abnormal radius is used for each type of sign language.

7. The sign language recognition method based on smart gloves according to claim 1, characterized in that: In step (4), before outputting the recognized gesture, a natural language processing technique based on context calibration is applied for verification to obtain a more accurate result.

Citation Information

Cited By

  • Automatic refrigerator transportation identification system

    CN120430707A

  • An automated refrigerator transport identification system

    CN120430707B