Real-time sign language recognition system and device based on deep learning

Abstract

The invention provides a real-time sign language recognition system and device based on deep learning, and relates to the technical field of video recognition. According to the real-time sign language recognition system and device based on deep learning, a camera is used for obtaining a real-time sign language RGB video, the video is subjected to frame extraction, an image left after frame extraction is subjected to image preprocessing, processed data is input into a Mediape network for key point detection, and then detected key point data is input into a Transform network for sign language classification. The system has the characteristics of real-time performance, high recognition rate and the like, key points of a human body can be detected in real time, sign language recognition can be carried out, sign language actions can be converted into understandable text or voice information by the sign language recognition system, so that the system helps deaf-mute people to communicate with non-sign language users without barriers, and the system has a wide application prospect and is worthy of popularization and application. The system can be deployed in deaf-mute schools, shopping malls, hospitals and other places to assist hearing-impaired people in communication.

Description

Technical Field

[0001] The present invention relates to the field of video recognition technology, and in particular to a real-time sign language recognition system and device based on deep learning. Background Art

[0002] Sign language is a visual language based on hand movements, gestures, and body postures, used by the deaf and mute to communicate and express themselves. However, for non-sign language users, understanding sign language and communicating can be a challenge. The emergence of sign language recognition technology provides more ways for the deaf and mute to communicate. By leveraging computer vision and machine learning technology, sign language recognition systems can convert sign language movements into understandable text or voice information, thereby helping the deaf and mute to communicate with non-sign language users without barriers.

[0003] The significance of sign language recognition is that it can promote the social inclusion and participation of the deaf. This technology enables the deaf to communicate more easily with family, friends and the general public, eliminating their barriers in face-to-face communication. In addition, sign language recognition technology also provides more educational and employment opportunities for the deaf, enabling them to participate in a wider range of social activities.

[0004] At present, sign language recognition technology has made some progress, but it still faces some challenges. The diversity and complexity of sign language make the design of sign language recognition algorithms complicated. In addition, there are differences between different sign language systems, which require adaptation and training for different sign language systems. This requires more research and development to improve the accuracy and practicality of sign language recognition. At present, there are two main ways to recognize sign language. One is to use wearable devices such as sign language recognition gloves to collect hand movements based on the sensors on the gloves, and perform sign language recognition based on these movement information. This method requires special wearable devices and is inconvenient. The other is a recognition method based on obtaining sign language video data from a camera, extracting features from the sign language video, and then using template matching or other machine learning methods to recognize the features. At present, the recognition rates of these two sign language recognition methods are low, and it is difficult to meet the needs of real-time sign language recognition.

[0005] To this end, we have developed a new real-time sign language recognition system and device based on deep learning. Summary of the invention

[0006] 1. Technical issues to be solved

[0007] In view of the shortcomings of the prior art, the present invention provides a real-time sign language recognition system and device based on deep learning, which solves the defects of the two sign language recognition methods in the prior art. One is to use wearable devices such as sign language recognition gloves to collect hand movements according to the sensors on the gloves, and perform sign language recognition based on these movement information. This method requires special wearable devices and is relatively inconvenient; the other is a recognition method based on obtaining sign language video data through a camera, extracting features from the sign language video, and then using template matching or other machine learning methods to recognize the features. At present, the recognition rates of these two sign language recognition methods are low, and it is difficult to meet the needs of real-time sign language recognition.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: a real-time sign language recognition system and device based on deep learning, including a camera module for image acquisition, a host module for data processing and analysis, and a display module for displaying the sign language recognition results. The camera transmits the acquired data to the data processing and analysis host module in real time, and after data preprocessing and neural network model inference, the results are transmitted to the display module, so that the user can obtain the meaning of the sign language video;

[0010] Among them, a camera is used to collect real-time sign language video data and extract frames of a fixed length. The Mediapipe network model is used to detect the human skeleton key points and facial key points of the extracted video frames. The key point data is input into the Transformer encoder network, the sign language classification results are output, and the results are returned to the sign language recognition system.

[0011] Preferably, the training and recognition method of the real-time sign language recognition model includes the following specific steps:

[0012] Step 1: Data collection and preprocessing:

[0013] Multiple people demonstrated 500 common sign language words repeatedly, and each sign language video recording lasted about 3 seconds;

[0014] The sign language video recording will be performed within 3 seconds after the system sends out the start recognition signal;

[0015] Randomly extract frames from the recorded video, extracting a total of 30 frames to form a video sequence F = [f1, f2.....f30];

[0016] Step 2: Face, body and hand key point detection:

[0017] Use the holistic model in Mediapipe to perform matrix detection on the key points of the human body in the video frame. Each frame contains the 3D position and confidence data of 33 key points of the body, the 3D position data of 468 key points of the face, and the 3D position data of 21 key points of the left and right hands.

[0018] Flatten all key point data of a single frame. 30 frames of data have a total of 30*1662 data.

[0019] The first dimension represents the time information of the video, and the second dimension represents the spatial information of the video action. Sign language is different from simple spatial gesture recognition and is continuous spatiotemporal information. Therefore, the above data points containing spatiotemporal information can be used as data elements for sign language recognition.

[0020] Step 3: Data labeling and training:

[0021] The obtained human body key point data is annotated, and the annotated videos are divided into 500 categories. The annotated video human body key point information and classification data are divided into a training set and a test set according to an 8:2 ratio;

[0022] The dataset is sent to the Transformer network for training, and the trained model is verified using the test set.

[0023] Preferably, the loss function used in the training process is a cross entropy loss function, which compares the predicted category with the actual category and imposes a penalty based on the difference between the probability and the actual expected value. The cross entropy loss function is defined as:

[0024]

[0025] Among them, t i The value is 0 or 1, which is the sample label value, p i is the probability that the sample prediction value is the i-th category.

[0026] Preferably, the Transformer network encoder comprises 4 layers, the number of heads in the multi-head attention mechanism is 6, the activation function is Relu, the number of hidden layer nodes of the feedforward neural network is 2048, the loss rate is set to 0.1, and then the output result is connected to a fully connected neural network to output the probability results of 500 classifications.

[0027] Preferably, when using the trained model for inference, the maximum probability value of the predicted result is used as the classification result. Adjusting the detection threshold can control the confidence of the target classification result, and discarding the confidence below the threshold. Only the detection results with confidence above the threshold are considered to be correct sign language classification results.

[0028] (III) Beneficial effects

[0029] The present invention provides a real-time sign language recognition system and device based on deep learning. It has the following beneficial effects:

[0030] The present invention uses a camera to obtain real-time sign language RGB video, extracts frames from the video, performs image preprocessing on the remaining images after the extraction, inputs the processed data into the Mediapipe network for key point detection, and then inputs the detected key point data into the Transformer network for sign language classification. The system has the characteristics of real-time performance and high recognition rate, and can detect key points of the human body in real time and perform sign language recognition. The sign language recognition system can convert sign language movements into understandable text or voice information, thereby helping the deaf-mute people to communicate with non-sign language users without barriers. It has a wide range of application prospects and can be deployed in deaf-mute schools, shopping malls, hospitals and other places to assist the hearing-impaired in communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a technical flow chart of the present invention;

[0032] Figure 2 It is a schematic diagram of the device of the present invention;

[0033] Figure 3 It is the software interface layout design diagram of the present invention;

[0034] Figure 4 It is the interface logic diagram of the present invention;

[0035] Figure 5 An example diagram of an interactive display module of the present invention;

[0036] Figure 6 This is a physical diagram of a prototype of the present invention;

[0037] Figure 7 The code map of key points of face, limbs, and hands extracted by MediaPipe of the present invention;

[0038] Figure 8 The Transformer network structure diagram of 500 classifications of the present invention;

[0039] Fig. 9 It is a detailed parameter diagram of the model training set of the present invention;

[0040] Fig.10 It is a training set accuracy curve diagram of the present invention;

[0041] Fig.11 It is a test set accuracy curve diagram of the present invention;

[0042] Fig.12It is the training set Loss curve diagram of the present invention;

[0043] Fig.13 It is the test set Loss curve diagram of the present invention;

[0044] Fig.14 This is the Top3 accuracy curve of the training set of the present invention;

[0045] Fig.15 This is the Top3 accuracy curve of the test set of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Example:

[0048] like Figure 1 - Fig.15 As shown, the embodiment of the present invention provides a real-time sign language recognition system and device based on deep learning, including a camera module for image acquisition (the training level used in this application is RGB sign language video, and the software function test is from an ordinary real-time acquisition RGB camera), a host module for data processing and analysis, and a display module for displaying sign language recognition results. The camera transmits the collected data to the data processing and analysis host module (Jetson Nano and other terminals) in real time, and after data preprocessing and neural network model inference, the results are transmitted to the display module, so that the user can obtain the meaning of the sign language video; such as:

[0049] (1) The acquisition module uses a camera to collect the user’s sign language movements and facial expressions in real time.

[0050] (2) Data processing: Use the Mediapipe method to obtain the user’s sign language movements and facial expression key point data in real time.

[0051] (3) Reasoning module: Use the trained Transformer encoding module to classify sign language actions.

[0052] (4) An interactive display module, including buttons for starting, pausing, and stopping sign language recognition, an area for displaying the video captured by the user in real time, and an area for displaying the sign language recognition results.

[0053] Specifically, a camera is used to collect real-time sign language video data and extract frames of a fixed length. The Mediapipe network model is used to detect the human skeleton key points and facial key points of the extracted video frames. The key point data is input into the Transformer encoder network, and the sign language classification results are output. The results are returned to the sign language recognition system. The trained model is used for inference to achieve real-time sign language recognition, and the recognition results are placed in the interactive display module for display.

[0054] The training and recognition method of the real-time sign language recognition model includes the following specific steps:

[0055] Step 1: Data collection and preprocessing:

[0056] Multiple people demonstrated 500 common sign language words repeatedly, and each sign language video recording lasted about 3 seconds;

[0057] The sign language video recording will be performed within 3 seconds after the system sends out the start recognition signal;

[0058] Randomly extract frames from the recorded video, a total of 30 frames are extracted to form a video sequence F = [f1, f2.....f30]; among them, randomly extracting frames from the video is to increase the robustness of the model and improve the generalization ability of the classification model.

[0059] Step 2: Face, body and hand key point detection:

[0060] Use the holistic model in Mediapipe to perform matrix detection on the key points of the human body in the video frame. Each frame contains the 3D position and confidence data of 33 key points of the body, the 3D position data of 468 key points of the face, and the 3D position data of 21 key points of the left and right hands.

[0061] Flatten all key point data of a single frame. 30 frames of data have a total of 30*1662 data.

[0062] The first dimension represents the time information of the video, and the second dimension represents the spatial information of the video action. Sign language is different from simple spatial gesture recognition and is continuous spatiotemporal information. Therefore, the above data points containing spatiotemporal information can be used as data elements for sign language recognition.

[0063] Step 3: Data labeling and training:

[0064] The obtained human body key point data is annotated, and the annotated videos are divided into 500 categories. The annotated video human body key point information and classification data are divided into a training set and a test set according to an 8:2 ratio;

[0065] The dataset is sent to the Transformer network for training, and the trained model is verified using the test set.

[0066] The loss function used in the training process is the cross entropy loss function. The cross entropy loss function compares the predicted category with the actual category and penalizes according to the difference between the probability and the actual expected value. The cross entropy loss function is defined as:

[0067]

[0068] Among them, t i The value is 0 or 1, which is the sample label value, p i is the probability that the sample prediction value is the i-th category.

[0069] The Transformer network encoder contains 4 layers, the number of heads in the multi-head attention mechanism is 6, the activation function is Relu, the number of hidden layer nodes of the feedforward neural network is 2048, the loss rate is set to 0.1, and then the output result is connected to a fully connected neural network to output the probability results of 500 classifications.

[0070] When using the trained model for inference, the maximum probability value of the predicted result is used as the classification result. Adjusting the detection threshold can control the confidence of the target classification result. Detection results with confidence values ​​lower than the threshold are discarded. Only detection results with confidence values ​​higher than the threshold are considered to be correct sign language classification results.

[0071] Application examples:

[0072] 1. Data Preprocessing

[0073] The video clips in the training data set are about 2-4 seconds long and 30 frames per second. First, 30 frames are evenly extracted from the video as sequence frames for extracting key points of the human body.

[0074] Using the pre-trained model provided by MediaPipe, the hand image is preprocessed to extract the hand key point sequence. Then the key point sequence is used as input to split the gesture sequence into multiple subsequences, each of which contains a fixed number of key points.

[0075] like Figure 7 As shown, MediaPipe is used to extract key point codes of human face, limbs, and hands.

[0076] 2. Model Training

[0077] like Figure 8As shown in Figure 2, the Transformer network is used to model and represent each subsequence. Specifically, each subsequence is taken as input and mapped into a continuous vector space through a multi-layer Transformer encoder. During the training process, the cross entropy loss function is used to classify the gesture sequence to predict the type of gesture.

[0078] 3. Model Testing

[0079] like Fig. 9 As shown in the figure, the model training set accuracy reached 99.87%, the Top3 accuracy was 100%, the validation set accuracy was 0.984, and the validation set Top3 accuracy was 0.998.

[0080] And, if Fig.10 - Fig.15 The results show that the accuracy of the 500-category sign language recognition dataset based on the Transfomer model architecture has reached a high level.

[0081] In summary, overall, the present invention can detect key points of the human body in real time and perform sign language recognition. The sign language recognition system can convert sign language movements into understandable text or voice information, thereby helping deaf-mute people to communicate with non-sign language users without barriers. When used specifically, it can use a camera to obtain real-time sign language RGB video, extract frames from the video, perform image preprocessing on the remaining images after the extraction of frames, input the processed data into the Mediapipe network for key point detection, and then input the detected key point data into the Transformer network for sign language classification. The system has the characteristics of real-time performance and high recognition rate, and has broad application prospects. It can be deployed in deaf-mute schools, shopping malls, hospitals and other places to assist hearing-impaired people in communication.

[0082] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the equivalents of the appended claims.

Claims

1. A real-time sign language recognition system and device based on deep learning, comprising a camera module for image acquisition, a host module for data processing and analysis, and a display module for displaying sign language recognition results, characterized in that: The camera transmits the collected data to the data processing and analysis host module in real time, and after data preprocessing and neural network model inference, the results are transmitted to the display module, so that the user can obtain the meaning of the sign language video; Among them, a camera is used to collect real-time sign language video data and extract frames of a fixed length. The Mediapipe network model is used to detect the human skeleton key points and facial key points of the extracted video frames. The key point data is input into the Transformer encoder network, the sign language classification results are output, and the results are returned to the sign language recognition system.

2. A real-time sign language recognition system and device based on deep learning according to claim 1, characterized in that: The training and recognition method of the real-time sign language recognition model includes the following specific steps: Step 1: Data collection and preprocessing: Multiple people demonstrated 500 common sign language words repeatedly, and each sign language video recording lasted about 3 seconds; The sign language video recording will be performed within 3 seconds after the system sends out the start recognition signal; Randomly extract frames from the recorded video, extracting a total of 30 frames to form a video sequence F = [f1, f2.....f30]; Step 2: Face, body and hand key point detection: Use the holistic model in Mediapipe to perform matrix detection on the key points of the human body in the video frame. Each frame contains the 3D position and confidence data of 33 key points of the body, the 3D position data of 468 key points of the face, and the 3D position data of 21 key points of the left and right hands. Flatten all key point data of a single frame. 30 frames of data have a total of 30*1662 data. The first dimension represents the time information of the video, and the second dimension represents the spatial information of the video action. Sign language is different from simple spatial gesture recognition and is continuous spatiotemporal information. Therefore, the above data points containing spatiotemporal information can be used as data elements for sign language recognition. Step 3: Data labeling and training: The obtained human body key point data is annotated, and the annotated videos are divided into 500 categories. The annotated video human body key point information and classification data are divided into a training set and a test set according to an 8:2 ratio; The dataset is sent to the Transformer network for training, and the trained model is verified using the test set.

3. The real-time sign language recognition system and device based on deep learning according to claim 2, characterized in that: The loss function used in the training process is the cross entropy loss function. The cross entropy loss function compares the predicted category with the actual category and penalizes according to the difference between the probability and the actual expected value. The cross entropy loss function is defined as: Among them, t i The value is 0 or 1, which is the sample label value, p i is the probability that the sample prediction value is the i-th category.

4. The real-time sign language recognition system and device based on deep learning according to claim 3, characterized in that: The Transformer network encoder contains 4 layers, the number of heads in the multi-head attention mechanism is 6, the activation function is Relu, the number of hidden layer nodes of the feedforward neural network is 2048, the loss rate is set to 0.1, and then the output result is connected to a fully connected neural network to output the probability results of 500 classifications.

5. A real-time sign language recognition system and device based on deep learning according to claim 4, characterized in that: When using the trained model for inference, the maximum probability value of the predicted result is used as the classification result. Adjusting the detection threshold can control the confidence of the target classification result. Detection results with confidence values ​​lower than the threshold are discarded. Only detection results with confidence values ​​higher than the threshold are considered to be correct sign language classification results.

Images