A method for converting sign language into speech in real time

Abstract

The present invention discloses a method for converting sign language into speech in real time, including the steps of: first, using a camera to collect video data of a signer; second, using a frame difference method to identify and extract the video data of the signer to obtain key frames; third, inputting the key frames into an OpenPose system to extract the key point coordinates of the signer's body and hand; fourth, converting the coordinates into digital feature information; fifth, dividing the digital feature information into a training set and a verification set; sixth, constructing a neural network model, inputting the training set into the neural network model for training, and verifying and updating it through a verification set to obtain a trained neural network; seventh, converting the sign language recognition result of the trained neural network into speech output in real time. The method of the present invention has simple steps, small data calculation amount, good robustness, can be effectively applied in sign language translation, improves the recognition speed while meeting the recognition accuracy, has good use effect, and is easy to promote and use.

Description

Technical Field

[0001] The invention belongs to the technical field of sign language recognition, and in particular relates to a method for converting sign language into speech in real time. Background Art

[0002] Affected by factors such as congenital deafness or acquired deafness, the main way for deaf-mute people to communicate with hearing people is sign language. Sign language is a language for the deaf-mute people to communicate with the outside world. Most hearing people have never learned sign language, and there are huge obstacles in their daily communication with the deaf-mute people. This issue has attracted widespread attention from the society.

[0003] With the rapid development of computer science and technology, sign language recognition has become a research hotspot in cutting-edge computer technology. At present, the research in the field of sign language recognition is mainly divided into two types: traditional recognition methods and deep learning-based recognition methods. Traditional sign language feature optimization methods include oriented gradient histogram pyramid (HOG), Grassmann covariance matrix model (HGCM), etc. Traditional classification models include K nearest neighbor classification (KNN), support vector machine (SVM), relevance vector machine (RVM), etc. Traditional time series modeling models include dynamic time warping (DTW) and hidden Markov model (HMM). Most of these methods rely on manual extraction of hand shape and direction features, which takes a lot of time. Moreover, most methods are only applicable to the recognition of static sign language, and complex time series relationships need to be established to recognize dynamic sign language. These factors have led to the limited development of traditional sign language recognition methods, which are difficult to meet people's needs for sign language recognition.

[0004] At present, with the substantial improvement of computer performance, deep learning technology is gradually emerging. Sign language recognition based on deep learning can obtain more comprehensive and abstract spatial and temporal features by using powerful learning and fitting capabilities, and can automatically model the transformation and details of sign language movements, which can avoid the subjectivity and tediousness of artificial experience feature extraction, effectively improve the sign language recognition rate, and break through the limitations of traditional sign language recognition methods. For example, Li et al. used a pyramid BiLSTM network to find key sign language movements, effectively solved the problem of temporal modeling of sign language, and achieved a high degree of accuracy. Kopuklu et al. proposed to integrate the motion information of sign language videos into static sign language images, and used CNN networks to separate the fused features, achieving good recognition results. Chai et al. divided sign language movements into RGB data and skeleton data, and sent them to both ends of the dual-stream RNN network for score fusion, and won the first place in the 2016 Chalearn gesture recognition competition. However, these methods have not paid attention to the large amount of redundant information contained in the sign language video itself. Redundant information is disruptive, and when gesture occlusion occurs in sign language, the recognition accuracy is low, making sign language more difficult to recognize and classify. It also takes up a lot of storage and computing resources, making it difficult to further improve recognition accuracy while meeting speed requirements. This has become a problem that needs to be solved. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for real-time conversion of sign language into speech in view of the deficiencies in the above-mentioned prior art. The method has simple steps, small amount of data calculation, convenient implementation, good robustness, can be effectively applied in sign language translation, improves recognition speed while meeting recognition accuracy, has good use effect, and is easy to promote and use.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for converting sign language into speech in real time, comprising the following steps:

[0007] Step 1: Use a camera to collect video data of the sign language user;

[0008] Step 2: Using a frame difference method to identify and extract the video data of the signer to obtain key frames;

[0009] Step 3: Input the key frame into the OpenPose system to extract the key point coordinates of the signer's body and hands;

[0010] Step 4: converting the key point coordinates of the human body and hand into digital feature information;

[0011] Step 5: Divide the digital feature information into a training set and a verification set;

[0012] Step 6: construct a neural network model, input the training set into the neural network model for training, and verify and update it through the verification set to obtain a trained neural network;

[0013] Step 7: Convert the sign language recognition results of the trained neural network into speech output in real time.

[0014] In the above method for converting sign language into speech in real time, the camera in step 1 is a monocular camera.

[0015] In the above-mentioned method for converting sign language into speech in real time, the video data of the signer in step 1 includes the signer's body video data and hand video data.

[0016] In the above-mentioned method for converting sign language into speech in real time, the key frame in step 2 is the video frame containing the most sign language action information, the position information of each decomposed action in the sign language action can be represented by the key frame, and multiple key frames constitute the timing information of the sign language action.

[0017] In the above-mentioned method for converting sign language into speech in real time, the specific process of inputting the key frame into the OpenPose system in step 3 and extracting the key point coordinates of the signer's body and hand includes:

[0018] Step 301, selecting 14 key points of the upper body of the sign language user;

[0019] Step 302: Select 21 key points of the left hand and 21 key points of the right hand of the signer;

[0020] Step 303, representing each key point with a two-dimensional feature matrix;

[0021] Step 304: Set the origin and convert the absolute coordinate values ​​of the two-dimensional feature matrix into relative coordinate values.

[0022] In the above-mentioned method for converting sign language into speech in real time, the specific process of converting the key point coordinates of the human body and the hand into digital feature information in step 4 includes:

[0023] Step 401, compressing the two-dimensional feature matrix under relative coordinates into a one-dimensional vector;

[0024] Step 402: normalize each value in the one-dimensional vector to obtain a digital eigenvalue;

[0025] Step 403: Combining the digital feature values, key frame numbers and sign language numbers to form a digital feature matrix.

[0026] In the above method for real-time conversion of sign language into speech, the training set and the validation set described in step 5 are divided in a ratio of 3:1.

[0027] In the above-mentioned method for converting sign language into speech in real time, the neural network model in step six is ​​a six-layer network structure, and the six-layer network structure includes an LSTM module, a Dropout layer and two fully connected layers. The LSTM module includes three LSTM layers, and each LSTM layer adds a RELU function to avoid falling into a local optimal solution. The Dropout layer is used after the LSTM module to randomly discard neurons to avoid overfitting; the RELU function is added to the fully connected layer and normalized to avoid the problem of internal covariate offset and speed up the training.

[0028] In the above-mentioned method for converting sign language into speech in real time, the specific process of inputting the training set into the neural network model for training, and verifying and updating it through the verification set to obtain the trained neural network includes:

[0029] Step 601: randomly initialize all parameters and weight values ​​in the neural network model;

[0030] Step 602: input the training set into the input layer and LSTM module of the neural network model;

[0031] Step 603: Calculate the weight matrix Wf and offset vector bf of the LSTM forget gate, the weight matrix Wi and offset variable bi of the input gate, the weight matrix Wc and offset variable bc of the memory control unit, and the weight matrix Wo and offset variable bo of the output gate through back propagation; and use gradient descent to update all weights and parameter values;

[0032] Step 604: The features calculated by the LSTM module are randomly discarded through the Dropout layer and then passed to the fully connected layer for feature weighted fusion;

[0033] Step 605: After the last fully connected layer, the Softmax function is used to classify the action features, the classification results are compared with the validation set and the weight value of the neural network model is updated;

[0034] Step 606: Repeat steps 602 to 605 until the number of training rounds reaches the set epoch value to obtain a trained neural network.

[0035] In the above-mentioned method for converting sign language into speech in real time, the specific process of converting the sign language recognition result of the trained neural network into speech output in real time in step seven includes: converting the sign language sentences with correct sequence into speech for output through the speech library Pyttsx3.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] 1. The method of the present invention has simple steps, small amount of data calculation, and is easy to implement.

[0038] 2. The present invention extracts key frames through the frame difference method, eliminates redundant action information in video data, and reduces memory usage and computing resources.

[0039] 3. The present invention extracts the key point coordinates of the signer's body and hands by inputting key frames into the OpenPose system, and performs coordinate conversion, thereby further removing redundant image information and taking into account the human skeleton information, thereby improving the robustness of recognition and avoiding low recognition accuracy caused by gesture occlusion in sign language.

[0040] 4. The present invention uses a lightweight convolutional neural network for recognition and can perform inference at a rate of 30FPS, meeting the real-time requirements of sign language translation into speech.

[0041] 5. The present invention can be effectively applied in sign language translation, improving recognition speed while meeting recognition accuracy, has good use effect, and is easy to promote and use.

[0042] In summary, the method of the present invention has simple steps, small amount of data calculation, easy implementation, good robustness, can be effectively applied in sign language translation, improves recognition speed while meeting recognition accuracy, has good use effect, and is easy to promote and use.

[0043] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the method of the present invention;

[0045] Figure 2 It is a network structure diagram of the neural network model of the present invention. DETAILED DESCRIPTION

[0046] like Figure 1 As shown, the method for converting sign language into speech in real time of the present invention comprises:

[0047] Step 1: Use a camera to collect video data of the sign language user;

[0048] Step 2: using a frame difference method to identify and extract the video data of the signer to obtain key frames;

[0049] Eliminate redundant motion information in the video data through step 2 to reduce memory usage;

[0050] Step 3: Input the key frame into the OpenPose system to extract the key point coordinates of the signer's body and hands;

[0051] Step 4: converting the key point coordinates of the human body and hand into digital feature information;

[0052] Through steps three and four, redundant image information is further removed to improve the robustness of recognition; coordinate transformation can eliminate the influence of physical differences between different people and the distance from the human body to the camera.

[0053] Step 5: Divide the digital feature information into a training set and a verification set;

[0054] Step 6: construct a neural network model, input the training set into the neural network model for training, and verify and update it through the verification set to obtain a trained neural network;

[0055] Step 7: Convert the sign language recognition results of the trained neural network into speech output in real time.

[0056] In this embodiment, the camera in step 1 is a monocular camera.

[0057] In this embodiment, the video data of the signer in step 1 includes the signer's body video data and hand video data.

[0058] In this embodiment, the key frame in step 2 is the video frame containing the most information about the sign language action. The position information of each decomposed action in the sign language action can be represented by the key frame, and multiple key frames constitute the timing information of the sign language action.

[0059] In this embodiment, the specific process of inputting the key frame into the OpenPose system in step 3 and extracting the key point coordinates of the signer's body and hand includes:

[0060] Step 301, selecting 14 key points of the upper body of the sign language user;

[0061] Step 302: Select 21 key points of the left hand and 21 key points of the right hand of the signer;

[0062] Step 303, representing each key point with a two-dimensional feature matrix;

[0063] Step 304: Set the origin and convert the absolute coordinate values ​​of the two-dimensional feature matrix into relative coordinate values.

[0064] In the specific implementation, the key point 0 is set as the origin, and the expression of coordinate transformation is:

[0065]

[0066] in, represents the relative coordinates of the i-th key point, K x and K y Respectively represent the absolute x-axis coordinate and y-axis coordinate of the i-th key point, b x and b y They represent the x-coordinate and y-coordinate of key point 0 respectively.

[0067] In this embodiment, the specific process of converting the key point coordinates of the human body and the hand into digital feature information in step 4 includes:

[0068] Step 401, compressing the two-dimensional feature matrix under relative coordinates into a one-dimensional vector;

[0069] The specific implementation expression is:

[0070]

[0071] in, Represents the one-dimensional coordinate vector of the i-th key point, row i and column i Represents the row and column of the i-th key point in the relative coordinate matrix, and n represents the dimension of the key point, which is 2 here.

[0072] Step 402: normalize each value in the one-dimensional vector to obtain a digital eigenvalue;

[0073] The specific implementation expression is:

[0074]

[0075] in, represents the maximum and minimum normalized values ​​of the i-th key point, x i represents the value of the i-th key point, x max Represents the maximum value of the sample data, x min Indicates the minimum value of the sample data.

[0076] Step 403: Combining the digital feature values, key frame numbers and sign language numbers to form a digital feature matrix.

[0077] In the specific implementation, 56 key points are extracted from each key frame, and a sign language action has a total of 56*n key points, where n is the number of key frames extracted, forming a (56+ID)×n digital feature matrix, where ID represents the sign language number, and the digital feature matrix is ​​output in the npy data format.

[0078] In this embodiment, the training set and the validation set described in step 5 are divided in a ratio of 3:1.

[0079] In this embodiment, Figure 2 As shown, the neural network model in step six is ​​a six-layer network structure, which includes an LSTM module, a Dropout layer and two fully connected layers. The LSTM module includes three LSTM layers, and each LSTM layer adds a RELU function to avoid falling into a local optimal solution. The Dropout layer is used after the LSTM module to randomly discard neurons to avoid overfitting; the RELU function is added to the fully connected layer and normalized to avoid the problem of internal covariate offset and speed up the training.

[0080] In the specific implementation, in order to prevent the distribution of inputs of each layer in the middle of the network from being greatly different each time the parameters are updated when using stochastic gradient descent to update the parameters, the RELU function is added after the convolution and normalized to avoid the problem of internal covariate shift and speed up the training.

[0081] In this embodiment, the specific process of inputting the training set into the neural network model for training in step 6, and verifying and updating it through the verification set to obtain the trained neural network includes:

[0082] Step 601: randomly initialize all parameters and weight values ​​in the neural network model;

[0083] Step 602: input the training set into the input layer and LSTM module of the neural network model;

[0084] Step 603: Calculate the weight matrix Wf and offset vector bf of the LSTM forget gate, the weight matrix Wi and offset variable bi of the input gate, the weight matrix Wc and offset variable bc of the memory control unit, and the weight matrix Wo and offset variable bo of the output gate through back propagation; and use gradient descent to update all weights and parameter values;

[0085] Step 604: The features calculated by the LSTM module are randomly discarded through the Dropout layer and then passed to the fully connected layer for feature weighted fusion;

[0086] Step 605: After the last fully connected layer, the Softmax function is used to classify the action features, the classification results are compared with the validation set and the weight value of the neural network model is updated;

[0087] Step 606: Repeat steps 602 to 605 until the number of training rounds reaches the set epoch value to obtain a trained neural network.

[0088] In this embodiment, the specific process of converting the sign language recognition results of the trained neural network into voice output in real time in step seven includes: converting the sign language sentences with correct sequence into voice output through the voice library Pyttsx3.

[0089] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for converting sign language into speech in real time, characterized in that: The following steps are involved: Step 1: Use a camera to collect video data of the sign language user; Step 2: using a frame difference method to identify and extract the video data of the signer to obtain key frames; Step 3: Input the key frame into the OpenPose system to extract the key point coordinates of the signer's body and hands; Step 4: converting the key point coordinates of the human body and hand into digital feature information; The specific process of converting the key point coordinates of the human body and hand into digital feature information in step 4 includes: Step 401, compressing the two-dimensional feature matrix under relative coordinates into a one-dimensional vector; Step 402: normalize each value in the one-dimensional vector to obtain a digital eigenvalue; Step 403, combining the digital feature values, key frame numbers and sign language numbers to form a digital feature matrix; Step 5: Divide the digital feature information into a training set and a verification set; Step 6: construct a neural network model, input the training set into the neural network model for training, and verify and update it through the verification set to obtain a trained neural network; The neural network model in step 6 is a six-layer network structure, which includes an LSTM module, a Dropout layer and two fully connected layers. The LSTM module includes three LSTM layers. Each LSTM layer adds a RELU function to avoid falling into a local optimal solution. The Dropout layer is used after the LSTM module to randomly discard neurons to avoid overfitting; the RELU function is added to the fully connected layer and normalized to avoid the problem of internal covariate offset and speed up the training; Step 7: Convert the sign language recognition results of the trained neural network into speech output in real time.

2. A method for real-time conversion of sign language into speech according to claim 1, characterized in that: The camera described in step 1 is a monocular camera.

3. A method for real-time conversion of sign language into speech according to claim 1, characterized in that: The video data of the signer in step 1 includes the signer's body video data and hand video data.

4. A method for real-time conversion of sign language into speech according to claim 1, characterized in that: The key frame in step 2 is the video frame containing the most information about the sign language action. The position information of each decomposed action in the sign language action can be represented by the key frame, and a plurality of the key frames constitute the timing information of the sign language action.

5. A method for real-time conversion of sign language into speech according to claim 1, characterized in that: The specific process of inputting the key frame into the OpenPose system and extracting the key point coordinates of the signer's body and hand as described in step 3 includes: Step 301, selecting 14 key points of the upper body of the sign language user; Step 302: Select 21 key points of the left hand and 21 key points of the right hand of the signer; Step 303, representing each key point with a two-dimensional feature matrix; Step 304: Set the origin and convert the absolute coordinate values ​​of the two-dimensional feature matrix into relative coordinate values.

6. A method for real-time conversion of sign language into speech according to claim 1, characterized in that: The training set and validation set described in step 5 are divided in a ratio of 3:

1.

7. A method for real-time conversion of sign language into speech according to claim 1, characterized in that: In step 6, the training set is input into the neural network model for training, and the validation set is used for validation and updating. The specific process of obtaining the trained neural network includes: Step 601: randomly initialize all parameters and weight values ​​in the neural network model; Step 602: input the training set into the input layer and LSTM module of the neural network model; Step 603: Calculate the weight matrix Wf and offset vector bf of the LSTM forget gate, the weight matrix Wi and offset variable bi of the input gate, the weight matrix Wc and offset variable bc of the memory control unit, and the weight matrix Wo and offset variable bo of the output gate through back propagation; and use gradient descent to update all weights and parameter values; Step 604: The features calculated by the LSTM module are randomly discarded through the Dropout layer and then passed to the fully connected layer for feature weighted fusion; Step 605: After the last fully connected layer, the Softmax function is used to classify the action features, the classification results are compared with the validation set and the weight value of the neural network model is updated; Step 606: Repeat steps 602 to 605 until the number of training rounds reaches the set epoch value to obtain a trained neural network.

8. A method for real-time conversion of sign language into speech according to claim 7, characterized in that: The specific process of converting the sign language recognition results of the trained neural network into speech output in real time as described in step 7 includes: converting the sign language sentences with correct sequence into speech output through the speech library Pyttsx3.

Images