A sign language recognition method and system based on LSTM and NLP

Abstract

The application belongs to the technical field of sign language recognition, and discloses a sign language recognition method and system based on LSTM and NLP, which contains two stages of model training and sign language recognition. In the model training stage, a timing key point sequence sample is obtained by using a sign language word unit data set and a gesture detection model, an LSTM-SL model is trained, a sign language representation four tuple is obtained by using sign language common corpus word segmentation, and a W2V-SL model is trained. The sign language recognition stage includes converting a sign language video to be recognized or a real-time sign language picture into a sign language word unit prediction sequence by using the LSTM-SL, outputting a sign language sentence conforming to a language specification by using a sliding window mechanism and the W2V-SL model, and proposing a sign language recognition system based on the method. The application simplifies the collection difficulty of timing characteristics of sign language recognition, effectively enhances the output effect of natural sentences, and improves the robustness and accuracy of the sign language recognition system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of sign language recognition, and particularly relates to a sign language recognition method and system based on LSTM and NLP. BACKGROUND

[0002] Sign language is the main communication tool used by the hearing impaired, and is a visual spatial language spontaneously created by the hearing impaired in their communication with each other. However, there are barriers to communication with others in today's society. Sign language recognition tools are a powerful means to help the hearing impaired integrate into the environment and enhance their sense of social belonging and self-worth.

[0003] Current sign language recognition technologies mainly fall into two categories: sign language recognition based on data gloves and sign language recognition based on vision. Sign language recognition systems based on data gloves can achieve accurate sign language recognition, but they require the use of heavy sensing devices, which is very inconvenient in actual production and life. In today's high-speed development of information technology, the quality of data collected by digital cameras has been guaranteed, so sign language recognition systems based on vision can achieve the same high-quality results while ensuring practicability.

[0004] According to the recognition object of the system, the sign language recognition task can be classified into isolated word sign language recognition and continuous sign language recognition. In a complex natural language system, a word is the basic unit of semantic expression. Isolated word sign language recognition is to recognize individual independent words, which is a discrete recognition method. Continuous sign language recognition is to recognize a complete sentence, which contains the context relationship between text sequences. Isolated word recognition focuses on the accuracy of model recognition gestures, while continuous recognition focuses more on the feature extraction method of video frame sequences and the context relationship of text. Combining the two can achieve better recognition results.

[0005] For the output of continuous sentences, most current methods extract the most probable words and then form sentences, which may have a large deviation in meaning when facing long and difficult sentences. Word embedding is a context-based model that quantifies the relationship between words in the form of word vectors, which can well calculate the semantic distance between isolated word sequences and language models in the corpus, so as to search for the most contextually appropriate continuous sign language sentence. The existing continuous sign language recognition system has not yet researched the technology of combining continuous sign language recognition and semantic distance measurement. SUMMARY

[0006] In order to overcome the above technical defects, the application provides a sign language recognition method and system based on LSTM and NLP, which is used for recognizing sign language vocabulary sequence from sign language video or real-time sign language picture and outputting sign language sentence conforming to language specification through natural language generation technology, mainly including two stages of model training and sign language recognition. In the model training stage, the timing key point sequence sample is obtained by using the sign language vocabulary data set and the posture detection model, the LSTM-SL model is trained, the sign language representation four-tuple is obtained by using the sign language commonly used corpus segmentation, and the W2V-SL model is trained; the sign language recognition stage includes converting the sign language video or real-time sign language picture to be recognized into sign language vocabulary prediction sequence through LSTM-SL, and outputting sign language sentence conforming to language specification by using sliding window mechanism and W2V-SL model.

[0007] In order to achieve the above purpose, the application is realized by the following technical scheme:

[0008] The application is a sign language recognition method and system based on LSTM and NLP, which includes the following steps:

[0009] The model training stage:

[0010] (1) Given a sign language vocabulary data set D={d1, d2, …, d n}, wherein d i ={vedio i , word i}, vedio i represents the sign language action video corresponding to the sign language vocabulary d i , and word i represents the natural language vocabulary corresponding to the sign language vocabulary. Each sign language vocabulary d i in D is converted into timing key point sequence data t i , thereby generating timing key point sequence sample data set T={t1, t2, …, t n} corresponding to the original sign language vocabulary data set D.

[0011] The specific steps are as follows:

[0012] (1a) The sign language video vedio i in d i is extracted to generate a frame picture sequence corresponding to the sign language vocabulary d i , that is, PIC i ={pic i,1 , pic i,2 , …, pic i,k}, wherein pic i,j is the jth frame picture of d i .

[0013] (1b) Using the sign language gesture key point detection model, i.e., the model provided by the gesture detection tool provided by MediaPipe, from PIC i For each image in the dataset, key points representing the sign language gesture features corresponding to sign language words are extracted, and a model is constructed based on the PIC. i The corresponding time-series key point sequence data P i ={kp i,1 ,kp i,2 ,…,kp i,k}, where kp i,j The tensor represents the coordinates of the key points of the sign language gesture in the j-th frame image.

[0014] (1c) is a sign language word d i word in i Construct a one-hot code tensor G relative to D i Calculate according to formula (2-1):

[0015]

[0016] (1d) is the sign language word d i Construct the corresponding time-series key point sequence data t i , represented as t i ={P i G i}

[0017] (2) Using the temporal key point sequence sample dataset T as input, train the LSTM neural network model to generate the LSTM-SL sign language word recognition model based on LSTM.

[0018] (3) Given a corpus of commonly used sign language phrases constructed from several sign language text sets, K = {k1, k2, ..., k n}, where k i For a text containing several sign language lexical units, calculate the sample dataset Q of sign language lexical unit representation quadruples based on the commonly used sign language corpus K. The specific steps are as follows:

[0019] (3a) Use word segmentation technology to extract each text k from K. i The lexical structure in the set NW = {word} corresponds to the sign language-related natural language lexical set NW. i |word i ∈K}; Calculate each sign language word in NW i Index position idx in K i and frequency of occurrence freq i Then, construct a sequence P of lexical-frequency triples, where P[i] = (idx) i ,word i,freq i ).

[0020] (3b) Taking the sequence of word-frequency triples P as input, for each element P[i] = (idx) i ,word i ,freq i ), based on its frequency (freq) i Calculate the corresponding sign language word i The weights are calculated according to formula (2-2):

[0021]

[0022] Where α is the smoothing constant, k is the weight coefficient, and SW is the set of sign language-related natural language lexicons. Formula (1-2) can change the weight values ​​of lexicons in the set of sign language-related natural language lexicons SW, increasing the coupling between sign language-related lexicons and the sign language lexicon recognition model LSTM-SL. Finally, a sample dataset Q of sign language lexicon representation quadruples corresponding to P is generated, where Q[i]=(idx i ,word i ,freq i weight i ).

[0023] (4) Using the sign language word representation quadruple sequence sample dataset Q as input, train the word2vec model to generate the sign language sentence natural language generation model W2V-SL based on word2vec.

[0024] Sign language recognition stage:

[0025] (1) For the sign language video or real-time sign language footage to be identified, convert the continuous sign language actions into the corresponding sign language word temporal key point sequence T. w ={tw1,tw2,…,tw n The input is fed into the LSTM-SL model to generate a sign language lexical prediction sequence S = (s1, s2, ..., s...). n ), where s i This represents the sign language lexical units predicted by the model. The specific steps are as follows:

[0026] (1a) The continuous sign language movements in the sign language video or real-time sign language screen to be identified are converted into the corresponding sign language lexical temporal key point sequence T at fixed frame intervals k using a sign language posture key point detection model (as described in step (2) of the model training stage). w ={tw1,tw2,…,tw n}, T w Substituting into the LSTM-SL model, we obtain the sign language word prediction sequence S.

[0027] (1b) In practical applications, the speed of sign language varies from person to person. For the sign language actions to be recognized, assuming the length of a sign language word is m frames and k < m, after dividing the sign language tokens with a fixed frame number k as the length, the obtained sign language token prediction sequence S has a length greater than the length of the sign language sentence represented in the actual sign language video, so as to obtain more candidate sign language tokens and ensure that the semantics are not omitted.

[0028] (2) Optimize the semantics of the sign language token prediction sequence S using a sliding window mechanism to generate a sign language prediction sequence S w =(s1, s2, …, s k ), specifically as follows:

[0029] (2a) For the sign language token s n in the sign language token prediction sequence S=(s1, s2, …, s i ), during the process of combining the sign language token sequence into a fluent sign language sentence, use a sliding window to slide in the prediction sequence S, and combine the sign language tokens in each window to form a sentence. The specific steps are as follows:

[0030] (2a1) Initialize the sliding window length to h, the sliding step size to 1, and the sliding sequence to the sign language token prediction sequence S=(s1, s2, …, s n ).

[0031] (2a2) If the length of the prediction sequence is less than h, that is, n < h, then directly output the sign language token prediction sequence S w =S. Otherwise, move the sliding window.

[0032] (2a3) When there are repeated predictions for the sign language token s i in the current window, only retain the first token and move the sliding window. After the sliding is completed, obtain the sign language prediction sequence S w =(s1, s2, …, s k ), where k ≤ n.

[0033] (3) Convert the sign language prediction sequence S w =(s1, s2, …, s k ) into a sentence vector V, input it into the word embedding model W2V - SL, and output the sign language sentence that conforms to the language specification and has the highest fluency by comparing with the similarity of common sign language sentences. The specific steps are as follows:

[0034] (3a) For each sign language token s w in the sign language prediction sequence S<0000​​​​​​), indicating that the word element s i The vector corresponding to the W2V-SL model when the center word is , where word2vec() is an algorithm used to convert sign language words into vectors. According to formula (2-3), the sentence vector V of the sign language prediction sequence is generated:

[0035]

[0036] (3b) For commonly used sign language phrases, establish a corpus E of commonly used sign language phrases, and calculate the sentence vector e of each phrase in E. i According to formula (2-4), calculate V and the expression e in the corpus of commonly used sign language phrases. i Cosine similarity:

[0037]

[0038] Where V i ,e i These represent the component vectors of vectors V and e, respectively. The sentence with the highest cosine similarity is output as the final fluent and linguistically correct sign language sentence.

[0039] The beneficial effects of this invention are: it can extract the movement information of sign language interpreters through a camera and a posture detection model, and process and train the collected data through a deep learning model.

[0040] This invention also incorporates a natural language generation model to improve the semantic content of the prediction during the sign language prediction process. It can accurately identify the semantics of the input sign language actions and generate sign language sentences that conform to language norms.

[0041] This invention simplifies the acquisition of temporal features in sign language recognition, enhances the final effect of sign language text, and makes the sign language recognition system more robust, faster in detection, and more accurate. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of a sign language gesture key point detection model.

[0043] Figure 2 This is a schematic diagram of the LSTM-SL neural network structure.

[0044] Figure 3 This is a diagram illustrating the overall framework of a sign language recognition system based on LSTM and NLP. Detailed Implementation

[0045] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0046] like Figure 3 As shown, this invention proposes a sign language recognition system based on LSTM and NLP. This system includes a two-stage processing procedure, specifically comprising:

[0047] I. Model Training Subsystem:

[0048] (1) Data acquisition module: Acquires sign language video or given sign language video read through camera.

[0049] (2) Sign language meta-dataset generation module: The sign language video is used to express the meaning of a specific independent sign language phrase with a fixed frame length as an information point. A sign language phrase is collected independently, and the collected sign language video is divided into a supervised form of sign language meta-dataset using the independent sign language phrase as the segmentation condition.

[0050] (3) Keypoint Extraction Module: Input sign language video images are processed at fixed frame intervals (k) using sign language posture keypoint detection models such as MediaPipe-holistic to convert them into corresponding image keypoints. Specifically, this includes 34 torso keypoints and 21 keypoints for each of the left and right hands. The detection results are as follows: Figure 1 As shown.

[0051] (4) Temporal Keypoint Sequence Generation Module: Through the keypoint extraction module, a k-frame keypoint sequence is obtained, P i ={kp i,1 ,kp i,2 ,…,kp i,k}, where kp i,j The tensor representing the coordinates of the sign language gesture keypoints in the j-th frame image is denoted by the sign language lexicon d. i word in i Construct a one-hot code tensor G relative to D i Generate the final time-series key point sequence data t. i ={P i G i} and the temporal key point sequence sample dataset T={t1,t2,…,t n}

[0052] (5) LSTM Model Architecture Module: Using the temporal keypoint sequence sample dataset T as input, an LSTM neural network model is trained (LSTM-LS). Since spatial modeling of the data has already been performed during pose detection, a Long Short-Term Memory (LSTM) neural network is directly used. Three gates establish a self-loop for the internal state of the LSTM neural network unit to analyze more diverse information over a time span. By extracting sequence features, the contextual relationship between individual sign language phrase frames is obtained, thus acquiring the semantic features of the sign language text. ReLU is selected as the activation function to eliminate linearity. The semantic features of the sign language sequence are connected in a pre-defined dense network (DenseNet). The model architecture is as follows: Figure 2 As shown.

[0053] (6) Sign Language Feature Quadruple Construction Module: Given a commonly used sign language corpus K constructed from several sign language text sets, use word segmentation technology to extract word units from each text in K to construct a sign language-related natural language word set NW = {word i |word i ∈K}; Calculate each sign language word in NW i Index position idx in K i and frequency of occurrence (freq) i Then, construct a sequence P of lexical-frequency triples, where P[i] = (idx) i ,word i ,freq i Each lexical unit is weighted according to its frequency of occurrence. Lexical units that appear in the LSTM-SL training are deweighted to ensure that they have an advantage in the downsampling during W2V-SL training, making it easier for the prediction results to hit the lexical units trained in LSTM-SL. Finally, the sign language lexical unit representation quadruple sequence sample dataset Q is obtained.

[0054] (7) W2V-SL training module: Input the sample dataset Q of sign language lexical representation quadruple sequence to train the word embedding language model W2V-SL.

[0055] II. Sign Language Recognition Subsystem

[0056] The sign language recognition subsystem includes:

[0057] 1. Preparation stage: Import video or real-time camera footage, extract key skeletal points of the human body using a pose detection tool, and generate the temporal key point sequence T of the sign language lexical units to be predicted. w ={tw1,tw2,…,tw n}

[0058] 2. Operational Phase:

[0059] (1) LSTM-SL prediction module: Substitute T w into the LSTM-SL model to obtain the sign language token prediction sequence S. Since the sign language speed varies among individuals in actual applications, for the sign language actions to be recognized, assuming the length of the sign language word is m frames and k < m, after dividing the sign language tokens with a fixed frame number k as the length, the length of the obtained sign language token prediction sequence S is greater than the length of the sign language sentence represented in the actual sign language video, thus obtaining more candidate sign language tokens to ensure that the semantics are not omitted. This will cause problems such as repeated recognition and misjudgment of sign language tokens. At the same time, considering the human-computer interaction of this sign language recognition system, the system will display the predicted tokens to the user in real time. For inaccurate recognition, the user can re-recognize by adjusting the sign language actions to increase the prediction accuracy. This will also cause problems such as repeated recognition and misjudgment of sign language tokens. To solve the above common problems, for the successfully recognized tokens, a sliding window module is established as follows:

[0060] (2) Sliding window module: When the sign language tokens in the current window are predicted to be repeated, only the first token is retained and the sliding window is moved. In this example, the length of the sliding window is set to g, that is, the sliding window will be used to detect every g tokens. After sliding, the sign language prediction sequence S w is obtained.

[0061] (3) W2V-SL prediction module: Input the sign language prediction sequence S w into the word embedding model W2V-SL and convert it into a sentence vector V.

[0062] (4) Similarity matching module: By comparing the similarity of the sentence vector V with the sentence vectors of common sign language sentences, such as comparing the cosine similarity, the group with the highest similarity degree is selected for matching, and finally the sign language sentence that conforms to the language specification is output.

[0063] Based on the sign language recognition system provided by the present invention, the sign language recognition method of the present invention is run. This sign language recognition method is used to recognize the sign language vocabulary sequence from a sign language video or a real-time sign language picture and output a sign language sentence that conforms to the language specification through natural language generation technology. Specifically, it includes the following steps:

[0064] Step 1. Model training stage, specifically including the following steps:

[0065] Step 1-1. Given a sign language token data set D = {d1, d2,..., d n}, where d i represents a sign language token, d i = {vedio i , word i}, vedio i represents the sign language token d iCorresponding sign language video, Word document i Representing the natural language lexicon corresponding to a sign language lexicon, each sign language lexicon d in the defined sign language lexicon dataset D. i Convert to time-series key point sequence data t i This generates a temporal keypoint sequence sample dataset T = {t1, t2, ..., t} corresponding to the original sign language lexical dataset D. n}

[0066] For sign language morpheme d i ={vedio i ,word i} Convert to time-series key point sequence data t i The details are as follows:

[0067] Step 1-1-1: Transfer the sign language word d i Sign language video in China i Extract a sequence of images with a fixed number of frames k and generate a sequence of sign language morphemes d. i The corresponding frame image sequence representation, i.e., PIC i ={pic i,1 ,pic i,2 ,…,pic i,k}, where pic i,j For d i The j-th frame image;

[0068] Step 1-1-2: Utilize the sign language gesture key point detection model to detect key points from PIC. i For each image in the dataset, key points representing the sign language gesture features corresponding to sign language words are extracted, and a model is constructed based on the PIC. i The corresponding time-series key point sequence data P i ={kp i,1 ,kp i,2 ,…,kp i,k}, where kp i,j The tensor representing the coordinates of key points of the sign language gesture in the j-th frame image;

[0069] Step 1-1-3, for sign language word d i word in i Construct a one-hot code tensor G relative to a given sign language lexical dataset D. i Calculate according to formula (1-1):

[0070]

[0071] Step 1-1-4, for sign language word d i Construct the corresponding time-series key point sequence data t i , represented as t i={P i G i}

[0072] Step 1-2: Using the temporal key point sequence sample dataset T obtained in Step 1-1 as input, train the LSTM neural network model to generate the LSTM-SL sign language word recognition model based on LSTM.

[0073] Steps 1-3: Given a corpus of commonly used sign language phrases K = {k1, k2, ..., kk} constructed from several sign language text sets. n}, where k i For a text containing several sign language lexical units, calculate the sample dataset Q of sign language lexical unit representation quadruples based on the commonly used sign language corpus K, specifically as follows:

[0074] Step 1-3-1: Extract each text k from the commonly used sign language corpus K using word segmentation technology. i The lexical structure in the corpus K corresponds to the sign language-related natural language lexical set NW = {word i |word i ∈K}; Calculate the sign language word in the natural language lexicon set NW related to sign language. i Index position idx in the commonly used sign language corpus K i and frequency of occurrence freq i Then, construct a sequence P of lexical-frequency triples, where P[i] = (idx) i ,word i ,freq i );

[0075] Step 1-3-2: Using the word-frequency triplet sequence P constructed in Step 1-3-1 as input, for each element in P, P[i] = (idx i ,word i ,freq i ), based on its frequency (freq) i Calculate the corresponding sign language word i The weights are calculated according to formula (1-2):

[0076]

[0077] Where α is the smoothing constant, k is the weight coefficient, and SW is the set of sign language related natural language lexicons. Formula (1-2) can change the weight values ​​of lexicons in the set of sign language related natural language lexicons SW, increase the coupling degree between sign language related lexicons and the sign language lexicon recognition model LSTM-SL, and finally generate a sample dataset Q of sign language lexicon representation quadruples corresponding to P, where Q[i]=(idx i ,word i,freq i ,weight i ).

[0078] Steps 1 - 4: Using the sign language word element representation quadruple sequence sample dataset Q obtained in Steps 1 - 3 as input, train the word2vec model to generate a sign language sentence natural language generation model W2V - SL based on word2vec.

[0079] Step 2: Sign language recognition stage, specifically including the following steps:

[0080] Step 2 - 1: For the sign language video or real - time sign language picture to be recognized, convert the continuous sign language actions into a corresponding sign language word element time - series key - point sequence T w ={tw1, tw2, …, tw n}, and input it into the sign language word element recognition model LSTM - SL in Step 1 - 2 to generate a sign language word element prediction sequence S=(s1, s2, …, s n ), where s i represents the sign language word element predicted by the model.

[0081] Generating the sign language word element prediction sequence S specifically includes:

[0082] Step 2 - 1 - 1: For the continuous sign language actions in the sign language video or real - time sign language picture to be recognized, at an interval of a fixed frame number length k, convert them into a corresponding sign language word element time - series key - point sequence T w ={tw1, tw2, …, tw n}, substitute T w into the LSTM - SL model to obtain the sign language word element prediction sequence S;

[0083] Step 2 - 1 - 2: For the sign language action to be recognized, assuming the length of the sign language word is m frames and taking k < m, after dividing the sign language word elements with a fixed frame number k as the length, the length of the obtained sign language word element prediction sequence S is greater than the length of the sign language sentence represented in the actual sign language video, so as to obtain more candidate sign language word elements and ensure that the semantics are not omitted.

[0084] Step 2 - 2: Use the sliding window mechanism to optimize the semantics of the sign language word element prediction sequence S to generate a sign language prediction sequence S w =(s1, s2, …, s k ).

[0085] Among them, optimizing the semantics of the sign language word element prediction sequence S using the sliding window is specifically as follows: For the sign language word element s n in the sign language word element prediction sequence S=(s1, s2, …, s i, during the process of combining sign language token sequences into fluent sign language sentences, a sliding window is used to slide in the prediction sequence S, and sign language tokens are combined into sentences in each window, which specifically includes the following steps:

[0086] Step 2-2-1: Initialize the sliding window length to h, the sliding step size to 1, and the sliding sequence to the sign language token prediction sequence S = (s1, s2, …, s n );

[0087] Step 2-2-2: If the length of the prediction sequence is less than h, that is, n < h, then directly output the sign language token prediction sequence S w = S, otherwise, move the sliding window;

[0088] Step 2-2-3: When there are repeated predictions for the sign language token s i in the current window, only retain the first token and move the sliding window. After the sliding is completed, obtain the sign language prediction sequence S w = (s1, s2, …, s k ), where k ≤ n.

[0089] Step 2-3: Convert the sign language prediction sequence S w = (s1, s2, …, s k ) generated in Step 2-2 into a sentence vector V, input it into the word embedding model W2V-SL, and output the sign language sentence that meets the language norms with the highest fluency through comparison with the similarity of common sign language sentences, which specifically includes the following steps:

[0090] Step 2-3-1: For each sign language token s w = (s1, s2, …, s k ) in the sign language prediction sequence S, calculate the corresponding word vector v i = word2vec(s i ), which represents the vector corresponding to the W2V-SL model when the token s i is the central word. Here, word2vec() is the algorithm used to convert sign language tokens into vectors. According to formula (1-3), generate the sentence vector V of the sign language prediction sequence: i

[0091]

[0092] Step 2-3-2: For common sign language sentences, establish a sign language common sentence corpus E, calculate the sentence vector e[[ID=—]] i of each sentence in E, and calculate the cosine similarity between V and the sentence e i in the sign language common sentence corpus according to formula (1-4):

[0093]

[0094] Where V i ,e i These represent the component vectors of vectors V and e, respectively. The sentence with the highest cosine similarity is output as the final fluent and linguistically correct sign language sentence.

[0095] This invention simplifies the acquisition of temporal features in sign language recognition, effectively enhances the output effect of natural language sentences, and improves the robustness and accuracy of the sign language recognition system.

[0096] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A sign language recognition method based on LSTM and NLP, used to identify sign language word sequences from sign language videos or real-time sign language footage, and output sign language sentences conforming to language norms through natural language generation technology, characterized in that: The sign language recognition method includes the following steps: Step 1, the model training stage, specifically including the following steps: Step 1-1: Given a sign language lexical dataset D = {d1, d2, ..., d...} n }, where d i Indicates sign language motif, d i ={vedio i ,word i }, vedio i The sign language word d i Corresponding sign language video, Word document i Representing the natural language lexicon corresponding to a sign language lexicon, each sign language lexicon d in the defined sign language lexicon dataset D. i Convert to time-series key point sequence data t i This generates a temporal keypoint sequence sample dataset T = {t1, t2, ..., t} corresponding to the original sign language lexical dataset D. n }; Step 1-2, taking the time series key point sequence sample data set T obtained in Step 1-1 as the input, training the LSTM neural network model to generate the sign language token recognition model LSTM-SL based on LSTM; Steps 1-3: Given a corpus of commonly used sign language phrases K = {k1, k2, ..., kk} constructed from several sign language text sets. n }, where k i For a text containing several sign language lexical units, calculate the sample dataset Q of sign language lexical unit representation quadruples based on the commonly used sign language corpus K. The specific steps include the following: Step 1-3-1: Extract each text k from the commonly used sign language corpus K using word segmentation technology. i The lexical structure in the corpus K corresponds to the sign language-related natural language lexical set NW = {word i |word i ∈K}; Calculate the sign language word in the natural language lexicon set NW related to sign language. i Index position idx in the commonly used sign language corpus K i and frequency of occurrence freq i Then, construct a sequence P of lexical-frequency triples, where P[i] = (idx) i ,word i ,freq i ); Step 1-3-2: Using the word-frequency triplet sequence P constructed in Step 1-3-1 as input, for each element in P, P[i] = (idx i ,word i ,freq i ), based on its frequency (freq) i Calculate the corresponding sign language word i The weights are calculated according to the following formula: Where α is the smoothing constant, k is the weighting coefficient, and SW is the set of natural language lexical units related to sign language. Finally, a dataset Q of four-tuple sequence samples of sign language lexical representations corresponding to P is generated, where Q[i] = (idx) i ,word i ,freq i weight i ); Step 1-4, taking the sign language token representation quadruple sequence sample data set Q obtained in Step 1-3 as the input, training the word2vec model to generate the sign language sentence natural language generation model W2V-SL based on word2vec; Step 2, the sign language recognition stage, specifically including the following steps: Step 2-1: For the sign language video or real-time sign language footage to be recognized, convert the continuous sign language actions into the corresponding sign language lexical temporal key point sequence T. w ={tw1,tw2,…,tw n The input is fed into the LSTM-SL sign language lexical recognition model in steps 1-2 to generate a sign language lexical prediction sequence S = (s1, s2, ..., s...). n ), where s i This represents the sign language lexical units predicted by the model; Step 2-2: Optimize the semantics of the sign language lexical prediction sequence S using the sliding window mechanism to generate the sign language prediction sequence S. w =(s1,s2,…,s k The semantic optimization of the sign language lexical prediction sequence S using a sliding window is specifically as follows: For the sign language lexical prediction sequence S = (s1, s2, ..., s...), ... n The sign language morpheme s in ) i In the process of combining sign language word sequences into fluent sign language sentences, a sliding window is used to slide across the prediction sequence S, combining sign language words into sentences within each window. Specifically, the steps include: Step 2-2-1: Initialize the sliding window length to h, the sliding step size to 1, and the sliding sequence to be the sign language word prediction sequence S = (s1, s2, ..., s...). n ); Step 2-2-2: If the predicted sequence length is less than h, i.e., n < h, then directly output the sign language token prediction sequence S w = S, otherwise, move the sliding window; Step 2-2-3, sign language morphemes s in the current window i When duplicate predictions occur, only the first word is retained. The sliding window is moved, and after the sliding is complete, the sign language prediction sequence S is obtained. w =(s1,s2,…,s k ), where k≤n; Step 2-3: The sign language prediction sequence S generated in step 2-2... w =(s1,s2,…,s k The vector is transformed into a sentence vector V, which is then input into the word embedding model W2V-SL. By comparing the similarity with commonly used sign language phrases, the model outputs the sign language phrase with the highest fluency that conforms to language norms.

2. The sign language recognition method based on LSTM and NLP according to claim 1, characterized in that: In step 1-1, for the sign language word d i ={vedio i ,word i } Convert to time-series key point sequence data t i The specific steps are as follows: Step 1-1-1: Transfer the sign language word d i Sign language video in China i Extract a sequence of images with a fixed number of frames k and generate a sequence of sign language morphemes d. i The corresponding frame image sequence representation, i.e., PIC i ={pic i,1 ,pic i,2 ,…,pic i,k }, where pic i,j For d i The j-th frame image; Step 1-1-2: Utilize the sign language gesture key point detection model to detect key points from PIC. i For each image in the dataset, key points representing the sign language gesture features corresponding to sign language words are extracted, and a model is constructed based on the PIC. i The corresponding time-series key point sequence data P i ={kp i,1 ,kp i,2 ,…,kp i,k }, where kp i,j The tensor representing the coordinates of key points of the sign language gesture in the j-th frame image; Step 1-1-3, for sign language word d i word in i Construct a one-hot code tensor G relative to a given sign language lexical dataset D. i Calculate according to the following formula: Step 1-1-4, for sign language word d i Construct the corresponding time-series key point sequence data t i , represented as t i ={P i G i } 3. The sign language recognition method based on LSTM and NLP according to claim 1, characterized in that: The generation of the sign language token prediction sequence S in Step 2-1 specifically includes the following steps: Step 2-1-1: Convert the continuous sign language movements in the sign language video or real-time sign language footage to be recognized into the corresponding sign language lexical temporal key point sequence T at fixed frame intervals of length k using a sign language posture key point detection model. w ={tw1,tw2,…,tw n }, T w Substituting into the LSTM-SL model, we obtain the sign language word prediction sequence S; Step 2-1-2, for the sign language action to be recognized, assuming that the length of the sign language word is m frames and k < m, after dividing the sign language tokens with a fixed frame number k as the length, the length of the obtained sign language token prediction sequence S is greater than the length of the sign language sentence represented in the actual sign language video, so as to obtain more candidate sign language tokens and ensure that the semantics are not omitted.

4. The sign language recognition method based on LSTM and NLP according to claim 1, characterized in that: In steps 2-3, the sign language prediction sequence S w =(s1,s2,…,s k Input a W2V-SL model and output fluent, language-compliant natural language sentences. The specific steps include the following: Step 2-3-1: For the sign language prediction sequence S w =(s1,s2,…,s k Each sign language morpheme s i Calculate the corresponding word vector v i =word2vec(s i ), indicating that the word element s i The vector corresponding to the W2V-SL model when the center word is used, where word2vec() is an algorithm used to convert sign language words into vectors. The sentence vector V of the sign language prediction sequence is generated according to the following formula: Step 2-3-2: For commonly used sign language phrases, establish a corpus E of commonly used sign language phrases, and calculate the sentence vector e of each phrase in E. i The sentence vector V is calculated according to the following formula, and the sentence e in the corpus of commonly used sign language phrases is compared with the sentence vector V. i Cosine similarity: Where V i ,e i The vectors representing the components of vectors V and e are used to output the sentence with the highest cosine similarity as the final fluent sign language sentence that conforms to language norms.

5. A sign language recognition system based on LSTM and NLP, characterized in that: The sign language recognition system is used to run the sign language recognition method according to any one of claims 1-4. The sign language recognition system includes a model training subsystem and a sign language recognition subsystem.

6. A sign language recognition system based on LSTM and NLP according to claim 5, characterized in that: The model training subsystem includes: (1) Data acquisition module: acquiring the sign language video read through the camera or a given sign language video; (2) Sign language token data set generation module: expressing a specific independent sign language phrase meaning with a fixed frame number length as an information point for the sign language video, independently collecting a certain sign language phrase, and dividing the collected sign language video into a supervised sign language token data set with the independent sign language phrase as the segmentation condition; (3) Key point extraction module: inputting the sign language video image, and converting it into the corresponding image key points at an interval of a fixed frame number length k through the sign language gesture key point detection model; (4) Temporal Keypoint Sequence Generation Module: Through the keypoint extraction module, a k-frame keypoint sequence is obtained, P i ={kp i,1 ,kp i,2 ,…,kp i,k }, where kp i,j The tensor representing the coordinates of the sign language gesture keypoints in the j-th frame image is denoted by the sign language lexicon d. i word in i Construct a one-hot code tensor G relative to D i Generate the final time-series key point sequence data t. i ={P i G i } and the temporal key point sequence sample dataset T={t1,t2,…,t n }; (5) LSTM model architecture module: taking the time series key point sequence sample data set T as the input, training the LSTM neural network model, LSTM-LS; (6) Sign Language Feature Quadruple Construction Module: Given a commonly used sign language corpus K constructed from several sign language text sets, use word segmentation technology to extract word units from each text in K to construct a sign language-related natural language word set NW = {word...} i |word i ∈K}; Calculate each sign language word in NW i Index position idx in K i and frequency of occurrence freq i Then, construct a sequence P of lexical-frequency triples, where P[i] = (idx) i ,word i ,freq i ); For each lexical unit, a weight is assigned based on its frequency of occurrence. Lexical units that appear during the training of LSTM-SL are weighted down to obtain the Q dataset of sign language lexical unit representation quadruples sequence samples. (7) W2V-SL training module: inputting the sign language token representation quadruple sequence sample data set Q, and training the word embedding language model W2V-SL.

7. A sign language recognition system based on LSTM and NLP according to claim 6, characterized in that: The sign language recognition subsystem (1) LSTM-SL prediction module: The temporal key point sequence T of the sign language lexical units to be predicted is generated. w ={tw1,tw2,…,tw n Substituting into the LSTM-SL model, we obtain the sign language word prediction sequence S; (2) Sliding window module: When the sign language word is predicted repeatedly in the current window, only the first word is retained and the sliding window is moved. After the sliding is completed, the sign language prediction sequence S is obtained. w ; (3) W2V-SL prediction module: The sign language prediction sequence S w The input word embedding model W2V-SL is transformed into sentence vectors V; (4) Similarity matching module: comparing the sentence vector V with the sentence vectors of common sign language sentences to find the group with the highest similarity for matching, and finally outputting the sign language sentence that conforms to the language specification.

Images