Sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM

Abstract

The present invention relates to the field of deep learning technology, and in particular, provides a sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM. The method includes constructing and obtaining an optimized sign language gesture sequence recognition model; by accessing a developed front-end webpage, and collecting sign language and gesture sequence videos through a camera device, and transmitting them back to a server for processing, obtaining processed sign language and gesture sequence information; inputting the processed sign language and gesture sequence information into an optimized sign language gesture sequence recognition model to obtain a recognition result; and transmitting the recognition result back to the front-end webpage, presenting it in the form of text and voice. The method improves the model algorithm, uses a multi-layer weighted residual self-attention network and a lightweight LSTM to realize continuous sign language and two-handed gesture sequence recognition, improves the popularity of sign language gesture recognition, reduces the complexity of operation, and expands the scope of application of hearing-impaired patients.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM. Background Art

[0002] The number of hearing-impaired patients is increasing every year. They can only communicate through sign language and gestures, which makes their social interaction extremely inconvenient and has a certain impact on their daily life and mental health. Sign language translation is the main direction to solve the problem of hearing-impaired patients communicating with others. However, there are very few sign language interpreters in society compared to hearing-impaired patients. Although the sign language translation software and wearable sign language translation devices on the market can realize the function of recognizing sign language, most of them recognize isolated words without considering the relationship between the context, which leads to deviations in sentence meaning. At the same time, there are also various problems in the use process, such as high cost, which is not conducive to popularization, complex operation and poor reusability.

[0003] The emergence of long short-term memory (LSTM) networks provides a new solution for continuous sign language and gesture recognition. LSTM models can better handle long-term dependencies, sequence data, variable-length inputs, and complex language structures, and have many advantages in language recognition tasks. In addition, most existing sign language recognition systems are limited to one-way communication between deaf-mute people and non-deaf-mute people, while there is still a relatively broad space for development in two-way communication. At the same time, there is also a gap in the problem of two-way communication between the deaf-mute and the blind, two almost completely isolated groups, which leads to the fact that most hearing-impaired patients cannot truly integrate into the lives of normal people, causing a serious disconnection from society. Summary of the invention

[0004] In view of this, the present invention provides a sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM, which is used to solve the problem that traditional models cannot obtain contextual information and cause sentence errors, improve the accuracy and popularity of sign language gesture recognition, reduce the complexity of operation, and expand the scope of application for hearing-impaired patients.

[0005] In a first aspect, the present invention provides a method for sign language gesture sequence recognition based on multi-layer weighted residual self-attention and LSTM, the method comprising:

[0006] Step 1: Construct a data processing function to preprocess the training data set and the test data set to obtain the preprocessed training data set and the test data set;

[0007] Step 2: Construct a sign language gesture sequence recognition model based on multi-layer weighted residual self-attention and long short-term memory network LSTM;

[0008] Step 3, iteratively training and optimizing the sign language gesture sequence recognition model in step 2 using the preprocessed training data set in step 1 to obtain an optimized sign language gesture sequence recognition model;

[0009] Step 4, loading the weights of the sign language gesture sequence recognition model optimized in step 3, and inputting the test data set preprocessed in step 1 into the sign language gesture sequence recognition model optimized in step 4 to obtain the test results;

[0010] Step 5: Access the developed front-end webpage and collect the sign language and gesture sequence video through the camera device, and transmit it back to the server for processing to obtain the processed sign language and gesture sequence information;

[0011] Step 6: Input the sign language and gesture sequence information processed in step 5 into the sign language and gesture sequence recognition model optimized in step 4 to obtain the recognition result; and transmit the recognition result back to the front-end web page and present it in the form of text and voice.

[0012] Optionally, after step 6, the method further includes uploading the successfully recognized two-handed gesture sign language and gesture sequence data to the cloud for use in the next training data set, and displaying them on the front-end web page for learning.

[0013] Optionally, step 1 includes:

[0014] Step 11: For the original video data, use the cv2 module to read and store it;

[0015] Step 12, after reading the original video data according to step 11, perform frame cutting processing on it. When performing frame cutting processing, fuzzy judgment is used to ensure the effectiveness of cutting each frame, that is, the gradient of each frame image is calculated, and images with gradients less than the gradient threshold are excluded. Finally, the images belonging to the same video after cutting are arranged in time series to obtain images of 1280×720 pixels each;

[0016] Step 13: Group the image sequences corresponding to all videos and set the batch size batchsize Set to 7 and adjust the order of each dimension to ;in b for batchsize , is the length of the extracted image sequence, W and H are the image width and image height respectively, where W The value of is 1280. H The value of is 720; each video is processed into a corresponding tensor ,in L is the total number of frames extracted, Cis the number of image channels.

[0017] Optionally, the step 2 includes:

[0018] Step 21: Divide the model into encoder Encoder and decoder Decoder Two parts;

[0019] Perform convolution on each image in the same video group to extract image features, using a 2×2 convolution kernel with a step size of 2 and adding a maximum pooling layer Maxpool , each image is 160×90 pixels; first, BatchNorm2d The function normalizes the data, and its formula is expressed as: ,in x is the value in the two-dimensional tensor of the incoming data, mean and var are the mean and variance respectively, e , gamma , beta is an adjustable parameter; secondly, use Relu Functions and Tanh Function, its formula expression is: , ,in For each value in the input tensor; finally, use the activation function LeakyReLU , its formula is expressed as: ,in is the input amount, is an adjustable parameter value, and the data after feature extraction is obtained; the obtained feature data is projected into a one-dimensional space to obtain the input tensor ,in b for batchsize , l is the length of the extracted image sequence, v is the feature vector;

[0020] Step 22: Convert the input tensor in step 21 , first go through Transformer Module encoder Encoder , perform multi-layer self-attention preprocessing, and its formula is expressed as: , ,in, attention (·) indicates self-attention processing; sorftmax (·) represents the normalization function; T Represents the transpose of a matrix; Indicates parameter, the value is 64; Mulatt A tensor representing the result of self-attention processing; comcat (·) represents the merging function of each header result; to All represent the results of self-attention heads; h Indicates h A self-attention head, Q , K , V They are tensors X With the weight matrix , , The result of multiplication is, Q , K , V Represent query tensor, key tensor, and value tensor respectively; is the learning parameter;

[0021] After the self-attention mechanism is processed, the weights of each part in the sequence are obtained and compared with the input tensor bit by bit. Multiplying, it is expressed as , ,in is the output tensor, is the input tensor, encoder (·) represents the multi-layer self-attention weight generation function; the feature information with attention weight processing is obtained and used as the encoder Encoder The final result is passed to the decoder Decoder middle;

[0022] Step 23: Pass the result to the decoder according to step 22 Decoder After that, the input result is analyzed and processed by a multi-layer weighted residual self-attention module. The multi-layer weighted residual self-attention module is connected. The multi-layer weighted residual self-attention module is divided into multiple weighted residual self-attention sub-modules. Each sub-module first divides the input into two paths, one of which is passed through a multi-layer linear network. mlp and normalization layer softmax Get weight , , which is expressed as ,in mlp(x) Indicates passing mlp Input processed by a multi-layer linear network, w Represents the generated weight, splitting it into , ; The other path passes through a linear layer and then three self-attention modules respectively. After weighted accumulation, it is finally residually connected and output. Its expression is ; ; ,in , and For the output of each level, , is the weight, x is the input tensor, is the processing result of the linear layer and activation function, Represents the self-attention processing function; after passing through the entire multi-layer weighted residual self-attention module, the result is passed to the LSTM network for time series prediction processing;

[0023] Step 24: The information processed in step 23 is passed into a multi-layer perceptron MLP network consisting of several layers of fully connected and non-fully connected layers. Tanh Function as activation function, using batchnorm Normalize the data and perform residual connections between layers.

[0024] Optionally, step 3 includes:

[0025] Step 31: Use cross entropy loss Cross Entropy Loss As a loss function, the cross entropy function expression is: ,in, N is the number of training samples, is the label of the statement, is the sentence predicted by the model, f Indicates f categories; iteratively update the model parameters by calculating the loss value with the true label;

[0026] Step 32, using the SGD stochastic gradient descent optimizer, and performing step-by-step training, first training with a learning rate of 0.0005; if there is no significant change in the training result, continue training with a learning rate of 0.0001, store the model parameters after every 15 rounds of training, and after comparing the recognition effects of each group of saved parameters, select a group of model parameters with the best recognition effect for subsequent prediction. Optionally, step 5 includes:

[0027] Step 51: Develop the front-end web page using conventional HTML, CSS, and JavaScript; getUsermidea The function calls the camera device to collect sign language and gesture sequence videos; when performing front-end and back-end data interaction, the network adaptive module is introduced, and the front-end web page sends detection messages to the server at a regular interval, and obtains the estimated value of the current network environment based on the response time and packet loss rate of the sent messages; if the estimated value of the current network environment is greater than the network environment threshold, the current network environment is poor, the front-end web page reduces the frame rate of the collected video, and the server changes the parameters of the model accordingly; use XMLHttpRequest The object makes an AJAX request to transmit the collected sign language and gesture videos and the estimated value of the current network environment to the backend server;

[0028] Step 52: The server scales the sign language and gesture video transmitted by the front-end web page, and performs frame capture on the video. When performing frame capture, fuzzy judgment is used to ensure the validity of each frame, that is, the gradient of each frame image is calculated, and images with gradients less than the gradient threshold are excluded. Finally, the captured images are arranged in time series to obtain an image sequence of 1280×720 pixels each, so as to obtain the processed sign language and gesture sequence information.

[0029] Optionally, step 6 includes:

[0030] The processed sign language and gesture sequence information is input into the optimized sign language and gesture sequence recognition model. After obtaining the corresponding recognition result, the text result is transmitted back to the front-end web page for display; and JavaScript In SpeechSynthesisUtterance The object pair recognition results are presented in the form of speech.

[0031] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM in the first aspect or any possible implementation of the first aspect.

[0032] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the device to execute the sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM in the first aspect or any possible implementation of the first aspect.

[0033] In the technical solution provided by the present invention, the method includes constructing a data processing function, preprocessing a training data set and a test data set to obtain a preprocessed training data set and a test data set; constructing a sign language gesture sequence recognition model based on multi-layer weighted residual self-attention and LSTM; iteratively training and optimizing the sign language gesture sequence recognition model through the preprocessed training data set to obtain an optimized sign language gesture sequence recognition model; loading the weights of the optimized sign language gesture sequence recognition model, and inputting the preprocessed test data set into the optimized sign language gesture sequence recognition model to obtain a test result; by accessing the developed front-end webpage, and collecting sign language and gesture sequence videos through a camera device, and transmitting them back to the server The device is used to process the processed sign language and gesture sequence information; the processed sign language and gesture sequence information is input into the optimized sign language and gesture sequence recognition model to obtain the recognition result; and the recognition result is transmitted back to the front-end web page and presented in the form of text and voice. This method improves the model algorithm, uses a multi-layer weighted residual self-attention network and a lightweight LSTM to realize continuous sign language and two-handed gesture sequence recognition, and improves the recognition accuracy, solves the problem that the traditional model cannot obtain contextual information and causes sentence errors, and selects a portable front-end web page to collect information, expands the scope of use of the system, improves the popularity of sign language and gesture recognition, reduces the complexity of operation, and expands the scope of application of hearing-impaired patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A flowchart of a sign language gesture sequence recognition method provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a sign language gesture sequence recognition model based on multi-layer weighted residual self-attention and LSTM provided in an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a feature extraction module of a sign language gesture sequence recognition model based on multi-layer weighted residual self-attention and LSTM provided in an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of a multi-layer weighted residual self-attention model of a sign language gesture sequence recognition model based on multi-layer weighted residual self-attention and LSTM provided in an embodiment of the present invention;

[0039] Figure 5 A schematic diagram of a front-end web page provided by an embodiment of the present invention;

[0040] Figure 6 A schematic diagram of another front-end webpage provided by an embodiment of the present invention;

[0041] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0045] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0046] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0047] Figure 1 A flowchart of a sign language gesture sequence recognition method provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0048] Step 1: Construct a data processing function to preprocess the training data set and the test data set to obtain the preprocessed training data set and the test data set.

[0049] In the embodiment of the present invention, step 1 includes:

[0050] Step 11: For the original video data, use the cv2 module to read and store it;

[0051] Step 12, after reading the original video data according to step 11, perform frame cutting processing on it. When performing frame cutting processing, fuzzy judgment is used to ensure the effectiveness of cutting each frame, that is, the gradient of each frame image is calculated, and images with gradients less than the gradient threshold are excluded. Finally, the images belonging to the same video after cutting are arranged in time series to obtain images of 1280×720 pixels each;

[0052] In the embodiment of the present invention, for example, the gradient threshold is set to 90. When the gradient of a certain frame image is 80, the image gradient is less than the gradient threshold, and the frame image is excluded; when the gradient of a certain frame image is 100, the image gradient is greater than the gradient threshold, and the frame image is retained.

[0053] Step 13: Group the image sequences corresponding to all videos and set the batch size batchsize Set to 7 and adjust the order of each dimension to ;in b for batchsize , is the length of the extracted image sequence, W and H are the image width and image height respectively, where W The value of is 1280. H The value of is 720; each video is processed into a corresponding tensor ,in L is the total number of frames extracted, C is the number of image channels.

[0054] In the embodiment of the present invention, the number of image channels is C The value of is 3.

[0055] In the embodiment of the present invention, before step 1, the step further includes obtaining a Chinese Sign Language dataset (CSL), and dividing the dataset into a training dataset and a test dataset in a ratio of 8:2.

[0056] Step 2: Construct a sign language gesture sequence recognition model based on multi-layer weighted residual self-attention and long short-term memory network LSTM.

[0057] In the embodiment of the present invention, Figures 2 to 4As shown, step 2 includes:

[0058] Step 21: Divide the model into encoder Encoder and decoder Decoder Two parts;

[0059] Perform convolution on each image in the same video group to extract image features, using a 2×2 convolution kernel with a step size of 2 and adding a maximum pooling layer Maxpool , each image is 160×90 pixels; first, BatchNorm2d The function normalizes the data, and its formula is expressed as: ,in x is the value in the two-dimensional tensor of the incoming data, mean and var are the mean and variance respectively, e , gamma , beta is an adjustable parameter; secondly, use Relu Functions and Tanh Function, its formula expression is: , ,in For each value in the input tensor; finally, use the activation function LeakyReLU , its formula is expressed as: ,in is the input quantity, is an adjustable parameter value, and the data after feature extraction is obtained; the obtained feature data is projected into a one-dimensional space to obtain the input tensor ,in b for batchsize , l is the length of the extracted image sequence, v is the feature vector;

[0060] Step 22: Convert the input tensor in step 21 , first go through Transformer Module encoder Encoder , perform multi-layer self-attention preprocessing, and its formula is expressed as: , ,in, attention (·) indicates self-attention processing; sorftmax (·) represents the normalization function; T Represents the transpose of a matrix; Indicates parameter, the value is 64; Mulatt A tensor representing the result of self-attention processing; comcat (·) represents the merging function of each header result; to All represent the results of self-attention heads; hIndicates h A self-attention head, Q , K , V They are tensors X With the weight matrix , , The result of multiplication is, Q , K , V Represent query tensor, key tensor, and value tensor respectively; is the learning parameter;

[0061] After the self-attention mechanism is processed, the weights of each part in the sequence are obtained and compared with the input tensor bit by bit. Multiplying, it is expressed as , ,in is the output tensor, is the input tensor, encoder (·) represents the multi-layer self-attention weight generation function; the feature information with attention weight processing is obtained and used as the encoder Encoder The final result is passed to the decoder Decoder middle;

[0062] Step 23: Pass the result to the decoder according to step 22 Decoder After that, the input result is analyzed and processed by a multi-layer weighted residual self-attention module. The multi-layer weighted residual self-attention module is connected. The multi-layer weighted residual self-attention module is divided into multiple weighted residual self-attention sub-modules. Each sub-module first divides the input into two paths, one of which is passed through a multi-layer linear network. mlp and normalization layer softmax Get weight , , which is expressed as ,in mlp(x) Indicates passing mlp Input processed by a multi-layer linear network, w Represents the generated weight, splitting it into , ; The other path passes through a linear layer and then three self-attention modules respectively. After weighted accumulation, it is finally residually connected and output. Its expression is ; ; ,in , and For the output of each level, , is the weight, x is the input tensor, is the processing result of the linear layer and activation function, Represents the self-attention processing function; after passing through the entire multi-layer weighted residual self-attention module, the result is passed to the LSTM network for time series prediction processing;

[0063] In the embodiment of the present invention, in order to prevent information loss caused by excessive concentration of attention, multiple layers of weighted residual self-attention are used to link between the attention processing layers.

[0064] Step 24: The information processed in step 23 is passed into a multi-layer perceptron MLP network consisting of several layers of fully connected and non-fully connected layers. Tanh Function as activation function, using batchnorm Normalize the data and perform residual connections between layers.

[0065] To sum up, firstly, the two-hand gesture video is frame-cut to obtain a time-series image sequence, which is then fed into the encoder. Encoder In encoder Encoder First, use a multi-layer convolutional network and maximum pooling to extract features, and project the two-dimensional features into a one-dimensional space to obtain the input tensor , then the input tensor After multiple layers of self-attention processing, the weights of each part in the sequence are obtained and compared with the tensor bit by bit. Multiply and pass the result to the decoder Decoder Decoder Decoder First, the encoder Encoder The results are put into a multi-layer weighted residual self-attention network, and then LSTM is used for time series prediction, and finally the results are obtained using a multi-layer perceptron MLP.

[0066] For the initially input 1280×720 size image, multi-layer convolution kernel 2×2 feature extraction is performed, and it is normalized after each convolution process, and the LeakyRelu activation function is used; then the generated two-dimensional features are subjected to the maximum pooling operation to obtain an image of 160×90 pixels, and finally the two-dimensional features are projected into a one-dimensional space.

[0067] In the multi-layer weighted residual self-attention model, the one-dimensional features obtained in the previous layer are first input into the single-layer self-attention network and multiplied by the weight matrix , , , get the query tensor Q , the key tensor K , value tensor V; After the self-attention mechanism is processed, the result of this layer is obtained, and after being accumulated with the processing results of each previous single-layer self-attention network and the weight product, it is used as the input of the next layer of multi-head self-attention network. After multiple layers are superimposed, a multi-layer weighted residual self-attention network is formed.

[0068] Step 3: Iteratively train and optimize the sign language gesture sequence recognition model in step 2 using the preprocessed training data set in step 1 to obtain an optimized sign language gesture sequence recognition model.

[0069] In the embodiment of the present invention, step 3 includes:

[0070] Step 31: Use cross entropy loss Cross Entropy Loss As a loss function, the cross entropy function expression is: ,in, N is the number of training samples, is the label of the statement, is the sentence predicted by the model, f Indicates f categories; iteratively update the model parameters by calculating the loss value with the true label;

[0071] Step 32: Use the SGD stochastic gradient descent optimizer and perform step-by-step training. First, use a learning rate of 0.0005 for training. If there is no obvious change in the training result, continue training with a learning rate of 0.0001. Store the model parameters after every 15 rounds of training. After comparing the recognition effects of each group of saved parameters, select a group of model parameters with the best recognition effect for subsequent predictions.

[0072] Step 4: load the weights of the sign language gesture sequence recognition model optimized in step 3, and input the test data set preprocessed in step 1 into the sign language gesture sequence recognition model optimized in step 4 to obtain the test results.

[0073] Step 5: Access the developed front-end webpage, collect sign language and gesture sequence videos through a camera device, and transmit them back to the server for processing to obtain processed sign language and gesture sequence information.

[0074] In the embodiment of the present invention, step 5 includes:

[0075] Step 51: Develop the front-end web page using conventional HTML, CSS, and JavaScript; getUsermideaThe function calls the camera device to collect sign language and gesture sequence videos; when performing front-end and back-end data interaction, the network adaptive module is introduced, and the front-end web page sends detection messages to the server at a regular interval, and obtains the estimated value of the current network environment based on the response time and packet loss rate of the sent messages; if the estimated value of the current network environment is greater than the network environment threshold, the current network environment is poor, the front-end web page reduces the frame rate of the collected video, and the server changes the parameters of the model accordingly; use XMLHttpRequest The object makes an AJAX request to transmit the collected sign language and gesture videos and the estimated value of the current network environment to the backend server;

[0076] In the embodiment of the present invention, for example, the network environment threshold is 0.75, when the estimated value of the current network environment is 0.62, the estimated value of the current network environment is less than the network environment threshold, which indicates that the current network environment is good; when the estimated value of the current network environment is 0.87, the estimated value of the current network environment is greater than the network environment threshold, which indicates that the current network environment is poor.

[0077] Step 52: The server scales the sign language and gesture video transmitted by the front-end web page, and performs frame capture on the video. When performing frame capture, fuzzy judgment is used to ensure the validity of each frame, that is, the gradient of each frame image is calculated, and images with gradients less than the gradient threshold are excluded. Finally, the captured images are arranged in time series to obtain an image sequence of 1280×720 pixels each, so as to obtain the processed sign language and gesture sequence information.

[0078] Step 6: Input the sign language and gesture sequence information processed in step 5 into the sign language and gesture sequence recognition model optimized in step 4 to obtain the recognition result; and transmit the recognition result back to the front-end web page and present it in the form of text and voice.

[0079] In the embodiment of the present invention, step 6 includes:

[0080] The processed sign language and gesture sequence information is input into the optimized sign language and gesture sequence recognition model. After obtaining the corresponding recognition result, the text result is transmitted back to the front-end web page for display; and the recognition result is presented in the form of speech using the SpeechSynthesisUtterance object in JavaScript.

[0081] In the embodiment of the present invention, Figure 5 As shown in the figure, after pressing the record button, the two-hand gesture recording starts, and after clicking again to end the recording, the video content is transmitted to the server for recognition and finally returned to the lower right frame; Figure 6 As shown, you can learn different sign languages ​​by clicking on the sentences on the right and the corresponding sign language videos will be displayed on the left.

[0082] In the embodiment of the present invention, after step 6, the successfully recognized two-handed gesture sign language and gesture sequence data are also uploaded to the cloud for use in the next training data set, and are displayed on the front-end web page for learning.

[0083] The video sent back by the front end will be stored on the server while the front end request is processed by PHP, so as to collect more sign language and gesture videos that are suitable for daily life, which will be used for future optimization and updating of the model to improve the effect of sign language recognition.

[0084] In this embodiment of the present invention, the training data set and the test data set are divided into a ratio of 8:2, the data set data is iterated for multiple times, and the average loss, the number of correct predictions and the accuracy rate are calculated for each time. Table 1 is part of the experimental results, including the test sentences, the total number of test videos, the number of correct predictions and the accuracy rate, as shown in Table 1.

[0085] Table 1

[0086] ;

[0087] As can be seen from Table 1, the average loss gradually decreases, the accuracy is improved by 56.2%, and the recognition accuracy of some sentences can reach more than 90%, which shows that better weight parameters of the model are obtained.

[0088] Each step of the embodiment of the present invention may be performed by an electronic device, which includes but is not limited to a mobile phone, a tablet computer, a portable PC, a desktop computer, etc.

[0089] In the technical solution provided by the present invention, the method includes constructing a data processing function, preprocessing a training data set and a test data set to obtain a preprocessed training data set and a test data set; constructing a sign language gesture sequence recognition model based on multi-layer weighted residual self-attention and LSTM; iteratively training and optimizing the sign language gesture sequence recognition model through the preprocessed training data set to obtain an optimized sign language gesture sequence recognition model; loading the weights of the optimized sign language gesture sequence recognition model, and inputting the preprocessed test data set into the optimized sign language gesture sequence recognition model to obtain a test result; by accessing the developed front-end webpage, and collecting sign language and gesture sequence videos through a camera device, and transmitting them back to the server The device is used to process the processed sign language and gesture sequence information; the processed sign language and gesture sequence information is input into the optimized sign language and gesture sequence recognition model to obtain the recognition result; and the recognition result is transmitted back to the front-end web page and presented in the form of text and voice. This method improves the model algorithm, uses a multi-layer weighted residual self-attention network and a lightweight LSTM to realize continuous sign language and two-handed gesture sequence recognition, and improves the recognition accuracy, solves the problem that the traditional model cannot obtain contextual information and causes sentence errors, and selects a portable front-end web page to collect information, expands the scope of use of the system, improves the popularity of sign language and gesture recognition, reduces the complexity of operation, and expands the scope of application of hearing-impaired patients.

[0090] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the above-mentioned embodiment of the sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM.

[0091] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM in the embodiment is implemented. To avoid repetition, they are not described one by one here.

[0092] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0093] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0094] The memory 212 may be an internal storage unit of the electronic device 21, such as a hard disk or memory of the electronic device 21. The memory 212 may also be an external storage device of the electronic device 21, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 21. Further, the memory 212 may also include both an internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 may also be used to temporarily store data that has been output or is to be output.

[0095] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM, characterized in that: The method comprises: Step 1: Construct a data processing function to preprocess the training data set and the test data set to obtain the preprocessed training data set and the test data set; Step 2: Construct a sign language gesture sequence recognition model based on multi-layer weighted residual self-attention and long short-term memory network LSTM; Step 3, iteratively training and optimizing the sign language gesture sequence recognition model in step 2 using the preprocessed training data set in step 1 to obtain an optimized sign language gesture sequence recognition model; Step 4, loading the weights of the sign language gesture sequence recognition model optimized in step 3, and inputting the test data set preprocessed in step 1 into the sign language gesture sequence recognition model optimized in step 4 to obtain the test results; Step 5: Access the developed front-end webpage and collect the sign language and gesture sequence video through the camera device, and transmit it back to the server for processing to obtain the processed sign language and gesture sequence information; Step 6: Input the sign language and gesture sequence information processed in step 5 into the sign language and gesture sequence recognition model optimized in step 4 to obtain a recognition result; and transmit the recognition result back to the front-end web page and present it in the form of text and voice; The step 2 comprises: Step 21: Divide the model into encoder Encoder and decoder Decoder Two parts; Perform convolution on each image in the same video group to extract image features, using a 2×2 convolution kernel with a step size of 2 and adding a maximum pooling layer Maxpool , each image is 160×90 pixels; first, BatchNorm2d The function normalizes the data, and its formula is expressed as: ,in x is the value in the two-dimensional tensor of the incoming data, mean and var are the mean and variance respectively, e , gamma , beta is an adjustable parameter; secondly, use Relu Functions and Tanh Function, its formula expression is: , ,in For each value in the input tensor; finally, use the activation function LeakyReLU , its formula is expressed as: ,in is the input quantity, is an adjustable parameter value, and the data after feature extraction is obtained; the obtained feature data is projected into a one-dimensional space to obtain the input tensor ,in b for batchsize , l is the length of the extracted image sequence, v is the feature vector; Step 22: Convert the input tensor in step 21 , first go through Transformer Module encoder Encoder , perform multi-layer self-attention preprocessing, and its formula is expressed as: , ,in, attention (·) indicates self-attention processing; sorftmax (·) represents the normalization function; T Represents the transpose of a matrix; Indicates parameter, the value is 64; Mulatt A tensor representing the result of self-attention processing; comcat (·) represents the merging function of each header result; to All represent the results of self-attention heads; h Indicates h A self-attention head, Q , K , V Tensor X With the weight matrix , , The result of multiplication is, Q , K , V Represent query tensor, key tensor, and value tensor respectively; is the learning parameter; After the self-attention mechanism is processed, the weights of each part in the sequence are obtained and compared with the input tensor bit by bit. Multiplying, it is expressed as , ,in is the output tensor, is the input tensor, encoder (·) represents the multi-layer self-attention weight generation function; the feature information with attention weight processing is obtained and used as the encoder Encoder The final result is passed to the decoder Decoder middle; Step 23: Pass the result to the decoder according to step 22 Decoder After that, the input result is analyzed and processed by a multi-layer weighted residual self-attention module. The multi-layer weighted residual self-attention module is connected. The multi-layer weighted residual self-attention module is divided into multiple weighted residual self-attention sub-modules. Each sub-module first divides the input into two paths, one of which is passed through a multi-layer linear network. mlp and normalization layer softmax Get weight , , which is expressed as ,in mlp(x) Indicates passing mlp Input processed by a multi-layer linear network, w Represents the generated weight, splitting it into , ; The other path passes through a linear layer and then three self-attention modules respectively. After weighted accumulation, it is finally residually connected and output. Its expression is ; ; ,in , and For the output of each level, , is the weight, x is the input tensor, is the processing result of the linear layer and activation function, Represents the self-attention processing function; after passing through the entire multi-layer weighted residual self-attention module, the result is passed to the LSTM network for time series prediction processing; Step 24: The information processed in step 23 is passed into a multi-layer perceptron MLP network consisting of several layers of fully connected and non-fully connected layers. Tanh Function as activation function, using batchnorm Normalize the data and perform residual connections between layers.

2. The method according to claim 1, characterized in that: After step 6, the method also includes uploading the successfully recognized two-handed gesture sign language and gesture sequence data to the cloud for use in the next training data set, and displaying them on the front-end web page for learning.

3. The method according to claim 1, characterized in that The step 1 comprises: Step 11: For the original video data, use the cv2 module to read and store it; Step 12, after reading the original video data according to step 11, perform frame cutting processing on it. When performing frame cutting processing, fuzzy judgment is used to ensure the effectiveness of cutting each frame, that is, the gradient of each frame image is calculated, and images with gradients less than the gradient threshold are excluded. Finally, the images belonging to the same video after cutting are arranged in time series to obtain images of 1280×720 pixels each; Step 13: Group the image sequences corresponding to all videos and set the batch size batchsize Set to 7 and adjust the order of each dimension to ;in b for batchsize , is the length of the extracted image sequence, W and H are the image width and image height respectively, where W The value of is 1280. H The value of is 720; each video is processed into a corresponding tensor ,in L is the total number of frames extracted, C is the number of image channels.

4. The method according to claim 1, characterized in that: The step 3 comprises: Step 31: Use cross entropy loss Cross Entropy Loss As a loss function, the cross entropy function expression is: ,in, N is the number of training samples, is the label of the statement, is the sentence predicted by the model, f Indicates f categories; iteratively update the model parameters by calculating the loss value with the true label; Step 32: Use the SGD stochastic gradient descent optimizer and perform step-by-step training. First, use a learning rate of 0.0005 for training. If there is no obvious change in the training result, continue training with a learning rate of 0.0001. Store the model parameters after every 15 rounds of training. After comparing the recognition effects of each group of saved parameters, select a group of model parameters with the best recognition effect for subsequent predictions.

5. The method according to claim 1, characterized in that The step 5 comprises: Step 51: Develop the front-end web page using conventional HTML, CSS, and JavaScript; getUsermidea The function calls the camera device to collect sign language and gesture sequence videos; when performing front-end and back-end data interaction, the network adaptive module is introduced, and the front-end web page sends detection messages to the server at a regular interval, and obtains the estimated value of the current network environment based on the response time and packet loss rate of the sent messages; if the estimated value of the current network environment is greater than the network environment threshold, the current network environment is poor, the front-end web page reduces the frame rate of the collected video, and the server changes the parameters of the model accordingly; use XMLHttpRequest The object makes an AJAX request to transmit the collected sign language and gesture videos and the estimated value of the current network environment to the backend server; Step 52: The server scales the sign language and gesture video transmitted by the front-end web page, and performs frame capture on the video. When performing frame capture, fuzzy judgment is used to ensure the validity of each frame, that is, the gradient of each frame image is calculated, and images with gradients less than the gradient threshold are excluded. Finally, the captured images are arranged in time series to obtain an image sequence of 1280×720 pixels each, so as to obtain the processed sign language and gesture sequence information.

6. The method according to claim 1, characterized in that The step 6 comprises: The processed sign language and gesture sequence information is input into the optimized sign language and gesture sequence recognition model. After obtaining the corresponding recognition result, the text result is transmitted back to the front-end web page for display; and JavaScript In SpeechSynthesisUtterance The object pair recognition results are presented in the form of speech.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM as described in any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the device to perform the sign language gesture sequence recognition method based on multi-layer weighted residual self-attention and LSTM as described in any one of claims 1 to 6.

Images