Static sign language recognition method based on bone angle features

Abstract

The invention discloses a static sign language recognition method based on skeleton angle features, and belongs to the field of sign language recognition, and the method comprises the steps: obtaining human hand skeleton joint point coordinate data in a historical data set based on a Medipe method; performing data preprocessing on the human hand skeleton joint point coordinate data to obtain a preprocessed data set; constructing a long-short-term memory network model, and training the long-short-term memory network model based on the preprocessed data set to obtain a sign language recognition model; and recognizing the static sign language based on the sign language recognition model. According to the scheme provided by the invention, hand positioning, cutting and hand image adjustment do not need to be carried out, the requirement for a training data set is not higher than that of a traditional scheme, and only a researcher needs to provide a clear hand image.

Description

Technical Field

[0001] The invention belongs to the field of sign language recognition, and in particular relates to a static sign language recognition method based on bone angle features. Background Art

[0002] Sign language recognition is the main language for the deaf community to communicate. As a visual language, sign language has a rich vocabulary and complete grammar. However, in daily life, it is difficult for hearing people to understand the content expressed by the deaf people's sign language due to lack of professional knowledge. This communication barrier brings inconvenience to the deaf people's daily life, especially in public places such as hospitals, banks, and stations. In order to achieve free communication between the deaf and the hearing, sign language recognition is a very important technical link.

[0003] Existing sign language recognition models generally face the problem of inaccuracy, mainly reflected in their reliance on standardized input of gestures. In practical applications, many systems have difficulty dealing with the diversity and deformation of gestures. When the hands are misplaced or in non-standard postures, the recognition accuracy of these models drops significantly, resulting in poor recognition results. Many systems also fail to fully utilize the spatial position and posture information of the hand, further reducing the accuracy and reliability of recognition. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a static sign language recognition method based on bone angle features, comprising:

[0005] Based on the Mediapipe method, the coordinate data of the human hand skeleton joint points in the historical data set are obtained;

[0006] Performing data preprocessing on the coordinate data of the human hand skeleton joint points to obtain a preprocessed data set;

[0007] Constructing a long short-term memory network model, and training the long short-term memory network model based on the preprocessed data set to obtain a sign language recognition model;

[0008] Static sign language is recognized based on the sign language recognition model.

[0009] Preferably, the process of obtaining the coordinate data of the human hand skeleton joint points in the historical data set based on the Mediapipe method includes:

[0010] Get historical data sets;

[0011] The historical data set is extracted based on the Mediapipe method to obtain the coordinate data of 21 human hand skeletal joint points.

[0012] Preferably, the process of preprocessing the coordinate data of the human hand skeleton joint points includes:

[0013] Acquiring a reference axis based on the human hand skeleton joint point coordinate data;

[0014] calculating the slope of each set of line segments as a vector based on the reference axis;

[0015] Obtaining the CosSin value of the angle between each group of vectors and the reference vector based on the slope, and obtaining the preprocessed data set based on the CosSin value;

[0016] Preferably, the expression for calculating the slope of each group of line segments as a vector based on the reference axis is:

[0017]

[0018] Where k is the slope, is the horizontal coordinate of the first point, is the ordinate of the first point, is the horizontal coordinate of the second point, is the ordinate of the second point;

[0019] The expression for obtaining the CosSin value of the angle between each set of vectors and the reference vector based on the slope is:

[0020] in, is the angle between the first joint of the finger and the reference axis, and k is the slope.

[0021] Preferably, the long short-term memory network model includes: an LSTM layer, an attention mechanism layer and a fully connected layer;

[0022] The LSTM layer is used to extract the dependencies and features between points;

[0023] The attention mechanism layer is used to assign weights to different skeleton key points to obtain weighted features;

[0024] The fully connected layer is used to classify the weighted features.

[0025] Preferably, the loss function of the long short-term memory network model is:

[0026] ;

[0027] Among them, C is the number of sign language categories, y i is the one-hot encoding label of the true category, y' i is the probability of the model predicting the output, λ is the regularization strength hyperparameter, and w j is the jth weight parameter in the neural network, and N is the total number of parameters in the neural network.

[0028] Preferably, the process of obtaining the sign language recognition model further includes: using accuracy and F1-Score as evaluation indicators of the sign language recognition model.

[0029] Preferably, the calculation expression of the accuracy is:

[0030] ;

[0031] Among them, Accuracy is the accuracy, Sum() is to calculate the sum of all values ​​in a data set, is the number of correctly classified samples, is the total sample number;

[0032] The calculation expression of the F1-Score is:

[0033] ;

[0034] in, is the F1 score, is the accuracy, is the recall rate.

[0035] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method is implemented when the processor executes the computer program.

[0036] On the other hand, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and wherein the method is implemented when the computer program is executed by a processor.

[0037] Compared with the prior art, the present invention has the following advantages and technical effects:

[0038] The present invention performs special data preprocessing on the input image, relinks the skeleton point information extracted from the gesture image of the original data set, recomposes the sine and cosine values ​​of the angles between the generated several skeleton points and the standard axis into a new data set, and combines the LSTM neural network for deep learning, thereby obtaining a model with stronger performance. This model can effectively recognize the diversity of gestures and different placement methods, ensure that a higher accuracy rate can be maintained under various postures, and enhance the robustness of the system. Compared with traditional sign language recognition schemes, the scheme proposed by the present invention does not require hand positioning, cropping and adjustment of hand images, and the requirements for training data sets are not as high as traditional schemes. Researchers only need to provide clear hand images, and there is no need to collect a large number of samples for the same gesture, different positions, and different angles. With the same number of valid image samples, the input data volume of the model of the method in this paper is reduced by 55% compared with the traditional method, which can effectively save the workload and resource consumption of research. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A point map outputted by an embodiment of the present invention;

[0041] Figure 2 is a data processing flow chart of an embodiment of the present invention;

[0042] Figure 3 This is a model training flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Embodiment 1

[0046] This embodiment provides a static sign language recognition method based on bone angle features, including:

[0047] Based on the Mediapipe method, the coordinate data of the human hand skeleton joint points in the historical data set are obtained;

[0048] Performing data preprocessing on the coordinate data of the human hand skeleton joint points to obtain a preprocessed data set;

[0049] Constructing a long short-term memory network model, and training the long short-term memory network model based on the preprocessed data set to obtain a sign language recognition model;

[0050] Static sign language is recognized based on the sign language recognition model.

[0051] This embodiment compares three different data set processing methods, namely, the sine and cosine values ​​(hereinafter referred to as LSTM_CosSin), two-dimensional coordinate points (hereinafter referred to as LSTM_2Coor), and three-dimensional coordinate points (hereinafter referred to as LSTM_3Coor) of this embodiment form three different primitives as the data input of the model. Among them, LSTM_CosSin uses Mediapipe to complete the acquisition of the coordinate data of the human hand skeleton joint points, and then uses the data processing solution proposed in this embodiment to convert the coordinate data to achieve data noise reduction, so that the feature data after the secondary conversion is used as the model input data. LSTM_2Coor and LSTM_3Coor are traditional sign language recognition model input data solutions, that is, only Mediapipe is used to acquire the coordinate data of the human skeleton joint points, and the skeleton joint point coordinate data is directly used as the input data of the sign language recognition model.

[0052] The datasets used in this implementation are derived from public datasets, and a total of four datasets are used: the American Sign Language digit dataset, the American Sign Language alphabet dataset, the Chinese Sign Language digit dataset, and the Chinese Sign Language alphabet dataset.

[0053] When the time series axis is introduced, the model of this embodiment can also recognize dynamic sign language, and the LSTM and attention mechanism show more advantages in processing these feature sequences with time series.

[0054] Among them, Mediapipe is used as a black box and outputs a bitmap, such as Figure 1 shown.

[0055] This embodiment defines that <5,17> is used as the reference axis (hereinafter referred to as the X-axis) among the skeleton points obtained from Mediapipe. Since the angle θ between the first joint of the finger and the reference axis X is relatively stable, no matter how the gesture angle is tilted, the effect on the angle θ is small. In other words, in the same gesture, the angle between the finger joint and the reference axis X is relatively stable, and will not produce huge differences due to changes in the gesture placement. Since each person's fingers are different in length and degree of curvature, the value of the angle θ may have a slight deviation. In order to reduce the impact of the fluctuation of the angle θ on the gesture features, this embodiment introduces the value pair of sinθ and cosθ to describe the features of gesture sign language, that is, the method LSTM_CosSin.

[0056] The data processing process is as follows Figure 2 As shown;

[0057] 1. Use MediaPipe to extract 21 skeleton key points;

[0058] 2. Remove two points 0 and 1 and rearrange the skeleton points to get the following four points:

[0059] Baseline: HAND_PALM_CONNECTIONS=[(5,17)];

[0060] First joint: FIRST_PHALANX=[(2,3),(5,6),(9,10),(13,14),(17,18)];

[0061] Second joint: SECOND_PHALANX=[(3,4),(6,7),(10,11),(14,15),(18,19)];

[0062] Third joint: THIRD_PHALANX=[(7,8),(11,12),(15,16),(19,20)];

[0063] 3. Determine the baseline vector as (5,17);

[0064] 4. Calculate the slope of each set of line segments as a vector.

[0065] Set points A(x1,y1), B(x2,y2) according to the slope calculation formula:

[0066]

[0067] Where k is the slope, is the horizontal coordinate of the first point, is the ordinate of the first point, is the horizontal coordinate of the second point, is the ordinate of the second point;

[0068] 5. Calculate the CosSin value of the angle between each set of vectors and the reference vector

[0069] According to the slope k and the formula, the cossin value can be obtained

[0070] ;

[0071] ;

[0072] in, is the angle between the first joint of the finger and the reference axis, and k is the slope.

[0073] The model training process is as follows Figure 3 As shown;

[0074] The model architecture is defined as follows:

[0075] 1. Input layer: The input data is the bone point features corresponding to each frame of the image, including the calculated slope and the sine and cosine values ​​of the angle, forming a data format like (N, 3), where N is the number of key points and 2 represents the two-dimensional features (i.e. sin(θ), cos(θ)).

[0076] 2. LSTM layer: Compared with ordinary RNN, it can extract more dependencies and features between points. The cross entropy loss function is selected in the LSTM architecture to deal with classification problems. At the same time, the L2 regularization term (weight decay, WeightDecay) is introduced to constrain model parameters and prevent overfitting. The final loss function is the weighted sum of the cross entropy loss and the L2 regularization term. The specific function expression is as follows:

[0077] Cross entropy loss function:

[0078] In the sign language recognition task, the goal is to predict the category label of the input gesture sequence through a neural network. Assuming that the sign language dataset contains C categories and the true label of each input sample is represented by one-hot encoding, the cross entropy loss is defined as follows:

[0079]

[0080] Where C is the number of sign language categories. i is the one-hot encoding label of the true category. If the sample belongs to category i, then y i =1, otherwise y i =0. y' i The probability of the model predicting the output is:

[0081]

[0082] Among them, z i and z j are the unnormalized outputs (logits) of the model on categories i and j, respectively.

[0083] L2 Regularization (Weight Decay):

[0084] In order to avoid overfitting of the model and improve the generalization ability, this embodiment uses L2 regularization to constrain the weight parameters of the neural network, which are defined as follows:

[0085]

[0086] in:

[0087] λ is a regularization strength hyperparameter, which is set to λ=0.001 in this embodiment.

[0088] w j is the jth weight parameter in the neural network.

[0089] N is the total number of parameters of the neural network.

[0090] The L2 regularization term penalizes parameters with larger weights, allowing the model to maintain smaller parameter values ​​during training and improve the stability of the model.

[0091] The final loss function:

[0092] Combining the cross entropy loss and the L2 regularization term, the final loss function of this embodiment is defined as follows:

[0093] L=L CE +L reg

[0094] Right now:

[0095]

[0096] The loss function of this embodiment is composed of the cross entropy loss L CE +L2 regularization is designed to improve the classification accuracy of the model, while controlling the complexity of the model and enhancing the generalization ability. In addition, combined with the AdamW optimizer, the training stability is improved, so that the model can achieve better performance in the sign language recognition task.

[0097] This embodiment uses a LSTM neural network, R represents the length of the input sequence, which can be any value, generally the number of samples in the training set, and n represents the dimension of the input data. LSTM contains 128 hidden layers and uses the ReLU activation function to solve the problem of traditional RNN explosion and reduce the computational complexity. The ReLU function formula is as follows:

[0098]

[0099] 3. Attention mechanism layer: weights can be assigned to different skeleton key points to highlight the more important joint points for sign language recognition. At the same time, through the attention mechanism, the model can better ignore the point pair relationships with large noises and only focus on stable feature angles. BahdanauAttention is used to calculate the importance of skeleton points. The attention weight is defined as follows:

[0100]

[0101] in, is the attention score of skeleton point i, e j Similarly, for the attention score of skeleton point j, W, V T , b are trainable parameters, which are weight matrix, projection vector, and bias vector h respectively. i is the feature of skeleton point i (it is the feature vector extracted by LSTM). This formula is used to measure the importance of skeleton point i in the whole sequence. After the score is normalized by Softmax, the final attention weight is formed. Finally, these attention weights are used to weight the sum of the features of all skeleton points to form a global representation as follows:

[0102] ;

[0103] It is a weighted summation of the global representation, which combines the information of all skeleton points and weights them according to their importance (i.e., attention weight). is the attention weight for each skeleton point.

[0104] The Softmax function expression is as follows:

[0105] ;

[0106] is the probability of category y given input x. Applies an exponential operation to the raw scores of category y, amplifying the impact of higher scores.

[0107] 4. Fully connected layer: Classify the weighted features combined with the attention mechanism, as follows:

[0108] (1) First layer: 256 neurons, using ReLU activation function

[0109] (2) Second layer: 128 neurons, using ReLU activation function

[0110] (3) Output layer: The number of neurons can be adjusted to the type of gesture, using the Softmax activation function

[0111] In terms of model evaluation, this embodiment designs accuracy and F1-Score as evaluation indicators for the sign language gesture recognition model. Accuracy represents the model's judgment of the batch of input samples, and the value range is [0,1]. Generally speaking, the higher the value tested, the better the model effect, and vice versa. The formula is as follows:

[0112] ;

[0113] Among them, Accuracy is the accuracy, Sum() is to calculate the sum of all values ​​in a data set, is the number of correctly classified samples, is the total number of samples.

[0114] F1-Score is the harmonic mean of Precision and Recall, and is a measure of classification problems. This indicator combines the results of Precision and Recall. The value range of F1-Score is [0,1]. Generally speaking, the higher the value tested, the better the model effect, and vice versa. The formula is as follows:

[0115] ;

[0116] in, is the F1 score, is the accuracy, is the recall rate.

[0117] After testing in this embodiment, the accuracy of the model trained by the LSTM_CosSin data set proposed in this embodiment is more stable and relatively higher than that of the other two traditional solutions, and at the same time, the amount of data required to generate the model is smaller and more arbitrary. Therefore, this embodiment determines that the method of this embodiment has better performance.

[0118] Model Identification:

[0119] 1. Use MediaPipe to output 21 skeleton points based on the incoming image;

[0120] 2. Obtain data according to the rules defined in the training and pass it into the model;

[0121] 3. After recognition, the model returns the first 10 letters recognized and their credibility.

[0122] On the other hand, this embodiment further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method is implemented when the processor executes the computer program.

[0123] On the other hand, this embodiment further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and wherein the method is implemented when the computer program is executed by a processor.

[0124] This embodiment performs special data preprocessing on the input image, relinks the skeleton point information extracted from the gesture image of the original data set, recomposes the sine and cosine values ​​of the angles between the generated several skeleton points and the standard axis into a new data set, and combines the LSTM neural network for deep learning, thereby obtaining a model with stronger performance. This model can effectively recognize the diversity of gestures and different placement methods, ensure that a higher accuracy rate can be maintained under various postures, and enhance the robustness of the system. Compared with the traditional sign language recognition scheme, the scheme proposed in this embodiment does not require hand positioning, cropping and adjustment of hand images, and the requirements for training data sets are not as high as traditional schemes. It only requires researchers to provide clear hand images, and there is no need to collect a large number of samples for the same gesture, different positions, and different angles. With the same number of valid image samples, the input data volume of the model of the method in this embodiment is reduced by 55% compared with the traditional method, which can effectively save the workload and resource consumption of research.

[0125] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A static sign language recognition method based on bone angle features, characterized in that: include: Based on the Mediapipe method, the coordinate data of the human hand skeleton joint points in the historical data set are obtained; Performing data preprocessing on the coordinate data of the human hand skeleton joint points to obtain a preprocessed data set; Constructing a long short-term memory network model, and training the long short-term memory network model based on the preprocessed data set to obtain a sign language recognition model; Recognize static sign language based on the sign language recognition model; The process of preprocessing the coordinate data of the human hand skeleton joint points includes: Acquiring a reference axis based on the human hand skeleton joint point coordinate data; calculating the slope of each set of line segments as a vector based on the reference axis; Obtaining the CosSin value of the angle between each group of vectors and the reference vector based on the slope, and obtaining the preprocessed data set based on the CosSin value; The long short-term memory network model includes: an LSTM layer, an attention mechanism layer and a fully connected layer; The LSTM layer is used to extract the dependencies and features between points; The attention mechanism layer is used to assign weights to different skeleton key points to obtain weighted features; The fully connected layer is used to classify the weighted features.

2. The method according to claim 1, characterized in that The process of obtaining the coordinate data of the human hand skeleton joint points in the historical data set based on the Mediapipe method includes: Get historical data sets; The historical data set is extracted based on the Mediapipe method to obtain the coordinate data of 21 human hand skeletal joint points.

3. The method according to claim 1, characterized in that The expression for calculating the slope of each set of line segments as a vector based on the reference axis is: ; Where k is the slope, is the horizontal coordinate of the first point, is the ordinate of the first point, is the horizontal coordinate of the second point, is the ordinate of the second point; The expression for obtaining the CosSin value of the angle between each set of vectors and the reference vector based on the slope is: ; ; in, is the angle between the first joint of the finger and the reference axis, and k is the slope.

4. The method according to claim 1, characterized in that: The loss function of the long short-term memory network model is: ； Among them, C is the number of sign language categories, y i is the one-hot encoding label of the true category, y' i is the probability of the model predicting the output, λ is the regularization strength hyperparameter, and w j is the jth weight parameter in the neural network, and N is the total number of parameters in the neural network.

5. The method according to claim 1, characterized in that The process of obtaining the sign language recognition model also includes: using accuracy and F1-Score as evaluation indicators of the sign language recognition model.

6. The method according to claim 5, characterized in that The calculation expression of the accuracy is: ； Among them, Accuracy is the accuracy, Sum() is to calculate the sum of all values ​​in a data set, is the number of correctly classified samples, is the total sample number; The calculation expression of the F1-Score is: ； in, is the F1 score, is the accuracy, is the recall rate.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images