A sign language recognition method, device and medium

Abstract

The application relates to a sign language recognition method, which comprises the following steps: collecting the bending degree data of each finger through a bending sensor and collecting the posture data of a hand through a six-axis sensor; combining the bending degree data of each finger and the posture data of the hand into sign language data to be recognized; inputting the sign language data to be recognized into a pre-trained sign language recognition model to output corresponding recognition results, wherein the recognition results are texts; and converting the texts of the recognition results into voice outputs. The application improves the accuracy of sign language recognition.

Description

Technical Field

[0001] This invention relates to the field of intelligent recognition technology, and in particular to a sign language recognition method, device, and computer-readable storage medium. Background Technology

[0002] Sign language is a means for deaf and mute people to communicate with the outside world, but the general public lacks understanding of sign language, creating barriers when communicating with deaf and mute individuals. Therefore, reliable methods for sign language recognition are needed.

[0003] The inventors of this application discovered in their research that existing sign language recognition technologies are mainly based on image processing technology, that is, by acquiring images of hand movements and then recognizing the sign language within them. Advances in image processing technology have brought increasingly accurate and efficient methods for gesture recognition. However, the effectiveness of image-based sign language recognition methods is limited by environmental factors, such as lighting, background conditions, shadows, and camera position; therefore, recognition accuracy still needs improvement. Summary of the Invention

[0004] To address the aforementioned problems, the present invention aims to provide a sign language recognition method, apparatus, and computer-readable storage medium that can improve the accuracy of sign language recognition.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] Firstly, this application provides a sign language recognition method, the method comprising:

[0007] The bending data of each finger is collected by a bending sensor, and the posture data of the hand is collected by a six-axis sensor.

[0008] The bending data of each finger and the posture data of the hand are combined to form the sign language data to be recognized;

[0009] The sign language data to be recognized is input into a pre-trained sign language recognition model, and the corresponding recognition result is output as text.

[0010] Convert the text of the recognition results into speech output.

[0011] In one implementation of this application, the sign language recognition model is a neural network model that incorporates an attention mechanism.

[0012] In one implementation of this application, the network architecture of the sign language recognition model includes: a one-dimensional convolutional neural network, and a multi-head attention mechanism module of a fused BiLSTM network connected to the one-dimensional convolutional neural network.

[0013] In one implementation of this application, the one-dimensional convolutional neural network includes a first part and a second part; the multi-head attention mechanism module is connected between the first part and the second part;

[0014] The first part of the one-dimensional convolutional neural network is used to extract data features of the sign language data to be recognized.

[0015] The multi-head attention mechanism module is used to further extract temporal features and action features from the data features;

[0016] The second part of the one-dimensional convolutional neural network is used to identify the recognition result corresponding to the sign language data to be recognized.

[0017] In one implementation of this application, the method further includes the step of training the sign language recognition model.

[0018] In one implementation of this application, the step of training the sign language recognition model includes:

[0019] Collect a set number of sign language samples containing the target sign language, and generate a text file for each sign language sample;

[0020] Each sign language sample is labeled with its corresponding sign language type using its filename;

[0021] Configure the network architecture of the sign language recognition model and initialize the network parameters;

[0022] The labeled sign language samples are input into the sign language recognition model with initialized network parameters for training. The network parameters of the model are adjusted until the model converges, and the trained sign language recognition model is obtained.

[0023] Secondly, this application provides a sign language recognition device, including: a bending sensor, a six-axis sensor, a processor, and an intelligent recognition module;

[0024] The bending sensor is used to collect bending data for each finger;

[0025] The six-axis sensor is used to collect hand posture data;

[0026] The processor is used to combine the bending data of each finger and the posture data of the hand into sign language data to be recognized.

[0027] The intelligent recognition module is used to input the sign language data to be recognized into a pre-trained sign language recognition model, output the corresponding recognition result, which is text; and convert the text of the recognition result into speech output.

[0028] In one implementation of this application, the bending sensor is a FLEX2.2; the six-axis sensor is an MPU6050; and the processor is an STM32 microcontroller.

[0029] In one implementation of this application, the bending sensor, the six-axis sensor, and the processor are disposed in a smart glove, and the processor is connected to the bending sensor and the six-axis sensor respectively.

[0030] The intelligent recognition module is used to deploy the sign language recognition module, recognize the recognition result corresponding to the sign language data to be recognized, and also to use the pyttsx3 library in Python to perform text-to-speech conversion;

[0031] The intelligent identification module is installed in a fixed terminal or mobile terminal that is communicatively connected to the intelligent glove.

[0032] Thirdly, this application provides a computer-readable storage medium storing a computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the sign language recognition method described in the first aspect.

[0033] The present invention has the following advantages due to the adoption of the above technical solutions: The present invention uses a designed sign language recognition model to effectively extract the features of dynamic sign language data collected by sensors, ensuring the accuracy of sign language recognition without being affected by the surrounding environment, and broadcasting the recognition results by voice, thereby facilitating communication between deaf and mute people and the outside world without requiring outsiders to learn sign language. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the sign language recognition method provided in an embodiment of the present invention;

[0035] Figure 2 This is a network structure diagram of the sign language recognition model in an embodiment of the present invention;

[0036] Figure 3 This is a network architecture diagram of the multi-head attention mechanism in an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the training process of the sign language recognition model according to an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0039] To address the urgent need to improve the accuracy of sign language recognition in existing technologies, this invention provides a sign language recognition method, apparatus, and computer-readable storage medium. The method includes: acquiring bending data of each finger using a bending sensor and acquiring hand posture data using a six-axis sensor; combining the bending data of each finger and the hand posture data to form sign language data to be recognized; inputting the sign language data to be recognized into a pre-trained sign language recognition model and outputting a corresponding recognition result, wherein the recognition result is text; and converting the text of the recognition result into speech for output.

[0040] Please refer to the accompanying drawings of the embodiments of the present invention for further details on the methods, apparatus and media provided by the present invention in the more detailed embodiments of the present invention.

[0041] like Figure 1 In one aspect of the embodiments of this application, a sign language recognition method is provided, specifically including:

[0042] S11 collects bending data of each finger through a bending sensor and hand posture data through a six-axis sensor.

[0043] S12, combine the bending data of each finger and the hand posture data into the sign language data to be recognized;

[0044] S13, input the sign language data to be recognized into a pre-trained sign language recognition model, and output the corresponding recognition result, wherein the recognition result is text;

[0045] S14, convert the text of the recognition result into speech output.

[0046] In this embodiment, the designed sign language recognition model effectively extracts the features of dynamic sign language data collected by the sensor, ensuring the accuracy of sign language recognition without being affected by the surrounding environment. The recognition results are then broadcast aloud, facilitating communication between deaf and mute individuals and the outside world without requiring external personnel to learn sign language.

[0047] The above-described methods and apparatus are described below with reference to more accompanying drawings and in more detail with respect to embodiments thereof.

[0048] In a more detailed embodiment of the present invention, a sign language recognition method based on sensor data and deep learning is provided, the specific implementation steps of which are as follows:

[0049] S1: The smart glove, composed of an STM32 microcontroller, five FLEX2.2 bending sensors, and an MPU6050 six-axis sensor, acquires sign language data from sensor data. Bending sensor data is acquired using a potential mapping method, with each FLEX2.2 sensor connected in series with a 15KΩ resistor to the PA1-PA5 interfaces of the STM32 to acquire the degree of finger bending. The MPU6050 six-axis accelerometer motion processing sensor acquires the hand's spatial posture. The MPU sensor uses an accelerometer and gyroscope to read information about the hand's posture, such as acceleration and angular velocity, to extract data on hand position and motion trajectory. The data from both sensors are acquired together by the STM32 and undergo preliminary data processing. Combining these two sensors allows for the capture of hand gestures unaffected by factors such as light intensity, background color, or motion obstruction.

[0050] S2: Input the sign language data to be recognized into the trained sign language recognition model through a serial port to obtain the recognition result output by the sign language recognition model; wherein, the sign language recognition model is composed of a multi-head attention mechanism module that connects a skip connection one-dimensional CNN and a BiLSTM network.

[0051] S3: Use the pyttsx3 library in Python to convert text to speech and broadcast the sign language recognition results.

[0052] In this embodiment of the application, the network architecture of the sign language recognition model is as follows: Figure 2 As shown, it includes a multi-head attention mechanism module that combines a one-dimensional CNN and a BiLSTM network.

[0053] First, a one-dimensional CNN model based on skip connections is designed.

[0054] Based on the functions of feature extraction and sign language recognition, the one-dimensional CNN model is divided into two parts. The first part aims to enhance the data feature extraction capability of the neural network model by fusing feature results from different layers, while the second part is responsible for the sign language recognition task. To capture the temporal sequence within sign language and further extract key action features, a multi-head attention mechanism module integrating BiLSTM is designed. This improved multi-head attention mechanism module is placed after the first part of the one-dimensional CNN model.

[0055] The first part of the one-dimensional CNN includes an input layer, convolutional layers, and max-pooling layers. The sign language sequence is converted into a matrix and used as input to the neural network model. The convolutional layers traverse the input matrix through a sliding window, extracting key feature information from the sign language matrix. To enhance the non-linearity of the neural network model and enable it to learn more complex patterns and features, the output of each convolutional layer undergoes a non-linear transformation using the ReLU activation function. Subsequently, max-pooling layers select the maximum value in each pooling window. The information within each pooling window is compressed into the most important feature value, thereby reducing the dimensionality of the extracted gesture feature matrix and eliminating redundant feature information in the gestures. The convolutional layer is the core of the CNN; the convolution operation can be represented as:

[0056]

[0057] in, This is the vector obtained after the j-th convolution in the k-th convolutional layer. Let i be the input vector of the k-th convolutional layer. These are the weights of the j-th convolution performed on the i-th input vector in the k-th convolutional layer. It is the bias vector of the j-th convolution of the k-th convolutional layer.

[0058] Furthermore, the CNN model employs skip connections to enhance the feature extraction capabilities of one-dimensional CNNs in sign language recognition. Fusing features obtained from convolutional layers through addition operations adds semantic information, making feature extraction more complex and in-depth. Specifically, after max pooling the features extracted from the third and fourth convolutional layers respectively, the corresponding elements of the two feature matrices are added to generate a new feature matrix. This new feature matrix is ​​then added to the feature matrix obtained from the last convolutional layer. Each convolutional layer can extract different features, and multi-level information is considered through feature addition operations. By fusing these features, the model can simultaneously utilize sign language feature information extracted from different convolutional layers.

[0059] The second part of the one-dimensional CNN includes max pooling layers, global average pooling layers, fully connected layers, and an output layer. Max pooling layers help the model focus on information important to the task while reducing the influence of unimportant features. Next, the global average pooling layer calculates the average of all elements in each sign language feature column, compressing the value of each feature column into a single value. This process transforms the feature matrix into a vector, effectively reducing dimensionality while retaining important information. Dropout is then added to mitigate overfitting and enhance the model's generalization ability. Fully connected layers are used for decision-making and classification, mapping the learned feature representations to each sign language category and outputting the corresponding category prediction. The softmax activation function is used to convert the prediction results into a probability distribution vector, where the values ​​represent the probability that the predicted sign language belongs to each category. Finally, the argmax function is used to find the maximum value in the vector, determining the category predicted by the model, thus achieving accurate sign language recognition.

[0060] Furthermore, the BiLSTM network model is integrated into the multi-head attention mechanism module to capture the internal temporal features of sign language and further extract key motion features. A schematic diagram of the network architecture is shown below. Figure 3 .

[0061] The multi-head attention mechanism module mainly consists of multiple self-attention mechanisms. Each self-attention mechanism extracts important gesture features from sign language based on three main inputs: query (Q), key (K), and value (V). Given a task-related query vector Q, it calculates the attention distribution with respect to K and appends it to V, thus calculating the attention. The self-attention mechanism uses a scaled dot product calculation method, and the calculation formula is as follows:

[0062]

[0063] Where d k is the dimension of the input vector.

[0064] The multi-head attention mechanism module directs the extracted gesture features to the weights W respectively. Q W K W V Multiplying yields the corresponding Q, K, and V. These Q, K, and V values ​​are then linearly mapped to h subspaces, with self-attention operations performed in each subspace. The results from each head are then concatenated, and finally, the concatenated result is processed through W. O The learnable parameters undergo a linear transformation to maintain consistency between the input and output dimensions, resulting in the final result of the multi-head attention mechanism: a feature matrix weighted by the importance of sign language features. The multi-head attention calculation formula is as follows:

[0065] MultiHead(Q,K,V)=Concat(head1,…,head h W O

[0066] where head i =Attention(QW i Q ,KW i K VW i V )

[0067] Sign language recognition is related to the dynamic evolution of gestures throughout the entire sign language expression process, exhibiting a strong temporal sequence. For example... Figure 3 As shown, to extract temporal features from sign language, a BiLSTM network model is placed after Q, K, and V. Subsequently, multi-head attention is used to extract feature importance, enabling the model to more effectively classify sign language actions. Furthermore, after the multi-head attention calculation, another BiLSTM network layer is introduced to better capture the bidirectional semantic dependencies of gestures and extract broader contextual information within the gestures. The two-layer BiLSTM temporal network structure allows the model to more comprehensively understand gesture features. Finally, a feature matrix containing temporal information is obtained, weighted by the importance of sign language features.

[0068] The improved multi-head attention mechanism module is placed after the first part of the described one-dimensional CNN architecture based on skip connections.

[0069] A well-designed sign language recognition model with a good network architecture needs to be pre-trained before deployment.

[0070] The specific training process includes the following steps:

[0071] Several sign language samples containing the target sign language were collected, and each sample was compiled into a text file. During the collection process, the speed of the sign language movements, the duration of glove wearing, hand temperature, humidity, glove stretching, and wear varied, making it suitable for use in most situations. Since the sensors used have a total of 11 attributes, the collected data was written into the text file as 11 columns.

[0072] Each sample is labeled with its corresponding sign language category using a filename. Different categories of sign language samples are mapped to different filenames, with the sample file named according to its category.

[0073] Configure the network architecture of the sign language recognition model and initialize the network parameters;

[0074] The labeled sign language samples are input into the sign language recognition model with initialized network parameters for training. The network parameters of the model are adjusted until the model converges, and the trained sign language recognition model is obtained.

[0075] Therefore, the trained model can be used to recognize the sign language data to be recognized and output the recognition results in text form.

[0076] In another aspect, this application also provides a sign language recognition device, including: a bending sensor, a six-axis sensor, a processor, and an intelligent recognition module;

[0077] The bending sensor is used to collect bending data for each finger;

[0078] The six-axis sensor is used to collect hand posture data;

[0079] The processor is used to combine the bending data of each finger and the posture data of the hand into sign language data to be recognized.

[0080] The intelligent recognition module is used to input the sign language data to be recognized into a pre-trained sign language recognition model, output the corresponding recognition result, which is text; and convert the text of the recognition result into speech output.

[0081] The sign language recognition device based on sensor data and deep learning provided in this invention inputs the acquired sign language data to be recognized into a sign language recognition model trained based on samples containing the target sign language via a serial port. It can automatically obtain the recognition result of the sign language to be recognized without the need for manual judgment of the sign language, thus realizing real-time recognition of sign language, which is of great significance for communication between deaf and mute people and the outside world.

[0082] In summary, this invention proposes a sign language recognition method and apparatus based on sensor data and deep learning. By designing, implementing, and training a sign language recognition model based on a skip-connection one-dimensional CNN and a multi-head attention mechanism module fused with BiLSTM, data collected by a smart glove equipped with an STM32 microcontroller, five FLEX2.2 bending sensors, and an MPU6050 six-axis sensor is transmitted to the model for recognition. The model's recognition results are then converted into speech for broadcast. To address the issues of visual sign language recognition being affected by the environment and the need to improve the accuracy of existing sensor-based sign language recognition methods, this invention provides a sign language recognition method and apparatus based on sensor data and deep learning.

[0083] The core of this method and device is to design an effective sign language recognition model. It adopts a one-dimensional CNN network model based on skip connections, integrates feature extraction results from different layers to improve the model's feature extraction capability, and designs a multi-head attention mechanism module that integrates BiLSTM to further extract key features of sign language while associating the internal temporal features of sign language. The improved multi-head attention mechanism is placed after the last convolutional layer of the CNN architecture, thereby combining the advantages of both to improve the accuracy of sign language recognition.

[0084] This invention proposes a sign language recognition method and device based on sensor data and deep learning. The method and device use a designed neural network model to effectively extract the features of dynamic sign language data collected by sensors, ensuring the accuracy of sign language recognition without being affected by the surrounding environment. The recognition results are then broadcast aloud, making it easier for deaf and mute people to communicate with the outside world without requiring external personnel to learn sign language.

[0085] In another aspect of the embodiments of this application, a computer storage medium is also provided.

[0086] This application also provides a computer-readable storage medium, which includes a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the above-described method. The specific implementation process will not be repeated here.

[0087] This application also provides a computer device. The computer device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the methods described above in the embodiment; to avoid repetition, these will not be elaborated further here. Alternatively, when the computer program is executed by the processor, it implements the functions of each model / unit in the device of the embodiment; to avoid repetition, these will not be elaborated further here.

[0088] Computer equipment can be desktop computers, laptops, handheld computers, servers, and cloud servers, etc. Computer equipment may include, but is not limited to, processors and memory. Those skilled in the art will understand that it may include more or fewer components than illustrated, or combine certain components, or different components; for example, computer equipment may also include input / output devices, network access devices, buses, etc.

[0089] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0090] Memory can be an internal storage unit of a computer device, such as a hard drive or RAM. Memory can also be an external storage device of a computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal and external storage units. Memory is used to store computer programs and other programs and data required by the computer device. Memory can also be used to temporarily store data that has been output or will be output.

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0092] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0093] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A sign language recognition method, characterized by, The method includes: The bending data of each finger is collected by a bending sensor, and the posture data of the hand is collected by a six-axis sensor. The bending data of each finger and the posture data of the hand are combined to form the sign language data to be recognized; The sign language data to be recognized is input into a pre-trained sign language recognition model, and the corresponding recognition result is output as text. Convert the text of the recognition results into speech output; The network architecture of the sign language recognition model includes: a one-dimensional convolutional neural network, and a multi-head attention mechanism module of a fused BiLSTM network connected to the one-dimensional convolutional neural network; The one-dimensional convolutional neural network includes a first part and a second part. The first part includes an input layer, a convolutional layer, and a max-pooling layer. The second part includes a max-pooling layer, a global average pooling layer, a fully connected layer, and an output layer. The multi-head attention mechanism module is connected between the first part and the second part. The first part of the one-dimensional convolutional neural network is used to extract data features of the sign language data to be recognized. The multi-head attention mechanism module is used to further extract temporal features and action features from the data features; The second part of the one-dimensional convolutional neural network is used to identify the recognition result corresponding to the sign language data to be recognized.

2. The sign language recognition method according to claim 1, characterized in that, The sign language recognition model is a neural network model that incorporates an attention mechanism.

3. The sign language recognition method according to claim 1, characterized in that, The method further includes the step of training the sign language recognition model.

4. The sign language recognition method according to claim 3, characterized in that, The steps for training the sign language recognition model include: Collect a set number of sign language samples containing the target sign language, and generate a text file for each sign language sample; Each sign language sample is labeled with its corresponding sign language type using its filename; Configure the network architecture of the sign language recognition model and initialize the network parameters; The labeled sign language samples are input into the sign language recognition model with initialized network parameters for training. The network parameters of the model are adjusted until the model converges, and the trained sign language recognition model is obtained.

5. A sign language recognition device for implementing the sign language recognition method as described in any one of claims 1 to 4, characterized in that, include: Bending sensor, six-axis sensor, processor, and intelligent recognition module; The bending sensor is used to collect bending data for each finger; The six-axis sensor is used to collect hand posture data; The processor is used to combine the bending data of each finger and the posture data of the hand into sign language data to be recognized. The intelligent recognition module is used to input the sign language data to be recognized into a pre-trained sign language recognition model and output the corresponding recognition result, which is text. The recognition results are then converted from text to speech output.

6. The sign language recognition device according to claim 5, characterized in that, The bending sensor is a FLEX2.2; the six-axis sensor is an MPU6050; and the processor is an STM32 microcontroller.

7. The sign language recognition device according to claim 5, characterized in that, The bending sensor, the six-axis sensor, and the processor are disposed in the smart glove, and the processor is connected to the bending sensor and the six-axis sensor respectively. The intelligent recognition module is used to deploy the sign language recognition module, recognize the recognition result corresponding to the sign language data to be recognized, and also to use the pyttsx3 library in Python to perform text-to-speech conversion; The intelligent identification module is installed in a fixed terminal or mobile terminal that is communicatively connected to the intelligent glove.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed, controls the device containing the computer-readable storage medium to perform the sign language recognition method according to any one of claims 1 to 4.

Images