Translation method based on convolutional neural network sign language translation glove
Through sign language translation gloves based on convolutional neural network, collecting and processing hand motion information, the problem of insufficient accuracy and real-time performance of existing sign language translation equipment is solved, and a high-precision and low-cost sign language translation solution is realized.
Patent Information
- Application Number
- CN202411710570.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-18
AI Technical Summary
The existing sign language translation equipment is not accurate and has poor real-time performance, which cannot meet the needs of efficient communication between deaf and mute people.
Sign language translation gloves based on convolutional neural network are adopted to collect hand movement information and finger bending information through attitude sensors and bending sensors, and train the convolutional neural network model to judge sign language information to achieve real-time translation.
It improves the accuracy and real-time nature of sign language translation, provides an efficient communication solution, and reduces equipment costs.
Smart Images
Figure CN119645227A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of embedded artificial intelligence, and in particular relates to a translation method based on convolutional neural network sign language translation gloves. Background Art
[0002] With the development of society and the advancement of science and technology, it has become particularly important to provide effective communication methods for special groups, such as the deaf-mute. Sign language, as an important non-verbal communication method, is crucial for the deaf-mute. However, the sign language translation equipment currently on the market often has problems such as low accuracy and poor real-time performance, and cannot meet the needs of efficient communication between the deaf-mute and the hearing people.
[0003] In recent years, the detection methods of sign language translation mainly rely on manual detection or automatic detection based on visual models. The manual detection method requires a lot of human resources, and the popularity of this method is limited due to the limited number of people who understand sign language. On the other hand, although the automatic detection method based on visual models has reduced the burden of human resources to a certain extent, its performance is often affected by factors such as the occlusion of hand movement details during sign language expression and excessive calculation, making it difficult to achieve a satisfactory level of detection accuracy. Therefore, the existing sign language translation detection technology still faces many challenges and urgently needs a more efficient and accurate solution. Summary of the invention
[0004] Purpose of the invention: The purpose of the present invention is to provide a translation method based on convolutional neural network sign language translation gloves. To solve the problem that there is no high-precision real-time sign language translation strategy in the current market. The gloves use posture sensors and bending sensors to collect overall hand movement information and local finger bending information, and judge sign language information by training convolutional neural networks to achieve real-time translation of sign language, providing a solution for communication for the deaf and mute.
[0005] Technical solution: A translation method based on a convolutional neural network sign language translation glove of the present invention comprises the following steps:
[0006] S1. Use sign language translation gloves to collect hand movement acceleration and finger bending movement information to quantify sign language movements;
[0007] S2, create a sign language dataset;
[0008] S3. Build and train a convolutional neural network model;
[0009] S4, deploy the trained convolutional neural network model on the STM32 microcontroller device;
[0010] S5. Use a convolutional neural network to detect the three-axis acceleration and three-axis angular velocity data of the hand to obtain the result of sign language translation.
[0011] Furthermore, the step S1 includes the following steps:
[0012] S11. Collect the overall movement information of the hand through the attitude sensor. The attitude sensor uses MPU6050. The SCL and SDA of MPU6050 are connected to the PB6 and PB7 ports of the STM32 microcontroller respectively. The recorded three-axis acceleration and three-axis gyroscope data of the hand are transmitted using the I2C communication protocol.
[0013] S12. The bending sensor is used to collect the finger bending movement information. The flex2.2 sensor based on the resistive carbon element has a normal resistance when not bent, increases when bent at 45°, and continues to increase when bent at 90°. The calculation formula of the bending sensor voltage is:
[0014]
[0015] The data formula for analog voltage output in STM32 is as follows:
[0016]
[0017] Among them, R1 refers to the current resistance value of the bending sensor, R2 is the resistor used for voltage division, V in It represents an input voltage of 5 volts, V out is the voltage output of the bending sensor, A0 refers to the analog voltage signal obtained by the sensor on the control unit;
[0018] S13. Use sensors to collect finger movement parameters involved in sign language expression, build a finger movement model, and quantify sign language movements.
[0019] Furthermore, the step S2 includes the following steps:
[0020] S21. Daily sign language gestures consist of two parts: finger movements and overall hand movements. Use bending sensors to collect finger movement information, and use posture sensors to collect overall hand movement information. When making a data set, let different experimenters wear sign language translation gloves, select 50-60 common sign language gestures, let the experimenters repeat each sign language gesture 20-30 times, and keep the collected sensor information locally.
[0021] S22. Save each set of data in a txt file. Each set of data includes six values, namely acceleration X-axis, acceleration Y-axis, acceleration Z-axis, angular velocity X-axis, angular velocity Y-axis, and angular velocity Z-axis. Check whether there are abnormal values or noise in the data, and clean the data with obvious collection errors.
[0022] Furthermore, the step S3 includes the following steps:
[0023] S31. Use TensorFlow and Keras libraries to build and train a convolutional neural network (CNN) for action recognition data collected from sensors.
[0024] S32. Construct the input layer of the convolutional neural network. The input layer is the first layer of the neural network. It receives the raw data input and passes it to the subsequent layers of the network. In the convolutional neural network CNN, the input layer matches the data shape expected by the network. For the CNN that processes time series data, the input data is three-dimensional, including: time step, number of features, and channel; the time step is the length of the sequence, indicating how many consecutive data points there are in the time series; the number of features indicates the number of features contained in each time step; the channel usually represents the color channel (such as RGB) in image processing. When processing univariate time series data, this dimension is 1; the network sets the time step to 150 and the number of features to 6, which corresponds to the three-axis acceleration and three-axis angular velocity data collected by the attitude sensor;
[0025] S33. Construct the convolutional layer of the convolutional neural network. The convolutional layer is the key part of feature extraction. The convolutional layer uses convolution operations to identify local patterns and features in the input data. The convolution operation involves a small matrix called a convolution kernel or filter, which slides over the input data and calculates the sum of the product of the convolution kernel and the input data at each position; this process generates a new two-dimensional array, called a feature map, which represents the activation level of a specific feature in the input data. The formula for the output feature map is:
[0026]
[0027] In the above formula, O l and O w are the row and column indices of the output feature map, respectively, and f l and f w are the row and column indices of the convolution kernel, X is the input data, K is the convolution kernel, and C is the number of channels of the input data. In addition, Stride and padding will also affect the size of the output feature map;
[0028] S34. Construct the flattening layer and fully connected layer of the convolutional neural network, which play different roles in the network. The main function of the flattening layer is to convert the multi-dimensional feature map output by the convolutional layer into a one-dimensional feature vector. The purpose of this is to input these features into the fully connected layer for further processing, while also reducing the number of network parameters, thereby simplifying the model structure. The fully connected layer is another important component in CNN. It is usually located at the end of the network and is used to integrate the features extracted by the convolutional layer and the pooling layer, and learn the complex nonlinear mapping relationship between the input features and the output for the final classification task. The flattening layer and the fully connected layer play a connecting role in CNN. The flattening layer is responsible for converting the multi-dimensional output of the convolutional layer into a one-dimensional feature vector, while the fully connected layer is responsible for integrating these features for the final classification or regression task. The combination of these two layers enables CNN to handle a variety of complex visual and sequence processing tasks;
[0029] S35. Construct the output layer of the convolutional neural network. The output layer is the last layer of the model. This network uses a fully connected layer followed by a Softmax activation function. The Softmax activation function is usually used for the output layer of multi-class classification problems. It converts the raw output of the fully connected layer (also called logits or scores) into a probability distribution so that the sum of the probabilities of all categories is 1.
[0030] Furthermore, in step S33, the convolutional neural network uses two convolutional layers. The first convolutional layer has 30 convolution kernels, each of which has a length of 3 and a step size of 3. The second convolution layer has 15 convolution kernels, each of which has a size of 3 and a step size of 3. The mathematical formula of the second convolution layer is similar to that of the first layer, but the number of filters and the size of the feature map are different. The second convolution layer can combine the features learned in the first layer to extract more complex features; at the same time, it has the ability to reduce dimensionality. Through step size and pooling operations, the second convolution layer can further reduce the spatial dimension of the data and reduce the amount of calculation. Through this hierarchical feature extraction and combination, the convolutional neural network can learn rich feature representations from the original data, thereby achieving excellent performance in this sequence processing task.
[0031] Further, in step S35, the Softmax activation function expression is:
[0032]
[0033] In the above formula, Z is the output of the fully connected layer, that is, the linear score vector, Z i is the i-th element in the score vector, e is the base of the natural logarithm, Softmax(Z) i is the probability of the i-th category after conversion, Z jRepresents the jth element in the linear score vector Z output by the fully connected layer.
[0034] For multi-class classification problems, Softmax ensures that the output can be interpreted as probabilities, so the model can predict the probability that the input data belongs to each category. In the network, the fully connected layer calculates the score of the input features for each category, and the Softmax activation function converts these scores into a probability distribution so that the output can be interpreted as the model's predicted probability for each category. Ultimately, the output of the model is a probability distribution, where each element represents the probability that the input data belongs to the corresponding category. During training, the goal of the model is to maximize the probability of the correct category. When predicting, the category with the highest probability is selected as the model's predicted output.
[0035] Further, the step S4 includes the following steps:
[0036] S41. Balance the performance and cost of the microcontroller, select STM32F103C8T6 as the controller, configure the engineering environment in Keil5 software, write the configuration file, define the system clock frequency as 72MHz, define the serial port baud rate as 115200, define the parameters required by the inertial measurement unit MPU6050, and define the threshold of the convolutional neural network output for action recognition;
[0037] S42. Write the functions of sensor data collection and convolutional neural network in different files respectively, write the main program file, call other header files and weight files of the neural network model, define an enumeration type eModel_Output, and list all possible model outputs, including Unrecognized and other specific actions; set static memory and allocate a static buffer static_buf for neural network operations.
[0038] Furthermore, the step S5 includes the following steps:
[0039] S51. The user puts on the sign language translation gloves and demonstrates the sign language movements. The model detects the three-axis acceleration and three-axis angular velocity based on the collected finger bending information and the overall hand movement information, and returns the category with the highest probability as the model's prediction output;
[0040] S52. The output information is printed on the serial port, which facilitates real-time communication between the speech-impaired and others.
[0041] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:
[0042] 1. The present invention adopts convolutional neural network technology to detect the three-axis acceleration and three-axis angular velocity collected by the posture sensor. Compared with the existing sign language translation detection scheme based on the visual model, it will not be subject to problems such as hand occlusion. The present invention has higher accuracy and smaller calculation amount, and provides a high-precision sign language translation scheme.
[0043] 2. The present invention is based on the STM32F103C8T6 microcontroller. Compared with the expensive computing platform required by the visual detection model to support its huge computing power, the present invention trains and detects sequence data, and the network structure is simpler and the computing power is smaller. The equipment cost is much lower than the sign language translation solution based on visual detection, which is convenient for subsequent product promotion.
[0044] 3. As a deep learning model, convolutional neural network (CNN) has achieved remarkable results in the fields of image recognition and video analysis. Through its powerful feature extraction capability, CNN can directly learn useful feature representations from input data without manual intervention in the feature selection and extraction process. This provides new possibilities for the automation of sign language translation. Compared with the traditional vision-based sign language recognition method, the sign language translation glove strategy based on CNN training of time series data has higher accuracy and less computational complexity, thereby improving the accuracy and real-time performance of sign language translation. Therefore, the present invention adopts convolutional neural network as the core algorithm to achieve higher sign language translation accuracy and better real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 This is a functional flow chart of a sign language translation glove based on convolutional neural network;
[0046] Figure 2 An example diagram of a typical dataset used for convolutional neural network training;
[0047] Figure 3 This is the structure diagram of the convolutional neural network deployed on STM32F103C8T6;
[0048] Figure 4 Physical design drawing of sign language translation gloves.
[0049] Reference numerals: 1. bending sensor data acquisition module; 2. bending sensor data processing module; 3. MPU6050 posture sensor. DETAILED DESCRIPTION
[0050] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.
[0051] See also Figure 1 and Figure 4The technical solution provided by this embodiment is a sign language translation glove based on a convolutional neural network, comprising the following steps:
[0052] S1, collection of hand movement acceleration and finger bending motion information to quantify sign language movements;
[0053] The MPU6050 attitude sensor is used to collect the overall movement information of the hand and obtain the three-axis acceleration and three-axis angular velocity. An important reason for choosing the MPU6050 attitude sensor is that it is small in size and easy to embed into the sign language translation gloves. The SCL and SDA of the MPU6050 are connected to the PB6 and PB7 ports of the STM32 microcontroller respectively, and the recorded hand three-axis acceleration and three-axis gyroscope data are transmitted using the I2C communication protocol;
[0054] Finger bending movement information is collected through the bending sensor. The flex2.2 sensor based on resistive carbon elements has normal resistance when not bent, increases resistance when bent at 45°, and continues to increase resistance when bent at 90°.
[0055] S2, create a sign language dataset;
[0056] Most of the sign language gestures used include two parts: the movement of the fingers and the movement of the whole hand. The bending sensor is used to collect the movement information of the fingers, and the posture sensor is used to collect the movement information of the whole hand. When making the data set, 5 different experimenters were asked to wear sign language translation gloves, 60 common sign language gestures were selected, and the experimenters were asked to repeat each sign language action 20 times, and the collected sensor information was saved locally. Each set of data includes six values, namely "acceleration X axis, acceleration Y axis, acceleration Z axis, angular velocity X axis, angular velocity Y axis, angular velocity Z axis". Check whether there are outliers or noise in the data, and clean the data with obvious collection errors.
[0057] S3. Build and train a convolutional neural network model;
[0058] Construct a convolutional neural network, set the time step of the input layer to 150, the number of features to 6, and the number of channels to 1; build two convolutional layers, the first convolutional layer has 30 convolution kernels, each of which has a length of 3 and a step of 3, and the second convolutional layer has 15 convolution kernels, each of which has a size of 3 and a step of 3, using the Relu activation function; build a flattening layer and an output layer, and the output layer uses a fully connected layer followed by a Softmax activation function.
[0059] S4, deploy the trained convolutional neural network model on the STM32 microcontroller device;
[0060] Configure the STM32 firmware library and development environment, use Arm Compiler 6.22, download CMSIS6.0.0, CMSIScompiler 2.1.0, Stm32F1xx DFP2.4.1, use STlink to burn the code to the STM32F103C8T6 microcontroller, and run to check the correctness of the configuration.
[0061] S5. Use a convolutional neural network to detect the six-axis acceleration and obtain the result of sign language translation;
[0062] Finally, the images and videos to be detected are input into the trained model for inference and prediction, and the results of sign language translation detection are obtained and output.
[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A translation method based on convolutional neural network sign language translation gloves, characterized in that: The steps include: S1. Use sign language translation gloves to collect hand movement acceleration and finger bending movement information to quantify sign language movements; S2, create a sign language dataset; S3. Build and train a convolutional neural network model; S4, deploy the trained convolutional neural network model on the STM32 microcontroller device; S5. Use a convolutional neural network to detect the three-axis acceleration and three-axis angular velocity data of the hand to obtain the result of sign language translation.
2. According to claim 1, a translation method based on convolutional neural network sign language translation gloves is characterized in that: The step S1 comprises the following steps: S11. Collect the overall movement information of the hand through the attitude sensor. The attitude sensor uses MPU6050. The SCL and SDA of MPU6050 are connected to the PB6 and PB7 ports of the STM32 microcontroller respectively. The recorded three-axis acceleration and three-axis gyroscope data of the hand are transmitted using the I2C communication protocol. S12. The bending sensor is used to collect the finger bending movement information. The flex2.2 sensor based on the resistive carbon element has a normal resistance when not bent, increases when bent at 45°, and continues to increase when bent at 90°. The calculation formula of the bending sensor voltage is: The data formula for analog voltage output in STM32 is as follows: Among them, R1 refers to the current resistance value of the bending sensor, R2 is the resistor used for voltage division, V in It represents an input voltage of 5 volts, V out is the voltage output of the bending sensor, A0 refers to the analog voltage signal obtained by the sensor on the control unit; S13. Use sensors to collect finger movement parameters involved in sign language expression, build a finger movement model, and quantify sign language movements.
3. The translation method based on convolutional neural network sign language translation gloves according to claim 1 is characterized in that: The step S2 comprises the following steps: S21. Daily sign language gestures consist of two parts: finger movements and overall hand movements. Use bending sensors to collect finger movement information, and use posture sensors to collect overall hand movement information. When making a data set, let different experimenters wear sign language translation gloves, select 50-60 common sign language gestures, let the experimenters repeat each sign language gesture 20-30 times, and keep the collected sensor information locally. S22. Save each set of data in a txt file. Each set of data includes six values, namely acceleration X-axis, acceleration Y-axis, acceleration Z-axis, angular velocity X-axis, angular velocity Y-axis, and angular velocity Z-axis. Check whether there are abnormal values or noise in the data, and clean the data with obvious collection errors.
4. The translation method based on convolutional neural network sign language translation gloves according to claim 1 is characterized in that: The step S3 comprises the following steps: S31. Use TensorFlow and Keras libraries to build and train a convolutional neural network (CNN) for action recognition data collected from sensors. S32. Construct the input layer of the convolutional neural network. The input layer is the first layer of the neural network. It receives the raw data input and passes it to the subsequent layers of the network. In the convolutional neural network CNN, the input layer matches the data shape expected by the network. For CNN that processes time series data, the input data is three-dimensional, including: time step, number of features, and channel; the time step is the length of the sequence, indicating how many consecutive data points there are in the time series; the number of features indicates the number of features contained in each time step; the channel usually represents the color channel in image processing, and this dimension is 1 when processing univariate time series data; the network sets the time step to 150 and the number of features to 6, which corresponds to the three-axis acceleration and three-axis angular velocity data collected by the attitude sensor; S33. Construct the convolutional layer of the convolutional neural network. The convolutional layer uses convolution operations to identify local patterns and features in the input data. The convolution operation involves a small matrix called a convolution kernel or filter, which slides over the input data and calculates the sum of the element-wise products of the convolution kernel and the input data at each position; this process generates a new two-dimensional array, called a feature map, which represents the degree of activation of a specific feature in the input data. The formula for the output feature map is: In the above formula, O l and O w are the row and column indices of the output feature map, respectively, and f l and f w are the row and column indices of the convolution kernel, X is the input data, K is the convolution kernel, and C is the number of channels of the input data; S34. Construct the flattening layer and fully connected layer of the convolutional neural network. The flattening layer converts the multi-dimensional feature map output by the convolutional layer into a one-dimensional feature vector. The fully connected layer is located at the end of the network and is used to integrate the features extracted by the convolutional layer and the pooling layer, and learn the complex nonlinear mapping relationship between the input features and the output to perform the final classification task. The flattening layer is responsible for converting the multi-dimensional output of the convolutional layer into a one-dimensional feature vector, while the fully connected layer is responsible for integrating these features to perform the final classification or regression task. S35. Construct the output layer of the convolutional neural network. The output layer is the last layer of the model. The convolutional neural network uses a fully connected layer followed by a Softmax activation function. The Softmax activation function converts the original output of the fully connected layer into a probability distribution so that the sum of the probabilities of all categories is 1.
5. The translation method based on convolutional neural network sign language translation gloves according to claim 4 is characterized in that: In step S33, the convolutional neural network adopts two convolutional layers. The first convolutional layer has 30 convolution kernels, each of which has a length of 3 and a step size of 3; the second convolution layer has 15 convolution kernels, each of which has a size of 3 and a step size of 3.
6. The translation method based on convolutional neural network sign language translation gloves according to claim 4 is characterized in that: In step S35, the Softmax activation function expression is: In the above formula, Z is the output of the fully connected layer, that is, the linear score vector, Z i is the i-th element in the score vector, e is the base of the natural logarithm, Softmax(Z) i is the probability of the i-th category after conversion, Z j Represents the jth element in the linear score vector Z output by the fully connected layer.
7. The translation method based on convolutional neural network sign language translation gloves according to claim 1 is characterized in that: The step S4 comprises the following steps: S41. Balance the performance and cost of the microcontroller, select STM32F103C8T6 as the controller, configure the engineering environment in Keil5 software, write the configuration file, define the system clock frequency as 72MHz, define the serial port baud rate as 115200, define the parameters required by the inertial measurement unit MPU6050, and define the threshold of the convolutional neural network output for action recognition; S42. Write the functions of sensor data collection and convolutional neural network in different files respectively, write the main program file, call other header files and weight files of the neural network model, define an enumeration type eModel_Output, and list all possible model outputs, including Unrecognized and other specific actions; set static memory and allocate a static buffer static_buf for neural network operations.
8. The translation method based on convolutional neural network sign language translation gloves according to claim 1 is characterized in that: The step S5 comprises the following steps: S51. The user puts on the sign language translation gloves and demonstrates the sign language movements. The model detects the three-axis acceleration and three-axis angular velocity based on the collected finger bending information and the overall hand movement information, and returns the category with the highest probability as the model's prediction output; S52. The output information is printed on the serial port, so that the speech-impaired person can communicate with others in real time.