Intelligent sign language translator based on convolutional neural network and translation method thereof

Abstract

The invention relates to an intelligent sign language translator based on a convolutional neural network and a translation method thereof, which are used for converting sign language actions into spoken languages or text output in real time, and comprise a sign language recognition module for capturing and analyzing the gesture actions by adopting the convolutional neural network CNN and converting sign languages into data which can be processed by a computer; the translation processing module is used for receiving the data from the sign language recognition module by adopting a sequence-to-sequence seq2seq model and processing the data to generate corresponding spoken language or character translation; and the interactive output module adopts a graphical user interface (GUI) and a speech synthesis technology and is used for displaying translation results in visual and auditory forms. According to the method, a deep learning model is adopted, visual features, gesture annotations and time sequence mapping of generated statements are directly learned, and feature correlation, entropy regularization of the gesture annotations and probability maximization of sentence decoding are measured in an end-to-end framework by using a joint loss optimization function; and the performance superior to that of other online translation models is realized under the condition of not introducing extra supervision.

Description

Technical Field

[0001] The present application relates to sign language translation, and in particular, to an intelligent sign language translator based on a convolutional neural network and a translation method thereof. Background Art

[0002] There are currently 430 million deaf people in the world, accounting for 5% of the total population. It is expected that this number will increase to 700 million by 2050, leading to a growing demand for sign language translators. This device can convert sign language into speech or other languages, greatly facilitating the communication and integration of people with hearing and speech impairments in education, medical care and administration.

[0003] The development of sign language translation software not only promotes communication within and outside the deaf community, but also provides them with broader social support and helps eliminate communication barriers. The significance of technological innovation is to draw social attention to the problems of the deaf and enhance care for them. Domestic and foreign research has focused on sign language translation technology. Foreign countries mainly use computer vision and machine learning for real-time recognition, although there are still challenges in accuracy in complex environments. Although domestic research started late, there has been some progress.

[0004] In summary, there are still major problems with sign language interpretation technology, as follows:

[0005] (1) The sign language vocabulary database is incomplete, not rich in variety, has low discrimination and large errors;

[0006] (2) High reliance on computers and poor convenience of use;

[0007] (3) It is difficult to accurately recognize sign language using image capture technology alone;

[0008] (4) The research subjects are narrow and the feedback is insufficient, which is not conducive to improvement.

[0009] Aiming at the communication needs and social integration barriers faced by the deaf-mute population, the present invention designs and implements a sign language translator based on a convolutional neural network (CNN) and a sequence-to-sequence (seq2seq) model. Summary of the invention

[0010] This application solves the technical problems raised in the background technology and discloses an intelligent sign language translator based on convolutional neural network and its translation method. The translator includes three main modules: sign language recognition module, translation processing module and interactive output module. A 3D convolutional neural network is used to process continuous frame actions, a seq2seq model is used to realize the conversion from sign language to text, and a graphical user interface and speech synthesis technology are used to optimize the user experience.

[0011] In order to solve the above technical problems, the technical solution provided by the present application is: an intelligent sign language translator based on a convolutional neural network, which is used to convert sign language movements into spoken or text output in real time, including:

[0012] The sign language recognition module uses a convolutional neural network (CNN) to capture and analyze gestures and convert sign language into computer-processable data.

[0013] The translation processing module uses a sequence-to-sequence seq2seq model to receive data from the sign language recognition module and process it to generate corresponding spoken or text translation;

[0014] The interactive output module uses a graphical user interface (GUI) and speech synthesis technology to display translation results in visual and auditory forms.

[0015] Furthermore, the sign language recognition module also includes a submodule for video data preprocessing, which performs processing including grayscale, normalization, data enhancement and size adjustment to adapt to the input requirements of the CNN model.

[0016] Furthermore, the data enhancement process also includes adjusting the brightness and contrast of sign language videos captured in different environments to improve the recognition accuracy of the model under various lighting conditions.

[0017] Furthermore, the translation processing module also includes a feature extraction submodule, which uses statistical and machine learning methods including chi-square test, mutual information and principal component analysis to select and optimize the feature set for training the seq2seq model from the preprocessed data.

[0018] Furthermore, the interactive output module also includes an adaptive user interface design submodule, which adjusts the display and playback parameters of text and voice output according to the user's settings to meet the user's personalized needs.

[0019] Furthermore, the interactive output module is also provided with a machine learning feedback mechanism for adjusting and optimizing the performance of each module according to the accuracy feedback of the user input.

[0020] A translation method using an intelligent sign language translator, the method comprising the following steps:

[0021] S1, capturing sign language movements through an intelligent sign language translator;

[0022] S2, converting the captured sign language movements into data through a sign language recognition module;

[0023] S3, sending the data to the translation processing module to generate corresponding spoken language or text;

[0024] S4. Display the translation results to the user through the interactive output module.

[0025] The present application has the following advantages: The present invention creates a mobile sign language translation system that enables hearing-impaired people to use it more conveniently, reduces dependence on computers, and creates a good communication environment. The present invention uses a deep learning model to directly learn the temporal mapping of visual features, gesture annotations, and generated sentences, and uses a joint loss optimization function to measure feature correlation, entropy regularization of gesture annotations, and probability maximization of sentence decoding in an end-to-end framework, achieving performance that is superior to other online translation models without introducing additional supervision. In addition, the present invention stores sign language databases in categories, calls synthetic text in partitions, expands the scope of application, is oriented to the general public, and is committed to collecting more comprehensive feedback.

[0026] Experimental results show that the translator performs well in terms of accuracy and practicality, and can effectively promote communication between hearing-impaired and non-hearing-impaired people. DETAILED DESCRIPTION

[0027] The present application is further described in detail below in conjunction with embodiments.

[0028] In the specific implementation of this application, an algorithm design and implementation of an intelligent sign language translator based on a convolutional neural network:

[0029] 1. Design ideas of sign language translator

[0030] The sign language translator designed and implemented by the present invention has the function of accurately recognizing sign language and translating it into spoken language or text in real time, thereby promoting communication between hearing-impaired and non-hearing-impaired people. The translator consists of three main modules: sign language recognition module, translation processing module and interactive output module, as follows:

[0031] 1. Sign language recognition module: The core algorithm is based on convolutional neural network (CNN). This module is responsible for capturing and analyzing gestures, converting complex sign language movements into computer-understandable data, and providing it to the subsequent translation processing module.

[0032] 2. Translation processing module: Designed using a sequence-to-sequence (seq2seq) model, this module receives data from the sign language recognition module, processes it through a deep neural network, and generates corresponding spoken or written translations to ensure the accuracy and real-time nature of the translation.

[0033] 3. Interactive output module: mainly uses graphical user interface (GUI) and speech synthesis technology. This module is responsible for presenting the translation results to users in a clear and easy-to-understand way, whether through text displayed on the screen or voice output through the speaker.

[0034] Aiming at a specific application scenario - daily communication with the hearing-impaired, the adaptability of sign language recognition to various indoor and outdoor environments is taken into consideration. After comparing several deep learning models, CNN was selected as the core algorithm of the sign language recognition module due to its superior performance in image recognition. The main purpose of the translation processing module is to map the recognized sign language movements to the corresponding language expressions, so the seq2seq model that can process serialized data is selected here. The interactive output module combines GUI design and text-to-speech (TTs) technology to ensure the intuitiveness and convenience of information transmission.

[0035] To test the effect of the translator, the present invention collects sign language video data under different environments and lighting conditions. By comparing the real translation results with the system output, the accuracy and practicality of the system are evaluated. On this basis, necessary model training optimization and function adjustment are carried out to meet the actual application needs.

[0036] 2. Data Collection and Data Preprocessing

[0037] The data set used in the present invention is a Chinese sign language database, covering 4414 standard Chinese sign language vocabularies, and a total of 331,050 vocabulary data of 30 individuals (13 males and 17 females). Each vocabulary data consists of RGB, depth and skeleton information. The entire data set can be divided into three subsets. The first subset consists of 26 letters and 10 numbers, the second subset consists of 500 daily vocabulary groups, and the third subset is a large vocabulary set. The data set contains 2000 Chinese SL vocabularies. Data preprocessing is an indispensable part of the present invention before training the model. Its purpose is to make the original data more regular and clear, so as to facilitate subsequent data analysis and image prediction. In the present invention, data preprocessing generally includes steps such as image preprocessing, resizing, and dividing the data set:

[0038] Preprocessing - grayscale, normalize, and perform data augmentation on the collected images to improve the training effect of the model.

[0039] Resize - resize all images to the uniform size required for CNN model input. Divide the dataset - divide the preprocessed images into training and test sets.

[0040] Storage format conversion: Save the image in a format that can be read by the CNN model and save the image path in CSV file format.

[0041] Depth images can effectively remove noise interference such as lighting and color in the background environment, allowing the model to more smoothly obtain distinguishing features under the depth image modality and improve the model recognition accuracy.

[0042] It is important to note that when performing data preprocessing, you also need to pay attention to the quality and integrity of the data. If the data has serious quality problems or too many missing data, it will be difficult to obtain accurate analysis results even with more sophisticated data preprocessing. Therefore, before performing data analysis, the quality and integrity of the data need to be fully evaluated and cleaned.

[0043] 3. Feature extraction and model training

[0044] The present invention uses a machine learning algorithm to automatically learn the features of raw data. First, data preprocessing is performed, including cleaning, normalization and standardization to eliminate noise and redundant information. Then, the most relevant and representative features are selected through statistical methods (such as chi-square test, mutual information) or model methods (such as decision trees, random forests). Then, the selected features are transformed to generate new features with more expressive power. Finally, principal component analysis is used for dimensionality reduction to transform the raw data into a linearly independent representation to better grasp the main characteristic components of the data.

[0045] In the first step, we prepare the training dataset, including input features and target variables, perform data cleaning and preprocessing, such as handling missing values, outliers, noise, feature extraction and normalization, and divide the dataset into training set, validation set and test set. We select convolutional neural network (CNN) and sequence to sequence (seq2seq) models for the sign language translation system.

[0046] In the second step, use the training data set to train the model and use an optimization algorithm (such as gradient descent) to update the model parameters to minimize the loss function. Record model performance indicators, such as accuracy and loss value, for subsequent evaluation and adjustment. After training, use the validation set to evaluate the model performance, calculate indicators such as accuracy, recall, and F1 value, and adjust the parameters based on the results to optimize the model performance. Finally, save the trained model for subsequent use.

[0047] The third step is to convert the gesture recognition data into a format acceptable to the model and input it into the trained model for prediction or classification. Output the model prediction results in a user-friendly way, such as displaying image classification results, generating voice responses, etc. Collect user feedback, continuously improve and optimize the model, and improve the practicality and accuracy of the algorithm.

[0048] 4. Experimental Results and Analysis

[0049] The loss function is of great significance in deep learning, mainly reflected in the following aspects:

[0050] Guiding model learning: The loss function provides the model with a learning direction. By minimizing the loss function, the model can gradually learn the inherent laws and characteristics of the data.

[0051] Evaluate model generalization ability: The loss function can not only evaluate model performance on the training set, but also evaluate the generalization ability of the model on the validation set and test set. If the model has a low loss value on the validation set and test set, it means that the model has good generalization ability.

[0052] Comparing different models: By comparing the loss values ​​of different models, you can evaluate the performance differences of these models on a specific task and thus select the model that best suits the current task.

[0053] The accuracy of the model established in the present invention gradually improved during the training process, and finally reached 100% validation set accuracy at the 60th epoch. The value of the loss function fluctuated greatly at the beginning, but as time went on, the loss value gradually decreased and stabilized. Although the loss value increased in some epochs, the overall performance of the model was steadily improved.

[0054] The learning rate has a complex control effect on the effective capacity of the model. A suitable learning rate can ensure that the model learns quickly while avoiding overfitting, thereby achieving good generalization on the validation set. This is because the appropriate update step size allows the model to gradually explore better solutions in the weight space, rather than stagnating in a local minimum or saddle point. Improper learning rate settings may cause oscillations during model training, that is, the loss value repeatedly increases and decreases, affecting the stability of the model. A suitable learning rate helps the model converge smoothly to the optimal solution.

[0055] The learning rate gradually decreases with each epoch, from the initial 0.001 to about 0.0005. After the 42nd epoch, the training speed of the model slows down, which may be due to the slow convergence caused by the reduction in learning rate. In summary, the model shows good learning ability and stability during training, and finally achieves a high verification accuracy.

[0056] Although the present invention has been shown and described primarily with reference to certain embodiments, it will be appreciated by those skilled in the art that numerous changes may be made thereto in terms of construction scheme and details, etc., without departing from the scope of protection defined by the claims. Thus, the scope of protection of the present invention is determined by the claims and includes all changes that fall within the meaning or equivalent scope of the claims.

Claims

1. An intelligent sign language translator based on convolutional neural network, characterized in that: Used to convert sign language movements into spoken or text output in real time, including: The sign language recognition module uses a convolutional neural network (CNN) to capture and analyze gestures and convert sign language into computer-processable data. The translation processing module uses a sequence-to-sequence seq2seq model to receive data from the sign language recognition module and process it to generate corresponding spoken or text translation; The interactive output module uses a graphical user interface (GUI) and speech synthesis technology to display translation results in visual and auditory forms.

2. The intelligent sign language translator based on convolutional neural network according to claim 1, characterized in that: The sign language recognition module also includes a submodule for video data preprocessing, which performs processing including grayscale, normalization, data enhancement and size adjustment to meet the input requirements of the CNN model.

3. The intelligent sign language translator based on convolutional neural network according to claim 2, characterized in that: The data enhancement process also includes adjusting the brightness and contrast of sign language videos captured in different environments to improve the recognition accuracy of the model under various lighting conditions.

4. The intelligent sign language translator based on convolutional neural network according to claim 1, characterized in that: The translation processing module also includes a feature extraction submodule, which uses statistical and machine learning methods including chi-square test, mutual information and principal component analysis to select and optimize the feature set for training the seq2seq model from the preprocessed data.

5. The intelligent sign language translator based on convolutional neural network according to claim 1, characterized in that: The interactive output module also includes an adaptive user interface design submodule, which adjusts the display and playback parameters of text and voice output according to the user's settings to meet the user's personalized needs.

6. The intelligent sign language translator based on convolutional neural network according to claim 5, characterized in that: The interactive output module is also equipped with a machine learning feedback mechanism to adjust and optimize the performance of each module based on the accuracy feedback of user input.

7. A translation method using the intelligent sign language translator according to any one of claims 1 to 6, the method comprising the following steps: S1, capturing sign language movements through an intelligent sign language translator; S2, converting the captured sign language movements into data through a sign language recognition module; S3, sending the data to the translation processing module to generate corresponding spoken language or text; S4. Display the translation results to the user through the interactive output module.