Method for constructing efficient sign language translation system based on deep learning

Abstract

According to the method for constructing the efficient sign language translation system based on deep learning, the accuracy and efficiency of sign language translation are remarkably improved through improvement in the aspects of data enhancement and preprocessing, model architecture optimization, real-time translation and interaction, system robustness and adaptability and the like. Through the application of the multi-modal deep learning model and in combination with the computer vision and natural language processing technology, the sign language can be better understood and translated, and diversified communication requirements of users are met. In addition, due to the self-adaptive learning ability and the multi-language support function of the system, the system can be continuously optimized and adapt to the requirements of different users, and the system has wide application prospects and social value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for constructing an efficient sign language translation system based on deep learning. Background Art

[0002] Sign language translation for the deaf and mute is of great significance in daily communication. However, existing sign language translation systems face problems such as low sign language recognition accuracy, slow translation speed, insufficient system training data, and high model complexity. The accuracy of sign language recognition is often affected by factors such as gesture changes, ambient lighting, and background interference, resulting in unreliable translation results. In addition, when processing complex gestures, existing systems often take a long time to translate, affecting the user's communication experience. With the development of deep learning technology, although some new methods have been proposed, there are still challenges such as insufficient model training data and high model complexity, which limit the widespread application of sign language translation systems. Therefore, a new solution is urgently needed to improve the accuracy and efficiency of sign language translation. Summary of the invention

[0003] The present invention provides a method for constructing an efficient sign language translation system based on deep learning, which is used to solve the above-mentioned problems.

[0004] The present invention provides the following technical solutions: A method for building an efficient sign language translation system based on deep learning is implemented by the following steps: Step 1: Data collection and annotation: Use multi-source data to collect sign language videos, including public data sets and videos voluntarily uploaded by users, and combine expert annotation and crowdsourcing annotation to ensure the accuracy and diversity of the data. Specifically, it includes the following steps: 1. Data source selection: First, select multiple data sources to collect sign language videos. These data sources include: – Public datasets: Use existing sign language video datasets, such as RWTH-PHOENIX-Weather, SignLanguage Recognition Dataset, etc. These datasets are usually verified by professional organizations and have high quality;

[0005] – Voluntary upload by users: By developing a user-friendly application, deaf users or sign language users are encouraged to upload their sign language videos. To protect user privacy, the system will ensure that all uploaded videos are anonymized;

[0006] 2. Data labeling: In order to ensure the accuracy and diversity of the data, a combination of expert labeling and crowdsourcing labeling is used: – Expert annotation: Invite sign language experts to annotate the collected videos to ensure that every gesture, action and expression is accurately identified and described. The experts will use standardized annotation tools to annotate the sign language content in each video and provide corresponding text descriptions.

[0007] – Crowdsourcing annotation: At the same time, crowdsourcing platforms (such as Amazon Mechanical Turk) are used for preliminary annotation. By setting clear annotation guidelines, ordinary users can participate in the annotation process and increase the diversity of data. The results of crowdsourcing annotation will be reviewed by experts to ensure the accuracy of the annotation.

[0008] 3. Data quality control: Strict quality control measures are implemented during the data collection and annotation process. Regular sampling checks are conducted on the annotation results to ensure the consistency and accuracy of the annotations. In addition, automated tools are used to conduct preliminary analysis of the videos to screen out videos of low quality or substandard videos to improve the overall quality of the dataset.

[0009] Through the above steps, the system can build a high-quality and diverse sign language video dataset, providing a solid foundation for subsequent model training.

[0010] Step 2: Data enhancement: Use data enhancement techniques such as rotation, scaling, flipping, etc. to increase the diversity of training data and improve the generalization ability of the model, thereby improving the accuracy and robustness of sign language recognition. The specific implementation method includes the following steps: 1. Rotation: Rotate the sign language video at random angles to simulate different shooting angles. Specifically, the system will randomly select an angle (for example, any angle between -30 degrees and 30 degrees) and then rotate the video frame. This method can help the model learn how gestures perform under different viewing angles and enhance its ability to recognize gestures.

[0011] 2. Scaling: By randomly scaling the video, the size of the gesture is changed to enhance the model's ability to recognize gestures at different distances. The system will randomly select a scaling factor (for example, any value between 0.8 and 1.2) and scale each frame. In this way, the model can adapt to the changes in gestures at different distances and improve its performance in practical applications.

[0012] 3. Flip: Flip the video horizontally or vertically to increase the diversity of gestures. The system will randomly decide whether to flip the video, and the flipped video will be regarded as a new training sample. This method can help the model learn the symmetry of gestures and further improve its recognition ability.

[0013] 4. Time warping: Random time warping is performed on the timeline of the video to simulate different speeds of gestures. Specifically, the system will randomly select certain frames for acceleration or deceleration to change the playback speed of the gesture. This method can help the model adapt to the gesture expression speed of different users and improve its recognition ability of dynamic gestures.

[0014] 5. Noise addition: Add random noise to the video to enhance the model's robustness to environmental interference. The system will perform noise processing on the video frames, such as adding Gaussian noise or salt and pepper noise. This method can help the model better cope with different shooting environments in practical applications and improve its recognition accuracy.

[0015] Through the above data enhancement technology, the system can effectively expand the training data set, generate diverse sign language video samples, and provide richer materials for subsequent deep learning model training.

[0016] Step 3: Build a multimodal deep learning model, combining computer vision and natural language processing technology, and use a multimodal deep learning model (such as combining CNN and Transformer architecture) to improve the accuracy and robustness of sign language recognition. The design of this model is divided into the following sub-steps: 1. Feature extraction: First, the sign language video is input to the computer vision module. This module uses a convolutional neural network (CNN) to extract spatial features in the video frames. Specifically, the system decomposes the video into consecutive frames and applies a pre-trained CNN (such as ResNet or EfficientNet) to each frame to extract the spatial features of the gestures.

[0017] At the same time, the text information corresponding to the sign language is input into the natural language processing module. This module uses the Transformer architecture, especially the self-attention mechanism, to capture the contextual information in the text. In this way, the system can understand the relationship between the sign language and its corresponding text.

[0018] 2. Feature connection: After feature extraction, the system connects the features from CNN and Transformer. Specifically, suppose the features extracted by CNN are , and the features extracted by Transformer are , then the feature fusion can be expressed as:

[0019] in The concatenation operation of the representation features. This feature fusion method can effectively combine visual and textual information and enhance the expressiveness of the model.

[0020] 3. Classification and output: fused features The input is sent to a multi-layer perceptron (MLP) classifier. The classifier consists of multiple fully connected layers, and the last layer uses a softmax activation function to output the category probability of the gesture. The specific output can be expressed as:

[0021] in is the weight matrix, is the bias term, is the gesture category, is the input feature.

[0022] 4. Optimization and training model: In order to optimize the model, the cross entropy loss function is used to measure the difference between the predicted category and the true category. The loss function can be expressed as:

[0023] in is the sample size, is the true label, is the probability predicted by the model.

[0024] The Adam optimization algorithm is then used for model training to accelerate convergence and improve model performance. The Adam algorithm combines the advantages of momentum and adaptive learning rate and is suitable for training deep learning models.

[0025] 5. Pruning and quantizing models: After training is completed, the system will apply model pruning technology to remove redundant neurons and connections to reduce the complexity of the model and computing resource consumption. Specifically, the system will evaluate the weight importance of each neuron and prune those neurons that have little impact on the final output.

[0026] In addition, the system will quantize the model and convert floating-point weights into low-precision integers to further improve inference speed and reduce memory usage.

[0027] Step 4: Implement real-time sign language recognition and translation. Using efficient inference engines (such as TensorRT) and edge computing devices, a real-time sign language recognition and translation solution is built to achieve real-time sign language recognition and translation and ensure a smooth user experience. The implementation steps of this solution are as follows: 1. Selection and configuration of inference engine: – Application of TensorRT: TensorRT is a high-performance inference engine developed by NVIDIA, designed for inference optimization of deep learning models. First, users need to convert the trained multimodal deep learning model into a format supported by TensorRT. The specific steps include exporting the model using frameworks such as TensorFlow or PyTorch, and then converting and optimizing it through TensorRT's model optimization tools to reduce the model's inference latency and memory usage.

[0028] 2. Deployment of edge computing devices: – Device selection: Choose suitable edge computing devices (such as NVIDIA Jetson series, Raspberry Pi, etc.) that have sufficient computing power to run the optimized model. The selection of devices should take into account factors such as processing power, power consumption, and cost.

[0029] –Environment configuration: Install the necessary software environment on the edge device, including CUDA, cuDNN, and TensorRT, to ensure that the model can run efficiently.

[0030] 3. Real-time data collection and processing: – Video input: The system collects the user’s sign language video in real time through the camera. The video stream will be transmitted to the edge computing device for processing. To improve processing efficiency, the system can decompose the video stream into continuous frames and pre-process each frame, such as scaling and normalization, to adapt to the input requirements of the model.

[0031] – Feature extraction and reasoning: On the edge device, the preprocessed video frames will be input into the optimized multimodal deep learning model. The model will simultaneously extract the spatial features of the gesture and the corresponding text information, perform real-time reasoning, and output the category and translation results of the gesture.

[0032] 4. Output of translation results: – Two-way translation function: The system supports two-way translation of sign language to text / speech and text / speech to sign language. For sign language input, the model will recognize the gestures and generate corresponding text or voice output; for text or voice input, the system will generate sign language animation and display the translation results in real time.

[0033] – User interface design: To improve the user experience, the system should be designed with a friendly user interface, display translation results in real time, and provide a feedback mechanism to allow users to evaluate the accuracy of the translation so that the system can perform adaptive learning and optimization.

[0034] Step 5: Adaptive learning. This system uses an adaptive learning mechanism, which aims to continuously optimize the sign language recognition model based on user usage habits and feedback, thereby improving recognition accuracy and translation quality. The implementation steps of this mechanism are as follows: 1. User behavior data collection: – Data collection: During the user’s use, the system records the user’s sign language input, translation results, and user feedback in real time. These data include the user’s input gestures, the system’s output translation text or voice, and the user’s satisfaction rating of the translation results. In this way, the system can obtain a large amount of user behavior data, providing a basis for subsequent model optimization.

[0035] – Data storage: The collected data will be stored in a database and managed in a structured manner. Each record should contain fields such as user ID, gesture category, translation result, user feedback, and timestamp for subsequent analysis.

[0036] 2. Feedback analysis and model adjustment: – Feedback processing: The system regularly analyzes user feedback to identify common errors and user dissatisfaction in translation results. For example, if multiple users are dissatisfied with the translation results of a certain gesture, the system will mark the gesture as "to be optimized".

[0037] – Model retraining: Based on the collected user feedback, the system will retrain the model. Specifically, the system will extract the training data related to the gestures to be optimized from the database, and adjust the parameters and structure of the model in combination with the information from user feedback. This process can adopt the incremental learning method to avoid training the model from scratch, saving computing resources and time.

[0038] 3. Dynamic model update: – Online learning mechanism: The system implements online learning function, allowing the model to be dynamically updated during user use. Every time the system completes a retraining, the optimized model will immediately replace the old model, ensuring that users can experience higher translation accuracy in subsequent use.

[0039] – Version control: To ensure the stability of the system, the system should perform version control for each model update. Users can choose to roll back to the previous model version to prevent the new model from being unstable or inaccurate.

[0040] 4. User interface and feedback mechanism: – Feedback interface design: The system should be designed with a user-friendly interface that allows users to provide feedback next to the translation results. Users can quickly express their satisfaction with the translation results through a simple rating system (such as star rating) or text input.

[0041] – Feedback incentive mechanism: In order to encourage users to actively provide feedback, the system can design incentive mechanisms, such as point rewards or function unlocking, to increase user engagement.

[0042] The beneficial effects achieved by the present invention are as follows: the present invention significantly improves the accuracy and efficiency of sign language translation through improvements in data enhancement and preprocessing, model architecture optimization, real-time translation and interaction, and system robustness and adaptability. Through the application of multimodal deep learning models, combined with computer vision and natural language processing technology, the present invention can better understand and translate sign language to meet the diverse communication needs of users. In addition, the system's adaptive learning ability and multilingual support functions enable it to continuously optimize and adapt to the needs of different users, and has broad application prospects and social value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is an overview of the steps of a method for building an efficient sign language translation system based on deep learning in the present invention; Figure 2 It is a sub-step diagram of step 1 of a method for building an efficient sign language translation system based on deep learning according to the present invention; Figure 3 It is a sub-step diagram of step 2 of a method for building an efficient sign language translation system based on deep learning according to the present invention; Figure 4 It is a sub-step diagram of step 3 of a method for building an efficient sign language translation system based on deep learning according to the present invention; Figure 5 It is a sub-step diagram of step 4 of a method for building an efficient sign language translation system based on deep learning according to the present invention; Figure 6 This is a sub-step diagram of step five of a method for building an efficient sign language translation system based on deep learning in the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Example: Figure 1-6 As shown, a method for building an efficient sign language translation system based on deep learning includes the following steps: Step 1: Data collection and annotation: 1. Data source selection: First, select multiple data sources to collect sign language videos. These data sources include: – Public datasets: Use existing sign language video datasets, such as RWTH-PHOENIX-Weather, SignLanguage Recognition Dataset, etc. These datasets are usually verified by professional organizations and have high quality.

[0046] – Voluntary upload by users: By developing a user-friendly application, deaf users or sign language users are encouraged to upload their sign language videos. To protect user privacy, the system will ensure that all uploaded videos are anonymized.

[0047] 2. Data labeling: In order to ensure the accuracy and diversity of the data, a combination of expert labeling and crowdsourcing labeling is used: – Expert annotation: Invite sign language experts to annotate the collected videos to ensure that every gesture, action and expression is accurately identified and described. The experts will use standardized annotation tools to annotate the sign language content in each video and provide corresponding text descriptions.

[0048] – Crowdsourcing annotation: At the same time, crowdsourcing platforms (such as Amazon Mechanical Turk) are used for preliminary annotation. By setting clear annotation guidelines, ordinary users can participate in the annotation process and increase the diversity of data. The results of crowdsourcing annotation will be reviewed by experts to ensure the accuracy of the annotation.

[0049] 3. Data quality control: Strict quality control measures are implemented during the data collection and annotation process. Regular sampling checks are conducted on the annotation results to ensure the consistency and accuracy of the annotations. In addition, automated tools are used to conduct preliminary analysis of the videos to screen out videos of low quality or substandard videos to improve the overall quality of the dataset.

[0050] In specific implementation: suppose we obtain 1,000 sign language videos from a public dataset and collect 500 videos through user uploads. Experts annotate these 1,500 videos to ensure that the gestures and corresponding text descriptions of each video are accurate. Subsequently, the crowdsourcing platform is used to perform preliminary annotations on these 1,500 videos, and finally the annotation results of 1,200 videos are confirmed after expert review.

[0051] Step 2: Data enhancement: 1. Rotation: Rotate the sign language video at random angles to simulate different shooting angles. Specifically, the system will randomly select an angle (for example, any angle between -30 degrees and 30 degrees) and then rotate the video frame. This method can help the model learn how gestures perform under different viewing angles and enhance its ability to recognize gestures.

[0052] 2. Scaling: By randomly scaling the video, the size of the gesture is changed to enhance the model's ability to recognize gestures at different distances. The system will randomly select a scaling factor (for example, any value between 0.8 and 1.2) and scale each frame. In this way, the model can adapt to the changes in gestures at different distances and improve its performance in practical applications.

[0053] 3. Flip: Flip the video horizontally or vertically to increase the diversity of gestures. The system will randomly decide whether to flip the video, and the flipped video will be regarded as a new training sample. This method can help the model learn the symmetry of gestures and further improve its recognition ability.

[0054] 4. Time warping: Random time warping is performed on the timeline of the video to simulate different speeds of gestures. Specifically, the system will randomly select certain frames for acceleration or deceleration to change the playback speed of the gesture. This method can help the model adapt to the gesture expression speed of different users and improve its recognition ability of dynamic gestures.

[0055] 5. Noise addition: Add random noise to the video to enhance the model's robustness to environmental interference. The system will perform noise processing on the video frames, such as adding Gaussian noise or salt and pepper noise. This method can help the model better cope with different shooting environments in practical applications and improve its recognition accuracy.

[0056] Specific implementation: Suppose we have a dataset of 100 sign language videos. First, the system will apply data augmentation techniques such as rotation, scaling, and flipping to each video. Specifically, for each video, the system may randomly select a rotation angle of 15 degrees, a scaling factor of 1.1, and a horizontal flip. After these processes, the original 100 videos will generate 300 enhanced videos (3 variants for each video). Next, the system will also perform time warping and noise addition on these videos to further increase the diversity of the dataset. Finally, after data augmentation processing, the size of the dataset will be expanded to 600 high-quality sign language videos, providing richer materials for the training of deep learning models, significantly improving the generalization ability and recognition accuracy of the model.

[0057] Step 3: Build a multimodal deep learning model: 1. Feature extraction: First, the sign language video is input into the computer vision module. This module uses a convolutional neural network (CNN) to extract spatial features in the video frames. Specifically, the system decomposes the video into consecutive frames and applies a pre-trained CNN (such as ResNet or EfficientNet) to each frame to extract the spatial features of the gestures. At the same time, the text information corresponding to the sign language is input into the natural language processing module. This module uses the Transformer architecture, especially the self-attention mechanism, to capture contextual information in the text. In this way, the system is able to understand the relationship between the sign language and its corresponding text.

[0058] 2. Feature connection: After feature extraction, the system connects the features from CNN and Transformer. Specifically, suppose the features extracted by CNN are , and the features extracted by Transformer are , then the feature fusion can be expressed as:

[0059] in The concatenation operation of the representation features. This feature fusion method can effectively combine visual and textual information and enhance the expressiveness of the model.

[0060] 3. Classification and output: fused features The input is sent to a multi-layer perceptron (MLP) classifier. The classifier consists of multiple fully connected layers, and the last layer uses a softmax activation function to output the category probability of the gesture. The specific output can be expressed as:

[0061] in is the weight matrix, is the bias term, is the gesture category, is the input feature.

[0062] 4. Optimize and train the model: In order to optimize the model, the system uses the cross entropy loss function to measure the difference between the predicted category and the true category. The loss function can be expressed as:

[0063] in is the sample size, is the true label, is the probability predicted by the model.

[0064] The Adam optimization algorithm is then used for model training to accelerate convergence and improve model performance. The Adam algorithm combines the advantages of momentum and adaptive learning rate and is suitable for training deep learning models.

[0065] 5. Pruning and quantizing models: After training is completed, the system will apply model pruning technology to remove redundant neurons and connections to reduce the complexity of the model and computing resource consumption. Specifically, the system will evaluate the weight importance of each neuron and prune those neurons that have little impact on the final output.

[0066] In addition, the system will quantize the model and convert floating-point weights into low-precision integers to further improve inference speed and reduce memory usage.

[0067] Specific implementation: Assume that we have a dataset containing 1,000 sign language videos and corresponding texts. First, the system decomposes each video into 30 frames and uses a pre-trained ResNet to extract the spatial features of each frame to obtain a feature matrix , whose dimensions are (Assume that ResNet outputs 2048-dimensional features). At the same time, the system inputs the corresponding text into the Transformer to obtain text features , whose dimensions are (Assume that Transformer outputs 512-dimensional features). Next, the system will and Connect and get fusion features , whose dimensions are These fused features are then fed into an MLP classifier for training, optimized using a cross-entropy loss function. After training, the system applies pruning and quantization techniques, ultimately resulting in an efficient multimodal deep learning model that enables real-time sign language recognition on edge computing devices. In this way, the system not only improves the accuracy of sign language recognition, but also significantly improves the inference speed and adaptability of the model.

[0068] Step 4: Realize real-time sign language recognition and translation: 1. Selection and configuration of inference engine: Application of TensorRT: TensorRT is a high-performance inference engine developed by NVIDIA, designed for inference optimization of deep learning models. First, users need to convert the trained multimodal deep learning model into a format supported by TensorRT. The specific steps include exporting the model using frameworks such as TensorFlow or PyTorch, and then converting and optimizing it through TensorRT's model optimization tools to reduce the model's inference latency and memory usage.

[0069] 2. Deployment of edge computing devices: Device selection: Choose suitable edge computing devices (such as NVIDIA Jetson series, Raspberry Pi, etc.) that have sufficient computing power to run the optimized model. The selection of devices should take into account factors such as processing power, power consumption, and cost.

[0070] –Environment configuration: Install the necessary software environment on the edge device, including CUDA, cuDNN, and TensorRT, to ensure that the model can run efficiently.

[0071] 3. Real-time data collection and processing: Video input: The system collects the user's sign language video in real time through the camera. The video stream will be transmitted to the edge computing device for processing. In order to improve processing efficiency, the system can decompose the video stream into continuous frames and pre-process each frame, such as scaling and normalization, to adapt to the input requirements of the model.

[0072] Feature extraction and reasoning: On the edge device, the preprocessed video frames will be input into the optimized multimodal deep learning model. The model will simultaneously extract the spatial features of the gesture and the corresponding text information, perform real-time reasoning, and output the category and translation results of the gesture.

[0073] 4. Output of translation results: Two-way translation function: The system supports two-way translation of sign language to text / speech and text / speech to sign language. For sign language input, the model will recognize the gestures and generate corresponding text or voice output; for text or voice input, the system will generate sign language animation and display the translation results in real time.

[0074] User interface design: To improve user experience, the system should be designed with a friendly user interface, display translation results in real time, and provide a feedback mechanism to allow users to evaluate the accuracy of the translation so that the system can perform adaptive learning and optimization.

[0075] In specific implementation: Assuming that the user inputs sign language through the camera, the system first decomposes the video stream into image data at 30 frames per second. After preprocessing, each frame of the image is input into a multimodal deep learning model optimized by TensorRT. The model will extract the spatial features of each frame and perform reasoning in combination with text information. For example, assuming that the user's gesture means "hello", the model will recognize the gesture and output the corresponding text "hello". At the same time, the system will convert the text into speech through the speech synthesis module, or generate the corresponding sign language animation, and display it in real time on the user interface. In this way, the system can achieve low-latency real-time sign language recognition and translation, significantly improving the user's communication experience.

[0076] Step 5: Adaptive learning: 1. User behavior data collection: – Data collection: During the user’s use, the system records the user’s sign language input, translation results, and user feedback in real time. These data include the user’s input gestures, the system’s output translation text or voice, and the user’s satisfaction rating of the translation results. In this way, the system can obtain a large amount of user behavior data, providing a basis for subsequent model optimization.

[0077] – Data storage: The collected data will be stored in a database and managed in a structured manner. Each record should contain fields such as user ID, gesture category, translation result, user feedback, and timestamp for subsequent analysis.

[0078] 2. Feedback analysis and model adjustment: – Feedback processing: The system regularly analyzes user feedback to identify common errors and user dissatisfaction in translation results. For example, if multiple users are dissatisfied with the translation results of a certain gesture, the system will mark the gesture as "to be optimized".

[0079] – Model retraining: Based on the collected user feedback, the system will retrain the model. Specifically, the system will extract the training data related to the gestures to be optimized from the database, and adjust the parameters and structure of the model in combination with the information from user feedback. This process can adopt the incremental learning method to avoid training the model from scratch, saving computing resources and time.

[0080] 3. Dynamic model update: Online learning mechanism: The system implements online learning function, allowing the model to be dynamically updated during user use. Every time the system completes a retraining, the optimized model will immediately replace the old model, ensuring that users can experience higher translation accuracy in subsequent use.

[0081] Version control: To ensure the stability of the system, the system should perform version control for each model update. Users can choose to roll back to the previous model version to prevent the new model from becoming unstable or inaccurate.

[0082] 4. User interface and feedback mechanism: Feedback interface design: The system should be designed with a user-friendly interface that allows users to provide feedback next to the translation results. Users can quickly express their satisfaction with the translation results through a simple rating system (such as star rating) or text input.

[0083] Feedback incentive mechanism: In order to encourage users to provide active feedback, the system can design incentive mechanisms, such as point rewards or function unlocking, to increase user engagement.

[0084] In specific implementation: suppose that when a user uses the sign language translation system, he inputs the gesture "thank you", and the system translates it into "thank you". If the user is not satisfied with the translation result and gives a low score, the system will record the feedback information. Subsequently, the system will analyze the translation accuracy of the gesture and find that multiple users are dissatisfied with the translation result. In the subsequent model retraining, the system will extract the training data related to the "thank you" gesture and make adjustments based on the user's feedback information. After retraining, the system will update the model and provide a more accurate translation result the next time the user inputs "thank you". Through this adaptive learning mechanism, the system can continuously optimize the accuracy of sign language translation and user experience to meet the needs of different users.

[0085] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing an efficient sign language translation system based on deep learning, characterized in that: It includes the following five steps: Step 1: Data collection and annotation; Step 2: Data enhancement; Step 3: Build a multimodal deep learning model; Step 4: Realize real-time sign language recognition and translation; Step 5: Adaptive learning.

2. According to the method of claim 1, it is characterized in that The step 1 includes the following 3 sub-steps:

1. Data source selection, including: public data sets and data sets voluntarily uploaded by users; 2. Data annotation, using a combination of expert annotation and crowdsourcing annotation; 3. Data quality control: During the process of data collection and labeling, regular sampling checks are conducted on the labeling results.

3. The method for constructing an efficient sign language translation system based on deep learning according to claim 1, characterized in that The step 2 includes the following 5 sub-steps:

1. Rotation: Rotate the sign language video at random angles to simulate different shooting angles, set a certain rotation angle, and then rotate the video frame; 2. Scaling: By randomly scaling the video, the size of the gesture is changed to enhance the model's ability to recognize gestures at different distances. The scaling factor is set and each frame is scaled.

3. Flip: Flip the video horizontally or vertically to increase the diversity of gestures, and randomly decide whether to flip the video; 4. Time warp: Random time warping is performed on the timeline of the video to simulate different speeds of gestures. Specifically, some frames are randomly selected to be accelerated or decelerated to change the playback speed of the gestures.

5. Noise addition: Add random noise to the video to enhance the model’s robustness to environmental interference, perform noise processing on the video frames, and add Gaussian noise or salt and pepper noise.

4. The method for constructing an efficient sign language translation system based on deep learning according to claim 1, characterized in that The step three includes the following five sub-steps:

1. Feature extraction: First, the sign language video is input into the computer vision module, and the convolutional neural network (CNN) is used to extract the spatial features in the video frame. The video is decomposed into consecutive frames, and the pre-trained CNN is applied to each frame to extract the spatial features of the gesture. At the same time, the text information corresponding to the sign language is input into the natural language processing module, and the self-attention mechanism of the Transformer architecture is used to capture the contextual information in the text; 2. Feature connection: After feature extraction, the system connects and fuses the features from CNN and Transformer; 3. Classification and output: Input the fused features into a multi-layer perceptron (MLP) classifier to output the category probability of the gesture; 4. Optimize and train the model: Use the cross entropy loss function to measure the difference between the predicted category and the true category; Then use the Adam optimization algorithm to train the model to accelerate convergence and improve model performance. The Adam algorithm combines the advantages of momentum and adaptive learning rate and is suitable for training deep learning models; 5. Pruning and quantizing models: After training, use model pruning technology to remove redundant neurons and connections to reduce the complexity of the model and computing resource consumption. In addition, convert floating-point weights to low-precision integers to further improve inference speed and reduce memory usage.

5. The method for constructing an efficient sign language translation system based on deep learning according to claim 1, characterized in that The step 4 includes the following 4 sub-steps:

1. Selection and configuration of inference engine: Export the model using frameworks such as TensorFlow or PyTorch, and then convert and optimize it using TensorRT's model optimization tools to reduce the model's inference latency and memory usage; 2. Deployment of edge computing devices: Select NVIDIA Jetson series or Raspberry Pi as edge computing devices; install the necessary software environment on the edge devices, including CUDA, cuDNN, and TensorRT, to ensure that the model can run efficiently; 3. Real-time data collection and processing: The user's sign language video is collected in real time through the camera. The video stream will be transmitted to the edge computing device for processing, and the video stream will be decomposed into continuous frames. Each frame will be pre-processed, scaled and normalized to meet the input requirements of the model; On the edge device, the preprocessed video frames are input into the optimized multimodal deep learning model to extract the spatial features of the gestures and the corresponding text information, perform real-time reasoning, and output the category and translation results of the gestures.

4. Output the translation results.

6. The method for constructing an efficient sign language translation system based on deep learning according to claim 1, characterized in that The step five includes the following four sub-steps:

1. User behavior data collection: When users use the system, the user's sign language input, translation results and user feedback information are recorded in real time. These data include the user's input gestures, the system's output translation text or voice, the user's satisfaction rating of the translation results, etc. The collected data is stored in the database and managed in a structured manner; 2. Feedback analysis and model adjustment: Set the system to regularly analyze user feedback, identify common errors in the translation results and situations where users are dissatisfied, and mark some gestures as "to be optimized"; based on the collected user feedback, retrain the model, extract training data related to the "to be optimized" gestures from the database, and adjust the model's parameters and structure based on the user feedback information; 3. Dynamic model update: Allow the model to be dynamically updated during user use. Every time the system completes a retraining, the optimized model will immediately replace the old model. At the same time, the system is set to perform version control for each model update; 4. User interface and feedback mechanism: Design a friendly user interface that allows users to provide feedback next to the translation results, and set up users to quickly express their satisfaction with the translation results through a simple rating system (such as star rating) or text input.

7. Step 2 according to claim 3, characterized in that The rotation angle in step 2 is any angle between -30 degrees and 30 degrees, and the scaling factor is any value between 0.8 and 1.2.

Images