Sign language translation method and system based on deep learning
By performing semantic parsing and grammatical correction on keyframe image sequences of sign language videos, and utilizing a bidirectional gated recurrent unit network and a lightweight TextCNN model, the problem of low accuracy in sign language translation is solved, achieving more efficient translation results.
Patent Information
- Application Number
- CN202511266484.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-05
- Publication Date
- 2025-11-14
AI Technical Summary
Existing deep learning models tend to overemphasize or neglect certain parts in sign language translation, resulting in low translation accuracy. They are particularly prone to confusion when dealing with similar sentence structures, and the segmentation is inaccurate, affecting the translation effect.
An image detection and recognition model is constructed. By extracting semantic parsing features from keyframe image sequences, grammatical and semantic corrections are performed on the gesture encoding sequences. Using a bidirectional gated recurrent unit network and a lightweight TextCNN model, the model recognizes and corrects sign language grammatical structures, generating natural language sentences that conform to the target language.
It improves the accuracy and coherence of sign language translation. The improved image detection and recognition model enhances the perception of global information, shortens the computation time, and improves translation efficiency and accuracy.
Smart Images
Figure CN120954098A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing and video analysis technology, and more specifically to a deep learning-based sign language translation method and system. Background Technology
[0002] Human-computer interaction, as a key technology connecting humans and computers, occupies an important position in the development of intelligentization.
[0003] With the widespread application of modern computer technology and artificial intelligence in the field of human-computer interaction, safer, more natural, and harmonious new interaction methods are emerging. Among these, gestures are a special language (symbols) used by humans to express information. The intention behind hand movements is generated by the brain, transmitted through neural pathways, and ultimately expressed in the form of gestures. In this sense, gestures can serve as a window for machines to understand human intentions. However, few people understand sign language. Sign language translation, by processing and analyzing sensory information from various modalities of the human hand and arm to identify sign language movements and decode the information expressed by the executor, is of great significance for fields such as medical rehabilitation, services, industry, and even the military.
[0004] Because sign language has its own unique grammar and word order as an independent language, and the continuous actions make segmentation quite difficult, inaccurate segmentation significantly impacts translation accuracy. Traditional deep learning models may overemphasize some parts while completely ignoring others. For example, when translating "I don't understand, could you please repeat it?", they might omit the word "repeat," resulting in the incorrect translation "I don't understand, could you please repeat it?" Furthermore, similar sentence structures like "How do I get to the train station?" and "Where is the restroom?" are easily confused, leading to incorrect translations and low accuracy. Summary of the Invention
[0005] To address the problems existing in the aforementioned fields, this invention proposes a deep learning-based sign language translation method and system. By constructing an image detection and recognition model, keyframe image sequences are identified to obtain the encoded sequence of gestures, which is then subjected to grammatical and semantic correction. Grammatical correction eliminates the differences between the grammatical structure of sign language and that of natural language. Semantic mapping converts the grammatically corrected encoded sequence into natural language text, and the natural language vocabulary sequence is combined into complete sentences according to the grammatical rules of the target language, improving the coherence between the encoded sequences and ensuring the accuracy of the translation.
[0006] To address the aforementioned technical problems, this invention discloses a deep learning-based sign language translation method, comprising the following steps: Obtain the keyframe image sequence from the sign language video information to be processed; Semantic parsing features of each sign language image in the keyframe image sequence are extracted and feature fusion is performed frame by frame to obtain the encoding sequence of the gesture; Based on the encoding sequence of the gesture, the sign language grammatical structure text corresponding to the encoding sequence is identified, the sign language grammatical structure text is converted into the target language structure text that conforms to the expression habits, and the encoding sequence after grammatical correction is obtained by performing grammatical disambiguation on the target language structure text. The grammatically corrected encoded sequence is used to obtain a natural language vocabulary sequence through semantic mapping. The natural language vocabulary sequence is then combined into a complete sentence according to the grammatical rules of the target language to generate a semantically corrected sign language translation text.
[0007] Preferably, the conversion of sign language grammatical structure text into target language structure text that conforms to expression habits specifically includes: By using a bidirectional gated recurrent unit network to perform temporal modeling on the feature vectors of semantic parsing features of consecutive frame sign language image sequences, the sign language grammatical structure text corresponding to the encoded sequence can be identified. By referencing the Chinese sign language grammar rule library, sign language structures are converted into target language structures that conform to expression habits.
[0008] Preferably, the step of semantically mapping the grammatically corrected encoded sequence to natural language vocabulary specifically involves: By learning the correspondence between gesture codes and words through a lightweight TextCNN model, the grammatically corrected encoding sequence is mapped to natural language words to generate a natural language word sequence.
[0009] Preferably, the basis for disambiguating the target language structure is to remove ambiguous target language structures based on the motion trajectory features of the gesture and the context.
[0010] Preferably, the generated semantically corrected sign language translation text is generated by recognition through an image detection and recognition model, which specifically includes: The network uses the YOLOv8 model as its main component, including a feature extraction module, a neck feature fusion module, and a head detection module, among which: The feature extraction module removes the C2f layer before the SPPF layer in the backbone of the YOLOv8 model's Backbone feature extraction module, replaces the C2f module with a Bottleneck Transformer after the SPPF layer, and retains the first three C2f layers to extract the semantic parsing features of each frame of sign language image in the keyframe image sequence. The Neck feature fusion module is used to perform feature fusion on the semantic parsing features frame by frame to generate the gesture encoding sequence; The Head detection module combines the first two standard convolutional layers of the upper and lower branches of the detection head, and replaces the standard convolutional layers with a combination of PConv and a 1×1 convolution. This module is used to perform grammatical and semantic correction on the encoded sequence of the gesture in sequence, generating semantically corrected sign language translation text.
[0011] Preferably, generating semantically corrected sign language translation text further includes rendering the sign language translation text as subtitles and overlaying it onto video frames, and outputting a video stream displaying the processed translation text.
[0012] Preferably, the acquisition of the keyframe image sequence in the sign language video information to be processed specifically involves capturing sign language movements in real time through a camera to generate a sign language video stream, or extracting the sign language image sequence to be processed from an existing video file.
[0013] Preferably, the process further includes preprocessing the sign language image sequence to be processed, specifically: The LabelImg tool was used to annotate and format the sign language image sequence. The annotated sign language image sequence was formatted to conform to the standard format of the YOLOv8 model input, and data rotation, cropping, and normalization were performed.
[0014] Preferably, it also includes a deep learning-based sign language translation system, comprising: The gesture encoding sequence acquisition module is used to acquire the key frame image sequence in the sign language video information to be processed; extract the semantic parsing features of each sign language image in the key frame image sequence and perform feature fusion frame by frame to obtain the gesture encoding sequence; The grammar correction module is used to identify the sign language grammar structure text corresponding to the encoding sequence of the gesture, convert the sign language grammar structure text into the target language structure text that conforms to the expression habits, and obtain the grammar-corrected encoding sequence by performing grammar disambiguation on the target language structure text. The semantic correction module is used to obtain a natural language vocabulary sequence from the grammatically corrected encoded sequence through semantic mapping, and to combine the natural language vocabulary sequence into a complete sentence according to the grammatical rules of the target language, thereby generating a semantically corrected sign language translation text.
[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes a deep learning-based sign language translation method. It extracts semantic parsing features from each frame of a sign language image sequence and fuses these features frame-by-frame to obtain the gesture encoding sequence. This enhances the model's ability to perceive global information and better extracts the basic features of the image. By identifying the sign language grammatical structure text corresponding to the gesture encoding sequence, the logical relationships between the encoding sequences are analyzed. The sign language grammatical structure text is then converted into target language structure text that conforms to expression habits, and grammatical disambiguation is performed to eliminate differences between the sign language grammatical structure text and the grammatical structure of natural language. To improve the coherence between the grammatically corrected encoding sequences, the grammatically corrected encoding sequences are further mapped to obtain a natural language vocabulary sequence. By learning the correspondence between gesture encoding and vocabulary, the natural language vocabulary sequence is combined into a complete sentence according to the grammatical rules of the target language, generating semantically corrected sign language translation text. During the translation process, the constructed image detection and recognition model enhances the model's ability to perceive global information and better extracts the basic features of the image by improving the Backbone feature extraction module. By replacing the standard convolutional layer in the Head detection module and calculating only some channels, the computational load and time cycle are shortened, thereby improving translation efficiency. Attached Figure Description
[0016] Figure 1 This is a flowchart of the deep learning-based sign language translation method proposed in this invention; Figure 2 A schematic diagram of a deep learning sign language translation system provided in an embodiment of the present invention; Figure 3 The image detection and recognition model architecture provided in the embodiments of the present invention; Figure 4 This is an architecture diagram of the BoTNet module in the image detection and recognition model of the present invention; Figure 5 This invention provides a Head detection module for an image detection and recognition model. Figure 6 The training curves are for the image detection and recognition model provided in this embodiment of the invention. Detailed Implementation
[0017] The following will refer to the appendices in the embodiments of the present invention. Figures 1-6 The technical solutions in the embodiments of the present invention will be clearly and completely described. It should be understood that the terminology used in the present invention is only for describing particular implementation methods and is not intended to limit the present invention.
[0018] Example like Figure 1 As shown, this invention proposes a deep learning-based sign language translation method, which includes the following steps: S1: Obtain the keyframe image sequence from the sign language video information to be processed; S2: Extract the semantic parsing features of each sign language image in the keyframe image sequence and perform feature fusion frame by frame to obtain the encoding sequence of the gesture; S3: Based on the encoding sequence of the gesture, identify the sign language grammatical structure text corresponding to the encoding sequence, convert the sign language grammatical structure text into the target language structure text that conforms to the expression habits, and obtain the grammatically corrected encoding sequence by performing grammatical disambiguation on the target language structure text. S4: Obtain a natural language vocabulary sequence from the grammar-corrected encoded sequence through semantic mapping, combine the natural language vocabulary sequence into a complete sentence according to the grammar rules of the target language, and generate a semantically corrected sign language translation text.
[0019] It also includes rendering sign language translation text as subtitles and overlaying it onto video frames, outputting a video stream that displays the processed translation text.
[0020] Specifically, in step S1, obtaining the keyframe image sequence in the sign language video information to be processed involves capturing sign language actions in real time through a camera to generate a sign language video stream, or extracting the sign language image sequence (i.e., keyframe image sequence) to be processed from an existing video file for subsequent sign language translation processing.
[0021] This also includes preprocessing the sign language image sequences to be processed, specifically: The LabelImg tool was used to annotate and format the sign language image sequence. The annotated sign language image sequence was formatted to conform to the standard format of the YOLOv8 model input, and data rotation, cropping, and normalization were performed.
[0022] In step S4, the semantically corrected sign language translation text is generated through the image detection and recognition model. The image detection and recognition model is based on the improved YOLOv8s framework and is optimized in three stages: feature extraction, feature fusion and detection, with the core objectives of lightweight and high-accuracy detection.
[0023] The network uses the YOLOv8 model as its main component, including a feature extraction module, a neck feature fusion module, and a head detection module. The improvement strategies for each module are as follows: Figure 3 As shown. Wherein:
[0024] like Figure 4The diagram shows the specific architecture of BoTNet. The feature extraction module of the image detection and recognition model proposed in this invention removes the C2f layer before the SPPF layer in the backbone feature extraction module of the YOLOv8 model, replaces the C2f module with a Bottleneck Transformer (BoTNet) after the SPPF layer, and retains the first three C2f layers. These layers are used to extract the semantic parsing features of each frame of the sign language image in the keyframe image sequence, thereby enhancing the model's ability to perceive global information and better extract the basic features of the image.
[0025] The Neck feature fusion module is used to perform feature fusion on a frame-by-frame basis on the semantic parsing features of each sign language image in the extracted keyframe image sequence to obtain the encoded sequence of the gesture.
[0026] like Figure 5 As shown, the Head detection module combines the first two standard convolutional layers of the upper and lower branches of the detection head, and replaces the standard convolutional layers with a combination of PConv and a 1×1 convolution. It only calculates for a portion of the channels, reducing FLOPs to 1 / 16 of that of a regular convolution. This is used to perform grammatical and semantic correction on the encoded sequence of the gesture sequentially.
[0027] The specific steps of grammar correction include: After feature fusion, the system obtains the semantic parsing features of each sign language image in the keyframe image sequence, i.e., the encoded sequence of gestures. Since the grammatical structure of sign language differs from that of natural language, grammatical correction is necessary.
[0028] 1) Contextual Relationship Analysis. A Bi-directional Gated Recurrent Unit (Bi-GRU) network is used to perform temporal modeling of the feature vectors of semantic parsing features from consecutive frame sign language image sequences. This analyzes the logical relationships between gestures and identifies the corresponding sign language grammatical structure text. For example, when the gesture sequence "toilet-go-me" is detected, the system will identify the corresponding "object-action-subject" sequence.
[0029] 2) Word order conversion. Based on the built-in Chinese sign language grammar rule library, the sign language grammar structure text is converted into the target language structure text. For example, "toilet-go-I" is converted into "I-go-toilet" to obtain the target language structure text that conforms to the expression habits.
[0030] 3) Ambiguity Disambiguation. For ambiguous gestures, since the grammatical structure of sign language text differs from that of natural language, the system performs grammatical disambiguation on the target language text by analyzing the movement trajectory features of the gestures and the contextual analysis, removing ambiguous target language text and thus obtaining a grammatically corrected encoding sequence. For example, if the subsequent gesture is "three o'clock," it is determined to be "time" rather than "circle."
[0031] The specific steps of semantic correction include: The grammatically corrected encoded sequence will be converted into natural language text.
[0032] 1) Semantic Mapping. A lightweight TextCNN model is used to semantically map the grammatically corrected encoded sequence to obtain a natural language vocabulary sequence. During training, the model learns the correspondence between gesture codes and words, and outputs the most probable natural language vocabulary sequence as the mapped natural language vocabulary sequence.
[0033] 2) Natural Language Generation. The mapped natural language vocabulary sequences are combined into complete sentences according to the grammatical rules of the target language to improve the coherence between the grammatically corrected encoded sequences.
[0034] 3) Real-time feedback. The system sets a confidence threshold. When the confidence of a gesture in a sentence is less than the confidence threshold of 0.92, a backtracking mechanism is automatically triggered for re-analysis. When the confidence of a gesture is greater than the confidence threshold of 0.92, a semantically corrected sign language translation is generated to ensure the accuracy of the translation.
[0035] In the training and testing phases of the image detection and recognition model, this invention first sets up a training environment, installs Anaconda3 and creates a Python virtual environment (such as yolov8_env), and configures CUDA, PyTorch and compatible Python versions.
[0036] In the data preparation phase, we collected sign language datasets (including self-shot videos and publicly available datasets such as the ChineseSign Language Dataset), labeled them using tools such as LabelImg, formatted them into the YOLO standard format, and performed data augmentation (rotation, cropping, normalization, etc.).
[0037] Configure yolov8s.yaml to define the image detection and recognition model structure, train it for 300 epochs to generate the best.pt weight file, and ensure that the image detection and recognition model achieves the required accuracy through validation metrics (such as box_loss, mAP, etc.).
[0038] After training, the image detection and recognition model was converted. First, the output layer of YOLOv8's yolo.py was modified to improve compatibility, and PyTorch's torch.onnx.export was used to convert best.pt to ONNX format (best.onnx). Simultaneously, the ONNX model architecture was verified to ensure no missing nodes. RKNN-Toolkit2 (v1.5.0) and a cross-compilation toolchain were installed on an Ubuntu virtual machine. A test.py script was written to call RKNN-Toolkit2 for quantization and conversion, generating the best.rknn file (using INT8 quantization to reduce model size and improve inference speed).
[0039] On the RK3568 development board, install a Linux system (such as Ubuntu or the official Rockchip image) and configure the necessary hardware peripherals (such as a camera and microphone).
[0040] Using rknn.config from RKNN-Toolkit2 for INT8 quantization, the image detection and recognition model's size is reduced by 75% and inference speed is increased by 2-3 times by decreasing the bit width of the weights and activation values (32-bit → 8-bit). A mixed-precision strategy is employed during quantization, preserving FP16 precision for sensitive layers (such as the detection head) to avoid precision loss.
[0041] Hardware instruction set optimization utilizes operator fusion techniques such as Conv2D / ReLU / Pooling supported by the NPU to merge consecutive operations into a single NPU instruction (e.g., Conv+ReLU), reducing memory access frequency. Operator support is analyzed via the rknn.query interface, and CPU-assisted computation is employed for unsupported operators (such as a custom CA attention module).
[0042] The rknn.optimize tool automatically prunes redundant computation nodes (such as invalid pooling layers), making the computation graph more compatible with the NPU pipeline.
[0043] The above method was used to achieve the model deployment goal on RK3568, and the actual test results showed that it met the 30FPS real-time requirement.
[0044] On the software side, install the RKNN inference library (librknnrt.so) and dependent environments such as OpenCV and Python3 to ensure that the image detection and recognition model can run smoothly.
[0045] Use ADB commands to push the converted best.rknn file, category label file (labels.txt), and executable program to the / userdata / directory on the development board. Write a CMakeLists.txt file to define the project structure and generate an ARM architecture executable file (such as demo) through cross-compilation, then push it to the development board. Run the program (. / demo best.rknn) in the development board terminal to test the sign language recognition effect in real time and monitor the output metrics (such as FPS and recognition accuracy).
[0046] To ensure optimal performance of the image detection and recognition model, we reduced accuracy loss by adjusting RKNN quantization parameters (such as dynamic range) and used NPU acceleration (rknn.inference) to replace CPU computation to improve inference speed.
[0047] In step S4, the translated text of the sign language image information to be processed is output, specifically including: Extract keyframe image sequences from the sign language video information to be processed; Feature extraction, semantic parsing, and feature fusion are performed on sign language image sequences, and sign language translation text is generated through grammar correction and translation processing. The translated text is rendered as subtitles and overlaid on video frames, and the output is a video stream displaying the processed translated text.
[0048] This invention also proposes a deep learning-based sign language translation system, such as... Figure 2 As shown, it includes: The gesture encoding sequence acquisition module is used to acquire the key frame image sequence in the sign language video information to be processed; extract the semantic parsing features of each sign language image in the key frame image sequence and perform feature fusion frame by frame to obtain the gesture encoding sequence; The grammar correction module is used to identify the sign language grammar structure text corresponding to the encoding sequence of the gesture, convert the sign language grammar structure text into the target language structure text that conforms to the expression habits, and obtain the grammar-corrected encoding sequence by performing grammar disambiguation on the target language structure text. The semantic correction module is used to obtain a natural language vocabulary sequence from the grammatically corrected encoded sequence through semantic mapping, and to combine the natural language vocabulary sequence into a complete sentence according to the grammatical rules of the target language, thereby generating a semantically corrected sign language translation text.
[0049] It also includes a display module, which renders the translated text as subtitles and displays them overlaid on the video frames, outputting a video stream of the processed translated text.
[0050] Compared to traditional text translation methods, the deep learning-based sign language translation method proposed in this invention improves translation accuracy by performing grammatical and semantic correction on the encoded sequence of acquired gestures. The constructed image detection and recognition model, in its Head detection module, replaces standard convolutional layers, performing calculations only on a subset of channels, thus reducing computational load and time, and consequently improving translation efficiency.
[0051] Case Analysis This example illustrates a hearing-impaired user making a basic Chinese sign language greeting, "Hello, I need help," to a staff member at a brightly lit shopping mall service center. The deep learning-based sign language translation system provided by this invention captures the gestures in real time using a camera mounted on an RK3568 development board, completing the recognition and translation within a 1.2-meter distance.
[0052] The system's RK3568 development board captures gesture images at 30 FPS via a camera. The BoT-C2f backbone network locks onto the hand region within 0.2 seconds, and the CA attention mechanism automatically filters out background noise such as billboards, accurately extracting hand contour features. A 160×160 high-resolution sampling layer locates 21 key hand points, achieving a fingertip coordinate confidence level of 0.94. The lightweight detection head employs PConv convolution combined with INT8 quantization technology, completing feature matching in 28ms with NPU acceleration and outputting the gesture code "GH-01".
[0053] Compared to the standard YOLOv8 model, this system demonstrates significant advantages in this scenario. When detecting the complex gesture of "right hand clenched into a fist and then left hand forming a V," the original model, lacking a CA attention mechanism, mistakenly identified a bystander's "hello" gesture as a "good" gesture in crowded or occluded environments. In contrast, the image detection and recognition model provided by this invention accurately focuses on the target region through coordinate attention, reducing the false alarm rate to 6.2%. Key performance indicator comparisons show that the model size was compressed from 89MB to 22MB, reducing RK3568 memory usage by 37%; single-frame processing latency was optimized from 42ms to 28ms, ensuring smooth dialogue; and particularly in continuous action recognition scenarios, the addition of a 160×160 sampling scale and BiGRU temporal modeling improved the accuracy of recognizing multiple gesture sequences, such as questions and descriptions, from 82.1% to 94.7%.
[0054] like Figure 6The figure shows the training curves of the image detection and recognition model provided by this invention, including train / box_loss, train / obj_loss, train / cls_loss, metrics / precision, metrics / recall, val / box_loss, val / obj_loss, val / cls_loss, metrics / mAP_0.5, and metrics / mAP_0.5: 0.95. Wherein:
[0055] train / box_loss corresponds to the training set bounding box regression loss, train / obj_loss corresponds to the training set object existence probability loss, train / cls_loss corresponds to the training set binary cross-entropy loss, metrics / precision corresponds to the precision of each class, and metrics / recall corresponds to the recall of each class.
[0056] val / box_loss corresponds to the bounding box regression loss on the validation set, val / obj_loss corresponds to the object presence probability loss on the validation set, val / cls_loss corresponds to the binary cross-entropy loss on the validation set, metrics / mAP_0.5 is used to measure the average detection accuracy of the model for all classes at a specific IoU threshold (0.5), and metrics / mAP_0.5:0.95 is used to calculate AP for IoU thresholds from 0.5 to 0.95 (a total of 10 points: 0.5, 0.55, ..., 0.95), and then take the arithmetic mean of these 10 APs.
[0057] pass Figure 6 These curves validate the effectiveness of the image detection and recognition model: after 40 rounds of training, the box loss steadily converges to below 0.02. The validation set metrics / mAP@0.5 reaches 91.2%, an improvement of 14.7 percentage points compared to the ordinary YOLOv8 model.
[0058] Therefore, the deep learning-based sign language translation method provided in this embodiment can quickly and accurately output translated text by extracting and fusing features from the acquired gesture information, and then performing grammatical and semantic corrections in sequence.
[0059] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
[0060] Furthermore, unless otherwise stated, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. All references to this specification are incorporated by way of citation to disclose and describe methods relating to those references. In the event of any conflict with any incorporated reference, the content of this specification shall prevail.
Claims
1. A deep learning-based sign language translation method, characterized in that, Includes the following steps: Obtain the keyframe image sequence from the sign language video information to be processed; Semantic parsing features of each sign language image in the keyframe image sequence are extracted and feature fusion is performed frame by frame to obtain the encoding sequence of the gesture; Based on the encoding sequence of the gesture, the sign language grammatical structure text corresponding to the encoding sequence is identified, the sign language grammatical structure text is converted into the target language structure text that conforms to the expression habits, and the encoding sequence after grammatical correction is obtained by performing grammatical disambiguation on the target language structure text. The grammatically corrected encoded sequence is used to obtain a natural language vocabulary sequence through semantic mapping. The natural language vocabulary sequence is then combined into a complete sentence according to the grammatical rules of the target language to generate a semantically corrected sign language translation text.
2. The deep learning-based sign language translation method according to claim 1, characterized in that, The process of converting sign language grammatical structure text into target language structure text that conforms to expression habits specifically involves: By using a bidirectional gated recurrent unit network to perform temporal modeling on the feature vectors of semantic parsing features of consecutive frame sign language image sequences, the sign language grammatical structure text corresponding to the encoded sequence can be identified. By referencing the Chinese sign language grammar rule library, sign language structures are converted into target language structures that conform to expression habits.
3. The deep learning-based sign language translation method according to claim 2, characterized in that, The process of semantically mapping the grammatically corrected encoded sequence to natural language vocabulary specifically involves: By learning the correspondence between gesture codes and words through a lightweight TextCNN model, the grammatically corrected encoding sequence is mapped to natural language words to generate a natural language word sequence.
4. The deep learning-based sign language translation method according to claim 3, characterized in that, The basis for performing grammatical disambiguation on the target language structure is to remove ambiguous target language structures based on the motion trajectory characteristics of gestures and the context.
5. The deep learning-based sign language translation method according to claim 4, characterized in that, The generated semantically corrected sign language translation text is identified through an image detection and recognition model, which specifically includes: The network uses the YOLOv8 model as its main component, including a feature extraction module, a neck feature fusion module, and a head detection module, among which: The feature extraction module removes the C2f layer before the SPPF layer in the backbone of the YOLOv8 model's Backbone feature extraction module, replaces the C2f module with a Bottleneck Transformer after the SPPF layer, and retains the first three C2f layers to extract the semantic parsing features of each frame of sign language image in the keyframe image sequence. The Neck feature fusion module is used to perform feature fusion on the semantic parsing features frame by frame to generate the gesture encoding sequence; The Head detection module combines the first two standard convolutional layers of the upper and lower branches of the detection head, and replaces the standard convolutional layers with a combination of PConv and a 1×1 convolution. This module is used to perform grammatical and semantic correction on the encoded sequence of the gesture in sequence, generating semantically corrected sign language translation text.
6. The deep learning-based sign language translation method according to claim 5, characterized in that, The process of generating semantically corrected sign language translation text also includes rendering the sign language translation text as subtitles and overlaying it onto video frames, and outputting a video stream that displays the processed translation text.
7. The deep learning-based sign language translation method according to claim 1, characterized in that, The acquisition of keyframe image sequences in the sign language video information to be processed specifically involves capturing sign language movements in real time through a camera to generate a sign language video stream, or extracting the sign language image sequences to be processed from existing video files.
8. The deep learning-based sign language translation method according to claim 7, characterized in that, This also includes preprocessing the sign language image sequences to be processed, specifically: The LabelImg tool was used to annotate and format the sign language image sequence. The annotated sign language image sequence was formatted to conform to the standard format of the YOLOv8 model input, and data rotation, cropping, and normalization were performed.
9. A deep learning-based sign language translation system, characterized in that, include: The gesture encoding sequence acquisition module is used to acquire keyframe image sequences from the sign language video information to be processed; Semantic parsing features of each sign language image in the keyframe image sequence are extracted and feature fusion is performed frame by frame to obtain the encoding sequence of the gesture; The grammar correction module is used to identify the sign language grammar structure text corresponding to the encoding sequence of the gesture, convert the sign language grammar structure text into the target language structure text that conforms to the expression habits, and obtain the grammar-corrected encoding sequence by performing grammar disambiguation on the target language structure text. The semantic correction module is used to obtain a natural language vocabulary sequence from the grammatically corrected encoded sequence through semantic mapping, and to combine the natural language vocabulary sequence into a complete sentence according to the grammatical rules of the target language, thereby generating a semantically corrected sign language translation text.