Sign language inter-translation and real-time communication system special for hearing and speech impaired people
By reconstructing sign language word order using binocular depth cameras and neural networks, and combining lip reading recognition and scene adaptation engines, the system solves the problem of real-time communication for people with hearing and speech impairments in complex scenarios. It achieves a low-cost, highly robust multimodal interaction system that supports bidirectional conversion between sign language and spoken language, and between lip reading and text.
Patent Information
- Application Number
- CN202511916879.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-18
- Publication Date
- 2026-03-20
AI Technical Summary
Existing technologies cannot effectively meet the real-time communication needs of people with hearing and speech impairments in complex scenarios. Traditional sign language interpreters are scarce and expensive. Existing equipment has high hardware costs, poor environmental adaptability, and low semantic conversion accuracy. Speech-to-text tools have low recognition rates in noisy environments, and the differences in word order between sign language and spoken language have not been resolved.
It uses a binocular depth camera to capture hand movements, combines xLSTM neural network modeling and Transformer architecture to reorganize sign language word order, integrates lip reading recognition module and scene adaptation engine to realize a multimodal interaction system, supports bidirectional conversion between sign language and spoken language, and lip reading and text, and optimizes hardware devices with embedded computing.
It achieves low-cost, highly robust multimodal interaction, resolves the differences in word order between sign language and spoken language, provides real-time and accurate communication in complex environments, and supports intelligent perception and alerts in scenarios such as healthcare and transportation.
Smart Images
Figure CN121708655A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of communication devices for people with hearing and speech impairments, and more specifically, to a sign language translation and real-time communication system specifically designed for people with hearing and speech impairments. Background Technology
[0002] There are over 466 million people with hearing and speech impairments worldwide (according to the World Health Organization), facing severe communication barriers in their daily lives. Traditional solutions rely on sign language interpreters or basic assistive devices, but professional sign language interpreters are scarce (only about 10,000 certified interpreters in China), and real-time services are expensive. With the development of artificial intelligence and computer vision technologies, research on sensor- or image-based sign language recognition is gradually emerging, such as data gloves and monocular camera gesture recognition systems. However, limited by hardware costs, environmental adaptability, and semantic conversion accuracy, these technologies have failed to effectively meet the real-time communication needs of hearing-impaired individuals in complex scenarios such as medical care, transportation, and social interactions. Against this backdrop, there is an urgent need for a low-cost, robust, multimodal interaction system to achieve barrier-free bidirectional conversion between sign language and spoken language. Current mainstream technologies have significant flaws: Hearing aids only amplify sound and cannot solve semantic understanding problems. They are also ineffective for people with severe hearing loss, and high-end models cost over 10,000 yuan. Data glove solutions (such as CyberGlove) require wearing dedicated sensors, with a single set costing over $2,000. They can only capture hand movements, ignoring body language and resulting in semantic loss, with an actual recognition accuracy of less than 70%. Visual recognition systems based on monocular cameras (such as MediaPipe Hands) are sensitive to ambient light, with a false recognition rate increasing by more than 30% in strong or low light conditions, and lacking depth information, they have difficulty distinguishing similar gestures. Speech-to-text tools (such as mobile phone voice assistants) have a recognition rate that drops sharply to 50% in noisy environments (>60dB) and cannot handle lip reading input; Existing sign language translation algorithms mostly output sign language word sequences directly, without addressing the word order differences between sign language and spoken language (e.g., the sign language "apple / I / eat" needs to be rearranged into "I eat apple"), resulting in semantic confusion in the output. Therefore, a sign language translation and real-time communication system specifically designed for people with hearing and speech impairments is proposed to address the above issues. Summary of the Invention
[0003] In order to overcome the above-mentioned deficiencies of the prior art, embodiments of the present invention provide a sign language translation and real-time communication system specifically for people with hearing and speech impairments, in order to solve the problems mentioned in the background art.
[0004] To achieve the above objectives, the present invention provides the following technical solution: a sign language translation and real-time communication system for people with hearing and speech impairments, comprising a mobile terminal and a portable hardware device; the mobile terminal is an intelligent device equipped with an operating system, and has a built-in sign language recognition module, lip reading module, speech recognition module, and scene adaptive engine; the portable hardware device includes a small binocular depth camera worn on the chest of the sign language user and an embedded development board for real-time acquisition of three-dimensional limb movement data; the system supports three interaction modes: Mode 1 captures sign language movements through the mobile terminal's camera or the portable device's camera, and outputs spoken text or speech after dynamic gesture recognition and word order recombination; Mode 2 captures the speaker's lip movements through the mobile terminal's front-facing camera, and generates highly robust text by combining the speech signal; Mode 3 captures speech through the mobile terminal's microphone and directly converts it into text output.
[0005] Preferably, the dynamic gesture recognition process of Mode 1 includes: using the YOLOv10 object detection algorithm to locate the hand region in the video frame in real time; extracting the 2.5D coordinates (X, Y, Z depth information) of 21 key points of the hand through the MediaPipe framework; using the xLSTM neural network to model the temporal dependency relationship of the key point sequence to recognize the semantics of dynamic gestures; and finally, through the multimodal semantic synthesis unit integrating the BERT pre-trained model, recombining the word sequence that conforms to the grammatical structure of sign language into a complete sentence according to the spoken word order. The recombination process preserves the contextual association and corrects word order differences, with an output delay of less than 0.5 seconds.
[0006] Preferably, the lip-reading recognition module of Mode 2 adopts the ChLipNet Chinese lip-reading recognition model, which includes a P2P sub-module and a P2CC sub-module. The P2P sub-module extracts lip image sequence features through a convolutional neural network and outputs a Pinyin character sequence through an n-xLSTM network. The P2CC sub-module converts the Pinyin sequence into a Chinese character sequence based on an Encoder-Decoder architecture. The recognition process is combined with a speech recognition auxiliary unit. When the ambient noise exceeds 65 decibels, a dual-modal fusion algorithm is activated to simultaneously analyze the lip movement trajectory and speech spectrum features to improve the recognition accuracy. It also supports a personalized learning engine to dynamically adapt to the user's lip shape features.
[0007] Preferably, the portable hardware device features a small binocular depth camera measuring 8.6cm × 2cm × 2cm, with a built-in infrared structured light projector, capable of generating millimeter-level precision depth point cloud data within a range of 0.1-3 meters; the embedded development board integrates a dynamic gesture recognition accelerator, processes key point temporal data using a lightweight xLSTM model, and connects to a 4G communication module to transmit the recognition results to the mobile terminal in real time; the device's wearing structure adopts a magnetic design, supporting adaptive adjustment of the horizontal angle by ±15 degrees to optimize the sign language capture range.
[0008] Preferably, the scene adaptive engine includes: a GPS positioning unit, an environmental sound classification model, and a situation reasoning algorithm; when a vehicle horn sound is detected and the positioning is in a road area, it triggers the vibration of the mobile terminal and the text warning of "Pay attention to vehicles"; when the hospital call broadcast sound is recognized and matches the user's appointment information, it pushes a medical visit reminder; when the doorbell frequency sound wave is analyzed and the positioning is in a residential area, it starts the "Visitor arrived" reminder; the response delay of all scenarios is less than 200 milliseconds.
[0009] Preferably, the specific implementation method of the word order reorganization algorithm is: establish a sign language vocabulary - spoken language vocabulary mapping library, calculate the semantic weights of each vocabulary in the sign language sequence through an attention mechanism, and use the Transformer architecture to generate a target sentence that conforms to the subject - predicate - object word order; for Chinese characteristics, set rules for complementing function words to automatically add grammatical components such as "de" and "le", and introduce a confusion matrix to correct homophone errors, so that the semantic fidelity of the reorganized sentence reaches more than 95%.
[0010] Preferably, it further includes a spoken language teaching subsystem, which includes: a sign language action standard library that stores 3D key point templates of more than 1000 standard gestures; a practical evaluation module that calculates the matching degree by comparing the Hausdorff distance between the user's gestures and the templates, and evaluates the action fluency based on the dynamic time warping algorithm; a personalized learning path engine that recommends special training courses according to the evaluation results, and the course content is classified according to medical, transportation, and social scenarios.
[0011] Preferably, the implementation of the bimodal fusion algorithm includes: constructing a joint feature space of lip movement - speech spectrum, and eliminating the differences between modalities through CCA canonical correlation analysis; adopting the Late Fusion strategy, and starting the speech feature weighting for segments with a lip - reading recognition confidence lower than 80%, and the weight calculation formula is: , where SNR is the signal - to - noise ratio and k is the environmental adaptation coefficient.
[0012] A sign language translation method includes: continuously collecting the body motion point cloud data of sign language users through a binocular depth camera, using the YOLOv10 algorithm to locate the hand bounding box in each frame of image, extracting the three - dimensional coordinates of 21 key points of the hand through the MediaPipe framework and constructing a time series, inputting the pre - trained xLSTM network classification model to identify the semantics of dynamic gestures, calling the multimodal semantic synthesis unit to reorganize the word order according to spoken language habits and add grammatical function words, and finally outputting the reorganized statement through the TTS engine or the mobile terminal screen. The whole process completes the calculation on the embedded development board side, and only transmits the text result to the mobile terminal to reduce the communication load.
[0013] A lip-reading recognition method includes: capturing 60fps lip movement video using a front-facing camera of a mobile terminal; using a Haar cascade classifier to locate the face region and segment the ROI lip region; inputting 15 consecutive frames of images into the P2P submodule of a ChLipNet model to generate a pinyin sequence; extracting contextual features through the Bi-LSTM encoder of the P2CC submodule; and decoding the sequence into a Chinese character sequence using an attention mechanism; when the environmental signal-to-noise ratio is below 15dB, simultaneously activating a microphone to collect speech signals; extracting MFCC features and performing feature-level fusion with the lip movement trajectory; and optimizing the recognition result through a support vector machine decision boundary.
[0014] The technical effects and advantages of this invention are as follows: Compared to existing technologies, this device integrates computer vision, deep learning, and natural language processing to construct a multimodal interactive system: It utilizes binocular depth cameras to non-contactly capture the 3D motion trajectories of hands and limbs, replacing high-cost data gloves to achieve full-domain motion recognition; it uses YOLOv10 to locate the hand region in real time, extracts key point coordinates using MediaPipe, and models temporal dependencies through an xLSTM neural network to accurately analyze dynamic gesture semantics; it innovatively integrates a multimodal semantic synthesis unit, employing a Transformer architecture to reorganize the order of sign language words and automatically complete grammatical function words, solving the problem of word order differences between sign language and spoken language; the lip-reading module uses a ChLipNet model to process lip images into pinyin sequences and then converts them into Chinese characters, integrating speech signals to form a dual-modal noise reduction mechanism; it also features a scene adaptive engine, linking environmental sound classification and GPS positioning technology to achieve intelligent perception and warnings in scenarios such as medical queuing and traffic avoidance. The system achieves embedded lightweight computing on the hardware side and provides a simple interactive interface on the mobile terminal, ultimately enabling natural, accurate, and real-time two-way communication between people with hearing and speech impairments and able-bodied individuals. Attached Figure Description
[0015] Figure 1 This is a system overall framework diagram of the present invention.
[0016] Figure 2 This is a flowchart illustrating the sign language translation workflow of the present invention.
[0017] Figure 3 This is a flowchart of the lip reading recognition process of the present invention.
[0018] Figure 4 This is a flowchart illustrating the scene linkage process of the present invention.
[0019] Figure 5 This is a basic framework diagram of the computer pronunciation evaluation of the present invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] Example 1 As attached Figure 1-5 As shown, (1) a sign language translation and real-time communication system for people with hearing and speech impairments includes a mobile terminal and a portable hardware device; the mobile terminal is an intelligent device with an operating system, and has a built-in sign language recognition module, lip reading recognition module, speech recognition module and scene adaptive engine; the portable hardware device includes a small binocular depth camera and an embedded development board worn on the chest of the sign language user, used to collect three-dimensional limb movement data in real time; the system supports three interaction modes: mode one captures sign language movements through the mobile terminal camera or portable device camera, and outputs spoken text or speech after dynamic gesture recognition and word order recombination. The system employs three modes: Mode 1 captures the speaker's lip movements using the mobile terminal's front-facing camera and combines this with the speech signal to generate robust text; Mode 2 captures speech using the mobile terminal's microphone and directly converts it into text. The system utilizes a combination of a mobile terminal and portable hardware: the mobile terminal is a smartphone with a customized app, integrating three core modules—sign language recognition, lip reading, and speech recognition—as well as a scene-adaptive engine; the portable hardware includes a chest-mounted binocular depth camera (such as the OV9282 sensor, 1280×800 resolution / 60fps) and an embedded development board (such as the Rockchip RK3566 chip). The binocular camera generates 3D point cloud data through infrared structured light projection, and the development board runs a lightweight gesture recognition model. When a hearing-impaired person wears the device to capture limb movements, a hearing-impaired person selects an interaction mode through the app, and the system dynamically activates sign language to speech, lip reading to text, or speech to text conversion based on the scenario.
[0022] (2) The dynamic gesture recognition process of Mode 1 includes: using the YOLOv10 object detection algorithm to locate the hand region in the video frame in real time, extracting the 2.5D coordinates (X, Y, Z depth information) of 21 key points of the hand through the MediaPipe framework, using the xLSTM neural network to model the temporal dependency relationship of the key point sequence to recognize the dynamic gesture semantics, and finally, through the multimodal semantic synthesis unit integrating the BERT pre-trained model, recombining the word sequence that conforms to the sign language grammar structure into a complete sentence according to the spoken word order. The recombination process retains the context association and corrects the word order difference, with an output delay of less than 0.5 seconds. The dynamic gesture recognition process begins with the real-time detection of the hand region by the YOLOv10 model. After outputting the bounding box coordinates, the cropped area image is input into the MediaPipe framework: the BlazePalm model locates the palm, and the hand landmark model calculates the 2.5D coordinates (X / Y pixel position + Z depth value) of 21 key points. The key point sequence of 30 consecutive frames is input into a three-layer xLSTM network (128 units per layer) to parse the dynamic gesture semantic encoding. The encoding is processed by the Transformer reorganization module, which adjusts the word order (e.g., "hospital / I / go" → "I go to the hospital") and completes grammatical function words through a multi-head attention mechanism. Finally, the TTS engine or screen outputs natural sentences.
[0023] (3) The lip reading recognition module of Mode 2 adopts the ChLipNet Chinese lip reading recognition model, which includes a P2P sub-module and a P2CC sub-module. The P2P sub-module extracts the lip image sequence features through a convolutional neural network and outputs the pinyin character sequence through an n-xLSTM network. The P2CC sub-module converts the pinyin sequence into a Chinese character sequence based on the Encoder-Decoder architecture. The recognition process is combined with a speech recognition auxiliary unit. When the ambient noise exceeds 65 dB, a dual-modal fusion algorithm is started to simultaneously analyze the lip movement trajectory and speech spectrum features to improve the recognition accuracy. It also supports a personalized learning engine to dynamically adapt to the user's lip shape features. The lip reading recognition is achieved through layered processing: after the Haar cascade classifier locates the face, the lip ROI region is segmented. 15 consecutive frames of images are input into the P2P sub-module of the ChLipNet model—the CNN convolutional layer—to extract spatial features. The n-xLSTM generates the pinyin sequence. The pinyin sequence is processed by the Bi-LSTM encoder of the P2CC sub-module to extract contextual features, and the attention decoder outputs the Chinese character sequence. When the ambient noise exceeds 65dB, speech signal acquisition is initiated simultaneously. MFCC features are fused with lip movement trajectories, and the results are weighted according to the signal-to-noise ratio adaptive formula. Final decision document.
[0024] (4) The portable hardware device features a small binocular depth camera measuring 8.6cm × 2cm × 2cm, with a built-in infrared structured light projector, capable of generating millimeter-level precision depth point cloud data within a range of 0.1-3 meters. The embedded development board integrates a dynamic gesture recognition accelerator, processes key point time-series data through a lightweight xLSTM model, and connects to a 4G communication module to transmit the recognition results to the mobile terminal in real time. The device's wearing structure adopts a magnetic design, supporting adaptive adjustment of ±15 degrees horizontally to optimize the sign language capture range. The portable hardware uses a parallel optical axis binocular camera (baseline distance 60mm), with a built-in 850nm infrared fill light, achieving a depth accuracy of ±2mm within a range of 0.3-3 meters. The embedded development board deploys an FP16 quantized xLSTM model, communicating with the mobile terminal via a 4G module (SIM7600CE). The magnetic housing design, combined with a ratchet adjustment mechanism (5 degrees / level), ensures that the camera's pitch angle adapts to the sign language capture needs of users of different heights.
[0025] (5) The scene adaptive engine includes: a GPS positioning unit, an environmental sound classification model, and a contextual reasoning algorithm; when a vehicle horn is detected and the location is in a road area, the mobile terminal is triggered to vibrate and display a "Beware of Vehicles" text warning; when a hospital call-up announcement is identified and matched with the user's appointment information, a visit reminder is pushed; when the doorbell frequency sound wave is analyzed and the location is in a residential area, a "Visitor Arrival" prompt is initiated; the response latency for all scenarios is less than 200 milliseconds. The scene adaptive engine continuously monitors the environmental audio stream: the YAMNet model identifies vehicle horns (2-4kHz frequency band), doorbell pulses (1-3kHz), and bus stop announcement voice features. When the GPS is located on the road and a horn is detected, the device is triggered to vibrate and display a "Vehicle Approaching" warning; in the hospital scenario, after the voice features match the appointment information, a "Please go to Clinic Room 3" prompt is pushed. The end-to-end latency from voiceprint recognition to action execution is strictly controlled within 200ms.
[0026] (6) The specific implementation method of the word order reorganization algorithm is as follows: establish a sign language vocabulary - spoken language vocabulary mapping library, calculate the semantic weights of each vocabulary in the sign language sequence through the attention mechanism, and use the Transformer architecture to generate a target sentence that conforms to the subject-predicate-object word order; for Chinese characteristics, set up a function word completion rule to automatically add grammatical components such as "de" and "le", and introduce a confusion matrix to correct homophone errors, so that the semantic fidelity of the reorganized sentence reaches more than 95%. Among them, the word order reorganization algorithm relies on a sign language - spoken language vocabulary mapping library of 5000 entries, and calculates the sequence weights through the Query-Key attention mechanism: for example, in the input sequence ["apple", "I", "eat"], "I" is placed at the beginning of the sentence due to the highest weight. The function word completion module dynamically adds grammatical components - add "le" when identifying a verb ("eat" → "ate"), and add "de" for the possessive relationship ("mom / bag" → "mom's bag"), so that the output sentence conforms to the spoken language habit.
[0027] (7) It also includes a spoken language teaching subsystem, which includes: a sign language action standard library that stores 3D key point templates of more than 1000 standard gestures; a practical evaluation module that calculates the matching degree by comparing the Hausdorff distance between the user's gestures and the templates, and evaluates the action fluency based on the dynamic time warping algorithm; a personalized learning path engine that recommends special training courses according to the evaluation results, and the course content is classified according to medical, transportation, and social scenarios. Among them, the spoken language teaching subsystem is implemented through a mobile terminal APP: the built-in sign language action standard library stores 3D key point templates of 1000 standard gestures (covering medical, transportation, and social scenarios); during practical evaluation, the user's camera captures a gesture video, and the system extracts a sequence of key points of consecutive frames, calculates the Hausdorff distance from the template to evaluate the spatial matching degree (a distance ≤ 5mm is judged as qualified), and at the same time applies the dynamic time warping algorithm (DTW) to align the time axes of the user's and template actions and calculate the trajectory similarity score; according to the evaluation results (comprehensive spatial and temporal scores), the personalized learning engine pushes special courses according to the weak scenario - for example, the "hospital scenario gesture pack" includes 20 high-frequency word gesture trainings such as registering and consulting a doctor.
[0028] (8) The implementation of the bimodal fusion algorithm includes: constructing a joint feature space of lip movement - speech spectrum, and eliminating the differences between modalities through CCA canonical correlation analysis; adopting the Late Fusion strategy, and starting the speech feature weighting for segments with a lip reading recognition confidence lower than 80%. The weight calculation formula is: , where SNR is the signal-to-noise ratio and k is the environmental adaptation coefficient. Among them, The dual-modal fusion process is as follows: The lip movement trajectory is used to generate a motion vector field via an optical flow algorithm; MFCC+Mel spectrum features are extracted from the speech signal; Canonical correlation analysis (CCA) is used to construct a joint feature space to eliminate modal differences; when the lip-reading confidence level is <80%, speech weighting is initiated, with the weighting formula... The k value is dynamically adjusted according to the environment (k=0.5 indoors, k=1.2 outdoors); the weighted bimodal features are input into the SVM classifier, and the decision boundary is optimized by max(0.7×P_lip +w×P_audio) to output the final text.
[0029] (9) A sign language translation method, comprising: continuously acquiring point cloud data of sign language users' limb movements using a binocular depth camera; locating the hand bounding box in each frame image using the YOLOv10 algorithm; extracting the three-dimensional coordinates of 21 key points of the hand and constructing a temporal sequence using the MediaPipe framework; inputting the data into a pre-trained xLSTM network classification model to recognize dynamic gesture semantics; calling a multimodal semantic synthesis unit to reorganize the word order according to spoken language habits and add grammatical function words; and finally outputting the reorganized sentence through a TTS engine or mobile terminal screen. The entire process is completed on the embedded development board, and only the text result is transmitted to the mobile terminal to reduce the communication load. Among them, the embedded optimization scheme realizes the local computation closed loop: the binocular camera synchronously outputs RGB frames and point cloud data, and the RK3566 chip runs the YOLOv10-Tiny model (2.5W power consumption) to crop the hand area in real time; MediaPipe Lite extracts the coordinates of 21 key points (<1KB per frame); after the xLSTM network completes the temporal analysis and word order reconstruction, it only transmits the text results to the mobile phone, reducing the communication load to 1 / 100 of that of video transmission.
[0030] (10) A lip reading recognition method, comprising: capturing 60fps lip movement video through the front-facing camera of a mobile terminal, using a Haar cascade classifier to locate the face region and segment the ROI lip region, inputting 15 consecutive frames of images into the P2P submodule of the ChLipNet model to generate a pinyin sequence, extracting contextual features through the Bi-LSTM encoder of the P2CC submodule, and decoding it into a Chinese character sequence by combining an attention mechanism; when the environmental signal-to-noise ratio is lower than 15dB, simultaneously starting the microphone to collect speech signals, extracting MFCC features and performing feature-level fusion with the lip movement trajectory, and then using a support vector machine. The decision boundary optimization results are as follows: The entire lip-reading recognition process is as follows: The front-facing camera of the mobile terminal captures 1080P face video at 60fps. After the Haar cascade classifier detects the face, the lip ROI (lower lip center point ± 15 pixels region) is segmented based on the YCbCr skin color model. 15 consecutive frames of ROI images are input into the P2P submodule of ChLipNet—a 4-layer CNN (3×3 convolutional kernel) to extract features, and an n-xLSTM (64 units) outputs a pinyin sequence. The pinyin sequence is encoded by a Bi-LSTM encoder (256 hidden layers) to generate a context vector, and the attention decoder is then used to... Calculate the probability distribution of Chinese characters; when the signal-to-noise ratio is <15dB, extract the speech MFCC features simultaneously, and recalculate the decoding weights after fusing them with the lip motion vector CCA.
[0031] Example 2: Multi-source data joint modeling scenario 1. System Initialization and Hardware Preparation The user wears a portable binocular depth camera (8.6cm×2cm×2cm) on their chest via a magnetic attachment. The camera's infrared illuminator (850nm wavelength) automatically turns on, generating depth point cloud data. The embedded development board (Rockchip RK3566 chip) starts a lightweight operating system, loading a pre-trained YOLOv10-Tiny model and an xLSTM gesture recognition network. The mobile app connects to the portable device via Bluetooth pairing. The initial interface offers three mode options: "sign language translation," "lip reading," and "voice input," as well as a scene warning switch.
[0032] 2. Sign Language Interpretation Workflow When a hearing-impaired person uses sign language: Step 1: 3D motion capture The binocular camera acquires RGB images and depth maps at 60fps, and generates 3D coordinates of key hand points (accuracy ±2mm) through parallax calculation, while simultaneously detecting limb movement trajectories.
[0033] Step 2: Real-time gesture recognition The YOLOv10-Tiny model locates the hand bounding box in the video frame on the embedded side (time-consuming < 50 ms), and the cropped area is input into the MediaPipe Lite framework to extract 21 key points (X / Y pixel coordinates + Z depth value); the sequence of 30 consecutive key points is input into a three-layer xLSTM network (128 units per layer), and a dynamic gesture semantic encoding is output (for example, encoding #203 corresponds to "hospital").
[0034] Step 3: Oral semantic recombination The semantic encoding is input into the Transformer recombination module, which adjusts the word order through attention weights (for example, reorganizes ["I", "hospital", "go"] into ["I", "go", "hospital"]), and automatically completes function words based on Chinese grammar rules (adds "le" → "I went to the hospital").
[0035] Step 4: Multimodal output The recombined text is transmitted to the mobile terminal through the 4G module, converted into voice broadcast by the TTS engine, and at the same time, the text and corresponding sign language animations are displayed on the screen (feedback delay < 0.5 seconds).
[0036] 3. Working process of lip reading recognition mode When a person with normal hearing speaks facing the mobile terminal camera: Step 1: Lip movement capture The front camera captures the face video at 60 fps, and the Haar cascade classifier locates the lip ROI area (160×80 pixels) and extracts 15 consecutive frames of images.
[0037] Step 2: Hierarchical semantic conversion The image sequence is input into the ChLipNet model: P2P sub-module: A 4-layer CNN (convolution kernel 3×3) extracts lip texture features, and the n-xLSTM outputs a pinyin sequence (such as "wo3 qu4 yi1 yuan4"); P2CC sub-module: The Bi-LSTM encoder analyzes the pinyin context, and the attention decoder generates a Chinese character sequence ("I go to the hospital").
[0038] Step 3: Anti-noise decision-making mechanism When the environmental noise > 65 dB, the microphone is synchronously started to collect speech, and the canonical correlation analysis (CCA) of the MFCC features and the lip movement trajectory is calculated; if the lip reading confidence < 80%, then according to the weight formula Fuse the speech features and output the final text to the screen.
[0039] 4. Speech input and scene linkage Speech-to-text: The mobile terminal's microphone captures speech, and the text is directly output through the on-device Transformer model (6 layers of encoder), supporting offline use.
[0040] Road safety: When GPS is located on the street and YAMNet detects a horn sound (2-4kHz pulse), the device vibrates and a full-screen "Caution: Vehicles" warning pops up. Medical reminder: When a call-up announcement (matching the user's appointment name) is detected within the hospital premises, the screen will display "Please go to XX clinic" and a countdown timer; Home doorbell: If a 1-3kHz pulse sound is detected and the location is a residential area, a flashing alert will be activated and a message "Visitor has arrived" will be displayed.
[0041] 5. Exception Handling and Optimization Mechanism Hand occlusion compensation: When key points are lost for 5 consecutive frames, the motion trajectory prediction algorithm (Kalman filter) is enabled to complete the data; Low-light enhancement: The camera's IR fill light automatically brightens in environments below 50 lux to ensure effective acquisition of depth information; Energy management: Enters sleep mode after 10 minutes without gesture input; vibration wake-up response time is less than 1 second.
[0042] Technical Effects of the Examples This process fully presents a closed-loop chain from hardware data acquisition (binocular camera point cloud + microphone audio) → embedded real-time processing (YOLOv10 localization + xLSTM temporal modeling) → semantic conversion optimization (Transformer word order reordering / ChLipNet layered decoding) → multi-terminal interaction (TTS broadcasting / graphical display). At the same time, it realizes intelligent collaboration between the physical environment and digital system through scene linkage engine, achieving the core goal of two-way barrier-free communication between sign language and spoken language.
[0043] Finally, the following points should be noted: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection", and "linkage" should be interpreted broadly, and can be mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change. Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other. In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A sign language translation and real-time communication system for people with hearing and speech impairments, characterized in that, The system includes mobile terminals and portable hardware devices. The mobile terminals are smart devices with an operating system, and include built-in sign language recognition, lip reading, speech recognition, and scene adaptation engines. The portable hardware devices include a small binocular depth camera worn on the chest of the sign language user and an embedded development board for real-time acquisition of three-dimensional body movement data. The system supports three interaction modes: Mode 1 captures sign language movements through the mobile terminal's camera or the portable device's camera, and outputs spoken text or speech after dynamic gesture recognition and word order reconstruction; Mode 2 captures the speaker's lip movements through the mobile terminal's front-facing camera, and generates highly robust text by combining the speech signal; Mode 3 captures speech through the mobile terminal's microphone and directly converts it into text for output.
2. The sign language translation and real-time communication system for people with hearing and speech impairments according to claim 1, characterized in that, The dynamic gesture recognition process of Mode 1 includes: using the YOLOv10 object detection algorithm to locate the hand region in the video frame in real time; extracting the 2.5D coordinates (X, Y, Z depth information) of 21 key points of the hand through the MediaPipe framework; using the xLSTM neural network to model the temporal dependency of the key point sequence to recognize the semantics of dynamic gestures; and finally, through the multimodal semantic synthesis unit integrating the BERT pre-trained model, recombining the word sequence that conforms to the grammatical structure of sign language into a complete sentence according to the spoken word order. The recombination process preserves the contextual association and corrects word order differences, with an output delay of less than 0.5 seconds.
3. The sign language translation and real-time communication system for people with hearing and speech impairments according to claim 1, characterized in that, The lip-reading recognition module in Mode 2 adopts the ChLipNet Chinese lip-reading recognition model, which includes a P2P submodule and a P2CC submodule. The P2P submodule extracts lip image sequence features through a convolutional neural network and outputs a Pinyin character sequence through an n-xLSTM network. The P2CC submodule converts the Pinyin sequence into a Chinese character sequence based on an Encoder-Decoder architecture. The recognition process is combined with a speech recognition auxiliary unit. When the ambient noise exceeds 65 decibels, a dual-modal fusion algorithm is activated to simultaneously analyze the lip movement trajectory and speech spectrum features to improve the recognition accuracy. It also supports a personalized learning engine to dynamically adapt to the user's lip shape features.
4. The sign language interpretation and real-time communication system for people with hearing and speech impairments according to claim 1, characterized in that, The portable hardware device has a small binocular depth camera measuring 8.6cm×2cm×2cm, and a built-in infrared structured light projector, which can generate depth point cloud data with millimeter-level accuracy within a range of 0.1-3 meters. The embedded development board integrates a dynamic gesture recognition accelerator, which processes key point time-series data through a lightweight xLSTM model and connects to a 4G communication module to transmit the recognition results to the mobile terminal in real time. The device's wearing structure adopts a magnetic design and supports adaptive adjustment of the horizontal angle by ±15 degrees to optimize the sign language capture range.
5. A sign language translation and real-time communication system for people with hearing and speech impairments according to claim 1, characterized in that, The scene adaptive engine includes: a GPS positioning unit, an environmental sound classification model, and a contextual reasoning algorithm; when a vehicle horn is detected and the location is in a road area, the mobile terminal is triggered to vibrate and display a "Watch out for vehicles" text warning; when a hospital call-up announcement is identified and matched with the user's appointment information, a visit reminder is pushed; when the doorbell frequency is analyzed and the location is in a residential area, a "Visitor arrives" prompt is activated; the response latency for all scenes is less than 200 milliseconds.
6. A sign language translation and real-time communication system for people with hearing and speech impairments according to claim 2, characterized in that, The specific implementation method of the word order recombination algorithm is as follows: establish a sign language vocabulary - spoken language vocabulary mapping library, calculate the semantic weights of each vocabulary in the sign language sequence through the attention mechanism, and use the Transformer architecture to generate a target sentence that conforms to the subject-predicate-object word order; for the characteristics of Chinese, set up a virtual word completion rule to automatically add grammatical components such as "de" and "le", and introduce a confusion matrix to correct homophone errors, so that the semantic fidelity of the recombined sentence reaches more than 95%.
7. A sign language translation and real-time communication system for people with hearing and speech impairments according to claim 1, characterized in that, It also includes a spoken language teaching subsystem, which includes: a sign language action standard library that stores 3D key point templates of more than 1000 standard gestures; a practical evaluation module that calculates the matching degree by comparing the Hausdorff distance between the user's gestures and the templates, and evaluates the action fluency based on the dynamic time warping algorithm; a personalized learning path engine that recommends special training courses according to the evaluation results, and the course content is classified according to medical, transportation, and social scenarios.
8. A sign language translation and real-time communication system for people with hearing and speech impairments according to claim 3, characterized in that, The implementation of the bimodal fusion algorithm includes: constructing a joint feature space of lip movement - speech spectrum, and eliminating the differences between modalities through CCA canonical correlation analysis; adopting the Late Fusion strategy, and starting the speech feature weighting for the segments with a lip reading recognition confidence lower than 80%. The weight calculation formula is: Where SNR is the signal-to-noise ratio and k is the environmental adaptability coefficient.
9. A sign language translation method based on the system of any one of claims 1-8, characterized in that, including: Continuously collect the body motion point cloud data of sign language users through a binocular depth camera, use the YOLOv10 algorithm to locate the hand bounding box in each frame of image, extract the three-dimensional coordinates of 21 key points of the hand through the MediaPipe framework and construct a time series, input it into a pre-trained xLSTM network classification model to identify the semantics of dynamic gestures, call the multimodal semantic synthesis unit to recombine the vocabulary order according to spoken language habits and add grammatical function words, and finally output the recombined sentence through the TTS engine or the mobile terminal screen. The whole process completes the calculation on the embedded development board side, and only transmits the text result to the mobile terminal to reduce the communication load.
10. A lip-reading recognition method based on the system of any one of claims 1-8, characterized in that, including: Capture a 60fps lip movement video through the front camera of the mobile terminal, use the Haar cascade classifier to locate the face area and segment the ROI lip area, input 15 consecutive frames of images into the P2P sub-module of the ChLipNet model to generate a pinyin sequence, extract the context features through the Bi-LSTM encoder of the P2CC sub-module, and decode it into a Chinese character sequence through the attention mechanism; when the environmental signal-to-noise ratio is lower than 15dB, synchronously start the microphone to collect the speech signal, extract the MFCC features and the lip movement trajectory for feature-level fusion, and optimize the recognition result through the support vector machine decision boundary.