Deep learning-based sign language-to-multilingual text speech mutual conversion method and application thereof

Abstract

The invention relates to a sign language-to-multilingual text speech mutual conversion method based on deep learning and application thereof, belongs to the field of artificial intelligence, man-machine interaction and barrier-free communication, and particularly relates to a sign language recognition, semantic understanding and multilingual speech synthesis method based on deep learning and an application system thereof. The sign language to voice text conversion method comprises the following steps: 1, an image acquisition and preprocessing module, 2, a hand key point detection module, 3, time sequence modeling and sign language recognition, 4, natural language understanding and semantic optimization, 5, a multi-language translation module, and 6, a text and voice synthesis output module. According to the sign language-to-multilingual text voice mutual conversion method based on deep learning and the application thereof, sign languages can be automatically, flexibly and accurately converted into voices and characters, and deaf-mutes are helped to efficiently communicate with common people; the system can be widely applied to hospitals, schools, government affair windows, public transportation and other scenes.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, human-computer interaction and barrier-free communication, and specifically to a deep learning-based sign language recognition, semantic understanding and multilingual speech synthesis method and its application system. Background Art

[0002] According to estimates from the China Disabled Persons' Federation and the Seventh National Population Census, the number of people with hearing impairments is approximately 27 million. The number of deaf people who actually use sign language is estimated to be between 10 and 20 million (including those with congenital and acquired hearing loss, as well as those who are deaf-mute). Globally, there are approximately 430 million people with hearing impairments, of whom approximately 360 million have severe hearing impairments (≥35 decibels). Approximately 70 million deaf people use sign language as their native language.

[0003] Sign languages ​​around the world are mainly divided into: ASL (American Sign Language), BSL (British Sign Language), CSL (Chinese Sign Language), and International Sign Language (IS). International Sign Language is a "hybrid sign language" used for international deaf conferences and cross-border communication. These sign languages ​​are not mutually exclusive and are independent languages, not dependent on spoken language.

[0004] Sign languages ​​in many countries or regions rely on gestures, fingers, arms, facial expressions, and the distance and size of movement amplitudes to express corresponding meanings. There are already some corresponding sign language translation software tools on the market, but most of these software tools rely on rule libraries or template matching, with low recognition efficiency and accuracy. They are also mostly limited to single-language text output (such as Chinese), and cannot achieve free conversion between multiple languages ​​or natural voice broadcasting. In addition, traditional systems have difficulty achieving a deep understanding of dynamic gestures, facial expressions, and contextual semantics, resulting in unnatural interactions, incoherent translations, and inaccurate expressions.

[0005] Furthermore, there is a lack of lightweight models that are compatible with multiple terminals (such as mobile phones, wearable devices, and computer equipment in public places), which limits the application scenarios. Summary of the Invention

[0006] The purpose of the present invention is to address the defects and shortcomings of the existing technology and provide a deep learning-based sign language to multilingual text-to-speech conversion method and its application. It can automatically and flexibly and accurately convert sign language into speech and text, helping the deaf and mute people to communicate efficiently with the general public; it can be widely used in hospitals, schools, government service windows, public transportation and other scenarios.

[0007] To achieve the above objectives, the method for converting sign language to speech text of the present invention is as follows: 1. Image acquisition and preprocessing module, 2. Hand key point detection module, 3. Time series modeling and sign language recognition, 4. Natural language understanding and semantic optimization, 5. Multilingual translation module, 6. Text-to-speech synthesis output module;

[0008] The image acquisition and preprocessing module in step 1: uses a camera, mobile device, or smart wearable device to capture the user's sign language video in real time; the video frame rate is recommended to be set at 25-30fps to ensure the integrity of the movement; the image is preprocessed by denoising, illumination normalization, background segmentation, etc. to ensure the accuracy of subsequent recognition;

[0009] The hand key point detection module in step 2 described above uses MediaPipe Hands or a self-developed high-precision regression network (such as one based on the BlazePose+Hourglass structure) to extract 21 hand key points from each frame of the image; the output is an array of 3D spatial coordinates (x, y, z) of the hand for each frame, and a timestamp is recorded; the data of both hands is fused and modeled, retaining the motion trajectory and symmetry information; this step realizes the "digital modeling" of the sign language movement, providing a basis for subsequent time series recognition;

[0010] Step 3 of the above, temporal modeling and sign language recognition: input the frame-level key point sequence into a temporal modeling network, such as Bi-LSTM, Transformer, TCN, etc.; use CTC (Connectionist Temporal Classification) loss or attention mechanism to train the model to achieve the mapping of "gesture sequence → sign language semantic unit"; support both continuous sign language recognition and phrase recognition modes to enhance semantic integrity;

[0011] The step 3 is to input the time series features composed of the hand key points (or whole body key points) extracted from each frame of the video into a temporal modeling network to learn its contextual dependencies, thereby recognizing the complete sign language semantics or text representation.

[0012] Step 4, natural language understanding and semantic optimization: uses pre-trained language models (such as BERT and RoBERTa) to perform grammatical correction, word order adjustment, and sentence reorganization on the recognized sign language text; supports contextual linkage, which can optimize the current semantic output based on the user's historical expressions; adds keyword extraction and intent recognition modules to facilitate subsequent language mapping; because the output of the sign language recognition model is a "literal translation" text, it often has problems such as improper word order, incomplete grammar, and ambiguous word meanings, which affect the accuracy of multilingual translation and speech synthesis. Therefore, this step introduces a pre-trained language model (such as BERT and RoBERTa) to perform natural language optimization on the initial text.

[0013] The multilingual translation module in step 5: uses a multilingual neural translation model (such as mBART, mT5, or Transformer-Base) to convert Chinese sign language text into the target language (such as English or Spanish). A lightweight translation model (such as Distilled mBART) can be executed locally on the terminal to reduce latency. It supports two-way translation (text to sign language) and allows users to set the target language.

[0014] The text-to-speech synthesis output module in step 6: performs speech synthesis based on the FastSpeech2+HiFi-GAN architecture to generate natural intonation; the model can switch voice packages to adapt to different genders, accents and languages; supports real-time voice stream output or text + voice dual output mode; supports voice dialogue return, and realizes voice recognition + virtual sign language display module for hearing sign language users; the system can be deployed on multiple platforms such as mobile phone apps, web pages, and smart screens.

[0015] It collects sign language videos through a camera, uses a convolutional neural network to extract the coordinates of key hand points, and converts them into time series data. It uses a bidirectional LSTM or Transformer model for dynamic action recognition and semantic analysis, and combines pre-trained language models to optimize grammatical and semantic structures. The recognition results are then translated into text in multiple languages ​​through a multilingual neural translation model. Finally, deep speech synthesis technologies such as FastSpeech2 and HiFi-GAN are used to convert the text into natural speech, realizing intelligent conversion from sign language to multilingual text and speech, with the advantages of high precision, low latency and good cross-language adaptability.

[0016] The method for converting speech and text to sign language is as follows: 1. Voice input or text input; 2. Text semantic analysis (natural language understanding NLU); 3. Sign language semantic mapping (Sign Gloss Mapping); 4. Gesture action sequence generation (Pose Sequence Generation); 5. Gesture rendering and output; Through speech recognition or text input, the system first extracts language semantics and maps them to sign language semantic units (Gloss), then generates continuous gesture action sequences through deep learning models or standard libraries, and finally outputs sign language through virtual signers or robots, achieving high-fidelity conversion from "auditory language" to "visual language".

[0017] In step 1, voice input or text input is used. If it is voice, a speech recognition engine (ASR, Automatic Speech Recognition) is used to convert the voice into text. Available models include: Wav2Vec2.0, DeepSpeech, Google ASR API, etc.

[0018] Step 2: Text semantic analysis (Natural Language Understanding (NLU)) to extract intent, entities, and semantic units, simplifying them into content that can be expressed in sign language; word segmentation + part-of-speech tagging (such as using BERT-CRF); entity recognition (NER); syntax tree parsing; and intent classification.

[0019] Step 3: Sign Gloss Mapping: Map text in ordinary Chinese or other languages ​​to sign language semantic symbols (gloss, sign language symbol words); establish a glossary of sign language mappings, using rule mapping, statistical models, or Transformer-based encoder-decoder models.

[0020] Step 4: Pose Sequence Generation: Deep learning-based action generation. Use a Seq2Seq model (e.g., Transformer / LSTM) to directly convert the gloss into a 3D hand keypoint sequence output format (hand joints per frame).

[0021] Step 5: Gesture rendering and output, Method 1: 3D hand animation, use a 3D rendering engine (such as Unity / Unreal / Blender) to drive a virtual sign language model and convert action key points into skeletal animation.

[0022] The working principle of this invention: It uses hand key point detection to recognize complex, continuous, dynamic sign language. It supports a variety of hand shapes and body combinations to adapt to different regional or national sign language standards. It reduces rigid "word-by-word translation" and outputs text expressions that are more in line with natural language habits. It can not only output Chinese, but also achieve simultaneous translation in languages ​​such as English and Spanish through a multilingual translation model.

[0023] The application scenarios of sign language to multilingual text-to-speech conversion include: public services, education, healthcare, justice, social interaction, and smart hardware. It is particularly important in promoting two-way communication between the deaf and hearing people. In government halls, hospitals, banks, courts and other window units, by deploying this system on touch-screen all-in-one machines or voice interactive terminals, hearing users can input voice or text, and the system will convert it into visual standard sign language animation or bionic robot sign language in real time, achieving barrier-free communication. In special education and rehabilitation teaching, the system can be used as an auxiliary teaching tool to support teachers in translating course content into sign language to improve learning efficiency. In personal terminal devices (such as smartphones and AR glasses), the system can be used as a "sign language translation assistant" plug-in to achieve an application experience that can be carried with you and used at any time. At the same time, it supports multilingual conversion between Chinese and English and other languages ​​in a multilingual environment, with good cross-national and cross-cultural scalability.

[0024] After adopting the above technical solution, the beneficial effects of the present invention are: it can automatically convert sign language into voice and text, helping the deaf and dumb to communicate efficiently with the general public; it can be widely used in hospitals, schools, government service windows, public transportation and other scenarios. The system can run on mobile phones, tablets, PCs or edge devices (such as Jetson); it provides channels for independent expression and understanding for people with disabilities, and promotes information equality; it is in line with the national development direction of intelligent assistive technology and has a good social influence. The present invention builds an efficient, natural and scalable cross-language sign language translation system through the integration of deep learning sign language recognition, multilingual understanding and speech synthesis technology, which greatly improves the human-computer interaction experience and barrier-free communication capabilities, and has significant social value and industrial potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 is a flow chart of the conversion from sign language to speech text in the present invention;

[0027] Figure 2 This is a flow chart of the conversion from speech text to sign language in the present invention;

[0028] Figure 3 Schematic diagram of the Yangtze River applied in Example 1 of the present invention. DETAILED DESCRIPTION

[0029] See Figure 1-2As shown, the method for converting sign language to speech text in this specific embodiment is as follows: 1. Image acquisition and preprocessing module, 2. Hand key point detection module, 3. Time series modeling and sign language recognition, 4. Natural language understanding and semantic optimization, 5. Multilingual translation module, 6. Text-to-speech synthesis output module;

[0030] The image acquisition and preprocessing module in step 1: uses a camera, mobile device, or smart wearable device to capture the user's sign language video in real time; the video frame rate is recommended to be set at 25-30fps to ensure the integrity of the movement; the image is preprocessed by denoising, illumination normalization, background segmentation, etc. to ensure the accuracy of subsequent recognition;

[0031] In step 1, the image denoising uses Gaussian filtering to perform local weighted averaging on the image. The weights are determined by a two-dimensional Gaussian distribution. This can smooth the image, remove high-frequency noise, and maintain a good edge state for the image. The specific algorithm is:

[0032]

[0033] Among them, σ is the standard deviation, which determines the degree of blur.

[0034] In step 1, illumination normalization requires that the image brightness be standardized to enhance the robustness of key point recognition in the presence of different illumination conditions in the acquisition environment. Gamma correction is used, specifically:

[0035] I′(x, y) = A·I(x, y) γ

[0036] in:

[0037] γ<1: Enhance dark details;

[0038] γ>1: compress highlights and suppress overexposure;

[0039] A: scaling factor, usually 1;

[0040] Normalize the image brightness range to make the model more robust to different lighting conditions.

[0041] The hand key point detection module in step 2 described above uses MediaPipe Hands or a self-developed high-precision regression network (such as one based on the BlazePose+Hourglass structure) to extract 21 hand key points from each frame of the image; the output is an array of 3D spatial coordinates (x, y, z) of the hand for each frame, and a timestamp is recorded; the data of both hands is fused and modeled, retaining the motion trajectory and symmetry information; this step realizes the "digital modeling" of the sign language movement, providing a basis for subsequent time series recognition;

[0042] The position coordinates (x, y) of 21 hand key points are extracted from each frame of the image to describe the structure and dynamic behavior of the gesture. Common key points are as follows:

[0043] 0: wrist;

[0044] 1-4: thumb;

[0045] 5-8: index finger;

[0046] 9-12: middle finger;

[0047] 13-16: ring finger;

[0048] 17~20: little finger;

[0049] The specific algorithm is:

[0050] 2.1. Feature Extraction: Use a CNN network (such as ResNet or MobileNet) to extract image features:

[0051] F=CNN(I)

[0052] Output dimensions are typically:

[0053] 2.2. Heatmap Generation

[0054] Use 1×1 convolution to map the feature map into K heat maps, each heat map represents the location distribution of a key point:

[0055] H k =Conv 1×1 (F), k=1,2,...,K

[0056] In each heatmap, the ideal distribution of keypoints is a two-dimensional Gaussian:

[0057]

[0058] 2.3. Coordinate Regression

[0059] Extract the maximum response point from each heat map as the keypoint coordinates:

[0060]

[0061] A more refined approach is to use a weighted average method (soft-argmax) to improve sub-pixel accuracy:

[0062]

[0063] 2.4. Timing smoothing and filtering: Use Kalman filtering or exponential sliding average to smooth out key point jitter between multiple frames and improve timing stability:

[0064]

[0065] Where α∈[0,1] is the smoothing coefficient, and the recommended value is about 0.7;

[0066] 2.5. Loss function (used during training),

[0067] The training goal is to minimize the difference (mean squared error) between the predicted heatmap and the true heatmap:

[0068]

[0069] 2.6. Output structure and obtain the two-dimensional coordinates of 21 key points:

[0070]

[0071] The confidence value of each key point (obtained from the maximum value of the heat map):

[0072]

[0073] This information will be fed into the next stage: gesture timing modeling (such as LSTM / Transformer) for dynamic sign language recognition.

[0074] Step 3 of the above, temporal modeling and sign language recognition: input the frame-level key point sequence into a temporal modeling network, such as Bi-LSTM, Transformer, TCN, etc.; use CTC (Connectionist Temporal Classification) loss or attention mechanism to train the model to achieve the mapping of "gesture sequence → sign language semantic unit"; support both continuous sign language recognition and phrase recognition modes to enhance semantic integrity;

[0075] Step 3 is to input the time series features composed of the hand key points (or whole body key points) extracted from each frame of the video into a temporal modeling network to learn its contextual dependencies, thereby recognizing the complete sign language semantics or text representation. The specific process is as follows:

[0076] Assume that the original input is a sign language video, extract T frames in total, and extract N key point coordinates (usually 2D or 3D coordinates) from each frame:

[0077] Key point vector for each frame:

[0078]

[0079] Or add confidence to become

[0080] The overall sequence input is:

[0081]

[0082] Where D = 2N or 3N.

[0083] Perform feature preprocessing on the frame, including normalization or standardization of X (such as Min-Max, Z-score); positional encoding can be added for Transformer network;

[0084] Input to the time series modeling network, using bidirectional LSTM;

[0085] The context representation h for each frame t Perform classification or sequence decoding;

[0086] Can be connected to CTC (Connected Temporal Classification) or Seq2Seq decoding module;

[0087] The final output is a Gloss sequence (intermediate sign language expression) or directly text output;

[0088] At the same time, facial expression features are integrated to improve semantic discrimination.

[0089] Step 4, natural language understanding and semantic optimization: uses pre-trained language models (such as BERT and RoBERTa) to perform grammatical correction, word order adjustment, and sentence reorganization on the recognized sign language text; supports contextual linkage, which can optimize the current semantic output based on the user's historical expressions; adds keyword extraction and intent recognition modules to facilitate subsequent language mapping; because the output of the sign language recognition model is a "literal translation" text, it often has problems such as improper word order, incomplete grammar, and ambiguous word meanings, which affect the accuracy of multilingual translation and speech synthesis. Therefore, this step introduces a pre-trained language model (such as BERT and RoBERTa) to perform natural language optimization on the initial text.

[0090] The multilingual translation module in step 5: uses a multilingual neural translation model (such as mBART, mT5, or Transformer-Base) to convert Chinese sign language text into the target language (such as English or Spanish). A lightweight translation model (such as Distilled mBART) can be executed locally on the terminal to reduce latency. It supports two-way translation (text to sign language) and allows users to set the target language.

[0091] The text-to-speech synthesis output module in step 6: performs speech synthesis based on the FastSpeech2+HiFi-GAN architecture to generate natural intonation; the model can switch voice packages to adapt to different genders, accents and languages; supports real-time voice stream output or text + voice dual output mode; supports voice dialogue return, and realizes voice recognition + virtual sign language display module for hearing sign language users; the system can be deployed on multiple platforms such as mobile phone apps, web pages, and smart screens.

[0092] It collects sign language videos through a camera, uses a convolutional neural network to extract the coordinates of key hand points, and converts them into time series data. It uses a bidirectional LSTM or Transformer model for dynamic action recognition and semantic analysis, and combines pre-trained language models to optimize grammatical and semantic structures. The recognition results are then translated into text in multiple languages ​​through a multilingual neural translation model. Finally, deep speech synthesis technologies such as FastSpeech2 and HiFi-GAN are used to convert the text into natural speech, realizing intelligent conversion from sign language to multilingual text and speech, with the advantages of high precision, low latency and good cross-language adaptability.

[0093] The method for converting speech and text to sign language is as follows: 1. Voice input or text input; 2. Text semantic analysis (natural language understanding NLU); 3. Sign language semantic mapping (Sign Gloss Mapping); 4. Gesture action sequence generation (Pose Sequence Generation); 5. Gesture rendering and output; Through speech recognition or text input, the system first extracts language semantics and maps them to sign language semantic units (Gloss), then generates continuous gesture action sequences through deep learning models or standard libraries, and finally outputs sign language through virtual signers or robots, achieving high-fidelity conversion from "auditory language" to "visual language".

[0094] In step 1, voice input or text input is used. If it is voice, a speech recognition engine (ASR, Automatic Speech Recognition) is used to convert the voice into text. Available models include: Wav2Vec2.0, DeepSpeech, Google ASR API, etc.

[0095] Step 2: Text semantic analysis (Natural Language Understanding (NLU)) to extract intent, entities, and semantic units, simplifying them into content that can be expressed in sign language; word segmentation + part-of-speech tagging (such as using BERT-CRF); entity recognition (NER); syntax tree parsing; and intent classification.

[0096] Step 3: Sign Gloss Mapping: Map the text in ordinary Chinese or other languages ​​to sign language semantic symbols (Gloss, sign language symbol words); establish a Gloss mapping vocabulary (Sign Language Glossary) using rule mapping, statistical model or Transformer-based encoder-decoder model

[0097] Step 4: Pose Sequence Generation: Deep learning-based action generation. Use a Seq2Seq model (e.g., Transformer / LSTM) to directly convert the gloss into a 3D hand keypoint sequence output format (hand joints per frame).

[0098] Step 5: Gesture rendering and output, Method 1: 3D hand animation, use a 3D rendering engine (such as Unity / Unreal / Blender) to drive a virtual sign language model and convert action key points into skeletal animation.

[0099] The working principle of this invention: It uses hand key point detection to recognize complex, continuous, dynamic sign language. It supports a variety of hand shapes and body combinations to adapt to different regional or national sign language standards. It reduces rigid "word-by-word translation" and outputs text expressions that are more in line with natural language habits. It can not only output Chinese, but also achieve simultaneous translation in languages ​​such as English and Spanish through a multilingual translation model.

[0100] The application scenarios of sign language to multilingual text-to-speech conversion include: public services, education, healthcare, justice, social interaction, and smart hardware. It is particularly important in promoting two-way communication between the deaf and hearing people. In government halls, hospitals, banks, courts and other window units, by deploying this system on touch-screen all-in-one machines or voice interactive terminals, hearing users can input voice or text, and the system will convert it into visual standard sign language animation or bionic robot sign language in real time, achieving barrier-free communication. In special education and rehabilitation teaching, the system can be used as an auxiliary teaching tool to support teachers in translating course content into sign language to improve learning efficiency. In personal terminal devices (such as smartphones and AR glasses), the system can be used as a "sign language translation assistant" plug-in to achieve an application experience that can be carried with you and used at any time. At the same time, it supports multilingual conversion between Chinese and English and other languages ​​in a multilingual environment, with good cross-national and cross-cultural scalability.

[0101] See Figure 3In this specific embodiment, the specific application scenario is the intelligent sign language assisted interactive terminal in the government service hall, which is installed in the self-service machine or manual window in the government service hall; connected to a high-definition camera, microphone and screen; and the system background is connected to the model and engine of the present invention.

[0102] Hearing-impaired citizens stand in front of the terminal and sign language to the camera; the system captures the gestures in real time and converts them into standard Chinese voice broadcast; the window staff responds after hearing the voice; the system then converts the voice content into corresponding sign language animation video feedback to the user, achieving barrier-free communication throughout the process without the need for a human sign language interpreter.

[0103] After adopting the above technical solution, the beneficial effects of the present invention are: it can automatically convert sign language into voice and text, helping the deaf and dumb to communicate efficiently with the general public; it can be widely used in hospitals, schools, government service windows, public transportation and other scenarios. The system can run on mobile phones, tablets, PCs or edge devices (such as Jetson); it provides channels for independent expression and understanding for people with disabilities, and promotes information equality; it is in line with the national development direction of intelligent assistive technology and has a good social influence. The present invention builds an efficient, natural and scalable cross-language sign language translation system through the integration of deep learning sign language recognition, multilingual understanding and speech synthesis technology, which greatly improves the human-computer interaction experience and barrier-free communication capabilities, and has significant social value and industrial potential.

[0104] The above description is only used to illustrate the technical solution of the present invention and is not intended to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A deep learning-based sign language to multilingual text-to-speech conversion method, characterized by: The method for converting sign language to speech text is as follows:

1. Image acquisition and preprocessing module, 2. Hand key point detection module, 3. Time series modeling and sign language recognition, 4. Natural language understanding and semantic optimization, 5. Multilingual translation module, 6. Text-to-speech synthesis output module.

2. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 1, characterized in that: The image acquisition and preprocessing module in step 1 described above uses a camera, mobile device, or smart wearable device to capture user sign language video in real time; the video frame rate is recommended to be set at 25 to 30 fps to ensure the integrity of the movement; the image is preprocessed by denoising, illumination normalization, background segmentation, etc. to ensure the accuracy of subsequent recognition.

3. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 2, characterized in that: In step 1, the image denoising uses Gaussian filtering to perform local weighted averaging on the image. The weights are determined by a two-dimensional Gaussian distribution. This can smooth the image, remove high-frequency noise, and maintain a good edge state for the image. The specific algorithm is: Among them, σ is the standard deviation, which determines the degree of blur.

4. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 2, characterized in that: In step 1, illumination normalization requires that the image brightness be standardized to enhance the robustness of key point recognition in the presence of different illumination conditions in the acquisition environment. Gamma correction is used, specifically: I′(x,y)=A·I(x,y) γ in: γ<1: Enhance dark details; γ>1: compress highlights and suppress overexposure; A: scaling factor, usually 1; Normalize the image brightness range to make the model more robust to different lighting conditions.

5. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 1, characterized in that: The hand key point detection module in step 2 uses MediaPipe Hands or a self-developed high-precision regression network (such as one based on the BlazePose+Hourglass structure) to extract 21 hand key points from each frame. The output is an array of 3D spatial coordinates (x, y, z) of the hand in each frame, and a timestamp is recorded. The data of both hands is fused and modeled, retaining the motion trajectory and symmetry information. This step realizes the "digital modeling" of sign language movements, providing a basis for subsequent time sequence recognition.

6. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 5, characterized in that: The position coordinates (x, y) of 21 hand key points are extracted from each frame of the image to describe the structure and dynamic behavior of the gesture. Common key points are as follows: 0: wrist; 1-4: thumb; 5-8: index finger; 9-12: middle finger; 13-16: ring finger; 17~20: little finger; The specific algorithm is: 2.

1. Feature Extraction: Use a CNN network (such as ResNet or MobileNet) to extract image features: F=CNN(I) Output dimensions are typically: 2.

2. Heatmap Generation Use 1×1 convolution to map the feature map into K heat maps, each heat map represents the location distribution of a key point: H k =Conv 1×1 (F), k = 1, 2, ..., K. In each heatmap, the ideal distribution of key points is a two-dimensional Gaussian: 2.

3. Coordinate Regression Extract the maximum response point from each heat map as the keypoint coordinates: A more refined approach is to use a weighted average method (soft-argmax) to improve sub-pixel accuracy: 2.

4. Timing smoothing and filtering: Use Kalman filtering or exponential sliding average to smooth out key point jitter between multiple frames and improve timing stability: Where α∈[0,1] is the smoothing coefficient, and the recommended value is about 0.7; 2.

5. Loss function (used during training), The training goal is to minimize the difference (mean squared error) between the predicted heatmap and the true heatmap: 2.

6. Output structure and obtain the two-dimensional coordinates of 21 key points: The confidence value of each key point (obtained from the maximum value of the heat map): This information will be fed into the next stage: gesture timing modeling (such as LSTM / Transformer) for dynamic sign language recognition.

7. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 1, characterized in that: Step 3 of the above-mentioned temporal modeling and sign language recognition involves inputting the frame-level key point sequence into a temporal modeling network, such as Bi-LSTM, Transformer, or TCN. The model is trained using a CTC (Connectionist Temporal Classification) loss or an attention mechanism to achieve a "gesture sequence → sign language semantic unit" mapping. Both continuous sign language recognition and phrase recognition modes are supported to enhance semantic integrity.

8. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 7, characterized in that: Step 3 is to input the time series features composed of the hand key points (or whole body key points) extracted from each frame of the video into a temporal modeling network to learn its contextual dependencies, thereby recognizing the complete sign language semantics or text representation. The specific process is as follows: Assume that the original input is a sign language video, extract T frames in total, and extract N key point coordinates (usually 2D or 3D coordinates) from each frame: Keypoint vector for each frame: Or add confidence to become The overall sequence input is: Where D = 2N or 3N. Perform feature preprocessing on the frame, including normalization or standardization of X (such as Min-Max, Z-score); positional encoding can be added for Transformer network; Input to the time series modeling network, using bidirectional LSTM; The context representation h for each frame t Perform classification or sequence decoding; Can be connected to CTC (Connected Temporal Classification) or Seq2Seq decoding module; The final output is a Gloss sequence (intermediate sign language expression) or directly text output; At the same time, facial expression features are integrated to improve semantic discrimination.

9. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 1, characterized in that: Step 4 of the natural language understanding and semantic optimization process involves using a pre-trained language model (e.g., BERT or RoBERTa) to perform grammatical correction, word order adjustment, and sentence reorganization on the recognized sign language text. It supports contextual linkage and can optimize the current semantic output based on the user's historical expressions; it adds keyword extraction and intent recognition modules to facilitate subsequent language mapping; since the output of the sign language recognition model is a "literal translation" text, there are often problems such as improper word order, incomplete grammar, and ambiguous word meanings, which affect the accuracy of multilingual translation and speech synthesis.

10. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 1, characterized in that: The multilingual translation module in step 5: uses a multilingual neural translation model (such as mBART, mT5, or Transformer-Base) to convert Chinese sign language text into the target language (such as English or Spanish); can execute a lightweight translation model (such as Distilled mBART) locally on the terminal to reduce latency; supports two-way translation (text to sign language) and supports users to set the target language.

11. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 1, characterized in that: The text-to-speech synthesis output module in step 6: performs speech synthesis based on the FastSpeech2+HiFi-GAN architecture to generate natural intonation; the model can switch voice packages to adapt to different genders, accents, and languages; supports real-time voice stream output or text + voice dual output mode; supports voice dialogue return, and realizes voice recognition + virtual sign language display module for hearing sign language users.

12. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 1, characterized in that: The method for converting speech and text to sign language is as follows:

1. Voice input or text input; 2. Text semantic analysis; 3. Sign language semantic mapping; 4. Gesture action sequence generation; 5. Gesture rendering and output; Through speech recognition or text input, the system first extracts language semantics and maps them to sign language semantic units (Gloss), then generates continuous gesture action sequences through deep learning models or standard libraries, and finally outputs sign language through virtual signers or robots, achieving high-fidelity conversion from "auditory language" to "visual language".

13. The deep learning-based sign language to multilingual text-to-speech conversion method according to claim 12, characterized in that: In step 1, voice input or text input is used. If it is voice, a speech recognition engine (ASR, Automatic Speech Recognition) is used to convert the voice into text. Available models include: Wav2Vec2.0, DeepSpeech, Google ASRAPI, etc.

14. The method for converting sign language to multilingual text and speech based on deep learning according to claim 12, characterized in that Step 2 involves text semantic analysis (natural language understanding (NLU)) to extract intent, entities, and semantic units, simplifying them into content that can be expressed in sign language; word segmentation + part-of-speech tagging (such as using BERT-CRF); entity recognition (NER); syntax tree parsing; and intent classification.

15. The method for converting sign language to multilingual text and speech based on deep learning according to claim 12, characterized in that Step 3: Sign Gloss Mapping, mapping ordinary Chinese or other language text to sign language semantic symbols (Gloss, sign language symbol words); establishing a Gloss mapping vocabulary (Sign Language Glossary) using rule mapping, statistical model or Transformer-based encoder-decoder model.

16. The method for converting sign language to multilingual text and speech based on deep learning according to claim 12, characterized in that Step 4: Pose Sequence Generation; deep learning-based action generation; using a Seq2Seq model (such as Transformer / LSTM) to directly convert gloss into a 3D hand key point sequence output format (hand joint points per frame).

17. The method for converting sign language to multilingual text and speech based on deep learning according to claim 12, characterized in that The step 5: gesture rendering and output, method 1: 3D hand animation, using a 3D rendering engine (such as Unity / Unreal / Blender) to drive a virtual sign language model, and converting action key points into skeletal animation.

18. Application of the deep learning-based sign language to multilingual text-to-speech conversion method, characterized by Application scenarios for sign language to multilingual text-to-speech conversion include public services, education, healthcare, justice, social interaction, and smart hardware, and are particularly important in promoting two-way communication between the deaf and hearing. In government offices, hospitals, banks, courts, and other window units, by deploying this system on touch-screen all-in-one machines or voice interaction terminals, hearing users can input voice or text, and the system will convert it into visual standard sign language animation or bionic robot sign language in real time, achieving barrier-free communication. In special education and rehabilitation teaching, the system can be used as an auxiliary teaching tool, supporting teachers to translate course content into sign language and improve learning efficiency. In personal terminal devices (such as smartphones and AR glasses), the system can be used as a "sign language translation assistant" plug-in, realizing an application experience of carrying it with you and allowing conversations at any time. At the same time, it supports multilingual conversion between Chinese and English, and has good cross-national and cross-cultural scalability.

19. The application of the deep learning-based sign language to multilingual text-to-speech conversion method according to claim 18, characterized in that: Specific application scenario: The intelligent sign language assisted interactive terminal in the government service hall is installed at the self-service machine or manual window in the government service hall; it is connected to a high-definition camera, microphone and screen; the model and engine of the present invention are connected to the system background; hearing-impaired citizens stand in front of the terminal and sign language to the camera; the system captures the gestures in real time and converts them into standard Chinese voice broadcast; the window staff responds after hearing the voice; the system then converts the voice content into corresponding sign language animation video feedback to the user.

Images