Cross-modal sign language recognition and real-time translation method

Abstract

The invention discloses a cross-modal sign language recognition and real-time translation method, and the method comprises the steps: fusing the multi-level sign language features of a hand key point, a hand shape, a motion track, a facial expression and a body posture, and combining context semantic understanding and grammatical structure analysis; high-precision continuous sign language recognition and bidirectional translation are realized through sign language grammar structure analysis, a context semantic inference mechanism, continuous sign language segmentation and recognition, a sign language dialect knowledge base and a virtual sign language generator, so that the communication efficiency between hearing impaired people and healthy hearing people is improved. According to the method, isolated sign language vocabularies can be recognized, grammar structures and contexts of sign languages can be understood, bidirectional translation of sign language-text / voice and text / voice-virtual sign language animations is achieved, and recognition and conversion of sign language dialects in different regions are supported.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, computer vision, and natural language processing, specifically to a cross-modal sign language recognition and real-time translation method, applicable to various application scenarios such as accessible public services, online education, video conferencing, emergency assistance, film and television subtitles, and sign language news broadcasting. Background Technology

[0002] With the rapid development of artificial intelligence technology, sign language recognition and translation, as an important bridge for communication between hearing-impaired and hearing individuals, has shown broad application prospects in multiple fields. Traditional sign language recognition methods mainly rely on single-modal information, such as gesture recognition based solely on key hand points, static gesture classification based solely on hand shape, or dynamic gesture recognition based solely on motion trajectory. However, sign language expressions have complex characteristics that are multi-dimensional and multi-layered, and single-modal information often cannot comprehensively and accurately capture the complete semantics of sign language.

[0003] Sign language, as a complete visual language system, encompasses not only hand gestures but also facial expressions, body postures, and spatial relationships. Facial expressions in sign language carry important tone and emotional information, such as grammatical functions like questioning, emphasis, and negation; body postures and spatial relationships are used to express grammatical structures and semantic relationships. Traditional sign language recognition systems suffer from the following problems:

[0004] 1. Insufficient feature extraction: Existing methods mainly focus on key points of the hand or hand shape information, ignoring important auxiliary information such as facial expressions and body posture. This results in low accuracy in recognizing grammatical structures such as interrogative sentences and negative sentences, and makes it impossible to understand the complete semantics of sign language.

[0005] 2. Difficulty in segmenting continuous sign language: Sign language expressions are continuous, unlike spoken language which has clear lexical boundaries. Most existing systems require manual segmentation or rely on simple time-series windows, and cannot automatically identify lexical boundaries in continuous sign language streams, thus limiting the system's practicality and real-time performance.

[0006] 3. Lack of contextual semantic understanding: Sign language contains many ambiguous gestures, the accurate meaning of which needs to be determined by combining the context and grammatical structure. Existing models lack dedicated grammatical parsing modules and contextual modeling mechanisms, making them unable to effectively handle ambiguous gestures and leading to recognition errors.

[0007] 4. Insufficient cross-dialect recognition capability: Sign languages ​​differ significantly between different countries and regions, such as Chinese Sign Language, American Sign Language, and British Sign Language. Existing systems are typically trained for a single sign language dialect and cannot recognize and convert sign language dialects from different regions, thus limiting the applicability of the system.

[0008] 5. Limited two-way translation capability: Most sign language recognition systems can only achieve one-way translation from sign language to text, and cannot achieve reverse translation from text / speech to sign language. Even if there is a reverse translation function, the generated virtual sign language animation is often not natural and smooth, and lacks coordination of facial expressions and body postures.

[0009] 6. Difficulty in meeting real-time requirements: In real-time interactive scenarios such as video conferencing and emergency assistance, the system needs to quickly and accurately recognize sign language and respond accordingly. However, existing deep learning models have high computational complexity and large inference latency, making it difficult to meet real-time requirements.

[0010] 7. High data annotation costs: Sign language recognition requires a large amount of labeled data, and sign language annotation requires professional sign language experts, resulting in high annotation costs and long cycles. Existing methods typically require a large amount of labeled data to achieve good performance, leading to high training costs.

[0011] Therefore, there is an urgent need for a sign language recognition and translation system that can fully utilize multimodal information, deeply integrate multi-level sign language features, accurately understand sign language grammatical structure and contextual semantics, and possess good real-time performance and cross-dialect adaptability. This invention addresses the aforementioned technical challenges by designing a multi-level feature fusion mechanism, a contextual semantic modeling module, a continuous sign language segmentation algorithm, and a virtual sign language generation model, providing an efficient and reliable technical solution for sign language recognition and translation. Summary of the Invention

[0012] To address the aforementioned issues, this invention provides a cross-modal sign language recognition and real-time translation method. By fusing multi-level sign language features such as hand key points, hand shapes, movement trajectories, facial expressions, and body postures, it achieves high-precision continuous sign language recognition and bidirectional translation. The system can recognize isolated sign language words, understand the grammatical structure and context of sign language, and realize bidirectional translation from sign language to text / speech and from text / speech to virtual sign language animation. It also supports the recognition and conversion of sign language dialects from different regions.

[0013] The specific plan is as follows:

[0014] A cross-modal sign language recognition and real-time translation method integrates multi-level sign language features, including key hand points, hand shapes, movement trajectories, facial expressions, and body postures. Combined with contextual semantic understanding and grammatical structure analysis, and employing sign language grammatical structure parsing, contextual semantic inference mechanisms, continuous sign language segmentation and recognition, a sign language dialect knowledge base, and a virtual sign language generator, it achieves high-precision continuous sign language recognition and bidirectional translation, thereby improving communication efficiency between hearing-impaired and hearing individuals. The method specifically includes the following steps:

[0015] S1. Multi-level sign language feature extraction: Design specialized feature extractors for three information sources: hand movements, facial expressions, and body postures, and construct multi-level sign language feature representations;

[0016] S11. Extraction of key points and hand shape features of the hand

[0017] For hand movements, a depth camera or binocular camera is used to acquire the 3D coordinate information of 21 key points of the hand; let the input hand key point sequence be... ,in Indicates the first 3D coordinates of 21 key points in the frame. For the number of frames;

[0018] First, calculate the relative positional features of key hand points, and then normalize them using the wrist as a reference point:

[0019]

[0020] in For the first The coordinates of key points on the wrist are determined; to capture the spatial structure of the hand, a graph convolutional network is used to model the hand skeleton structure; 21 key points are constructed into a graph structure. ,in The set of nodes (21 key points). It is a set of edges (edges connecting adjacent key points);

[0021] A spatiotemporal graph convolutional network (ST-GCN) is used to extract hand structural features. ST-GCN captures the spatial relationships between key points of the hand through graph convolution operations and captures dynamic changes in the time dimension through temporal convolution. In specific implementation, firstly, spatial graph convolution is performed on each key point and its neighboring nodes to extract local spatial features; then, one-dimensional convolution is performed on the temporal dimension to capture motion patterns; finally, higher-level semantic features are extracted step by step through multiple stacked spatiotemporal graph convolutional layers.

[0022]

[0023] in For graph convolution feature dimensions, For the first The structural features of the hand in the frame;

[0024] Simultaneously, hand shapes are classified and recognized. Key hand points are input into the hand shape classifier to identify hand shape categories (such as clenched fist, open hand, pointing hand, heart shape, etc.). The hand shape classifier is implemented using a multi-layer fully connected network: first, the 3D coordinates of 21 key points are flattened into 63-dimensional vectors, then feature transformation is performed through multiple fully connected layers, and finally, the probability distribution of various hand shapes is output through a Softmax layer.

[0025]

[0026] in For the number of hand type categories, Probability distribution for hand shape classification;

[0027] S12. Hand movement trajectory feature extraction

[0028] To capture the dynamic characteristics of hand movements, the motion trajectory of key hand points is calculated; a motion velocity vector is defined:

[0029]

[0030] in The time interval is used to more comprehensively capture motion characteristics and simultaneously calculate the motion direction angle to describe the directional changes of hand movements.

[0031]

[0032] The velocity vector and direction angle are combined to form the motion feature vector. A bidirectional long short-term memory network is used to extract the temporal features of the motion trajectory:

[0033]

[0034] in For the motion feature dimension, the BiLSTM network processes the sequence through forward and backward LSTMs respectively, and finally concatenates the hidden states in the two directions to obtain the final temporal feature representation.

[0035] S13. Facial Expression Feature Extraction

[0036] For facial expressions, a Facial Action Unit (AU) recognition system is used to extract expression features; let the input facial image sequence be... ,in Indicates the first Frame facial images;

[0037] First, facial landmark detectors are used to locate facial landmarks, and then Active User Area (AU) features are extracted. The AU extractor is implemented using a convolutional neural network: first, a pre-trained CNN is used to extract feature representations of the facial image, and then a fully connected layer is used to regress the intensity value of each AU unit. Each AU unit corresponds to a specific facial muscle action (e.g., AU1 corresponds to the inner corner of the eyebrow raising, AU12 corresponds to the corner of the mouth raising), and the regression network outputs an intensity value of 0-5.

[0038]

[0039] in For AU feature dimensions, Including the intensity values ​​of the action units such as raising eyebrows, lowering the corners of the mouth, and widening the eyes;

[0040] To capture temporal changes in facial expressions, a temporal attention mechanism is employed:

[0041]

[0042] in The weight matrix is ​​a learnable matrix. For attention weight vectors, The weighted facial expression features;

[0043] S14. Body posture feature extraction

[0044] For body posture, extract the upper body tilt angle and head movements; let the body key point sequence be... ,in Indicates the first Frames Key points of the body;

[0045] Calculate the upper body tilt angle:

[0046]

[0047] The head motion encoder is implemented using an LSTM network: the coordinate sequence of key points on the head is input into the LSTM, and the temporal patterns of head movements are captured through a memory mechanism; the hidden states of the LSTM encode the motion features of the head in the time dimension, including nodding, shaking, left and right turning, and other motion patterns.

[0048]

[0049] in This represents the dimension of head movement features.

[0050] S2. Multi-level feature fusion: After completing the feature extraction of the three levels of hand, face and body posture, in order to make full use of the complementarity of features at different levels, a spatiotemporal attention mechanism is designed to fuse multi-level features to obtain multi-level fused features.

[0051] Receive extraction , , , , and Features are used to achieve deep fusion through projection and attention mechanisms;

[0052] First, the features at different levels are projected onto a unified feature space:

[0053]

[0054] in , , Let be the projection matrix. To unify the dimension of the feature space, This indicates vector concatenation;

[0055] A multi-head cross-attention mechanism is employed to compute the correlation between features at different levels. This mechanism maps input features to three matrices: query, key, and value. Attention weights are obtained by calculating the similarity between the query and the key, and then the values ​​are summed using weighted averages. The feature space is divided into multiple subspaces, each of which independently computes attention. Finally, the outputs of all heads are concatenated. Specifically, hand features are used as the query, and facial and body features are used as keys. The degree of attention given by hand features to facial and body features is calculated to capture the semantic correlation between features at different levels.

[0056]

[0057] The calculation for each attention head is as follows: , For the number of attention heads, For each head dimension, For the first The projection matrix of the head, The output projection matrix is ​​simplified as follows:

[0058]

[0059] in , , , The learnable parameter matrix;

[0060] The final multi-level fusion features are obtained through weighted summation:

[0061]

[0062] in For learnable fusion weight coefficients, satisfying , This is the final multi-level fusion feature.

[0063] S3, Contextual semantic modeling and syntax parsing; obtaining multi-level fusion features Subsequently, in order to understand the grammatical structure and contextual semantics of sign language, a sign language grammatical structure parsing and contextual semantic inference mechanism was designed to obtain a semantic feature sequence containing grammatical and contextual information;

[0064] The output fused feature sequence As input, through syntax parsing and context inference, the output is a semantic representation containing both syntactic and contextual information. This provides more accurate semantic features for subsequent sign language recognition;

[0065] S31. Analysis of Sign Language Grammar Structure

[0066] Sign language has a unique grammatical structure, including subject-verb-object order, tense expression, interrogative sentences, and negative sentences. This invention constructs a sign language grammar knowledge base and defines a set of grammatical rules. ;

[0067] For the input sign language feature sequence A bidirectional LSTM network is used for grammar parsing. The parser first encodes the sign language feature sequence using a bidirectional LSTM to capture contextual information. Then, it focuses on relevant rules in the grammar rule base through an attention mechanism. Finally, it outputs grammatical structure features through a fully connected layer, including the grammatical role of the current gesture (subject, predicate, object, etc.), tense information (past, present, future), and tone information (interrogative, negative, emphasis, etc.).

[0068]

[0069] in For grammatical features, It contains grammatical structure information for the current moment (such as subject-verb-object roles, tense, mood, etc.);

[0070] S32, Contextual Semantic Inference

[0071] To handle ambiguous gestures, a contextual attention mechanism is used to infer the accurate meaning; the size of the context window is defined as follows: Calculate the semantic similarity between the current gesture and the gestures in the context:

[0072]

[0073] Context-weighted features:

[0074]

[0075] in This is the distance attenuation coefficient. , For attenuation parameters;

[0076] The final semantic representation combines syntactic and contextual information:

[0077]

[0078] in For semantic fusion matrix, This refers to the semantic feature dimension.

[0079] S4. Continuous sign language segmentation and recognition; obtaining semantic feature sequences containing grammatical and contextual information. Subsequently, in order to achieve automatic segmentation and recognition of continuous sign language, connectionist temporal classification (CTC) loss or attention mechanism is used to achieve end-to-end recognition;

[0080] The semantic feature sequence is encoded into a hidden representation, then mapped to the vocabulary space, and finally the text recognition result is output through CTC decoding or sequence-to-sequence translation.

[0081] S41, Continuous recognition based on CTC

[0082] The semantic feature sequence is input into a bidirectional LSTM encoder; the bidirectional LSTM encoder consists of multiple layers of bidirectional LSTM, with the forward LSTM and backward LSTM processing the contextual information of the sequence respectively, and finally the forward and backward hidden states are concatenated to obtain an encoded representation containing complete contextual information:

[0083]

[0084] in and These are the hidden states of the forward and backward LSTMs, respectively. The hidden layer dimension of the bidirectional LSTM encoder;

[0085] By mapping fully connected layers to the vocabulary space, the encoded features are mapped to the vocabulary probability distribution:

[0086]

[0087] in For the size of the sign language vocabulary list, For the first The word probability distribution of the frame and The weight matrix and bias vector of the output layer;

[0088] The CTC loss function is used for training to achieve end-to-end recognition without manual segmentation. The CTC algorithm addresses the issue of inconsistent input and output sequence lengths by introducing blank labels, allowing the model to output sequences of arbitrary lengths. It also uses dynamic programming to calculate the probability of all possible alignment paths. The CTC loss function is defined as follows:

[0089]

[0090] in The actual label sequence, Output sequence for the model, To align the path, The set of all alignment paths that can be mapped to the actual label sequence. For path In the A label for a moment;

[0091] S42, Sequence Translation Based on Attention Mechanism

[0092] As an alternative to CTC, a Transformer architecture is used to implement sequence-to-sequence translation. The Transformer decoder consists of multiple stacked decoder layers, each containing a self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network. The self-attention mechanism allows the decoder to focus on different positions in the generated sequence, while the encoder-decoder attention mechanism allows the decoder to focus on relevant parts of the input semantic feature sequence. In the specific implementation, the generated prefix sequence is first embedded and positionally encoded, and then processed through multiple decoder layers, each containing residual connections and layer normalization.

[0093]

[0094] in For the output sequence, This is the already generated prefix sequence.

[0095] The vocabulary prediction probability is calculated using the Softmax function:

[0096]

[0097] in To output the projection matrix, the decoder hidden state is mapped to the vocabulary space.

[0098] S5. Sign Language Dialect Knowledge Base and Cross-Dialect Translation: After completing sign language recognition, a sign language dialect knowledge base is constructed to support the recognition and conversion of sign language dialects in different regions.

[0099] Multi-level fusion features of received output It identifies dialect types through a dialect encoder and uses a cross-dialect mapping matrix to achieve feature conversion between different dialects, thereby supporting cross-dialect sign language recognition and translation;

[0100] S51, Dialect Feature Coding

[0101] For sign language in different dialects, a dialect encoder is used to extract features. The dialect encoder is implemented using a fully connected network, with an independent encoder network designed for each dialect. The dialect encoder receives multi-level fused features and maps general features to dialect-specific feature representations through multiple fully connected layers and non-linear activation functions, capturing the unique expression patterns of that dialect.

[0102]

[0103] in Indicates the first Different dialects (e.g., CSL for Chinese Sign Language, ASL for American Sign Language). As a characteristic specific to dialects, and dialect encoder parameters;

[0104] S52, Cross-dialect mapping

[0105] Design a cross-dialect mapping matrix to map features from one dialect to another dialect space:

[0106]

[0107] in To use dialect To dialect The mapping matrix.

[0108] S6. Virtual Sign Language Generation: To achieve reverse translation from text / speech to sign language, a virtual sign language generation model is designed.

[0109] Together with the aforementioned sign language recognition, it forms a complete two-way translation system: the recognition module realizes the translation of sign language to text, and the generation module realizes the translation of text to sign language; the generation module receives text input, obtains semantic representation through text encoder, then generates gesture feature sequence, and finally converts it into 3D virtual sign language animation containing hand movements, facial expressions and body posture;

[0110] S61. Text-to-Gesture Sequence Generation

[0111] For input text First, semantic representations are obtained through a text encoder; the text encoder uses a pre-trained language model (such as BERT or GPT) to encode the text sequence into context-sensitive word vector representations:

[0112]

[0113] A sequence-to-sequence model is used to generate gesture sequences; the gesture decoder is implemented using a Transformer decoder architecture: the text encoder processes the text semantic sequence, and the gesture decoder autoregressively generates the gesture feature sequence; at each time step, the gesture decoder, based on the generated gesture sequence and the text representation from the text encoder, focuses on relevant parts of the text through an attention mechanism to generate the next gesture feature.

[0114]

[0115] in For text encoding sequence, where For the first The characteristics of a gesture For gesture features;

[0116] S62, 3D sign language animation generation

[0117] The gesture features are converted into a 3D hand keypoint sequence; the keypoint decoder is implemented using a multi-layer fully connected network, mapping the gesture features to the 3D coordinates of 21 keypoints; the keypoint decoder gradually expands the feature dimension through multiple fully connected layers, finally outputting a 21×3 coordinate matrix:

[0118]

[0119] in and For keypoint decoder parameters;

[0120] Simultaneously, corresponding facial expressions and body poses are generated; the facial expression decoder uses a fully connected network to map gesture features into AU feature vectors; the body pose decoder also uses a fully connected network to map gesture features into body tilt angles.

[0121]

[0122]

[0123] in For facial expression decoder parameters, These are the parameters for the body pose decoder;

[0124] Motion smoothing algorithms are used to ensure smooth animation:

[0125]

[0126] in For smoothing parameters.

[0127] The beneficial effects of this invention are as follows:

[0128] 1. By deeply integrating multi-level features such as key hand points, hand shape, movement trajectory, facial expression, and body posture, this invention improves the accuracy of word recognition by 20-25 percentage points and the accuracy of continuous sign language recognition by 18-22 percentage points compared with single-modal sign language recognition methods, and can more accurately understand the complete semantics and grammatical structure of sign language.

[0129] 2. This invention innovatively designs a grammar parsing module and a contextual semantic modeling mechanism, enabling it to understand the subject-verb-object order, tense expressions, interrogative sentences, negative sentences, and other grammatical structures of sign language, and to infer the accurate meaning of ambiguous gestures through context. This capability significantly improves the system's accuracy in understanding complex sign language expressions, a feature not found in existing sign language recognition systems.

[0130] 3. This invention employs a CTC loss or attention mechanism to achieve end-to-end continuous sign language recognition, automatically identifying word boundaries in continuous sign language streams without manual segmentation. This technology overcomes the limitations of traditional methods that require manual annotation of word boundaries, enabling the system to process natural and fluent sign language expressions, significantly improving the system's practicality and user experience.

[0131] 4. Supports the recognition and mutual conversion of multiple sign language dialects, including Chinese Sign Language, American Sign Language, and British Sign Language, and achieves automatic conversion between sign language dialects from different regions through a cross-dialect mapping matrix. This capability expands the system's applicability, enabling it to serve hearing-impaired individuals from diverse cultural backgrounds.

[0132] 5. It can not only translate sign language into text / speech, but also translate text / speech into virtual sign language animation. The generated virtual sign language animation includes coordinated hand movements, facial expressions, and body postures, which is smooth and natural, and can accurately convey semantic information, providing a complete solution for two-way communication between hearing and hearing-impaired people.

[0133] 6. This innovative approach incorporates facial expression AU features and body posture information into the sign language recognition system, fully utilizing the grammatical functions and semantic information (such as interrogation, emphasis, and negation) carried by this information in sign language. This design significantly improves the system's accuracy in recognizing grammatical structures such as interrogative and negative sentences.

[0134] 7. An adaptive multi-level feature fusion mechanism was designed. When the quality of features at a certain level is poor or missing (e.g., the hand is occluded, or insufficient lighting causes the facial image to be blurred), the system can automatically adjust the fusion weights and rely on information from other levels to maintain recognition accuracy. This mechanism improves the system's robustness in complex environments.

[0135] 8. Through optimization techniques such as model quantization, feature caching, parallel processing, and sliding window, the complete recognition process time is controlled within 500ms while maintaining high accuracy, and the virtual sign language generation time is controlled within 300ms. This meets the response speed requirements of real-time interactive scenarios and supports large-scale commercial deployment.

[0136] 9. This invention can be applied to various fields such as accessible public services, online education, video conferencing, emergency assistance, film and television subtitles, sign language news broadcasting, intelligent assistants, and sign language dictionaries, and has broad market application prospects and significant social value. Especially in scenarios requiring real-time bidirectional translation and cross-dialect support, this invention possesses irreplaceable technical advantages.

[0137] 10. The technical framework of this invention has good scalability, can easily integrate more modal information (such as eye tracking, physiological signals, etc.), can be extended to more sign language dialects, and can be adapted to computing resources of different scales (from edge devices to cloud servers), providing a solid foundation for subsequent technology iteration and functional enhancement. Attached Figure Description

[0138] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0139] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0140] As shown in the figure, this invention provides a cross-modal sign language recognition and real-time translation method, taking the sign language translation service in a public service hall as an application scenario. The specific solution is as follows:

[0141] S1, System Architecture and Data Acquisition

[0142] A sign language recognition and translation system is constructed, comprising a data acquisition module, a multi-level feature extraction module, a semantic understanding module, a sign language recognition module, a translation output module, and a virtual sign language generation module.

[0143] Data Acquisition Configuration: Depth Camera: Intel RealSense D435 depth camera, 640×480 pixels resolution, depth accuracy ±1mm, 30fps frame rate, used to acquire 3D coordinates of key hand points; High-Definition Camera: 1080p webcam, 30fps frame rate, used to capture facial expressions and body posture; Audio Acquisition: Voice input is acquired using a directional microphone (for reverse translation), sampling rate 16kHz, 16-bit quantization.

[0144] The system processes input data in real time, recognizing and translating each sign language segment (usually 2-5 seconds).

[0145] S2, Implementation of Multi-level Sign Language Feature Extraction

[0146] S21. Extraction of key points and hand shape features of the hand.

[0147] Hand detection and key point localization were performed on depth images captured by a depth camera, and 3D coordinate sequences of 21 hand key points were extracted. (3 seconds, 30fps), where .

[0148] Normalization is performed using the wrist as a reference point:

[0149]

[0150] A hand skeleton graph structure was constructed, and a spatiotemporal graph convolutional network (ST-GCN) was used to extract structural features. The number of graph convolutional layers was set to 3, with each layer having an output dimension of 64, resulting in a final feature dimension of [missing value]. :

[0151]

[0152] The hand shape classifier uses a 3-layer fully connected network. The input is the normalized keypoint coordinates, and the output is the probability distribution of 15 hand shape categories. ):

[0153]

[0154] S22, Hand movement trajectory feature extraction

[0155] Calculate the velocity vector of key points on the wrist:

[0156]

[0157] Simultaneously calculate the angle of the direction of motion:

[0158]

[0159] The velocity vector and direction angle are combined to form the motion feature vector. A bidirectional LSTM network (hidden layer dimension) is used. Extracting temporal features of motion trajectories:

[0160]

[0161] S23, Facial Expression Feature Extraction

[0162] Face detection and alignment are performed on facial image sequences captured by a high-definition camera, and facial key points are extracted. The OpenFace toolkit is used to extract the intensity values ​​of 17 AU units, and the feature dimensions are... :

[0163]

[0164] Facial expression information is aggregated using a temporal attention mechanism:

[0165]

[0166] Attention weights are calculated as follows:

[0167]

[0168] in , To obtain facial expression features .

[0169] S24. Body posture feature extraction

[0170] OpenPose was used to detect key body points and extract the upper body tilt angle.

[0171]

[0172] The head motion encoder uses an LSTM network (hidden layer dimension). Extracting head movement features:

[0173]

[0174] S3, Implementation of Multi-level Feature Fusion

[0175] After completing the aforementioned multi-level feature extraction, the extracted hand, facial, and body posture features are fused according to the aforementioned multi-level feature fusion design. This fusion module receives all features extracted from the aforementioned hand keypoint and hand shape feature extraction, hand motion trajectory feature extraction, facial expression feature extraction, and body posture feature extraction modules, and achieves deep fusion through projection and attention mechanisms.

[0176] Projecting features at different levels onto a unified 256-dimensional feature space ( The projection process is as follows:

[0177]

[0178] in , , Each parameter is consistent with the aforementioned multi-level feature fusion module.

[0179] An 8-head cross-attention mechanism is used to calculate intermodal associations:

[0180]

[0181] in , , , .

[0182] Final fusion features:

[0183]

[0184] In this embodiment, the fusion weight is set to It emphasizes the dominant role of hand movements while making full use of auxiliary information such as facial expressions and body posture.

[0185] S4. Implementation of Contextual Semantic Modeling

[0186] After obtaining the multi-level fused feature sequence output by the aforementioned multi-level feature fusion module, contextual semantic modeling and syntax parsing are performed according to the aforementioned design for contextual semantic modeling and syntax parsing. This module uses the fused feature sequence... As input, semantic features are output through syntax parsing and context inference. This provides a more accurate semantic representation for subsequent identification.

[0187] S41. Syntax Structure Analysis

[0188] Building a sign language grammar knowledge base containing 50 grammar rules This includes subject-verb-object order, tense expression, interrogative sentence markers, and negative sentence markers.

[0189] For fused feature sequences A bidirectional LSTM is used for syntax parsing.

[0190]

[0191] Among them, the syntactic feature dimension This is consistent with the aforementioned sign language grammar structure parsing module.

[0192] S42, Contextual Semantic Inference

[0193] Set the context window size Calculate the semantic similarity between the current gesture and the gestures in the context:

[0194]

[0195] Context-weighted features are calculated as follows:

[0196]

[0197] Where the distance attenuation coefficient Attenuation parameter This is consistent with the aforementioned context semantic inference module.

[0198] The final semantic representation fusion is as follows:

[0199]

[0200] in semantic feature dimension All parameters are completely consistent with those of the aforementioned context semantic inference module.

[0201] S5. Implementation of continuous sign language recognition

[0202] After obtaining the semantic feature sequence output by the aforementioned contextual semantic modeling module, end-to-end recognition is achieved using CTC loss, following the design of continuous recognition based on CTC. This module encodes the semantic feature sequence into a hidden representation, then maps it to the vocabulary space, and finally outputs the final text recognition result through CTC decoding.

[0203] Input the semantic feature sequence into a bidirectional LSTM encoder (hidden layer dimension) ):

[0204]

[0205] in and These are the hidden states of the forward and backward LSTMs, respectively.

[0206] Mapped to the vocabulary space via a fully connected layer:

[0207]

[0208] Vocabulary size (Includes commonly used sign language vocabulary) and The weight matrix and bias vector of the output layer are consistent with those of the aforementioned CTC-based continuous recognition module.

[0209] Sequence alignment and recognition are performed using a CTC decoder, following the aforementioned CTC loss function. The program is trained and outputs a text sequence. For example, given a 3-second sign language video, the recognition result would be "Hello, I would like to conduct some business".

[0210] S6. Implementation of Sign Language Dialect Recognition and Conversion

[0211] After completing sign language recognition, to support cross-dialect applications, sign language dialect recognition and conversion are performed according to the aforementioned design of sign language dialect knowledge base and cross-dialect translation. This module receives the multi-level fused features output by the aforementioned multi-level feature fusion module. It identifies dialect types through a dialect-specific encoder and uses a cross-dialect mapping matrix to achieve feature conversion between different dialects.

[0212] Construct a knowledge base that includes three dialects: Chinese Sign Language (CSL), American Sign Language (ASL), and British Sign Language (BSL).

[0213] For input sign language, the dialect type is first identified. A dialect classifier is used:

[0214]

[0215] Assuming the language is identified as CSL, to convert it to ASL, cross-dialect mapping is used:

[0216]

[0217] in , Mapping matrix It is trained on paired multi-dialect data through contrastive learning, and is consistent with the aforementioned cross-dialect mapping module.

[0218] S7. Implementation of Virtual Sign Language Generation

[0219] After completing sign language recognition, virtual sign language generation is performed according to the aforementioned design for virtual sign language generation to achieve reverse translation. This module receives text input, obtains semantic representation through a text encoder, generates a sequence of gesture features, and finally converts it into a 3D virtual sign language animation.

[0220] For the text input "Hello, how can I help you?", the semantic representation is first obtained through a text encoder. The BERT-based Chinese model is used as the text encoder, and the output dimension is... :

[0221]

[0222] in This is consistent with the aforementioned text-to-gesture sequence generation module.

[0223] A sequence-to-sequence model is used to generate a sequence of gestures. Assume 10 gestures are generated:

[0224]

[0225] Among them, gesture feature dimension This is consistent with the aforementioned text-to-gesture sequence generation module.

[0226] Convert gesture features into 3D hand key points:

[0227]

[0228] Simultaneously generate corresponding facial expressions and body postures:

[0229]

[0230] Motion smoothing algorithms are used to ensure smooth animation:

[0231]

[0232] The smoothing parameter is set to It is consistent with the aforementioned 3D sign language animation generation module.

[0233] Render and generate 3D virtual sign language animation, which is then displayed on the screen.

[0234] In summary, this invention innovatively encodes 3D coordinates of key hand points, hand shape classification, motion trajectory, facial expression AU features, and body posture in a unified manner. It extracts features at each level through a spatiotemporal graph convolutional network, bidirectional LSTM, and temporal attention mechanism, and achieves deep fusion using a multi-head cross-attention mechanism. This mechanism fully leverages the complementarity of features at different levels, significantly improving the accuracy and robustness of sign language recognition.

[0235] This invention designs a dedicated sign language grammar parsing module, constructs a sign language grammar knowledge base, and understands the subject-verb-object order, tense expressions, interrogative sentences, negative sentences, and other grammatical structures of sign language. It employs a contextual attention mechanism to calculate the semantic similarity between the current gesture and historical gestures, and infers the accurate meaning of ambiguous gestures through weighted fusion. This mechanism effectively solves the ambiguity problem in sign language recognition and improves the accuracy of semantic understanding.

[0236] This invention employs a CTC loss or attention mechanism to achieve end-to-end continuous sign language recognition, automatically identifying word boundaries in continuous sign language streams without manual segmentation. Similar to continuous speech recognition technology, this method can handle natural and fluent sign language expressions, significantly improving the system's practicality and user experience.

[0237] This invention constructs a knowledge base containing multiple sign language dialects, designs dialect-specific encoders and cross-dialect mapping matrices, and realizes the recognition and mutual conversion of sign language dialects from different regions. By training cross-dialect mapping relationships through comparative learning, the system can adapt to sign language expressions from different cultural backgrounds, thus expanding the system's applicability.

[0238] This invention, based on text or voice input, uses a sequence-to-sequence model to generate gesture feature sequences, and collaboratively generates 3D virtual sign language animations through a keypoint decoder, facial expression decoder, and body posture decoder. A motion smoothing algorithm is employed to ensure the smoothness and naturalness of the animation, achieving high-quality sign language reverse translation.

[0239] The above embodiments are merely typical illustrative methods of the present invention, and the scope of protection of the present invention is not limited thereto. All equivalent substitutions and improvements made under the concept of the present invention should fall within the scope of protection. It should be emphasized that any modifications or minor adjustments made by those skilled in the art without departing from the basic principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for cross-modal sign language recognition and real-time translation, characterized in that, By integrating multi-layered sign language features such as key hand points, hand shapes, movement trajectories, facial expressions, and body postures, and combining contextual semantic understanding and grammatical structure analysis, this system achieves high-precision continuous sign language recognition and bidirectional translation through sign language grammatical structure parsing, contextual semantic inference mechanisms, continuous sign language segmentation and recognition, a sign language dialect knowledge base, and a virtual sign language generator. This improves communication efficiency between hearing-impaired and hearing individuals. Specifically, the system includes the following steps: S1. Multi-level sign language feature extraction: Design specialized feature extractors for three information sources: hand movements, facial expressions, and body postures, and construct multi-level sign language feature representations; S2. Multi-level feature fusion: After completing the feature extraction of the three levels of hand, face and body posture, in order to make full use of the complementarity of features at different levels, a spatiotemporal attention mechanism is designed to fuse multi-level features to obtain multi-level fused features. S3. Contextual Semantic Modeling and Syntax Parsing: After obtaining multi-level fusion features, in order to understand the grammatical structure and contextual semantics of sign language, a sign language grammatical structure parsing and contextual semantic inference mechanism is designed to obtain a semantic feature sequence containing grammatical and contextual information. S4. Continuous Sign Language Segmentation and Recognition: After obtaining the semantic feature sequence containing grammatical and contextual information, in order to achieve automatic segmentation and recognition of continuous sign language, connection-temporal classification (CTC) loss or attention mechanism is adopted to achieve end-to-end recognition. S5. Sign Language Dialect Knowledge Base and Cross-Dialect Translation: After completing sign language recognition, a sign language dialect knowledge base is constructed to support the recognition and conversion of sign language dialects in different regions. S6. Virtual Sign Language Generation: To achieve reverse translation from text / speech to sign language, a virtual sign language generation model is designed.

2. The cross-modal sign language recognition and real-time translation method according to claim 1, characterized in that, Step S1 specifically includes: S11. Extraction of key points and hand shape features of the hand For hand movements, a depth camera or binocular camera is used to acquire the 3D coordinate information of 21 key points of the hand; let the input hand key point sequence be... ,in Indicates the first 3D coordinates of 21 key points in the frame. For the number of frames; First, calculate the relative positional features of key hand points, and then normalize them using the wrist as a reference point: in For the first The coordinates of key points on the wrist are determined; to capture the spatial structure of the hand, a graph convolutional network is used to model the hand skeleton structure; 21 key points are constructed into a graph structure. ,in For a set of nodes, Let it be the set of edges; A spatiotemporal graph convolutional network (ST-GCN) is used to extract hand structural features. ST-GCN captures the spatial relationships between key points of the hand through graph convolution operations and captures dynamic changes in the time dimension through temporal convolution. In specific implementation, firstly, spatial graph convolution is performed on each key point and its neighboring nodes to extract local spatial features; then, one-dimensional convolution is performed on the temporal dimension to capture motion patterns; finally, higher-level semantic features are extracted step by step through multiple stacked spatiotemporal graph convolutional layers. in For graph convolution feature dimensions, For the first The structural features of the hand in the frame; Simultaneously, hand shapes are classified and recognized. Key hand points are input into a hand shape classifier to identify hand shape categories. The hand shape classifier is implemented using a multi-layer fully connected network: first, the 3D coordinates of 21 key points are flattened into 63-dimensional vectors; then, feature transformation is performed through multiple fully connected layers; finally, the probability distribution of various hand shapes is output through a Softmax layer. in For the number of hand type categories, Probability distribution for hand shape classification; S12. Hand movement trajectory feature extraction To capture the dynamic characteristics of hand movements, the motion trajectory of key hand points is calculated; a motion velocity vector is defined: in The time interval is used to more comprehensively capture motion characteristics and simultaneously calculate the motion direction angle to describe the directional changes of hand movements. The velocity vector and direction angle are combined to form the motion feature vector. A bidirectional long short-term memory network is used to extract the temporal features of the motion trajectory: in For the motion feature dimension, the BiLSTM network processes the sequence through forward and backward LSTMs respectively, and finally concatenates the hidden states in the two directions to obtain the final temporal feature representation. S13. Facial Expression Feature Extraction For facial expressions, the facial action coding unit (AU) recognition system is used to extract expression features; let the input facial image sequence be... ,in Indicates the first Frame facial images; First, facial landmark detectors are used to locate facial landmarks, and then Active Feature (AU) features are extracted. The AU extractor is implemented using a convolutional neural network: first, a pre-trained CNN is used to extract feature representations of the facial image, and then a fully connected layer is used to regress the intensity value of each AU unit. Each AU unit corresponds to a specific facial muscle movement, and the regression network outputs an intensity value of 0-5. in For AU feature dimensions, Including the intensity values ​​of the action units such as raising eyebrows, lowering the corners of the mouth, and widening the eyes; To capture temporal changes in facial expressions, a temporal attention mechanism is employed: in The weight matrix is ​​a learnable matrix. For attention weight vectors, The weighted facial expression features; S14. Body posture feature extraction For body posture, extract the upper body tilt angle and head movements; let the body key point sequence be... ,in Indicates the first Frames Key points of the body; Calculate the upper body tilt angle: The head motion encoder is implemented using an LSTM network: the coordinate sequence of head key points is input into the LSTM, and the temporal patterns of head movements are captured through a memory mechanism; the hidden states of the LSTM encode the motion features of the head in the time dimension, including nodding, shaking, and left and right turning motion patterns. in This represents the dimension of head movement features.

3. The cross-modal sign language recognition and real-time translation method according to claim 2, characterized in that, In step S2, the extracted data is received. , , , , and Features are used to achieve deep fusion through projection and attention mechanisms; First, the features at different levels are projected onto a unified feature space: in , , Let be the projection matrix. To unify the dimension of the feature space, This indicates vector concatenation; A multi-head cross-attention mechanism is employed to compute the correlation between features at different levels. This mechanism maps input features to three matrices: query, key, and value. Attention weights are obtained by calculating the similarity between the query and the key, and then the values ​​are summed using weighted averages. The feature space is divided into multiple subspaces, each of which independently computes attention. Finally, the outputs of all heads are concatenated. Specifically, hand features are used as the query, and face and body features are used as keys. The degree of attention given by hand features to face and body features is calculated to capture the semantic correlation between features at different levels. The calculation for each attention head is as follows: , For the number of attention heads, For each head dimension, For the first The projection matrix of the head, The output projection matrix is ​​simplified as follows: in , , , The learnable parameter matrix; The final multi-level fusion features are obtained through weighted summation: in For learnable fusion weight coefficients, satisfying , This is the final multi-level fusion feature.

4. The cross-modal sign language recognition and real-time translation method according to claim 3, characterized in that, In step S3, the output fused feature sequence As input, through syntax parsing and context inference, the output is a semantic representation containing both syntactic and contextual information. This provides more accurate semantic features for subsequent sign language recognition; specifically including: S31. Analysis of Sign Language Grammar Structure Construct a sign language grammar knowledge base and define a set of grammar rules. ; For the input sign language feature sequence A bidirectional LSTM network is used for grammar parsing. The parser first encodes the sign language feature sequence using a bidirectional LSTM to capture contextual information. Then, it focuses on relevant rules in the grammar rule base through an attention mechanism. Finally, it outputs grammatical structure features through a fully connected layer, including the grammatical role, tense information, and tone information of the current gesture. in For syntactic feature dimension, It contains grammatical structure information for the current moment (such as subject-verb-object roles, tense, mood, etc.); S32, Contextual Semantic Inference To handle ambiguous gestures, a contextual attention mechanism is used to infer the accurate meaning; the size of the context window is defined as follows: Calculate the semantic similarity between the current gesture and the gestures in the context: Context-weighted features: in This is the distance attenuation coefficient. , For attenuation parameters; The final semantic representation combines syntactic and contextual information: in For semantic fusion matrix, This refers to the semantic feature dimension.

5. The cross-modal sign language recognition and real-time translation method according to claim 4, characterized in that, In step S4, the semantic feature sequence is encoded into a hidden representation, then mapped to the vocabulary space, and finally the text recognition result is output through CTC decoding or sequence-to-sequence translation; specifically including: S41, Continuous Recognition Based on CTC The semantic feature sequence is input into a bidirectional LSTM encoder; the bidirectional LSTM encoder consists of multiple layers of bidirectional LSTM, with the forward LSTM and backward LSTM processing the contextual information of the sequence respectively, and finally the forward and backward hidden states are concatenated to obtain an encoded representation containing complete contextual information: in and These are the hidden states of the forward and backward LSTMs, respectively. The hidden layer dimension of the bidirectional LSTM encoder; By mapping fully connected layers to the vocabulary space, the encoded features are mapped to the vocabulary probability distribution: in For the size of the sign language vocabulary list, For the first The word probability distribution of the frame and The weight matrix and bias vector of the output layer; The CTC loss function is used for training to achieve end-to-end recognition without manual segmentation. The CTC algorithm addresses the issue of inconsistent input and output sequence lengths by introducing blank labels, allowing the model to output sequences of arbitrary lengths. It also uses dynamic programming to calculate the probability of all possible alignment paths. The CTC loss function is defined as: in The actual label sequence, Output sequence for the model, To align the path, The set of all alignment paths that can be mapped to the actual label sequence. For path In the A label for a moment; S42, Sequence Translation Based on Attention Mechanism As an alternative to CTC, the Transformer architecture is used to implement sequence-to-sequence translation. The Transformer decoder consists of multiple stacked decoder layers, each containing a self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network. In the specific implementation, the generated prefix sequence is first embedded and positionally encoded, and then processed through multiple decoder layers, each containing residual connections and layer normalization. in For the output sequence, The generated prefix sequence; The vocabulary prediction probability is calculated using the Softmax function: in This is for outputting the projection matrix.

6. The cross-modal sign language recognition and real-time translation method according to claim 5, characterized in that, In step S5, the multi-level fusion features of the received output are... This system identifies dialect types through a dialect encoder and utilizes a cross-dialect mapping matrix to achieve feature transformation between different dialects, thereby supporting cross-dialect sign language recognition and translation; specifically including: S51, Dialect Feature Coding For sign language in different dialects, a dialect encoder is used to extract features. The dialect encoder is implemented using a fully connected network, with an independent encoder network designed for each dialect. The dialect encoder receives multi-level fused features and maps general features to dialect-specific feature representations through multiple fully connected layers and non-linear activation functions, capturing the unique expression patterns of that dialect. in Indicates the first dialects As a characteristic specific to dialects, and dialect encoder parameters; S52, Cross-dialect mapping Design a cross-dialect mapping matrix to map features from one dialect to another dialect space: in To use dialect To dialect The mapping matrix.

7. The cross-modal sign language recognition and real-time translation method according to claim 6, characterized in that, In step S6, a complete bidirectional translation system is formed with the aforementioned sign language recognition: the recognition module translates sign language into text, and the generation module translates text into sign language; the generation module receives text input, obtains semantic representation through a text encoder, then generates a gesture feature sequence, and finally converts it into a 3D virtual sign language animation containing hand movements, facial expressions, and body postures; specifically including: S61. Text-to-Gesture Sequence Generation For input text First, semantic representations are obtained through a text encoder; the text encoder uses a pre-trained language model to encode the text sequence into context-sensitive word vector representations: A sequence-to-sequence model is used to generate gesture sequences; the gesture decoder is implemented using a Transformer decoder architecture: the text encoder processes the text semantic sequence, and the gesture decoder autoregressively generates the gesture feature sequence; at each time step, the gesture decoder, based on the generated gesture sequence and the text representation from the text encoder, focuses on relevant parts of the text through an attention mechanism to generate the next gesture feature. in For text encoding sequence, where For the first The characteristics of a gesture For gesture features; S62, 3D sign language animation generation The gesture features are converted into a 3D hand keypoint sequence; the keypoint decoder is implemented using a multi-layer fully connected network, mapping the gesture features to the 3D coordinates of 21 keypoints; the keypoint decoder gradually expands the feature dimension through multiple fully connected layers, finally outputting a 21×3 coordinate matrix: in and For keypoint decoder parameters; Simultaneously, corresponding facial expressions and body poses are generated; the facial expression decoder uses a fully connected network to map gesture features into AU feature vectors; the body pose decoder also uses a fully connected network to map gesture features into body tilt angles. in For facial expression decoder parameters, These are the parameters for the body pose decoder; Motion smoothing algorithms are used to ensure smooth animation: in For smoothing parameters.

Images