A lightweight real-time sign language recognition and translation method and system for municipal window scenarios

Abstract

The application discloses a kind of municipal window scene-oriented lightweight real-time sign language recognition and translation method and system, it is related to artificial intelligence and computer vision technical field.The application includes: the three-dimensional coordinates of 53 key points are extracted by human posture estimation model, and after normalizing with the midpoint of double shoulder as origin, key points are divided into left hand group, right hand group and upper body group according to anatomical structure, respectively projected and fused into 256-dimensional embedding features;Time series modeling is carried out by learnable position encoding and 6-layer Transformer encoder, and sign language word class probability is output by mask mean pooling and classification head;Hand motion speed-based three-state finite state automaton is used to automatically detect sign language action boundary;The recognized vocabulary sequence is converted into natural Chinese sentence by multi-level priority rule template.The application realizes the automatic recognition and translation of sign language in a low-delay, high-precision, privacy-safe manner on a consumer-grade hardware platform.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and more specifically to a lightweight real-time sign language recognition and translation method and system for municipal window scenarios. Background Technology

[0002] Sign language, as a visual language using hand gestures, facial expressions, and body postures, possesses a complete grammatical system and rich expressive capabilities, serving as an important tool for hearing-impaired individuals to participate in social life and access public services. However, in public service settings such as government service windows, hearing-impaired individuals face significant communication barriers when handling government matters such as household registration, social security, medical insurance, housing provident fund inquiries, tax declarations, and marriage registration, because staff generally lack sign language communication skills.

[0003] Currently, the mainstream method for addressing communication issues among the hearing-impaired at government service windows still relies on human sign language interpreters. However, professional sign language interpreters are extremely scarce, far from meeting the enormous needs of the hearing-impaired in daily government affairs. Furthermore, human translation services suffer from high costs, long waiting times for appointments, and limited service coverage, preventing many grassroots government service windows from providing effective communication support to the hearing-impaired. While written communication is an alternative, it is limited by the literacy abilities of some hearing-impaired individuals, and is inefficient and prone to misunderstandings, failing to meet the complex communication needs of government affairs. Therefore, developing an automated sign language recognition and translation system has significant social and practical value.

[0004] Existing sign language recognition technologies mainly fall into the following categories, but all of them have varying degrees of technical shortcomings, making it difficult to meet the actual deployment needs of municipal service window scenarios: The first category is sign language recognition methods based on RGB video. These methods directly use the original RGB video frames captured by the camera as input, and extract the spatiotemporal features in the video through convolutional neural networks (CNN) or three-dimensional convolutional networks (3D-CNN) to classify and recognize sign language actions. The main drawbacks of this type of method are: (1) huge computational load. Processing high-resolution video frames requires a large amount of GPU computing power. The number of model parameters usually exceeds tens of millions or even hundreds of millions, making it difficult to achieve real-time inference on consumer-grade hardware; (2) serious privacy and security risks. The system needs to continuously store and process original image data containing biometric information such as user facial features and body appearance. In the sensitive scenario of government services involving the security of citizens' personal information, the storage and transmission of original video data increases the risk of privacy leakage; (3) highly sensitive to lighting conditions. The indoor lighting conditions of different government service windows vary significantly. Changes in natural light, fluorescent light flickering, backlighting, and other factors can all lead to a significant decrease in recognition accuracy; (4) lack of robustness to changes in the background environment. In the government service window scenario, there is frequent personnel flow and complex and changeable background. Methods based on original pixels are easily affected by background interference.

[0005] The second category is sign language recognition methods based on depth cameras. This type of method uses RGB-D depth cameras (such as Microsoft Kinect, Intel RealSense, etc.) to simultaneously acquire color images and depth information, and constructs a three-dimensional human skeleton model through the depth map to achieve the capture and recognition of sign language movements. Although this type of method improves the recognition accuracy and anti-background interference ability to a certain extent, it has the following prominent defects: (1) The equipment cost is high. The unit price of a dedicated depth camera is usually over several thousand yuan. Large-scale deployment to tens of thousands of grassroots government service windows across the country will generate huge hardware procurement costs; (2) Deployment and installation are inconvenient. Depth cameras have strict requirements on installation location, angle, working distance, etc., and require dedicated equipment brackets and cables, which increases the transformation cost and construction difficulty of existing government service windows; (3) The effective working distance of depth cameras is limited, usually within the range of 0.5 meters to 4.5 meters. The accuracy of depth data drops sharply beyond this range, and the actual physical space layout of government service windows and the standing distance of users are often difficult to control precisely.

[0006] The third category is sign language recognition methods based on skeletal keypoints. These methods first extract the coordinates of human skeletal keypoints from ordinary RGB videos using human pose estimation models (such as MediaPipe, OpenPose, etc.), and then use the keypoint sequence as input for sign language recognition. Compared with directly processing the original video pixels, keypoint representation has advantages such as low data dimensionality, strong resistance to background interference, and privacy friendliness. However, existing skeletal keypoint-based sign language recognition schemes still have the following shortcomings: (1) Existing schemes generally use large-scale Transformer models or graph convolutional networks (GCN), with the number of parameters usually in the tens of millions, and the inference latency in the hundreds of milliseconds or even seconds, which cannot meet the low latency requirements of face-to-face real-time interaction in municipal window scenarios; (2) Most existing schemes are designed for continuous sign language recognition tasks, and are not well adapted to the communication mode of municipal windows, which is mainly composed of isolated words and phrases; (3) There is a lack of effective automatic sign language action segmentation mechanism, which cannot accurately determine the start and end boundaries of sign language actions in continuous video streams, and requires manual button triggering or fixed-duration interception, which reduces the natural interactive experience of the system.

[0007] The fourth category is the limitations of existing sign language translation solutions. Converting the recognized sign language gloss sequence into fluent natural language sentences is a key step in sign language translation systems. Existing solutions mainly use end-to-end neural machine translation models (such as Seq2Seq, Transformer translation models, etc.), but such methods have the following problems: (1) The training data requirements are huge, requiring a large number of paired sign language-text parallel corpora, while high-quality parallel corpora in the sign language field are extremely scarce, especially professional corpora for specific fields (such as government services) are severely lacking; (2) The translation results of neural translation models lack interpretability and controllability. In government services, a scenario with extremely high accuracy requirements, unpredictable translation errors may lead to serious business processing errors; (3) Model updates and maintenance are difficult. When it is necessary to add or adjust specific words and expressions, it is necessary to collect data again and retrain the model, resulting in slow response speed and high maintenance costs.

[0008] In summary, existing technical solutions have varying degrees of shortcomings in terms of lightweight design, real-time performance, privacy protection, ease of deployment, and translation flexibility, making it difficult to fully meet the practical application needs of real-time sign language recognition and translation in municipal service window scenarios.

[0009] Therefore, proposing a lightweight real-time sign language recognition and translation method and system for municipal window scenarios to solve the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0010] In view of this, the present invention provides a lightweight real-time sign language recognition and translation method and system for municipal window scenarios, which solves the problems of existing sign language recognition technology in municipal window scenarios, such as high inference latency due to large model parameters, privacy risks caused by raw video data processing, lack of automated sign language action boundary detection mechanism, and insufficient flexibility of translation scheme. Thus, it realizes automatic sign language recognition and translation on consumer-grade hardware platforms in a low-latency, high-precision, and privacy-secure manner.

[0011] To achieve the above objectives, the present invention provides the following technical solution: A lightweight real-time sign language recognition and translation method for municipal service window scenarios includes the following steps: S1. Video frame acquisition and preprocessing: The video frame sequence is acquired by an RGB camera at a preset frame rate, and the video frames are preprocessed by color space conversion and size normalization to obtain preprocessed video frames. S2. Human Key Point Extraction: The human pose estimation model is used to detect human key points in the preprocessed video frames and extract the three-dimensional coordinates of all key points. S3. Key point normalization: Using the midpoint of the user's shoulders as the origin and the Euclidean distance between the shoulders as the unit length, the coordinates of all key points are normalized by translation and scaling. S4. Grouped Keypoint Embedding: According to the human anatomical structure, all keypoints are divided into three groups: left hand keypoints, right hand keypoints, and upper body keypoints. The three groups of keypoints are mapped into three embedding vectors through independent linear projection layers. The three embedding vectors are concatenated and processed by a fusion linear layer, layer normalization, and Dropout layer to obtain a unified embedding feature. S5. Position Encoding and Temporal Modeling: Learnable position encoding is added to the embedded features to inject frame sequence position information. Then, a multi-layer Transformer encoder is used to perform self-attention modeling on the temporal feature sequence to capture the dependencies of sign language actions in the temporal dimension. S6. Masked Mean Pooling and Classification: Based on the validity mask of each frame, the temporal features output by the Transformer encoder are subjected to masked mean pooling, and the variable-length temporal features are aggregated into a global representation vector of fixed dimensions. The probability distribution of each sign language word class is output through the classification head. S7. Sign Language Segmentation State Machine: Based on the displacement velocity of the left-hand group key points and the right-hand group key points between adjacent frames, a three-state finite state automaton including idle state, sign language performance state and hold state is designed to automatically detect the start and end boundaries of sign language actions in continuous video streams, and trigger model inference of S4-S6 when the sign language action is detected to end, so as to realize the automatic segmentation of sign language actions. S8. Template Matching Translation: The identified sign language word sequence is converted into natural Chinese sentences through a multi-level priority rule template engine; S9. Speech Synthesis Output: The translated Chinese sentences are converted into speech signals through a speech synthesis engine and played through a speaker, realizing a complete translation chain from sign language expression to speech output for hearing-impaired people.

[0012] Optionally, S2 extracts the 3D coordinates of 53 key points, including 21 key points on the left hand (1 on the left wrist, 4 on the left thumb, 4 on the left index finger, 4 on the left middle finger, 4 on the left ring finger, and 4 on the left little finger); 21 key points on the right hand (1 on the right wrist, 4 on the right thumb, 4 on the right index finger, 4 on the right middle finger, 4 on the right ring finger, and 4 on the right little finger); and 11 key points on the upper body (nose tip, left inner corner of eye, right inner corner of eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist). Each key point contains three coordinate components: x, y, and z, for a total of 53 × 3 = 159 dimensional feature vectors.

[0013] Optionally, in S3, the specific details of performing translation and scaling normalization on the coordinates of all key points, using the midpoint between the user's shoulders as the origin and the Euclidean distance between the shoulders as the unit length, are as follows: Let the coordinates of the key point on the left shoulder be... The coordinates of the key point on the right shoulder are The origin of the calculation coordinate system is the midpoint between the shoulders. ; Calculate the scaling factor as the Euclidean distance between the shoulders. When d is less than the threshold At that time, set d=1.0 to avoid division by zero errors; perform the operation on each coordinate component of all 53 key points. Normalization operation.

[0014] Optionally, in S4, the three sets of keypoints are mapped to three sets of embedding vectors through independent linear projection layers. These three sets of embedding vectors are then concatenated and processed through a fusion linear layer, layer normalization, and a Dropout layer to obtain the unified embedding feature. The key points of the left-hand group are taken from the 0th to the 62nd dimension of the 159-dimensional vector, which is a total of 63 dimensions. They are then mapped to a 128-dimensional embedding through a linear layer Linear(63,128). The keypoints of the right-hand group are taken from the 63rd to the 125th dimension, a total of 63 dimensions, and mapped to a 128-dimensional embedding through a linear layer Linear(63,128); The key points of the upper body group are taken from the 126th to the 158th dimension, a total of 33 dimensions, and mapped to a 64-dimensional embedding through a linear layer Linear(33,64); After concatenating the three sets of embeddings along the feature dimension to obtain a 320-dimensional vector, the vector is processed by a linear fusion layer (Linear(320,256)), a LayerNorm(256) normalization layer, and a Dropout(p=0.1) random deactivation layer to output a 256-dimensional unified embedding feature.

[0015] Optionally, the learnable location encoding added to the embedded features in S5 is of dimension [dimensionality missing]. The learnable parameter matrix, where, To the maximum supported sequence length, The embedding dimension is used; the learnable parameter matrix is ​​initialized with a truncated normal distribution and a standard deviation of 0.02; during forward propagation, the first T rows of the parameter matrix are taken and added element-wise to the input embedding, where T is the actual length of the current input sequence.

[0016] Optionally, the multi-layer Transformer encoder in S5 can be configured as follows: embedding dimension =256, number of attention heads nhead=8, internal dimension of feedforward network dff=1024, number of encoder layers numlayers=6, dropout ratio is 0.1, activation function is GELU, pre-norm structure is used, that is, layer normalization is performed before self-attention and feedforward network, batch_first is set to True to make the input tensor shape (B,T,D).

[0017] Optionally, the specific details of masked mean pooling of the temporal features output by the Transformer encoder based on the validity mask of each frame in S6 are as follows: Construct a boolean-type effective mask based on the actual effective frame count (lengths) of each sample. The effective frame position is True, and the padding position is False; Expand to Shape and convert to floating-point type; multiply the Transformer encoder output element-wise with the expanded mask and sum along the time dimension to obtain... The shape features are summed; the mask is summed along the time dimension to obtain the effective frame count and the lower bound is set to 1.0 to avoid division by zero; the sum of features is divided by the effective frame count to obtain the mask mean pooling result.

[0018] Optionally, the classification head in S6 consists of a linear layer Linear(256,256), a GELU activation function, a Dropout (p=0.3) random deactivation layer, and a linear layer Linear(256,num_classes) connected in sequence, where num_classes is the number of sign language vocabulary categories.

[0019] Optionally, the state transition conditions for the three-state finite state automaton in S7, which includes an idle state, a sign language performance state, and a hold state, are as follows: The condition for transitioning from the idle state to the sign language execution state is: the average motion velocity of the left-hand keypoints and the right-hand keypoints in the current frame. ,in, The threshold for initiating movement; The conditions for transitioning from the sign language state (SIGNING) to the hold state (HOLD) are: the average movement speed of the key points in the left hand group and the key points in the right hand group. continued Frame, in which The threshold for determining stillness. The threshold for the number of consecutive still frames; The condition for transitioning from the HOLD state to the IDLE state is: automatic transition after completing one model inference recognition; The hand movement speed v is calculated as follows: take the Euclidean norm of the three-dimensional coordinate difference between the current frame and the previous frame for 21 left-hand key points and 21 right-hand key points, and take the average value of the 42 key points.

[0020] Optionally, the S7 also includes a cooldown period anti-repetition mechanism: after a sign language recognition is completed, a cooldown counter is started and set to a preset number of cooldown frames. During the cooling-off period, even if the boundary of sign language movement is detected, no new inference recognition is triggered. The cooling counter is decremented by 1 for each frame of video processed until the counter reaches zero and then normal detection is restored. It also includes a sliding window frame buffering mechanism: setting the maximum buffer capacity. Set minimum effective window size A circular queue is used to store video frames and their corresponding key point data. When the number of buffered frames exceeds... The earliest frame is automatically discarded; if the number of buffered frames is insufficient when the state machine triggers inference. Then skip this reasoning.

[0021] Optionally, the model inference phases S4-S6, triggered when the sign language action is detected to have ended in S7, employ an automatic mixed-precision inference mode on the GPU device, performing some computations in FP16 half-precision floating-point format.

[0022] Optionally, the neural network models used in S4-S6 are pre-trained models, and the training samples of the models are augmented by a combination of the following data augmentation methods: spatial jitter enhancement, temporal resampling enhancement, keypoint random discard enhancement, and horizontal mirror enhancement. Horizontal mirror enhancement is achieved by swapping the left-hand keypoint group with the right-hand keypoint group and inverting all x-coordinates.

[0023] Optionally, S8 performs matching according to the priority order of direct word mapping, verb-object structure template, interrogative sentence template, state description template, material list template, and catch-all concatenation, converting sign language vocabulary into fluent and natural Chinese sentences; among them, The first priority is direct word mapping; when the recognition result is a single gloss word, the corresponding Chinese expression is directly returned from the table. The second priority is verb-object structure template matching; the gloss sequence is matched with predefined verb-object structure patterns containing placeholders. The third priority is interrogative sentence template matching. The fourth priority is state description template matching. The fifth priority is bill of materials template matching. The sixth priority is fallback concatenation; when none of the above templates match, the default format is used to output a single gloss, and multiple glosses are connected by commas before output.

[0024] A lightweight real-time sign language recognition and translation system for municipal service window scenarios, applying any of the above-mentioned lightweight real-time sign language recognition and translation methods for municipal service window scenarios, includes a video acquisition module, a key point extraction module, a preprocessing module, a grouping embedding module, a temporal modeling module, a classification and recognition module, a real-time reasoning module, a translation module, a speech synthesis module, and a user interaction module connected in sequence. The video capture module is used to capture video frame sequences through an RGB camera; The key point extraction module is used to extract the three-dimensional coordinates of 53 human key points from video frames and encode them into 159-dimensional feature vectors. The preprocessing module is used to normalize the coordinates of key points with the midpoint of the shoulders as the origin and the distance between the shoulders as the unit length. The grouping and embedding module is used to divide key points into left-hand key points, right-hand key points and upper-body key points according to anatomical structure, and then project and stitch them together to form a unified dimension of embedding features. The temporal modeling module includes a learnable positional encoder and a multi-layer Transformer encoder for temporal self-attention modeling of embedded feature sequences; The classification and recognition module includes a masked mean pooling layer and a classification head, which are used to output the probability distribution of sign language word classes; The real-time inference module includes a three-state finite state automaton and a sliding window frame buffer, which are used to automatically detect the boundaries of sign language actions and trigger model inference. The translation module includes a multi-level priority rule template engine for converting sign language word sequences into natural Chinese sentences; The speech synthesis module is used to convert Chinese sentences into speech signals for output. The user interaction module provides a graphical user interface for video preview, recognition result display, and dialogue recording.

[0025] Optionally, the rule templates for the translation module are stored and managed in YAML format configuration files, which support the dynamic addition and adjustment of translation rules without modifying the program code or retraining the model.

[0026] Optionally, the system adopts a front-end and back-end separated web application architecture. The back-end provides RESTful API and WebSocket real-time streaming communication interface. After the key points are extracted, the original video frame data is discarded, and subsequent processing is based solely on the key point coordinate data.

[0027] As can be seen from the above technical solution, compared with the prior art, the present invention provides a lightweight real-time sign language recognition and translation method and system for municipal window scenarios, which has the following beneficial effects: (1) Lightweight model design, suitable for edge deployment: This invention controls the total number of model parameters to about 5.5M (millions) through grouped key point embedding and compact Transformer encoder architecture, which is much lower than existing video-based sign language recognition models (usually 50M-200M parameters) and key point recognition models based on large-scale Transformer (usually 20M-80M parameters); the lightweight model design enables the system to achieve efficient inference in consumer-grade GPU (such as NVIDIA GTX 1650 and above) or even CPU environment, without the need for expensive server-grade GPU hardware support; the time for a single model forward inference is less than 30 milliseconds (measured on NVIDIA RTX 3060 GPU), which is significantly lower than the 100-500 milliseconds inference time usually required by existing solutions; the model file size is only about 22MB (FP32 format), which can be easily deployed to the existing computing devices of government service windows without the need to purchase additional high-performance computing hardware; (2) High recognition accuracy to meet the practical needs of government service scenarios: The Top-1 recognition accuracy of this invention on the isolated sign language word dataset exceeds 85%, and the Top-5 recognition accuracy exceeds 95%; for 200-500 government-related sign language words frequently used in municipal service windows (such as "ID card", "household registration book", "payment", "inquiry", "processing", "get number", etc.), through targeted training data collection and model fine-tuning, the recognition accuracy can be further improved to over 90%; the high-precision recognition capability ensures the reliability and practicality of the system in actual government service scenarios; (3) End-to-end real-time response and natural interactive experience: The end-to-end processing latency (from the completion of the sign language action to the output of Chinese text and speech) of this invention is controlled within 500 milliseconds, of which key point extraction is about 15 milliseconds / frame, model inference is about 30 milliseconds, template matching translation is less than 1 millisecond, and speech synthesis is about 200 milliseconds; with the help of a three-state finite state automaton based on motion speed, the system can automatically detect the start and end boundaries of the sign language action without the user having to manually trigger the recording, thus realizing a natural interactive experience similar to a face-to-face sign language interpreter; the cool-down anti-repetition mechanism effectively avoids false triggering between consecutive actions, improving the stability and smoothness of the system in actual use; (4) Strong privacy protection mechanism, suitable for sensitive government scenarios: The core technical route of this invention is based on the representation of skeletal key points. After the key points are extracted, the original RGB video frame data is discarded. All subsequent processing steps (normalization, embedding, inference, translation) are based only on the abstract coordinate data of 53 key points. The key point data does not contain personal privacy information such as the user's facial appearance, skin color, and clothing. This fundamentally eliminates the risk of video data leakage of personal privacy from the technical architecture level. (5) Convenient deployment and low modification cost: This invention only requires a regular USB / network camera to collect video input, without the need to purchase dedicated hardware equipment such as depth cameras and infrared sensors; the system can be installed and run on the computer terminals of existing government service windows, utilizing existing network infrastructure and display devices, with extremely low modification costs; the system supports the WebSocket real-time streaming communication protocol, which can be flexibly integrated with existing government service information systems; the modular system architecture design allows each functional component to be upgraded and maintained independently, reducing long-term operation and maintenance costs; (6) Highly adaptable to scenarios and broad vocabulary coverage: This invention has been specially optimized for municipal window government service scenarios: at the vocabulary level, it focuses on covering 200-500 high-frequency words in government scenarios; at the translation level, the predefined multi-level rule templates are specially designed for typical sentence structures in government scenarios (such as inquiry, processing, material submission, status query, etc.), and the translation results are accurate and natural; the template management mechanism based on YAML configuration files enables maintenance personnel to flexibly add and adjust translation rules without modifying the code or retraining the model, and quickly respond to business changes; (7) Improved data augmentation strategies enhance model generalization ability: During the training phase, this invention employs four data augmentation strategies: spatial jitter, temporal resampling, random key point discarding, and horizontal mirroring. These strategies effectively improve the robustness and generalization ability of the model to variations in actual scenarios such as different sign language expression speeds, key point detection noise, partial occlusion, and differences in left and right hand usage habits, thereby reducing the reliance on large-scale labeled data. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0029] Figure 1 A flowchart of a lightweight real-time sign language recognition and translation method for municipal window scenarios provided by the present invention; Figure 2 A schematic diagram of the structure of a lightweight real-time sign language recognition and translation system for municipal service window scenarios provided by the present invention; Figure 3 The network structure diagram for embedding key points in grouping provided by this invention; Figure 4 This is a diagram illustrating the overall architecture of the model provided by the present invention. Figure 5 The sign language segmentation state machine transition diagram provided by the present invention; Figure 6 The real-time inference flowchart provided for this invention; Figure 7 The template matching translation flowchart provided by this invention; Figure 8 A schematic diagram of the interactive interface layout for deploying government service windows provided by the present invention; Figure 9 This is a schematic diagram showing the distribution of 53 key points on the human body provided by the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] See Figure 1 As shown, this invention discloses a lightweight real-time sign language recognition and translation method for municipal service window scenarios, including the following steps: S1. Video frame acquisition and preprocessing: The video frame sequence is acquired by an RGB camera at a preset frame rate, and the video frames are preprocessed by color space conversion and size normalization to obtain preprocessed video frames. S2. Human Key Point Extraction: The human pose estimation model is used to detect human key points in the preprocessed video frames and extract the three-dimensional coordinates of all key points. S3. Key point normalization: Using the midpoint of the user's shoulders as the origin and the Euclidean distance between the shoulders as the unit length, the coordinates of all key points are normalized by translation and scaling. S4. Grouped Keypoint Embedding: According to the human anatomical structure, all keypoints are divided into three groups: left hand keypoints, right hand keypoints, and upper body keypoints. The three groups of keypoints are mapped into three embedding vectors through independent linear projection layers. The three embedding vectors are concatenated and processed by a fusion linear layer, layer normalization, and Dropout layer to obtain a unified embedding feature. S5. Position Encoding and Temporal Modeling: Learnable position encoding is added to the embedded features to inject frame sequence position information. Then, a multi-layer Transformer encoder is used to perform self-attention modeling on the temporal feature sequence to capture the dependencies of sign language actions in the temporal dimension. S6. Masked Mean Pooling and Classification: Based on the validity mask of each frame, the temporal features output by the Transformer encoder are subjected to masked mean pooling, and the variable-length temporal features are aggregated into a global representation vector of fixed dimensions. The probability distribution of each sign language word class is output through the classification head. S7. Sign Language Segmentation State Machine: Based on the displacement velocity of the left-hand group key points and the right-hand group key points between adjacent frames, a three-state finite state automaton including idle state, sign language performance state and hold state is designed to automatically detect the start and end boundaries of sign language actions in continuous video streams, and trigger model inference of S4-S6 when the sign language action is detected to end, so as to realize the automatic segmentation of sign language actions. S8. Template Matching Translation: The identified sign language word sequence is converted into natural Chinese sentences through a multi-level priority rule template engine; S9. Speech Synthesis Output: The translated Chinese sentences are converted into speech signals through a speech synthesis engine and played through a speaker, realizing a complete translation chain from sign language expression to speech output for hearing-impaired people.

[0032] Furthermore, S2 extracts the 3D coordinates of 53 key points. Among them, there are 21 key points on the left hand, including 1 on the left wrist, 4 on the left thumb, 4 on the left index finger, 4 on the left middle finger, 4 on the left ring finger, and 4 on the left little finger; 21 key points on the right hand, including 1 on the right wrist, 4 on the right thumb, 4 on the right index finger, 4 on the right middle finger, 4 on the right ring finger, and 4 on the right little finger; and 11 key points on the upper body, namely the tip of the nose, the inner corner of the left eye, the inner corner of the right eye, the left ear, the right ear, the left shoulder, the right shoulder, the left elbow, the right elbow, the left wrist, and the right wrist. Each key point contains three coordinate components: x, y, and z, for a total of 53 × 3 = 159 dimensional feature vectors.

[0033] Furthermore, in S3, the specific details of translating and scaling the coordinates of all key points using the midpoint between the user's shoulders as the origin and the Euclidean distance between the shoulders as the unit length are as follows: Let the coordinates of the key point on the left shoulder be... The coordinates of the key point on the right shoulder are The origin of the calculation coordinate system is the midpoint between the shoulders. ; Calculate the scaling factor as the Euclidean distance between the shoulders. When d is less than the threshold At that time, set d=1.0 to avoid division by zero errors; perform the operation on each coordinate component of all 53 key points. Normalization operation.

[0034] Furthermore, in S4, the three sets of keypoints are mapped to three sets of embedding vectors through independent linear projection layers. These three sets of embedding vectors are then concatenated and processed through a fusion linear layer, layer normalization, and Dropout layer to obtain the unified embedding feature. The key points of the left-hand group are taken from the 0th to the 62nd dimension of the 159-dimensional vector, which is a total of 63 dimensions. They are then mapped to a 128-dimensional embedding through a linear layer Linear(63,128). The keypoints of the right-hand group are taken from the 63rd to the 125th dimension, a total of 63 dimensions, and mapped to a 128-dimensional embedding through a linear layer Linear(63,128); The key points of the upper body group are taken from the 126th to the 158th dimension, a total of 33 dimensions, and mapped to a 64-dimensional embedding through a linear layer Linear(33,64); After concatenating the three sets of embeddings along the feature dimension to obtain a 320-dimensional vector, the vector is processed by a linear fusion layer (Linear(320,256)), a LayerNorm(256) normalization layer, and a Dropout(p=0.1) random deactivation layer to output a 256-dimensional unified embedding feature.

[0035] In a specific embodiment, the specific content of the grouped keypoint embedding is as follows: 1. Key point vector grouping Let the input keypoint tensor be... Where B is the batch size, T is the number of time steps (frames), and 159 = 53 × 3 is the keypoint feature dimension. The 159-dimensional vector is divided into three sub-vectors along the feature dimension according to anatomical structure: Left-handed subvectors: It includes the x, y, and z coordinates of 21 key points on the left; Right-handed subvector: It includes the x, y, and z coordinates of 21 key points on the right side; Upper body vector: It includes the x, y, and z coordinates of 11 key points on the upper body; 2. Grouped linear projection The three sets of subvectors are feature-mapped through independent linear projection layers: Left hand embedded: ,in , Output ; Right-hand embedding: ,in , Output ; Upper body embedded: ,in , Output ; The number of parameters for the three projection layers are as follows: left hand 63×128+128=8,192, right hand 63×128+128=8,192, upper body 33×64+64=2,176, totaling 18,560 parameters.

[0036] 3. Splicing and Fusion

[0037] Concatenate the three sets of embedding vectors along the feature dimension:

[0038] Where 320 = 128 + 128 + 64.

[0039] The concatenated vectors are then processed by a fusion network:

[0040] in , LayerNorm performs layer normalization on 256-dimensional features with a Dropout rate of 0.1.

[0041] The number of parameters in the fusion layer is: 320×256+256=82,176 (linear layer)+256×2=512 (γ and β parameters of LayerNorm), totaling 82,688 parameters.

[0042] 4. Summary of Dimensional Changes

[0043] The complete dimensional change process is as follows: (B,T,159)→Grouping→(B,T,63),(B,T,63),(B,T,33)→Projection→(B,T,128),(B,T,128),(B,T,64)→Stabbing→(B,T,320)→Blending→(B,T,256); The total number of parameters in the grouped keypoint embedding module is 18,560 + 82,688 = 101,248, approximately 0.1M parameters.

[0044] Furthermore, in S5, the learnable location encoding added to the embedded features is of dimension [dimensional value missing]. The learnable parameter matrix, where, To the maximum supported sequence length, The embedding dimension is used; the learnable parameter matrix is ​​initialized with a truncated normal distribution and a standard deviation of 0.02; during forward propagation, the first T rows of the parameter matrix are taken and added element-wise to the input embedding, where T is the actual length of the current input sequence.

[0045] Specifically, This is the embedding dimension, with a default value of 200.

[0046] Furthermore, the multi-layer Transformer encoder in S5 is configured as follows: embedding dimension =256, number of attention heads nhead=8, internal dimension of feedforward network dff=1024, number of encoder layers numlayers=6, dropout ratio is 0.1, activation function is GELU, pre-norm structure is used, that is, layer normalization is performed before self-attention and feedforward network, batch_first is set to True to make the input tensor shape (B,T,D).

[0047] Furthermore, the specific details of the masked mean pooling of the temporal features output by the Transformer encoder based on the validity mask of each frame in S6 are as follows: Construct a boolean-type effective mask based on the actual effective frame count (lengths) of each sample. The effective frame position is True, and the padding position is False; Expand to Shape and convert to floating-point type; multiply the Transformer encoder output element-wise with the expanded mask and sum along the time dimension to obtain... The shape features are summed; the mask is summed along the time dimension to obtain the effective frame count and the lower bound is set to 1.0 to avoid division by zero; the sum of features is divided by the effective frame count to obtain the mask mean pooling result.

[0048] Furthermore, the classification head in S6 consists of a linear layer Linear(256,256), a GELU activation function, a Dropout (p=0.3) random deactivation layer, and a linear layer Linear(256,num_classes) connected in sequence, where num_classes is the number of sign language vocabulary categories.

[0049] Furthermore, the state transition conditions of the three-state finite state automaton in S7, which includes an idle state, a sign language performance state, and a hold state, are as follows: The condition for transitioning from the idle state to the sign language execution state is: the average motion velocity of the left-hand keypoints and the right-hand keypoints in the current frame. ,in, The threshold for initiating movement; The conditions for transitioning from the sign language state (SIGNING) to the hold state (HOLD) are: the average movement speed of the key points in the left hand group and the key points in the right hand group. continued Frame, in which The threshold for determining stillness. The threshold for the number of consecutive still frames; The condition for transitioning from the HOLD state to the IDLE state is: automatic transition after completing one model inference recognition; The hand movement speed v is calculated as follows: take the Euclidean norm of the three-dimensional coordinate difference between the current frame and the previous frame for 21 left-hand key points and 21 right-hand key points, and take the average value of the 42 key points.

[0050] Furthermore, the S7 also includes a cooldown period anti-repetition mechanism: after a sign language recognition is completed, a cooldown counter is started and set to a preset number of cooldown frames. During the cooling-off period, even if the boundary of sign language movement is detected, no new inference recognition is triggered. The cooling counter is decremented by 1 for each frame of video processed until the counter reaches zero and then normal detection is restored. It also includes a sliding window frame buffering mechanism: setting the maximum buffer capacity. Set minimum effective window size A circular queue is used to store video frames and their corresponding key point data. When the number of buffered frames exceeds... The earliest frame is automatically discarded; if the number of buffered frames is insufficient when the state machine triggers inference. Then skip this reasoning.

[0051] Furthermore, in S7, the model inference phases S4-S6, triggered when the sign language action is detected to have ended, employ an automatic mixed-precision inference mode on the GPU device, performing some computations in FP16 half-precision floating-point format.

[0052] In a specific embodiment, the sign language segmentation state machine contains the following: 1. State Definition The state machine contains three states, represented by an enumeration type: IDLE (Idle State): The system is in a waiting state, no valid sign language movement is detected, the camera continues to capture video frames and calculate the hand movement speed, but does not trigger model inference.

[0053] SIGNING: The system detects that the user is performing sign language expression. Video frames are continuously stored in the frame buffer, and the system continuously monitors the speed of hand movements, waiting for the signal that the action has ended.

[0054] HOLD (Hold State): The system detects that the sign language action has ended (the hand stops moving), triggers the model to infer and recognize the sign language action in the buffer, and automatically transitions back to the IDLE state after the inference is completed.

[0055] 2. Calculation of hand movement speed

[0056] Let the 159-dimensional keypoint vector extracted from the current frame be... The first 126 dimensions of the hand key points are taken and reshaped into a (42,3) matrix representing the three-dimensional coordinates of the 42 hand key points:

[0057] Calculate the point-by-point Euclidean displacement of the hand keypoints relative to the previous frame:

[0058] The hand movement speed is the average of the displacements of all 42 key points:

[0059] For the first frame (without a previous frame reference), set .

[0060] 3. State transition conditions

[0061] The state transition rules are as follows: (1) IDLE→SIGNING: When The transition is triggered when the motion initiation threshold (default 0.02) is reached, and the idle counter (idle_counter) is reset to 0.

[0062] (2) SIGNING→HOLD: When When the idle time threshold (default 0.005) is reached, the idle counter `idle_counter` is incremented by 1; otherwise, it is reset to 0. When `idle_counter` ≥ `n_idle` (the threshold for consecutive idle frames, default 10 frames), a transition is triggered, and model inference is executed.

[0063] (3) HOLD→IDLE: Automatic transfer after inference is completed, and idle_counter is reset to 0.

[0064] 4. Cooling-off period anti-repetition mechanism

[0065] The system maintains a cooling counter, `cooldown_counter`. After each model inference, `cooldown_counter` is set to `cooldown_frames` (default 30 frames, approximately 1 second). Thereafter, `cooldown_counter` is decremented by 1 for each video frame processed. While `cooldown_counter` is greater than 0, even if the state machine triggers the `SIGNING→HOLD` transition, model inference is not performed, and an empty result is returned directly. This mechanism effectively prevents repeated recognition due to short pauses between sign language words in continuous sign language expressions.

[0066] Furthermore, the neural network models used in S4-S6 are pre-trained models, and the training samples of the models are expanded by a combination of the following data augmentation methods: spatial jitter enhancement, temporal resampling enhancement, keypoint random discard enhancement, and horizontal mirror enhancement. The horizontal mirror enhancement is achieved by swapping the left-hand keypoint group with the right-hand keypoint group and inverting all x-coordinates.

[0067] Furthermore, S8 performs matching according to the priority order of direct word mapping, verb-object structure templates, interrogative sentence templates, state description templates, material list templates, and catch-all splicing, converting sign language vocabulary into fluent and natural Chinese sentences; among them, The first priority is direct word mapping; when the recognition result is a single gloss word, the corresponding Chinese expression is directly returned from the table. The second priority is verb-object structure template matching; the gloss sequence is matched with predefined verb-object structure patterns containing placeholders. The third priority is interrogative sentence template matching. The fourth priority is state description template matching. The fifth priority is bill of materials template matching. The sixth priority is fallback concatenation; when none of the above templates match, the default format is used to output a single gloss, and multiple glosses are connected by commas before output.

[0068] In a specific embodiment, the template matching translation content is as follows: 1. Template configuration file structure Translation templates are stored in YAML format configuration files.

[0069] 2. Template matching algorithm

[0070] Template matching is performed in the following six priority order: Priority 1: Direct word mapping. When the recognition result contains only a single gloss word, search in the single mapping table. If a match is found, the mapped Chinese expression is returned directly. If no match is found, continue to the next priority.

[0071] Priority 2-5: Pattern template matching, sequentially trying four types of templates: action_object, question, status, and material. For each rule in each type of template, the rule's pattern (pattern list) is compared element-by-element with the input gloss sequence. If the elements in the pattern are ordinary strings, they must be exactly equal to the gloss at the corresponding position; If an element in the pattern is a placeholder enclosed in curly braces (such as "{doc}"), then the gloss value at the corresponding position will be recorded as the value of that placeholder. When the pattern and gloss sequence have the same length and all non-placeholder positions match perfectly, the corresponding placeholder in the template is replaced with the recorded placeholder value to generate the final Chinese sentence.

[0072] Priority 6: fallback stitching. When all templates do not match, a single gloss is output in the default format "(sign language recognition) {gloss}", and multiple glosses are connected with commas and output in the format "(sign language recognition) {glosses}".

[0073] 3. Translation Example

[0074] Table 1 Translation Examples

[0075] In one specific embodiment, the following is included: This embodiment provides experimental verification data and comparative analysis results for the system.

[0076] 1. Experimental Dataset

[0077] The experiment used the following dataset for validation: (1) DEVISIGN-L sign language dataset: contains 2,000 isolated sign language words, each of which was recorded 5 times by 8 signers, for a total of 80,000 video samples. It is divided into training set, validation set and test set in an 8:1:1 ratio.

[0078] (2) Self-collected government affairs scenario dataset: For 300 commonly used government affairs-related sign language words at municipal service windows, 15 hearing-impaired volunteers were invited to record in a simulated government affairs window environment. Each person recorded 10 times for each word, for a total of 45,000 video samples. The data collection environment simulated the lighting conditions and background environment of different government affairs windows.

[0079] 2. Comparison of model parameter count and inference speed

[0080] Table 2 Comparison of Model Parameter Quantity and Inference Speed

[0081] All inference times in the table above were measured on an NVIDIA RTX 3060 GPU (12GB VRAM), with a batch size of 1 and an input sequence length of 64 frames.

[0082] 3. Comparison of recognition accuracy

[0083] Table 3 Comparison of recognition accuracy on the DEVISIGN-L test set

[0084] Table 4 Comparison of recognition accuracy on self-collected government affairs scenario datasets

[0085] Experimental results show that the recognition accuracy of the method in this invention on a general sign language dataset is comparable to that of large-scale models (such as SignBERT, with 87.5M parameters), only about 2 percentage points lower, but with only 6.3% of the parameters and an inference speed 11.4 times faster. On a dataset optimized for government affairs scenarios, the recognition accuracy of the method in this invention reaches 90.3%, surpassing the MS-G3D method (85.8%) with a larger number of parameters, demonstrating the effectiveness and advantages of the grouped keypoint embedding design in municipal window scenarios.

[0086] 4. End-to-end latency test

[0087] Table 5. Measurement results of complete end-to-end processing latency (from sign language action completion to voice output) on the NVIDIA RTX 3060 platform.

[0088] The end-to-end latency is within 300 milliseconds, which is far below the human perception threshold for natural conversation latency (typically 500 milliseconds to 1 second), meeting the requirements for real-time interaction.

[0089] 5. State machine segmentation accuracy test

[0090] Table 6 evaluates the sign language action segmentation accuracy of the state machine on a test set containing 200 consecutive sign language videos (each containing 3-8 consecutive sign language words).

[0091] The results show that the three-state finite state automaton based on hand movement speed can effectively detect the start and end boundaries of sign language actions. With the help of the cooldown mechanism, the false trigger rate can be controlled within an acceptable range.

[0092] 6. Data-enhanced ablation experiments

[0093] Table 7 shows the ablation experiment results on the government affairs scenario dataset, after removing various data augmentation methods.

[0094] Ablation experiments show that all four data augmentation strategies contribute positively to improving the model's generalization performance, with temporal resampling augmentation contributing the most (2.4%). This is because the difference in expression speed among different sign language users is one of the main factors affecting recognition accuracy.

[0095] A lightweight, real-time sign language recognition and translation system for municipal service window scenarios, such as... Figure 2 As shown, a lightweight real-time sign language recognition and translation method for municipal window scenarios, applying any of the above, includes a video acquisition module, a key point extraction module, a preprocessing module, a grouping embedding module, a temporal modeling module, a classification and recognition module, a real-time inference module, a translation module, a speech synthesis module, and a user interaction module connected in sequence. The video capture module is used to capture video frame sequences through an RGB camera; The key point extraction module is used to extract the three-dimensional coordinates of 53 human key points from video frames and encode them into 159-dimensional feature vectors. The preprocessing module is used to normalize the coordinates of key points with the midpoint of the shoulders as the origin and the distance between the shoulders as the unit length. The grouping and embedding module is used to divide key points into left-hand key points, right-hand key points and upper-body key points according to anatomical structure, and then project and stitch them together to form a unified dimension of embedding features. The temporal modeling module includes a learnable positional encoder and a multi-layer Transformer encoder for temporal self-attention modeling of embedded feature sequences; The classification and recognition module includes a masked mean pooling layer and a classification head, which are used to output the probability distribution of sign language word classes; The real-time inference module includes a three-state finite state automaton and a sliding window frame buffer, which are used to automatically detect the boundaries of sign language actions and trigger model inference. The translation module includes a multi-level priority rule template engine for converting sign language word sequences into natural Chinese sentences; The speech synthesis module is used to convert Chinese sentences into speech signals for output. The user interaction module provides a graphical user interface for video preview, recognition result display, and dialogue recording.

[0096] Furthermore, the translation module's rule templates are stored and managed in YAML format configuration files, supporting the dynamic addition and adjustment of translation rules without modifying the program code or retraining the model.

[0097] Furthermore, the system adopts a front-end and back-end separated web application architecture. The back-end provides RESTful API and WebSocket real-time streaming communication interface. After the key points are extracted, the original video frame data is discarded, and subsequent processing is based solely on the key point coordinate data.

[0098] In a specific embodiment, such as Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 As shown, it includes the following: 1. Hardware Environment Configuration The system's hardware requirements are as follows: Processor: Intel Core i5 or higher, or AMD Ryzen 5 or higher x86_64 architecture CPU; Memory: at least 8GB DDR4; Graphics card: NVIDIA GeForce GTX 1650 or higher (with at least 4GB VRAM), or CPU inference mode if a dedicated graphics card is unavailable; Camera: USB or webcam supporting 720p or higher resolution, with a frame rate of at least 30fps; Audio output: standard speakers or desktop speakers. In typical municipal window deployment scenarios, the above hardware configuration is essentially the same as the standard configuration of existing government office terminals, requiring no additional dedicated hardware procurement.

[0099] 2. Software Architecture Design

[0100] The system adopts a front-end and back-end separated web application architecture. The back-end is built on the FastAPI asynchronous web framework based on the Python language, providing RESTful API and WebSocket real-time streaming communication interface; the core of the system consists of the following functional modules: (1) Video Acquisition Module: The system camera is called through the OpenCV library to continuously acquire RGB video frames at a frame rate of 30fps. The video frames are then converted to color space (BGR to RGB) and normalized in size before being sent to the key point extraction module. This module supports multiple input source configurations, including local USB cameras, network IP cameras, and RTSP video streams.

[0101] (2) Keypoint Extraction Module: Human keypoint detection is implemented based on the Google MediaPipe Holistic solution. MediaPipe Holistic can simultaneously detect keypoints of the human hand, face, and body posture. This system selects 53 keypoints (21 for the left hand, 21 for the right hand, and 11 for the upper body) from the detection results. The x, y, and z coordinates of each keypoint are extracted to form a 159-dimensional keypoint feature vector. When keypoint detection fails in a frame (such as when the user briefly leaves the screen or the hand is occluded), the keypoint vector of that frame is filled with a vector of all zeros to ensure the continuity of subsequent processing links.

[0102] (3) Preprocessing Module: The extracted keypoint sequences are normalized using the midpoint between the shoulders as the origin and the distance between the shoulders as the unit length. Specifically, the midpoint of the three-dimensional coordinates of the left shoulder keypoint (the 5th point in the upper body group, coordinate column indices 141-143) and the right shoulder keypoint (the 6th point in the upper body group, coordinate column indices 144-146) is calculated as the origin, and the Euclidean distance between the shoulders is calculated as the scaling factor. The x, y, and z coordinates of all 53 keypoints are normalized by translation and division by the scaling factor. This normalization method eliminates the influence of differences in body scale among different users and changes in the relative position and distance between the user and the camera on the keypoint coordinates, so that the model has a consistent recognition ability for users of different body types and positions. In addition, the preprocessing module is also responsible for padding or truncating variable-length keypoint sequences to a fixed length max_len (default 64 frames) and recording the actual effective length of each sequence.

[0103] (4) Group Embedding Module: Based on the group keypoint embedding mechanism, the 159-dimensional keypoint vector is divided into three functional groups according to anatomical structure: left-hand keypoints (0-62, 63), right-hand keypoints (63-125, 63), and upper-body keypoints (126-158, 33). The three groups of keypoints are mapped to embedding vectors of different dimensions (128 for left hand, 128 for right hand, and 64 for upper body) through independent linear projection layers, and then concatenated into a 320-dimensional vector in the feature dimension. Finally, a unified 256-dimensional embedding feature is obtained by processing through a fusion linear layer, layer normalization, and Dropout layer. The advantages of this group embedding design are: hand keypoints carry the main semantic information of sign language actions and are given a larger embedding dimension (128 dimensions each) to retain richer detailed features; upper-body keypoints mainly provide auxiliary contextual information of body posture and use a smaller embedding dimension (64 dimensions) to reduce the number of parameters; independent projection of each group avoids mutual interference between features of different body parts.

[0104] (5) Temporal Modeling Module: This module includes a learnable positional encoder and a 6-layer Transformer encoder. The learnable positional encoder maintains a learnable parameter matrix of shape (200, 256), initialized with a truncated normal distribution (standard deviation 0.02). During forward propagation, the first T rows are taken and added element-wise with the input embedding to inject frame sequence position information into the model. The Transformer encoder uses a Pre-Norm structure (i.e., layer normalization is performed before the self-attention and feedforward networks). Each layer contains an 8-head self-attention mechanism (d_model=256, 32 dimensions per head) and a feedforward network (1024 internal dimensions, GELU activation function), with a Dropout rate of 0.1. The encoder accepts src_key_padding_mask input to ignore padding frame positions, ensuring that the self-attention mechanism only calculates attention weights between valid frames.

[0105] (6) Classification and Recognition Module: This module includes a masked mean pooling layer and two classification heads. The masked mean pooling layer constructs a Boolean mask based on the actual number of valid frames for each sample, averaging the feature vectors at the valid frame positions to aggregate the variable-length temporal features (B,T,256) into a fixed-dimensional global representation (B,256). The classification head consists of a linear layer (Linear(256,256)), GELU activation, Dropout (p=0.3), and a linear layer (Linear(256,num_classes)), outputting the logits of each sign language word class. During the inference stage, the logic is converted to a probability distribution using the softmax function, and the class corresponding to the highest probability is taken as the recognition result. Low-confidence results are filtered out using a confidence threshold (default 0.6).

[0106] (7) Real-time Inference Module: This module integrates a three-state finite state automaton and a sliding window frame buffer to achieve automatic detection and inference of sign language actions in continuous video streams. The frame buffer is implemented using a circular queue with a capacity of 90 frames, while simultaneously maintaining the original video frame buffer and the keypoint buffer. Figure 9 As shown, the upper body group has 11 key points (numbered 0-10) and 33 dimensions; the left hand group has 21 key points and 63 dimensions; the right hand group has 21 key points and 63 dimensions; totaling 53 key points and 159 dimensions. The state machine determines state transitions based on the average displacement velocity of the 42 hand key points between adjacent frames. A cooling-off period mechanism is activated after each recognition, suppressing new inference triggers during a 30-frame (approximately 1-second) cooling-off period to prevent repeated recognitions due to brief pauses in continuous sign language expressions.

[0107] (8) Translation Module: Based on a multi-level rule template engine driven by YAML configuration files, it converts the recognized gloss word sequences into natural Chinese sentences. The template configuration file defines various patterns such as word mapping tables, verb-object structure templates, interrogative sentence templates, state description templates, and material list templates. During matching, various templates are tried in order of priority, and pattern matching with placeholders and automatic filling are supported. Operation and maintenance personnel can directly edit the YAML configuration file to add or modify translation rules without modifying the program code or retraining the model.

[0108] (9) Speech Synthesis Module: Converts the translated Chinese sentences into speech signals through a text-to-speech (TTS) engine and plays them through a speaker. The system supports integration with multiple TTS engines, and online TTS services or local offline TTS engines can be selected according to the actual deployment environment.

[0109] (10) User interaction module: Provides a web browser-based graphical user interface, including a real-time camera preview area (which can overlay skeletal key points), a recognition status indicator, current recognition results and confidence level display, a dialogue record area, and operation control buttons. The interface design follows accessibility design principles, and the font size and color contrast meet visual accessibility standards.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0111] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A lightweight real-time sign language recognition and translation method for municipal service window scenarios, characterized in that, Includes the following steps: S1. Video frame acquisition and preprocessing: The video frame sequence is acquired by an RGB camera at a preset frame rate, and the video frames are preprocessed by color space conversion and size normalization to obtain preprocessed video frames. S2. Human Key Point Extraction: The human pose estimation model is used to detect human key points in the preprocessed video frames and extract the three-dimensional coordinates of all key points. S3. Key point normalization: Using the midpoint of the user's shoulders as the origin and the Euclidean distance between the shoulders as the unit length, the coordinates of all key points are normalized by translation and scaling. S4. Grouping Key Point Embedding: Divide all key points into three groups according to human anatomy: left hand key points, right hand key points, and upper body key points. The three sets of key points are mapped into three sets of embedding vectors through independent linear projection layers. The three sets of embedding vectors are concatenated and processed by a fusion linear layer, layer normalization and dropout layer to obtain a unified embedding feature. S5. Position Encoding and Temporal Modeling: Learnable position encoding is added to the embedded features to inject frame sequence position information. Then, a multi-layer Transformer encoder is used to perform self-attention modeling on the temporal feature sequence to capture the dependencies of sign language actions in the temporal dimension. S6. Masked Mean Pooling and Classification: Based on the validity mask of each frame, the temporal features output by the Transformer encoder are subjected to masked mean pooling, and the variable-length temporal features are aggregated into a global representation vector of fixed dimensions. The probability distribution of each sign language word class is output through the classification head. S7. Sign Language Segmentation State Machine: Based on the displacement velocity of the left-hand group key points and the right-hand group key points between adjacent frames, a three-state finite state automaton including idle state, sign language performance state and hold state is designed to automatically detect the start and end boundaries of sign language actions in continuous video streams, and trigger model inference of S4-S6 when the sign language action is detected to end, so as to realize the automatic segmentation of sign language actions. S8. Template Matching Translation: The identified sign language word sequence is converted into natural Chinese sentences through a multi-level priority rule template engine; S9. Speech Synthesis Output: The translated Chinese sentences are converted into speech signals through a speech synthesis engine and played through a speaker, realizing a complete translation chain from sign language expression to speech output for hearing-impaired people.

2. The lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, S2 extracts the 3D coordinates of 53 key points, including 21 key points on the left hand (1 on the left wrist, 4 on the left thumb, 4 on the left index finger, 4 on the left middle finger, 4 on the left ring finger, and 4 on the left little finger); 21 key points on the right hand (1 on the right wrist, 4 on the right thumb, 4 on the right index finger, 4 on the right middle finger, 4 on the right ring finger, and 4 on the right little finger); and 11 key points on the upper body (nose tip, left inner corner of eye, right inner corner of eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist). Each key point contains three coordinate components: x, y, and z, for a total of 53 × 3 = 159 dimensional feature vectors.

3. The lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, In S3, the specific steps for translating and scaling the coordinates of all keypoints, using the midpoint between the user's shoulders as the origin and the Euclidean distance between the shoulders as the unit length, are as follows: Let the coordinates of the key point on the left shoulder be... The coordinates of the key point on the right shoulder are The origin of the coordinate system is the midpoint between the shoulders. ; Calculate the scaling factor as the Euclidean distance between the shoulders. When d is less than the threshold At that time, set d=1.0 to avoid division by zero errors; perform the operation on each coordinate component of all 53 key points. Normalization operation.

4. A lightweight real-time sign language recognition and translation method for municipal window scenarios according to claim 1 or 2, characterized in that, In S4, the three sets of keypoints are mapped to three sets of embedding vectors through independent linear projection layers. These three sets of embedding vectors are then concatenated and processed through a fusion linear layer, layer normalization, and Dropout layer to obtain the unified embedding feature. The key points of the left-hand group are taken from the 0th to the 62nd dimension of the 159-dimensional vector, which is a total of 63 dimensions. They are then mapped to a 128-dimensional embedding through a linear layer Linear(63,128). The keypoints of the right-hand group are taken from the 63rd to the 125th dimension, a total of 63 dimensions, and mapped to a 128-dimensional embedding through a linear layer Linear(63,128); The key points of the upper body group are taken from the 126th to the 158th dimension, a total of 33 dimensions, and mapped to a 64-dimensional embedding through a linear layer Linear(33,64); After concatenating the three sets of embeddings along the feature dimension to obtain a 320-dimensional vector, the vector is processed by a linear fusion layer (Linear(320,256)), a LayerNorm(256) normalization layer, and a Dropout(p=0.1) random deactivation layer to output a 256-dimensional unified embedding feature.

5. A lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, In S5, the learnable location encoding added to the embedded features has a dimension of [missing information]. The learnable parameter matrix, where, To the maximum supported sequence length, The embedding dimension is used; the learnable parameter matrix is ​​initialized with a truncated normal distribution and a standard deviation of 0.02; during forward propagation, the first T rows of the parameter matrix are taken and added element-wise to the input embedding, where T is the actual length of the current input sequence.

6. A lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, The multi-layer Transformer encoder in S5 is configured as follows: embedding dimension =256, number of attention heads nhead=8, internal dimension of feedforward network dff=1024, number of encoder layers numlayers=6, dropout ratio is 0.1, activation function is GELU, pre-norm structure is used, that is, layer normalization is performed before self-attention and feedforward network, batch_first is set to True to make the input tensor shape (B,T,D).

7. A lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, The specific details of the masked mean pooling of the temporal features output by the Transformer encoder based on the validity mask of each frame in S6 are as follows: Construct a boolean-type effective mask based on the actual effective frame count (lengths) of each sample. The effective frame position is True, and the padding position is False; Expand to Shape and convert to floating-point type; multiply the Transformer encoder output element-wise with the expanded mask and sum along the time dimension to obtain... The shape features are summed; the mask is summed along the time dimension to obtain the effective frame count and the lower bound is set to 1.0 to avoid division by zero; the sum of features is divided by the effective frame count to obtain the mask mean pooling result.

8. A lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, The classification head in S6 consists of a linear layer Linear(256,256), a GELU activation function, a Dropout (p=0.3) random deactivation layer, and a linear layer Linear(256,num_classes), where num_classes is the number of sign language vocabulary categories.

9. A lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, The state transition conditions of the three-state finite state automaton S7, which includes an idle state, a sign language performance state, and a hold state, are as follows: The condition for transitioning from the idle state to the sign language execution state is: the average motion velocity of the left-hand keypoints and the right-hand keypoints in the current frame. ,in, The threshold for initiating movement; The conditions for transitioning from the sign language state (SIGNING) to the hold state (HOLD) are: the average movement speed of the key points in the left hand group and the key points in the right hand group. continued Frame, in which The threshold for determining stillness. The threshold for the number of consecutive still frames; The condition for transitioning from the HOLD state to the IDLE state is: automatic transition after completing one model inference recognition; The hand movement speed v is calculated as follows: take the Euclidean norm of the three-dimensional coordinate difference between the current frame and the previous frame for 21 left-hand key points and 21 right-hand key points, and take the average value of the 42 key points.

10. A lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, The S7 also includes a cooldown period anti-repetition mechanism: after a sign language recognition is completed, a cooldown counter is started and set to a preset number of cooldown frames. During the cooling-off period, even if the boundary of sign language movement is detected, no new inference recognition is triggered. The cooling counter is decremented by 1 for each frame of video processed until the counter reaches zero and then normal detection is restored. It also includes a sliding window frame buffering mechanism: setting the maximum buffer capacity. Set minimum effective window size A circular queue is used to store video frames and their corresponding key point data. When the number of buffered frames exceeds... The earliest frame is automatically discarded; if the number of buffered frames is insufficient when the state machine triggers inference. Then skip this reasoning.

11. A lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, In S7, the model inference phases S4-S6, triggered when the sign language action is detected to have ended, employ an automatic mixed-precision inference mode on the GPU device, performing some computations in FP16 half-precision floating-point format.

12. A lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, The neural network models used in S4-S6 are pre-trained models, and the training samples of the models are expanded by a combination of the following data augmentation methods: spatial jitter enhancement, temporal resampling enhancement, keypoint random discard enhancement, and horizontal mirror enhancement. Horizontal mirror enhancement is achieved by swapping the left-hand keypoint group with the right-hand keypoint group and inverting all x-coordinates.

13. A lightweight real-time sign language recognition and translation method for municipal service window scenarios according to claim 1, characterized in that, S8 performs matching according to the priority order of direct word mapping, verb-object structure template, interrogative sentence template, state description template, material list template, and catch-all concatenation, converting sign language vocabulary into fluent and natural Chinese sentences; among them, The first priority is direct word mapping; when the recognition result is a single gloss word, the corresponding Chinese expression is directly returned from the table. The second priority is verb-object structure template matching; the gloss sequence is matched with predefined verb-object structure patterns containing placeholders. The third priority is interrogative sentence template matching. The fourth priority is state description template matching. The fifth priority is bill of materials template matching. The sixth priority is fallback concatenation; when none of the above templates match, the default format is used to output a single gloss, and multiple glosses are connected by commas before output.

14. A lightweight real-time sign language recognition and translation system for municipal service window scenarios, characterized in that, The lightweight real-time sign language recognition and translation method for municipal window scenarios according to any one of claims 1-13 includes a video acquisition module, a key point extraction module, a preprocessing module, a group embedding module, a temporal modeling module, a classification and recognition module, a real-time inference module, a translation module, a speech synthesis module, and a user interaction module connected in sequence. The video capture module is used to capture video frame sequences through an RGB camera; The key point extraction module is used to extract the three-dimensional coordinates of 53 human key points from video frames and encode them into 159-dimensional feature vectors. The preprocessing module is used to normalize the coordinates of key points with the midpoint of the shoulders as the origin and the distance between the shoulders as the unit length. The grouping and embedding module is used to divide key points into left-hand key points, right-hand key points and upper-body key points according to anatomical structure, and then project and stitch them together to form a unified dimension of embedding features. The temporal modeling module includes a learnable positional encoder and a multi-layer Transformer encoder for temporal self-attention modeling of embedded feature sequences; The classification and recognition module includes a masked mean pooling layer and a classification head, which are used to output the probability distribution of sign language word classes; The real-time inference module includes a three-state finite state automaton and a sliding window frame buffer, which are used to automatically detect the boundaries of sign language actions and trigger model inference. The translation module includes a multi-level priority rule template engine for converting sign language word sequences into natural Chinese sentences; The speech synthesis module is used to convert Chinese sentences into speech signals for output. The user interaction module provides a graphical user interface for video preview, recognition result display, and dialogue recording.

15. A lightweight real-time sign language recognition and translation system for municipal service window scenarios according to claim 14, characterized in that, The translation module's rule templates are stored and managed in YAML format configuration files, allowing for the dynamic addition and adjustment of translation rules without modifying the program code or retraining the model.

16. A lightweight real-time sign language recognition and translation system for municipal service window scenarios according to claim 14, characterized in that, The system adopts a front-end and back-end separated web application architecture. The back-end provides RESTful API and WebSocket real-time streaming communication interface. After the key points are extracted, the original video frame data is discarded, and subsequent processing is based solely on the key point coordinate data.

Images