Sign language recognition method based on kinematics semantic description and Mama architecture
By transforming sign language labels into kinematic semantic descriptions and adopting a sign language recognition method based on the Mamba architecture, the problems of impoverished text modal semantics and insufficient long text encoding in existing technologies are solved, achieving efficient sign language recognition and low-cost training, and improving recognition accuracy and generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGZHOU UNIVERSITY
- Filing Date
- 2026-01-22
- Publication Date
- 2026-05-01
AI Technical Summary
Existing sign language recognition methods suffer from problems such as impoverished text modal semantics, insufficient long text encoding capabilities, and high training costs for all parameters, which limit the model's generalization ability and reduce recognition performance.
Employing kinematic semantic description and the Mamba architecture, we transform sign language labels into kinematic semantic descriptions and utilize pre-trained text and visual encoders for feature extraction and similarity calculation. Combined with an alternating freeze training strategy, we reduce computational costs and GPU memory usage.
It improves the semantic alignment accuracy and generalization ability of the model, solves the problem of loss of key information caused by text truncation, and reduces training costs.
Smart Images
Figure CN121963316A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of sign language recognition technology, and in particular relates to a sign language recognition method based on kinematic semantic description and Mamba architecture. Background Technology
[0002] In recent years, significant progress has been made in sign language isolated word recognition methods based on multimodal contrastive learning. Architectures such as ActionCLIP have achieved cross-modal retrieval and recognition tasks by aligning video and text features. However, existing methods still have several key shortcomings.
[0003] First, text-based modalities suffer from semantic poverty. Current methods typically use highly abstract sign language labels as text input. These labels cannot describe specific kinematic details such as hand shape, trajectory, position, and expression, making it difficult for the model to establish a deep connection between visual actions and text semantics, thus limiting its generalization ability.
[0004] Secondly, the model's encoding capability for long texts is insufficient. If detailed action descriptions are introduced to enrich the semantics, the text length often far exceeds the processing limit of mainstream Transformer encoders. Limited by the preset context length, the model has to forcibly truncate long texts, resulting in the loss of crucial motion information and severely impairing recognition performance.
[0005] Finally, full parameter fine-tuning is costly. Full parameter training of large-scale visual encoders requires enormous GPU memory resources, and the significant differences between video and text modalities make direct joint training difficult, limiting the application of the technology on conventional hardware.
[0006] Therefore, there is an urgent need for a sign language recognition scheme that can fully utilize rich kinematic semantics, efficiently process long sequence descriptions, and has controllable training costs. Summary of the Invention
[0007] In view of the above-mentioned deficiencies of the prior art, this invention proposes a sign language recognition method based on kinematic semantic description and Mamba architecture. The technical solution designed in this invention includes the following steps: Obtain sign language videos and their corresponding sign language tags; Transforming sign language labels into kinematic semantic descriptions; The kinematic semantic description is encoded using a pre-trained text encoder to obtain text feature vectors; Visual feature vectors are obtained by encoding sign language videos using a visual encoder. The similarity between visual feature vectors and text feature vectors is calculated to classify and recognize sign language videos.
[0008] Preferably, the step of converting sign language tags into kinematic semantic descriptions includes: For a single action sequence, the sign language vocabulary is mapped to a unique kinematic description; For sign language words in the format XY, determine the corresponding semantic item based on index X, and determine the signage description based on variant Y; For complex sign language vocabulary containing multi-stage actions, map and preserve segmented descriptive structures to generate long sequence text; For sign language labels with multiple meanings, a dual disambiguation matching mechanism is used to determine the corresponding kinematic semantic description.
[0009] Preferably, the step of using a dual disambiguation matching mechanism to determine the corresponding kinematic semantic description includes: Parse the index information in the sign language tags and match it with the semantic order in the preset dictionary; When the index information is unclear, the English translation associated with the sign language label is introduced, and the final match is determined by calculating the semantic similarity between the English translation and the description of each candidate meaning in the dictionary.
[0010] Preferably, the text encoder uses the state-space model Mamba.
[0011] Preferably, the visual encoder includes a visual Transformer ViT main network and integrates a low-rank adaptive LoRA module in the attention layer.
[0012] Preferably, the model training process of the method employs an alternating freeze strategy, which includes: In the first stage, the backbone parameters of the visual encoder are frozen, and the temporal fusion part of the text encoder and the visual encoder is trained. In the second stage, the text encoder and time-series fusion part are frozen, and only the parameters of the backbone network of the visual encoder are adjusted.
[0013] Preferably, the classification and recognition of the sign language video includes: Extract the visual feature vector from the sign language video to be identified; The candidate sign language vocabulary set is transformed into a kinematic semantic description set, and the corresponding text feature vector library is extracted by the text encoder. Calculate the similarity between the visual feature vector and each vector in the text feature vector library; The sign language label corresponding to the text feature vector with the highest similarity is used as the recognition result.
[0014] Beneficial effects: 1. This application introduces standardized kinematic descriptions to replace abstract Gloss labels, enabling the model to learn the essential details of actions such as hand shape, trajectory, and position, fundamentally solving the problem of poor semantic modality in text and significantly improving the semantic alignment accuracy and generalization ability of the model. 2. This application uses the state-space model Mamba instead of the traditional Transformer as the text encoder, and leverages its linear computational complexity and excellent long sequence modeling capabilities to solve the problem of key information loss caused by text truncation. 3. This application integrates the LoRA module into the visual encoder and designs an alternating freeze training strategy. While achieving efficient alignment of video and text features, it only requires fine-tuning a very small number of parameters, which greatly reduces the memory usage and computational cost. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of a model framework of a preferred embodiment of the present invention. Detailed Implementation
[0016] The embodiments of the present invention will be described in detail below. The embodiments described below are implemented based on the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the embodiments described below.
[0017] This invention designs a sign language recognition method based on kinematic semantic description and the Mamba architecture, such as... Figure 1 As shown, the technical solution includes the following steps, specifically: Obtain sign language videos and their corresponding sign language tags; Transforming sign language labels into kinematic semantic descriptions; The kinematic semantic description is encoded using a pre-trained text encoder to obtain text feature vectors; Visual feature vectors are obtained by encoding sign language videos using a visual encoder. The similarity between visual feature vectors and text feature vectors is calculated to classify and recognize sign language videos.
[0018] Specifically, the sign language videos and tags are derived from publicly available Chinese sign language isolated word datasets, including NationCSL-DP, NMFs-CSL, and SLR500, and are divided into training and testing sets according to a preset ratio.
[0019] In addition, according to the National Common Sign Language Dictionary, sign language tags (gloss) are systematically transformed into specific kinematic descriptions. The descriptive text of compound words and mixed-type words is usually longer than 200 tokens, which forms the data basis for introducing the Mamba model for long sequence modeling.
[0020] Preferably, the sign language labels are converted into kinematic semantic descriptions, including: For a single action sequence, the sign language vocabulary is mapped to a unique kinematic description; For sign language words in the format XY, determine the corresponding semantic item based on index X, and determine the signage description based on variant Y; For complex sign language vocabulary containing multi-stage actions, map and preserve segmented descriptive structures to generate long sequence text; For sign language labels with multiple meanings, a dual disambiguation matching mechanism is used to determine the corresponding kinematic semantic description.
[0021] Preferably, a dual disambiguation matching mechanism is used to determine the corresponding kinematic semantic description, including: Parse the index information in the sign language tags and match it with the semantic order in the preset dictionary; When the index information is unclear, the English translation associated with the sign language label is introduced, and the final match is determined by calculating the semantic similarity between the English translation and the description of each candidate meaning in the dictionary.
[0022] Specifically, the dual disambiguation matching mechanism follows a priority strategy: index matching is performed first; when the index is unclear, the English translations provided with the dataset are used as auxiliary semantic anchors for similarity matching. For a small number of words not included in the dictionary, a fallback mechanism is used to directly retain the original sign language labels as text input.
[0023] In addition, before model processing, the input sign language video needs to be preprocessed, including fixed frame sampling (64 frames in this embodiment) and standardization processing.
[0024] Preferably, the text encoder uses the state-space model Mamba.
[0025] Preferably, the visual encoder includes a visual Transformer ViT main network and integrates a low-rank adaptive LoRA module in the attention layer.
[0026] The construction of the visual encoder includes: using ViT for frame-by-frame spatial feature extraction and injecting a low-rank adaptation matrix into its attention layer to achieve efficient parameter fine-tuning; using a temporal fusion module (adopting the temporal Transformer from ActionCLIP) to perform temporal modeling of frame-level features, ultimately outputting a 512-dimensional video feature vector. During training, the ViT backbone is frozen, and only the LoRA parameters are updated.
[0027] Preferably, the model training process of the method employs an alternating freeze strategy, which includes: In the first stage, the backbone parameters of the visual encoder are frozen, and the temporal fusion part of the text encoder and the visual encoder is trained. In the second stage, the text encoder and time-series fusion part are frozen, and only the parameters of the backbone network of the visual encoder are adjusted.
[0028] Specifically, the training process is as follows: In the first stage (text adaptation stage), the ViT backbone is frozen, and the Mamba text encoder and temporal fusion module are trained with all parameters. The goal is to adapt the text encoder to the sign language semantic space. In the second stage (visual fine-tuning stage), the Mamba and temporal fusion modules are frozen, and only the LoRA parameters in ViT are trained. The goal is to fine-tune visual features based on the defined text feature space to achieve accurate alignment. Training uses the AdamW optimizer and a cosine annealing learning rate strategy, and the optimal model is saved using validation set accuracy as the monitoring metric.
[0029] Preferably, the sign language video is classified and identified, including: Extract the visual feature vector from the sign language video to be identified; The candidate sign language vocabulary set is transformed into a kinematic semantic description set, and the corresponding text feature vector library is extracted by a text encoder. Calculate the similarity between the visual feature vector and each vector in the text feature vector library; The sign language label corresponding to the text feature vector with the highest similarity is used as the recognition result.
[0030] Specifically, cosine similarity is used for similarity calculation. During prediction, features of all candidate words describing the text are pre-calculated and a feature library is built. Then, the features of the video to be tested are compared with all features in the library, and the highest similarity is taken as the recognition result.
[0031] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A sign language recognition method based on kinematic semantic description and Mamba architecture, characterized in that, include: Obtain sign language videos and their corresponding sign language tags; Transforming sign language labels into kinematic semantic descriptions; The kinematic semantic description is encoded using a pre-trained text encoder to obtain text feature vectors; Visual feature vectors are obtained by encoding sign language videos using a visual encoder. The similarity between visual feature vectors and text feature vectors is calculated to classify and recognize sign language videos.
2. The sign language recognition method based on kinematic semantic description and Mamba architecture according to claim 1, characterized in that, The process of converting sign language labels into kinematic semantic descriptions includes: For a single action sequence, the sign language vocabulary is mapped to a unique kinematic description; For sign language words in the format XY, determine the corresponding semantic item based on index X, and determine the signage description based on variant Y; For complex sign language vocabulary containing multi-stage actions, map and preserve segmented descriptive structures to generate long sequence text; For sign language labels with multiple meanings, a dual disambiguation matching mechanism is used to determine the corresponding kinematic semantic description.
3. The sign language recognition method based on kinematic semantic description and Mamba architecture according to claim 2, characterized in that, The method of using a dual disambiguation matching mechanism to determine the corresponding kinematic semantic description includes: Parse the index information in the sign language tags and match it with the semantic order in the preset dictionary; When the index information is unclear, the English translation associated with the sign language label is introduced, and the final match is determined by calculating the semantic similarity between the English translation and the description of each candidate meaning in the dictionary.
4. The sign language recognition method based on kinematic semantic description and Mamba architecture according to claim 1, characterized in that, The text encoder uses the state-space model Mamba.
5. The sign language recognition method based on kinematic semantic description and Mamba architecture according to claim 1, characterized in that, The visual encoder includes a visual Transformer ViT main network and integrates a low-rank adaptive LoRA module in the attention layer.
6. The sign language recognition method based on kinematic semantic description and Mamba architecture according to claim 1, characterized in that, The model training process of the method employs an alternating freeze strategy, which includes: In the first stage, the backbone parameters of the visual encoder are frozen, and the temporal fusion part of the text encoder and the visual encoder is trained. In the second stage, the text encoder and time-series fusion part are frozen, and only the parameters of the backbone network of the visual encoder are adjusted.
7. The sign language recognition method based on kinematic semantic description and Mamba architecture according to claim 1, characterized in that, The classification and recognition of sign language videos includes: Extract the visual feature vector from the sign language video to be identified; The candidate sign language vocabulary set is transformed into a kinematic semantic description set, and the corresponding text feature vector library is extracted by the text encoder. Calculate the similarity between the visual feature vector and each vector in the text feature vector library; The sign language label corresponding to the text feature vector with the highest similarity is used as the recognition result.