A lightweight gesture recognition model training method and apparatus for sign language interaction
By constructing a lightweight student model and combining dynamic temporal convolution and temporal contrastive learning, the problems of real-time interaction and large-scale labeled data dependence in sign language recognition technology on mobile terminals are solved, achieving high-precision and robust sign language recognition results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG VOCATIONAL COLLEGE OF SPECIAL EDUCATION
- Filing Date
- 2026-03-16
- Publication Date
- 2026-06-02
AI Technical Summary
Existing vision-based sign language recognition technologies struggle to achieve real-time interaction on resource-constrained mobile devices, and their reliance on multimodal information and large-scale, finely labeled data leads to decreased recognition accuracy and insufficient generalization ability.
We construct a lightweight student model, train it by freezing the multimodal teacher model for knowledge distillation, combine dynamic temporal convolution and temporal contrastive learning to adaptively model the temporal dependence of sign language actions, and enhance feature representation capabilities through a cross-attention mechanism to reduce dependence on large-scale labeled data.
It achieves high accuracy and robustness in sign language recognition with low computational power consumption, adapts to complex scenarios, and improves the model's real-time processing and generalization capabilities on mobile terminals.
Smart Images
Figure CN122135435A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of sign language recognition technology, and more specifically, to a lightweight gesture recognition model training method and apparatus for sign language interaction. Background Technology
[0002] Sign language, as the primary communication tool for the deaf and mute community, is a complex visual language encompassing gestures, facial expressions, and body postures. Sign language recognition technology aims to automatically convert sign language videos into text or speech using computer vision and artificial intelligence methods, thereby breaking down communication barriers between the deaf and hearing communities and possessing significant social and practical value. In recent years, with the rapid development of deep learning technology, sign language recognition research has made remarkable progress. The mainstream recognition methods are mainly divided into two categories: those based on wearable devices (such as data gloves) and those based on vision. Vision-based methods require no special hardware, offer a user-friendly experience, and have become a current research hotspot.
[0003] However, existing vision-based sign language recognition technologies still face many challenges in practical applications. First, sign language recognition involves the comprehensive utilization of multimodal information, including hand appearance, movement trajectory, and joint structure. Existing technologies typically use single-modal input (such as using only RGB video), which easily loses motion blur and depth information in fast-moving or hand self-occlusion scenarios, leading to a decrease in recognition accuracy. Although fusing multimodal information such as optical flow or skeletal points can improve recognition performance, the computational overhead of multi-stream parallel inference is huge, making it difficult to achieve real-time interaction on resource-constrained mobile terminals (such as smartphones and AR / VR glasses). Second, sign language gestures are characterized by long temporal dependencies and drastic changes in local motion speed. Existing lightweight models typically use fixed temporal convolutional kernels, which cannot dynamically adjust the receptive field according to the speed of hand movements, resulting in loss of detail in low-speed fine gestures or temporal aliasing in high-speed movements, affecting the accuracy of continuous sign language recognition. In addition, the annotation cost of sign language data is extremely high, especially frame-level sequence alignment annotation, which requires a large amount of professional knowledge and human resources, limiting the model's generalization ability and large-scale application in vertical fields.
[0004] In summary, how to absorb the advantages of multimodal information while ensuring low computational power consumption, achieve adaptive temporal modeling of sign language actions, and reduce dependence on large-scale fine-labeled data are the technical problems that urgently need to be solved in the field of sign language recognition. Summary of the Invention
[0005] The technical problem to be solved by this invention is how to absorb the advantages of multimodal information while ensuring low computing power consumption, realize adaptive temporal modeling of sign language actions, and reduce the dependence on large-scale fine-labeled data. In view of the above-mentioned defects of the prior art, this invention provides a lightweight gesture recognition model training method and device for sign language interaction.
[0006] The technical solution adopted by this invention to solve its technical problem is: on the one hand A lightweight gesture recognition model training method for sign language interaction, characterized by the following steps: Acquire sign language video data, which includes a series of consecutive RGB image frames; Construct at least two teacher models with different modalities, including a first teacher model and a second teacher model. The first teacher model takes an RGB image frame sequence as input to extract appearance features; the second teacher model takes modal data reflecting hand movement information as input to extract motion features; and freezes the parameters of the teacher models during training. A lightweight student model is constructed, which takes only RGB image frame sequences as input and includes a dynamic temporal convolution module. The dynamic temporal convolution module dynamically adjusts the receptive field according to the local motion speed of the input features to adaptively model the temporal dependence of sign language actions under different motion speeds. The teacher model is used to train the student model using knowledge distillation, and the knowledge of the teacher model is transferred to the student model by minimizing the distillation loss function. A temporal contrastive learning task is introduced during training. By constructing temporal positive and negative sample pairs and minimizing the contrastive loss, the sensitivity of the student model to temporal order is enhanced. After training, the parameters of the student model are saved for lightweight deployment.
[0007] Preferably, the second teacher model is an optical flow teacher model, with the optical flow map stacking sequence between adjacent frames as input; or the second teacher model is a skeletal point teacher model, with the hand key point sequence or the hand key point rendering heatmap as input.
[0008] Preferably, the dynamic temporal convolution module includes: A motion speed sensor is used to calculate local motion speed based on the features of adjacent frames, and to generate a weight vector corresponding to the motion speed through a mapping network. Multiple parallel temporal convolutional branches, each with a different dilation rate, correspond to temporal receptive fields of different scales. The attention fusion unit is used to perform a weighted summation of the outputs of each branch based on the weight vector to obtain the fused temporal features.
[0009] Preferably, the distillation loss function includes: The classification distillation loss is used to measure the difference between the output probability distribution of the student model and the output probability distribution of the teacher model. Feature distillation loss is used to measure the difference between the intermediate layer features of the student model and the intermediate layer features of the teacher model.
[0010] Preferably, the feature distillation loss is calculated through a cross-modal attention transfer mechanism, using the intermediate layer features of the student model as the query and the intermediate layer features of the teacher model as the key and value, and calculating the alignment loss between student features and teacher features through cross-attention.
[0011] Preferably, the time-series comparison learning task includes: Positive sample pairs are constructed from the same sign language video clip through temporal resampling; Negative sample pairs are constructed by randomly shuffling the frame order of different sign language video clips or the same clip. A temporal projection head is connected after a specific iteration of the student model to map positive and negative sample pairs to the contrast space; The InfoNCE loss function is used to minimize the distance between positive sample pairs and maximize the distance between negative sample pairs.
[0012] Preferably, after training, the model is further included in a lightweight deployment step: the student model is reparameterized to merge the multi-branch structure of the dynamic temporal convolution module into a single standard temporal convolution layer; and the merged model is quantized with INT8 to generate an inference engine file suitable for mobile devices.
[0013] on the other hand A lightweight gesture recognition model training device for sign language interaction, comprising: The data acquisition module is used to acquire sign language video data, wherein the sign language video verses include a continuous sequence of RGB image frames; A teacher model storage module is used to store at least two teacher models with different modalities. The teacher models include a first teacher model and a second teacher model. The first teacher model takes an RGB image frame sequence as input to extract appearance features. The second teacher model takes modal data reflecting hand movement information as input to extract motion features. The parameters of the teacher models are frozen during training. A student model building module is used to build a lightweight student model. The student model takes only RGB image frame sequences as input and includes a dynamic temporal convolution module. The dynamic temporal convolution module dynamically adjusts the receptive field of its temporal convolution according to the local motion speed of the input features to adaptively model the temporal dependence of sign language actions under different motion speeds. The distillation training module is used to perform knowledge distillation training on the student model using the teacher model, and to transfer the knowledge of the teacher model to the student model by minimizing the distillation loss function. The contrastive learning module is used to introduce a temporal contrastive learning task during training. By constructing temporal positive and negative sample pairs and minimizing the contrastive loss, the sensitivity of the student model to temporal order is enhanced. The model saving module is used to save the parameters of the student model after training, for use in lightweight deployment.
[0014] The beneficial effects of this invention are as follows: 1. By freezing three teacher models (RGB, optical flow, and skeletal points), multi-dimensional information such as appearance texture, motion boundaries, and joint structures is distilled into a single-stream RGB student model. Although the student model relies only on RGB input during inference, its recognition accuracy in complex scenarios such as fast movement and hand self-occlusion approaches that of the large teacher model. A cross-attention mechanism is employed for feature distillation, enabling student features to actively query and absorb motion edge and skeletal structure information from teacher features. This results in alignment accuracy higher than traditional MSE loss, further enhancing feature representation capabilities.
[0015] 2. The dynamic temporal convolution module adjusts the dilation rate weights of the temporal convolution in real time based on the local motion speed (calculated through inter-frame feature similarity). It automatically expands the receptive field to capture large displacements during fast movements (such as waving) and contracts the receptive field to preserve details during slow, fine movements (such as finger bending). This resolves the contradiction that fixed convolution kernels cannot simultaneously handle fast and slow movements, significantly improving the temporal boundary localization accuracy of continuous sign language recognition. By constructing positive and negative samples through temporal resampling and frame shuffling, the model is forced to learn the causal order of actions, avoiding misrecognition of reversed or disordered gestures and enhancing the model's sensitivity to temporal logic.
[0016] 3. Temporal contrastive learning serves as an auxiliary task, utilizing massive amounts of unlabeled video data without additional annotation. By distinguishing between original segments and out-of-order segments, the model automatically learns the temporal structure of gestures, reducing the need for expensive frame-level annotations and improving the model's generalization ability in vertical domains.
[0017] 4. The student model uses only RGB as input, and its backbone employs a lightweight CNN with dynamic temporal convolutions, keeping the number of parameters under 5M. After training, the dynamic temporal module is reparameterized and fused into a single standard convolution, eliminating dynamic routing overhead and further improving inference speed. After training, the model undergoes INT8 quantization, compressing its size to a few MB. On Snapdragon CPUs, it can achieve real-time processing speeds exceeding 30 FPS, meeting the interactive needs of smartphones, AR / VR glasses, and other mobile devices.
[0018] 5. Even if the information of a certain modality is damaged (such as changes in illumination affecting RGB, or motion blur affecting optical flow), the multimodal representations learned by the student model through distillation can still remain stable, thus improving the robustness of the model. Temporal contrastive learning makes the model less sensitive to disturbances such as frame rate changes and local jitter, making it more adaptable in practical applications. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a schematic diagram illustrating the steps of a lightweight gesture recognition model training method for sign language interaction according to an embodiment of this application.
[0020] Figure 2 This is a schematic diagram of the composition of a lightweight gesture recognition model training device for sign language interaction according to an embodiment of this application. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention. Example 1
[0022] Preferred embodiments of the present invention, for example Figure 1 As shown, a lightweight gesture recognition model training method for sign language interaction includes the following steps: S1. Acquire sign language video data, which includes a series of consecutive RGB image frames. As an optional embodiment, this may include sampling at a fixed frame rate (e.g., 25fps), segmenting the video into fixed-length segments (e.g., T=32 frames), with each segment serving as a training sample. Each frame is scaled to a uniform size (e.g., 224×224 pixels) and normalized, mapping pixel values to the [0,1] interval. For teacher models requiring optical flow or skeletal point modalities, offline calculation of optical flow maps and skeletal point heatmaps is performed: dense optical flow between adjacent frames is calculated using the TV-L1 algorithm and stacked to form a temporal optical flow map (e.g., stacking 5 frames of optical flow with 10 channels); hand pose estimation algorithms (e.g., MediaPipe or Hand4Whole) are used to extract key points for both hands and face, rendering them as Gaussian heatmaps (generating a heatmap for each channel of key point) to preserve spatial structure information.
[0023] S2. Construct at least two teacher models with different modalities, including a first teacher model and a second teacher model. The first teacher model takes an RGB image frame sequence as input to extract appearance features; the second teacher model takes modal data reflecting hand movement information as input to extract motion features. The parameters of the teacher models are frozen during training. As an optional embodiment, the first teacher model can adopt an architecture based on VideoSwin Transformer or I3D, taking an RGB sequence as input and outputting classification logits and intermediate layer feature maps. The second teacher model can be an optical flow teacher model, taking a stacked sequence of optical flow maps between adjacent frames as input, using a P3D or C3D architecture; or the second teacher model can be a skeletal point teacher model, taking a sequence of hand keypoints or a heatmap rendered from hand keypoints as input, using an ST-GCN or ST-TR architecture. Each teacher model is pre-trained on a large sign language dataset to ensure strong feature extraction capabilities.
[0024] S3. Construct a lightweight student model. The student model takes only RGB image frame sequences as input and includes a dynamic temporal convolution module. The dynamic temporal convolution module dynamically adjusts the receptive field based on the local motion velocity of the input features to adaptively model the temporal dependence of sign language actions at different motion velocities. As an optional embodiment, the dynamic temporal convolution module includes a motion velocity sensor, multiple parallel temporal convolution branches, and an attention fusion unit. The motion velocity sensor calculates the local motion velocity based on the features of adjacent frames and generates a weight vector corresponding to the motion velocity through a mapping network. Each of the multiple parallel temporal convolution branches has a different dilation rate, corresponding to temporal receptive fields of different scales. The attention fusion unit performs a weighted summation of the outputs of each branch based on the weight vector to obtain the fused temporal features.
[0025] S4. The teacher model is used to train the student model using knowledge distillation. Knowledge from the teacher model is transferred to the student model by minimizing the distillation loss function. As an optional embodiment, the distillation loss function includes classification distillation loss and feature distillation loss. Classification distillation loss measures the difference between the output probability distribution of the student model and the output probability distribution of the teacher model. Feature distillation loss measures the difference between the intermediate features of the student model and the intermediate features of the teacher model; the feature distillation loss is calculated through a cross-modal attention transfer mechanism, using the intermediate features of the student model as the query and the intermediate features of the teacher model as the key and value, and calculating the alignment loss between the student and teacher features through cross-attention.
[0026] S5. Introduce a temporal contrastive learning task during training. By constructing temporal positive and negative sample pairs and minimizing the contrastive loss, the sensitivity of the student model to temporal order is enhanced.
[0027] S51. Construct positive sample pairs from the same sign language video clip through temporal resampling; S52. Construct negative sample pairs from different sign language video clips or by randomly shuffling the frame order of the same clip. S53. Connect a temporal projection head after a specific iteration of the student model to map positive and negative sample pairs to the contrast space. S54. Use the InfoNCE loss function to minimize the distance between positive sample pairs and maximize the distance between negative sample pairs.
[0028] S6. Lightweight Model Deployment Steps: After training, save the parameters of the student model for lightweight deployment, perform structural reparameterization on the student model, merge the multi-branch structure of the dynamic temporal convolution module into a single standard temporal convolution layer, and perform INT8 quantization on the fused model to generate an inference engine file suitable for mobile devices. Example 2
[0029] A lightweight gesture recognition model training device for sign language interaction, referring to Figure 2 It includes a data acquisition module, a teacher model storage module, a student model construction module, a distillation training module, a comparative learning module, and a model saving module.
[0030] The data acquisition module is used to acquire sign language video data, which includes a continuous sequence of RGB image frames. As an optional embodiment, the module is also responsible for preprocessing the video data, including frame sampling, size normalization, pixel normalization, and offline calculation of optical flow maps and skeletal point heatmaps for the teacher model.
[0031] The teacher model storage module is used to store at least two different modalities of teacher models. The teacher models include a first teacher model and a second teacher model. The first teacher model takes RGB image frame sequences as input and is used to extract appearance features. The second teacher model takes modal data reflecting hand movement information as input and is used to extract motion features. The parameters of the teacher models are frozen during training.
[0032] The student model building module is used to build a lightweight student model. The student model only takes RGB image frame sequences as input and includes a dynamic temporal convolution module. The dynamic temporal convolution module dynamically adjusts the receptive field of its temporal convolution according to the local motion speed of the input features to adaptively model the temporal dependence of sign language actions under different motion speeds. The specific components of the dynamic temporal convolution module include a motion speed sensor, multiple parallel temporal convolution branches, and an attention fusion unit.
[0033] The distillation training module is used to train the student model using the teacher model through knowledge distillation, and to transfer the knowledge of the teacher model to the student model by minimizing the distillation loss function.
[0034] The contrastive learning module is used to introduce a temporal contrastive learning task during training. By constructing temporal positive and negative sample pairs and minimizing the contrastive loss, it enhances the sensitivity of the student model to temporal order.
[0035] The model saving module is used to save the parameters of the student model after training for lightweight deployment. As an optional embodiment, a model lightweighting module is also included, used to perform structural reparameterization on the student model after training, fusing the multi-branch structure of the dynamic temporal convolution module into a single standard temporal convolutional layer, and performing INT8 quantization on the fused model to generate an inference engine file suitable for mobile devices.
[0036] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A lightweight gesture recognition model training method for sign language interaction, characterized in that, Includes the following steps: Acquire sign language video data, which includes a series of consecutive RGB image frames; Construct at least two teacher models with different modalities, including a first teacher model and a second teacher model. The first teacher model takes an RGB image frame sequence as input to extract appearance features; the second teacher model takes modal data reflecting hand movement information as input to extract motion features. The parameters of the teacher model are frozen during training; A lightweight student model is constructed, which takes only RGB image frame sequences as input and includes a dynamic temporal convolution module. The dynamic temporal convolution module dynamically adjusts the receptive field according to the local motion speed of the input features to adaptively model the temporal dependence of sign language actions under different motion speeds. The teacher model is used to train the student model using knowledge distillation, and the knowledge of the teacher model is transferred to the student model by minimizing the distillation loss function. A temporal contrastive learning task is introduced during training. By constructing temporal positive and negative sample pairs and minimizing the contrastive loss, the sensitivity of the student model to temporal order is enhanced. After training, the parameters of the student model are saved for lightweight deployment.
2. The lightweight gesture recognition model training method for sign language interaction according to claim 1, characterized in that, The second teacher model is an optical flow teacher model, which takes the stacked sequence of optical flow maps between adjacent frames as input; or the second teacher model is a skeletal point teacher model, which takes the sequence of hand key points or the heat map of hand key points as input.
3. The lightweight gesture recognition model training method for sign language interaction according to claim 1, characterized in that, The dynamic temporal convolution module includes: A motion speed sensor is used to calculate local motion speed based on the features of adjacent frames, and to generate a weight vector corresponding to the motion speed through a mapping network. Multiple parallel temporal convolutional branches, each with a different dilation rate, correspond to temporal receptive fields of different scales. The attention fusion unit is used to perform a weighted summation of the outputs of each branch based on the weight vector to obtain the fused temporal features.
4. The lightweight gesture recognition model training method for sign language interaction according to claim 1, characterized in that, The distillation loss function includes: The classification distillation loss is used to measure the difference between the output probability distribution of the student model and the output probability distribution of the teacher model. Feature distillation loss is used to measure the difference between the intermediate layer features of the student model and the intermediate layer features of the teacher model.
5. The lightweight gesture recognition model training method for sign language interaction according to claim 4, characterized in that, The feature distillation loss is calculated through a cross-modal attention transfer mechanism, using the intermediate layer features of the student model as queries and the intermediate layer features of the teacher model as keys and values, and calculating the alignment loss between student features and teacher features through cross-attention.
6. The lightweight gesture recognition model training method for sign language interaction according to claim 1, characterized in that, The time-series comparison learning task includes: Positive sample pairs are constructed from the same sign language video clip through temporal resampling; Negative sample pairs are constructed by randomly shuffling the frame order of different sign language video clips or the same clip. A temporal projection head is connected after a specific iteration of the student model to map positive and negative sample pairs to the contrast space; The InfoNCE loss function is used to minimize the distance between positive sample pairs and maximize the distance between negative sample pairs.
7. The lightweight gesture recognition model training method for sign language interaction according to claim 1, characterized in that, After training, the model is further deployed in a lightweight manner: the student model is reparameterized to merge the multi-branch structure of the dynamic temporal convolution module into a single standard temporal convolutional layer; and the merged model is quantized with INT8 to generate an inference engine file suitable for mobile devices.
8. A lightweight gesture recognition model training device for sign language interaction, characterized in that, include: The data acquisition module is used to acquire sign language video data, wherein the sign language video verses include a continuous sequence of RGB image frames; A teacher model storage module is used to store at least two teacher models with different modalities. The teacher models include a first teacher model and a second teacher model. The first teacher model takes an RGB image frame sequence as input and is used to extract appearance features. The second teacher model takes modal data reflecting hand movement information as input to extract motion features; the parameters of the teacher model are frozen during training. A student model building module is used to build a lightweight student model. The student model takes only RGB image frame sequences as input and includes a dynamic temporal convolution module. The dynamic temporal convolution module dynamically adjusts the receptive field of its temporal convolution according to the local motion speed of the input features to adaptively model the temporal dependence of sign language actions under different motion speeds. The distillation training module is used to perform knowledge distillation training on the student model using the teacher model, and to transfer the knowledge of the teacher model to the student model by minimizing the distillation loss function. The contrastive learning module is used to introduce a temporal contrastive learning task during training. By constructing temporal positive and negative sample pairs and minimizing the contrastive loss, the sensitivity of the student model to temporal order is enhanced. The model saving module is used to save the parameters of the student model after training, for use in lightweight deployment.