A sign language recognition method and system
By combining self-supervised pre-training and minimally labeled fine-tuning, and using wrist optical sensors and inertial measurement units to collect data, efficient sign language recognition was achieved. This solved the problems of high data labeling costs and weak model generalization ability in traditional sign language recognition technology, and improved recognition accuracy and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI UNIV OF TECH
- Filing Date
- 2026-03-27
- Publication Date
- 2026-07-10
Smart Images
Figure CN122365044A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of sign language recognition technology. Specifically, this invention relates to a sign language recognition method and system. Background Technology
[0002] Sign language recognition has significant application value in communication, human-computer interaction, and rehabilitation training for people with hearing or speech impairments. Existing wearable sign language recognition solutions typically integrate multiple sources of sensors, such as optical sensors and IMUs, to achieve recognition. For example, methods that use wrist information detection sensors and IMU multimodal data, along with convolutional neural networks for classification, can achieve high recognition accuracy.
[0003] Chinese Patent 119964250B provides a sign language recognition method and system based on a ConvLSTM model. The method includes: acquiring strain signal data under different hand movements and annotating the strain signal data with semantic information to obtain a dataset; performing multi-scale spatiotemporal decomposition on the strain signal data according to a preset multidimensional hierarchical transformation (MHT), obtaining high-frequency and low-frequency components based on the decomposition results, and reconstructing the high-frequency and low-frequency components to obtain reconstructed features; using the reconstructed features as input to an improved ConvLSTM model and the annotated semantic information as output to the improved ConvLSTM model, iteratively training the improved ConvLSTM model to obtain a sign language recognition model; and inputting the target strain signal data of the hand movement to be recognized into the sign language recognition model to obtain the semantic information corresponding to the target strain signal data.
[0004] In existing technologies, the data collected by sensors usually requires manual action segmentation and labeling, which is time-consuming and costly. Secondly, when the number of labeled samples is limited or the user's wearing method changes, the model's generalization ability is weak and the recognition performance is prone to decline. Summary of the Invention
[0005] The present invention aims to provide a sign language recognition method and system to reduce data annotation costs and achieve accurate sign language recognition even with limited sample data.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] This invention provides a sign language recognition method, comprising the following steps:
[0008] Step 1: Acquire multi-channel time-series data collected by wrist optical sensors and inertial measurement units, and construct motion segment samples;
[0009] Step 2: Perform self-supervised pre-training based on action segment samples to obtain a temporal feature encoder for sign language recognition;
[0010] Step 3: Label a small number of action segment samples, load the pre-trained temporal feature encoder onto the small number of labeled action segment samples and perform classification training to obtain a sign language classification model;
[0011] Step 4: Based on the continuous stream data collected by the wrist optical sensor and inertial measurement unit, the action trigger segmentation and window construction are performed, and the sign language classification model is used for online reasoning to output the sign language category result.
[0012] In step one, the data collected by the wrist optical sensor and the data collected by the inertial measurement unit are aligned using timestamps.
[0013] In step two, the self-supervised pre-training employs a dual-network structure comprising an online network and a target network. At least two augmented views are generated for the same action segment, and their representations are trained. The parameters of the target network are updated using an exponential moving average. Augmented views include at least one of amplitude scaling, noise perturbation, temporal occlusion, or random temporal clipping. The temporal feature encoder comprises a local temporal feature extraction module and a long-range dependency modeling module. The local temporal feature extraction module uses a one-dimensional convolutional network, and the long-range dependency modeling module uses a Transformer encoding network.
[0014] In step three, classification training uses either the linear evaluation mode of the frozen encoder or the end-to-end fine-tuning mode of the unfrozen encoder.
[0015] In step four, the action triggering segment specifically involves calculating the energy index for the continuous stream data and comparing it with the background threshold.
[0016] This invention provides a system for applying a sign language recognition method, comprising a wrist optical acquisition module, an inertial measurement module, a preprocessing and synchronization module, a self-supervised pre-training module, a minimally labeled fine-tuning module, and a real-time inference module. The wrist optical acquisition module uses a wrist optical sensor to acquire image information of the user's wrist region. The inertial measurement module uses an inertial measurement unit to acquire the inertial features of the user's wrist movement. The preprocessing and synchronization module standardizes and synchronizes the data acquired by the wrist optical acquisition module and the inertial measurement module. The self-supervised pre-training module trains a temporal feature encoder using the information acquired by the wrist optical acquisition module and the inertial measurement module. The minimally labeled fine-tuning module labels a small number of action segment samples and performs classification training to obtain a sign language classification model. The real-time inference module acquires the data input from the wrist optical acquisition module and the inertial measurement module in real time, performs online inference, and outputs the sign language category result.
[0017] The technical effects of this invention are as follows:
[0018] (1) This invention reduces the cost of data annotation and makes the method more practical. It adopts a mode that combines self-supervised pre-training with fine-tuning with few annotations. Through the BYOL self-supervised learning framework, a large number of unlabeled action segment samples are pre-trained, and a high-performance temporal feature encoder can be obtained without manual annotation. Subsequently, only a small number of action segment samples need to be annotated to complete the classification training, which greatly reduces the workload and cost of manual annotation and solves the problem of high dependence on large-scale labeled data and high annotation cost of traditional sign language recognition models.
[0019] (2) This invention reduces model inference latency while ensuring recognition accuracy by combining a lightweight coding network that integrates local temporal feature extraction and long-range dependency modeling. The temporal feature encoder integrates a local temporal feature extraction module (one-dimensional convolutional network) and a long-range dependency modeling module (Transformer coding network), which can accurately extract local detail features of actions and effectively capture long-range temporal dependencies of actions, thereby enhancing the model's adaptability to different action amplitudes and environmental interference.
[0020] (3) The multimodal data fusion of the present invention is accurate, which improves the accuracy of sign language recognition. The wrist optical sensor and the inertial measurement unit (IMU) are used to collect data in a coordinated manner. The optical features and inertial features are accurately synchronized in time through timestamp alignment. A time sequence input with a unified dimension is constructed and standardized. The input is then fed into a unified time sequence coding network to complete the multimodal information fusion modeling, which effectively makes up for the limitations of data collected by a single sensor, thereby improving the accuracy of sign language classification and recognition.
[0021] (4) The present invention has outstanding real-time performance and rapid response. By comparing the energy index of the continuous stream data with the preset background threshold, the sign language action is quickly triggered and segmented, and a fixed-length action segment is accurately extracted and a model input window is constructed. Combined with a lightweight temporal feature encoder, the model inference time and the total system response time are effectively shortened. At the same time, the relevant time parameters are recorded by logging, which facilitates optimization and adjustment. It can meet the needs of real-time sign language recognition and is suitable for real-time scenarios such as daily communication. Attached Figure Description
[0022] This manual includes the following figures, which illustrate the following:
[0023] Figure 1 This is a flowchart of a sign language recognition method and system according to the present invention;
[0024] Figure 2 This is a block diagram illustrating the logical structure of a sign language recognition method and system according to the present invention.
[0025] Figure 2The modules are labeled as follows: 1. Wrist optical acquisition module; 2. Inertial measurement module; 3. Preprocessing and synchronization module; 4. Self-supervised pre-training module; 5. Few-label fine-tuning module; 6. Real-time inference module. Detailed Implementation
[0026] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, in order to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention, and to facilitate its implementation.
[0027] This invention provides a sign language recognition method, comprising the following steps:
[0028] Step 1: Acquire multi-channel time-series data collected by wrist optical sensors and inertial measurement units, and construct motion segment samples;
[0029] Step 2: Perform self-supervised pre-training based on action segment samples to obtain a temporal feature encoder for sign language recognition;
[0030] Step 3: Label a small number of action segment samples, load the pre-trained temporal feature encoder onto the small number of labeled action segment samples and perform classification training to obtain a sign language classification model;
[0031] Step 4: Based on the continuous stream data collected by the wrist optical sensor and inertial measurement unit, the action trigger segmentation and window construction are performed, and the sign language classification model is used for online reasoning to output the sign language category result.
[0032] In step one, the data collected by the wrist optical sensor and the data collected by the inertial measurement unit are aligned using timestamps.
[0033] In step two, the self-supervised pre-training employs a dual-network structure comprising an online network and a target network. At least two augmented views are generated for the same action segment, and their representations are trained. The parameters of the target network are updated using an exponential moving average. Augmented views include at least one of amplitude scaling, noise perturbation, temporal occlusion, or random temporal clipping. The temporal feature encoder comprises a local temporal feature extraction module and a long-range dependency modeling module. The local temporal feature extraction module uses a one-dimensional convolutional network, and the long-range dependency modeling module uses a Transformer encoding network.
[0034] In step three, classification training uses either the linear evaluation mode of the frozen encoder or the end-to-end fine-tuning mode of the unfrozen encoder.
[0035] In step four, the action triggering segment specifically involves calculating the energy index for the continuous stream data and comparing it with the background threshold.
[0036] This invention provides a system for applying a sign language recognition method, comprising a wrist optical acquisition module, an inertial measurement module, a preprocessing and synchronization module, a self-supervised pre-training module, a minimally labeled fine-tuning module, and a real-time inference module. The wrist optical acquisition module uses a wrist optical sensor to acquire image information of the user's wrist region. The inertial measurement module uses an inertial measurement unit to acquire the inertial features of the user's wrist movement. The preprocessing and synchronization module standardizes and synchronizes the data acquired by the wrist optical acquisition module and the inertial measurement module. The self-supervised pre-training module trains a temporal feature encoder using the information acquired by the wrist optical acquisition module and the inertial measurement module. The minimally labeled fine-tuning module labels a small number of action segment samples and performs classification training to obtain a sign language classification model. The real-time inference module acquires the data input from the wrist optical acquisition module and the inertial measurement module in real time, performs online inference, and outputs the sign language category result.
[0037] The present invention provides a detailed description of a sign language recognition method and system.
[0038] The computer acquires multi-channel time-series data collected by wrist optical sensors and inertial measurement units, and constructs motion segment samples. The optical sensors are used to acquire image information of the wrist region and extract statistical features of multiple regions of interest (ROIs). The inertial measurement unit (IMU) is used to acquire triaxial acceleration and triaxial angular velocity data, thereby obtaining inertial features that reflect the wrist motion state.
[0039] A feature encoder for sign language recognition is obtained through self-supervised pre-training based on action segment samples. Specifically, this invention first uses the BYOL self-supervised learning framework to pre-train unlabeled action segment samples. Fixed-length action segment samples are segmented from continuously acquired multi-channel temporal data, and a first enhanced view and a second enhanced view are generated for the same action segment. The enhancement methods include at least one of random temporal cropping, amplitude scaling, Gaussian noise perturbation, and temporal occlusion. Then, an online network and a target network are constructed. The online network includes a temporal feature encoder, a projection head, and a prediction head, while the target network includes a temporal feature encoder and a projection head. The temporal feature encoder is composed of a one-dimensional convolutional network and a Transformer encoding network. The online network is trained by minimizing the distance loss between the online network's predicted representation and the target network's output representation, and the target network parameters are updated using an exponential moving average method, thereby obtaining a pre-trained temporal feature encoder for downstream tasks. Specifically, the online network includes a temporal feature encoder, a projection head, and a prediction head, while the target network includes a temporal feature encoder and a projection head. The first and second augmented views, obtained by augmenting the same action segment using two different methods, are input into the online network and the target network, respectively, to obtain the predicted representation of the online network. and and the output representation of the target network and The preferred pre-training loss is symmetric distance loss.
[0040]
[0041] Among them, the normalized cosine distance is preferred for the single-term distance loss:
[0042]
[0043] During training, the target network serves as a reference branch and is updated directly without gradient backpropagation. Its parameters are updated from the online network parameters using an exponential moving average, with the update formula as follows:
[0044]
[0045] in, Indicates online network parameters, Indicates the target network parameters. The momentum coefficient is preferably between 0.99 and 0.999, and more preferably, it is gradually increased from 0.99 to 0.996 during training. Through this method, the online network learns temporal feature representations that are robust to amplitude perturbations, noise perturbations, and temporal occlusions without the need for manual annotation. Furthermore, the temporal feature encoder parameters in the online network are retained after pre-training for downstream classification training with fewer annotations.
[0046] A small number of action segment samples are labeled, and a pre-trained temporal feature encoder is loaded onto these labeled action segment samples for classification training to obtain a sign language classification model. Specifically, a small number of training samples are selected from the segmented labeled action segment samples using stratified sampling, based on the principle of consistent category proportions, to maintain a balanced distribution of samples across categories. The small number of labeled samples is preferably 10% to 50% of the total labeled samples, more preferably 10%, 30%, or 50%. In one embodiment, when the total number of labeled samples is 300, the small number of labeled samples can be 30, 90, or 150, respectively. Subsequently, the temporal feature encoder parameters obtained in the supervised pre-training stage are loaded, a classification head is constructed after the temporal feature encoder, and training is performed using an end-to-end fine-tuning method, i.e., the encoder parameters are unfrozen and updated together with the classification head, thereby obtaining the sign language classification model.
[0047] The system executes action triggering segmentation and window construction based on continuous stream data collected by wrist optical sensors and inertial measurement units, and uses a sign language classification model for online inference to output sign language category results. Specifically, after model training, real-time continuous stream data is collected using a contactless wristwatch for online recognition. This real-time recognition process includes: continuously collecting multi-channel data from wrist optical sensors and inertial measurement units; detrending the continuous stream data to obtain residual signals reflecting action changes; calculating the energy index of the residual signal within a fixed-length sliding window, and comparing the energy index with a preset background threshold to detect the start of a sign language action. The energy index is preferably the sum of squares or the average of the sum of squares of the signals within the window. Let the window length be... Window class The residual signal at each sampling point is The current window energy It can be represented as:
[0048]
[0049] in Preferably, 5 to 20 frames are selected, more preferably 10 frames. The preset background threshold is preferably obtained adaptively based on the continuous stream data during the static background phase. Specifically, background data of a preset length can be collected during the motionless background phase, divided into multiple windows, the background energy of each window can be calculated, and then the average value can be taken to obtain the background energy benchmark. Preset background threshold It can be represented as:
[0050]
[0051] in The threshold amplification factor is preferably 5 to 50, more preferably 20 to 40, and in one embodiment, it is 30. This factor is applied to the current energy over several consecutive windows. Greater than the threshold When a sign language action is detected, a fixed-length action segment is extracted and a model input window is constructed. The constructed data is then input into the trained sign language classification model for online inference, outputting the corresponding sign language category and its confidence level. At the same time, the single model inference time and the total system response time from action triggering to recognition output are recorded, and the relevant information is written to a log file.
[0052] This invention employs a time-aligned feature stitching method for multimodal fusion. Specifically, the optical features acquired by the wrist optical sensor and the IMU features acquired by the inertial measurement unit are first aligned in the time dimension. Then, a time-series input with a unified dimension is constructed, and the samples are standardized. Finally, the input is fed into a unified time-series coding network for modeling to achieve the fusion representation of multimodal information.
[0053] The beneficial effects of the present invention are described in detail below.
[0054] This invention reduces data annotation costs and enhances the practicality of the method. It employs a combination of self-supervised pre-training and minimal annotation fine-tuning. Using the BYOL self-supervised learning framework, it pre-trains on a large number of unlabeled action segment samples, achieving a high-performance temporal feature encoder without manual annotation. Subsequent annotation of only a small number of action segment samples is sufficient for classification training, significantly reducing the workload and cost of manual annotation. This solves the problems of high dependence on large-scale labeled data and high annotation costs in traditional sign language recognition models.
[0055] This invention reduces model inference latency while maintaining recognition accuracy by combining a lightweight encoding network that integrates local temporal feature extraction and long-range dependency modeling. The temporal feature encoder integrates a local temporal feature extraction module (one-dimensional convolutional network) and a long-range dependency modeling module (Transformer encoding network), which can accurately extract local detailed features of actions and effectively capture long-range temporal dependencies of actions, thereby enhancing the model's adaptability to different action amplitudes and environmental interference.
[0056] This invention achieves precise multimodal data fusion, improving the accuracy of sign language recognition. It employs a wrist-worn optical sensor and an inertial measurement unit (IMU) to collaboratively acquire data. Time stamp alignment enables precise temporal synchronization of optical and inertial features, constructing a unified-dimensional temporal input and standardizing it. This input is then fed into a unified temporal coding network to complete multimodal information fusion modeling, effectively overcoming the limitations of data acquired from a single sensor and thus improving the accuracy of sign language classification and recognition.
[0057] This invention excels in real-time performance and rapid response. By comparing the energy index of continuous streaming data with a preset background threshold, it achieves rapid triggering and segmentation of sign language actions, accurately extracts fixed-length action segments, and constructs a model input window. Combined with a lightweight temporal feature encoder, it effectively shortens the model inference time and the total system response time. At the same time, by logging relevant time parameters, it facilitates optimization and adjustment, meeting the needs of real-time sign language recognition and adapting to immediate scenarios such as daily communication.
[0058] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution; or the direct application of the inventive concept and technical solution to other situations without modification, are all within the protection scope of the present invention.
Claims
1. A sign language recognition method, characterized in that, Includes the following steps: Step 1: Acquire multi-channel time-series data collected by wrist optical sensors and inertial measurement units, and construct motion segment samples; Step 2: Perform self-supervised pre-training based on action segment samples to obtain a temporal feature encoder for sign language recognition; Step 3: Label a small number of action segment samples, load the pre-trained temporal feature encoder onto the small number of labeled action segment samples and perform classification training to obtain a sign language classification model; Step 4: Based on the continuous stream data collected by the wrist optical sensor and inertial measurement unit, the action trigger segmentation and window construction are performed, and the sign language classification model is used for online reasoning to output the sign language category result.
2. The sign language recognition method as described in claim 1, characterized in that: In step one, the data collected by the wrist optical sensor and the data collected by the inertial measurement unit are aligned using timestamps.
3. The sign language recognition method as described in claim 1, characterized in that: In step two, the self-supervised pre-training adopts a dual-network structure that includes an online network and a target network. It generates at least two augmented views for the same action segment and trains their representations.
4. The sign language recognition method as described in claim 3, characterized in that: The parameters of the target network are updated using an exponential moving average.
5. The sign language recognition method as described in claim 3, characterized in that: Enhanced views include at least one of amplitude scaling, noise perturbation, temporal occlusion, or random temporal clipping.
6. The sign language recognition method as described in claim 1, characterized in that: The temporal feature encoder includes a local temporal feature extraction module and a long-range dependency modeling module.
7. The sign language recognition method as described in claim 6, characterized in that: The local temporal feature extraction module uses a one-dimensional convolutional network, while the long-range dependency modeling module uses a Transformer coding network.
8. The sign language recognition method as described in claim 1, characterized in that: In step three, classification training uses either the linear evaluation mode of the frozen encoder or the end-to-end fine-tuning mode of the unfrozen encoder.
9. The sign language recognition method as described in claim 1, characterized in that: In step four, the action triggering segment specifically involves calculating the energy index for the continuous stream data and comparing it with the background threshold.
10. A system for applying the sign language recognition method as described in any one of claims 1-9, characterized in that: It includes a wrist optical acquisition module, an inertial measurement module, a preprocessing and synchronization module, a self-supervised pre-training module, a minimally labeled fine-tuning module, and a real-time inference module; The wrist optical acquisition module uses a wrist optical sensor to acquire image information of the user's wrist area; The inertial measurement module uses an inertial measurement unit to collect the inertial characteristics of the user's wrist movement. The preprocessing and synchronization module is used to standardize and synchronize the data acquired by the wrist optical acquisition module and the inertial measurement module. The self-supervised pre-training module is used to train the temporal feature encoder based on information collected by the wrist optical acquisition module and the inertial measurement module; The minimal annotation fine-tuning module is used to annotate a small number of action fragment samples and perform classification training to obtain a sign language classification model; The real-time inference module is used to collect data from the wrist optical acquisition module and the inertial measurement module in real time and perform online inference to output sign language category results.