Wearable sign language translation method based on multi-scale time sequence compression and large language model

Abstract

The invention discloses a wearable sign language translation method based on multi-scale time sequence compression and a large language model, belongs to the technical field of sign language translation, and solves the problems that an existing wearable sign language translation method is unbalanced in inter-modal information density, poor in cross-sentence generalization and excessively depends on sign language word labeling. In order to solve the problem of density imbalance between high-frequency myoelectricity-motion signals and sign language low-frequency semantics, a multi-scale time sequence compression word segmentation device is provided, the signals are smoothly compressed into myoelectricity-motion tokens with high semantic density, and key semantics are reserved while the calculation complexity is reduced. And a big language is introduced into the field, and the syntactic understanding and generalization ability of the model in an open scene is improved by utilizing powerful text priori knowledge of the big language. Meanwhile, a two-stage training strategy without sign language word labeling is provided, the problem of LLM knowledge degradation caused by signal noise is solved, and end-to-end training without sign language word supervision is achieved.

Description

Technical Field

[0001] This invention relates to the field of sign language translation technology, specifically to a wearable sign language translation method based on multi-scale temporal compression and a large language model. Background Technology

[0002] Sign language translation technology is mainly divided into two categories: visual solutions and wearable device solutions. Visual solutions, based on computer vision technology, capture sign language movements through a camera and use deep learning models for translation. While this type of method is constantly improving in performance, it is limited by issues such as lighting conditions, visual occlusion, and user privacy. Furthermore, it requires the user to always be within the camera's field of view, making seamless daily interaction difficult. In contrast, wearable device solutions collect bioelectrical and motion signals by wearing sensors (such as sEMG armbands and IMU watches), offering advantages such as portability and privacy security. The most mature solution is the fusion of surface electromyography (sEMG) signals with motion signals (IMU).

[0003] While sEMG-IMU-based approaches have made some progress, they still have shortcomings in areas such as signal semanticization, model generalization, and reliance on data annotation. First, existing methods suffer from an information density imbalance across the semantic span between signal features and natural language. Common sEMG sampling rates are 200Hz, while the semantic change rate of sign language gestures is only about 2.4Hz. Second, existing methods fail to effectively utilize the prior knowledge of pre-trained models, resulting in insufficient generalization ability in cross-sentence scenarios. Finally, existing methods largely rely on intermediate layer sign language word (Gloss) annotations for CTC training, requiring sign language experts to meticulously annotate each sign language sample, leading to an over-reliance on expert data annotations.

[0004] In summary, existing wearable sign language translation methods suffer from problems such as an imbalance in information density between modalities, poor cross-sentence generalization, and over-reliance on sign language word annotation. Summary of the Invention

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a wearable sign language translation method based on multi-scale temporal compression and a large language model. This method solves the problems of intermodal information density imbalance, poor cross-sentence generalization, and over-reliance on sign language word annotation in existing wearable sign language translation methods.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: A wearable sign language translation method based on multi-scale temporal compression and a large language model is provided, including the following steps: S1. Acquire sEMG and IMU signals in real time through wearable devices, and preprocess the sEMG and IMU signals; S2. Extract features from the preprocessed sEMG signal and IMU signal to obtain sEMG features and IMU features respectively. Use the IMU features to generate a gated attention mask, and weight the sEMG features by using the gated attention mask. Then, concatenate and fuse the weighted sEMG features with the IMU features to obtain multimodal fusion features. S3. Input the multi-modal fusion features into the pre-trained multi-scale temporal compression word segmenter; the multi-scale temporal compression word segmenter includes a multi-scale temporal compression module and a cross-modal bridge. The multi-scale temporal compression module outputs sign language action features with different temporal receptive fields through multiple parallel convolutional branches, each with different convolutional kernel sizes and dilation rates. The sign language action features output by all parallel convolutional branches are fused and temporally downsampled to obtain multi-scale temporal compressed features. The multi-scale temporal compressed features are input into the cross-modal bridge to obtain electromyography-motion tokens. S4. Project the electromyography-motion tokens into the input embedding space of the pre-trained large language model to obtain the semantic embedding sequence. Then, translate and decode the semantic embedding sequence through the large language model to generate natural language text.

[0007] Furthermore, the method for preprocessing the sEMG signal is as follows: input the sEMG signal into the spatial calibration module for spatial calibration, and then resample the calibrated sEMG signal at the frequency.

[0008] Furthermore, the weighted expression for sEMG features is as follows: ; ; ; in, and These are the preprocessed sEMG signal and IMU signal, respectively; Conv For a one-dimensional convolution with a kernel size of 3 and padding of 1; For batch normalization; Use the GELU activation function; and These are sEMG features and IMU features, respectively. For gating attention mask; For element-wise multiplication; Weighted sEMG features; This is a transpose.

[0009] Furthermore, the expression for the sign language action features output by the network layer with parallel convolutional branches is as follows: in, For the first In the nth parallel convolution branch, the th The sign language gesture features output by the layer; Use the GELU activation function; For batch normalization; The kernel size and dilation rate are respectively and Convolutional layers; For multimodal fusion features in the multi-scale temporal compression module Input features of the layer.

[0010] Furthermore, the expression for electromyographic-motor tokens is: ; ; ; in, and These are multi-scale temporal compression features and electromyography-motor tokens, respectively. and These are self-attention enhancement features and convolutional enhancement features, respectively. Input features for multi-head self-attention mechanism; It is a feedforward network; This is a multi-head self-attention mechanism; For layer normalization; This is a convolutional module.

[0011] Furthermore, the expression for generating natural language text is: in, For a given and input Under the condition that the next token is The probability of; For the first The historical word sequence already generated by the large language model at each time step; For large language models in the first The next token predicted at each time step; It is a semantic embedding sequence; It is a normalized exponential function; and These are the output projection matrix and the output projection bias, respectively. For large language models in the first The top-level hidden state at each time step; To give input Generate natural language text sequences under certain conditions The joint probability; For multiplication; For time step The total number.

[0012] Furthermore, the multi-scale temporal compression module and the large language model are pre-trained using a two-stage progressive training strategy, which includes: SA freezes all training parameters in the large language model and trains the multi-scale temporal compression module for a preset number of rounds based on the real labels of natural language text through a loss function. SB unfreezes all training parameters in the large language model and, based on the real labels of the natural language text, trains the multi-scale temporal compression module and all training parameters in the large language model for a preset number of rounds by minimizing the loss function.

[0013] Furthermore, the expression for the loss function is: in, The loss function; These are the training parameters from step SA or step SB.

[0014] Compared with the prior art, the present invention has the following significant advantages: 1. First, addressing the density imbalance between high-frequency electromyographic-motor signals and low-frequency semantics in sign language, a multi-scale temporal compression word segmenter is proposed. This smoothly compresses the signals into high-semantic-density electromyographic-motor tokens, reducing computational complexity while preserving key semantics. Second, for the first time, a large language pre-model is introduced into this field, leveraging its powerful textual prior knowledge to improve the model's syntactic understanding and generalization capabilities in open scenarios. Furthermore, the entire approach constitutes an end-to-end training framework. By directly using the final natural language text as the learning target, it completely eliminates the reliance on the expensive and expert-annotated intermediate step of sign language words (Gloss). This significantly reduces the cost and barrier of data annotation and avoids performance bottlenecks caused by Gloss annotation errors or information loss. It solves the problems of intermodal information density imbalance, poor cross-sentence generalization, and over-reliance on sign language word annotation in existing wearable sign language translation methods.

[0015] 2. By using a learnable spatial calibration matrix and instance normalization, the inconsistent signal distribution of sEMG signals caused by minor sensor placement shifts and differences in skin conductivity among different users is effectively eliminated. Frequency resampling provides the convenience of integer-fold downsampling for subsequent multi-scale temporal compression, avoiding feature loss. Compared to linear interpolation, this method can more effectively suppress aliasing and preserve the high-frequency spectral characteristics of bioelectrical signals to the greatest extent.

[0016] 3. By utilizing relatively stable IMU motion features to generate gating masks for sEMG features, it is possible to dynamically enhance electromyographic features related to the current hand movement, while suppressing motion artifacts and noise caused by sensor friction, muscle fatigue, etc., thus achieving more robust and accurate multimodal feature fusion.

[0017] 4. By using convolutional layers with different kernels and dilation rates in parallel, the model can simultaneously capture features at different time scales in sign language movements—from short-range, subtle finger movements to long-range, macroscopic arm movements. This multi-scale perception capability ensures that the compressed features can fully preserve the rich temporal information of sign language, providing information assurance for accurate translation.

[0018] 5. The Conformer structure of the cross-modal bridge combines the global dependency modeling capability of multi-head self-attention (MHSA) with the local feature extraction capability of convolutional modules (ConvModule). It can effectively capture the long-range temporal dependencies required for sign language grammar and make up for the shortcomings of pure attention mechanism in capturing local details, generating higher quality feature representations that are aligned with the semantic space of the language model.

[0019] 6. By leveraging the generative capabilities of large language models, the signal features processed at the front end are smoothly translated into natural language text, avoiding the strong constraints on sequence alignment and Gloss dependence of traditional CTC training.

[0020] 7. Due to the significant differences in feature distribution between sEMG signals, IMU signals, and natural language, directly performing end-to-end joint training between a randomly initialized MTCT encoder and a pre-trained large language model can easily lead to the destruction of pre-trained weights by initial noise gradients, resulting in knowledge degradation. Therefore, a progressive two-stage training approach is proposed to achieve cross-modal translation in two stages without sign language word supervision. Stage 1 freezes the LLM, forcing the MTCT to map the feature distribution learned from sEMG-IMU signals to the semantic space of the large language model, thus completing the initial word segmentation operation without destroying the pre-trained weights. Stage 2 freezes the LLM for global fine-tuning to adapt to the unique grammatical structure of sign language and further improve the feature extraction capability of the MTCT. Attached Figure Description

[0021] Figure 1 This is a structural block diagram of a wearable sign language translation method based on multi-scale temporal compression and a large language model.

[0022] Figure 2 This is a flowchart of the attention gating mechanism.

[0023] Figure 3 This is a block diagram of the multi-scale temporal compression module. Detailed Implementation

[0024] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0025] Example 1 Wearable sign language translation technology based on surface electromyography (sEMG) and inertial measurement units (IMUs) holds great potential in eliminating communication barriers for the hearing impaired due to its portable and seamless interaction. However, existing methods still suffer from problems such as intermodal information density imbalance, poor cross-sentence generalization, and over-reliance on sign language gloss annotation. Therefore, this embodiment provides a wearable sign language translation method based on multi-scale temporal compression and a large language model, defining the sEMG-IMU sign language translation task as a process that compiles multimodal time series data. To discrete text The probability mapping problem. This represents the 8-channel sEMG signal with a 200Hz sampling rate collected by the Myo wristband. This indicates an IMU signal containing 3D acceleration, 3D angular velocity, and 4D quaternions; , , and For the 1st, 2nd, Nth, and ith tokens, Given a vocabulary, the ultimate training objective is to maximize the conditional probabilities. ,refer to Figure 1 The specific steps include: S1. Acquire sEMG and IMU signals in real time through wearable devices, and preprocess the sEMG and IMU signals.

[0026] S2. Extract features from the preprocessed sEMG and IMU signals to obtain sEMG features and IMU features respectively. Use the IMU features to generate a gated attention mask, and weight the sEMG features by using the gated attention mask. Then, concatenate and fuse the weighted sEMG features with the IMU features to obtain multimodal fusion features.

[0027] S3. Input the multi-modal fusion features into the pre-trained multi-scale temporal compression word segmenter; the multi-scale temporal compression word segmenter includes a multi-scale temporal compression module and a cross-modal bridge. The multi-scale temporal compression module outputs sign language action features with different temporal receptive fields through multiple parallel convolutional branches, each containing different convolutional kernel sizes and dilation rates. The sign language action features output by all parallel convolutional branches are fused and temporally downsampled to obtain multi-scale temporal compressed features. The multi-scale temporal compressed features are input into the cross-modal bridge to obtain electromyography-motion tokens. S4. Project the electromyography-motion tokens into the input embedding space of the pre-trained large language model to obtain the semantic embedding sequence. Then, translate and decode the semantic embedding sequence through the large language model to generate natural language text.

[0028] For step S1, the method for preprocessing the sEMG signal is as follows: The sEMG signal is input into the spatial calibration module for spatial calibration, and the calibrated sEMG signal is then resampled at the frequency.

[0029] In this embodiment, considering the issues of inconsistent spatial distribution and large relative amplitude differences in the original sEMG signals due to slight shifts in wearing position and dynamic variations in skin conductivity among different subjects, a spatial calibration module is introduced in the preprocessing stage to eliminate these shifts. Unlike the manual calibration in existing technologies, this embodiment uses a learnable rotation matrix initialized as an identity matrix. and bias To automatically correct rotational misalignment caused by sensor wearing, the calibrated signal is further normalized to eliminate amplitude differences between samples. The expression for calibrating the sEMG signal is: in, It is an sEMG signal; and These are the learnable rotation matrix and the bias, respectively; For instance normalization function; The sEMG signal is for calibration.

[0030] Furthermore, since the original sEMG signal has a frequency of 200Hz, multiple binary downsampling operations will result in non-integer frame lengths (e.g., This can lead to feature loss or redundant padding. Therefore, this embodiment uses a band-limited Sinc interpolation algorithm based on a Kaiser window to... Frequency resampling to This frequency has excellent divisibility properties and can support up to 6 binary downsampling operations. Compared to linear interpolation, this method can more effectively suppress aliasing and preserve the high-frequency spectral characteristics of bioelectrical signals to the greatest extent.

[0031] Regarding step S2, in existing technologies for sign language recognition using only a single modality, IMU-based systems generally outperform sEMG systems. This phenomenon is usually due to the characteristics of the two signals: IMU signals can capture the macroscopic movement trajectory of the limbs, have low-frequency and stable distribution characteristics, and are relatively stable; while sEMG signals, although containing a large amount of hand information, are prone to motion artifacts due to friction between the sensor and skin during large-scale limb movements, resulting in noisy features and less stability. Based on this phenomenon, this embodiment proposes a motion-guided lightweight attention gating mechanism, referring to... Figure 2 This mechanism uses the motion features of the IMU to predict a gated attention mask for sEMG features. By detecting specific motion patterns of the IMU, it enhances the corresponding spatiotemporal features of the sEMG, thereby reducing noise caused by non-motion. Specifically, when the IMU detects a specific limb trajectory, the gating unit activates the sEMG features related to the semantics of that motion, thus focusing attention on motion-related features and reducing the influence of irrelevant features. Specifically, step S2 further includes: Let the preprocessed sEMG signal be input as IMU signal input is First, a two-stream feature extractor is constructed to process the two modes separately, projecting the two signals onto a unified high-dimensional feature space. Its expression is: in, and These are sEMG features and IMU features, respectively; Conv This represents a one-dimensional convolution with a kernel size of 3 and padding of 1. For batch normalization; Use the GELU activation function; This is a transpose.

[0032] Then, Input a lightweight attention-gated unit. This unit is based on IMU features. Learn a feature-point level attention weight mask This measures the importance of each sEMG feature point. The expression is: Among them, Conv This represents a one-dimensional convolution with a kernel size of 3 and padding of 1. (Gateway mask) The closer the element value is to 1, the stronger the correlation between the current IMU motion feature pattern and the feature point. Conversely, the closer it is to 0, the weaker the correlation between the feature point and the feature point. The electromyographic signal near this point may be affected by motion artifacts or be in a resting state, thus suppressing this noisy feature.

[0033] Next, the generated gating mask will be... and Perform element-wise multiplication (HadamardProduct) to obtain the weighted sEMG features. , This process uses stable upper limb movements as a priori, and performs gating processing on the electromyographic signals, effectively filtering out electromyographic noise generated by irrelevant movements.

[0034] Finally, and The splicing is performed along the channel dimension, and through a... Convolutional layers perform cross-modal feature fusion to obtain the final multimodal fused features. , .

[0035] For step S3, considering that most semantics of sign language consists of rapid, short-range hand movements and slow, long-range upper limb movements, single-scale convolutional kernels, limited by fixed receptive fields, struggle to extract modal features at different scales while compressing samples. Therefore, this embodiment proposes a Multi-scale Temporal Compression Unit (MTCU). This unit uses convolutional branches with different receptive fields to extract sign language movement features at different scales in parallel, capturing multi-scale features of both short-range details (e.g., finger flexion / extension) and long-range movements (e.g., arm waving). Simultaneously, each MTCU performs a 2x downsampling through a convolution with a stride of 2. These are then stacked to form a Multi-scale Temporal Compression Module (MTCM), as described above. Figure 3 .

[0036] Specifically, in step S3, let the first The input features of the layer MTCU are To construct multi-scale receptive field coverage and achieve feature downsampling, the MTCU's parallel branch set is as follows: There are four branches in total. All branches use step size Perform timing compression.

[0037] The specific functions of each branch are defined as follows: (a) Point-by-point projection branch ( (a) Using only one-dimensional convolution with a kernel size of 1 for cross-channel information interaction and dimensionality reduction, responsible for preserving the instantaneous features and linear mapping relationships of the signal. (b) Local sensing branch (c) Mid-range sensing branch: A one-dimensional convolution with a kernel size of 3 combined with non-linear activation is used to capture local high-frequency details (such as varicose fingers). (d) Long-range dilated branch ( ): A one-dimensional convolution with a kernel size of 5 and padding of 2 is used to further expand the receptive field to extract motion semantics over a medium time span. This branch employs a dilated convolution with a kernel size of 3, padding of 2, and dilation of 2. This branch significantly expands the receptive field without increasing the number of parameters, specifically designed to capture long-range temporal dependencies. The output of each branch... The expression is: in, For the first In the nth parallel convolution branch, the th The sign language gesture features output by the layer; Use the GELU activation function; For batch normalization; The kernel size and dilation rate are respectively and Convolutional layers; For multimodal fusion features in multi-scale temporal compression units Input features of the layer.

[0038] To fuse multi-scale features, the outputs of all branches are concatenated along the channel dimension and then reassembled along the channel dimension using a 1x1 convolution. To mitigate the gradient degradation problem in deep networks, residual connections and residual paths are introduced. Alignment is achieved in the time dimension using 1x1 convolutions with a stride of 2. Finally, the... Layer output Defined as: in, , , and They are respectively , , and The output; Features of splicing; To fuse the weights of the convolutions; This is the GELU activation function.

[0039] In this embodiment, (preferred) Multi-scale timing compression units are stacked to form a multi-scale timing compression module. The multi-scale timing compression module compresses the time resolution of the original signal. This allows for the transformation of long, continuous signals into compact, high-semantic-density features.

[0040] For step S3, although the MTCM module has compressed the long temporal signal into high-information-density features, these features are still extracted by a convolutional network and are subject to distance limitations. This is not ideal for sign language grammar, which requires handling long-distance dependencies. Furthermore, directly feeding the output of the convolutional network into the subsequent LLM would lead to training difficulties due to excessively large feature distribution differences. Therefore, this embodiment adds a cross-modal bridge after the MTCM module to form the final multi-scale temporal compression tokenizer (MTCT) to alleviate the modal differences between signal features and LLM semantic features. The cross-modal bridge is composed of… The Transformer module consists of multiple Conformer layers. Compared to the standard Transformer, the Conformer introduces a convolutional enhancement mechanism, which effectively fuses global interactions and local features. Specifically, each Conformer block consists of two feedforward networks, a multi-head self-attention module, and a convolutional module. This allows for the compression of multi-scale temporal features from the input. Its forward propagation process is defined as follows: ; ; ; in, and These are multi-scale temporal compression features and electromyography-motor tokens, respectively. and These are self-attention enhancement features and convolutional enhancement features, respectively. Input features for multi-head self-attention mechanisms; It is a feedforward network; This is a multi-head self-attention mechanism; For layer normalization; The convolutional module's specific computation process includes: First, normalizing the input features using layer normalization; second, expanding the feature channel dimensions through pointwise convolutional layers and using gated linear units for feature selection; then, extracting local temporal features through one-dimensional deep convolutional layers; next, normalizing the features after deep convolution using group normalization and activating them using the Swish activation function; finally, restoring the feature channel dimensions through pointwise convolutional layers and performing residual connections with the input features for output.

[0041] For the large language model in step S4, considering that existing methods typically train their decoders only on small-scale sign language datasets, lacking a general understanding of language and often relying on costly sign language word annotations as intermediate supervision, resulting in poor generalization in open-ended contexts across sentences, this embodiment selects mT5 (MultilingualT5) as the decoder for converting EMG-IMU tokens to natural language text. This improves the system's understanding of general syntactic structures and deep semantics, leveraging its powerful linguistic priors to enhance the accuracy and generalization of sign language translation. Specifically, after MTCT maps continuous sEMG-IMU signals to the semantic space of mT5, mT5 can utilize its linguistic priors learned through pre-training on the large-scale multilingual corpus mC4 to automatically correct grammatical errors and infer the semantics of unseen sentences. This allows for training even without sign language word annotations, outputting text that conforms to natural language norms.

[0042] This embodiment formalizes the decoding process as an autoregressive generation task conditioned on continuous bioelectrical features. First, a linear adapter is introduced to project the output tokens of MTCT onto the input embedding space of mT5, resulting in a continuous semantic embedding sequence. During the inference phase, the model bases its decisions on the input features. and the historical word sequence generated at the current moment For the next token The prediction is based on the posterior probability distribution, and the prediction expression is: in, For a given and input Under the condition that the next token is The probability of; For the first The historical word sequence already generated by the large language model at each time step; For large language models in the first The next token predicted at each time step; It is a semantic embedding sequence; It is a normalized exponential function; and These are the output projection matrix and the output projection bias, respectively. For large language models in the first The top-level hidden state at each time step.

[0043] The entire target text The generation probability is defined as the product of the conditional probabilities at each step, and its expression is: in, To give input Generate natural language text sequences under certain conditions The joint probability; For multiplication; For time step The total number.

[0044] To efficiently search for the global optimum in a huge hypothesis space, a beam search strategy is adopted in the decoding phase. This involves maintaining a top-valued matrix. A number of candidate paths are used to balance the accuracy and diversity of the generated paths.

[0045] Model training employs end-to-end optimization using Maximum Likelihood Estimation (MLE). This embodiment defines the objective function as the standard Cross-Entropy Loss, aiming to minimize the Negative Log-Likelihood (NLL) of the true label sequence. in, The loss function; These are trainable parameters.

[0046] As a further aspect of this embodiment, since the sEMG-IMU signal and natural language differ significantly in feature distribution, directly performing end-to-end joint training of the randomly initialized MTCT encoder and the pre-trained mT5 could easily lead to the initial noise gradients corrupting the pre-trained weights and causing knowledge degradation. Therefore, the multi-scale temporal compression unit and the large language model are pre-trained using a two-stage progressive training strategy, which includes: SA freezes all training parameters in the large language model and trains the multi-scale temporal compression unit for a preset number of rounds based on the real labels of natural language text through a loss function.

[0047] At this point, during the modality alignment stage, MTCT is forced to map the feature distribution learned from the sEMG-IMU signal to the semantic space of mT5, thereby completing the initial word segmentation operation of the signal without destroying the pre-trained weights. The optimization objective of this stage is as follows: in, These are the optimal training parameters for MTCT. For all training parameters of MTCT; Freeze all training parameters in the large language model.

[0048] SB unfreezes all training parameters in the large language model and, based on the real labels of the natural language text, trains the multi-scale temporal compression unit and all training parameters in the large language model for a preset number of rounds by minimizing the loss function.

[0049] After MTCT has acquired preliminary semantic representation capabilities, the mT5 parameters are unfrozen. A full-scale fine-tuning was performed to adapt to the unique grammatical structure of sign language and further improve the feature extraction capabilities of MTCT. The optimization objectives for this stage are as follows: in, This represents the optimal set of trainable parameters in MTCT and large language models. This represents all trainable parameters in a large language model.

[0050] In summary, the beneficial effects of this solution are as follows: 1. The sEMG-IMU sign language translation system emg2sign is proposed. It is the first to introduce pre-trained LLM into sEMG-IMU sign language translation to improve the lack of generalization of existing methods in cross-sentence scenarios.

[0051] 2. A multi-scale temporal compression word segmenter (MTCT) is proposed. This module smoothly compresses high-frequency signals into high semantic density tokens to improve the information density imbalance between sEMG-IMU modality and text modality.

[0052] 3. A two-stage training strategy without sign language word annotation is proposed, which trains the signal segmenter MTCT and the pre-trained model mT5 in stages to improve the knowledge degradation problem of LLM caused by signal noise gradient.

[0053] Example 2 This embodiment is a further limitation based on Embodiment 1. Its purpose is to provide an experiment on a wearable sign language translation method based on multi-scale temporal compression and a large language model. Other parts not mentioned refer to Embodiment 1 or the prior art.

[0054] It should be noted that in this embodiment, the wearable sign language translation method based on multi-scale temporal compression and a large language model is referred to as emg2sign. The dataset used to evaluate the translation performance in this embodiment comes from the sEMG-IMU sign language corpus constructed in the prior art (Wang Q, Zheng Z, Wang Q, et al. Generalizations of wearable device placements and sentences insignia language recognition with transformer-based model[J]. IEEE Transactions on Mobile Computing, 2024, 23(10): 10046-10059.). This corpus was designed under the guidance of sign language experts to cover everyday expressions in real-world scenarios such as banking, healthcare, and education to ensure the authenticity and everyday relevance of the data. However, since only a portion of the dataset of this corpus is publicly available, this embodiment uses the currently publicly available portion, denoted by SL*.

[0055] Statistical analysis of the SL* data revealed that it included 11 subjects and a total of 14,215 samples. Although the sample size is smaller than the complete dataset described in the original paper, it still contained 689 sign language words and 400 sentences. The sentence lengths ranged from 3 to 12 words, with an average sentence length of 6.1 seconds. The sEMG-IMU signal lengths ranged from 800 to 3800 seconds (4 to 19 seconds), with an average length of 1165 seconds (5.8 seconds).

[0056] To evaluate the model's generalization ability in different scenarios, three test sets were set up for different scenarios: (a) Unseen repetition. Due to the non-stationarity and randomness of sEMG signals, the distribution of micropotentials can drift even when the same user repeats the same action due to muscle fatigue or changes in attention. Therefore, this test set was set up to record different repetitions of visible sentences performed by visible subjects to evaluate the model's generalization ability in everyday use scenarios.

[0057] (b) Unseen subject test. Due to significant physiological differences in muscle anatomy, subcutaneous fat thickness, and skin conductivity among individuals, the distribution of sEMG signals for the same sign language gestures varies greatly across different users. Therefore, this test set was configured to record visible repetitions of visible sentences performed by different subjects to evaluate the model's generalization ability in cross-person scenarios.

[0058] (c) Unseensentence Test. Due to the high complexity of natural language, different sentences have different syntactic structures and semantic logic. If the model only identifies the corresponding natural language label based on the signal without learning lexical and syntactic structures, the model's ability to perform across sentences will be greatly reduced. Therefore, this test set is set to unseensent sentences from the test subjects to evaluate the model's generalization ability in cross-sentence scenarios. In this embodiment, the ratio of training set:validation set:test set is 10:1:2.

[0059] To quantitatively evaluate the performance of the sign language translation method in Example 1 on continuous sign language translation tasks and to maintain comparability with existing work, this example uses word error rate (WER) as the core evaluation metric. WER is a common standard for measuring the difference between predicted and true sequences in sequence generation tasks. It is based on edit distance and is defined as follows: in, The total number of sign language words in the ground truth sequence; (Substitution) (Deletion) and (Insertion) represents the minimum number of replacement, deletion, and insertion operations required to convert the model-generated predicted sequence into a true sequence. In the three test subsets mentioned above, a lower WER value indicates a higher translation accuracy of the model.

[0060] In this embodiment, the experiment was implemented based on the PyTorchLightning framework and trained on a single NVIDIA RTX 3090 (24GB) GPU. To improve computational efficiency, BF16 mixed precision was used, with a batch size of 32 and a maximum number of iterations of 30. The optimizer used was AdamW (initial learning rate 1.5e-4), along with a warmup mechanism and gradient pruning strategy to stabilize training.

[0061] In terms of model architecture, the encoder embedding dimension is set to 256, and the decoder loads pre-trained mT5-base weights. To prevent overfitting and improper updates to the pre-trained weights, this embodiment introduces an early stopping mechanism and a phased encoder warm-up strategy. To ensure reproducibility, the global random seed is fixed at 3407.

[0062] Specifically, this paper compares the word error rate (WER) of emg2sign with that of the existing mainstream method easyhear. The experimental results are shown in Table 1. As shown in Table 1, in the (a) Unseen repetition test set, emg2sign achieved a WER of 6.84%, a significant improvement compared to the baseline method easyhear's 16.43%, with an error rate reduction of nearly 10 percentage points. This result indicates that emg2sign can better handle changes in the muscle state of the same subject and sensor wear offset (such as muscle fatigue or sensor friction).

[0063] In the (b) Unseen subject test set, emg2sign achieved a WER of 17.09%, maintaining a low error rate and even outperforming easyhear's WER in the unseen sample dataset. This demonstrates that emg2sign can effectively identify the intrinsic patterns of electromyographic-motor signals, rather than simply memorizing individual physiological characteristics or fixed waveform patterns, thus effectively overcoming the domain shift problem caused by differences in anatomical structures among different users.

[0064] Finally, in the (c) Unseensentence test set, even when faced with sentences that the same user had never seen before, emg2sign still achieved a WER of 24.97%. This result indicates that the model does not merely perform simple phrase-level pattern matching, but truly learns the lexical and syntactic rules of sign language, possessing the generalization ability to perform semantic reasoning in unseen contexts, and completing the generalization from signal recognition to semantic understanding.

[0065] As a further measure in this embodiment, in order to verify the effectiveness of each core module in the emg2sign framework, this embodiment conducted detailed ablation experiments on three test sets, and the results are shown in Table 2.

[0066] As can be seen from Table 2: 1. Importance of Pre-trained Language Model (mT5): In this experiment, an untrained Transformer was used instead of mT5. Removing the mT5 decoder significantly reduced the model's performance on all test sets, with WERs soaring to 47.5%, 54.45%, and 59.55%, respectively. This demonstrates that in sign language translation tasks, introducing a large language model plays a crucial role in understanding semantic logic, correcting grammatical errors, and handling unseen syntactic structures.

[0067] 2. Contribution of Multi-Scale Temporal Compression: In this experiment, the multi-scale compression mechanism was removed, and only a single-scale processing of the signal was performed using a convolutional kernel of 3. Experimental results show that the model performance decreased significantly, with WER degrading by 7.02%, 7.66%, and 9.35% on the three test sets of no-sample, no-subject, and no-sentence (reaching 13.86%, 24.75%, and 34.32%, respectively). This indicates that a single resolution alone cannot simultaneously process short-range hand muscle movements and long-range upper limb movements in sEMG signals, while the multi-scale mechanism plays an important role in integrating local micro-features with global macro-semantics.

[0068] 3. Contribution of Motion-Guided Gating Mechanism: In this experiment, the gating unit used to fuse sEMG and IMU signals was removed, and the two modal features were directly concatenated. Experimental results show that after removing the gating mechanism, the WER of the model on the three test sets increased to varying degrees, at 7.04%, 17.68%, and 25.91%, respectively. This indicates that by introducing motion signals as gating guidance, the model can effectively suppress background noise or motion artifacts during non-action periods, thereby dynamically selecting more discriminative EMG features and enhancing the robustness of the model in complex dynamic scenes.

[0069] 4. Contribution of the Cross-Modal Bridge: In this experiment, the cross-modal bridge module connecting the encoder and decoder was removed, resulting in a significant drop in model performance. The WERs surged to 42.54%, 51.5%, and 58.35%, respectively, rendering the model unusable. Its negative impact was second only to removing the pre-trained model. This result demonstrates that due to the significant distributional differences between the continuous physical signal space of sEMG and the discrete semantic space of LLM, simple connections cannot achieve effective feature mapping. A bridge capable of capturing global long-range dependencies plays a crucial role in achieving modal alignment between signal and text.

[0070] As a further embodiment, a compression ratio ablation experiment was also conducted, and the results are shown in Table 3.

[0071] Table 3 shows that after compression, at a frequency of 24Hz, the model almost fails to converge, with a WER approaching 80%. This is because the excessively long sequence length not only increases the computational burden but also introduces a large amount of non-semantic redundant noise, interfering with the model's temporal modeling. As the frequency decreases (to 24Hz), the model becomes less efficient. 12Hz At 6Hz, the model performance is significantly improved, achieving optimal performance with word error rates of 6.84%, 17.09%, and 24.97% in scenarios without samples, subjects, and sentences, respectively. However, when further compressed to 3Hz, the model performance actually declines. This phenomenon can be explained by the Nyquist sampling theorem: the average output rate of sign language actions is approximately 2.4Hz. According to the theorem, to recover the signal without distortion, the sampling frequency must be greater than twice the highest frequency of the signal, i.e., above 4.8Hz. The sampling rate of 3Hz is below this critical value, leading to undersampling and frequency aliasing, resulting in the loss of crucial motion details and affecting recognition accuracy. In this embodiment, the compression frequency is ultimately set to 6Hz. 6Hz not only satisfies the Nyquist sampling condition, ensuring the integrity of motion information, but also provides a finer-grained temporal resolution than 3Hz, thus balancing the model's recognition performance and real-time performance.

[0072] As a further aspect of this embodiment, in order to demonstrate the impact of different encoder freezing rounds on the final performance of the model, this embodiment also conducted a two-stage training ablation experiment, and the results are shown in Table 4.

[0073] Table 4 illustrates that, since MTCT is randomly initialized while mT5 has pre-trained weights, direct joint training can easily corrupt the decoder's pre-trained knowledge due to gradient noise. Using a phased unfreezing strategy, the model achieved the best results (WER 6.84%) after freezing for 3 rounds. This is because moderate warm-up allows the decoder to adapt to the input embedding distribution first. In contrast, if the freeze time is too short (e.g., 1 round), the encoder has not yet converged and stabilized, causing the WER to rise to 12.89%; if the freeze time is too long (e.g., 5 rounds), it leads to underfitting (WER 10%). This experiment verifies the importance of two-stage training.

Claims

1. A wearable sign language translation method based on multi-scale temporal compression and a large language model, characterized in that, Including the following steps: S1. Acquire sEMG and IMU signals in real time through wearable devices, and preprocess the sEMG and IMU signals; S2. Extract features from the preprocessed sEMG signal and IMU signal to obtain sEMG features and IMU features respectively. Use the IMU features to generate a gated attention mask, and weight the sEMG features by using the gated attention mask. Then, concatenate and fuse the weighted sEMG features with the IMU features to obtain multimodal fusion features. S3. Input the multi-modal fusion features into the pre-trained multi-scale temporal compression word segmenter; the multi-scale temporal compression word segmenter includes a multi-scale temporal compression module and a cross-modal bridge. The multi-scale temporal compression module outputs sign language action features with different temporal receptive fields through multiple parallel convolutional branches, each with different convolutional kernel sizes and dilation rates. The sign language action features output by all parallel convolutional branches are fused and temporally downsampled to obtain multi-scale temporal compressed features. The multi-scale temporal compressed features are input into the cross-modal bridge to obtain electromyography-motion tokens. S4. Project the electromyography-motion tokens into the input embedding space of the pre-trained large language model to obtain a semantic embedding sequence, and translate and decode the semantic embedding sequence through the large language model to generate natural language text.

2. The wearable sign language translation method based on multi-scale temporal compression and a large language model according to claim 1, characterized in that, The method for preprocessing sEMG signals is as follows: input the sEMG signal into the spatial calibration module for spatial calibration, and then resample the calibrated sEMG signal at the frequency.

3. The wearable sign language translation method based on multi-scale temporal compression and a large language model according to claim 1, characterized in that, The expression for weighting sEMG features is: ； ； ； in, and These are the preprocessed sEMG signal and IMU signal, respectively; Conv It is a one-dimensional convolution; For batch normalization; Use the GELU activation function; and These are sEMG features and IMU features, respectively. For gating attention mask; For element-wise multiplication; Weighted sEMG features; This is a transpose.

4. The wearable sign language translation method based on multi-scale temporal compression and a large language model according to claim 1, characterized in that, The multi-scale temporal compression module includes multiple multi-scale temporal compression units. Each multi-scale temporal compression unit includes parallel convolutional branches with different kernel sizes and dilation rates. The expression for the sign language action features output by the network layer of each parallel convolutional branch in each multi-scale temporal compression unit is as follows: in, For the first In the nth parallel convolution branch, the th The sign language gesture features output by the layer; Use the GELU activation function; For batch normalization; The kernel size and dilation rate are respectively and Convolutional layers; For multimodal fusion features in the multi-scale temporal compression module Input features of the layer.

5. The wearable sign language translation method based on multi-scale temporal compression and a large language model according to claim 1, characterized in that, The expression for electromyographic-motor tokens is: ； ； ； in, and These are multi-scale temporal compression features and electromyography-motor tokens, respectively. and These are self-attention enhancement features and convolutional enhancement features, respectively. Input features for multi-head self-attention mechanism; It is a feedforward network; This is a multi-head self-attention mechanism; For layer normalization; This is a convolutional module.

6. The wearable sign language translation method based on multi-scale temporal compression and a large language model according to claim 5, characterized in that, The expression for generating natural language text is: in, For a given and input Under the condition that the next token is The probability of; For the first The historical word sequence already generated by the large language model at each time step; For large language models in the first The next token predicted at each time step; It is a semantic embedding sequence; It is a normalized exponential function; and These are the output projection matrix and the output projection bias, respectively. For large language models in the first The top-level hidden state at each time step; To give input Generate natural language text sequences under certain conditions The joint probability; For multiplication; For time steps The total number.

7. The wearable sign language translation method based on multi-scale temporal compression and a large language model according to claim 6, characterized in that, The multi-scale temporal compression module and the large language model are pre-trained using a two-stage progressive training strategy, which includes: SA freezes all training parameters in the large language model and trains the multi-scale temporal compression module for a preset number of rounds based on the real labels of natural language text through a loss function. SB unfreezes all training parameters in the large language model and, based on the real labels of the natural language text, trains the multi-scale temporal compression module and all training parameters in the large language model for a preset number of rounds by minimizing the loss function.

8. The wearable sign language translation method based on multi-scale temporal compression and a large language model according to claim 7, characterized in that, The expression for the loss function is: in, The loss function; These are the training parameters from step SA or step SB.

Images