Continuous sign language recognition method and system based on hierarchical attention feature fusion

Abstract

This invention proposes a continuous sign language recognition method and system based on hierarchical attention feature fusion. The method involves processing sign language videos, outputting multi-level, multi-scale feature maps through a visual encoder. These maps are then processed by a cross-scale aggregation module containing a hierarchical attention network before being fed into a classifier for word classification. The CTC decoder outputs a word sequence. Simultaneously, a two-layer fully connected network performs binary classification prediction to determine whether each frame is a boundary frame for a sign language word. An online self-distillation mechanism based on exponential moving average (EMA) is established, using an exponential moving average version of the model parameters as the EMA teacher model. The student model continuously learns to obtain a more stable word sequence output. A multi-level regularization framework is constructed, coordinating optimization at four levels: label space, temporal space, feature space, and parameter space. This invention effectively improves the model's generalization ability under small sample conditions, thereby reducing the word error rate (WER) in continuous sign language recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a continuous sign language recognition method and system based on hierarchical attention feature fusion. Background Technology

[0002] Continuous Sign Language Recognition (CSLR) aims to automatically convert continuous sign language video sequences into corresponding word sequences, and is an important research direction at the intersection of computer vision and natural language processing. In existing technologies, mainstream CSLR methods employ an end-to-end framework of "visual encoder + temporal modeling + CTC decoding." Specifically, a convolutional neural network (such as ResNet) is first used to extract visual features frame-by-frame from the input sign language video; then, a temporal modeling of the frame-level features is performed using a temporal convolutional network or a bidirectional long short-term memory network (BiLSTM) to capture the contextual dependencies of sign language actions; finally, a weakly supervised training is performed using the connection-temporal classification (CTC) loss function, achieving end-to-end sequence learning without frame-level annotation. However, existing methods typically only use single-scale features from the last layer of the visual encoder, failing to fully utilize the multi-scale intermediate features at each stage of the network, resulting in insufficient fusion of hand details and global pose information.

[0003] To improve the recognition performance and robustness of models, researchers have attempted to introduce knowledge distillation techniques into continuous sign language recognition tasks. A typical approach is to construct a teacher-student framework, pre-training a large-capacity teacher model and then transferring its knowledge to a lightweight student model. Another approach is to design a two-branch network structure, allowing the two branches to learn from each other through distillation, thereby improving the model's generalization ability through consistency constraints. However, these methods either require pre-training independent teacher models, increasing training costs and procedural complexity, or employ a two-branch structure, increasing the number of model parameters and inference overhead. Furthermore, existing distillation methods fail to effectively utilize the temporal evolution information of model parameters during training (such as exponential moving averages), making it difficult to obtain stable and high-quality distilled signals.

[0004] In summary, current continuous sign language recognition methods generally have the following shortcomings: 1. Lack of Adaptability in Multi-Scale Visual Feature Fusion: Sign language involves multi-scale spatiotemporal information such as hand details, arm movements, and facial expressions. Existing methods, when fusing multi-scale features from different stages of convolutional networks, typically use fixed weight concatenation or simple summation, failing to dynamically adjust the contribution of features at each level according to the characteristics of different samples. This results in the model being unable to focus on shallow detail features for fine-grained gestures, nor on deep semantic features for large-amplitude limb movements, leading to the easy loss of key sign language action information.

[0005] 2. Insufficient Temporal Boundary Awareness: Continuous sign language videos contain the consecutive expressions of multiple sign language words, with blurred and difficult-to-precise temporal boundaries between words. Existing methods lack explicit modeling of the temporal boundaries of sign language words; the models rely solely on the implicit alignment mechanism of CTC loss, failing to actively perceive and predict action segmentation positions. This leads to over-insertion or over-deletion errors when processing long sequences, affecting the final recognition accuracy.

[0006] 3. Lack of efficient online self-distillation mechanisms: Existing knowledge distillation methods require pre-training of independent teacher models or the use of dual-branch network structures, resulting in complex training processes and high computational costs. These methods fail to utilize the exponential moving average (EMA) version of the same model during training as a teacher, making it difficult to achieve efficient online self-distillation. EMA models can provide stable prediction signals with smooth temporal data and are a natural high-quality teacher, but existing CSLR methods have not fully explored their distillation potential.

[0007] 4. Lack of systematic regularization strategies: Public datasets for continuous sign language recognition are limited in size (e.g., PHOENIX-2014 has only about 6000 training samples), making deep networks prone to overfitting during training. While existing methods have introduced some regularization techniques, they are mostly applied piecemeal, lacking a systematic regularization framework tailored to the characteristics of the CSLR task. Cooperative optimization strategies for techniques such as label smoothing, temporal occlusion enhancement, and spectral enhancement have not been fully studied, and the generalization ability of the models still has significant room for improvement. Summary of the Invention

[0008] This invention proposes a continuous sign language recognition method and system based on hierarchical attention feature fusion. It can adaptively fuse multi-scale visual features, explicitly model the temporal boundaries of sign language words, achieve efficient online self-distillation using the EMA model, and effectively improve the generalization ability of the model under small sample conditions through a systematic multi-level regularization framework, thereby reducing the word error rate (WER) of continuous sign language recognition.

[0009] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A continuous sign language recognition method based on hierarchical attention feature fusion includes: S1. The sign language video outputs multi-scale feature maps of multiple levels through a visual encoder. It is then spatially pooled and fully connected mapped through a cross-scale aggregation module containing a hierarchical attention network. Local temporal modeling is then performed through a temporal convolutional network. The temporal feature sequence is obtained through a bidirectional LSTM and fed into a classifier for word classification. The word sequence is then output by the CTC decoder. S2. The temporal feature sequence is fed into the classifier and then subjected to binary classification prediction through a two-layer fully connected network to determine whether each frame is a boundary frame of sign language words. S3. Establish an online self-distillation mechanism based on exponential moving average (EMA). Use the exponential moving average version of the model parameters during the training process in steps S1 and S2 as the EMA teacher model. The student model continuously learns from the time-smooth EMA teacher model to obtain a more stable word sequence output. S4. Construct a multi-level regularization framework to collaboratively optimize steps S1-S3 from four levels: label space, temporal space, feature space, and parameter space.

[0010] Furthermore, the cross-scale aggregation module in step S1 includes: S101. Perform global average pooling along the time dimension on the temporal features of multiple levels in the multi-scale feature map to obtain the global description vector of each level. S102. Input the global description vector into a shared attention network, which consists of two fully connected layers, compressing the feature dimension from C to C / 4 in sequence, and then mapping it to a scalar attention score. S103. Perform Softmax normalization on the attention score to obtain the weight coefficients of each level; S104. Use weighting coefficients to weight and scale the temporal features of each level to achieve adaptive fusion.

[0011] Furthermore, step S2 includes: S201. Receive the time-series feature sequence with dimensions (T, B, C), where T is the time-series length, B is the batch size, and C is the feature dimension. S202, The first fully connected network maps the feature dimension from C to C / 2, followed by the ReLU activation function and Dropout regularization; S203, the second fully connected network maps the feature dimension from C / 2 to 2, and outputs the probability distribution of each frame belonging to the boundary / non-boundary.

[0012] Furthermore, the online self-distillation mechanism based on the exponential moving average (EMA) in step S3 includes: S301. Parameter Maintenance: During training, a set of shadow parameters is maintained for all trainable parameters of the model; after each training step, the shadow parameters are updated according to the following formula: ; in, For the shadow parameters updated in step t, The shadow parameters are updated at step t-1; These are the current model parameters. The attenuation coefficient; S302, Teacher Prediction Generation: In each training batch, the EMA shadow parameters are temporarily applied to the model, and a gradient-free forward propagation is performed to obtain the output probability distribution of the teacher model. Then, the original model parameters are restored. S303, KL divergence distillation loss: The KL divergence between the output probability distributions of the student model and the EMA teacher model is calculated as the distillation loss; a temperature parameter is introduced to soften the probability distribution. : ; in For KL divergence distillation loss, and These are the output logits of the student model and the teacher model, respectively; KL represents the KL divergence calculation. S304. Joint optimization of losses: The KL divergence distillation loss is weighted and summed with the CTC loss from step S1 and the boundary prediction loss from step S2, and then joint optimization is performed end-to-end.

[0013] Furthermore, step S4 includes: S401, Label Space Regularization-Label Smoothing CTC Loss: When calculating CTC loss, hard labels are converted into soft labels, and a certain proportion of probability quality is evenly distributed to non-target categories to prevent the model from being overconfident in the training labels. S402, Temporal Spatial Regularization - Temporal Occlusion Enhancement: Randomly occlude several consecutive frames on the temporal feature sequence and fill them with zero values ​​or random noise to force the model to learn a more robust temporal representation; S403, Feature Space Regularization - Spectrum Enhancement: Randomly occlude several continuous channels in the feature channel dimension to simulate feature loss and enhance the model's robustness to feature perturbations; S404, Parameter Space Regularization - EMA and Early Stopping Strategy: EMA is used to smooth the model parameters over time and reduce parameter oscillations; at the same time, an early stopping strategy is adopted to monitor the WER index on the validation set and terminate training early when there is no improvement after several consecutive rounds to prevent overfitting.

[0014] In another aspect, this invention proposes a continuous sign language recognition system based on hierarchical attention feature fusion, comprising: Main task module: The sign language video outputs multi-scale feature maps of multiple levels through a visual encoder, which then performs spatial pooling and fully connected mapping through a cross-scale aggregation module containing a hierarchical attention network. Local temporal modeling is then performed through a temporal convolutional network, and a temporal feature sequence is obtained through a bidirectional LSTM. This sequence is then fed into a classifier for word classification, and the word sequence is output by the CTC decoder. Temporal boundary prediction module: While the temporal feature sequence is fed into the classifier, binary classification prediction is performed through a two-layer fully connected network to determine whether each frame is a boundary frame of sign language words; EMA Self-Distillation Module: An online self-distillation mechanism based on exponential moving average (EMA) is established. The exponential moving average version of the model parameters during the training process of the cross-scale aggregation module and the temporal boundary prediction module is used as the EMA teacher model. The student model continuously learns from the temporally smooth EMA teacher model to obtain a more stable word sequence output. Multi-level regularization framework module: The main task module, temporal boundary prediction module, and EMA self-distillation module are collaboratively optimized from four levels: label space, temporal space, feature space, and parameter space.

[0015] Furthermore, the cross-scale aggregation module in the main task module includes: Average pooling unit: Performs global average pooling along the time dimension on the temporal features of multiple levels in the multi-scale feature map to obtain the global description vector of each level; Attention Unit: The global description vector is input into a shared attention network, which consists of two fully connected layers that compress the feature dimension from C to C / 4 and then map it to a scalar attention score. Normalization unit: Performs Softmax normalization on the attention score to obtain the weight coefficients of each level; Fusion Unit: Weighted scaling of temporal features at each level is achieved using weight coefficients to realize adaptive fusion.

[0016] Furthermore, the temporal boundary prediction module includes: Receiving unit: Receives the time-series feature sequence with dimensions (T, B, C), where T is the time-series length, B is the batch size, and C is the number of feature dimension channels; First layer mapping unit: The first fully connected network maps the feature dimension from C to C / 2, followed by ReLU activation function and Dropout regularization; The second layer mapping unit: The second layer of the fully connected network maps the feature dimension from C / 2 to 2, and outputs the probability distribution of each frame belonging to the boundary / non-boundary.

[0017] Furthermore, the EMA self-distillation module includes: Parameter Maintenance Unit: During training, a set of shadow parameters is maintained for all trainable parameters of the model; after each training step, the shadow parameters are updated according to the following formula: ;in, For the shadow parameters updated in step t, The shadow parameters are updated at step t-1; These are the current model parameters. The attenuation coefficient; Teacher prediction generation unit: In each training batch, the EMA shadow parameters are first temporarily applied to the model, a gradient-free forward propagation is performed to obtain the output probability distribution of the teacher model, and then the original model parameters are restored. KL divergence distillation loss unit: Calculates the KL divergence between the output probability distributions of the student model and the EMA teacher model as the distillation loss; a temperature parameter is introduced to soften the probability distribution. : ;in For KL divergence distillation loss, and These are the output logits of the student model and the teacher model, respectively; KL represents the KL divergence calculation. Joint Loss Optimization Unit: The KL divergence distillation loss is weighted and summed with the CTC loss of the main task module and the boundary prediction loss of the time series boundary prediction module, and then jointly optimized end-to-end.

[0018] Furthermore, the multi-level regularization framework module includes: Label space regularization unit: When calculating CTC loss, hard labels are converted into soft labels, and a certain proportion of probability quality is evenly distributed to non-target categories to prevent the model from being overconfident in the training labels. Temporal spatial regularization unit: Randomly occludes several consecutive frames on the temporal feature sequence and fills them with zero values ​​or random noise, forcing the model to learn a more robust temporal representation; Feature space regularization unit: randomly occludes several continuous channels in the feature channel dimension to simulate feature loss and enhance the model's robustness to feature perturbations; Parameter space regularization unit: The model parameters are smoothed over time using EMA to reduce parameter oscillations; at the same time, an early stopping strategy is adopted to monitor the WER index on the validation set. When the model no longer improves after several consecutive rounds, the training is terminated in advance to prevent overfitting.

[0019] Compared to traditional continuous sign language recognition methods, which have limitations in handling multi-scale temporal feature fusion and training stability, especially prone to overfitting and inaccurate word boundary segmentation on small datasets, this invention proposes a continuous sign language recognition method and system based on hierarchical attention feature fusion. This method employs a continuous sign language recognition framework, introduces a cross-scale aggregation module, and combines dynamic hierarchical attention mechanisms, boundary prediction assistance tasks, and online self-distillation of EMA techniques. Through these innovative improvements, this invention significantly enhances the model's recognition accuracy in complex sign language scenarios, while maintaining high recognition accuracy while preserving computational efficiency. The main technical effects of this invention include: 1. Cross-Scale Aggregation Module (CSA): CSA fully leverages the complementarity of features extracted from different depths of convolutional neural networks. Shallow networks retain rich spatial details and edge textures, suitable for capturing fine-grained information such as finger shape changes and palm orientation; deep networks encode more abstract semantic information, helping to understand global movement patterns such as arm movement trajectories and body postures. Traditional methods often only use the last layer of features or simple concatenation, making it difficult to balance the relationship between details and semantics. Compared to traditional single-layer features, the CSA module performs spatial pooling and channel mapping on the features of the three layers separately, unifying the feature dimensions before feeding them into a shared temporal modeling network. Subsequently, global average pooling is used to obtain the global representation of the features of each layer as input to the hierarchical attention network. The hierarchical attention network dynamically calculates the contribution weights of each layer through a two-layer fully connected structure and Softmax normalization. Finally, the learned weights are used to perform a weighted summation of the temporal features of each layer to obtain the fused multi-scale feature representation. This design enhances the model's adaptability to different types of sign language movements: for words with fine gestures, shallow detail features receive higher weights; for words with large-amplitude body movements, the contribution of deep semantic features increases. Experimental results show that the CSA module can improve WER by approximately 0.6% compared to single-level features.

[0020] 2. Dynamic Hierarchical Attention Mechanism: The hierarchical attention mechanism is a core component of the CSA module, inspired by the successful application of attention mechanisms in multi-task learning. Unlike fixed-weight feature fusion, the hierarchical attention mechanism dynamically allocates attention resources based on the specific characteristics of the input sequence. Regarding sample adaptability, fixed-weight fusion applies the same fusion ratio to all samples, failing to distinguish the different feature requirements of different types of sign language movements. The hierarchical attention mechanism, by learning the weight distribution related to the input, enables the model to make personalized fusion decisions for each sample. In terms of long-range dependency modeling, by introducing global average pooling to obtain global representations for each level, the hierarchical attention network can determine the fusion strategy based on the statistical characteristics of the entire sequence. This is particularly important for word recognition, which requires understanding the context for disambiguation. Regarding adaptability to complex backgrounds, when the recording environment is complex, the hierarchical attention mechanism automatically reduces the weights of disturbed levels, effectively suppressing the impact of background noise on the recognition results.

[0021] 3. Boundary Prediction Auxiliary Task: The alignment process of the CTC loss function is implicit; the model does not directly learn the boundary positions between words, leading to over-insertion (incorrectly segmenting a word into multiple parts) and over-deletion (incorrectly merging multiple adjacent words). This invention designs a boundary prediction module, constructing a binary classification prediction head based on frame-level features to determine whether each frame is near a word boundary. Boundary label generation uses a simplified rule: consecutive frames with the same label are marked as boundary frames, and the rest are marked as non-boundary frames. The boundary prediction loss is used as an auxiliary loss and jointly optimized with the CTC main loss, with a weight coefficient set to 0.3. During the inference phase, the boundary prediction results do not directly participate in decoding, but the training process has already injected boundary awareness into the shared feature representation. Compared to pure CTC training, this explicit boundary modeling method forces the model to pay attention to the temporal segmentation points between words while learning sequence recognition, thereby strengthening the model's temporal boundary awareness. Experiments show that after introducing boundary prediction, both over-insertion and over-deletion errors are significantly reduced.

[0022] 4. Online Self-Distillation Mechanism of EMA: Knowledge distillation is an effective means to improve model performance, but traditional offline distillation requires pre-training a convergent teacher model, increasing training complexity and time cost. This invention uses the Exponential Moving Average (EMA) technique to achieve online self-distillation, cleverly solving this problem. The EMA teacher is automatically maintained by the moving average of the student model parameters, without separate training. The decay coefficient is set to 0.999, meaning that new parameters contribute only 0.1% of the weight, ensuring the smoothness of the teacher parameters. The KL divergence distillation loss weight is set to 0.5. Compared with traditional distillation methods, this mechanism significantly reduces training costs. Simultaneously, the EMA parameters integrate historical information from multiple training steps, making their predictions more stable than instantaneous parameters, and can serve as more reliable soft labels to guide student learning. As training progresses, the quality of the EMA teacher gradually improves, and the student model can continuously benefit from the improved teacher, forming a virtuous cycle. By minimizing the KL divergence between the student and EMA teacher output distributions, the model is encouraged to produce consistent predictions on similar inputs, achieving progressive knowledge transfer and effectively improving the model's generalization ability.

[0023] 5. Multi-level Regularization Framework: In the field of sign language recognition, publicly available datasets are relatively small (e.g., PHOENIX-2014 contains only about 6000 samples), making model overfitting a prominent issue. This invention constructs a multi-level regularization framework covering the entire training process, co-constraining the model from four different spaces. In the label space, label smoothing technology is used to soften hard labels into a smooth distribution, avoiding overconfidence in the model based on training samples; the smoothing coefficient is set to 0.1. In the temporal space, a temporal occlusion strategy is used to randomly occlude consecutive frames of the input sequence during training, forcing the model to learn to use contextual information for prediction, enhancing robustness to temporal gaps. In the feature space, borrowing from the SpecAugment technique in speech recognition, random occlusion is performed along the feature dimension to prevent the model from over-relying on specific feature channels. In the parameter space, the EMA mechanism itself has a parameter smoothing regularization effect, combined with an early stopping strategy based on validation set performance, to terminate training in time before the model begins to overfit. Compared to a single regularization method, these four levels of regularization complement each other and work together at different stages of the training process, forming a complete anti-overfitting system. Under limited data conditions, it effectively suppresses overfitting and significantly enhances the model's generalization performance on unseen samples.

[0024] 6. Overall Performance Improvement: By comprehensively applying the above technical solutions, the word error rate (WER) of this invention on the PHOENIX-2014 dataset test set decreased from 17.5% of the baseline model to 17.33%. Considering that the performance of existing methods on this dataset is already quite saturated, this improvement has significant practical implications. More importantly, the various technical innovations of this invention have good scalability and versatility: the CSA module and hierarchical attention mechanism can be easily embedded into other visual backbone networks; the EMA online self-distillation mechanism has no special requirements for network structure and is suitable for various sequence modeling tasks; the components of the multi-level regularization framework can be flexibly combined and configured according to specific scenarios. This invention provides a complete continuous sign language recognition solution, which effectively improves recognition accuracy and generalization ability through multiple targeted technical improvements while maintaining model simplicity and efficiency. Attached Figure Description

[0025] Figure 1 This is an overall technical framework diagram of Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the cross-scale aggregation module CSA and the hierarchical attention mechanism in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the online self-distillation mechanism of EMA in Embodiment 4 of the present invention; Figure 4 This is a schematic diagram of the multi-level regularization framework of Embodiment 5 of the present invention. Detailed Implementation

[0026] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0027] First, the technical terms involved in the present invention will be described.

[0028] CSLR: Continuous Sign Language Recognition, which refers to the task of automatically converting a continuous sign language video sequence into a corresponding vocabulary sequence.

[0029] CTC: Connectionist Temporal Classification, a loss function for sequence-to-sequence learning that does not require frame-level alignment annotation.

[0030] ResNet: Residual Network, a deep convolutional neural network that solves the problem of difficult training of deep networks through residual connections.

[0031] BiLSTM: Bidirectional Long Short-Term Memory, a recurrent neural network that can capture bidirectional temporal dependencies.

[0032] EMA: Exponential Moving Average, which performs temporal smoothing on model parameters to obtain a more stable version of the parameters.

[0033] KL divergence: Kullback-Leibler divergence, which is used to measure the difference between two probability distributions and is often used in knowledge distillation.

[0034] WER Word Error Rate: The main evaluation metric for continuous sign language recognition. The lower it is, the higher the recognition accuracy.

[0035] CSA: Cross-Scale Aggregation.

[0036] TCN: Temporal Convolutional Network.

[0037] Over-insertion: In CTC (Connectionist Temporal Classification) decoding, over-insertion refers to the situation where the model incorrectly splits a vocabulary into multiple vocabularies. For example: The true label is "hello", but the model outputs "you hello hello", repeating the recognition of "hello" twice. Reason: The implicit alignment mechanism of CTC lacks clear boundary constraints, resulting in repeated output on adjacent similar frames.

[0038] Over-deletion: Opposite to over-insertion, over-deletion refers to the model incorrectly merging multiple adjacent words into one word. For example, the true label is "How are you", but the model outputs "How are", missing "you". Reason: The model has insufficient discrimination ability at word boundaries and misjudges consecutive different words as a continuation of the same word.

[0039] Shadow Parameters: In the EMA (Exponential Moving Average) mechanism, shadow parameters are a smoothed copy of the model parameters.

[0040] Based on the above description, the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0041] Embodiment 1: This embodiment proposes the overall technical framework of a continuous sign language recognition method based on hierarchical attention feature fusion, as Figure 1 shown, including the following processing procedures: Input the sign language video and send it to the ResNet visual encoder for frame-by-frame feature extraction. The ResNet visual encoder outputs feature maps at multiple levels at different depth stages. In this embodiment, it is set to three levels of feature maps, namely Stage2 feature map, Stage3 feature map, and Stage4 feature map.

[0042] The feature maps at the three levels are sent to the Cross-Scale Aggregation Module (CSA) for adaptive fusion. The CSA contains a hierarchical attention network inside, which is used to dynamically calculate the contribution weights of each level. The fused features are subjected to local temporal modeling through the Temporal Convolutional Network (TCN), and then the global temporal dependencies are captured through a bidirectional LSTM to generate a global temporal feature sequence.

[0043] The output of the bidirectional LSTM is sent to the boundary prediction head for the auxiliary boundary prediction task on the one hand and to the classifier for word classification on the other hand. Finally, the CTC decoder outputs the word sequence. In the training stage, the EMA module maintains an exponentially weighted moving average version of the model parameters, and the output of the classifier is constrained for consistency through the KL divergence distillation loss.

[0044] This embodiment introduces the cross-scale aggregation module and combines technologies such as the dynamic hierarchical attention mechanism, the boundary prediction auxiliary task, and EMA online self-distillation, significantly improving the recognition accuracy of the model in complex sign language scenarios, while maintaining computational efficiency and ensuring high recognition accuracy.

[0045] Embodiment 2: This embodiment mainly elaborates on the implementation of the cross-scale aggregation module and the hierarchical attention mechanism described in Embodiment 1.

[0046] As Figure 2 As shown, the detailed structure of the cross-scale aggregation module is as follows: The Stage 2, Stage 3, and Stage 4 feature maps are processed in three parallel branches. Each branch first maps the feature maps of different sizes to a 512-dimensional feature vector through a spatial pooling convolutional mapping layer. Specifically, the Stage 2 feature map has a size of (B, 64, T, 56, 56), which is mapped to 512 dimensions after 4×4 average pooling and 2×2 convolution; the Stage 3 feature map has a size of (B, 128, T, 28, 28), which is mapped to 512 dimensions after 4×4 convolution; and the Stage 4 feature map has a size of (B, 256, T, 14, 14), which is mapped to 512 dimensions after 2×2 convolution.

[0047] The features of the three branches are temporally modeled using a Temporal Convolutional Network (TCN). The TCN employs a two-layer one-dimensional convolutional structure with a kernel size of 5 and 1024 output channels. Subsequently, each branch is pooled along the temporal dimension using a global average pooling layer to obtain global descriptor vectors for each layer. These three global descriptor vectors are then fed into a shared hierarchical attention network, which consists of two fully connected layers: the first layer compresses the 1024 dimensions to 256 dimensions and activates it using ReLU; the second layer maps the 256 dimensions to a one-dimensional scalar score. The attention scores of the three layers are normalized using a Softmax normalization layer, and the resulting weight coefficients are then used to weight and fuse the temporal features of each layer, ultimately outputting a fused multi-scale feature representation.

[0048] This embodiment enhances the model's adaptability to different types of sign language movements: for words with fine gestures, shallow detail features receive higher weights; for words with large-amplitude limb movements, the contribution of deep semantic features increases. Experimental results show that the CSA module can improve WER by approximately 0.6% compared to single-level features. The hierarchical attention mechanism, by learning the weight distribution related to the input, enables the model to make personalized fusion decisions for each sample. Regarding long-range dependency modeling, by introducing global average pooling to obtain the global representation of each level, the hierarchical attention network can determine the fusion strategy based on the statistical characteristics of the entire sequence. In terms of adaptability to complex backgrounds, when the recording environment is complex, the hierarchical attention mechanism can automatically reduce the weights of disturbed levels, effectively suppressing the impact of background noise on the recognition results.

[0049] Example 3: This embodiment mainly provides a detailed explanation of the implementation of boundary prediction described in Embodiment 1.

[0050] The boundary prediction head receives frame-level features from the bidirectional LSTM output and performs binary classification prediction through a two-layer fully connected network. Specific parameter configurations are as follows: The input dimension is the output dimension of the bidirectional LSTM (1024 dimensions); the first fully connected network layer maps the 1024 dimensions to 512 dimensions, followed by a ReLU activation function and a Dropout regularization with a probability of 0.2; the second fully connected network layer maps the 512 dimensions to 2 dimensions and outputs the probability distribution of each frame belonging to the boundary / non-boundary.

[0051] The generation of boundary pseudo-labels adopts a simplified rule: based on the CTC alignment results, the start and end frames of the time segment corresponding to each word are marked as boundary frames (label value 1), and the remaining frames are marked as non-boundary frames (label value 0). The padding position is marked as -1 and ignored when calculating the loss. The boundary prediction loss adopts the cross-entropy loss function, with a loss weight set to 0.3, and is jointly optimized with the main task CTC loss.

[0052] Boundary prediction results are not directly involved in decoding, but their training process has already injected boundary awareness into the shared feature representation. Compared to pure CTC training, this explicit boundary modeling approach forces the model to pay attention to temporal segmentation points between words while learning sequence recognition, thereby strengthening the model's temporal boundary awareness. Experiments show that after introducing boundary prediction, both over-insertion and over-deletion errors are significantly reduced.

[0053] Example 4: This embodiment mainly provides a detailed explanation of the implementation of the EMA online self-distillation mechanism in Embodiment 1.

[0054] like Figure 3 As shown, the EMA online self-distillation mechanism employs a student-teacher dual-branch structure. The input sign language video is fed into both the student and teacher models simultaneously. The student model comprises a ResNet visual encoder, a cross-scale aggregation module (CSA), and a bidirectional LSTM. Its parameters are updated via gradient descent in each training step before being output. The teacher model has the same network structure as the student model, but its parameters are maintained by the EMA module using an exponential moving average.

[0055] The parameter update formula for the EMA module is: ,in For the shadow parameters updated in step t, The shadow parameters are updated at step t-1; The attenuation coefficient α is set to 0.999 for the current model parameters.

[0056] Consistency between student model output and teacher model output is constrained using KL divergence distillation loss, and a temperature parameter is introduced. : ; in For KL divergence distillation loss, and The output logits of the student model and teacher model are respectively; KL represents the KL divergence calculation; temperature parameter. The value is set to 8.0, and the loss weight is set to 0.5. In each training batch, the EMA shadow parameters are first temporarily applied to the teacher model for gradient-free forward propagation to obtain soft labels, and then the student model parameters are restored. Knowledge distillation is achieved by minimizing the KL divergence.

[0057] The online self-distillation mechanism of EMA significantly reduces training costs. Furthermore, EMA parameters integrate historical information from multiple training steps, resulting in more stable predictions compared to instantaneous parameters, making them a more reliable soft label for guiding student learning. As training progresses, the quality of the EMA teacher model gradually improves, allowing the student model to continuously benefit from the improved teacher, creating a virtuous cycle. By minimizing the KL divergence between the student and EMA teacher output distributions, the model produces consistent predictions on similar inputs, achieving progressive knowledge transfer and effectively enhancing the model's generalization ability.

[0058] Example 5: This embodiment focuses on the overall process in Embodiment 1, and uses a multi-level regularization framework for collaborative constraints.

[0059] like Figure 4 As shown, the multi-level regularization framework employs collaborative constraints from four different levels. The input sign language video, after processing by the main network, outputs a word sequence, subject to the following four levels of regularization during training: Label space regularization: Label smoothing technique is used to convert hard labels into soft labels when calculating CTC loss. The smoothing coefficient is set to 0.1, which evenly distributes part of the probability mass to non-target categories to prevent the model from being overconfident in the training labels.

[0060] Temporal spatial regularization: A temporal occlusion enhancement strategy is adopted, which randomly occludes consecutive frames of the input global temporal feature sequence during training. The occlusion ratio is set to 10%, and the maximum length of the occluded block is 15% of the sequence length, forcing the model to learn to use contextual information for prediction.

[0061] Feature space regularization: Borrowing from the SpecAugment technique, continuous channels are randomly occluded along the feature channel dimension, with the occlusion ratio set to 10% and the maximum width of the occluded block being 15% of the feature dimension, thereby enhancing the model's robustness to feature perturbations.

[0062] Parameter space regularization consists of two components: the EMA module and the early stopping strategy module. The EMA module smooths the model parameters over time to reduce parameter oscillations; the early stopping strategy module monitors the validation set WER metric, with a patience parameter set to 7 epochs. When the validation set performance no longer improves after several consecutive epochs, training is terminated early to prevent overfitting.

[0063] Example 6: This embodiment verifies the results obtained using the above method.

[0064] The results of the comparative experiments on the PHOENIX-2014 dataset are shown in the table below:

[0065] The experimental results show that all the techniques proposed in this invention can effectively reduce the word error rate. When applied together, the WER is reduced by about 2.0 percentage points compared with the traditional baseline model on the test set.

[0066] Example 7: This embodiment proposes a continuous sign language recognition system based on hierarchical attention feature fusion, including: Main task module: The sign language video outputs multi-scale feature maps of multiple levels through a visual encoder, which then performs spatial pooling and fully connected mapping through a cross-scale aggregation module containing a hierarchical attention network. Local temporal modeling is then performed through a temporal convolutional network, and a temporal feature sequence is obtained through a bidirectional LSTM. This sequence is then fed into a classifier for word classification, and the word sequence is output by the CTC decoder. Temporal boundary prediction module: While the temporal feature sequence is fed into the classifier, binary classification prediction is performed through a two-layer fully connected network to determine whether each frame is a boundary frame of sign language words; EMA Self-Distillation Module: An online self-distillation mechanism based on exponential moving average (EMA) is established. The exponential moving average version of the model parameters during the training process of the cross-scale aggregation module and the temporal boundary prediction module is used as the EMA teacher model. The student model continuously learns from the temporally smooth EMA teacher model to obtain a more stable word sequence output. Multi-level regularization framework module: The main task module, temporal boundary prediction module, and EMA self-distillation module are collaboratively optimized from four levels: label space, temporal space, feature space, and parameter space.

[0067] The cross-scale aggregation module in the main task module includes: Average pooling unit: Performs global average pooling along the time dimension on the temporal features of multiple levels in the multi-scale feature map to obtain the global description vector of each level; Attention Unit: The global description vector is input into a shared attention network, which consists of two fully connected layers that compress the feature dimension from C to C / 4 and then map it to a scalar attention score. Normalization unit: Performs Softmax normalization on the attention score to obtain the weight coefficients of each level; Fusion Unit: Weighted scaling of temporal features at each level is achieved using weight coefficients to realize adaptive fusion.

[0068] The time series boundary prediction module includes: Receiving unit: Receives the time-series feature sequence with dimensions (T, B, C), where T is the time-series length, B is the batch size, and C is the number of feature dimension channels; First layer mapping unit: The first fully connected network maps the feature dimension from C to C / 2, followed by ReLU activation function and Dropout regularization; The second layer mapping unit: The second layer of the fully connected network maps the feature dimension from C / 2 to 2, and outputs the probability distribution of each frame belonging to the boundary / non-boundary.

[0069] The EMA self-distillation module includes: Parameter Maintenance Unit: During training, a set of shadow parameters is maintained for all trainable parameters of the model; after each training step, the shadow parameters are updated according to the following formula: ;in, For the shadow parameters updated in step t, The shadow parameters are updated at step t-1; These are the current model parameters. The attenuation coefficient; Teacher prediction generation unit: In each training batch, the EMA shadow parameters are first temporarily applied to the model, a gradient-free forward propagation is performed to obtain the output probability distribution of the teacher model, and then the original model parameters are restored. KL divergence distillation loss unit: Calculates the KL divergence between the output probability distributions of the student model and the EMA teacher model as the distillation loss; a temperature parameter is introduced to soften the probability distribution. : ;in For KL divergence distillation loss, and These are the output logits of the student model and the teacher model, respectively; KL represents the KL divergence calculation. Joint Loss Optimization Unit: The KL divergence distillation loss is weighted and summed with the CTC loss of the main task module and the boundary prediction loss of the time series boundary prediction module, and then jointly optimized end-to-end.

[0070] The multi-level regularization framework module includes: Label space regularization unit: When calculating CTC loss, hard labels are converted into soft labels, and a certain proportion of probability quality is evenly distributed to non-target categories to prevent the model from being overconfident in the training labels. Temporal spatial regularization unit: Randomly occludes several consecutive frames on the temporal feature sequence and fills them with zero values ​​or random noise, forcing the model to learn a more robust temporal representation; Feature space regularization unit: randomly occludes several continuous channels in the feature channel dimension to simulate feature loss and enhance the model's robustness to feature perturbations; Parameter space regularization unit: The model parameters are smoothed over time using EMA to reduce parameter oscillations; at the same time, an early stopping strategy is adopted to monitor the WER index on the validation set. When the model no longer improves after several consecutive rounds, the training is terminated in advance to prevent overfitting.

[0071] The continuous sign language recognition system based on hierarchical attention feature fusion proposed in this embodiment can achieve the continuous sign language recognition method based on hierarchical attention feature fusion described in Embodiments 1-5, and has the same technical effect.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A continuous sign language recognition method based on hierarchical attention feature fusion, characterized in that, include: S1. The sign language video outputs multi-scale feature maps of multiple levels through a visual encoder. It is then spatially pooled and fully connected mapped through a cross-scale aggregation module containing a hierarchical attention network. Local temporal modeling is then performed through a temporal convolutional network. The temporal feature sequence is obtained through a bidirectional LSTM and fed into a classifier for word classification. The word sequence is then output by the CTC decoder. S2. The temporal feature sequence is fed into the classifier and then subjected to binary classification prediction through a two-layer fully connected network to determine whether each frame is a boundary frame of sign language words. S3. Establish an online self-distillation mechanism based on exponential moving average (EMA). Use the exponential moving average version of the model parameters during the training process in steps S1 and S2 as the EMA teacher model. The student model continuously learns from the time-smooth EMA teacher model to obtain a more stable word sequence output. S4. Construct a multi-level regularization framework to collaboratively optimize steps S1-S3 from four levels: label space, temporal space, feature space, and parameter space.

2. The continuous sign language recognition method based on hierarchical attention feature fusion according to claim 1, characterized in that, The cross-scale aggregation module in step S1 includes: S101. Perform global average pooling along the time dimension on the temporal features of multiple levels in the multi-scale feature map to obtain the global description vector of each level. S102. Input the global description vector into a shared attention network, which consists of two fully connected layers, compressing the feature dimension from C to C / 4 in sequence, and then mapping it to a scalar attention score. S103. Perform Softmax normalization on the attention score to obtain the weight coefficients of each level; S104. Use weighting coefficients to weight and scale the temporal features of each level to achieve adaptive fusion.

3. The continuous sign language recognition method based on hierarchical attention feature fusion according to claim 1, characterized in that, Step S2 includes: S201. Receive the time-series feature sequence with dimensions (T, B, C), where T is the time-series length, B is the batch size, and C is the feature dimension. S202, The first fully connected network maps the feature dimension from C to C / 2, followed by the ReLU activation function and Dropout regularization; S203, the second fully connected network maps the feature dimension from C / 2 to 2, and outputs the probability distribution of each frame belonging to the boundary / non-boundary.

4. The continuous sign language recognition method based on hierarchical attention feature fusion according to claim 1, characterized in that, The online self-distillation mechanism based on the exponential moving average (EMA) in step S3 includes: S301. Parameter Maintenance: During training, a set of shadow parameters is maintained for all trainable parameters of the model; after each training step, the shadow parameters are updated according to the following formula: ; in, For the shadow parameters updated in step t, The shadow parameters are updated at step t-1; These are the current model parameters. The attenuation coefficient; S302, Teacher Prediction Generation: In each training batch, the EMA shadow parameters are temporarily applied to the model, and a gradient-free forward propagation is performed to obtain the output probability distribution of the teacher model. Then, the original model parameters are restored. S303, KL divergence distillation loss: The KL divergence between the output probability distributions of the student model and the EMA teacher model is calculated as the distillation loss; a temperature parameter is introduced to soften the probability distribution. : ； in For KL divergence distillation loss, and These are the output logits of the student model and the teacher model, respectively; KL represents the KL divergence calculation. S304. Joint optimization of losses: The KL divergence distillation loss is weighted and summed with the CTC loss from step S1 and the boundary prediction loss from step S2, and then joint optimization is performed end-to-end.

5. The continuous sign language recognition method based on hierarchical attention feature fusion according to claim 1, characterized in that, Step S4 includes: S401, Label Space Regularization-Label Smoothing CTC Loss: When calculating CTC loss, hard labels are converted into soft labels, and a certain proportion of probability quality is evenly distributed to non-target categories to prevent the model from being overconfident in the training labels. S402, Temporal Spatial Regularization - Temporal Occlusion Enhancement: Randomly occlude several consecutive frames on the temporal feature sequence and fill them with zero values ​​or random noise to force the model to learn a more robust temporal representation; S403, Feature Space Regularization - Spectrum Enhancement: Randomly occlude several continuous channels in the feature channel dimension to simulate feature loss and enhance the model's robustness to feature perturbations; S404, Parameter Space Regularization - EMA and Early Stopping Strategy: EMA is used to smooth the model parameters over time and reduce parameter oscillations; at the same time, an early stopping strategy is adopted to monitor the WER index on the validation set and terminate training early when there is no improvement after several consecutive rounds to prevent overfitting.

6. A continuous sign language recognition system based on hierarchical attention feature fusion, characterized in that, include: Main task module: The sign language video outputs multi-scale feature maps of multiple levels through a visual encoder, which then performs spatial pooling and fully connected mapping through a cross-scale aggregation module containing a hierarchical attention network. Local temporal modeling is then performed through a temporal convolutional network, and a temporal feature sequence is obtained through a bidirectional LSTM. This sequence is then fed into a classifier for word classification, and the word sequence is output by the CTC decoder. Temporal boundary prediction module: While the temporal feature sequence is fed into the classifier, binary classification prediction is performed through a two-layer fully connected network to determine whether each frame is a boundary frame of sign language words; EMA Self-Distillation Module: An online self-distillation mechanism based on exponential moving average (EMA) is established. The exponential moving average version of the model parameters during the training process of the cross-scale aggregation module and the temporal boundary prediction module is used as the EMA teacher model. The student model continuously learns from the temporally smooth EMA teacher model to obtain a more stable word sequence output. Multi-level regularization framework module: The main task module, temporal boundary prediction module, and EMA self-distillation module are collaboratively optimized from four levels: label space, temporal space, feature space, and parameter space.

7. The continuous sign language recognition system based on hierarchical attention feature fusion according to claim 6, characterized in that, The cross-scale aggregation module in the main task module includes: Average pooling unit: Performs global average pooling along the time dimension on the temporal features of multiple levels in the multi-scale feature map to obtain the global description vector of each level; Attention Unit: The global description vector is input into a shared attention network, which consists of two fully connected layers that compress the feature dimension from C to C / 4 and then map it to a scalar attention score. Normalization unit: Performs Softmax normalization on the attention score to obtain the weight coefficients of each level; Fusion Unit: Weighted scaling of temporal features at each level is achieved using weight coefficients to realize adaptive fusion.

8. The continuous sign language recognition system based on hierarchical attention feature fusion according to claim 6, characterized in that, The time series boundary prediction module includes: Receiving unit: Receives the time-series feature sequence with dimensions (T, B, C), where T is the time-series length, B is the batch size, and C is the number of feature dimension channels; First layer mapping unit: The first fully connected network maps the feature dimension from C to C / 2, followed by ReLU activation function and Dropout regularization; The second layer mapping unit: The second layer of the fully connected network maps the feature dimension from C / 2 to 2, and outputs the probability distribution of each frame belonging to the boundary / non-boundary.

9. The continuous sign language recognition system based on hierarchical attention feature fusion according to claim 6, characterized in that, The EMA self-distillation module includes: Parameter Maintenance Unit: During training, a set of shadow parameters is maintained for all trainable parameters of the model; after each training step, the shadow parameters are updated according to the following formula: ;in, For the shadow parameters updated in step t, The shadow parameters are updated at step t-1; These are the current model parameters. The attenuation coefficient; Teacher prediction generation unit: In each training batch, the EMA shadow parameters are first temporarily applied to the model, and a gradient-free forward propagation is performed to obtain the output probability distribution of the teacher model. Then the original model parameters are restored. KL divergence distillation loss unit: Calculates the KL divergence between the output probability distributions of the student model and the EMA teacher model as the distillation loss; a temperature parameter is introduced to soften the probability distribution. : ;in For KL divergence distillation loss, and These are the output logits of the student model and the teacher model, respectively; KL represents the KL divergence calculation. Joint Loss Optimization Unit: The KL divergence distillation loss is weighted and summed with the CTC loss of the main task module and the boundary prediction loss of the time series boundary prediction module, and then jointly optimized end-to-end.

10. The continuous sign language recognition system based on hierarchical attention feature fusion according to claim 6, characterized in that, The multi-level regularization framework module includes: Label space regularization unit: When calculating CTC loss, hard labels are converted into soft labels, and a certain proportion of probability quality is evenly distributed to non-target categories to prevent the model from being overconfident in the training labels. Temporal spatial regularization unit: Randomly occludes several consecutive frames on the temporal feature sequence and fills them with zero values ​​or random noise, forcing the model to learn a more robust temporal representation; Feature space regularization unit: randomly occludes several continuous channels in the feature channel dimension to simulate feature loss and enhance the model's robustness to feature perturbations; Parameter space regularization unit: The model parameters are smoothed over time using EMA to reduce parameter oscillations; at the same time, an early stopping strategy is adopted to monitor the WER index on the validation set. When the model no longer improves after several consecutive rounds, the training is terminated in advance to prevent overfitting.

Images