Degradation-robust quality-aware adaptive fusion continuous sign language recognition method

Abstract

The present application relates to the technical field of continuous sign language recognition, and particularly provides a quality-aware adaptive fusion continuous sign language recognition method for degradation robustness. The method comprises the following steps: extracting an RGB frame sequence and a skeleton key point sequence; obtaining a video frame sequence of training input; inputting the video frame sequence into an RGB encoding branch and inputting the skeleton key point sequence into a skeleton encoding branch to obtain frame-level features extracted respectively; performing time sequence convolution on the frame-level features to obtain time-dimensionally aligned RGB time sequence features and skeleton time sequence features; using a quality-aware adaptive fusion module QAAF to estimate learnable fusion weights according to the RGB time sequence features and obtaining fused time sequence features; inputting the fused time sequence features into a BiLSTM network and a CTC module to output a sign language vocabulary sequence. The method improves the robustness and accuracy of the continuous sign language recognition system in the actual deployment environment.

Description

Technical Field

[0001] This invention relates to the field of continuous sign language recognition technology, and in particular to a degradation-robust quality-aware adaptive fusion continuous sign language recognition method. Background Technology

[0002] Sign language is the primary means of communication for the deaf community. Continuous Sign Language Recognition (CSLR) aims to identify continuous sign sentences from unsegmented video streams, holding significant research and application value in computer vision and human-computer interaction. In recent years, deep learning-based CSLR methods have made significant progress. Mainstream methods employ CNN feature extraction + temporal modeling + CTC decoding pipeline, achieving good performance under laboratory conditions. However, in real-world deployment scenarios, video quality is frequently affected by the following factors: low resolution (long-distance shooting or low-end acquisition equipment); exposure environment (excessive indoor lighting); sensor noise (electronic noise from low-quality cameras); motion blur (imaging blur caused by rapid movements in sign language); and compression artifacts (lossy compression during video transmission over a network).

[0003] Existing research systematically reveals the severe impact of common image degradation (noise, blur, compression, etc.) on deep neural networks. In the field of CSLR, existing methods mainly rely on the RGB single modality. These degradation conditions directly destroy the quality of RGB features, leading to a sharp decline in recognition performance and severely limiting the practical usability of CSLR systems. In multimodal sign language recognition, skeletal keypoints can be pre-extracted from the original clear frames (e.g., via MediaPipe), and their data is unaffected by RGB video degradation during the inference stage. Therefore, skeletal features are inherently degradation-robust and can serve as a reliable source of information when the quality of RGB features degrades. However, simple fixed-weight fusion (such as...) It cannot adapt to different degrees of degradation. Summary of the Invention

[0004] In view of this, the present invention provides a degradation-robust quality-aware adaptive fusion continuous sign language recognition method to improve the robustness and accuracy of continuous sign language recognition systems in practical deployment environments.

[0005] In a first aspect, the present invention provides a degradation-robust quality-aware adaptive fusion continuous sign language recognition method, the method comprising: Step 1, Data Acquisition and Bimodal Preprocessing: Acquire the raw continuous sign language video and extract the RGB frame sequence in parallel. and the corresponding skeletal key point sequence ; Step 2, Asymmetric Degradation Enhancement (DAT): During the training phase, RGB frame sequences are processed with preset probabilities. Random quality degradation is applied to the RGB frames to obtain the video frame sequence of the training input. ; Step 3, Dual-branch feature encoding: Encode the video frame sequence Input the RGB encoding branch and the skeletal keypoint sequence Input the skeletal encoding branch to obtain the extracted frame-level features. and ; Step 4, Temporal Convolution Modeling and Feature Alignment: [This section likely refers to a step or step, but the context is unclear and requires further information.] and Perform temporal convolutions separately to obtain time-aligned RGB temporal features. With skeletal temporal features ; Step 5, Quality-Aware Adaptive Fusion: Using the Quality-Aware Adaptive Fusion module (QAAF) based on... Estimate the learnable fusion weights and... and Perform convex combination weighted fusion to obtain fused temporal features. ; Step 6, Timing Decoding and Result Output: This involves fusing the timing features. The input temporal features are processed by a bidirectional long short-term memory network (BiLSTM) and a connection to a temporal classification and decoding module (CTC) to output a sign language vocabulary sequence.

[0006] Optionally, step 1 includes: Obtain the raw continuous sign language video and decode it into an RGB frame sequence, denoted as: ; Skeletal keypoint sequence extracted from raw frames and time-aligned , denoted as: .

[0007] Optionally, step 2 includes: During the training phase, random image quality degradation is applied only to RGB frames to obtain a video frame sequence. Skeletal key point sequence Keeping the clear estimate unchanged, video frame sequence The expression is: ; in, Indicates the first Image degradation operators, This represents the probability of applying degradation to RGB frames during the training phase.

[0008] Optionally, step 3 includes: The RGB encoding branch uses ResNet-34 pre-trained on ImageNet to extract frame-level spatial features from video frame sequences. The feature map is obtained after four residual stages. Frame features are obtained through global average pooling. Frame features are arranged chronologically to form a frame-level feature sequence of RGB coding branches. Subsequently, it is fed into the temporal convolutional modeling layer; The skeleton coding branch uses the spatiotemporal graph convolutional network ST-GCN to perform spatiotemporal coding on the keypoint sequence; Graph Structure: Construct an undirected graph over the 75 keypoints provided by MediaPipe, with the adjacency matrix denoted as... ; Normalized adjacency matrix: ; ST-GCN blocks: Each layer contains spatial graph convolutions and temporal convolutions, with residual connections. ; Encoder structure: 9-layer ST-GCN, with channels gradually increasing. Finally, global average pooling is performed on the keypoint dimension. Convolutional mapping to 512 dimensions, frame-level feature sequence of the skeletal coding branch: .

[0009] Optionally, step 4 includes: Time-aligned RGB timing features With skeletal temporal features Their expressions are as follows: ; .

[0010] Optionally, step 5 includes: Image quality degradation will alter the statistical distribution of RGB frame-level / temporal features; under a fixed-capacity MLP, RGB temporal features... Quality estimation input vector As a low-dimensional sufficient statistic, it is used for scalar weights related to network inference and fusion; given the TemporalConv output of the RGB encoding branch. The estimated learnable fusion weights process is as follows: ; ; in, , , As MLP weights, Dropout is applied after the first two ReLU layers. For learnable bias parameters, For the Sigmoid function; The expression for the fused temporal features is: .

[0011] Optionally, step 6 includes: Fusing temporal features The BiLSTM and classification head are input, and then CTC decoding is used to obtain the sign language vocabulary sequence. The expression for the final output sign language vocabulary sequence recognition result is as follows: ; The expression for the auxiliary branch is: ; The corresponding CTC target is written as: .

[0012] Optionally, TemporalConv employs a K5-P2-K5-P2 structure, consisting of two 1D convolutional layers with a kernel of 5 and two max pooling layers with a stride of 2, downsampling the temporal dimension by a factor of 4 while increasing the number of channels from 512 to 1024. ; BiLSTM uses a 2-layer bidirectional LSTM, with a hidden dimension of 512 in each direction and a total output dimension of 1024. ; The RGB encoding branch and the skeleton encoding branch each contain independent TemporalConv and BiLSTM, used to assist the supervision path; the fusion path only passes through QAAF. The classification header will not repeat TemporalConv; it contains a total of 3 BiLSTM groups and 2 TemporalConv groups.

[0013] The technical solution provided by this invention includes data acquisition and bimodal preprocessing: acquiring raw continuous sign language video and extracting RGB frame sequences in parallel. and the corresponding skeletal key point sequence Asymmetric Degradation Enhancement (DAT): During the training phase, RGB frame sequences are augmented with preset probabilities. Random quality degradation is applied to the RGB frames to obtain the video frame sequence of the training input. Dual-branch feature encoding: Encoding video frame sequences... Input the RGB encoding branch and the skeletal keypoint sequence Input the skeletal encoding branch to obtain the extracted frame-level features. and Temporal Convolutional (TCON) Modeling and Feature Alignment: [This section appears to be incomplete and requires further context.] and Perform temporal convolutions separately to obtain time-aligned RGB temporal features. With skeletal temporal features Quality-Aware Adaptive Fusion: Utilizing the Quality-Aware Adaptive Fusion (QAAF) module based on... Estimate the learnable fusion weights and... and Perform convex combination weighted fusion to obtain fused temporal features. Temporal decoding and output: fusing temporal features The input temporal features are processed by a bidirectional long short-term memory network (BiLSTM) and a connected temporal classification and decoding module (CTC) to output a sign language vocabulary sequence. This method improves the robustness and accuracy of the continuous sign language recognition system in real-world deployment environments. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart of a degradation-robust quality-aware adaptive fusion continuous sign language recognition method provided in an embodiment of the present invention; Figure 2 The degradation comparison images provided in the embodiments of the present invention are as follows: (a) is the original image under clear conditions; (b) is the image under mild motion blur degradation; (c) is the image under moderate motion blur degradation; (d) is the image under severe motion blur degradation; (e) is the image under mild Gaussian noise degradation; (f) is the image under moderate Gaussian noise degradation; (g) is the image under severe Gaussian noise degradation; (h) is the image under mild low resolution degradation; (i) is the image under moderate low resolution degradation; (j) is the image under severe low resolution degradation; (k) is the image under mild exposure degradation; (l) is the image under moderate exposure degradation; (m) is the image under severe exposure degradation; (n) is the image under mild JPEG compression degradation; (o) is the image under moderate JPEG compression degradation; and (p) is the image under severe JPEG compression degradation. Figure 3The following are grouped bar charts comparing the word error rate (WER) under different degradation conditions provided in the embodiments of the present invention: (a) is a comparison chart of the present invention and the comparison method under the clear test condition (clean); (b) is a comparison chart of the present invention and the comparison method under the low resolution degradation condition; (c) is a comparison chart of the present invention and the comparison method under the exposure degradation condition; (d) is a comparison chart of the present invention and the comparison method under the Gaussian noise degradation condition; (e) is a comparison chart of the present invention and the comparison method under the motion blur degradation condition; and (f) is a comparison chart of the present invention and the comparison method under the JPEG compression degradation condition. Figure 4 The WER heatmap of word error rate on the CE-CNSL-D multi-condition test set provided in this embodiment of the invention; Figure 5 The WER convergence curve of the verification word set error rate during the training process is provided for the embodiments of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0018] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0019] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0020] Figure 1The flowchart of the degradation-robust quality-aware adaptive fusion continuous sign language recognition method provided in the embodiments of the present invention is as follows: Figure 1 As shown, the method includes: QAAF-CSLR is a two-branch multimodal framework comprising an RGB encoding branch (ResNet-34), a skeleton encoding branch (ST-GCN), a quality-aware adaptive fusion module (QAAF), and asymmetric degradation enhancement (DAT) during training. Overall robustness relies on the synergy of these three components: the skeleton branch provides representations on the inference side that do not degrade with RGB image quality; DAT on the training side ensures the RGB encoder contacts the degradation distribution in the fusion path; and QAAF performs weighted integration of the two TemporalConv features at the fusion point. QAAF operates on the TemporalConv outputs of both branches. , (Time step has been aligned to) The BiLSTM in each branch is only used for auxiliary supervision and knowledge distillation, and is not used as input to QAAF. The main recognition path is QAAF fusion followed by... .

[0021] Step 1, Data Acquisition and Bimodal Preprocessing: Acquire the raw continuous sign language video and extract the RGB frame sequence in parallel. and the corresponding skeletal key point sequence .

[0022] In this embodiment of the invention, step 1 includes: Obtain the raw continuous sign language video and decode it into an RGB frame sequence, denoted as: ; Skeletal keypoint sequence extracted from raw frames and time-aligned (75 key points of MediaPipe) Coordinates and confidence level), denoted as: .

[0023] In this embodiment of the invention, the goal of step 1 is to construct the synchronous input for the subsequent dual-branch network: the RGB encoding branch uses... (During training, the DAT is further enhanced by asymmetric degeneration), and the skeletal coding branch is used. ,in Aligned with labels on the timeline, serving as a source of stable geometric cues in degraded scenarios.

[0024] Step 2, Asymmetric Degradation Enhancement (DAT): During the training phase, RGB frame sequences are processed with preset probabilities. Random quality degradation is applied to the RGB frames to obtain the video frame sequence of the training input. .

[0025] DAT constitutes the core training mechanism of this paper: under a dual-branch structure, random degradation is only applied to RGB, and the skeleton encoding branch always corresponds to clear keypoint estimation, thus minimizing empirical risk and systematically covering both degraded and clean input distributions. Ablation studies show that removing DAT leads to a sharp increase in WER under various degradation conditions, and its marginal effect is greater than replacing QAAF with fixed fusion. QAAF and DAT are sequentially coupled in the algorithm (DAT determines the training distribution, QAAF...). (Spatial fusion completes) The following conclusions are drawn: The main empirical gain in degradation robustness comes from the overall representation learning supported by DAT and skeletal branches, and QAAF provides supplementary fusion degrees of freedom on this basis.

[0026] In this embodiment of the invention, step 2 includes: During the training phase, random image quality degradation is applied only to RGB frames, with a probability of... The video frame sequence is obtained. Skeletal key point sequence Keeping the clear estimate unchanged, video frame sequence The expression is: ; in, Indicates the first Image degradation operators (low resolution, exposure, Gaussian noise, motion blur, JPEG compression, etc.) This represents the probability of applying degradation to RGB frames during the training phase.

[0027] In this embodiment of the invention, step 2 achieves the construction of a training distribution that is perturbed by RGB but maintains a stable skeleton, providing a robust foundation for subsequent fusion learning across degradation.

[0028] Step 3, Dual-branch feature encoding: Encode the video frame sequence Input the RGB encoding branch and the skeletal keypoint sequence Input the skeletal encoding branch to obtain the extracted frame-level features. and .

[0029] In this embodiment of the invention, step 3 includes: The RGB encoding branch (ResNet-34 + pooling + TemporalConv) uses ResNet-34 pre-trained on ImageNet to extract frame-level spatial features from video frame sequences. The feature map is obtained after four residual stages. Frame features are obtained through global average pooling. Frame features are arranged chronologically to form a frame-level feature sequence of RGB coding branches. Subsequently, it is fed into the temporal convolutional modeling layer; The skeleton encoding branch (ST-GCN+TemporalConv) uses a spatial temporal graph convolutional network (ST-GCN) to perform spatiotemporal encoding on the keypoint sequence; Graph Structure: Construct an undirected graph over the 75 keypoints provided by MediaPipe, with the adjacency matrix denoted as... ; Normalized adjacency matrix: ; The key points are divided into four groups according to anatomical regions, as shown in Table 1: Table 1. Division of Key Anatomical Regions ; In addition, cross-regional edges are added to enhance the coordination between the left and right hands and the torso and face: left wrist (node ​​0) - left shoulder (49), right wrist (21) - right shoulder (50), nose (42) - facial contour chain starting point (53).

[0030] ST-GCN blocks: Each layer contains spatial graph convolutions ( Convolution + graph multiplication) and temporal convolution ( Convolution), with residual connections: ; Encoder structure: 9-layer ST-GCN, with channels gradually increasing. Finally, global average pooling is performed on the keypoint dimension. Convolutional mapping to 512 dimensions, frame-level feature sequence of the skeletal coding branch: .

[0031] Step 4, Temporal Convolution Modeling and Feature Alignment: [This section likely refers to a step or step, but the context is unclear and requires further information.] and Perform temporal convolutions separately to obtain time-aligned RGB temporal features. With skeletal temporal features .

[0032] In this embodiment of the invention, step 4 includes: Time-aligned RGB timing features With skeletal temporal features Their expressions are as follows: ; .

[0033] Step 5, Quality-Aware Adaptive Fusion: Using the Quality-Aware Adaptive Fusion module (QAAF) based on... Estimate the learnable fusion weights and... and Perform convex combination weighted fusion to obtain fused temporal features. .

[0034] QAAF is introduced in the fusion layer. The TemporalConv outputs two branches of RGB encoding and two branches of skeletal encoding. , Convex combination is employed. Its design goal is to mitigate the suboptimal nature of fixed weighting in sharp-degraded mixed distributions by implicitly inferring fusion weights related to input quality using RGB representation. Based on data recorded on the test set... Statistical data shows that, under the current MLP format, video-level pooling, and multi-task loss settings, Highly concentrated in the near-saturation range, the RGB branch still dominates in the fusion; therefore, the WER improvement brought by QAAF to the relatively fixed fusion is limited but consistent. The main empirical source of robustness should be discussed in conjunction with ablation and DAT, rather than attributed to QAAF alone. The design motivation is shown in Table 2.

[0035] Table 2 Design Motivation ; Under the settings of this invention, the skeleton is estimated offline from clear frames, and QAAF only from... estimate And adjust the relative weights of the two TemporalConv features.

[0036] In this embodiment of the invention, step 5 includes: Image quality degradation will alter the statistical distribution of RGB frame-level / temporal features; under a fixed-capacity MLP, RGB temporal features... Quality estimation input vector As a low-dimensional sufficient statistic, it is used for scalar weights related to network inference and fusion; given the TemporalConv output of the RGB encoding branch. The estimated learnable fusion weights process is as follows: ; ; in, , , As MLP weights, Dropout (0.2 and 0.1) is applied after the first two ReLU layers. For learnable bias parameters, For the Sigmoid function; Output bias initialization: initialize the scalar bias Initialize to This makes the Sigmoid output approximately [value missing] in the early stages of training. This assigns a higher prior weight to the RGB encoding branch; this setting is related to the DAT probability. This aligns with the batch statistics that primarily use clear samples, which helps stabilize the early optimization process. After convergence... It remains close to upper saturation under various conditions, which is consistent with "significantly reduced during degradation". The idealized trajectory is not consistent, so the behavior of QAAF will be discussed based on the measured distribution.

[0037] The contributions of the two features are dynamically adjusted based on the RGB quality status, and the fused temporal features are output; the expression for the fused temporal features is as follows: .

[0038] in Created via broadcast for all channels and time steps.

[0039] From a system behavior perspective, under the combined effect of DAT and the bimodal skeleton, the WER curve of the complete model on CE-CNSL-D increases relatively smoothly with degradation intensity. This phenomenon reflects the overall adaptation of the encoder-fusion-decoder, rather than being explained by a single scalar weighting mechanism. Compared to the fixed weighted baseline with no QAAF, the introduction of QAAF reduces the degradation-average WER and the clear test WER by approximately 1.5 and 1.9 percentage points, respectively. The improvement is limited but consistent across all settings.

[0040] QAAF additional parameter overhead: by count, It accounts for only a small percentage of the total parameters, making it a lightweight design.

[0041] Step 6, Timing Decoding and Result Output: This involves fusing the timing features. The input temporal features are processed by a bidirectional long short-term memory network (BiLSTM) and a connection to a temporal classification and decoding module (CTC) to output a sign language vocabulary sequence.

[0042] In this embodiment of the invention, step 6 includes: Fusing temporal features The BiLSTM and classification head are input, and then CTC decoding is used to obtain the sign language vocabulary sequence. The expression for the final output sign language vocabulary sequence recognition result is as follows: ; The expression for the auxiliary branch is: ; The corresponding CTC target is written as: .

[0043] The above completes the end-to-end sequence recognition from the fused temporal representation to the final sign language vocabulary sequence.

[0044] In this embodiment of the invention, the temporal convolution TemporalConv adopts a K5-P2-K5-P2 structure, consisting of two layers of one-dimensional convolution with a kernel of 5 and two layers of max pooling with a stride of 2. This downsamples the temporal dimension by a factor of 4 while simultaneously increasing the number of channels to a uniform dimension (from 512 to 1024), thereby ensuring that the two feature paths can be fused in both time and channel dimensions. ; BiLSTM uses a 2-layer bidirectional LSTM, with a hidden dimension of 512 in each direction and a total output dimension of 1024. ; The RGB encoding branch and the skeleton encoding branch each contain independent TemporalConv and BiLSTM, used to assist the supervision path; the fusion path only passes through QAAF. The classification header will not repeat TemporalConv; it contains a total of 3 BiLSTM groups and 2 TemporalConv groups.

[0045] In this embodiment of the invention, auxiliary supervision, knowledge distillation, and joint optimization are employed. In addition to the main path, auxiliary BiLSTM-CTC supervision is added to the two branches, and sequence distillation constraints are constructed to jointly train the total loss: ; ; in: The fusion branch CTC loss (main loss) is used to guide the final recognition result; , These are the auxiliary CTC losses for the RGB encoding branch BiLSTM output and TemporalConv output, respectively; , These are the auxiliary losses corresponding to the skeletal coding branches; , These are the sequence knowledge distillation losses; Weighting coefficients, weight settings: , , , .

[0046] In this embodiment of the invention, the above steps, through the joint constraints of main loss, auxiliary loss, and distillation loss, improve the stability and generalization ability of the fusion path and branch encoder, and finally form a robust model that can perform unified reasoning under both clear and degenerate conditions.

[0047] The dataset CE-CNSL of this invention is a Chinese continuous sign language dataset for complex environments. Constructed by Harbin Engineering University, it contains 5988 continuous sign language videos, 3515 words, and 12 translators (including 2 hearing-impaired individuals), covering over 70 real-life scenarios. The breakdown is shown in Table 3. Table 3 Dataset partitioning ; Skeletal keypoints were extracted using MediaPipe, containing 75 keypoints (42 for hands + 11 for upper body + 22 for face).

[0048] In embodiments of the present invention, such as Figure 2 As shown in (a) to (p), CE-CNSL-D (Degradation Assessment Benchmark): A degradation assessment benchmark is constructed based on the CE-CNSL test set. As shown in Table 4, 5 types of degradation are used, each with 3 degrees (mild, moderate, severe), for a total of 15 degradation conditions. In addition to the original clear condition, there are a total of 16 test conditions, as shown in Table 5, with implementation details.

[0049] Visual example of the CE-CNSL-D evaluation protocol: 1 sharpness condition and 5 types of synthetic degradation (low resolution, exposure, Gaussian noise, motion blur, JPEG compression), 3 intensity levels for each type, for a total of 16 test configurations.

[0050] Low resolution (simulating long-distance shooting): ; First, downsample using INTER_AREA to Then upsample back to the original size using INTER_LINEAR, parameters ;parameter 112, 96, and 72 correspond to mild, moderate, and severe cases at low resolution, respectively. Exposure (simulated overexposure): ; Gamma correction brightens the overall image; parameters ;parameter Values ​​of 1.5, 2.5, and 4.0 correspond to light, medium, and heavy exposure, respectively. Gaussian noise (analog sensor noise): ; parameter ;parameter 15, 30, and 50 represent mild, moderate, and severe Gaussian noise, respectively. Motion blur (simulating fast motion): ; for Horizontal motion blur kernel, parameters ;parameter 5, 11, and 19 correspond to mild, moderate, and severe motion blur, respectively. JPEG compression (simulating lossy transmission): ; parameter (Quality factor, the lower the value, the more serious the problem); parameters 30, 15, and 5 correspond to light, medium, and heavy JPEG respectively; Table 4 Degradation Types ; Table 5 Implementation Details ; Network and Temporal: Both the RGB encoding branch and the skeleton encoding branch are followed by the same TemporalConv structure after the frame-level features, reducing the temporal dimension to a lower level. The number of channels is increased to 1024, and then fed into the auxiliary BiLSTM (used for branch CTC and distillation) and the main BiLSTM (used for final CTC) after fusion with QAAF. The hidden dimension is unified to 1024 to facilitate the alignment of the two-path features at the fusion point.

[0051] Optimization and Scheduling: Adam is used in conjunction with weight decay to stabilize multi-task training; MultiStepLR multiplies the learning rate by 1 / 2 at the 20th and 30th epochs. This allows for more refined fusion and auxiliary heads at a lower learning rate in later stages. A batch size of 4 was used to accommodate the memory usage of dual-branch, multiple BiLSTM, and CTC within 24GB of GPU memory. Training consisted of 35 epochs, consistent with the optimal epoch statistics from the early stopping records in the validation set.

[0052] DAT and QAAF: DAT applies random degradation to RGB frames with a probability of 0.35, ensuring that the skeleton always corresponds to a sharp keypoint; this probability is a trade-off between exposing the degradation distribution and preserving enough clean samples to learn appearance details. QAAF's MLP takes the 1024-dimensional channels of the TemporalConv output as input, and maps them to a Sigmoid scalar via a 1024→512→128→1 mapping. The initial output bias is set to 1.5 to ensure optimal performance during the initial training phase. A slight bias towards RGB is beneficial for stable optimization in early iterations where most samples are clear.

[0053] Multitasking and Distillation: , , , The main CTC, RGB / skeleton-assisted CTC, and sequence knowledge distillation are weighted and fused separately to ensure that each encoder and fusion path obtains gradient signals; distillation temperature Used to soften the teacher (BiLSTM) distribution, making it easier for students (TemporalConv) to learn the temporal structure.

[0054] Implementation and Decoding: During training, PyTorch automatic mixed precision (FP16 + GradScaler) is enabled to accelerate the process and save GPU memory; during evaluation, Beam Search CTC decoding with a beamwidth of 10 is used on the fused branch output, consistent with common CSLR practices. The input spatial resolution is... The skeleton consists of 75 MediaPipe points, each... Total 75×3 dimensions.

[0055] Data augmentation (in addition to DAT): Apply random cropping to the training RGB values. Horizontal flip ( ) and timing length scaling ( This is to reduce overfitting to fixed viewpoints and fixed frame rates; the above enhancements only apply to RGB, and the bones are still sent to ST-GCN after being aligned with the label time.

[0056] In this embodiment of the invention, an ablation experiment was conducted, and the results of the ablation experiment are shown in Table 6.

[0057] Table 6 Ablation Experiment Results ; The validation set WER represents the validation set performance corresponding to the best epoch during training; the clear test WER and the degenerate average WER are both derived from evaluation results on the test set. The degenerate average WER is the mean of 15 degenerate conditions (excluding clear).

[0058] In this embodiment of the invention, ablation analysis is performed: DAT: Compared to the complete model, removing DAT reduced the WER for clear tests from 47.74% to 46.34%, but increased the average WER for 15 degradation types from 50.09% to 72.39% (a deterioration of 22.30 percentage points), and approached random guessing levels under conditions such as noise_30 and blur_19. This result supports considering DAT as an indispensable training regularization in the implementation of this invention, used to expose the RGB degradation distribution; without DAT, even with the dual-branch and QAAF retained, it is still difficult to maintain a usable recognition rate on unseen degradation types.

[0059] QAAF as a fixed-weighted fusion: While retaining DAT and bi-branch, replacing QAAF with a fixed weight increases the degenerate average WER from 50.09% to 51.58% (+1.49 percentage points), and the clarity test from 47.74% to 49.65% (+1.91 percentage points). QAAF can be characterized as a fusion improvement with limited magnitude and consistent direction, and its marginal contribution is less than that of removing DAT.

[0060] The relative roles of skeletal branches, DAT, and QAAF: The w / o QAAF configuration still achieves 51.58% of the degenerate average WER, indicating that skeletal geometry and DAT-induced RGB representation bear the main robustness in the implementation of this invention; QAAF in Additional adaptive weighting is provided spatially, further reducing WER.

[0061] In this embodiment of the invention, degradation robustness is evaluated, as shown in Table 7.

[0062] Table 7 compares the WER of each configuration under 16 test conditions. ; In embodiments of the present invention, such as Figure 3 As shown in (a) to (f), the horizontal axis represents the degradation condition grouping and intensity level (mild / moderate / severe), and the vertical axis represents the word error rate (WER). The lower the value, the better the recognition performance. In the figure, the blue, orange-red, golden, and gray bars correspond to the present invention (QAAF-CSLR), without DAT, without QAAF (fixed fusion), and TFNet baseline, respectively. Figure 3 (a) in the text represents the WER under the clean condition, which is used to show the WER changes of each method under the condition of no degradation test; Figure 3 (b) in the figure is used to show the WER changes under three degradation conditions of low resolution: light, moderate, and severe. The horizontal axis represents light degradation in order of severity. =112, Moderate =96, Severe =72 (represents the downsampling side length; the smaller the value, the more severe the degradation). Figure 3 (c) in the diagram is used to show the WER changes under three degradation conditions: light, medium, and heavy. The horizontal axis represents light degradation in order of severity. Moderate Severe ( The larger the value, the brighter the image; Figure 3 (d) in the figure is used to show the WER change under three degradation conditions: mild, moderate, and severe Gaussian noise. The horizontal axis represents mild noise in descending order of severity. Moderate Severe ( The larger the size, the stronger the noise. Figure 3 (e) in the figure is used to show the WER changes under three degradation conditions of light / medium / heavy motion blur, with the horizontal axis representing light in order. Moderate Severe ( (This refers to the size of the blur kernel; the larger the kernel, the stronger the blur). Figure 3 (f) in the figure is used to show the WER change under three degradation conditions of JPEG compression: light, medium, and heavy. The horizontal axis represents light degradation in order of severity. Moderate Severe (quality factor) The smaller the value, the greater the compression distortion. In this embodiment of the invention, Figure 3 To illustrate the recognition performance and robustness of this invention in both clear and multi-class degradation scenarios, it is evident that under most degradation conditions and varying intensities, the column height corresponding to this invention is generally lower than that of the comparative method, indicating that this invention can effectively reduce WER. Simultaneously, the WER significantly increases under conditions of noise, blur, and low resolution without DAT, demonstrating that the asymmetric degradation enhancement (DAT) during training contributes significantly to degradation robustness. The absence of QAAF (fixed fusion) shows consistent degradation compared to this invention, indicating that quality-aware adaptive fusion (QAAF) can further improve performance. Overall, this invention exhibits better stability and practicality in complex image degradation scenarios.

[0063] Degradation robustness of QAAF-CSLR: The complete method exhibits relatively gradual performance degradation under all degradation conditions. Taking the most severe degradation as an example (lr_72, ex_4.0, ns_50, bl_19, jp_5), the WER increases by a maximum of 6.90 percentage points from 47.74% in the clean condition to 54.64% (JPEG quality factor 5). This behavior is consistent with the degradation distribution of DAT exposure, the stability of skeletal branches on the inference side, and the learning of fusion paths. The catastrophic degradation with / no DAT: The ablation configuration without DAT achieves the best WER (46.34%) under clean conditions, but its performance deteriorates sharply under degenerate conditions. Specifically, under Gaussian noise ( ) and fuzzy ( Under these conditions, WER spiked to 91.27% and 86.63% respectively, almost completely losing its recognition ability. The root cause of this phenomenon is that this configuration never encountered degraded samples during the training phase, and its feature representation relied entirely on clean data distribution, lacking generalization ability when faced with distribution shifts.

[0064] Differentiated impacts of degradation types: The degree of impact of different degradation types on the system varies significantly. For w / oDAT configurations, low resolution and noise have the most severe impacts (lr_72: 85.63%, ns_50: 90.74%), while exposure degradation has a relatively milder impact (ex_1.5: 46.92%), reflecting the inherent differences in the robustness of ResNet-34 to different degradation types.

[0065] In embodiments of the present invention, such as Figure 4 As shown, the horizontal axis represents the test conditions, indicating the 16 test conditions of CE-CNSL-D, in the following order: clean; low resolution (light / medium / heavy) (lr_112, lr_96, lr_72); light / medium / heavy exposure (ex_1.5, ex_2.5, ex_4.0); light / medium / heavy Gaussian noise (ns_15, ns_30, ns_50); light / medium / heavy motion blur (bl_5, bl_11, bl_19); and light / medium / heavy JPEG compression. (jp_30, jp_15, jp_5), where light / medium / heavy represent progressively increasing degradation intensity; the vertical axis represents the model configuration, indicating the models participating in the comparison, namely the present invention (QAAF-CSLR), without DAT, without QAAF (fixed fusion), and TFNet baseline; the meaning of color and value: the number in each grid is the WER (%) of the model under the corresponding test conditions; the color bar on the right represents the numerical value, the darker the color, the higher the WER and the worse the recognition performance, and the lighter the color, the lower the WER and the better the recognition performance.

[0066] This invention maintains a low WER under most degradation conditions, and the performance decline is more gradual with increasing degradation, indicating stronger robustness. After removing DAT (without DAT), the WER increases significantly under conditions such as noise, blur, and low resolution, indicating that DAT contributes significantly to degradation robustness. After removing QAAF (fixed fusion), there is a consistent degradation compared to this invention, indicating that quality-aware adaptive fusion can further reduce WER. The TFNet baseline has a high WER under severe degradation, verifying the advantages of this invention in complex image quality degradation scenarios.

[0067] In embodiments of the present invention, such as Figure 5 As shown, the horizontal axis represents the number of training epochs, indicating the number of times the model parameters complete one iteration of the full training set; the vertical axis represents the word error rate (WER, %) on the validation set, with lower values ​​indicating better recognition performance. The four curves in the figure correspond to: this invention (QAAF-CSLR), without QAAF (fixed fusion), without DAT, and the TFNet baseline, respectively.

[0068] QAAF-CSLR (Full): Within 35 epochs, the validation set WER steadily decreased from the initial 79.72% to 47.91%, producing 22 optimal updates, with a smooth convergence process. w / o QAAF (Fixed Fusion): The convergence trend is similar to the full version, but the final WER converges to 49.93%, 2.02 percentage points higher than the full version, reflecting the lower efficiency of fixed fusion in utilizing the training signal. w / o DAT (No Degeneracy Augmentation): Converged to the lowest WER (47.12%) on the validation set (clean conditions) because its training process is not affected by degeneracy noise, allowing for sufficient optimization on a clean distribution. However, this advantage comes at the cost of a severe lack of degeneracy robustness.

[0069] Figure 5 Used to characterize the convergence speed, convergence stability, and final validation performance of different methods during training. Figure 5 As can be seen, with the increase of training rounds, the overall WER of each method decreases and gradually converges. The curve of this invention maintains a low WER in the mid-to-late stages and the decrease process is stable, demonstrating good optimization stability and generalization ability. Compared with the method without QAAF (fixed fusion), the WER of this invention is lower after convergence, indicating that the quality-aware adaptive fusion mechanism can further improve recognition performance. The curve without DAT can achieve a low WER on the clear validation set, but combined with the degradation test results, it is clear that it is not good at generalizing to degradation scenarios. The TFNet baseline curve is generally higher and converges more slowly, indicating that its performance and stability are weaker than those of this invention under the settings of this invention. In summary, this invention verifies the effective convergence characteristics and relative advantages of this invention in the training phase.

[0070] In this embodiment of the invention, a comparison with the baseline method is provided: TFNet (actually using the MSTNet architecture) is a multi-scale temporal continuous sign language recognition method proposed for the CE-CNSL dataset. It only uses RGB unimodal sign language and lacks degradation robustness design. Under the same CE-CNSL-D evaluation protocol, TFNet achieves a WER of 50.82% under clean conditions and an average WER of 73.07% under 15 degradation conditions, which is on par with w / o DAT (72.39%). Specifically, TFNet almost completely fails under severe degradation such as Gaussian noise (noise_30: 86.91%), motion blur (blur_19: 88.72%), and JPEG compression (jpeg_5: 87.01%), exhibiting a catastrophic degradation similar to w / o DAT. In comparison, QAAF-CSLR has an average degradation WER of 50.09% under the same degradation conditions, which is 22.98 percentage points lower than TFNet, mainly reflecting the effectiveness of the dual-modal path and DAT training; compared with w / oQAAF, QAAF brings an additional reduction of about 1.49 percentage points.

[0071] This invention proposes QAAF-CSLR (Quality-Aware Adaptive Fusion for CSLR): based on parallel encoding of the RGB encoding branch (ResNet-34) and the skeleton encoding branch (ST-GCN), quality-aware adaptive fusion (QAAF) is introduced at the TemporalConv output; during the training phase, asymmetric degradation enhancement (DAT) is used, that is, random degradation is applied only to RGB frames with a certain probability, while the skeleton sequence is estimated based on clear frames and aligned with time labels during the preprocessing phase, and decoupled from RGB degradation during testing under the evaluation settings of this invention. It is important to emphasize that the overall robustness is contributed by bimodal complementarity, the training distribution induced by DAT, and the fusion path. Their relative importance should be judged by ablation experiments: on 15 degradation test conditions, removing DAT increased the average WER from 50.09% to 72.39% (an absolute increase of 22.30 percentage points); when QAAF was replaced with fixed-weighted fusion, the average degradation WER was 51.58%, and the clear test WER increased from 47.74% to 49.65% (increases of 1.49 and 1.91 percentage points, respectively). This invention further constructs the CE-CNSL-D benchmark, covering 16 test configurations including one clear condition and three intensity levels for each of five degradation types. Results on CE-CNSL and CE-CNSL-D show that the proposed method improves the WER in both clean and degradation settings compared to the RGB unimodal strong baseline; among these, the gains from DAT and the bimodal skeleton are the most significant, while QAAF provides a smaller but consistent improvement over fixed fusion.

[0072] The core value of DAT is demonstrated by experimental results, which show that under the ablation settings of this invention, the removal of DAT has the greatest marginal impact on the degraded average WER. From w / o DAT to the complete model, the degraded average WER decreased from 72.39% to 50.09% (an absolute change of 22.30 percentage points); while when QAAF was replaced with fixed fusion, the corresponding difference was 1.49 percentage points (51.58% → 50.09%).

[0073] The role of multimodal complementarity in degradation scenarios can be observed through conditional analysis with / o DAT configuration. Different degradation types exhibit vastly different degrees of damage to RGB features. Exposure degradation (ex_1.5: 46.92%) has a relatively small impact on performance because ResNet-34 has some inherent robustness to global brightness changes; however, Gaussian noise (ns_30: 91.27%) and motion blur (blur_19: 86.63%) almost completely destroy the usability of RGB features. In these extreme degradation scenarios, the value of the skeletal keypoint modality is particularly prominent—since keypoints are pre-extracted from the original clear frame, their information is unaffected by RGB degradation, providing reliable information anchors for the fusion system.

[0074] QAAF and actual measurement , The high concentration across 16 test conditions indicates that the fusion weights in the current implementation have limited sensitivity to image quality conditions, with the RGB branch performing particularly well. Convex combinations of spaces still dominate. This empirical phenomenon differs from the literal meaning of quality-perceived adaptive fusion; therefore, this invention reports it truthfully in the experimental section. The distribution is discussed together with the finite marginal gain of w / o QAAF to avoid inconsistencies between the method and observations. Further extensions are needed in estimator architecture, supervisory signals, or spatiotemporal granularity to obtain more resolution modal weighting in degenerate scenarios.

[0075] In this embodiment of the invention, the original continuous sign language video is acquired, and the RGB frame sequence is extracted in parallel. and the corresponding skeletal key point sequence ; RGB frame sequence Input RGB encoding branch, then convert the skeletal keypoint sequence The skeletal encoding branch is input, and RGB temporal features are obtained after temporal convolution. With skeletal temporal features During the training phase, random image quality degradation is applied to RGB frames with preset probabilities to obtain the video frame sequence used for training. The quality-aware adaptive fusion module then... Estimating fusion weights ,right , Weighted fusion is performed to obtain fusion time-series features. ;Will The fused temporal features are processed through a bidirectional long short-term memory network and a connection-based temporal classification decoder to output the final sign language vocabulary sequence. During the training phase, degradation is applied only to the RGB values, and the skeleton is always estimated to correspond to sharp frames, forming asymmetric degradation enhancement (DAT). The system uses a learnable scalar and performs a convex combination of the two TemporalConv outputs. Based on the CE-CNSL-D multi-condition degradation evaluation protocol, this invention significantly reduces the word error rate under video quality degradation conditions, improving the robustness and accuracy of the continuous sign language recognition system in real-world deployment environments.

[0076] The technical solution provided by this invention includes data acquisition and bimodal preprocessing: acquiring raw continuous sign language video and extracting RGB frame sequences in parallel. and the corresponding skeletal key point sequence Asymmetric Degradation Enhancement (DAT): During the training phase, RGB frame sequences are augmented with preset probabilities. Random quality degradation is applied to the RGB frames to obtain the video frame sequence of the training input. Dual-branch feature encoding: Encoding video frame sequences... Input the RGB encoding branch and the skeletal keypoint sequence Input the skeletal encoding branch to obtain the extracted frame-level features. and Temporal Convolutional (TCON) Modeling and Feature Alignment: [This section appears to be incomplete and requires further context.] and Perform temporal convolutions separately to obtain time-aligned RGB temporal features. With skeletal temporal features Quality-Aware Adaptive Fusion: Utilizing the Quality-Aware Adaptive Fusion (QAAF) module based on... Estimate the learnable fusion weights and... and Perform convex combination weighted fusion to obtain fused temporal features. Temporal decoding and output: fusing temporal features The input temporal features are processed by a bidirectional long short-term memory network (BiLSTM) and a connected temporal classification and decoding module (CTC) to output a sign language vocabulary sequence. This method improves the robustness and accuracy of the continuous sign language recognition system in real-world deployment environments.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A degradation-robust quality-aware adaptive fusion continuous sign language recognition method, characterized in that, The method includes: Step 1, Data Acquisition and Bimodal Preprocessing: Acquire the raw continuous sign language video and extract the RGB frame sequence in parallel. and the corresponding skeletal key point sequence ; Step 2, Asymmetric Degradation Enhancement (DAT): During the training phase, RGB frame sequences are processed with preset probabilities. Random quality degradation is applied to the RGB frames to obtain the video frame sequence of the training input. ; Step 3, Dual-branch feature encoding: Encode the video frame sequence Input the RGB encoding branch and the skeletal keypoint sequence Input the skeletal encoding branch to obtain the extracted frame-level features. and ; Step 4, Temporal Convolution Modeling and Feature Alignment: [This section likely refers to a step or step, but the context is unclear and requires further information.] and Perform temporal convolutions separately to obtain time-aligned RGB temporal features. With skeletal temporal features ; Step 5, Quality-Aware Adaptive Fusion: Using the Quality-Aware Adaptive Fusion module (QAAF) based on... Estimate the learnable fusion weights and... and Perform convex combination weighted fusion to obtain fused temporal features. ; Step 6, Timing Decoding and Result Output: This involves fusing the timing features. The input temporal features are processed by a bidirectional long short-term memory network (BiLSTM) and a connected temporal classification and decoding module (CTC) to output a sign language vocabulary sequence. Step 5 includes: Image quality degradation will alter the statistical distribution of RGB frame-level / temporal features; under a fixed-capacity MLP, RGB temporal features... Quality estimation input vector As a low-dimensional sufficient statistic, it is used for scalar weights related to network inference and fusion; given the TemporalConv output of the RGB encoding branch. The estimated learnable fusion weights process is as follows: ； ； in, , , As MLP weights, Dropout is applied after the first two ReLU layers. For learnable bias parameters, For the Sigmoid function; The expression for the fused temporal features is: 。 2. The method according to claim 1, characterized in that, Step 1 includes: Obtain the raw continuous sign language video and decode it into an RGB frame sequence, denoted as: ； Skeletal keypoint sequence extracted from raw frames and time-aligned , denoted as: 。 3. The method according to claim 2, characterized in that, Step 2 includes: During the training phase, random image quality degradation is applied only to RGB frames to obtain a video frame sequence. Skeletal key point sequence Keeping the clear estimate unchanged, video frame sequence The expression is: ； in, Indicates the first Image degradation operators, This represents the probability of applying degradation to RGB frames during the training phase.

4. The method according to claim 3, characterized in that, Step 3 includes: The RGB encoding branch uses ResNet-34 pre-trained on ImageNet to extract frame-level spatial features from video frame sequences. The feature map is obtained after four residual stages. Frame features are obtained through global average pooling. Frame features are arranged chronologically to form a frame-level feature sequence of RGB coding branches. Subsequently, it is fed into the temporal convolutional modeling layer; The skeleton coding branch uses the spatiotemporal graph convolutional network ST-GCN to perform spatiotemporal coding on the keypoint sequence; Graph Structure: Construct an undirected graph over the 75 keypoints provided by MediaPipe, with the adjacency matrix denoted as... ; Normalized adjacency matrix: ; ST-GCN blocks: Each layer contains spatial graph convolutions and temporal convolutions, with residual connections. ； Encoder structure: 9-layer ST-GCN, with channels gradually increasing. Finally, global average pooling is performed on the keypoint dimension. Convolutional mapping to 512 dimensions, frame-level feature sequence of the skeletal coding branch: 。 5. The method according to claim 4, characterized in that, Step 4 includes: Time-aligned RGB timing features With skeletal temporal features Their expressions are as follows: ； 。 6. The method according to claim 1, characterized in that, Step 6 includes: Fusing temporal features The BiLSTM and classification head are input, and then CTC decoding is used to obtain the sign language vocabulary sequence. The expression for the final output sign language vocabulary sequence recognition result is as follows: ； The expression for the auxiliary branch is: ； The corresponding CTC target is written as: 。 7. The method according to claim 6, characterized in that, TemporalConv employs a K5-P2-K5-P2 structure, consisting of two 1D convolutional layers with a kernel size of 5 and two max pooling layers with a stride of 2, downsampling the temporal dimension by a factor of 4 while increasing the number of channels from 512 to 1024. ； BiLSTM uses a 2-layer bidirectional LSTM, with a hidden dimension of 512 in each direction and a total output dimension of 1024. ； The RGB encoding branch and the skeleton encoding branch each contain independent TemporalConv and BiLSTM, used to assist the supervision path; the fusion path only passes through QAAF. The classification header will not repeat TemporalConv; it contains a total of 3 BiLSTM groups and 2 TemporalConv groups.

Images