Sign language translation system and method based on lightweight mask enhancement

Abstract

The invention discloses a sign language translation system and method based on lightweight mask enhancement, and the method comprises the following steps: 1, employing an inter-frame difference technology to recognize and reject redundant frames having no substantial contribution to motion expression, and further distinguishing the foreground and background information of sign language motion through a local dynamic mask module; step 2, inputting the sign language video optimized by the lightweight mask module to a visual module; accurate and comprehensive visual features are generated; 3, converting the visual features obtained from the visual module into language features suitable for language processing; and step 4, the language module receives the language features converted by the mapping module, and processes the language features in combination with an encoder-decoder model mBART to improve the quality of the output text. According to the method, the accuracy and efficiency of sign language-to-spoken language translation are remarkably improved; the method not only optimizes the use of computing resources, but also greatly enhances the readability and semantic coherence of translation results, and provides powerful technical support for realizing more popular and convenient sign language communication.

Description

Technical Field

[0001] The present invention belongs to the technical field of sign language translation, and in particular relates to a sign language translation system and method based on lightweight mask enhancement. Background Art

[0002] The core of the sign language translation task lies in accurately converting sign language movement videos into corresponding spoken text sentences. A significant technical challenge in this process is the huge semantic gap between sign language movements and spoken sentences. Unlike traditional natural language spoken sentences, which mainly rely on sound expression, sign language, as a visual language, relies not only on changes in gestures but also incorporates information from multiple dimensions such as facial expressions and body posture to express itself. This fundamental difference makes it extremely complex to directly map sign language movements to spoken sentences. To address this challenge, current sign language translation technology mainly adopts two methods: Sign2Text (S2T) and Sign2Gloss2Text (S2G2T). These two methods have their own characteristics and aim to bridge the semantic gap between sign language and spoken language through different paths.

[0003] The Sign2Text method uses an end-to-end network structure and attempts to generate spoken sentences directly from sign language videos. Its advantage is that the model structure is relatively simple and intuitive, which can reduce the risk of error propagation. However, the Sign2Text method also has obvious limitations. First, due to the lack of intermediate representation, the model performs poorly when dealing with complex long-term dependency problems. This means that when faced with longer or more complex sign language sequences, the Sign2Text model may experience inaccurate translation. Secondly, the Sign2Text method has extremely high requirements for data processing, and data annotation is costly and time-consuming, which makes its promotion in practical applications challenging. In addition, the method is relatively difficult in network debugging and model understanding.

[0004] In contrast, the Sign2Gloss2Text approach effectively alleviates some of the issues of the Sign2Text approach by introducing gloss sequences as intermediate markers. Within this framework, the sign language translation task is divided into two stages: the Sign2Gloss stage, a fine-grained visual action classification task that aims to convert a given sign language video into a corresponding gloss sequence; and the Gloss2Text stage, the language translation task, which aims to translate the gloss sequence into the corresponding spoken text. This modular design allows for independent optimization of each module, facilitating intermediate debugging. Furthermore, the relative ease of gloss annotation reduces data acquisition costs. However, while the Sign2Gloss2Text approach addresses some of these issues, it also introduces new challenges. The separate task structure increases system complexity, and the multi-step processing significantly increases computational resource requirements. More importantly, errors in the gloss translation process directly impact the subsequent text generation process, further increasing the difficulty of translating the entire sentence and compromising the final expression quality. Furthermore, gloss is a discrete representation of the language modality and does not encode the spatiotemporal visual information contained in the sign language video, such as facial expressions. Directly using it as an intermediate modality can result in reduced translation performance. For example, hearing-impaired people often express the adverb "extremely" through exaggerated facial expressions, and this information is ignored in the gloss annotation. However, if you want to generate a complete and semantically accurate translation sentence, you must also take these adverbs into account. Therefore, using gloss as the intermediate mode of the entire sign language translation model is not optimal.

[0005] Another challenge facing sign language translation is the uncertainty of sign language data. Sign language translation models must focus on key motion areas in the video, such as the signer's hand movements, body posture, and facial expressions. However, in real-world scenarios, there are many uncontrollable factors, such as complex background changes, varying lighting conditions, camera angles, and individual characteristics of the signer, such as skin color, clothing, and movement amplitude. These factors can interfere with sign language recognition and translation, affecting the accuracy and stability of the model.

[0006] Therefore, how to accurately extract sign language movement features and enhance the system's generalization ability becomes the key to building an efficient and stable sign language translation system. Summary of the Invention

[0007] To overcome the shortcomings of the aforementioned prior art, the present invention aims to provide a sign language translation system and method based on lightweight mask enhancement, which efficiently processes sign language video data, accurately extracts key visual features, effectively converts visual and language modality information, and generates natural and fluent spoken text. By integrating a lightweight mask module for redundant frame removal and foreground-background differentiation, a visual module specifically designed to capture spatiotemporal features, a mapping module that compensates for the limitations of gloss sequences, and a language module that utilizes an advanced encoder-decoder model for text generation, the present invention significantly improves the accuracy and efficiency of sign language-to-spoken translation. This not only optimizes the use of computing resources but also greatly enhances the readability and semantic coherence of the translation results, providing strong technical support for achieving more widespread and convenient sign language communication.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A sign language translation system based on lightweight mask enhancement, including a lightweight mask module, a vision module, a mapping module, and a language module;

[0010] The lightweight mask module takes the sign language video as input, removes redundant frames and combines local dynamic masking technology to distinguish the sign language movements from background interference;

[0011] The visual module converts the sign language video optimized by the lightweight mask module into effective visual features, providing data support for subsequent mapping and translation tasks;

[0012] The mapping module converts the visual features extracted by the vision module into language features suitable for input by the language module;

[0013] The language module uses the language features converted by the mapping module and combines it with the encoder-decoder model mBART to predict the corresponding spoken text sentences.

[0014] A sign language translation method based on lightweight mask enhancement includes the following steps:

[0015] Step 1. Data preprocessing and redundant frame removal:

[0016] During the initial processing of sign language video data, frame differencing technology is used to identify and remove redundant frames that do not contribute substantially to the expression of the action. A local dynamic mask module based on spatial attention technology is used to further distinguish the foreground and background information of the sign language action, allowing the model to focus more on the key action information in the input sequence.

[0017] Step 2. Extract effective visual features:

[0018] Input the sign language video optimized by the lightweight mask module into the vision module; generate accurate and comprehensive visual features as input for the next step of processing;

[0019] Step 3. Conversion of visual features to language features

[0020] Using the mapping module, the visual features obtained from the vision module are converted into language features suitable for language processing;

[0021] Step 4. Generate spoken text

[0022] The language module receives the language features converted by the mapping module and processes them in combination with the encoder-decoder model mBART to improve the quality of the output text.

[0023] In step 1, the changed area of ​​the inter-frame difference method is quickly detected by calculating the difference between adjacent frame images;

[0024] When dealing with scenes with slow-moving targets, a two-frame difference method is used;

[0025] When body movements or facial expressions change rapidly, the three-frame difference method is used to extract motion information.

[0026] The calculation process of the two-frame difference method can be divided into four steps:

[0027] (1) Difference image: First, perform a subtraction operation on the grayscale values ​​of pixels at the same position in two consecutive frames of the sign language video data, i.e., fn(x, y) and fn-1(x, y), and take the absolute value of the result to obtain the difference image Dn;

[0028] (2) Image binarization analysis: Use the threshold T to binarize each pixel in the differential image Dn to generate the image Rn. In this process, if the grayscale value of a pixel is 0, the pixel is identified as part of the background; and a pixel with a grayscale value of 255 is considered to be a sign of motion.

[0029] (3) Connectivity analysis: a process used to identify connected sets of foreground pixels in a binary image, i.e., to determine which pixels belong to the same moving object;

[0030] (4) Capturing motion information: Based on the results of connectivity analysis, further extract information about each moving object. Finally, through the above processing steps, an image containing the complete moving object is extracted;

[0031] D n (x,y)=|f n (x,y)-f n-1 (x,y)|

[0032]

[0033] The three-frame difference method is specifically as follows:

[0034] Select three consecutive frames from the video stream, and then perform differential operations on two adjacent frames. n and D n+1 Calculate according to the following formula and extract motion information after processing;

[0035] D′ n (x,y)=|f n+1 (x,y)-f n (x,y)|∩|f n (x,y)-f n-1 (x,y)|

[0036] By adding a light-sensitive term to the threshold T to adapt to scenes with changing lighting, the impact of lighting changes on target motion judgment is suppressed, as shown in the following formula. If the scene lighting changes slightly, the value of this term will tend to zero; when the lighting changes drastically, the value will increase significantly to dynamically increase the judgment threshold.

[0037]

[0038] Where λ is the light suppression coefficient, N A is the total number of pixels in the area to be detected.

[0039] In step 1, a local dynamic mask module (denoted as Mask module) based on spatial attention technology is used to eliminate background effects;

[0040] For the input feature map F that removes redundant frames that have no substantial contribution to the expression of the action, the Mask module will first calculate an attention score M based on F S (F), or it can be called spatial weight, the calculation formula of the Mask module attention score is as follows:

[0041]

[0042] Where F is the input feature map, the size is (B, C, T, H, W), where B is the batch size, C is the number of channels, T is the time dimension, H is the height, and W is the width; MaxPooling (·) and AveragePooling (·) in the above formula refer to the use of maximum pooling and average pooling on F in the channel dimension, respectively, to obtain two tensors of size (B, 1, T, H, W), and then after the concatenation (·) operation, a tensor of size (B, 2, T, H, W) is obtained. The conv in the above formula k1 (·) and conv k2(·) is specifically implemented as a three-dimensional convolution operation with a kernel height and width of k1 and k2 and a time dimension of 2. k1 and k2 are hyperparameters that can be adjusted according to actual conditions. k1 is set to 3 to capture subtle sign language movement features, and k2 is set to 31 to capture large-scale human motion areas. Then, two tensors M of size (B, 1, T, H, W) are added. S1 (F) and M S2 (F) is added to obtain the fusion features of both large and small scales; in the above formula, f(·) is an arbitrary activation function, which is sigmoid.

[0043] For the fusion feature of size (B, 1, T, H, W), the final attention score is obtained through the sigmoid function, which is the spatial weight M of size (B, 1, T, H, W). S (F); spatial weight M S The value of each pixel in (F) represents the importance of the corresponding position in the input feature map F, and M is obtained. S After (F), it is used to mask the input feature map F, thereby extracting the foreground and suppressing the background. The mask operation is defined as Operation, the specific formula is as follows:

[0044]

[0045] Use two convolution kernels of different sizes to extract features of different scales in sign language movements, fuse them and output attention scores; finally, The operation is to multiply the elements at the same position in different channels of the input feature map by the element value of each corresponding position in the tensor to obtain the final output; among them, the attention score of (B, 1, T, H, W) is equivalent to the required Mask.

[0046] In step 2, the visual module architecture includes a video backbone network and a lightweight head network. The video backbone network is responsible for extracting rich spatiotemporal feature information from the sign language video; the lightweight head network is used to further encode information in the time dimension.

[0047] The visual module first uses the S3D video backbone network to extract the spatiotemporal feature information in the video, and then further encodes the temporal information through a lightweight head network to ensure the understanding of the action features in the video and their changes over time.

[0048] In step 3, the mapping module uses a fully connected multilayer perceptron (MLP) with two hidden layers to convert the visual features extracted by the vision module into language features suitable for processing by the language module. The mapping module receives the visual features output from the head network of the vision module and converts them into language features for subsequent use. The entire translation network framework can achieve end-to-end training and learning under the joint supervision of CTC loss and translation loss.

[0049] The hidden layer is used to supplement the spatiotemporal visual information of the sign language video that cannot be fully encoded by the gloss sequence, providing the language module with richer and more detailed video spatiotemporal information and language clues.

[0050] In step 4, the encoder first converts the input language features into an intermediate representation that contains the key information of the sign language. Then, the decoder generates natural, fluent, and semantically accurate spoken sentences based on this intermediate representation, while taking into account the immediate context information and the semantic coherence of the entire sequence, thereby improving the quality of the output text.

[0051] Beneficial effects of the present invention:

[0052] Through the above steps, the present invention provides an efficient and accurate sign language to spoken language translation method, which significantly improves the accuracy and readability of the translation results.

[0053] The present invention optimizes the input data by introducing the inter-frame difference method. Specifically, this method does not process all video frames comprehensively, but only calculates the differences between multiple consecutive frames to intelligently remove redundant frames that have little impact on the final translation result from the input sign language video frame sequence. This optimization greatly reduces the resource consumption during the training process, avoids the high computational overhead caused by processing a large amount of redundant information in traditional methods, effectively reduces the difficulty of data modeling and improves the modeling efficiency of the model. In addition, by removing redundant frames, the system can focus more on the analysis and processing of key frames to ensure that every important sign language movement detail is captured, thereby improving the accuracy and fluency of the translation. In general, this method based on the inter-frame difference method provides an efficient and accurate solution for the pre-processing of sign language videos, allowing subsequent visual feature extraction and translation tasks to proceed more smoothly, ultimately achieving a higher quality sign language to text conversion effect.

[0054] This paper proposes a local dynamic masking module based on spatial attention technology and introduces two different convolutional kernel sizes within this module to specifically capture diverse features in sign language videos. Specifically, the small convolutional kernel is specifically designed to capture subtle sign language motion features, such as the signer's facial expressions, while the large convolutional kernel focuses on extracting large-scale motion features of the human body, such as arm movements and body posture. By fusing the features extracted by these two convolutional kernels, we can obtain a comprehensive and detailed representation of motion features. This multi-layered feature extraction approach improves the model's ability to understand both subtle and significant motions, reduces the impact of video background on model accuracy and stability, and further optimizes overall system performance, resulting in more accurate sign language video-to-text conversion. Compared to traditional methods, this approach does not require comprehensive processing of all features, but instead selectively selects and integrates key information, reducing unnecessary computational overhead and improving efficiency. This allows for the effective capture and interpretation of everything from subtle facial expressions to large body movements, providing solid technical support for high-quality sign language translation. This strategy significantly enhances the practicality and reliability of the system, ensuring a more efficient and accurate sign language interpretation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is the overall system framework diagram of the present invention.

[0056] Figure 2 Schematic diagram of the lightweight mask module of the present invention.

[0057] Figure 3 Schematic diagram of the visual module of the present invention.

[0058] Figure 4 Schematic diagram of the two-frame difference method of the present invention.

[0059] Figure 5 Schematic diagram of the three-frame difference method of the present invention.

[0060] Figure 6 Schematic diagram of the local dynamic mask module of the present invention.

[0061] Figure 7 This is an analysis diagram of the simulation results of the present invention. DETAILED DESCRIPTION

[0062] The present invention will be further described in detail below with reference to the accompanying drawings.

[0063] The technical solution of the present invention relates to a sign language translation method based on lightweight mask enhancement. To facilitate a more intuitive understanding of the structure, principles, and operational relationships of the various components of the invention, the technical solution is described in detail below with reference to the accompanying drawings.

[0064] Structure Description

[0065] like Figure 1 As shown in the figure, the sign language translation system based on lightweight mask enhancement includes four modules: a lightweight mask module that removes redundant frames and distinguishes foreground and background information, a visual module that extracts effective visual features from sign language videos, a mapping module that performs feature conversion, and a language module that generates spoken sentences based on language features.

[0066] The modules are interconnected to form the overall framework of the sign language translation model proposed in the present invention.

[0067] Lightweight Mask Module: This module addresses the massive data processing challenges of sign language translation tasks. It uses frame differencing techniques during data preprocessing to remove redundant frames, effectively reducing unnecessary data. Furthermore, it utilizes spatial attention techniques to distinguish foreground from background information, enabling the model to focus on key motion information in the input sequence.

[0068] Specifically, if Figure 2 This module consists of two main submodules: a redundant frame removal module and a local dynamic mask module. The redundant frame removal module uses inter-frame difference analysis to analyze video sequences, identifying and removing redundant frames that do not contribute substantially to the expression of the action, thereby optimizing training efficiency without losing any critical information. Simultaneously, the local dynamic mask module, based on spatial attention technology, distinguishes foreground sign language movements from background interference, allowing the model to focus on the actual action area, significantly enhancing learning effectiveness.

[0069] In summary, the lightweight mask module aims to reduce the computational load while improving model training results, providing solid technical support for building an efficient and accurate sign language translation system.

[0070] Vision Module: The purpose of the vision module is to convert the sign language video processed by the lightweight mask module into valid visual features. This visual feature will then serve as the input of the mapping module.

[0071] Specifically, the architecture of the visual module is as follows Figure 3 As shown in the figure, it consists of two parts: one is the S3D video backbone network that can extract the spatiotemporal feature information in the sign language video, and the other is the lightweight head network for further encoding the temporal information.

[0072] For the video backbone network, the present invention uses the S3D network as its underlying architecture, specifically for basic video feature extraction, given its excellent balance between model performance and inference speed. The video backbone network is designed to extract dense visual feature representations for accurate gloss sequence prediction. To this end, the video backbone network specifically utilizes the first four blocks of the S3D network, which efficiently extract spatiotemporal feature information from the input sign language video, providing high-quality visual feature representations for subsequent processing.

[0073] For the lightweight head network, in order to further encode temporal information, the head network designed by the present invention includes a projection block and a temporal convolution block. Specifically, the projection block consists of a temporal linear layer, a batch normalization layer and a ReLU activation layer. This block is responsible for the preliminary conversion of the S3D features extracted by the video backbone network to adapt to the subsequent temporal dimension processing requirements. The temporal convolution block contains two temporal convolution layers, a linear translation layer and a ReLU activation layer. This block aims to strengthen the understanding of the temporal relationship between video frames and extract more refined temporal feature information through multi-layer processing. The head network receives the S3D features from the video backbone network as input, and after the above-mentioned structural processing, generates a gloss feature representation, which characterizes the characteristics of the gloss category in a high-dimensional space. Then, these features are fed into the linear classifier and the frame-based gloss probability is calculated by the Softmax function. In the Sign2Gloss task stage, the gloss probability extracted by the visual module can be used to calculate CTC

[0074] (ConnectionistTemporalClassification) loss and input it into the CTC decoder to achieve the prediction of gloss sequence.

[0075] This design not only ensures the capture of action features in the video, but also strengthens the understanding of how these actions change over time, thereby providing more accurate and comprehensive input for subsequent processing.

[0076] Mapping module: The mapping module aims to build a bridge between the visual modality and the language modality to supplement the spatiotemporal visual information of the sign language video that cannot be fully encoded by the gloss sequence. In the Sign2Gloss2Text framework, although the vision module can convert the sign language video into gloss sequences, and the translation module can convert these gloss sequences into spoken sentences, directly using the gloss sequence as an intermediate representation is not the optimal choice. The mapping module is specifically implemented as a multi-layer perceptron with two hidden layers. By converting the visual features extracted by the vision module into language features suitable for the input of the language module, it provides the language module with more video spatiotemporal information and language information clues, enhancing the understanding ability of the sign language translation model, thereby making the conversion from sign language video to spoken sentences more accurate and rich. This helps to overcome the limitations of gloss representation and improve the performance of the entire sign language translation system.

[0077] Language module: Using the language features converted by the mapping module and combined with the encoder-decoder model mBART, the language module is used to accurately calculate and predict the corresponding spoken text sentences.

[0078] First, the encoder processes the input language features and converts them into an intermediate representation that contains the key information of the sign language. Subsequently, the decoder generates natural, fluent, and semantically accurate spoken sentences based on this intermediate representation. The decoder not only takes into account the immediate contextual information, but also understands the semantic coherence of the entire sequence, thereby improving the quality of the output text. In this way, the system can not only effectively convert sign language content into spoken text, but also greatly improve the accuracy and readability of the translation results. This conversion process from language features to spoken text is a key step in realizing an efficient and accurate sign language translation system.

[0079] The various modules work closely together to promote the construction of the sign language translation model in order to improve translation efficiency and model performance.

[0080] Principle Description

[0081] This invention uses a lightweight mask module to remove redundant frames and distinguish the background information of the video, uses a mapping module as a bridge for feature conversion to provide the language module with more sufficient visual feature information, and transfers the knowledge of other general domain datasets to the sign language translation task through gradual pre-training, thereby reducing the resource consumption required for model training and testing, improving model translation efficiency, and enhancing model performance.

[0082] Principle of lightweight mask module

[0083] (1) Principle of redundant frame removal module

[0084] Keyframes are the most representative frames in a video and can summarize the entire video content. The correlation analysis of keyframes is based on the content changes in the video. Usually, the selected frames should reflect the main movements and scene changes in the video. In the isolated sign language recognition task, the average length of the processed single action video is 4 seconds, and some are as short as 1 second. The movement details are relatively evenly distributed on the time scale. The sign language videos processed in the sign language translation task may contain several sign language words, with an overall longer duration, generally containing more than 100 frames. In these long sign language videos, there may be a large number of redundant frames, which affects the data processing efficiency. Using the key frame extraction algorithm to remove redundant frames from the input samples and remove the pauses between sign language actions can achieve the purpose of improving the convergence speed of the model.

[0085] The present invention proposes a redundant frame removal technology for filtering sign language videos based on the inter-frame difference method, wherein the inter-frame difference method is a simple and efficient target detection algorithm, and the changed area can be quickly detected by differential calculation of adjacent frame images.

[0086] like Figure 4 As shown, the calculation process of the two-frame difference method can be divided into four steps: (1) Differential image. First, the grayscale values ​​of the pixels at the same position in two consecutive frames, namely fn(x, y) and fn-1(x, y), are subtracted, and the absolute value of the result is taken to obtain the differential image Dn; (2) Image binarization analysis. Each pixel in the differential image Dn is binarized using the threshold T to generate the image Rn, as shown in the following formula. In this process, if the grayscale value of the pixel is 0, the pixel is identified as part of the background; and the pixel with a grayscale value of 255 is regarded as a sign of motion; (3) Connectivity analysis. This step is used to identify the set of connected foreground pixels in the binary image, that is, to determine which pixels belong to the same moving object. This step is crucial for understanding the shape, size and position of different objects in the image. Common connectivity analysis algorithms include 4-connectivity and 8-connectivity methods, which consider the relationship between the pixel and its neighbors in the four directions of up, down, left, and right, or in eight directions including the diagonal; (4) Capturing motion information. Based on the results of the connectivity analysis, we can further extract information about each moving object, such as its bounding box, center of mass location, area, and other features. This information is crucial for high-level tasks such as object tracking and gesture recognition. For example, in sign language recognition, specific gestures can be identified by analyzing motion patterns between consecutive frames. Ultimately, through the above processing steps, an image containing the complete moving object can be extracted.

[0087] D n (x,y)=|f n (x,y)-f n-1 (x,y)|

[0088]

[0089] The two-frame difference method is suitable for processing scenes with slow target movement, such as slow hand movements and facial expressions. When the body movements or facial expressions change quickly, the three-frame difference method can be used to extract motion information. The principle of the three-frame difference method is similar to the two-frame difference method, and it can also be divided into the above four steps for operation, but it differs in the calculation and processing of the difference image, such as Figure 5 Specifically, three consecutive frames are selected from the video stream, and then the difference operation is performed on two adjacent frames. n and D n+1 Calculate according to the following formula and extract motion information after processing.

[0090] D′ n (x,y)=|f n+1 (x,y)-f n (x,y)|∩|f n (x,y)-f n-1 (x,y)|

[0091] To adapt the algorithm to scenes with changing lighting, we can add a lighting-sensitive term to the threshold T to suppress the impact of lighting changes on target motion judgment, as shown in the following formula. If the scene lighting changes slightly, the value of this term will tend to zero; when the lighting changes drastically, the value will increase significantly, dynamically raising the judgment threshold.

[0092]

[0093] Where λ is the light suppression coefficient, N A is the total number of pixels in the area to be detected.

[0094] By applying frame differencing to the original input frame sequence, unnecessary frames in the video can be effectively identified and removed, significantly reducing the amount of data required to be processed. This method optimizes data flow and improves processing efficiency, making the analysis and understanding of video content more efficient.

[0095] (2) Principle of local dynamic mask module

[0096] Spatial attention is a method for selecting salient regions within a feature map based on the spatial relationships between different feature maps. It computes an attention score for the input feature map and uses this score as a weighting factor for the values ​​at the same pixel location across different channels of the input feature map, generating an output feature map containing only salient regions. However, for sign language tasks, a challenge lies in the fact that background objects in the video can interfere with the recognition of sign language movements. Factors such as the color of the signer's clothing, unrelated passersby, and background changes can all introduce noise. Furthermore, the habits of different signers can lead to inconsistent movement positions and amplitudes, making it even more difficult for the model to capture salient regions within the scene. Common translation models typically use the entire frame as input to the network model. During feature extraction, the image is scaled, resulting in a significant loss of movement details, reducing the ability to express behavioral features in the core regions. However, the backgrounds of real-world scenarios are complex and varied, making it impossible to fully capture all backgrounds due to data collection incompleteness. Therefore, a local dynamic masking module (denoted as the Mask module) based on spatial attention is proposed to eliminate background influences, thereby focusing the model's attention on spatial regions such as the signer's hands, face, and upper body in the video. For the input feature map F, the Mask module will first calculate an attention score M based on F. S (F), which can also be called spatial weight, the calculation formula of the Mask module attention score is as follows:

[0097]

[0098] Where F is the input feature map, with a size of (B, C, T, H, W), where B is the batch size, C is the number of channels, T is the time dimension, H is the height, and W is the width. In the above formula, MaxPooling (·) and AveragePooling (·) refer to the use of maximum pooling and average pooling on F in the channel dimension, respectively, to obtain two tensors of size (B, 1, T, H, W), which are then subjected to concatenation (·) operations to obtain a tensor of size (B, 2, T, H, W). The conv in the above formula is k1 (·) and conv k2 (·) is specifically implemented as a three-dimensional convolution operation with a kernel height and width of k1 and k2 and a time dimension of 2. k1 and k2 are hyperparameters that can be adjusted according to actual conditions. In this paper, k1 is set to 3 to capture subtle sign language movement features, and k2 is set to 31 to capture large-scale human motion areas. Then, two tensors M of size (B, 1, T, H, W) are S1 (F) and M S2(F) is added to obtain the fusion feature containing both large and small scales. In the above formula, f(·) can be any activation function, and in this paper it is sigmoid. Therefore, for the fusion feature of size (B, 1, T, H, W), the final attention score is obtained after the sigmoid function, which is the spatial weight M of size (B, 1, T, H, W). S (F). Spatial weight M S The value of each pixel in (F) represents the importance of the corresponding position in the input feature map F. Embedding this module in the model can play a role in weakening irrelevant background. S After (F), it is used to mask the input feature map F, thereby extracting the foreground and suppressing the background. The mask operation is defined as Operation, the specific formula is as follows:

[0099]

[0100] The operating mechanism of the Mask module is as follows Figure 6 As shown in , it uses two convolution kernels of different sizes to extract features of different scales in sign language movements, and then outputs the attention score after fusion. S (F) is not a single value, but a tensor of size (B, 1, T, H, W). The operation is to multiply the elements at the same position in different channels of the input feature map by the element value of each corresponding position in the tensor to obtain the final output. Among them, the attention score of size (B, 1, T, H, W) calculated here is equivalent to the required Mask. This paper expects that the Mask corresponding to each frame can achieve a value close to 0 in the background part and a value close to 1 in the parts related to sign language movements such as the hands and head, so that when the Mask is "multiplied" with the video or feature map, it can play a role in removing the background. In order to enable the model to retain all spatial features in the initial stage and avoid the gradient disappearance caused by too few transferred features, the initial value of the bias value of the 3D convolution of the module can be set to a larger value (in the present invention, it is set to 3), so that the initial Mask value at all spatial positions is close to 1. Subsequently, the model gradually learns how to remove background interference through training data. Since the designed Mask module is intended to weaken the influence of the background, it is very important for the module to obtain a large amount of spatial information to distinguish important objects from irrelevant objects. In the above-mentioned Mask module design, the input of the convolution is a feature map with only two channels. Therefore, in order to obtain more spatial information, conv k2(·) A large-size convolution kernel is used. In addition, the proposed local dynamic mask module is an independent module that does not change the size of the input feature map, so it can be flexibly applied to other models.

[0101] Principle of the mapping module

[0102] The mapping module utilizes a fully connected multilayer perceptron (MLP) with two hidden layers. Its primary task is to convert the visual features extracted by the vision module into language features suitable for processing by the language module. Specifically, the mapping module receives the visual features output by the vision module's head network and converts them into language features for subsequent use. Thanks to the inclusion of the mapping module, the entire translation network framework can achieve end-to-end training and learning under the joint supervision of the CTC loss and the translation loss. This design not only simplifies the training process but also improves the model's learning efficiency and accuracy, making the process of translating sign language videos into text smoother and more efficient.

[0103] Action relationship description

[0104] The action relationships between modules are connected through the flow of knowledge and information, as described below:

[0105] Lightweight mask module

[0106] The lightweight masking module's primary task is to address the massive and complex data processing challenges encountered during sign language translation. To address this challenge, the module implements a series of optimization measures to significantly reduce unnecessary data while ensuring that critical information is not lost.

[0107] Specifically, during the data preprocessing stage, advanced redundant frame removal technology is employed. By meticulously analyzing video sequences, this module intelligently identifies and removes redundant frames that do not contribute substantially to the expression of movements, significantly improving training efficiency. Furthermore, to further enhance model learning, the lightweight masking module incorporates a local dynamic masking module based on spatial attention technology. This effectively distinguishes foreground sign movements from background noise, allowing the model to focus more closely on the actual sign movement areas, improving the accuracy of feature extraction and understanding. In this way, the lightweight masking module not only reduces computational burden but also significantly improves model training. This process is the starting point for optimizing the entire sign language translation system, as it provides the foundation for data reduction and feature optimization for subsequent model training and performance improvement. By removing redundant frames during preprocessing and incorporating local dynamic masking technology to distinguish sign movements from background noise, the processed data from this module is used as input for the vision module, reducing unnecessary data while enhancing the model's focus on key information. This lays a solid foundation for subsequent reasoning and learning, enabling the entire system to achieve higher translation accuracy and efficiency while reducing computational burden.

[0108] Vision Module

[0109] The purpose of the vision module is to convert the sign language video optimized by the lightweight mask module into effective visual features. This process is a key link in the entire system optimization chain because it provides important data support for subsequent mapping and translation tasks.

[0110] Specifically, the vision module first receives the processed video data and converts it into visual features suitable for further analysis, which then serve as input to the mapping module.

[0111] like Figure 3 As shown in the figure, the visual module architecture consists of two parts: one is the video backbone network, which is responsible for extracting rich spatiotemporal feature information from the sign language video; the other is the lightweight head network, which is used to further encode information in the time dimension.

[0112] This design not only ensures accurate capture of action details within the video but also enhances the model's ability to understand how movements change over time. This allows the vision module to provide more accurate and comprehensive input information, laying a solid foundation for subsequent tasks. This architectural design ensures high accuracy while effectively improving processing efficiency, enabling smoother and more accurate sign language-to-text translation.

[0113] Mapping Module

[0114] The mapping module aims to bridge the gap between visual and linguistic modalities, supplementing the spatiotemporal visual information in sign language videos that gloss sequences cannot fully capture. In the Sign2Gloss2Text framework, while the vision module can convert sign language videos into gloss sequences, and the translation module converts these gloss sequences into spoken sentences, relying directly on gloss sequences as an intermediate representation has limitations and cannot fully capture the full information in the original video.

[0115] The mapping module converts the rich visual features extracted by the vision module into language features suitable for language module input, which not only makes up for the shortcomings of gloss representation, but also provides the language module with more video spatiotemporal information and language clues. This design enables the language module to understand the sign language content more comprehensively, thereby generating more accurate and rich spoken sentences. Specifically, the mapping module deeply processes the visual features and converts them into a form that not only contains key action information but also is easy for the language model to understand, which greatly enhances the understanding and expression capabilities of the entire translation system. In this way, the mapping module improves the accuracy of the conversion process from sign language video to spoken sentence, effectively overcomes the limitations of gloss as an intermediate representation, and improves the performance of the entire sign language translation system. Therefore, the mapping module plays a vital role in realizing an efficient and accurate sign language translation system, promoting barrier-free communication between people from different language backgrounds.

[0116] Language Module

[0117] By leveraging the language features converted by the mapping module and combining it with the encoder-decoder model mBART, the language can accurately calculate and predict the corresponding spoken text sentences. This process is the final step in the entire sign language translation system and provides the core technical support for efficient and accurate sign language-to-text conversion.

[0118] First, the encoder performs a deep analysis of the linguistic features obtained from the sign language video after processing by the mapping module, converting them into an intermediate representation containing key information. During this process, the encoder leverages advanced deep learning techniques to capture the complex structure and dynamic changes in the linguistic features, ensuring the integrity and accuracy of the information. This intermediate representation not only condenses the core content of the original sign language video but also lays the foundation for subsequent text generation. Next, the decoder generates natural, fluent, and semantically accurate spoken sentences based on this intermediate representation. When generating text, the decoder not only considers immediate context but also understands the semantic coherence of the entire sequence, significantly improving the quality of the output text. This approach not only enhances the understanding of the input features but also ensures that the generated spoken sentences are more consistent with the expressions used in real-world communication. This design not only enables the system to effectively convert sign language content into high-quality spoken text, but also significantly improves the accuracy and readability of the translation results. Therefore, the conversion process from linguistic features to spoken text is not only a key step in achieving an efficient and accurate sign language translation system but also a vital bridge for promoting cross-language communication.

[0119] Combine Figure 1 The relationship between each module in the entire system is closely linked and coordinated with each other. The present invention can significantly improve the efficiency of the sign language translation model, reduce the consumption of computing resources, while maintaining high model performance, and promote the automation process of sign language translation.

[0120] This paper proposes a redundant frame removal technique based on inter-frame differencing. This technique aims to intelligently remove redundant frames that have little impact on the final translation result from an input sign language video frame sequence by calculating the differences between multiple consecutive frames. Specifically, this method first performs a detailed analysis of consecutive frames in the sign language video. By calculating the degree of change between adjacent frames, it identifies frames that contribute little or nothing to the expression of the gestures. These redundant frames often contain repeated information or minor variations, which do not contribute substantially to the understanding of the sign language gestures. After determining which frames are redundant, the system automatically removes them, reducing unnecessary data processing. This process not only optimizes the data flow and reduces the computational burden, but also improves overall processing efficiency and model performance. Furthermore, by removing redundant frames, the system can focus more on analyzing and processing key frames, ensuring that every important sign language gesture detail is captured, thereby improving the accuracy and fluency of the translation. Overall, this inter-frame differencing-based method provides an efficient and accurate solution for sign language video preprocessing, enabling smoother subsequent visual feature extraction and translation tasks, ultimately achieving higher-quality sign language-to-text conversion. The application of this technology has greatly improved the performance of the entire sign language translation system, making it more practical and reliable.

[0121] When identifying redundant frames, in addition to using the inter-frame difference method for inference, methods such as the K-means clustering algorithm can also be used for inference. In this alternative approach, video frames can be treated as points in a high-dimensional space and clustered using the K-means algorithm. K-means is a commonly used clustering algorithm that divides a dataset into K clusters, making the data points within the same cluster as similar as possible, while the data points in different clusters are as different as possible. Specifically, it clusters the data by inputting the feature vector extracted from each frame into the K-means algorithm. The appropriate K value (i.e., the number of key frames required) is determined based on the length and complexity of the video. Then, for each cluster, the frame closest to the cluster center is selected as the key frame, and the remaining frames are redundant frames.

[0122] The proposed local dynamic masking module utilizes convolutional kernels of two different sizes to specifically capture diverse features in sign language videos. Specifically, the small convolutional kernel is used to capture subtle sign language motion features, such as facial expressions of the signer, while the large convolutional kernel focuses on extracting large-scale motion features, such as arm movements and body posture. First, the small convolutional kernel meticulously maps the location of each subtle motion and extracts corresponding feature values, which serve as the basis for capturing subtle variations. Next, the large convolutional kernel processes the larger motion region and extracts corresponding feature values ​​to cover the entire motion framework. By fusing the features extracted by these two convolutional kernels, we obtain a refined and comprehensive motion feature representation. This multi-layered feature extraction approach not only improves the model's ability to understand both subtle and significant motions but also optimizes overall system performance, making the conversion from sign language videos to text more accurate and smooth. This allows for the effective capture and interpretation of everything from subtle facial expressions to large body movements, providing solid technical support for high-quality sign language translation.

[0123] The core idea behind designing the number and size of convolution kernels is to obtain multi-scale and more comprehensive motion information. Small kernels are intended to capture subtle sign language motion features, such as changes in a sign language performer's facial expressions, while large kernels can extract large-scale motion region features of the human body, such as arm movements and body posture. By fusing the features extracted by these two kernels, a detailed and comprehensive motion feature representation can be obtained. The current solution uses a simple addition of the features extracted by the two kernels for fusion, but other fusion methods are also possible. For example, the two features can be superimposed and then convolved with a small kernel to obtain the fused feature. Although this approach increases computational cost and model complexity compared to simple addition, it may provide greater expressiveness and flexibility.

[0124] Simulation results:

[0125] A detailed simulation analysis was conducted on the recognition and translation effects of the system proposed in the present invention on the test set. The simulation results show that, regardless of whether the lightweight mask module is integrated, the visual module has a very high recognition accuracy for basic weather vocabulary (such as "REGEN" (rain), "SCHNEE" (snow) and "GEWITTER" (thunderstorm)) and common verb phrases (such as "VERSCHWINDEN" (disappear), "BLEIBEN" (keep) and "KOMMEN" (arrive)), and there is almost no misrecognition. However, for vocabulary related to region and direction (such as "REGION" (region), "NORD" (north), "NORDOST" (northeast)), misrecognition or confusion often occurs. Further analysis shows that, regardless of whether the lightweight mask module is added, the translation model of the present invention can generate natural and fluent results that are close to the reference translation when processing weather descriptions. However, for translations involving place and time, there are often large differences from the reference translation.

[0126] Specifically, the comparison Figure 7 The recognition sequence of the visual module in the reference gloss sequence was compared with the reference gloss sequence. It was found that both the basic visual module and the visual module combined with the lightweight mask module mistakenly recognized "REGION" in the reference gloss sequence as "ALPEN" (Alps). However, it is worth noting that only the basic visual module misidentified "NORD" (North) as "NORDOST" (Northeast), which shows that the recognition performance of the system has been improved after the lightweight mask module was added. In addition, by comparing Figure 7 The sentence translated by the sign language translation model and the reference translation shown in the figure shows that both models mistranslate the time "night" as "daytime" and also have some deviations in the location description. Overall, the translation model combined with the lightweight mask module achieves translation quality closer to the reference translation, showing higher accuracy and better semantic coherence.

[0127] In summary, although there is still room for improvement in the recognition of certain specific types of vocabulary, the present invention has significantly improved the performance of the sign language to spoken language translation system as a whole, especially by introducing the lightweight mask module.

Claims

1. A sign language translation system based on lightweight mask enhancement, characterized by: Includes lightweight mask module, vision module, mapping module and language module; The lightweight mask module takes the sign language video as input, removes redundant frames and combines local dynamic masking technology to distinguish the sign language movements from background interference; The visual module converts the sign language video optimized by the lightweight mask module into effective visual features; The mapping module converts the visual features extracted by the vision module into language features suitable for input by the language module; The language module uses the language features converted by the mapping module and combines it with the encoder-decoder model mBART to predict the corresponding spoken text sentences.

2. A sign language translation method based on lightweight mask enhancement, characterized in that: The following steps are involved: Step 1. During the initial processing of sign language video data, frame differencing techniques are used to identify and remove redundant frames that do not contribute substantially to the expression of the gestures. A local dynamic masking module based on spatial attention technology is then used to further distinguish the foreground and background information of the sign language gestures. Step 2. Input the sign language video optimized by the lightweight mask module into the vision module; Generate accurate and comprehensive visual features as input for the next processing; Step 3. Use the mapping module to convert the visual features obtained from the vision module into language features suitable for language processing; Step 4. The language module receives the language features converted by the mapping module and processes them in combination with the encoder-decoder model mBART to improve the quality of the output text.

3. The sign language translation method based on lightweight mask enhancement according to claim 2, characterized in that: In step 1, the changed area of ​​the inter-frame difference method is quickly detected by calculating the difference between adjacent frame images; When processing scenes with slow-moving targets, the two-frame difference method is used; When body movements or facial expressions change rapidly, the three-frame difference method is used to extract motion information.

4. The sign language translation method based on lightweight mask enhancement according to claim 3, characterized in that: The calculation process of the two-frame difference method is divided into four steps: (1) First, perform a subtraction operation on the grayscale values ​​of pixels at the same position in two consecutive frames of the sign language video data, i.e., fn(x, y) and fn-1(x, y), and take the absolute value of the result to obtain the differential image Dn; (2) Using the threshold T, each pixel in the differential image Dn is binarized to generate the image Rn. In this process, if the grayscale value of a pixel is 0, the pixel is identified as part of the background; and a pixel with a grayscale value of 255 is considered a sign of motion. (3) The process of identifying connected sets of foreground pixels in a binary image, i.e., determining which pixels belong to the same moving object; (4) Based on the results of the connectivity analysis, further extract information about each moving object. Finally, through the above processing steps, an image containing the complete moving object is extracted; D n (x,y)=∣f n (x,y)-f n-1 (x,y)∣ 5. The sign language translation method based on lightweight mask enhancement according to claim 3, characterized in that: The three-frame difference method is specifically as follows: Select three consecutive frames from the video stream, and then perform differential operations on two adjacent frames. n and D n+1 Calculate according to the following formula and extract motion information after processing; By adding a light-sensitive term to the threshold T to adapt to scenes with light changes, the influence of light changes on target motion judgment can be suppressed, as shown in the following formula: If the scene illumination changes slightly, the value will tend to zero; when the illumination changes dramatically, the value will increase significantly to dynamically increase the decision threshold; Where λ is the light suppression coefficient, N A is the total number of pixels in the area to be detected.

6. The sign language translation method based on lightweight mask enhancement according to claim 3, characterized in that: In step 1, the local dynamic mask module based on spatial attention technology is used to eliminate background effects; For the input feature map F that removes redundant frames that have no substantial contribution to the expression of the action, the Mask module will first calculate an attention score M based on F S (F), or it can be called spatial weight, the calculation formula of the Mask module attention score is as follows: Where F is the input feature map with size (B, C, T, H, W), where B is the batch size, C is the number of channels, T is the time dimension, H is the height, and W is the width; MaxPooling (·) and AveragePooling (·) in the above formula refer to the use of maximum pooling and average pooling on F in the channel dimension, respectively, to obtain two tensors of size (B, 1, T, H, W), and then the concatenation (·) operation to obtain a tensor of size (B, 2, T, H, W); conv in the above formula k1 (·) and conv k2 (·) is specifically implemented as a three-dimensional convolution operation with a kernel height and width of k1 and k2 and a time dimension of 2. k1 and k2 are adjusted as hyperparameters according to actual conditions. k1 is used to capture subtle sign language movement features, and k2 is used to capture large-scale human motion areas. Two tensors M of size (B, 1, T, H, W) are converted into S1 (F) and M S2 (F) is added to obtain the fusion features of both large and small scales; where f(·) is an arbitrary activation function, which is sigmoid; For the fusion feature of size (B, 1, T, H, W), the final attention score is obtained through the sigmoid function, which is the spatial weight M of size (B, 1, T, H, W). S (F); spatial weight M S The value of each pixel in (F) represents the importance of the corresponding position in the input feature map F, and M is obtained. S (F), use it to mask the input feature map F, and define the mask operation as Operation, the specific formula is as follows: Use two convolution kernels of different sizes to extract features of different scales in sign language movements, fuse them and output attention scores; finally, The operation is to multiply the elements at the same position in different channels of the input feature map by the element value of each corresponding position in the tensor to obtain the final output; among them, the attention score of (B, 1, T, H, W) is equivalent to the required Mask.

7. The sign language translation method based on lightweight mask enhancement according to claim 6, characterized in that: In step 2, the visual module architecture includes a video backbone network and a lightweight head network. The video backbone network is responsible for extracting rich spatiotemporal feature information from the sign language video; the lightweight head network is used to further encode information in the time dimension. The visual module first uses the S3D video backbone network to extract the spatiotemporal feature information in the video, and then further encodes the temporal information through a lightweight head network to ensure the understanding of the action features in the video and their changes over time.

8. The sign language translation method based on lightweight mask enhancement according to claim 7, characterized in that: In step 3, the mapping module uses a fully connected multilayer perceptron with two hidden layers to convert the visual features extracted by the vision module into language features suitable for processing by the language module; The mapping module receives the visual features output from the visual module head network and converts them into language features for subsequent use. The entire translation network framework can achieve end-to-end training and learning under the joint supervision of CTC loss and translation loss; The hidden layer is used to supplement the spatiotemporal visual information of the sign language video that cannot be fully encoded by the gloss sequence, providing the language module with richer and more detailed video spatiotemporal information and language clues.

9. The sign language translation method based on lightweight mask enhancement according to claim 8, characterized in that: In step 4, the encoder first converts the input language features into an intermediate representation that contains the key information of the sign language. Then, the decoder generates natural, fluent, and semantically accurate spoken sentences based on this intermediate representation, while taking into account the immediate context information and the semantic coherence of the entire sequence to improve the quality of the output text.

Images