A sign language translation method, system, electronic device and storage medium

Abstract

A sign language translation method, system, electronic device and storage medium relate to the field of video natural language generation, and alleviate the problems that existing sign language translation technology cannot consider gesture dynamic change and semantic information. A sign language translation method comprises a video processing step, uniformly sampling an RGB sign language video to be recognized to obtain a visual image sequence; a visual feature extraction step, obtaining visual features by passing the visual image sequence through a visual encoder; a language feature extraction step, encoding a plurality of predetermined sign language labels to obtain a plurality of corresponding text semantic features; a semantic alignment step, aligning the text semantic features and the visual features to obtain aligned text semantic features and visual features; and a translation step, completing the translation of the RGB sign language video to be recognized. The present application is suitable for the field of sign language recognition and translation, and is especially suitable for the field of semantic understanding of continuous and natural sign language videos.

Description

Technical Field

[0001] This invention relates to the field of video natural language generation, and more specifically to the field of sign language translation. Background Technology

[0002] Sign language translation is a key research task in the fields of human-computer interaction and accessible information access. Its goal is to automatically translate sign language using visual models to help people with hearing impairments overcome communication barriers. The core challenge lies in the joint modeling of fine-grained temporal motion patterns and semantic discriminative information.

[0003] Because sign language relies on hand shapes, movement trajectories, and the coordinated action of hands, face, and body, and possesses a unique linguistic structure and logic, automated sign language translation is far more complex than ordinary motion recognition. Furthermore, differences exist in gesture execution among different sign language users, and the semantic meaning of gestures is heavily influenced by context. These characteristics present significant challenges to automated sign language recognition modeling. Therefore, designing a translation model that can simultaneously capture dynamic changes in gestures and semantic information is of great importance for sign language translation.

[0004] In recent years, deep models pre-trained for vision and language have shown great potential in various vision tasks. These models possess strong spatial features and semantic understanding capabilities by mapping static images and natural language descriptions to a shared semantic space. However, sign language vocabulary is highly similar at the static appearance level, and the translation process often relies only on short-term, local motion differences for differentiation, lacking the inherent ability to model the temporal dynamic information of videos, resulting in inaccurate sign language translation.

[0005] In the prior art, Chinese patent document CN114492796A discloses "a multi-task learning sign language translation method based on syntax tree". This technical solution translates sign language words to form a sign language word sequence to complete a short description of the sign language video. The essence of this technical solution is still only translating the static appearance of sign language, and does not obtain dynamic information from the sign language video.

[0006] In summary, existing sign language translation technologies suffer from several problems: they cannot simultaneously capture the dynamic changes in gestures and semantic information; there is a high degree of similarity in the static appearance of sign language; and sign language translation is inaccurate. Summary of the Invention

[0007] This invention alleviates the problems of existing sign language translation technologies, such as the inability to simultaneously consider dynamic changes in gestures and semantic information, the high similarity of static appearances in sign language, and inaccurate sign language translation.

[0008] Option 1: A sign language translation method, comprising the following steps: Video processing steps: Uniformly sample the RGB sign language video to be recognized to obtain a visual image sequence; Visual feature extraction step: The visual image sequence is passed through a visual encoder to obtain visual features; Language feature extraction steps: Encode several pre-set sign language tags to obtain corresponding text semantic features; Semantic alignment step: Perform the following operations on each of the several text semantic features to obtain several sets of aligned text semantic features and visual features: The visual features and the text semantic features are used to obtain aligned text semantic features and visual features through a cross-modal semantic perception model; Translation steps: Based on the semantic alignment steps, several sets of aligned text semantic features and visual features are used to obtain several similarities; based on the several similarities, a matching score matrix is ​​obtained; based on the matching score matrix, the translation of the RGB sign language video to be identified is completed.

[0009] Furthermore, in one embodiment of the present invention, the visual encoder includes an improved Transformer encoder, a temporal enhancement module, and a pooling module, wherein the visual image sequence is processed sequentially by the improved Transformer encoder, the temporal enhancement module, and the pooling module to obtain visual features.

[0010] Furthermore, in one embodiment of the present invention, the improved Transformer encoder is to add an inter-frame differential sensing layer (FDA) between the self-attention layer (MSA) and the feedforward neural network layer (FFN) of the existing Transformer encoder. The self-attention layer MSA is used to obtain the corresponding self-attention feature sequence based on the visual image sequence. The inter-frame differential sensing layer (FDA) is used to obtain inter-frame differential sensing features based on the self-attention feature sequence. .

[0011] Furthermore, in one embodiment of the present invention, the inter-frame differential sensing layer (FDA) is used to obtain a visual token sequence from the self-attention feature sequence through dimension mapping. ; Also used to pass

[0012]

[0013] Obtain instantaneous motion characteristics and acceleration characteristics ,in, ; Also used to pass

[0014] Obtain inter-frame differential information ,in, for Linear projection between; combining the inter-frame difference information of each image in the visual image sequence into an inter-frame difference information sequence motion token; It is also used to combine the inter-frame difference information sequence motion token with the visual token sequence. By splicing, we can obtain ; Also used to pass

[0015] Obtain initial features ; It is also used to define the initial features. After removing the inter-frame difference information sequence motion token, the modulated visual token sequence is obtained. ; Also used to pass

[0016] Obtain inter-frame differential sensing features .

[0017] Furthermore, in one embodiment of the present invention, the cross-modal semantic perception model includes a visual text attention module and a text visual attention module; The visual text attention module is used to focus on the visual features. and the text semantic features Obtain the aligned text semantic features ; The text visual attention module is used to focus on the visual features. and the text semantic features Obtain the aligned visual features .

[0018] Furthermore, in one embodiment of the present invention, the text semantic features ,pass

[0019] Obtain, among which, This is a multi-head attention mechanism, where the superscript 't' represents the text modality; superscript... The visual modality is represented; the subscripts Q, K, and V represent the query, key, and value roles in the attention mechanism, respectively. This is the projected learnable weight matrix.

[0020] Furthermore, in one embodiment of the present invention, the visual feature ,pass

[0021] get.

[0022] Option 2: A sign language translation system, comprising the following modules: Module 1 is used to uniformly sample the RGB sign language video to be recognized to obtain a visual image sequence; Module 2 is used to obtain visual features by passing the visual image sequence through a visual encoder; Module 3 is used to encode several sign language tags to obtain corresponding text semantic features; Module four is used to perform the following operations on each of the several text semantic features to obtain several sets of aligned text semantic features and visual features: The visual features and the text semantic features are used to obtain aligned text semantic features and visual features through a cross-modal semantic perception model; Module 5 is used to obtain several similarities based on several sets of aligned text semantic features and visual features obtained in Module 4; to obtain a matching score matrix based on the several similarities; and to complete the translation of the RGB sign language video to be identified based on the matching score matrix.

[0023] Option 3: An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the method described in Scheme 1.

[0024] Option 4: A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in Option 1.

[0025] The sign language translation method, system, electronic device, and storage medium described in this invention are based on motion perception, effectively alleviating the problems of existing sign language translation technologies, such as the inability to simultaneously consider dynamic changes in gestures and semantic information, the high similarity of static appearances in sign language, and inaccurate sign language translation. Specific beneficial effects include: 1. The sign language translation method described in this invention establishes a general sign language translation framework, SignMotion, based on pure visual input through video processing, visual feature extraction, language feature extraction, semantic alignment, and translation. This framework specifically integrates motion perception and cross-modal semantic interaction mechanisms to simultaneously model the temporal dynamics and linguistic semantic information of sign language without requiring complex human posture information as prior guidance. Furthermore, it can achieve collaborative translation of the temporal structure of sign language actions and linguistic semantic information, improving temporal modeling capabilities, reducing cross-modal semantic alignment bias, and avoiding reliance on additional modal information, thereby improving the accuracy of sign language translation.

[0026] 2. The sign language translation method of this invention employs an inter-frame differential perceptual layer (FDA) in the visual feature extraction step. This effectively suppresses static background interference and explicitly extracts the differential information between adjacent frames at both the frame and feature levels, injecting it as attention guidance into the self-attention calculation. Through the FDA, the visual encoder can significantly enhance its sensitivity to short-term motion trajectories, transient displacements, and subtle changes in hand details while maintaining its original spatial representation capabilities. This allows for better differentiation of sign language words that are highly similar in static appearance but different in motion patterns.

[0027] 3. The sign language translation method described in this invention employs a cross-modal semantic awareness model (CSA) in its semantic alignment step. This bridges the gap between visual dynamics and categorical text semantics, enabling fine-grained semantic interaction between visual features and text category embeddings. By optimizing text branch features using visual feature representations, the visual and text representations of sign languages ​​within the same category are more tightly coupled in the vector space, while dissimilar categories exhibit greater separation. This approach balances gesture dynamics with semantic information, avoiding inaccurate translation. This process does not introduce additional supervision signals but is instead embedded as a lightweight cross-modal alignment mechanism within the overall model.

[0028] 4. The sign language translation method described in this invention forms a complementary mechanism between the inter-frame differential perception layer (FDA) and the cross-modal semantic perception model (CSA). The former amplifies motion signals and improves temporal representation from the visual end, while the latter tightens category prototypes and improves cross-modal alignment from the semantic end, which can effectively improve the performance of sign language recognition in terms of fine-grained discrimination and semantic consistency.

[0029] The method described in this invention is applicable to the field of sign language recognition and translation, and is particularly suitable for high-precision, end-to-end semantic understanding and category discrimination of continuous, natural sign language videos. Furthermore, this application can also be widely applied to the following specific scenarios and sub-fields: educational sign language translation, legal sign language translation, public service sign language interaction, remote video communication assistance, and the development of sign language video databases. Attached Figure Description

[0030] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of the sign language translation method described in Implementation Method 1. Detailed Implementation

[0031] Various embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. The embodiments described with reference to the drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0032] Implementation Method 1: A sign language translation method described in this implementation method includes the following steps: Video processing steps: Uniformly sample the RGB sign language video to be recognized to obtain a visual image sequence; Visual feature extraction step: The visual image sequence is passed through a visual encoder to obtain visual features; Language feature extraction steps: Encode several pre-set sign language tags to obtain corresponding text semantic features; Semantic alignment step: Perform the following operations on each of the several text semantic features to obtain several sets of aligned text semantic features and visual features: The visual features and the text semantic features are used to obtain aligned text semantic features and visual features through a cross-modal semantic perception model; Translation steps: Based on the semantic alignment steps, several sets of aligned text semantic features and visual features are used to obtain several similarities; based on the several similarities, a matching score matrix is ​​obtained; based on the matching score matrix, the translation of the RGB sign language video to be identified is completed.

[0033] In this embodiment, during the translation step, the similarity... pass

[0034] Obtain, among which, For the first A visual feature, For the first Textual semantic features.

[0035] In this embodiment, a matching score matrix is ​​obtained based on the similarity. This matching score matrix is ​​a two-dimensional matrix of dimension B×K, where B represents the input video batch size and K represents the total number of preset sign language tags. The element S in the b-th row and k-th column of this matrix... b,k That is, the visual feature vector of the b-th video sample. The text semantic feature vector corresponding to the kth sign language label The cosine similarity between them is calculated as follows: .

[0036] The matching score matrix is ​​preferably obtained through the CLIP (Contrastive Language–Image Pretraining) model, used to achieve cross-modal image-text matching and zero-shot classification. This ensures the effectiveness of feature space alignment, and further enhances feature quality and alignment accuracy through the aforementioned inter-frame differential sensing module (FDA) and cross-modal semantic sensing module (CSA), thereby achieving better recognition performance.

[0037] In this embodiment, the translation of the RGB sign language video to be recognized is completed based on the matching score matrix. Specifically: All predefined sign language category labels are converted into a set of text semantic feature vectors using a text encoder. The similarity between visual features and each text feature is calculated to form a Bi. A matching score matrix of K (B is the batch size, K is the total number of categories). For each video sample (i.e., each row of the matrix), the category with the highest score is selected as the final translation / recognition result. The similarity is preferably cosine similarity.

[0038] The sign language translation method described in this embodiment proposes a motion-aware visual-language sign language translation framework, SignMotion. Based on a unified visual-text representation, SignMotion constructs an end-to-end trainable dual-branch system tailored to the temporal and semantic characteristics of sign language vocabulary. This preserves the semantic transfer capabilities of large-scale visual-language pre-training and achieves competitive performance on multiple publicly available sign language datasets. Furthermore, its modular design balances temporal dynamic modeling with semantic refinement, eliminating the need for hand skeleton points or pose information.

[0039] This implementation uses only the original RGB video as input in the video processing step, avoiding complex and error-prone pose estimation algorithms, simplifying the system process, and improving deployment flexibility and robustness.

[0040] This implementation introduces an inter-frame differential sensing layer (FDA) and a cross-modal semantic sensing model (CSA) to improve the ability to recognize fine-grained sign language movements without adding extra modal information.

[0041] Implementation Method 2: This implementation method further defines the sign language translation method described in Implementation Method 1. In this implementation method, the visual encoder includes an improved Transformer encoder, a temporal enhancement module, and a pooling module. The visual image sequence is processed sequentially through the improved Transformer encoder, the temporal enhancement module, and the pooling module to obtain visual features.

[0042] This embodiment further defines the visual feature extraction step and provides an example of a visual encoder. In this embodiment, the improved Transformer encoder extracts the deep global semantic information of the video segment; the temporal enhancement module uses channel attention as the core to model and compress the inter-frame relationship in the temporal dimension in order to learn the temporal dependency of sign language actions.

[0043] Implementation Method 3: This implementation method is a further limitation of the sign language translation method described in Implementation Method 2. In this implementation method, the improved Transformer encoder is to add an inter-frame difference sensing layer (FDA) between the self-attention layer (MSA) and the feedforward neural network layer (FFN) of the existing Transformer encoder. The self-attention layer MSA is used to obtain the corresponding self-attention feature sequence based on the visual image sequence. The inter-frame differential sensing layer (FDA) is used to obtain inter-frame differential sensing features based on the self-attention feature sequence. .

[0044] In this embodiment, the feedforward neural network layer (FFN) is used to perform the same nonlinear transformation on each input feature vector to obtain a feature sequence. This transformation typically consists of two fully connected layers with an activation function sandwiched in between, preferably GELU or ReLU.

[0045] Its mathematical expression is usually as follows: FFN(X in = W2 Activation(W1 X in +b1)+b2 Where W1, b1, W2, b2 are learnable weights and bias parameters.

[0046] The feedforward neural network layer FFN is used to perform further nonlinear feature transformation and dimension mapping on the features extracted by the MSA and FDA layers to enhance the expressive power of the model.

[0047] This embodiment further refines the visual encoder, illustrating an improved Transformer encoder. This method introduces an inter-frame differential perception layer (FDA) that feeds enhanced features back to the backbone network via residual connections, enabling the visual encoder to possess explicit motion perception capabilities at each layer. In this way, the FDA effectively amplifies key dynamic cues in sign language videos, helping to suppress interference from background and irrelevant static information, providing a more discriminative visual representation foundation for subsequent temporal aggregation and cross-modal semantic alignment. In this embodiment, the FDA is embedded within the self-attention layer (MSA), guiding the model to focus more on discriminative temporal change regions without introducing additional supervision or prior knowledge of human anatomy, effectively enhancing the model's sensitivity to temporal changes in sign language actions. Implementation Method Four: This implementation method further defines the sign language translation method described in Implementation Method Three. In this implementation method, the inter-frame differential perception layer (FDA) is used to obtain a visual token sequence from the self-attention feature sequence through dimensional mapping. ; Also used to pass

[0048]

[0049] Obtain instantaneous motion characteristics and acceleration characteristics ,in, ; Also used to pass

[0050] Obtain inter-frame differential information ,in, for Linear projection between; combining the inter-frame difference information of each image in the visual image sequence into an inter-frame difference information sequence motion token; It is also used to combine the inter-frame difference information sequence motion token with the visual token sequence. By splicing, we can obtain ; Also used to pass

[0051] Obtain initial features ; It is also used to define the initial features. After removing the inter-frame difference information sequence motion token, the modulated visual token sequence is obtained. ; Also used to pass

[0052] Obtain inter-frame differential sensing features .

[0053] In this embodiment, the splicing is performed along the spatial token sequence dimension.

[0054] In this embodiment, the For visual input sequence, ,in, For the number of feature tokens, Batch Size For frames (Time steps), For feature dimensions; preferably for the visual input sequence Perform data augmentation.

[0055] In this implementation, the FDA introduces frame-difference information (FDI) based on the continuous-time modeling concept. The motion information can be naturally described in the form of time derivatives. The first time derivative reflects the instantaneous velocity, while the second time derivative describes the acceleration characteristics of the motion change. Together, they describe the dynamic evolution of the action. , where v(t) and a(t) represent the velocity and acceleration characteristics of the sign language movement in the feature space, respectively.

[0056] In practical engineering implementation, since video is a discretely sampled sequence, the aforementioned continuous derivative can be approximated by finite differences. Given a length of... The video sequence, its first The feature representation corresponding to the frame is denoted as Then the first-order and second-order inter-frame differences can be expressed as:

[0057]

[0058] in, and In mathematics, this corresponds to the discrete approximation of the continuous-time derivative, which can effectively capture key information such as hand displacement, direction changes, and movement rhythm.

[0059] This implementation further refines the improved Transformer encoder, illustrating it with an example of the inter-frame difference perception layer (FDA). This method explicitly models inter-frame differences within the Transformer, effectively capturing instantaneous dynamic changes in sign language movements and significantly improving the ability to distinguish subtle differences in motion. It highlights hand movement areas through inter-frame difference information to capture dynamic changes in adjacent frames, enhancing fine-grained motion modeling capabilities. This avoids the difficulty in distinguishing sign languages ​​that are highly similar in spatial structure but different in motion patterns due to their static appearance in a single frame.

[0060] Implementation Method 5: This implementation method further defines the sign language translation method described in Implementation Method 1. In this implementation method, the cross-modal semantic perception model includes a visual text attention module and a text visual attention module. The visual text attention module is used to focus on the visual features. and the text semantic features Obtain the aligned text semantic features ; The text visual attention module is used to focus on the visual features. and the text semantic features Obtain the aligned visual features .

[0061] This implementation further defines the semantic alignment step and provides an example of a cross-modal semantic perception model. It consists of two symmetrical but complementary attention calculation processes: visual-guided text semantic perception and text-guided visual semantic perception. This implementation interactively aligns visual and text features through a bidirectional cross-attention mechanism, enabling mutual guidance and dynamic modulation between them to achieve fine-grained semantic alignment. It realizes bidirectional, adaptive interaction between visual and text features, effectively bridging the semantic gap between modalities and improving matching accuracy.

[0062] Implementation Method Six: This implementation method further defines the sign language translation method described in Implementation Method Five. In this implementation method, the text semantic features... ,pass

[0063] Obtain, among which, This is a multi-head attention mechanism, where the superscript 't' represents the text modality; superscript... The visual modality is represented; the subscripts Q, K, and V represent the query, key, and value roles in the attention mechanism, respectively. This is the projected learnable weight matrix.

[0064] In this embodiment, the multi-head attention mechanism Attn( This is used to model long-range dependencies between elements in an input sequence. It is also used to fuse textual semantic features T and visual features V to achieve cross-modal alignment and information enhancement.

[0065] Original text semantic features (N is the number of text tokens, C is the embedding dimension) as the basic representation; cross-modal attention is used. Extract visual context information related to the current text token from visual feature V; add the attention output (with the same shape as T) to the original T residual to obtain the enhanced text semantic features. .Right now It is a text semantic feature obtained by visually guided attention residual enhancement based on the original text feature T.

[0066] This implementation further defines the cross-modal semantic perception model, focusing on text semantic features. An example is provided to illustrate that this implementation adaptively reweights text semantics based on the current video content, enabling text features to focus on the semantic dimensions most relevant to the current video motion pattern, thereby mitigating the semantic overlap problem between different categories of text prototypes. This implementation can more accurately distinguish visually similar but semantically different gestures with the help of enhanced text priors. By optimizing text branch features using visual feature representations, the visual and text representations of the same category of sign language are more closely integrated in the vector space, resulting in greater separation between dissimilar categories. This helps the Cross-Modal Semantic Awareness (CSA) model reduce ambiguity caused by relying solely on visual or textual representations.

[0067] Implementation Method Seven: This implementation method further defines the sign language translation method described in Implementation Method Five. In this implementation method, the visual features... ,pass

[0068] get.

[0069] This implementation further defines the cross-modal semantic perception model, focusing on visual features. The explanation guides visual features to further converge on the semantic space, which helps the cross-modal semantic perception model (CSA) reduce ambiguity caused by relying solely on visual or textual features.

[0070] Implementation method eight: A sign language translation system according to this implementation method includes the following modules: Module 1 is used to uniformly sample the RGB sign language video to be recognized to obtain a visual image sequence; Module 2 is used to obtain visual features by passing the visual image sequence through a visual encoder; Module 3 is used to encode several sign language tags to obtain corresponding text semantic features; Module four is used to perform the following operations on each of the several text semantic features to obtain several sets of aligned text semantic features and visual features: The visual features and the text semantic features are used to obtain aligned text semantic features and visual features through a cross-modal semantic perception model; Module 5 is used to obtain several similarities based on several sets of aligned text semantic features and visual features obtained in Module 4; to obtain a matching score matrix based on the several similarities; and to complete the translation of the RGB sign language video to be identified based on the matching score matrix.

[0071] Implementation Method Nine: An electronic device according to this implementation method includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the method described in any one of embodiments one through seven.

[0072] Embodiment 10: A computer-readable storage medium according to this embodiment stores a computer program, which, when executed by a processor, implements the method described in any one of Embodiments 1 to 7.

[0073] Implementation method eleven: A training method for a sign language translation system, wherein the training set used includes the WLASL dataset, IllumSLR dataset, SLR500 dataset and Kinetics-400 dataset; The WLASL dataset includes subsets of different sizes, such as WLASL100, WLASL300, or WLASL2000, to evaluate the performance of the model at different vocabulary sizes. The IllumSLR dataset includes subsets of IllumSLR100, IllumSLR200, IllumSLR500, or IllumSLR700. This dataset contains sign language videos under different lighting conditions to verify the robustness of the model in complex lighting environments. The SLR500 dataset is a large-scale continuous sign language recognition dataset containing 500 words. The Kinetics-400 dataset is a pre-trained related dataset, a classic human action recognition dataset used for pre-training models.

[0074] In this embodiment, the sign language translation system is trained using a loss function based on the training set. The sign language translation system is the same as the sign language translation system described in Implementation Method 8.

[0075] The loss function is a symmetric cross-entropy loss function, configured to process the visual input sequence after blending enhancements (such as Mixup or CutMix) and its corresponding soft labels. It is configured to minimize the modal difference between visual features and text features. During the calculation, the loss function introduces a blending enhancement coefficient λ to weight and sum the supervision signals of two different samples to adapt to the visual input sequence X. aug This improves the model's generalization ability to complex backgrounds and lighting variations. Specifically, the visual input sequence X... aug Its corresponding label Y aug The total loss function L is formed by linearly combining the original label Y1 and the mixed label Y2 according to the mixing coefficient λ. total Including the visual-to-text classification loss L v2t And text-to-visual classification loss L t2v The calculation formula is as follows:

[0076] Among them, the classification loss from visual to text L v2t Used to measure the probability distribution of model predictions and the hybrid augmentation label Y aug The differences between them are calculated as follows:

[0077] Where λ is the mixing coefficient, H( ) represents the cross-entropy calculation function, P v2t Let L represent the probability distribution of visual to text predictions by the model; similarly, let L be the text-to-visual classification loss. t2v The same weighting method is used for calculation.

[0078] In this embodiment, the training parameters used include learning rate, batch size, optimizer, and number of training epochs. The learning rate adopts a hierarchical learning rate strategy, with the base learning rate of the visual encoder set to 1e-4 and the learning rate of the text encoder set to 5e-5. The batch size is dynamically adjusted according to the video memory capacity, and in the script configuration, the preferred range is 8 to 32. The optimizer is preferably the AdamW optimizer, with the weight decay factor set to 0.05; The number of training rounds is used for end-to-end training on the training dataset, and the total number of training rounds is preferably 50 to 100 rounds.

Claims

1. A sign language translation method, characterized in that, Includes the following steps: Video processing steps: Uniformly sample the RGB sign language video to be recognized to obtain a visual image sequence; Visual feature extraction step: The visual image sequence is passed through a visual encoder to obtain visual features; Language feature extraction steps: Encode several pre-set sign language tags to obtain corresponding text semantic features; Semantic alignment step: Perform the following operations on each of the several text semantic features to obtain several sets of aligned text semantic features and visual features: The visual features and the text semantic features are used to obtain aligned text semantic features and visual features through a cross-modal semantic perception model; Translation steps: Based on the semantic alignment steps, several sets of aligned text semantic features and visual features are used to obtain several similarities; based on the several similarities, a matching score matrix is ​​obtained; based on the matching score matrix, the translation of the RGB sign language video to be identified is completed.

2. The sign language translation method according to claim 1, characterized in that, The visual encoder includes an improved Transformer encoder, a temporal enhancement module, and a pooling module. The visual image sequence is processed sequentially through the improved Transformer encoder, the temporal enhancement module, and the pooling module to obtain visual features.

3. The sign language translation method according to claim 2, characterized in that, The improved Transformer encoder is to add an inter-frame differential sensing layer (FDA) between the self-attention layer (MSA) and the feedforward neural network layer (FFN) of the existing Transformer encoder. The self-attention layer MSA is used to obtain the corresponding self-attention feature sequence based on the visual image sequence. The inter-frame differential sensing layer (FDA) is used to obtain inter-frame differential sensing features based on the self-attention feature sequence. .

4. The sign language translation method according to claim 3, characterized in that, The inter-frame differential sensing layer (FDA) is used to obtain a visual token sequence from the self-attention feature sequence through dimension mapping. ; Also used to pass Obtain instantaneous motion characteristics and acceleration characteristics ,in, ; Also used to pass Obtain inter-frame differential information ,in, for Linear projection between; combining the inter-frame difference information of each image in the visual image sequence into an inter-frame difference information sequence motion token; It is also used to combine the inter-frame difference information sequence motion token with the visual token sequence. By splicing, we can obtain ; Also used to pass Obtain initial features ; It is also used to define the initial features. After removing the inter-frame difference information sequence motion token, the modulated visual token sequence is obtained. ; Also used to pass Obtain inter-frame differential sensing features .

5. The sign language translation method according to claim 1, characterized in that, The cross-modal semantic perception model includes a visual text attention module and a text visual attention module; The visual text attention module is used to focus on the visual features. and the text semantic features Obtain the aligned text semantic features ; The text visual attention module is used to focus on the visual features. and the text semantic features Obtain the aligned visual features .

6. The sign language translation method according to claim 5, characterized in that, The semantic features of the text ,pass Obtain, among which, This is a multi-head attention mechanism, where the superscript 't' represents the text modality; superscript... The visual modality is represented; the subscripts Q, K, and V represent the query, key, and value roles in the attention mechanism, respectively. This is the projected learnable weight matrix.

7. The sign language translation method according to claim 5, characterized in that, The visual features ,pass get.

8. A sign language translation system, characterized in that, Includes the following modules: Module 1 is used to uniformly sample the RGB sign language video to be recognized to obtain a visual image sequence; Module 2 is used to obtain visual features by passing the visual image sequence through a visual encoder; Module 3 is used to encode several sign language tags to obtain corresponding text semantic features; Module four is used to perform the following operations on each of the several text semantic features to obtain several sets of aligned text semantic features and visual features: The visual features and the text semantic features are used to obtain aligned text semantic features and visual features through a cross-modal semantic perception model; Module 5 is used to obtain several similarities based on several sets of aligned text semantic features and visual features obtained in Module 4; to obtain a matching score matrix based on the several similarities; and to complete the translation of the RGB sign language video to be identified based on the matching score matrix.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-7.

Images