A novel word-level contrastive learning framework for sign language translation and a sign language translation system
By introducing the word-level contrastive learning framework ConSLT, the performance limitations of low-resource sign language translation systems are addressed, resulting in more accurate and fluent sign language translation applicable to various sign language processing tasks.
Patent Information
- Application Number
- CN202310062798.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-16
- Publication Date
- 2025-12-23
- Estimated Expiration
- 2043-01-16
AI Technical Summary
Existing sign language translation systems suffer from insufficient translation performance under low-resource conditions due to difficulties in dataset collection and annotation, making accurate and fluent translation impossible.
A novel word-level contrastive learning framework, ConSLT, is employed, which includes modules for video input, visual extraction, sign language encoding, sentence embedding, sign language decoding, contrastive learning, and loss calculation. By constructing positive and negative pairs, the model is optimized using CTC, cross-entropy, and KL divergence loss functions to achieve end-to-end sign language translation.
Significantly improves the accuracy and fluency of sign language translation under low resource conditions, with a performance improvement of over 1.4 BLEU, and is suitable for various sign language-related natural language processing and vision tasks.
Smart Images
Figure CN116092191B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the fields of computer vision, natural language processing and sign linguistics, and in particular, to a novel word-level contrastive learning framework for sign language translation and a sign language translation system. BACKGROUND
[0002] Sign language is a visual language used in daily communication by the hearing impaired community. Sign language translation (SLT) can play an important role in narrowing the communication gap between the hearing impaired and the normal people. Therefore, SLT has received increasing attention in the research community. SLT takes sign language videos as input and generates a natural spoken sentence. The most popular SLT dataset currently contains less than 9K parallel sign language videos, sign language vocabulary and spoken sentences. The collection and annotation of sign language datasets are very difficult and expensive. Therefore, the SLT task is essentially a low-resource problem.
[0003] Currently, contrastive learning has become a very popular technique in the fields of computer vision and natural language processing, and its performance is very stable. Recent work has also demonstrated the great potential of contrastive learning in low-resource scenarios. SUMMARY
[0004] The first object of the present application is to break through the translation bottleneck of sign language in low-resource conditions, take advantage of contrastive learning, alleviate the low-resource situation of sign language translation, improve the performance of sign language translation, and provide a novel word-level contrastive learning framework ConSLT (Contrastive Framework for Sign Language Translation) for sign language translation.
[0005] The second object of the present application is to provide a sign language translation method based on contrastive learning that can also learn good sign language representations in low-resource situations.
[0006] The third object of the present application is to provide a sign language translation system based on contrastive learning that is more accurate and fluent.
[0007] The novel word-level contrastive learning framework (referred to as ConSLT framework) for sign language translation comprises a video input module, a visual extraction module, a sign language encoding module, a sentence embedding module, a sign language decoding module, a contrastive learning module, a loss calculation module and an output module.
[0008] The video input module is used for sign language translation data selection modeling, and the sign language video is input into the model in the form of video frames;
[0009] The visual extraction module is used for extracting features of the sign language video by using a convolutional neural network for each frame of the sign language video, so as to obtain sequence features containing sign language visual information.
[0010] The sign language encoding module is used for obtaining a hidden representation by using an encoder-decoder for the received visual feature vector; the sign language encoding module is provided with an attention mechanism;
[0011] The sentence embedding module is used for inputting a corresponding spoken language sentence of the sign language video as an answer in a training stage;
[0012] The sign language decoding module is used for generating a translation result by using a feature vector of the encoding module after receiving an output vector of the sign language encoding module; the sign language decoding module is provided with a mask attention mechanism;
[0013] The contrast learning module is used for constructing a positive example pair and a negative example pair in a training stage;
[0014] The loss calculation module is used for calculating an alignment loss of a sign language word annotation by using a connectionist temporal classification (CTC) loss function, calculating a translation loss of a generated word by using a cross-entropy loss function, and calculating a contrast loss of a decoding module output obtained by outputting the sign language video twice by using a KL divergence;
[0015] The output module is used for outputting a spoken language sentence obtained by using a sign language translation model for the sign language video.
[0016] The present application provides a sign language translation method based on contrast learning, comprising the following steps:
[0017] 1) sign language corpus selection modeling;
[0018] 2) sign language visual feature extraction;
[0019] 3) end-to-end sign language video conversion;
[0020] 4) sentence embedding in a training stage;
[0021] 5) constructing a positive example pair and a negative example pair;
[0022] 6) sign language translation model loss calculation;
[0023] 7) sign language translation result output.
[0024] In step 1), the sign language corpus selection modeling is performed by inputting sign language videos in the sign language corpus into the model in the form of video frames through a video input module;
[0025] The sign language corpus includes sign language videos, sign language word annotations, and spoken sentence triplets, denoted as D={(x,z,y)}∈(X,Z,Y); wherein X, Z, and Y represent sign language videos, sign language word annotations, and spoken sentences, respectively;
[0026] Specifically, sign language videos representing T frames, is the corresponding sign language word annotation sequence, is the corresponding spoken sentence; the sign language word annotation sequence is a word-by-word sign language transcription, and due to the significant difference between sign language word annotations and spoken grammar, the word order of the sign language word annotation sequence is different from that of the spoken sentence; T z and T y are the lengths of the sign language word annotation sequence and the spoken sentence, respectively.
[0027] In step 2), the sign language visual feature extraction is performed by using a convolutional neural network to extract features from the sign language video for each frame of the sign language video in the visual extraction module, obtaining a sequence of features containing sign language visual information.
[0028] In step 3, the end-to-end sign language video conversion is performed by sending the extracted visual feature vector to the sign language encoding module, obtaining a hidden representation through the encoder-decoder; after receiving the output vector of the sign language encoding module, the sign language decoding module generates a translation result through the feature vector of the encoding module;
[0029] Specifically, the sign language video in the sign language corpus is modeled and converted from video to text using a transformer-based encoder-decoder structure;
[0030] The sign language encoding module is used to learn meaningful spatiotemporal representations and sign language representations, and to encode visual information into a text space by aligning sign language word annotations in sign language videos, providing assistance for sign language translation tasks; the sign language word annotations in the sign language video are aligned using CTC loss; the sign language encoding module outputs a feature vector containing sign language semantic information; the sign language encoding module has an attention mechanism, which breaks the restriction of input sequence length, and is a technology that enables the model to focus on important information and fully learn and absorb;
[0031] The frame-level representation of the sign language video is extracted by a pre-trained sign language embedding network; positional information is added to the frame-level representation by a positional encoding (PE) method for the input embedding of the encoder; the input is sent to the transformer encoder module to generate a hidden representation of the sign language video; these operations are expressed as:
[0032] f i =SignEmbedding(x i ), (1)
[0033]
[0034]
[0035] where i∈[1,Tx] denotes the i-th frame of the video, SignEmbedding denotes the pre-trained sign language embedding network, PE denotes the input embedding of the encoder adding position information to the frame-level representation, Encoder denotes the encoder;
[0036] After receiving the vector of the output of the sign language encoding module, there is a begin identifier, and after identifying the begin, the sign language decoding module generates a translation result through the feature vector of the encoding module; each step of the decoding stage outputs a word from the output sequence; the output of each step is input to the bottom decoder in the next time step, so that the decoding result is output to the higher layer; position encoding is embedded and added in these decoder inputs to indicate the position of each word, and the process is repeated until the end identifier appears, indicating that the decoding module has completed the output.
[0037] The sign language decoding module has a mask attention mechanism, mainly because decoding is a sequential operation process, and when predicting a word at the k-th time step, only the first k-1 prediction results can be seen, so the subsequent part of the content needs to be masked; at the same time, in the attention mechanism of the decoder, K and V come from the output of the encoder, and Q comes from the last output of the decoder;
[0038] These operations are expressed as:
[0039] w t = WordEmbedding(y t ), (4)
[0040]
[0041]
[0042] o t = softmax(Wh t +b). (7)
[0043] where the conditional probability p(y|x) and the objective function are calculated as follows:
[0044]
[0045]
[0046] In step 4), the specific steps of the training phase sentence embedding can be: in the training phase, the sentence embedding module inputs the corresponding spoken language sentence y of the sign language video x as the answer in the training phase; the spoken language sentence y is converted into a sentence embedding, and the character representation in the text is converted into a vector representation. In order to capture the relationship between words in a high-dimensional space, a linear layer is used to project the character representation into a high-dimensional space.
[0047] In step 5), the specific steps of constructing positive and negative pairs can be:
[0048] A symbol video x is input into the model twice by using different dropout sampling; due to the dropout mechanism in the transformer, part of the units are randomly discarded, and for each word y t Two different hidden representations are obtained, denoted as h t And For each word y t A positive pair is constructed
[0049] For each word in the decoding step, the ConSLT framework takes it and the enhanced version generated by dropout as a positive pair; ConSLT does not use all other sentences or words in the batch as negative examples, but randomly selects K words in the vocabulary that are not in the current sentence as negative examples to mine different negative examples; given a sentence y, all the word sets in the sentence y are denoted as S; from a candidate word set All words in the vocabulary V that are not in the current sentence S are randomly selected as K words, and for each word y t A negative sample subset is constructed y - is looked up through a negative example embedding lookup table A negative pair is constructed for each word
[0050] In step 6), the specific steps of the sign language translation model loss calculation can be: in the loss calculation module, the alignment loss of the aligned sign language word annotation in the sign language encoding module is calculated using the CTC loss function, the translation loss of the generated words in the sign language decoding module is calculated using the cross-entropy loss function, and the contrastive loss of the decoding module output obtained by outputting the sign language video twice is calculated using the KL divergence.
[0051] In step 7), the sign language translation result output is the spoken language sentence obtained by the sign language translation model through the output module.
[0052] The application also provides a sign language translation system based on contrastive learning, comprising a video input unit, a sentence input unit, a video feature extraction module, a sign language encoding module, a sign language decoding module, a contrastive learning module, a storage unit, a loss calculation module and a text output unit.
[0053] The video input unit is used for sign language translation data selection modeling, and the sign language video is input into the system in the form of video frames.
[0054] The sentence input unit is used for inputting an existing data set containing sign language video corresponding spoken language sentences as answers in the training stage.
[0055] The video feature extraction module is used for extracting features of the sign language video using a convolutional neural network for each frame of the sign language video to obtain sequence features containing sign language visual information.
[0056] The sign language encoding module is used for sending the extracted visual feature vector to the sign language encoding module to obtain hidden representations through an encoder-decoder.
[0057] The sign language decoding module is used for generating a translation result through the feature vector of the encoding module after receiving the output vector of the sign language encoding module.
[0058] The contrastive learning module is used for constructing positive and negative pairs required by the model through the contrastive learning module in the training stage.
[0059] The storage unit is used for storing sign language translation data.
[0060] The loss calculation module is used for calculating the alignment loss of the sign language word annotation in the sign language encoding module using the CTC loss function, calculating the translation loss of the generated words in the sign language decoding module using the cross-entropy loss function, and calculating the contrastive loss of the decoding module output obtained by outputting the sign language video twice using the KL divergence.
[0061] The text output unit is used for outputting the spoken language sentence output obtained by the sign language translation model.
[0062] The application has the following advantages:
[0063] 1) The application first explores the contrastive learning of sign language translation from the perspective of natural language processing. Contrastive learning directly uses data itself as supervision information, and the information provided by the data itself is more abundant than the label information, so that more abundant feature representations of sign language data can be learned.
[0064] 2) The application proposes a new word-level contrastive learning method for sign language translation. Word-level contrastive learning is a self-supervised learning method that provides a new insight for alleviating the low-resource problem of sign language translation due to insufficient training data. Through word-level contrastive learning, good sign language representation can be learned even in low-resource situations, making the sign language translation system more accurate and fluent.
[0065] 3) The ConSLT framework proposed in the application has always performed better than the baseline, and the framework is not limited to a model and can be applied to different models. The method is widely applicable to various sign language-related natural language processing and visual tasks, and has good application prospects and application value. BRIEF DESCRIPTION OF DRAWINGS
[0066] Figure 1 A sign language translation framework based on contrastive learning.
[0067] Figure 2 A sign language translation method flowchart based on contrastive learning.
[0068] Figure 3 A sign language translation system schematic diagram based on contrastive learning. DETAILED DESCRIPTION
[0069] In order to better understand the above technical solutions, the content of the application and the exemplary embodiments will be described in more detail below in conjunction with the drawings. Although the exemplary embodiments of the application are shown in the drawings, it should be understood that the application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the application and to fully convey the scope of the application to those skilled in the art.
[0070] The content of the application aims to learn effective word representations by introducing word-level contrastive learning in the decoding process of sign language translation. In addition, the method proposed in the application is not limited to a model and can adapt to different model configurations.
[0071] As Figure 1 The sign language translation framework (ConSLT) based on contrastive learning according to the embodiments of the application comprises a video input module, a visual extraction module, a sign language encoding module, a sentence embedding module, a sign language decoding module, a contrastive learning module, a loss calculation module, and an output module.
[0072] The video input module is used for sign language translation data selection modeling, and the sign language video is input into the model in the form of video frames.
[0073] The visual extraction module is used for feature extraction of the sign language video using a convolutional neural network for each frame of the sign language video to obtain sequence features containing sign language visual information.
[0074] The sign language coding module is used to obtain a hidden representation through an encoder-decoder for the received visual feature vector;
[0075] The sentence embedding module is used to correspond the input sign language video to a spoken language sentence as an answer in the training stage;
[0076] The sign language decoding module is used to generate a translation result through the feature vector of the coding module after receiving the output vector of the sign language coding module;
[0077] The contrast learning module is used to construct a positive example pair and a negative example pair in the training stage;
[0078] The loss calculation module is used to calculate an alignment loss of the aligned sign language word annotation in the sign language coding module using a CTC (Connectionist Temporal Classification) loss function, calculate a translation loss of the generated word in the sign language decoding module using a cross-entropy loss function, and calculate a contrast loss of the decoding module output obtained by outputting the sign language video twice using a KL divergence;
[0079] The output module is used to output the spoken language sentence obtained by the sign language translation model for the sign language video.
[0080] The word-level contrast learning framework ConSLT proposed in the application does not introduce additional structures, and for the first time introduces a contrast learning method, which can adapt to different models, and improves the translation performance of the cascade baseline by more than 1.4 BLEU, so as to realize more accurate and fluent sign language translation in a low resource condition. First, the sign language translation data selection modeling is performed in the video input module, and the sign language video is input into the model in the form of a video frame; then in the visual extraction module, the convolutional neural network is used to extract the features of the sign language video for each frame of the sign language video to obtain sequence features containing sign language visual information; the extracted visual feature vector is sent to the sign language coding module to obtain a hidden representation through an encoder-decoder; the sentence embedding module inputs the corresponding spoken language sentence of the sign language video as an answer in the training stage; after receiving the output vector of the sign language coding module, the sign language decoding module generates a translation result through the feature vector of the coding module; in the training stage, the contrast learning module constructs a positive example pair and a negative example pair; in the loss calculation module, a CTC loss function is used to calculate an alignment loss of the aligned sign language word annotation in the sign language coding module, a cross-entropy loss function is used to calculate a translation loss of the generated word in the sign language decoding module, and a KL divergence is used to calculate a contrast loss of the decoding module output obtained by outputting the sign language video twice; finally, the output module outputs the spoken language sentence obtained by the sign language translation model for the sign language video.
[0081] As Figure 2As shown, in an embodiment of the present application, a sign language translation method based on contrastive learning is provided, and the process includes:
[0082] Step 1: input the sign language video.
[0083] In the sign language translation data selection modeling of the video input module, the sign language video is input into the model in the form of video frames.
[0084] Step 2: obtain the sign language feature representation.
[0085] In the visual extraction module, the convolutional neural network is used to extract the features of the sign language video for each frame of the sign language video, and the sequence features containing the visual information of the sign language are obtained;
[0086] Step 3: input the sign language feature into the encoder.
[0087] The extracted visual feature vector is sent to the sign language encoding module, and the hidden representation is obtained through the encoder-decoder. The sign language encoding module is composed of multiple transformer encoder modules. The purpose of the sign language encoding module is to learn meaningful spatiotemporal representation and sign language representation, and to encode the visual information into the text space by aligning the sign language word annotation in the sign language video, thereby providing help for the sign language translation task. Here, CTC loss is used to align the sign language word annotation in the sign language video. The output of the sign language encoding module is a feature vector containing semantic information of the sign language.
[0088] The sign language encoding module has an attention mechanism, which can get rid of the length limit of the input sequence and is a technology that enables the model to focus on important information and learn and absorb it fully.
[0089] Step 4: input the feature vector containing semantic information of the sign language at the output end of the encoder into the decoder.
[0090] The decoder module is composed of multiple transformer decoder stacks. After receiving the output vector of the sign language encoding module, there will be a begin identifier. After recognizing the begin, the sign language decoding module generates a translation result through the feature vector of the encoding module. Each step of the decoding stage outputs a word from the output sequence. The output of each step is input to the bottom decoder in the next time step, and the decoding result is output to the higher layer. Position encoding is embedded and added in these decoder inputs to indicate the position of each word. Repeat this process until the end identifier is reached, indicating that the decoding module has completed the output.
[0091] The sign language decoding module employs a masking attention mechanism primarily because decoding is a sequential process. When predicting a word at the k-th time step, only the first k-1 predictions are visible, necessitating a masking operation on subsequent parts. Furthermore, in the decoder's attention mechanism, K and V originate from the encoder's output, while Q comes from the decoder's previous output.
[0092] Step 5: Construct contrastive learning pairs of positive and negative examples.
[0093] This invention employs different dropout sampling methods, feeding a symbolic video x into the model twice. Because the dropout mechanism randomly discards some units in the transformer, it can provide sampling for each word y. t We obtain two distinct hidden representations, denoted as h. t and For each word y t Construct positive pairs
[0094] Using all other words within the batch as negatives is a natural choice. However, it has two main drawbacks: first, the diversity of words in the mini-batch is insufficient; second, due to contextual dependencies, different words within the same sentence cannot be inferred. To address this, this invention proposes a novel negative sample sampling strategy. Given a sentence y, the set of all words in sentence y is represented as S. First, from a candidate word set... K words are randomly selected from (all words in vocabulary V that are not in the current sentence S), and each word y is... t Construct a negative sample subset Then, y - Lookup table using a negative example embedding Therefore, negative pairs can be constructed for each word.
[0095] Step 6: Calculate the contrastive learning loss.
[0096] In the loss calculation module, the CTC loss function is used to calculate the alignment loss of the sign language word annotations in the sign language encoding module, the cross-entropy loss function is used to calculate the translation loss of the generated words in the sign language decoding module, and the KL divergence is used to calculate the contrast loss of the decoding module output obtained by outputting the sign language video twice.
[0097] This invention employs KL divergence as the distance metric for contrastive learning because KL loss is more rigorous than cosine loss. For example, given two vectors a = [1.0, 2.0, 3.0] and b = [20.0, 40.0, 60.0], cos(a, b) = 1, but KL(a, b) = 4.04. Experimental results have already demonstrated the effectiveness of the method presented in this invention.
[0098] The goal of contrastive learning is to minimize the following loss.
[0099]
[0100]
[0101] where τ is a temperature hyperparameter. Here the bidirectional KL-divergence is computed because the KL-divergence is an asymmetric measure.
[0102] During the training process of ConSLT, the model is optimized by jointly minimizing the translation loss and the contrastive training loss.
[0103]
[0104] where α is a hyperparameter to balance the two training losses.
[0105] Step 7: Sign language translation result output.
[0106] Finally, the spoken sentence obtained through the sign language translation model is output through the output module.
[0107] The present application provides a sign language translation system based on contrastive learning as shown in Figure 3 The specific description is as follows:
[0108] The sign language translation system based on contrastive learning includes a video input unit, a sentence input unit, a video feature extraction module, a sign language encoding module, a sign language decoding module, a contrastive learning module, a storage unit, a loss calculation module, and a text output unit.
[0109] The video input unit is used for sign language translation data selection modeling, and the sign language video is input into the system in the form of video frames.
[0110] The sentence input unit is used to input an existing data set containing sign language video corresponding spoken sentences as answers in the training phase.
[0111] The video feature extraction module is used to extract features of the sign language video using a convolutional neural network for each frame of the sign language video, obtaining sequence features containing sign language visual information.
[0112] The sign language encoding module is used to send the extracted visual feature vector to the sign language encoding module to obtain hidden representations through the encoder-decoder.
[0113] The sign language decoding module is used to generate translation results through the feature vector of the encoding module after receiving the output vector of the sign language encoding module.
[0114] The contrast learning module is used to construct positive example pairs and negative example pairs by the contrast learning module in the training stage.
[0115] The storage unit is used to store sign language translation data.
[0116] The loss calculation module is used to calculate the alignment loss of the aligned sign language word annotation in the sign language encoding module using the CTC loss function, calculate the translation loss of the generated words in the sign language decoding module using the cross-entropy loss function, and calculate the contrast loss of the decoding module output obtained by outputting the sign language video twice using the KL divergence calculation.
[0117] The text output unit is used to output the spoken language sentence output obtained by the sign language translation model from the sign language video.
[0118] To sum up, the present application provides a novel word-level contrast learning framework and sign language translation system for sign language translation, which first explores the contrast learning of sign language translation from the perspective of natural language processing, and proposes a new word-level contrast learning method for sign language translation. The present application first performs sign language translation data selection modeling in the video input module, inputs the sign language video in the form of video frames into the model; then in the visual extraction module, the convolutional neural network is used to extract the features of the sign language video for each frame of the sign language video to obtain sequence features containing visual information of the sign language; the extracted visual feature vector is sent to the sign language encoding module to obtain hidden representation through the encoder-decoder; the sentence embedding module inputs the corresponding spoken language sentence of the sign language video as the answer in the training stage; after receiving the output vector of the sign language encoding module, the sign language decoding module generates a translation result through the feature vector of the encoding module; in the training stage, positive example pairs and negative example pairs are constructed through the contrast learning module; in the loss calculation module, the alignment loss of the aligned sign language word annotation in the sign language encoding module is calculated using the CTC loss function, the translation loss of the generated words in the sign language decoding module is calculated using the cross-entropy loss function, and the contrast loss of the decoding module output obtained by outputting the sign language video twice is calculated using the KL divergence calculation; finally, the output module outputs the spoken language sentence output obtained by the sign language translation model from the sign language video. The ConSLT method proposed in the present application does not introduce additional structures, first introduces the method of contrast learning, can adapt to different models, improves the translation performance of the baseline by more than 1.4 BLEU, and realizes more accurate and fluent sign language translation in the case of low resources. The method is widely applicable to various sign language related natural language processing and visual tasks, and has good application prospect and application value.
[0119] The present application is compared with the existing method in terms of actual technical effect, as shown in Table 1:
[0120] Table 1. Comparison of experimental results on the data set PHOENIIX14T
[0121]
[0122] Note: In end-to-end translation, SL-Transformer [3] is the baseline model of the present invention; in cascaded approach, STMC-Transformer [4] is the baseline model of the present invention. Performance improvement is above 1.4 BLUE-4.
[0123] References:
[0124] [1] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden, “Neural sign language translation,” in Proc. of CVPR, 2018.
[0125] [2] Pan Xie, Mengyi Zhao, and Xiaohui Hu, “Pisltrc: Position-informed sign language transformer with content-aware convolution,” IEEE TMM, 2021.
[0126] [3] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden, “Sign language transformers: Joint end-to-end sign language recognition and translation,” in Proc. of CVPR, 2020.
[0127] [4] Kayo Yin and Jesse Read, “Better sign language translation with STMC-transformer,” in Proc. of COLING, 2020.
Claims
1. A sign language translation method based on contrastive learning, characterized in that... Includes the following steps: 1) Sign language corpus selection and modeling; The sign language corpus selection and modeling involves selecting and modeling the sign language corpus in the video input module, and inputting the sign language videos in the corpus into the model in the form of video frames; The sign language corpus includes a triplet of sign language video, sign language annotations, and spoken sentences, denoted as D = {(x,z,y)}∈(X,Z,Y); where X,Z,Y represent the sign language video, sign language annotations, and spoken sentences, respectively. Specifically, This represents a sign language video of a T-frame. It is the corresponding sign language word sequence. This corresponds to the spoken sentence; the sign language transcription sequence is a word-for-word sign language transcription, and due to the significant differences between sign language transcription and spoken language grammar, the word order of the sign language transcription sequence differs from that of the spoken sentence; T z and T y These are the lengths of the sign language word sequence and the spoken sentence, respectively; 2) Sign language visual feature extraction; The sign language visual feature extraction is performed by using a convolutional neural network to extract features from each frame of the sign language video in the visual extraction module, so as to obtain a sequence of features containing sign language visual information. 3) End-to-end sign language video conversion; The end-to-end sign language video conversion involves sending the extracted visual feature vectors to the sign language encoding module, where the encoder and decoder obtain the hidden representations. After receiving the output vectors from the sign language encoding module, the sign language decoding module uses the feature vectors from the encoding module to generate the translation results. Specifically, after modeling the sign language videos in the sign language corpus, a transformer-based encoder-decoder structure is used to convert the video to text; The sign language encoding module is used to learn meaningful spatiotemporal representations and sign language representations. By aligning sign language annotations in sign language videos, visual information is encoded into the text space, which helps with sign language translation tasks. CTC loss is used to align sign language annotations in sign language videos. The sign language encoding module outputs feature vectors containing sign language semantic information. The sign language encoding module incorporates an attention mechanism, which is a technique that allows the model to focus on and fully learn important information, freeing it from the limitation of input sequence length. The frame-level representation of the sign language video is extracted by a pre-trained sign language embedding network; positional encoding (PE) is used to add positional information to the encoder's input embedding into the frame-level representation; the input is fed into the transformer encoder module to generate the hidden representation of the sign language video; these operations are described as follows: F i =SignEmbedding(x i ), (1) Where i∈[1,Tx] represents the i-th frame of the video; SigEmbedding represents the pre-trained sign language embedding network; PE represents the encoder's input embedding with positional information added to the frame-level representation; and Encdoer represents the encoder. After receiving the output vector from the sign language encoding module, there is a "begin" identifier. Upon recognizing "begin", the sign language decoding module generates the translation result using the feature vector from the encoding module. Each step in the decoding stage outputs a word from the output sequence. The output of each step is input into the bottom decoder in the next time step, causing its decoding result to be output to a higher layer. Positional encoding is embedded and added to these decoder inputs to indicate the position of each word. This process is repeated until the "end" identifier is reached, indicating that the decoding module has completed the output. The sign language decoding module has a masking attention mechanism, mainly because decoding is a sequential operation. When predicting a word at the k-th time step, only the first k-1 prediction results can be seen, so the subsequent parts need to be masked. At the same time, in the decoder's attention mechanism, K and V come from the encoder's output, while Q comes from the decoder's previous output. These operations are described as follows: w t =WordEmbedding(y t ), (4) o t =softmax(Wh t +b). (7) The conditional probability and objective function of p(y|x) are calculated as follows: 4) Sentence embedding during training: During the training phase, the sentence embedding module takes the spoken sentence y corresponding to the sign language video x as the input and uses it as the answer for the training phase. The spoken sentence y is converted into a sentence embedding, and the text representation in the text is converted into a vector representation. In order to capture the relationship between words in the high-dimensional space, a linear layer is used to project the text representation into the high-dimensional space. 5) Construct positive and negative pairs, the specific steps are as follows: Using different dropout sampling methods, a symbolic video x is fed into the model twice; because the dropout mechanism randomly discards some units in the transformer, for each word y... t We obtain two distinct hidden representations, denoted as h. t and For each word y t Construct positive pairs For each word in the decoding step, the ConSLT framework pairs it with the enhanced version generated by dropout as a positive example; instead of using sentences or words from all other batches as negative examples, ConSLT randomly selects K words from the vocabulary that are not in the current sentence as negative examples to mine different negative examples; given a sentence y, the set of all words in sentence y is represented as S; from a candidate word set... K words are randomly selected from (all words in vocabulary V that are not in the current sentence S), and each word y is... t Construct a negative sample subset y - Lookup table using a negative example embedding Build negative pairs for each word 6) Loss calculation of sign language translation model. The specific steps are as follows: In the loss calculation module, the CTC loss function is used to calculate the alignment loss of the sign language encoding module to align the sign language annotations, the cross-entropy loss function is used to calculate the translation loss of the generated words in the sign language decoding module, and the KL divergence is used to calculate the contrast loss of the decoding module output obtained by outputting the sign language video twice. 7) Output the sign language translation results.
2. The sign language translation method based on contrastive learning as described in claim 1, characterized in that... In step 7), the sign language translation result is output by the output module, which outputs the spoken sentence obtained through the sign language translation model.