Annotation-free sign language translation method based on fusion attention

Abstract

The invention discloses an unannotated sign language translation method based on fusion attention. The unannotated sign language translation method comprises the following steps: acquiring a sign language video and a sign language spoken language text corresponding to the sign language video; constructing an initial sign language translation model based on a Transform architecture, and training the initial sign language translation model only through a sign language video and a sign language spoken language text to obtain an unannotated sign language translation model; in the training process, internal attention of the pseudo annotations and attention between the pseudo annotations are calculated and fused, and meanwhile, a contrast loss combined cross entropy loss training model is introduced; and translating the sign language video to be translated through the sign language translation model to obtain spoken language text output. The effect is improved in the aspect of annotation-free sign language translation, and compared with a previous sign language translation method, the sign language translation method has higher accuracy and application potential.

Description

Technical Field

[0001] The invention belongs to the technical field of sign language translation, and in particular relates to an annotation-free sign language translation method based on fusion attention. Background Art

[0002] According to statistics from the World Federation of the Deaf, there are more than 70 million hearing-impaired people in the world. Sign language, as a visual language, is mainly used by the hearing-impaired. Sign language users use a combination of gestures, facial expressions and body movements to express the information they want to convey. Barrier-free communication is an important way for the hearing-impaired to break the information island and conduct equal social communication. The main requirement for barrier-free communication for the hearing-impaired is that normal hearing people can understand the sign language expressions of the hearing-impaired. With the development of artificial intelligence technology, especially the progress of computer vision research and natural language processing research, this requirement has become possible.

[0003] Sign language translation technology can be divided into two major research directions based on whether annotations are used: annotation-based sign language translation and non-annotated sign language translation. In the process of sign language translation, annotations as an intermediate representation can help the model better understand the sign language video and promote the learning of semantic boundary positions. These methods that directly or indirectly use annotations have achieved good results in sign language translation. However, annotations are expensive and cumbersome to obtain, and sign language experts need to spend a lot of time on annotation. Therefore, in recent years, more and more research has begun to turn to non-annotated sign language translation methods, directly learning to translate sign language videos into spoken text, making sign language translation methods more versatile and providing the possibility of utilizing a wider range of sign language resources.

[0004] The present invention is different from previous sign language translation models. The GASLT model only pays attention to the local attention of the markers, but does not consider the contextual information of the markers. The EvSign model uses sign language data captured by an event camera to train the model, and uses relevant annotations during the training process. The present invention, without the help of annotations, not only pays attention to the internal attention of the pseudo-annotations, but also integrates the attention between pseudo-annotations, making the video markers more relevant. At the same time, the contrast loss is introduced to narrow the gap between cross-modalities to achieve annotation-free sign language translation. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes an unannotated sign language translation method based on fusion attention to solve the problems existing in the above prior art.

[0006] To achieve the above object, the present invention provides an unannotated sign language translation method based on fusion attention, comprising the following steps:

[0007] Obtain a sign language video and a sign language spoken text corresponding to the sign language video;

[0008] Building an initial sign language translation model based on the Transformer architecture, and training the initial sign language translation model only through the sign language video and the sign language spoken text to obtain an unannotated sign language translation model;

[0009] During the training process, the pseudo-annotation internal attention and pseudo-annotation inter-attention are calculated and fused, and the contrast loss and cross-entropy loss are introduced to train the model.

[0010] The sign language video to be translated is translated by using the unannotated sign language translation model to obtain a spoken text output.

[0011] Preferably, the method for training the initial sign language translation model comprises:

[0012] Obtain visual features of sign language videos through pre-trained I3D models;

[0013] Inputting the visual features into a visual encoder, wherein the visual encoder extracts pseudo-annotation intra-attention and pseudo-annotation inter-attention of the visual features based on a fusion attention mechanism;

[0014] Merge the attention between pseudo-annotations and the attention within pseudo-annotations to obtain sign language-text features;

[0015] Embed the real sign language spoken text in the text decoder to obtain the representation vector of each text;

[0016] The sign language-text features and the representation vector of each text are softly aligned through the contrastive loss function, and the model is optimized by combining the contrastive loss and cross entropy loss to obtain the sign language translation model.

[0017] Preferably, the method for the sign language translation model to translate the sign language video to be translated comprises:

[0018] Obtain the visual features of the sign language video to be translated through the pre-trained I3D model;

[0019] Inputting the visual features into a visual encoder, wherein the visual encoder calculates pseudo-annotation intra-attention and pseudo-annotation inter-attention of the visual features based on a fusion attention mechanism;

[0020] Merge the attention between pseudo-annotations and the attention within pseudo-annotations to obtain sign language-text features;

[0021] The sign language-text features are decoded by a text decoder to obtain spoken text.

[0022] Preferably, the method for acquiring visual features of a sign language video comprises:

[0023] The pre-trained I3D model is used to obtain the initial video features and the linear layer is used to convert the video features into the dimensions of the visual encoder and perform batch normalization. The normalized features are processed to obtain the embedded features of each frame.

[0024] Preferably, the method of embedding authentic sign language spoken text comprises:

[0025] The word vectors of real sign language spoken text are converted into text representations of each word using the pre-trained subword embeddings in BPEmb as the initialization of the embedding layer.

[0026] Preferably, the method for obtaining the sign language-text feature comprises:

[0027] The visual features are divided according to the windows, and attention is calculated for each window to obtain the result of the pseudo-annotation internal attention. The attention calculation of the window mark is performed to obtain the attention map of the relationship between each window. The attention map between pseudo-annotations is fused to weight the result vector of the pseudo-annotation internal attention, and then a residual connection is performed with the result of the pseudo-annotation internal attention to obtain the sign language-text feature.

[0028] Preferably, the expression of the pseudo-annotation internal attention is:

[0029]

[0030] In the formula, It is a feature tag, which consists of a video tag and a window tag. q , W k and W v Represent the three linear layers for query, key and value calculation in regular self-attention, respectively. Attention is a standard self-attention.

[0031] The expression of the attention between the pseudo annotations is:

[0032]

[0033]

[0034] In the formula, and are the video frame label and window label within each window extracted from the output of IaPG, respectively; LN is the layer normalization layer, and Af is the activation function.

[0035] Preferably, the model introduces contrast loss and cross entropy loss to jointly train the unannotated sign language translation model, and the total loss function expression of the model is:

[0036]

[0037] is the cross entropy loss;

[0038] The contrast loss function expression is:

[0039]

[0040] In the formula, {v i ,t i} is a positive sample pair, {v i ,t j}(i≠j) is a negative sample pair, s(.) is a similarity function, and τ is a temperature hyperparameter. The model is trained by combining contrast loss and cross entropy loss to narrow the gap between the two modalities.

[0041] Compared with the prior art, the present invention has the following advantages and technical effects:

[0042] The present invention discloses a method for non-annotated sign language translation based on fusion attention, comprising the following steps: obtaining a sign language video and a sign language spoken text corresponding to the sign language video; constructing an initial sign language translation model based on a Transformer architecture, and training the initial sign language translation model only through the sign language video and the sign language spoken text to obtain a non-annotated sign language translation model; calculating pseudo-annotation internal attention and pseudo-annotation inter-attention during the training process, and fusing the two, while introducing a contrast loss combined with a cross entropy loss training model; translating the sign language video to be translated through the sign language translation model to obtain a spoken text output. The present invention has achieved improved results in non-annotated sign language translation, and has higher accuracy and application potential than previous sign language translation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 A schematic diagram of a sign language translation model according to an embodiment of the present invention;

[0045] Figure 2 is a fused attention graph of an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of IaPG according to an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of visualizing fusion attention according to an embodiment of the present invention;

[0048] Figure 5 This is a diagram showing the effect of window size on model performance according to an embodiment of the present invention;

[0049] Figure 6 is a distribution diagram of frames corresponding to each gloss in an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of selecting a text encoding model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] Embodiment 1

[0054] This embodiment provides an unannotated sign language translation method based on fusion attention, including the following steps:

[0055] Obtain a sign language video and a sign language spoken text corresponding to the sign language video;

[0056] An initial sign language translation model is built based on the Transformer architecture, and the initial sign language translation model is trained only with sign language videos and sign language spoken text to obtain an unannotated sign language translation model.

[0057] During the training process, the pseudo-annotation internal attention and pseudo-annotation inter-attention are calculated and fused, and the contrast loss and cross-entropy loss are introduced to train the model.

[0058] The sign language video to be translated is translated through the unannotated sign language translation model to obtain spoken text output.

[0059] Furthermore, the method for training the initial sign language translation model includes:

[0060] Obtain visual features of sign language videos through pre-trained I3D models;

[0061] The visual features are input into the visual encoder, which calculates the pseudo-annotation intra-attention and pseudo-annotation inter-attention of the visual features based on the fusion attention mechanism;

[0062] Merge the attention between pseudo-annotations and the attention within pseudo-annotations to obtain sign language-text features;

[0063] Embed the real sign language spoken text in the text decoder to obtain the representation vector of each text;

[0064] The sign language-text features and the representation vector of each text are softly aligned through the contrast loss function, and the model is optimized by combining the contrast loss and the cross entropy loss to obtain the sign language translation model (such as Figure 1 ).

[0065] The video text embedding method is as follows:

[0066] First, the input video and natural language text are embedded. For the video feature x′ extracted by TSPNet t , first use a linear layer to convert the video features to the dimension of the visual encoder, and then perform batch normalization (BN). The Softsign function is used for processing, which can make the gradient flow more stable and reduce the gradient vanishing and explosion problems, and is more suitable for processing image features. In this way, the embedded features of each frame are obtained For text embedding, we first use the pre-trained subword embeddings in BPEmb as the initialization of the embedding layer (Emb). This process is actually based on the way annotations are formed. Multiple subwords make up a word similar to multiple video frames making up a annotation. Then, we use a method similar to visual feature embedding to convert the word vectors into text representations for each word. These operations are formulated as follows, where f_pos(·) represents the position information injected into the feature, W 1 , b 1 , W 2 , b 2 as a parameter.

[0067]

[0068]

[0069] The principle of the visual encoder is as follows:

[0070] The processed video embedding and the additional window mark are input into the visual encoder. First, the embedding is layer-normalized, and then the fused attention is calculated. According to the window division, attention is first calculated for each window to obtain the result of the pseudo-annotation internal attention. Next, the attention of the window mark is calculated to obtain the attention map of the relationship between each window. The attention map between pseudo-annotations is fused in to weight the result vector of the pseudo-annotation internal attention, and then a residual connection is performed with the result of the pseudo-annotation internal attention to obtain the result of the fused attention. The fused attention structure proposed in this embodiment is as follows: Figure 2 shown.

[0071] Pseudo-annotation internal attention (IaPG) focuses on the sign language video frames representing each annotation, imitating the role of annotations to make the internal relationship of sign language video frames corresponding to the same annotation closer. Pseudo-annotation internal attention is built on the window-based self-attention calculation. Figure 3 As shown in the figure, first, the input video features are split into windows. Each window can be regarded as the corresponding input of each annotation, and different windows correspond to different annotations. At the same time, a window tag is created for each window as a global representation, and is used to simulate the contextual relationship of the sign language video in the following pseudo-annotation attention module. The pseudo-annotation internal attention is calculated for all video tags in each window. The implementation of pseudo-annotation internal attention can be summarized as follows:

[0072]

[0073] in, It is a feature tag, which consists of a video tag and a window tag. q , W k and W v Represents three linear layers for query, key, and value computation in regular self-attention. Attention is a standard self-attention with d k Represents the scaling factor:

[0074]

[0075] Pseudo Inter-annotation Attention (IrPG) starts from the global perspective and makes each annotation pay attention to other annotations, thus enhancing the contextual relationship between annotations. A single tag is used to encapsulate the core information of each window. The window tag can be initialized as a fixed zero vector. When passing through IaPG, information interaction will occur between the window tag and the tags in each window. Therefore, the window tag can learn the global representation of the window.

[0076] The pseudo-annotation attention is mainly used to fuse the information of different windows and obtain the final representation of the input features. Specifically, the present invention extracts the video frame markers within each window from the output of IaPG and window tags Window Tags It is used to calculate the attention relationship between windows and generate an attention map after passing through the layer normalization layer (LN) and the GELU activation function. At the same time, the present invention directly converts the feature vector obtained by the pseudo-annotation internal attention calculation result into As the value branch of IrPG without any other additional operations. With the attention map and frame feature vector, attention calculation can be performed between windows to achieve global information exchange. IrPG also uses residual connection. The implementation process of IrPG is as follows:

[0077]

[0078] In this embodiment, GELU is used as the activation function.

[0079] Sign language translation is a cross-modal learning task from video to text. Therefore, a method is needed to build a bridge between video and text. To this end, the "contrastive loss" function is used to align text and image in semantic space. This model only trains the visual encoder and text decoder. The text encoder is not directly trained additionally. Instead, cross-modal soft alignment (CSA) is achieved with the help of the existing text model. During the training process, the contrastive loss and cross entropy loss are combined to achieve unannotated sign language translation.

[0080] The model introduces contrast loss and cross entropy loss to jointly train the unannotated sign language translation model. The total loss function expression of the model is:

[0081]

[0082] is the cross entropy loss; for each sign language video, the average of all vectors output by the visual encoder is calculated to represent the vector v of the entire sign language video i , using the text encoder to get the representation vector of each text as t i Among them, {v i ,t i} is considered as a positive sample pair, {v i ,t j}(i≠j) is regarded as a negative sample pair. The contrast loss is calculated as follows, s(.) is the similarity function, and τ is the temperature hyperparameter:

[0083]

[0084] The following experiments were also conducted in this embodiment:

[0085] Use the following data set:

[0086] RWTH-PHOENIX Weather 2014T (PHOENIX14T) dataset is widely used to train and evaluate sign language translation models from sign language video input to spoken output. This dataset collects daily news and weather forecast broadcasts of German public television station PHOENIX from 2009 to 2011. The dataset contains a total of 8257 video-annotation-text pairs, including 7096 training sets, 519 validation sets, and 642 test sets. CSL-Daily is a large public Chinese sign language dataset, whose content mainly revolves around the daily life of the deaf community. The topics involved are very wide, including family life, health care, school life, banking services, shopping, social interaction, etc., and contains a total of 20654 video-annotation-text pairs. The dataset contains a total of 20654 video-annotation-text pairs, including 18401 training sets, 1077 validation sets, and 1176 test sets.

[0087] The scoring criteria are as follows:

[0088] With reference to previous sign language translation papers, etc., the present invention uses BLEU and ROUGE-L scores to evaluate the translation performance of the model. These two indicators are the most commonly used evaluation indicators in machine translation. BLEU-n represents the weighted average translation accuracy from 1-gram to n-gram. Generally, the present invention uses an equal weight approach, that is, the weights from 1-gram to n-gram are all 1 / n. ROUGE-L uses the longest common subsequence length between the predicted text and the reference text to calculate the F1 score, which can effectively reflect the semantic similarity and coverage between them. Higher BLEU and ROUGE-L scores indicate better translation performance.

[0089] The untranslated sign language translation model SLTFA is based on the Transformer architecture, and fusion attention is added to the encoder. The model consists of 2 layers of encoders and 2 layers of decoders, in which the head value of the multi-head attention mechanism is 8, and the hidden layer dimension is 384. The present invention uses Xavier initialization to initialize the network, uses the Adam optimizer, the initial learning rate is 0.0005, and the batch size is set to 32. Using plateau learning rate scheduling, the patience coefficient and the decline coefficient are adjusted to 9 and 0.5 respectively. Dropout of 0.3 and 0.5 are also used on the encoder and decoder layers to reduce overfitting. The BPEmb model with a vocabulary of 50,000 and an embedding dimension of 300 is used, the window size is set to 5, and the weights of the cross entropy loss and the contrastive learning loss L are both set to 1.

[0090] The quantitative comparison is as follows:

[0091] The proposed SLTFA model was experimented on the PHOENIX14T dataset and compared with seven unannotated sign language translation models. The experimental results are shown in Table 1. The GASLT model achieved the best result, while the model proposed in the present invention improved the ROUGE score by 0.96 and the BLEU-4 score by 1.25 compared with GASLT in the experiment, indicating that the model of the present invention has achieved a major breakthrough in long sequence translation. At the same time, the BLEU-1 and BLEU-2 scores were improved by 0.81 and 1.05 respectively, indicating that the model of the present invention can also maintain high performance in short sequence translation.

[0092] At the same time, the two models were compared on the CSL-Daily dataset. For Joint-SLT, the experimental results reproduced by Yin et al. in the unannotated setting were used. The results are shown in Table 2. Although the BLEU-4 score is lower than that of the GASLT model, it shows that the model still has a lot of room for improvement in the long sequence translation of Chinese. But overall, the method of the present invention outperforms the Joint-SLT and GASLT models on the CSL-Daily dataset. The results in Tables 1 and 2 prove that the model of the present invention achieves better results in unannotated sign language translation, further verifying the correctness of the fusion of attention within pseudo-annotations and attention between annotations, and supporting the important role of these mechanisms in improving translation performance in practice.

[0093] The test set in the dataset was classified according to the gender of the sign language presenter to verify the translation performance of the model, and the results are shown in Tables 3 and 4. The experimental results show that the number of videos expressed by female presenters is significantly higher than that of male presenters, and accordingly, the scores of female presenters are also higher, highlighting the importance of data volume. However, in some indicators, such as BLEU-4 of the PHOENIX14T dataset and BLEU-4 of the CSL-Daily dataset, the difference between the two is small, indicating that the model can ignore the influence of the gender of the presenter to a certain extent.

[0094] Table 1

[0095]

[0096]

[0097] Table 2

[0098]

[0099] Table 3

[0100]

[0101] Table 4

[0102]

[0103] The ablation experiment is as follows:

[0104] Four sets of ablation experiments are designed and implemented to systematically study and analyze the optimal combinations of various components and parameters and their impact on the model performance.

[0105] Effectiveness of Fusion Attention and Cross-modal Soft Alignment:

[0106] In this experiment, the effectiveness of the model is verified by comparing different attention mechanisms, and the experimental results are shown in Table 5. In the self-attention mechanism, each query will pay attention to all keys, and it is not easy to distinguish the video frames corresponding to different annotations, and it is difficult to determine which contents belong to the same annotation. Although the pseudo-annotation internal attention mechanism uses windows to divide annotations, it ignores the relationship between annotations and cannot fully process sentences composed of multiple annotations. The fusion attention proposed in the present invention not only considers the internal logic of multiple sign language video frames forming an annotation, but also considers the internal logic of multiple annotations forming a spoken sentence. Therefore, compared with other attention mechanisms, the model of the present invention has achieved better experimental results. Figure 4 This is the visualization result of fusion attention, where the window size is 5, and the 0th one corresponds to the additional global window mark. It can be seen that the window mark interacts well with the internal features of the window, and the weights of the internal features of the window are not much different, which indirectly shows that the window division is appropriate and each window should belong to the same gloss. In addition, under the three attention mechanisms, after adding cross-modal soft alignment, the experimental results are improved, which proves that the use of contrast loss effectively reduces the cross-modal distance and improves the performance of the model.

[0107] Table 5

[0108]

[0109] Window size for attention inside pseudo-annotations:

[0110] Each annotation is often composed of multiple frames. In this experiment, the optimal parameter setting of the window size when calculating the attention within the pseudo annotation is verified. The results are as follows: Figure 5 As shown in the figure, the experimental results show that when the window is set to 5, the SLTFA model has the best translation performance. On the PHOENIX14T test set, the ratio of the number of video frames to the number of gloss words is counted, as shown in the figure. Figure 6 As shown in the figure, when the average number of video frames corresponding to each gloss is between 4-6 frames and 9-11 frames, the cumulative frequencies of these two intervals are relatively high, with 5 and 10 as the center points. Selecting 5 as the window number can take into account these two high-frequency intervals at the same time, thus proving the reliability and effectiveness of the window of 5.

[0111] Weights for cross-modal soft alignment:

[0112] During the experiment, it was found that the value of contrast loss was lower than the cross entropy loss. In order to balance the two, an experiment on the weight of contrast loss was conducted to expand the contrast loss by integer multiples. However, the experimental results are shown in Table 6. When the weight ratio of contrast loss and cross entropy loss is 1:1, the effect is best. This may be because the text encoding is generated by other models and does not participate in the optimization of the entire model during training. If the contrast loss weight is too high, the model may focus too much on the alignment of sign language video features with text features, while ignoring the importance of the translation process.

[0113] Table 6

[0114]

[0115] Choice of text encoding model:

[0116] In order to achieve cross-modal soft alignment, the text model pre-trained by other scholars is used to encode the spoken text, so the encoding effects of different models are experimentally compared. Among them, Bert represents the "bert-base-german-uncased" model trained by the MDZDigital Library team, which is a Bert model trained on a large amount of German data. Sentence-Bert modifies the pre-trained Bert and uses twin networks and ternary network structures to obtain semantically meaningful text sentence embedding. The experimental results are as follows Figure 7 As shown in Figure 2, the results encoded using the Sentence-Bert model are better than those of the Bert model. This is because Sentence-Bert is stronger in understanding the entire sentence and can effectively capture sentence-level semantic information, thereby better supervising the visual representation of sign language videos.

[0117] The quality comparison is as follows:

[0118] Table 7 shows two example translations of GASLT and the model SLTFA of this application on the PHOENIX14T test set. In the first example, SLTFA produced a very accurate translation, indicating that it has a certain translation ability, while GASLT also expressed the main meaning, but it was not completely accurate, with an extraneous translation of "und machen sie es gut". In the second example, GASLT did not translate the correct meaning. Except for "am freitag", other words have no direct relationship with the real result. Compared with the correct result, the translation result of the SLTFA model has certain differences in expression, but the semantics are basically the same, which may be due to the diversity of descriptions from annotations to spoken language. As shown in Table 8, there is also such a situation in the training data. When the annotations are the same, the translation results are roughly the same, but there are differences in expression. Table 9 shows the translation results of SLTFA on the CSL-Daily test set. Although the word order is different in the first example, the words are the same and the meaning is the same. In the second example, for the same sign language video, the translation results of two sign language presenters are shown, and the results show that the difference in presenters has a certain impact on the model.

[0119] Comparison of translation results: The meaning of the wavy line is exactly the same as the reference. The double underline indicates the same meaning but different expression. The dotted line means it is completely wrong.

[0120] Table 7

[0121]

[0122]

[0123] Table 8

[0124]

[0125] Table 9

[0126]

[0127] Furthermore, the method for the sign language translation model to translate the sign language video to be translated includes:

[0128] Obtain the visual features of the sign language video to be translated through the pre-trained I3D model;

[0129] The visual features are input into the visual encoder, which calculates the pseudo-annotation intra-attention and pseudo-annotation inter-attention of the visual features based on the fusion attention mechanism;

[0130] Merge the attention between pseudo-annotations and the attention within pseudo-annotations to obtain sign language-text features;

[0131] The sign language-text features are decoded by a text decoder to obtain the spoken text.

[0132] The present invention has the following technical effects:

[0133] (1) Based on the inherent logic of sign language video to spoken language translation, a new fusion attention is proposed. It calculates attention from two perspectives: between pseudo-annotations and within pseudo-annotations, and fuses them to better understand sign language videos.

[0134] (2) Contrastive loss is introduced for soft alignment, and natural language is used to supervise visual representation to complete the translation from visual representation to natural language, effectively reducing the gap between cross-modalities.

[0135] (3) Experimental results show that the model proposed in this paper has achieved certain improvements in unannotated sign language translation and has higher accuracy and application potential than previous sign language translation methods.

[0136] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for unannotated sign language translation based on fusion attention, characterized in that: The following steps are involved: Obtain a sign language video and a sign language spoken text corresponding to the sign language video; Building an initial sign language translation model based on the Transformer architecture, and training the initial sign language translation model only through the sign language video and the sign language spoken text to obtain an unannotated sign language translation model; During the training process, the attention within the pseudo-annotations and the attention between the pseudo-annotations are calculated and fused, and the contrast loss and cross-entropy loss are introduced to train the model. The sign language video to be translated is translated by using the unannotated sign language translation model to obtain a spoken text output.

2. The method for unannotated sign language translation based on fusion attention according to claim 1, characterized in that: The method for training the initial sign language translation model includes: Obtain visual features of sign language videos through pre-trained I3D models; Inputting the visual features into a visual encoder, wherein the visual encoder calculates pseudo-annotation intra-attention and pseudo-annotation inter-attention of the visual features based on a fusion attention mechanism; Merge the attention between pseudo-annotations and the attention within pseudo-annotations to obtain sign language-text features; Embed the real sign language spoken text in the text decoder to obtain the representation vector of each text; The sign language-text features and the representation vector of each text are softly aligned through the contrastive loss function, and the model is optimized by combining the contrastive loss and cross entropy loss to obtain an unannotated sign language translation model.

3. The method for unannotated sign language translation based on fusion attention according to claim 1, characterized in that: The method for the sign language translation model to translate the sign language video to be translated includes: Obtain the visual features of the sign language video to be translated through the pre-trained I3D model; Inputting the visual features into a visual encoder, wherein the visual encoder calculates pseudo-annotation intra-attention and pseudo-annotation inter-attention of the visual features based on a fusion attention mechanism; Merge the attention between pseudo-annotations and the attention within pseudo-annotations to obtain sign language-text features; The sign language-text features are decoded by a text decoder to obtain spoken text.

4. The method for unannotated sign language translation based on fusion attention according to claim 2, characterized in that: The method for obtaining visual features of a sign language video comprises: The pre-trained I3D model is used to obtain the initial video features and the linear layer is used to convert the video features into the dimensions of the visual encoder and perform batch normalization. The normalized features are processed to obtain the embedded features of each frame.

5. The method for unannotated sign language translation based on fusion attention according to claim 2, characterized in that: The method for embedding real sign language spoken text includes: The word vectors of real sign language spoken text are converted into text representations of each word using the pre-trained subword embeddings in BPEmb as the initialization of the embedding layer.

6. The method for unannotated sign language translation based on fusion attention according to claim 2, characterized in that: Methods for obtaining sign language-text features include: The visual features are divided according to the windows, and attention is calculated for each window to obtain the result of the pseudo-annotation internal attention. The attention calculation of the window mark is performed to obtain the attention map of the relationship between each window. The attention map between pseudo-annotations is fused to weight the result vector of the pseudo-annotation internal attention, and then a residual connection is performed with the result of the pseudo-annotation internal attention to obtain the sign language-text feature.

7. The method for unannotated sign language translation based on fusion attention according to claim 6, characterized in that: The expression of the pseudo-annotation internal attention is: In the formula, It is a feature tag, which consists of a video tag and a window tag. q 、w k and w v Represent the three linear layers used for query, key and value calculation in regular self-attention. Attention is a standard self-attention. The expression of the attention between the pseudo annotations is: In the formula, and are the video frame label and window label within each window extracted from the output of IaPG, respectively; LN is the layer normalization layer, and Af is the activation function.

8. The method for unannotated sign language translation based on fusion attention according to claim 2, characterized in that: The model introduces contrast loss and cross entropy loss to jointly train the unannotated sign language translation model. The total loss function expression of the model is: is the cross entropy loss; The contrast loss function expression is: In the formula, {v i , t i } is a positive sample pair, {v i , t j }(i≠j) is a negative sample pair, s(.) is the similarity function, and T is the temperature hyperparameter.

Images