Annotation-free sign language translation method based on hierarchical dilatation attention and multi-modal semantic reconciliation

Abstract

The invention provides a non-annotation sign language translation method based on hierarchical dilatation attention and multi-modal semantic reconciliation, and relates to the technical field of sign language translation. The method comprises the following steps: extracting visual features of sign language video frames, superposing position codes, and sending the visual features into an attention distraction visual encoder to generate hierarchical encoding features; in the training stage, visual features and text embedding are aligned by adopting multi-modal semantic harmonic loss; and executing cross attention fusion in a text decoder, outputting target language probability distribution, and finally generating a translation sequence by combining a cross entropy loss optimization network. According to the method, accurate translation of complex sign language semantics under the condition of no manual annotation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sign language translation, in particular to a non-annotation sign language translation method based on hierarchical expansion attention and multi-modal semantic reconciliation. BACKGROUND

[0002] Sign language is an important tool for communication between deaf people. Unlike other language methods, it needs to be expressed through actions. Most able-bodied people have not systematically learned sign language and do not understand it, so there is a certain obstacle in communication between deaf people and able-bodied people, leading to deaf people constantly encountering various challenges in the process of trying to integrate into society. Sign language translation refers to translating sign language videos into spoken language texts that conform to the logic of normal communication and exchange, which is a significant task in the field of information accessibility, and can help able-bodied people understand the needs of the deaf and promote social inclusiveness. There are great differences in the order and grammar rules between sign language videos and spoken language texts, while annotations have the same order as sign language. Therefore, in the process of sign language translation, annotations as an intermediate representation can help the model better understand sign language videos and promote the learning of semantic boundary positions. Direct or indirect use of annotations has achieved good results in sign language translation. However, the acquisition of annotations is expensive and tedious, requiring sign language experts to spend a lot of time on annotation. Therefore, in recent years, more and more research has turned to non-annotation sign language translation methods, which directly learn to translate sign language videos into spoken language texts, although it is more challenging, but makes the sign language translation method more general, and also provides the possibility of using more extensive sign language resources.

[0003] The sign language translation task can be regarded as a sequence-to-sequence translation problem, so this paper chooses to build a non-annotation sign language translation model based on the Transformer framework. In the process of sign language expression, multiple frames of sign language video frames represent the same annotation. If the attention of the Transformer is directly used to process visual information, the attention to local information will be lost, resulting in insufficient attention between video frames within the same annotation. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a non-annotation sign language translation method based on hierarchical expansion attention and multi-modal semantic reconciliation, which significantly improves the expression ability and translation accuracy of the model for complex sign language videos.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A non-annotation sign language translation method based on hierarchical expansion attention and multi-modal semantic reconciliation, comprising:

[0007] visual features are generated for each video frame of a sign language video sequence and position encodings are superimposed, and text embeddings are generated for each text subword of a natural language text and position encodings are superimposed;

[0008] the visual features are fed into a visual encoder based on an expanded attention mechanism to obtain a sequence of hierarchical visual encoding features;

[0009] in the training phase, the sequence of visual encoding features and the text embeddings are cross-modal aligned based on a multi-modal semantic harmony loss function;

[0010] in each layer of the text decoder, the output of the previous decoding layer is taken as a query, and cross-attention fusion is performed on the sequence of visual encoding features as key-value to generate a probability distribution of the target language;

[0011] a cross-entropy loss is calculated based on the probability distribution of the target language, and is linearly weighted with the multi-modal semantic harmony loss function to update the network parameters;

[0012] a target language text sequence is generated according to the probability distribution of the target language until a terminal symbol is output, and a final translation sequence is obtained.

[0013] Preferably, the size of the i-th layer attention window of the visual encoder is w i = 2 i ; wherein w i is the size of the i-th layer attention window; the complexity of the i-th layer attention window is O(w i × t); wherein t is the number of video frames.

[0014] Preferably, the attention calculation formula of the i-th layer attention window is:

[0015]

[0016] wherein, is the query vector corresponding to the i-th layer of the t-th frame; and are the key and value vectors corresponding to the local window of the t-th frame, respectively; d k is the dimension of the key vector, is the attention of the i-th layer attention window.

[0017] Preferably, the calculation formula of the multi-modal semantic harmony loss function is:

[0018]

[0019] wherein L mmtl is the multi-modal semantic harmony loss function, d anchor,positive (i) represents the similarity between the text embedding and the positive sample visual feature, danchor,negative (i) is the difference between the text embedding and the negative sample video, alpha is a key hyperparameter, N is the number of triplets, and represents the number of triplet samples participating in the loss function calculation in a batch or an iteration.

[0020] Preferably, the calculation formula of the cross-attention mechanism of the text decoder is:

[0021]

[0022] wherein Q (i) is the query of the i-th layer of the text decoder, from the output of the previous layer of the text decoder, and are the key and value from the i-th layer of the visual encoder, T represents the transpose operation of the key matrix output by the visual encoder at the i-th layer, is the cross-attention mechanism of the text decoder.

[0023] Preferably, the calculation formula of the cross-entropy loss is:

[0024]

[0025] wherein, is the cross-entropy loss, y t is the index of the word at the t-th position of the target sentence in the vocabulary, P t is the probability distribution predicted by the model at the t-th step, and T' is the length of the target language sequence.

[0026] Preferably, the formula for linearly weighting the multi-modal semantic harmony loss function is:

[0027]

[0028] wherein, is the total loss, is the cross-entropy loss, L mmtl is the multi-modal semantic harmony loss function, and lambda is the weight for adjusting the proportion of the loss function.

[0029] According to the specific embodiments provided by the present application, the following technical effects are disclosed:

[0030] By introducing the expanded attention mechanism, the present application effectively expands the time receptive field of the model at different levels, realizes the collaborative modeling of local and global sign language time sequence information, and combines the multi-modal semantic harmony loss function to align and optimize the video and text embedding space, thereby improving the semantic mapping ability between sign language video and natural language text. The overall network can also fully excavate the semantic association between multi-modalities under the condition of no annotation, thereby improving the robustness and expression accuracy of the translation model in real application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0032] Figure 1 The method flowchart provided for the embodiments of the present application is as shown in

[0033] Figure 2 The model structure schematic diagram provided for the embodiments of the present application is as shown in

[0034] Figure 3 The expanded attention window schematic diagram provided for the embodiments of the present application is as shown in

[0035] Figure 4 The multi-modal triple loss schematic diagram provided for the embodiments of the present application is as shown in

[0036] Figure 5 The cross-attention schematic diagram provided for the embodiments of the present application is as shown in

[0037] Figure 6 The model layer number adjustment schematic diagram provided for the embodiments of the present application is as shown in

[0038] Figure 7 The BPEmb model vocabulary size selection schematic diagram provided for the embodiments of the present application is as shown in DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0041] Figure 1 The method flowchart provided for the embodiments of the present application is as shown in Figure 1 As shown in the method flowchart provided for the embodiments of the present application, the present application provides a kind of unannotated sign language translation method based on hierarchical expansion attention and multi-modal semantic reconciliation, comprising:

[0042] Step 100: generate visual features for each video frame of the sign language video sequence and superimpose position encoding, and generate text embedding for each text subword of the natural language text and superimpose position encoding;

[0043] Step 200: send the visual features into the visual encoder based on the expanded attention mechanism to obtain a sequence of hierarchical visual coding features;

[0044] Step 300: in the training phase, align the visual coding feature sequence and the text embedding based on the multi-modal semantic harmony loss function;

[0045] Step 400: in each layer of the text decoder, perform cross-attention fusion with the output of the previous decoding layer as the query and the visual coding feature sequence as the key-value to generate the probability distribution of the target language;

[0046] Step 500: calculate the cross-entropy loss based on the probability distribution of the target language, and linearly weight it with the multi-modal semantic harmony loss function to update the network parameters;

[0047] Step 600: generate the target language text sequence according to the probability distribution of the target language until the termination symbol is output to obtain the final translation sequence.

[0048] The specific model structure of the embodiment is as shown in Figure 2 The technical route is as follows:

[0049] Video text embedding: the video and the text are respectively initially embedded, and the position information is added, so that the video and the text representation conform to the input of the Transformer.

[0050] Visual encoder: the model does not use the traditional self-attention mechanism, but uses the expanded attention mechanism proposed, in the first layer, the single video frame focuses on only adjacent frames, and in the second layer, the frame can see frames in a larger range after it, and so on, the attention range of the frame gradually expands to a larger range of context.

[0051] Multi-modal semantic harmony: in the training process, a multi-modal triplet loss is introduced to match the text and the corresponding video, and the text space is harmonized with the video space, so that the visual encoder can better obtain the representation of the visual features and reduce the gap between the modalities.

[0052] Text decoder: the output of the visual encoder is taken as the input to generate text conforming to the logic of spoken language. In addition, since the perception range of each layer of the encoder for the video frame is different, the encoder has hierarchical feature expression, so the output of each layer of the encoder is transmitted to the decoder.

[0053] Similar to general sequence-to-sequence learning tasks, the model first embeds the input video and natural language text. For the video features x t , the video features are first converted to the dimension of the visual encoder using a linear layer, and then batch normalization (BN) is performed. Unlike the commonly used ReLU activation function, the Softsign function is used in this method because it can make the gradient flow more stable and reduce the problem of gradient vanishing and explosion, and is more suitable for processing image features. Through this method, the embedding features of each frame are obtained For text embedding, the pre-trained subword embedding in BPEmb is used as the initialization of the embedding layer (Emb). This approach is chosen because the logic of multiple subwords forming a word is similar to multiple video frames forming an annotation, so similar composition methods can help improve the accuracy of sign language translation tasks. To unify the text representation and the video representation, a method similar to visual feature embedding is used to convert the word vector into a text representation for each word Both sign language video and text have sequential information, and sign language video also has strong temporal information. In this method, the position information is added to the relevant representation according to the traditional Transformer position encoding method. The operation formula is as follows:

[0054]

[0055] where f pos (·) represents the injection of position information into the feature, and W1, b1, W2, b2 are parameters.

[0056] Optionally, the purpose of the visual encoder is to enrich the semantic information of the initial embedding of the video and fully consider the context of the video frames to improve the accuracy of sign language translation. In the sign language translation task, the quantity relationship between each annotation and its corresponding video frame is usually uncertain, and the number of video frames corresponding to each annotation is usually different, i.e., the number of video frames corresponding to a sign language action is affected by the complexity of the sign language action and the performance of the sign language demonstrator, making it have certain uncertainty. In the traditional self-attention mechanism, it is assumed that a single video frame has the same perception range for each layer, i.e., it can pay attention to all other video frames. This attention method makes it difficult to flexibly handle the uncertainty of the number of video frames, so it is difficult to form an effective representation while considering semantic information.

[0057] Traditional attention mechanism perceives too much range, considering global information, but ignoring local information, because video frames are more closely related to their neighbors. If only local information is considered, limiting the range of each layer of attention mechanism within the same window may lead to insufficient global information acquisition. Therefore, this paper proposes an expansion attention mechanism more consistent with sign language movements, which can gradually focus on video frames of different ranges by expanding the attention range of each layer step by step. In this way, the model can capture short-term and long-term cross-frame dependencies at each layer, gradually cover more context information, and better match the relationship between video frames and annotations.

[0058] First, focus on local features, then gradually expand the attention range to capture global information. The attention range of each self-attention layer is limited within a local window of size w, for example, for the t-th frame, only the attention weights of the frames within its local window need to be calculated. Then, the size of the local window w is expanded at the i-th layer, i.e. w = 2 i ,i = 1, 2…, as shown in Figure 3 , the attention range is gradually expanded as the number of layers increases. Attention calculation is performed on the divided window, for the t-th frame in the video sequence, the attention calculation at the i-th layer can be expressed as:

[0059]

[0060] where, is the query vector corresponding to the t-th frame at the i-th layer; and

[0061] are the key and value vectors corresponding to the video frames within the local window of the t-th frame; d k is the dimension of the key vector, which is used as a normalization factor to avoid large inner products.

[0062] As shown in Figure 2 , the video feature sequence that has completed the initial embedding enters multiple encoder blocks (Encoder Blocks). Each encoder block contains a layer normalization, an expansion self-attention mechanism, a feedforward neural network, and an Add&Norm structure.

[0063] The expansion attention mechanism designed in this way can better match the generation logic of sign language videos. First, since the attention range of each layer gradually increases, the SLTHDA model can better capture time-dependent relationships of different lengths, and understand global information while obtaining local information. In addition, the expansion attention mechanism can to some extent reduce the limitations brought by fixed window size, so that the model can better adapt to changes in sign language videos.

[0064] Meanwhile, compared with the traditional global self-attention mechanism, the expansion attention mechanism gradually expands the attention range, effectively reducing the computational complexity. Specifically, for a sign language video with t video frames, the computational complexity of the i-th layer and the local window with size w i = 2 i is O(w i x t), while the computational complexity of the traditional self-attention is O(t 2 ). The hidden benefit of this method is to enable the model to effectively process longer video sequences, avoiding the problem of low efficiency caused by the high computational complexity of the traditional Transformer in long sequence processing.

[0065] In the sign language translation task, it is necessary to translate sign language into spoken language text that conforms to the logic of normal communication and exchange. Compared with text-to-text machine translation, there is a clear semantic difference between video and text, so the direct mapping from video to text is more challenging and difficult. Therefore, the model proposes to use a multimodal triplet loss (MMTL) to harmonize the multimodal semantics. Through this loss function, the features of video and text are more matching and discriminative, thereby optimizing the encoder, so that the SLTHDA model can still maintain a certain sign language translation accuracy when testing.

[0066] Specifically, first, use the text encoding model to encode the sign language and spoken language text offline, which can reduce the difficulty of model training to some extent. Then, the text feature is regarded as the anchor, and the corresponding video feature is the positive sample, while the other video features in the same batch are negative samples. Using the multimodal triplet loss, the model will learn to make the distance between the text feature and the positive sample video feature as close as possible, and the distance between the text feature and the negative sample video feature as far as possible, as shown in Figure 4 MMTL optimizes the model by calculating the Euclidean distance difference between the anchor, positive sample and negative sample, so that the relative position of the text feature and the video feature in the feature space is more reasonable. The calculation formula of the loss formula is as follows:

[0067]

[0068] where d anchor,positive (i) is the Euclidean distance between the text feature and the corresponding video feature, that is, the similarity between the text and the positive sample video feature, d anchor,negative(i) is the Euclidean distance between the text feature and the negative sample video feature, that is, the difference between the text and the negative sample video. Alpha is usually called margin, as a key hyperparameter, it sets the minimum interval threshold of positive and negative sample pairs, so that the samples form a distinguishable decision boundary in the feature space, if the difference between the distance of the positive sample and the distance of the negative sample is less than alpha, the loss will increase, so as to further pull the positive sample and push the negative sample.

[0069] By minimizing MMTL, the model can effectively reduce the semantic distance between the text and the positive sample video, and pull away the semantic distance between the text and the negative sample video, and use the text space to reconcile the video space. On this basis, the SLTHDA model not only learns more distinguishable video features in the training process, optimizes the visual encoder, but also effectively reduces the gap between the video and the text modal, so that the output features of the encoder can contain more semantic information, which is crucial for the subsequent text generation of the decoder. At the same time, the model can also learn how to find the common representation of video and text in the multi-modal space, reconcile the video feature space and the text feature space, so as to get as close as possible to the text-to-text machine translation method, thereby improving the translation accuracy.

[0070] In addition, the text decoder is constructed based on the Transformer structure, which is trained jointly with the visual encoder to generate text that conforms to the logic of sign language spoken text. At the visual encoder end, the model uses an expanded attention mechanism, and the dependency relationship between video frames has both local details and global semantic information. Each layer of video frame semantic information will have different emphases, and each layer will have different outputs. In the traditional Transformer model, the decoder usually only receives the output from the last layer of the encoder. Obviously, this way cannot fully utilize the output of the expanded attention, and cannot effectively utilize the multi-level temporal information. In order to fully utilize the feature information of each layer of the visual encoder, this paper proposes a hierarchical mode. The model not only relies on the output of the last layer, but also transmits the output of each layer of the encoder to the corresponding layer of the decoder. In this way, the decoder can fuse the corresponding layer feature representation from the encoder at each layer through the cross-attention mechanism.

[0071] In the second attention mechanism of the decoder, that is, the cross-attention mechanism, the query comes from the output of the previous layer of the decoder itself, and the key and value come from the output features of the i-th layer of the encoder, as shown in Figure 5 By using this way, the decoder can effectively fuse the semantic feature information from the corresponding layer of the encoder at each layer. The cross-attention mechanism calculation formula of the decoder is as follows:

[0072]

[0073] where Q (i) is the query of the i-th layer of the decoder, from the output of the previous layer of the decoder, and are the key and value from the i-th layer of the encoder.

[0074] Through this hierarchical mode, the text decoder can not only utilize the output of the last layer of the visual encoder, but also make full use of the multi-level feature information from each layer of the encoder, which enables the decoder to capture fine-grained local information and global semantic information, better understand the time sequence dependence in the video, and thus improve the accuracy of sign language translation.

[0075] Further, the text decoder generates text by utilizing the information of the visual decoder. Since the task of the sign language translation model belongs to machine translation, the model is still optimized by using a cross-entropy loss function when being constructed. The effect of the SLTHDA model is as follows, and let the target sentence be y=(y1,y2,…,y t ), wherein y t is the index of the word at the t-th position of the target sentence in the vocabulary table, and the probability distribution predicted by the model at the t-th step is P t , then the loss function is:

[0076]

[0077] The SLTHDA jointly uses a multi-modal triplet loss and a cross-entropy loss for optimization in the training process. In the process of calculating the loss, a related weight is added to adjust the relationship between the two loss functions, so as to achieve the optimum. In the testing process, the cross-entropy loss is used to optimize the model, because the additional text features cannot be used to reconcile the video features. In the model training process, the loss function is as follows, wherein λ is the weight for adjusting the proportion of the loss function.

[0078]

[0079] As an optional implementation, the embodiment first performs related experiments of the SLTHDA model on the PHOENIX14T and CSL-Daily data sets, and the results are shown in Tables 1 and 2. The experimental results of the model are compared with other public and better-performing unannotated sign language translation models to verify the effectiveness of the SLTHDA model. The actual translation effect in the test set on the PHOENIX14T data set is observed and analyzed to further verify the translation ability of the SLTHDA model.

[0080] Quantitative comparison:

[0081] On the PHOENIX14T dataset, as shown in Table 1, the SLTHDA model showed obvious advantages, and performed better than other models in the table in terms of BLEU-1, BLEU-2 and BLEU-4 scores. The SLTHDA model has similar results to the GASLT model, although the result is slightly worse in BLEU-3, but it has improved in other indicators, and has a large improvement compared to other models, which shows that the SLTHDA model has better capture ability in short-term and long-term dependencies, can better capture the information of visual features, and can better handle the complex relationship between sign language video and text, effectively improving the performance of sign language translation. At the same time, in terms of ROUGE score, the proposed SLTHDA model also achieved a result of 40.15, which is 0.29 higher than the GASLT model, further verifying the advantages of SLTHDA in the fluency and relevance of generated text.

[0082] Table 1 Experimental results on the PHOENIX14T dataset

[0083]

[0084] Table 2 Experimental results on the CSL-Daily dataset

[0085]

[0086]

[0087] On the CSL-Daily dataset, the experimental results are shown in Table 2, and the SLTHDA model still shows strong performance, especially in terms of BLEU-1, BLEU-2 and BLEU-3 scores, which are 7.83, 2.65 and 0.94 higher than the GASLT model respectively, while the BLEU-4 scores of the two models are similar, which shows that the SLTHDA model has a significant advantage in handling translation quality, and also shows that the SLTHDA model has greater advantages in capturing semantic accuracy on the CSL-Daily dataset. At the same time, the model improves the ROUGE score by 7.86 compared to the optimal result, which also shows that the proposed SLTHDA model has strong ability to generate more accurate and relevant text.

[0088] The SLTHDA model achieves good results on the PHOENIX14T and CSL-Daily datasets, which shows that the model also has certain translation ability in the case of multiple languages, and proves that the structure of the SLTHDA model is effective.

[0089] Qualitative comparison:

[0090] To visually verify the translation effect of SLTHDA model, the translation cases of test set in PHOENIX14T dataset are qualitatively analyzed in this embodiment, and the translation results of reference translation (Ref), GASLT model and SLTHDA model in two examples are listed in Table 3. As can be seen from the table, for the short spoken text "guten abend liebezuschauer" (good evening, dear audience), the translation results of the two models are correct and consistent, which shows that in the case of short spoken text and small number of actions in sign language video, both SLTHDA model and GASLT model can accurately capture the meaning of gestures and achieve good sign language translation effect. However, in the second example "und nun die wettervorhersage für morgen freitag den fünfzehnten oktober" (now is the weather forecast for tomorrow Friday, October 15), both models have some time information errors. As can be seen from Table 3, the first half of the sentence translated by the GASLT model is correct, but the day of the week is wrong, translating Friday as Tuesday, and the date and month are not translated, even introducing irrelevant information such as "snow". Moreover, the grammar of the translation result is also problematic. In German, "den + certain date" is the correct format, but "Schnee" (snow) is a common noun, not a date. Although the translation result of the SLTHDA model in this paper has some deviations, the grammar structure of the translation result is closer to the reference translation, and the understanding ability of the sign language video is stronger, and the text generation ability is also stronger, which shows that by continuously expanding the window, the model has stronger context understanding ability.

[0091] Although the SLTHDA model performs well in the actual translation results compared with the GASLT model, it translates "fünfzehnten Oktober" (October 15) to "vierzehnten Februar" (February 14), which is an error in the translation of dates and months. In order to further analyze the model's understanding of sign language videos, this embodiment demonstrates relevant sign language video frames for "15 (fünfzehnten)" and "14 (vierzehnten)" and "Oktober (October)" and "Februar (February)". From the results, it can be observed that these gestures only have slight differences in shape or position. For example, "15" and "14" are both right-hand finger up-and-down movements for two times, "15" is five fingers, and "14" is four fingers. The sign language actions for "October" and "February" both involve holding the right hand, with slight differences in the amplitude and slightly different swinging trajectories, but the positions of the fists are similar. From the sign language video frames, these gestures have slight differences in details, which may be easily ignored or not fully encoded during visual embedding, leading to the model's misjudgment of 15 as 14 and confusion of October as February in the subsequent translation process. This explains why SLTHDA has time information bias in the overall syntactic correctness in Table 3.

[0092] From the qualitative results, it can be seen that the SLTHDA model uses hierarchical expansion attention mechanism, which can capture local and global information to some extent, thus being more superior in the overall sentence structure and grammatical correctness, and having certain advantages in longer sentence translation, but still has certain limitations in recognizing the details of sign language actions.

[0093] Table 3 Quality results on PHOENIX14T dataset

[0094]

[0095]

[0096] Further, this embodiment conducts an ablation experiment. In this embodiment, the performance advantages of the SLTHDA model will be introduced in detail through the ablation experiment on the PHOENIX14T dataset. This section designs and conducts six groups of ablation experiments, including the effectiveness of key modules, the comparison between expansion attention and fixed window, the adjustment of model layers, the weight of multi-modal semantic reconciliation, the margin of multi-modal triplet loss, and the selection of BPEmb encoding model, to systematically study and analyze the optimal combination of each component and parameter and its influence on the model performance.

[0097] Effectiveness of key modules:

[0098] In this experiment, the effectiveness of the proposed expanded attention mechanism, hierarchical pattern and multi-modal semantic reconciliation is verified. Under the condition of keeping other conditions consistent, the corresponding module is removed for experiment, compared with the best model and analyzed. Table 4 presents the experimental comparison results, where w / o indicates the ablation experiment of removing the corresponding module. From the results, it can be seen that the complete SLTHDA model reaches the highest score in each index, which also shows that the proposed modules cooperate with each other and form a synergistic effect in the whole, effectively improving the translation performance of the model.

[0099] From Table 4, it can be seen that after replacing the expanded attention module with the traditional attention mechanism, the model has a decrease in BLEU and ROUGE scores, which are 3.84, 4.11, 3.86, 3.57 and 4.08 respectively, with a relatively obvious decrease, which shows that the expanded attention can effectively capture short and long distance temporal dependencies and improve the interaction representation between video frames by expanding the attention range layer by layer, obtaining video features with rich semantic information, thereby significantly improving the translation quality. Compared with SLTHDA, after removing the hierarchical pattern, the BELU and ROUGE scores also decrease slightly, which are 0.54, 0.96, 1.27, 1.30 and 0.53 respectively, although the decrease is smaller than when the expanded attention is removed, but it still shows that the introduced hierarchical pattern has a positive effect on the model performance, verifying that this module can make the decoder fully utilize the semantic information of each level of the encoder, and enhance the comprehensive understanding of local information and global information of video frames. From the table, it can be seen that when the multi-modal semantic reconciliation module is removed, compared with SLTHDA, the BLEU and ROUGE scores decrease by 1.12, 1.05, 1.17, 1.22 and 1.46 respectively, which also decrease to a certain extent, which shows that multi-modal semantic reconciliation can affect the representation of video features, optimize the video encoder, effectively reduce the gap between video and text modalities, and alleviate the problem of inconsistent feature spaces between cross-modalities.

[0100] When the expanded attention module is removed, the decrease compared with SLTHDA is more obvious than when the hierarchical pattern and multi-modal semantic reconciliation module are removed, which also indirectly shows the importance and effectiveness of the expanded attention mechanism, which plays a key role in sign language video understanding. At the same time, from Table 4, it can be found that removing any one of the modules will cause the model to have different degrees of performance degradation, further proving the effectiveness of the cooperation of these modules.

[0101] Table 4 Effectiveness of key modules

[0102]

[0103] Comparison of expanded attention and fixed window:

[0104] The SLTHDA model proposed in this embodiment uses an expanded attention mechanism to capture visual features with more semantic information. In the same layer, the attention range is consistent for each frame. This form is equivalent to a fixed window, but the window of each layer is changing. To verify the effectiveness of the variable window designed by the expanded attention mechanism, the embodiment conducts relevant experiments. The fixed window of each layer in the entire model is set with the same parameters, and the other settings are the same as those of the SLTHDA model, and then the model is compared with the present model. The specific comparison is shown in Table 5. The table shows the performance of the model in BLEU and ROUGE indicators under different attention ranges. In the experimental setting, the attention range of SLTHDA is gradually expanded, the number of layers of the SLTHDA model is 5, and according to the window mechanism of the expanded attention, the attention range is fixed at 2, 4, 8, 16 and 32 respectively, to observe the influence of different attention ranges on the performance of the model.

[0105] As can be seen from Table 5, when the attention range is 4, the BLEU-1 reaches the optimal value of 40.06, indicating that the short-range attention mechanism can effectively capture local information and improve the matching degree at the word level, and has certain advantages in short-distance dependency modeling. However, BLEU-4 and ROUGE decrease compared with SLTHDA, indicating that relying only on local information may not be enough to improve the quality of long sequence translation, and the support of global information is still needed. With the expansion of the attention range, the model cannot continuously improve in BLEU-1, BLEU-2 and other indicators, but there is a slight fluctuation in BLEU-4 and ROUGE indicators. Especially when the attention range is 8, the ROUGE score reaches the best performance of 40.32, indicating that when the attention range is large, the global information modeling capability can be enhanced to a certain extent, which helps to improve the overall fluency of the text. However, its BLEU-4 is still lower than that of the complete model, with a gap of 7.3%, indicating that relying only on the expansion of the attention range may not be enough to improve the translation accuracy. In addition, when the attention range is increased to 32, the model has a slight rebound in BLEU and ROUGE scores compared with the attention range of 16, but the indicators are still lower than those of SLTHDA, which indicates that too large attention range may lead to the loss of local information and affect the matching ability of phrases. However, this configuration achieves a ROUGE score of 39.84, indicating that it still has certain advantages in modeling the overall sentence structure.

[0106] SLTHDA achieves the highest value in BLEU-2, BLEU-3 and BLEU-4 indicators, and performs well in BLEU-1 and ROUGE, indicating that the strategy of gradually expanding the attention range can effectively combine local and global information, enabling the model to find a balance between short-distance dependency modeling and long-distance dependency capturing, thereby improving the accuracy of sign language translation.

[0107] Table 5 Experimental comparison of dilated attention and fixed window

[0108]

[0109] Adjustment of the number of model layers:

[0110] The number of model layers affects model performance. As the number of layers increases, the model's learning and representation capabilities also increase. However, if the number of layers is too large, it may affect the model's efficiency and performance. For the SLTHDA model, due to the inclusion of the expanded attention mechanism, each layer has a different attention range. The number of layers affects the model's global and local perception capabilities. Therefore, determining the number of model layers is a key experiment.

[0111] The experimental results are as follows Figure 6 As shown in the figure, as the number of SLTHDA model layers increases from 3 to 5, the BLEU and ROUGE metrics show an overall upward trend, demonstrating that deeper network structures can, to a certain extent, help capture the more complex temporal and semantic dependencies between sign language video and text. When the SLTHDA model layer number is set to 5, the BLEU-4 score reaches 16.55, and the ROUGE score increases to 40.15, achieving optimal or near-optimal performance in all metrics. This also shows that, given the current data scale and task complexity, moderately deepening the network layer number can effectively enhance the expressive power of sign language translation models.

[0112] However, when the model's layers were increased to 6, both BLEU and ROUGE metrics declined to some extent: BLEU-1 dropped from 39.16 to 38.71, BLEU-4 dropped from 16.55 to 15.47, and ROUGE dropped from 40.15 to 39.56. Overall model performance declined. This may be due to increased training difficulty caused by the network's excessive depth, or increased overfitting of features, which limited further improvement in model performance. Overall, the 5-layer network structure achieved a relatively ideal balance between capturing multi-level semantic information and maintaining trainability.

[0113] Optionally, multimodal semantic reconciliation utilizes text features to adjust the video feature space during model training, using a multimodal triplet loss to achieve this. When jointly optimizing the model, cross-entropy loss and multimodal triplet loss require careful consideration to find a balance between these two loss functions to achieve optimal model performance. Table 6 shows the impact of different MMTL weights on model performance when jointly training with cross-entropy loss.

[0114] To explore the optimal weight setting of the multi-modal triple loss, the experiment first tried a larger range, setting the weight value to 1, 5, 10, 15, 20, 25, and 30. The experimental results showed that when the weight was set to 10 and 15, the model performed better, indicating that the model could well balance the influence of text features on video features, effectively reducing the difficulty of cross-modal translation. To further refine the model's effect, the experiment continued in the interval of 10 to 15, hoping to obtain the best effect. The experimental results are shown in Table 6. It was finally found that when the weight was set to 12, the model achieved the best performance on BLEU-2, BLEU-3, BLEU-4, and ROUGE indicators, with values of 27.10, 20.55, 16.55, and 40.15, respectively, while BLEU-1 only differed by 0.06 from the optimal result.

[0115] These experimental results verify the importance of properly adjusting the multi-modal triple loss weight for optimizing cross-modal feature alignment, and also show that the SLTHDA model has high robustness in capturing the semantic relationship between video and text. Lower weight values result in insufficient cross-modal alignment constraints, making it difficult to sufficiently narrow the distance between text embeddings and corresponding video embeddings, while higher weight values may result in excessive cross-modal alignment constraints, affecting the final generation quality.

[0116] Table 6 Weight of multi-modal semantic harmonization

[0117]

[0118] Further, the margin parameter in the multi-modal triple loss, denoted by α, has a certain degree of influence on the model performance. When the margin value is small, the loss value is more likely to approach 0, but the model is less likely to distinguish similar data samples. When the margin value is large, not only is it difficult for the loss value to approach 0, but the network also fails to converge. However, setting a large margin value can more clearly distinguish similar data samples. Therefore, setting a reasonable margin value is crucial.

[0119] As can be seen from Table 7, as the margin parameter increases from 0.1 to 1.0, the SLTHDA model shows a certain fluctuation trend in various indicators. Among them, when margin = 0.2, the SLTHDA model reaches the highest value in BLEU-2, BLEU-3 and BLEU-4, although it does not achieve the best effect in BLEU-1 and ROUGE indicators, but it is not much different from the best effect, such as the ROUGE score only differs by 0.12. In addition, as can be seen from Table 7, for the values of BLEU-3 and BLEU-4, when margin is 0.2, except that individual results are close to the highest value, other results are quite different from the highest value, therefore, from the whole, when margin = 0.2, the SLTHDA model can obtain better performance in different indicators. At the same time, when the margin value is 0.7 to 1.0, it can be seen that BLEU-1 and BLEU-2 have better results, indicating that when the margin value is too large, the generated short sequence translation text has more ability, but it still has certain limitations in fluency and long sequence text generation.

[0120] When margin is 0.2, the SLTHDA model can maximize the distance between the text and the positive sample video, realize the focus on the positive sample, maintain the separation from the negative sample video, and fully distinguish the features of different videos, thereby improving the translation performance to a certain extent.

[0121] Table 7 margin adjustment of multi-modal triple loss

[0122]

[0123] In the SLTHDA model, the pre-trained subword embeddings in BPEmb are used as the initialization of the embedding layer. BPEmb is a set of pre-trained subword embeddings, which is based on Byte Pair Encoding (BPE) and trained on Wikipedia corpus for 275 languages. Its main use is as input for neural models in natural language processing, including German and Chinese. In this section, the impact of different BPE vocabulary sizes on model performance is explored. The vocabulary size is the sum of the number of BPE merge operations and the number of characters in the training data. The number of BPE merge operations determines whether the generated symbol sequence is biased towards short (few merge operations) or long (many merge operations). Using a small number of merge operations will generate a large number of single, double, and triple words, while performing a large number of merge operations will generate symbols representing the most common words. With fewer operations, the vocabulary size will also decrease, and more data will be needed to learn how to combine these symbols into meaningful units (such as words). More operations will allow many common words to have their own symbols, requiring more data to train good embeddings for these longer symbols, which is suitable for high-resource languages but may not be as suitable for some low-resource languages. Therefore, the size of the BPE vocabulary will affect the performance of the model.

[0124] To achieve the optimal experimental results, experiments on the size of the BPE vocabulary were conducted, Figure 7 The experimental results of the SLTHDA model on the PHOENIX14T dataset are shown, where 25k (25000), 50k (50000), 100k (100000), and 200k (200000) represent the size of the BPE vocabulary used. From the figure, it can be observed that when the vocabulary size is set to 50k, the SLTHDA model achieves the best performance in BLEU-2, BLEU-3, BLEU-4, and ROUGE indicators, especially in the ROUGE indicator, which improves by 2.12 compared to the result with a vocabulary size of 25k, and has a certain gap with other results, with a relatively significant improvement.

[0125] The BLEU-4 of the model under the 25k vocabulary size is only 14.63, and using a smaller vocabulary table leads to a decrease in subword granularity, which affects the model's ability to handle low-frequency words and compound words. As the vocabulary table further increases, although there is a certain improvement in BLEU-1, the optimal result of 39.36 is achieved under the 200k vocabulary, but other indicators do not achieve the best performance. Using a larger vocabulary table can divide the text into more subword units, but it may lose some of the semantic integrity of the words. A moderate subword division can preserve more word information and effectively alleviate the problem of low-frequency words, thereby promoting the model's learning of the text semantics.

[0126] The ablation experiment verifies that the SLTHDA model can effectively segment low-frequency words while maintaining sufficient semantic integrity on the PHOENIX14T German weather dataset using a 50k size BPE vocabulary, thereby making the text embedding more expressive, and proves the importance of moderate vocabulary size for the cross-modal translation task, and provides a strong experimental basis for the design of the text embedding module of the present research.

[0127] The various embodiments are described in a progressive manner in the specification, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be mutually referred to.

[0128] The principles and implementation modes of the present application are described by applying specific examples in this paper, and the above description of the examples is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for unannotated sign language translation based on hierarchical dilated attention and multimodal semantic reconciliation, characterized by: include: Generate visual features and superimpose position codes for each video frame of the sign language video sequence, and generate text embedding and superimpose position codes for each text subword of the natural language text; The visual features are fed into a visual encoder based on the dilated attention mechanism to obtain a hierarchical visual encoding feature sequence; During the training phase, cross-modal alignment is performed on the visual encoding feature sequence and the text embedding based on a multimodal semantic reconciliation loss function; At each layer of the text decoder, the output of the previous decoding layer is used as a query, and the visual encoding feature sequence is used as a key-value to perform cross-attention fusion to generate a target language probability distribution; Calculating the cross entropy loss with the target language probability distribution and linearly weighting it with the multimodal semantic reconciliation loss function to update the network parameters; A target language text sequence is generated according to the target language probability distribution until a terminator is output to obtain a final translation sequence.

2. The method for unannotated sign language translation based on hierarchical expanded attention and multimodal semantic reconciliation according to claim 1 is characterized in that: The size of the attention window of the i-th layer of the visual encoder is w i =2 i ; Among them, w i is the size of the attention window of the i-th layer; the complexity of the attention window of the i-th layer is O(w i ×t); where t is the number of video frames.

3. The method for unannotated sign language translation based on hierarchical expanded attention and multimodal semantic reconciliation according to claim 2 is characterized in that: The attention calculation formula of the i-th layer attention window is: in, is the query vector corresponding to the t-th frame at layer i; and are the key and value vectors corresponding to the video frame in the local window of the t-th frame; d k is the dimension of the key vector, is the attention of the i-th layer attention window.

4. The method for unannotated sign language translation based on hierarchical expanded attention and multimodal semantic reconciliation according to claim 1 is characterized in that: The calculation formula of the multimodal semantic reconciliation loss function is: Among them, L mmtl is the multimodal semantic reconciliation loss function, d anchor,positive (i) represents the similarity between the text embedding and the positive sample video features, d anchor,negative (i) represents the difference between the text embedding and the negative sample video, α is the key hyperparameter, and N is the number of triplets, which indicates the number of triplet samples participating in the loss function calculation in one batch or one iteration.

5. The method for unannotated sign language translation based on hierarchical expanded attention and multimodal semantic reconciliation according to claim 1 is characterized in that: The calculation formula of the cross attention mechanism of the text decoder is: Among them, Q (i) is the query of the i-th layer of the text decoder, which comes from the output of the previous layer of the text decoder, and are the keys and values ​​from the i-th layer of the visual encoder, T represents the transpose operation of the key matrix output by the visual encoder at the i-th layer, is the cross attention mechanism of the text decoder.

6. The method for unannotated sign language translation based on hierarchical expanded attention and multimodal semantic reconciliation according to claim 1, characterized in that: The calculation formula of the cross entropy loss is: in, is the cross entropy loss, y t is the index of the word at position t in the target sentence in the vocabulary, P t is the probability distribution predicted by the model at step t, and T' is the length of the target language sequence.

7. The method for unannotated sign language translation based on hierarchical expanded attention and multimodal semantic reconciliation according to claim 1, characterized in that: The formula for linearly weighting the multimodal semantic reconciliation loss function is: in, is the total loss, is the cross entropy loss, L mmtl is the multimodal semantic reconciliation loss function, and λ is the weight for adjusting the proportion of the loss function.

Images