Sign language translation method and device based on cross-modal retrieval enhancement, equipment and medium

Abstract

The application discloses a sign language translation method and device based on cross-modal retrieval enhancement, equipment and medium, relates to the field of sign language translation, and comprises the following steps: spatial and motion features are extracted by using a visual encoder, and global sign language visual features are obtained through down-sampling and dimension mapping; word and sentence features are extracted by using a text encoder; cross-modal retrieval is performed in the word and sentence features by taking the global sign language visual features as a query vector, and a fusion feature is formed by adaptive weighting; text semantic is injected into the global sign language visual features by using a cross-modal encoder to obtain an enhanced visual sequence; and after splicing with a prompt word, the enhanced visual sequence is input into an LLM to generate a sign language translation result. Through the history context dynamic retrieval denoising guided by the visual features, the context redundant noise of the multi-round dialogue is effectively removed; the preposed multi-task joint loss and double alignment are used to eliminate the modal difference gap, thereby effectively inhibiting the semantic deviation and generation illusion of the large model under the complex context, and the sign language translation accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of sign language translation technology, and in particular to a sign language translation method, apparatus, device, and medium based on cross-modal retrieval enhancement. Background Technology

[0002] Sign language translation (SLT) is a cross-modal generation task that converts continuous sign language videos into natural language text. One approach is gloss-based sign language translation, which first uses a Continuous Sign Language Recognition (CSLR) model to map video frames into discrete sequences of intermediate words (Gloss), and then uses a translation model to combine these sequences into fluent natural language sentences. However, this approach relies on high-quality, costly, manually annotated Gloss data. Another approach is gloss-free end-to-end sign language translation, which directly establishes a mapping network from the original video stream to the target text. It utilizes a Large Language Model (LLM) with hundreds of billions of pre-trained corpora as the text generation foundation. A visual extractor obtains the visual feature matrix from the sign language video, and a projection layer maps it to the input space of the LLM. Leveraging the powerful logical reasoning and language generation capabilities of the LLM, the translated sign language text is directly output.

[0003] However, due to the significant modal differences between visual features and discrete natural language text features, simply using tensor concatenation to fuse sign language visual features with historical dialogue text features will introduce severe historical dialogue context noise into the translation process due to the lack of fine-grained retrieval and filtering of historical dialogue text. This distracts the attention mechanism from focusing on the current core visual action, severely interfering with the semantic inference of sign language actions by the LLM decoder, and is also prone to causing sequence length explosion and memory overflow. Especially when dealing with heterogeneous multimodal features of long video sequences, it is impossible to effectively and deeply fuse the significantly different multimodal features, leading to gradient vanishing and difficulty in training convergence in the sign language translation network. This results in huge computational overhead while failing to guarantee the accuracy of the sign language translation results. Summary of the Invention

[0004] This invention provides a sign language translation method, apparatus, device, and medium based on cross-modal retrieval enhancement to address the technical problems of severe contextual noise in multi-turn dialogue scenarios and low accuracy of sign language translation results due to large differences in heterogeneous multimodal features in long video sequences.

[0005] In a first aspect, embodiments of the present invention provide a sign language translation method based on cross-modal retrieval enhancement, comprising: S101 uses a visual encoder to extract sign language spatial features and sign language motion features from sign language videos, and performs feature downsampling fusion and dimension mapping to form global sign language visual features; S102, using a text encoder to extract word text features and sentence text features from the historical dialogue text corresponding to the sign language video; S103 utilizes global sign language visual features as query vectors, performs cross-modal retrieval with word text features and sentence text features respectively, obtains word-level retrieval features and sentence-level retrieval features, and performs adaptive weighted fusion to form fused features; S104 utilizes a cross-modal encoder to inject the textual semantics contained in the fused features into the global sign language visual features through cross-attention, resulting in an enhanced visual sequence; S105, the enhanced visual sequence is concatenated with the preset cue word feature matrix to obtain a multimodal input long sequence, which is then input into a trained large language model for autoregressive text generation to obtain the sign language translation result.

[0006] Secondly, embodiments of the present invention provide a sign language translation device based on cross-modal retrieval enhancement, comprising: The visual encoding module is used to extract sign language spatial features and sign language motion features from sign language videos using a visual encoder, and to perform feature downsampling fusion and dimension mapping to form global sign language visual features; The text encoding module is used to extract word text features and sentence text features from the historical dialogue text corresponding to the sign language video using a text encoder. The cross-modal retrieval fusion module is used to utilize global sign language visual features as query vectors, and perform cross-modal retrieval with word text features and sentence text features respectively to obtain word-level retrieval features and sentence-level retrieval features, and then perform adaptive weighted fusion to form fused features; The cross-modal coding enhancement module is used to inject the textual semantics contained in the fused features into the global sign language visual features through cross-attention, thereby obtaining an enhanced visual sequence; The autoregressive text generation module is used to concatenate the enhanced visual sequence with the preset cue word feature matrix to obtain a multimodal input long sequence, which is then input into a trained large language model for autoregressive text generation to obtain sign language translation results.

[0007] Thirdly, embodiments of the present invention provide an electronic device, including: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described sign language translation method based on cross-modal retrieval enhancement.

[0008] Fourthly, embodiments of the present invention provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the aforementioned sign language translation method based on cross-modal retrieval enhancement.

[0009] This invention provides a sign language translation method, apparatus, device, and medium based on cross-modal retrieval enhancement. The method extracts global visual features of sign language from sign language videos and corresponding word and sentence text features from historical dialogues. It then uses the global visual features as an active query vector for cross-modal retrieval and adaptive weighted fusion. Next, it injects the purified historical text semantics into the visual sequence using cross-attention to form an enhanced visual sequence. Finally, it concatenates this sequence with a feature matrix derived from prompt words and inputs it into a large language model for autoregressive generation to obtain the sign language translation result. Simultaneously, it constructs a multi-task joint loss function using visual-target text alignment loss, visual-historical dialogue context alignment loss, and large language model autoregressive generation loss. This function is then combined with real target translation text to optimize the large language model for cross-modal and context alignment. By accurately denoising the historical context centered on the current sign language action, redundant noise in multi-turn sign language dialogues is effectively eliminated, avoiding the dilution of LLM attention weights. By utilizing a multi-task joint loss function with pre-alignment combined with double alignment, when facing the modal difference gap between high-dimensional continuous video streams and discrete natural language symbols, the visual encoder is forced not only to accurately capture isolated sign language actions, but also to deeply perceive the historical text context. This reduces the cognitive mapping pressure of LLM, effectively suppresses semantic deviation and generative illusions generated by large models in complex contexts, and improves the accuracy of sign language translation. Attached Figure Description

[0010] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a sign language translation method based on cross-modal retrieval enhancement as described in Embodiment 1 of the present invention; Figure 2 This is a structural diagram of the contextual information fusion network described in Embodiment 1 of the present invention; Figure 3 This is a flowchart of a sign language translation method based on cross-modal retrieval enhancement as described in Embodiment 2 of the present invention; Figure 4 This is a structural diagram of the 1D temporal convolutional network described in Embodiment 2 of the present invention; Figure 5 This is a network structure diagram of the cross-modal retrieval enhancement module described in Embodiment 2 of the present invention; Figure 6This is a network structure diagram of the cross-modal encoder described in Embodiment 2 of the present invention; Figure 7 This is a schematic diagram of the structure of a sign language translation device based on cross-modal retrieval enhancement according to Embodiment 3 of the present invention; Figure 8 This is a structural diagram of the electronic device described in Embodiment 4 of the present invention. Detailed Implementation

[0011] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0012] Example 1 Figure 1 The flowchart below illustrates a sign language translation method based on cross-modal retrieval enhancement as described in Embodiment 1 of this invention. Guided by sign language visual features, it dynamically denoises and refines historical dialogue text, accurately extracting fine-grained word text features and macro-level sentence contextual information features from historical dialogues. This achieves pre-alignment and deep cross-modal fusion of high-purity historical dialogue text features with current sign language visual features, suppressing severe contextual noise and the generation illusion of large language models in historical dialogue texts, thereby improving translation accuracy in multi-turn sign language actions and complex dialogue scenarios. Specifically, the method includes the following steps: S101 uses a visual encoder to extract sign language spatial features and sign language motion features from sign language videos, and performs feature downsampling fusion and dimension mapping to form global sign language visual features.

[0013] Sign language videos are high-dimensional continuous visual streams containing complex three-dimensional spatial and temporal evolutions. When performing gesture recognition on sign language videos, it is necessary to first convert the video frame sequence into a high-dimensional, high-density semantic sequence. However, directly inputting the high-dimensional, high-density semantic sequence into an intelligent model can lead to problems such as computational overhead and memory overflow. Therefore, a visual encoder is first used to extract sign language spatial features, which represent static visual spatial topological information such as hand shape, facial expressions, and body posture, as well as sign language motion features, which represent spatiotemporal dynamic features of the hand, such as hand movement trajectory and limb displacement. The visual encoder consists of two parallel processing branches: an image encoder branch and a video encoder branch, which are used to extract sign language spatial features and sign language motion features, respectively. Then, a downsampling network is used to compress the long sequence, removing redundant and meaningless frames. Finally, it is projected onto a dimension consistent with the hidden space of the large language model to obtain global sign language visual features.

[0014] S102 uses a text encoder to extract word text features and sentence text features from the historical dialogue text corresponding to the sign language video.

[0015] In multi-turn sign language dialogues, there is often a large amount of contextual noise that is irrelevant to the current sign language content to be translated. Furthermore, sign language is highly context-dependent. By using the word embedding layer and word segmenter of the text encoder to encode features of unstructured historical dialogue text, a structured multi-level text vector is formed. Fine-grained word-level representations of word text features are extracted, which can be vector representations of each word unit. Additionally, sentence text features of the entire sentence semantic representation are extracted by masking average pooling of word unit features, which serve as the basis for cross-modal denoising text features.

[0016] S103 utilizes global sign language visual features as query vectors, performs cross-modal retrieval with word text features and sentence text features respectively, obtains word-level retrieval features and sentence-level retrieval features, and performs adaptive weighted fusion to form fused features.

[0017] Historical dialogue texts often contain redundant information unrelated to sign language actions. Due to the context-dependent nature of sign language actions, global sign language visual features representing the current visual action can be used as query vectors. These vectors are then used for cross-modal retrieval and matching with word and sentence text features. Cosine similarity to the current visual action is calculated for each feature, and weights are dynamically assigned based on this similarity. Words and sentences highly relevant to the sign language action are assigned higher weights, while those less relevant or irrelevant are assigned lower weights. This process suppresses noise and removes irrelevant words. The weighted summation forms a fusion feature that incorporates the sign language action's context. This approach preserves fine-grained word-level semantics highly relevant to the current action while extracting macro-level sentence-level contextual information to form the fusion feature, effectively suppressing the dilution of attention from contextual noise in large language models.

[0018] S104 utilizes a cross-modal encoder to inject the textual semantics contained in the fused features into the global sign language visual features through cross-attention, resulting in an enhanced visual sequence.

[0019] Because visual features and textual features have significant modal differences, especially since visual features in sign language videos represent spatial and motion information, an attention mechanism is used. Global sign language visual features are used as the main body, and fused features extracted from historical text are used as keys and values. Based on attention weights, the semantic information of the historical text is injected into the global sign language visual features. While preserving the original temporal attributes of the sign language actions, contextual information from historical dialogues containing contextual semantics is added to the global sign language visual features, forming an enhanced visual sequence. Furthermore, because the frame sequence of sign language videos is long, cross-attention can be used for injection, along with layer normalization and residual connection topology, to avoid memory overflow issues caused by sequence length explosion.

[0020] S105, the enhanced visual sequence is concatenated with the preset cue word feature matrix to obtain a multimodal input long sequence, which is then input into a trained large language model for autoregressive text generation to obtain the sign language translation result.

[0021] Large language models often rely on corresponding cue word templates to issue instructions guiding their targeted execution of sign language translation tasks, while reducing output bias and illusions. By quantizing the cue word templates into a feature matrix and concatenating it with an enhanced visual sequence in the physical dimension, the cue word feature matrix provides explicit generation instructions and task execution specifications to the large language model, while the enhanced visual sequence provides multimodal semantic input. The cue word feature matrix can be obtained by converting the preset cue word text through the large language model's built-in word segmenter and word embedding model. Then, it is concatenated with the enhanced visual sequence to obtain a long multimodal input sequence containing multimodal sign language features and generation instructions. This sequence is then input into a pre-trained large language model, which, during the autoregressive text generation process, can generate sign language translation results that conform to the semantics of the current sign language action and do not deviate from the historical context, based on the high-purity visual and contextual information contained within.

[0022] Optionally, a multi-task joint loss function is constructed to jointly train the visual encoder, the large language model adapter, the cross-modal enhanced visual sequence generation module, the cross-modal encoder, and the autoregressive decoder of the large language model. The multi-task joint loss function is constructed based on the dual InfoNCE multimodal pre-alignment loss and introduces a dynamically learnable temperature scaling factor. The log probability distribution used to smooth Softmax includes a visual-target text alignment loss to narrow the distance between global sign language visual features and real target translation text features, a visual-historical dialogue context alignment loss to narrow the distance between global sign language visual features and high-purity context features in historical dialogue text, and a large language model autoregressive generation loss to optimize the sign language translation result text generated by the large language model decoder.

[0023] Due to the significant modal differences between visual and textual features—sign language videos being high-dimensional continuous video streams while spoken text is an abstract, structured sequence of discrete symbols—directly inputting them into large language models can easily lead to vanishing gradients and difficulty in convergence. Therefore, a multi-task joint loss function with pre-alignment can be constructed. This pre-contrast alignment loss forces continuous visual features to converge towards the discrete textual latent space, effectively mitigating semantic deviation and illusion problems caused by misalignment of heterogeneous modal features, reducing the cognitive mapping pressure on the large language model decoder, smoothing the network gradient descent, and exponentially increasing convergence efficiency. An example is the visual-target text alignment loss. To narrow down the geometric distance between visual features and the corresponding ground truth target text in the high-dimensional latent space, let the current sample batch size be... Global sign language visual features are The corresponding target translation text features are First calculate and The inner product is obtained The similarity matrix is ​​calculated, with diagonal elements representing positive sample pairs and off-diagonal elements representing negative sample pairs; then, the cross-entropy loss function is used to calculate the visual-to-text direction (...). loss Similarly, calculate the text-to-visual orientation ( loss The final visual-target text alignment loss is defined as a bidirectional symmetric average. This ensures the balance of spatial mapping optimization.

[0024] However, simply aligning visual features with target text features is insufficient to resolve the ambiguity of sign language gestures and the illusion problem in the translated text generated by Large Language Models (LLMs). Furthermore, a visual-historical dialogue context alignment loss method is also employed. To bridge the gap between visual features and historical dialogue context information, the underlying visual encoder is required not only to accurately encode isolated sign language actions but also to include historical context information, endowing the model with context awareness and semantic bias prevention capabilities. This includes: calculating the cosine similarity between word text features and target translated text features, extracting Softmax normalized weights to weight and sum the word text features, and then performing L2 normalization to obtain high-purity single-sentence features. Then, the cosine similarity between the high-purity single-sentence features and target translated text features is calculated, and Softmax normalized weights are extracted to weight and sum the high-purity single-sentence features, and then L2 normalization is performed to obtain high-purity context features. Finally, the visual-historical dialogue context alignment loss is calculated using the bidirectional symmetric InfoNCE contrastive loss function.

[0025] Because historical dialogue texts typically contain multiple sentences, each with varying relevance between textual features and visual features, a large amount of vocabulary irrelevant to the current sign language action semantics exists, creating significant contextual noise. This noise negatively impacts cross-modal semantic alignment, causing visual features to be misled by noisy words in the historical dialogue text, leading to alignment failure and incorrect model learning. Therefore, during training, it's necessary to first use the target translation text (i.e., the correct sign language translation result, which can be considered as labeled data during model training) as a guide to extract effective contextual information from the historical dialogue text according to similarity weights. This removes noise before performing cross-modal alignment. For example, the textual features of each word in the historical dialogue text are first calculated. Features of the target translation text The cosine similarity between the words is calculated, and then a softmax operation is performed to obtain the similarity weight between each word in the historical dialogue text and the target translation text. This weight is then used to evaluate the features of each word text. Perform a weighted summation to utilize the features of the target translated text. The relevance value weight of each word in the historical text is evaluated to filter out noise in the historical dialogue text and obtain high-purity single-sentence features. Then, the characteristics of high-purity single sentences were analyzed. After performing L2 norm normalization, the target translation text features are compared with those of the target translation text. Calculate the cosine similarity and perform a softmax operation again to obtain the similarity weights between the high-purity single-sentence features and the target translated text features. Then, apply the high-purity single-sentence features... We perform weighted summation and then L2 norm normalization to obtain high-purity contextual features that incorporate contextual information from each individual sentence. This includes historical dialogue context information that is highly relevant to the target translation text, such as... Figure 2 The diagram shown is a structural diagram of the context information fusion network. Finally, the cross-entropy loss function is also used to calculate the visual-to-context direction (...). loss and context to visual direction ( loss The final visual-historical dialogue context alignment loss is .

[0026] Traditional large language model (LLM) translation systems rely solely on the autoregressive generation error at the decoder end for long-link backpropagation, which easily leads to extremely slow updates in the underlying visual encoder. This necessitates addressing the issue of autoregressive generation loss in large language models. This involves enforcing pre-trained feature space alignment. For example, taking the pre-trained large language model Qwen3 based on an Encoder-Decoder architecture, during the forward propagation phase of training, the decoder employs a teacher forcing strategy and uses a strict causal attention mask to prevent future information leakage. The LLM decoder predicts the true target translation sequence token by token. At that moment LLM is based on multimodal input and the true prior prefix sequence Calculate the probability distribution of the next word. Finally, the standard negative log-likelihood loss function is used to calculate the text generation loss of the LLM decoder translation result. Ultimately, the multimodal pre-alignment loss of dual InfoNCE is reduced. , Autoregressive generation loss for large language models Through hyperparameters λ 1. λ 2. λ 3. By performing linear combination, a multi-task joint loss function for unified back-updating of gradients across the entire network is constructed. .

[0027] This embodiment extracts global visual features of sign language from sign language videos and corresponding word and sentence text features from historical dialogues. The global visual features are then used as active query vectors for cross-modal retrieval and adaptive weighted fusion. Cross-attention is then used to inject the purified historical text semantics into the visual sequence to form an enhanced visual sequence. Finally, this enhanced visual sequence is concatenated with the feature matrix of the prompt words and input into a large language model for autoregressive generation to obtain the sign language translation result. Simultaneously, a multi-task joint loss function is constructed using perceptual-target text alignment loss, visual-historical dialogue context alignment loss, and large language model autoregressive generation loss. This function is then combined with real target translation text to optimize the large language model for cross-modal and contextual alignment. By accurately denoising the historical context centered on the current sign language action, redundant noise in multi-turn sign language dialogues is effectively eliminated, avoiding the dilution of LLM attention weights. By utilizing a multi-task joint loss function with pre-alignment combined with double alignment, when facing the modal difference gap between high-dimensional continuous video streams and discrete natural language symbols, the visual encoder is forced not only to accurately capture isolated sign language actions, but also to deeply perceive the historical text context. This reduces the cognitive mapping pressure of LLM, effectively suppresses semantic deviation and generative illusions generated by large models in complex contexts, and improves the accuracy of sign language translation.

[0028] Example 2 Figure 3 This is a flowchart of a sign language translation method based on cross-modal retrieval enhancement according to Embodiment 2 of the present invention. This embodiment is based on the above embodiment and optimized. In this embodiment, S103 is specifically optimized as follows: Calculate the word-level cosine similarity matrix between word text features and global sign language visual features, and use the dynamic Top-N mask filling algorithm to remove word text features whose word-level cosine similarity does not meet the preset dynamic truncation threshold. The remaining word text features are weighted and summed, and then average pooling and normalization are performed to obtain word-level retrieval features; Calculate the sentence-level cosine similarity matrix between sentence text features and global sign language visual features and convert it into weights. Then perform weighted summation and normalization to obtain sentence-level retrieval features. The cosine similarity between word-level retrieval features, sentence-level retrieval features, and global sign language visual features is calculated and concatenated. Then, word-level adaptive weights and sentence-level adaptive weights are calculated to perform weighted summation on word-level retrieval features and sentence-level retrieval features respectively, resulting in fused features.

[0029] Accordingly, the sign language translation method based on cross-modal retrieval enhancement provided in this embodiment specifically includes: S201 uses a visual encoder to extract sign language spatial features and sign language motion features from sign language videos, and performs feature downsampling fusion and dimension mapping to form global sign language visual features.

[0030] Specifically, the process involves acquiring the sign language video data to be translated and the corresponding historical dialogue text. The sign language video data is then input into a trained visual encoder. The image encoder within the visual encoder performs multi-scale checkerboard segmentation and reconstruction of the sign language video data to extract spatial features. Finally, the video encoder within the visual encoder extracts sign language motion features based on sliding window time-slice sampling.

[0031] Because people with disabilities need to express meaning through sign language, sign language video frame sequences are usually extremely long. Furthermore, the cross-modal gap between visual and textual representations makes alignment in the feature dimension difficult. When acquiring sign language video data to be translated, it is also necessary to obtain the corresponding historical dialogue text representing its context. To extract visual features representing sign language movement in three-dimensional space, the acquired sign language video data to be translated is input into a trained visual encoder. The visual encoder consists of two branches: an image encoder and a video encoder. First, the high-resolution image (video frames of the sign language video data) is segmented into multiple sub-square tensors, which are then fed in parallel into the pre-trained image encoder CLIP-ViT to extract multi-scale features. Afterward, inverse merging is performed to restore the complete spatial topology. Then, the feature maps extracted at different scales are aligned to a uniform size through region interpolation and stitched together in the channel dimension to obtain a highly rich spatial feature sequence, forming the sign language spatial features. Then, based on the sliding window time-slice sampling strategy, the original sign language video is divided into multiple overlapping video segments with a fixed window size and overlap step. For example, to reduce memory usage, the window size is set to 16 frames and the overlap step is 8. Non-overlapping video segmentation methods include 0-15 frames, 16-31 frames, 32-47 frames, etc. However, if a key sign language action happens to occur between frames 10 and 20, it will be segmented into two different segments, resulting in disjointed and fragmented sign language actions. Therefore, overlapping segmentation is used, cutting the video into 0-15 frames, 8-23 frames, 16-31 frames, and so on, with adjacent segments overlapping by 8 frames. This data is then fed into a pre-trained video encoder, VideoMAE, to extract spatiotemporal features, obtaining a motion feature sequence, thus forming the sign language motion features. .

[0032] The spatial and motor features of sign language are length-padded and concatenated in the time dimension to obtain concatenated features. These features are then downsampled using a large language model adapter and then normalized by a multilayer perceptron to achieve the same dimensionality as the hidden layer of the large language model decoder, resulting in global sign language visual features.

[0033] Since unprocessed video frame sequences are typically extremely long, to reduce computational overhead and memory usage, and to avoid the computational complexity of the LLM self-attention mechanism exponentially increasing with the length of the video frame sequence, downsampling is required to exponentially compress the video frame sequence. This is achieved by incorporating sign language spatial features along the time dimension. and sign language movement characteristics After length padding, the pieces are then spliced ​​to obtain the splicing feature. Then it is fed into a 1D temporal convolutional network configured with residual connections, the network structure of which is as follows: Figure 4 As shown, the large language model adapter downsamples the spliced ​​features. This network is composed of multiple stacked Conv1D, BatchNorm1D, ReLU, and MaxPool1D blocks, i.e., 1D convolution, batch normalization, activation function (ReLU), and 1D max pooling are sequentially connected and then stacked. First, a 1D convolution kernel (Conv1D) with a kernel size of 5 slides along the time axis, examining 5 frames simultaneously and fusing the information from the 5 frames into 1 frame. Then, a 1D max pooling (MaxPool1D) with a stride of 2 is used to halve the sequence length, retaining only the most representative features from two adjacent frames. After each convolution and pooling operation, the effective length of the time series is reduced, thus condensing a long video frame sequence of hundreds of frames into a high-density semantic sequence of tens of frames. Next, the number of visual feature channels needs to be matched with the latent space of the LLM. This is done by constructing a multilayer perceptron (MLP) with the activation function GELU to strictly map the visual features to the hidden layer dimension of the large language model. For example, the large language model LLM can be Qwen3, with its hidden layer dimension... Then, invalid frames in the sequence padding are cleared by element-wise multiplication of the Boolean mask matrix to accurately exclude them. The features of the remaining valid frames are averaged and then normalized using L2 norm to obtain global sign language visual features with global semantic information. .

[0034] S202 utilizes a text encoder to extract word text features and sentence text features from the historical dialogue text corresponding to the sign language video.

[0035] Specifically, the historical dialogue text is input into the trained text encoder, and the word segmenter in the text encoder is used to perform word conversion and encoding to obtain word text features.

[0036] During continuous multi-turn dialogues between hearing and hearing-impaired individuals, both spoken language and sign language are converted into natural language text and recorded as historical dialogue text data. The text encoder utilizes a pre-trained large language model word embedding layer combined with masked pooling to extract sentence text features for each sentence. and word text features of each word in all sentences The historical dialogue text data contains multiple sentences of historical dialogue text. First, it is input into the pre-trained large language model's built-in tokenizer (such as Qwen3 tokenizer) to convert the strings in the historical dialogue text into LLM-readable token IDs. Then, it is fed into the large model's built-in embedding layer for encoding to obtain word text features. This is used to characterize the fine-grained semantic features of each word in the historical dialogue text.

[0037] The word text features are processed by masked average pooling and normalization to obtain the sentence text features.

[0038] To further obtain a macroscopic semantic representation of the sentence, the mask matrix is ​​first expanded and then combined with the word feature vectors. The multiplication process involves a mask matrix automatically generated by the pre-trained large-scale model's tokenizer (such as Qwen3 tokenizer) during batch tokenization and padding of historical dialogue text. All padded lexical features are forcibly set to zero to eliminate noise interference. Then, the remaining word feature vectors are summed along their length dimension and divided by the number of effective words in the sentence to calculate the average feature vector representing the global semantics of the sentence. Finally, L2 norm normalization is performed to obtain the sentence text features that condense the macroscopic semantics. .

[0039] S203, calculate the word-level cosine similarity matrix between word text features and global sign language visual features, and use the dynamic Top-N mask filling algorithm to remove word text features whose word-level cosine similarity does not meet the preset dynamic truncation threshold.

[0040] Historical dialogue texts often contain a large amount of vocabulary irrelevant to the semantics of the current sign language action. In such cases, a cross-modal retrieval enhancement module can be used to perform cross-modal enhanced retrieval on the historical dialogue text. This removes irrelevant words, eliminates contextual noise interference, prevents LLM from being misled by redundant information, and focuses LLM's attention on background words strongly related to the current sign language action, reducing the probability of semantic deviation and improving translation accuracy in complex contexts. For example... Figure 5 The diagram shown illustrates the network structure of the cross-modal retrieval enhancement module. The inputs to the cross-modal retrieval enhancement module include: global sign language visual features. As the active query vector (Query) in the entire retrieval process, and sentence text features and word text features Searching for visual features of sign language using word-level search paths and matching them with current global sign language visual features. Highly relevant, fine-grained word text features, for example, in a word-level retrieval path, first calculate the text features of each word. With current global sign language visual features Word-level cosine similarity matrix To eliminate contextual noise in historical dialogue text, a fixed Top-N truncation method cannot adapt to changes in the effective sentence length. For short sentences, a fixed large truncation N value introduces too many noisy words, while for long sentences, a fixed small truncation N value results in the loss of key information. Therefore, a dynamic Top-N masking and filling algorithm based on the effective sentence length is used to remove irrelevant words, retaining only the top N words with the highest cosine similarity. This allows the number of retained words to adaptively fluctuate with the actual information content of the sentence, based on the effective sentence length. Dynamically set cutoff threshold .

[0041] An optional implementation of this embodiment is to introduce [a method / mechanism] during training to increase robustness. The random fluctuation range, and the preset dynamic cutoff thresholds include: Cut off the lower boundary The calculation formula is: ; Cut off the upper boundary The calculation formula is: ; According to the lower bound and truncation of the upper boundary A uniformly distributed random offset is introduced to calculate the dynamic truncation threshold. The formula is as follows: ; in, The effective length of the sentence. The random offsets are uniformly distributed. A dynamic floating mechanism is used to avoid information loss or noise residue caused by hard truncation, enabling the model to maintain high-purity retrieval performance when faced with historical dialogue texts of different lengths and information densities.

[0042] S204. The remaining word text features are weighted and summed, and then average pooling and normalization are performed to obtain word-level retrieval features.

[0043] Simultaneously, a mask matrix is ​​constructed to forcibly set the similarity of mask elements ranked after the dynamic truncation threshold N in the word-level cosine similarity matrix to negative infinity. Then, the probability distribution is calculated using the Softmax function. At this point, the weights of words masked to negative infinity become 0, thus obtaining the high-purity weights of the remaining top N words with the highest similarity. Based on this weight Weighted summation yields the refined single-sentence features. Since the input historical dialogue text contains multiple sentences, mean pooling and L2 norm normalization are performed on the refined single-sentence features to obtain word-level retrieval features. By aggregating high-purity core keywords, it represents the semantic information of keywords that most accurately represent the core actions of sign language videos in a whole dialogue containing multiple sentences. It is fine-grained single semantic information that can complement sentence-level macro semantic information.

[0044] S205: Calculate the sentence-level cosine similarity matrix between the sentence text features and the global sign language visual features, convert it into weights, and then perform weighted summation and normalization to obtain the sentence-level retrieval features.

[0045] Within the sentence-level retrieval path, the search is performed using the sentence-level retrieval path to find visual features matching the current global sign language. The relevant sentence-level macro-contextual information is first used to calculate the sentence text features of each sentence. With current global sign language visual features Sentence-level cosine similarity matrix Then, the weights of each sentence are generated using the Softmax function. By using this weight to perform a weighted summation on all sentences and then performing L2 norm normalization, sentence-level retrieval features with macro-contextual information are obtained. .

[0046] S206. Calculate the cosine similarity between word-level retrieval features, sentence-level retrieval features, and global sign language visual features, and then concatenate them. Next, calculate word-level adaptive weights and sentence-level adaptive weights to perform weighted summation on word-level retrieval features and sentence-level retrieval features respectively, and obtain the fused features.

[0047] In the dual-path dynamic adaptive fusion path, before inputting global sign language visual features into the LLM decoder, they are first used as active query conditions to retrieve effective information across modalities in historical dialogue text. Since the auxiliary explanatory value of different granularities for the current sign language action is dynamically changing, the dual-path dynamic adaptive fusion path adaptively fuses word-level fine-grained features with sentence-level macro-contextual information, avoiding the reduction in information entropy or noise dilution of the fused features caused by fixed-weight fusion. For example, word-level retrieval features are calculated separately. Sentence-level retrieval features With global sign language visual features The cosine similarity is calculated, and then the two obtained similarity scores are concatenated to form a similarity matrix. The matrix is ​​then adaptively calculated using the Softmax function. and weight and For example, Softmax([0.85,0.65])=[0.55,0.45], will be calculated according to their respective weights. and Weighted summation yields the fusion characteristics. This includes both the current global sign language visual features The highly relevant fine-grained word-level text information, along with sentence-level text information from the macro-context, ensures the high purity and high information entropy of the fused features, effectively suppressing the interference of noise dilution on the attention of large language models.

[0048] S207 utilizes a cross-modal encoder to inject the textual semantics contained in the fused features into the global sign language visual features through cross-attention, resulting in an enhanced visual sequence.

[0049] Specifically, the global sign language visual features are used as the attention query vector matrix, and the fused features are expanded in sequence dimension to serve as the key matrix and value matrix.

[0050] After obtaining fused features through cross-modal retrieval, the retrieved effective information is fused and enhanced to provide high-purity, high-value textual cues for LLM, thereby improving the accuracy of LLM-generated translated text. This involves integrating long-sequence global sign language visual features. The query vector matrix (Query, Q), used as an attention mechanism, is applied to the fused features. After expanding the sequence length dimension, the result is a key (K) and value (V) matrix. The formula for cross-attention is as follows:

[0051] When entering the cross-attention block, the fused features after sequence dimension expansion no longer need to be manually segmented, but are instead used as the whole input. The attention block contains two independent linear projection layers. The fused features after sequence expansion are multiplied by two different weights to generate a K matrix for calculating the relevance to video features and a V matrix for providing actual text enhancement cues. This effectively avoids the length explosion caused by directly concatenating sequence dimensions or the inability of shallow attention to deeply fuse textual prior knowledge and capture deep spatiotemporal semantic interactions.

[0052] Calculate the dot product similarity between the query vector matrix and the normalized key matrix, perform a scaling activation operation, and then multiply it with the normalized value matrix to obtain the text semantic interaction features.

[0053] To ensure the numerical stability of the inner product during attention calculation and to avoid gradient anomalies caused by excessive variance differences between different modalities, a cross-modal encoder is first used to process the fused features of the input. Perform first-level normalization :

[0054] Then calculate the global sign language visual features Each frame's query vector and the normalized key matrix Calculate the dot product similarity and divide by the scaling factor. Then, a Softmax operation is performed to effectively prevent the vanishing gradient and training instability caused by excessively large inner product values. This is then combined with the normalized value matrix. Multiplication yields textual semantic interaction features The formula is as follows:

[0055] in, for K The dimension value is used to densely and uniformly broadcast pure text semantics into each frame of the visual sequence according to the corresponding adaptive weights, avoiding sequence length explosion.

[0056] The text semantic interaction features and global sign language visual features are additively fused based on residual connection topology to obtain intermediate features.

[0057] To prevent excessive textual information from masking the original visual action features, the scaling dot product attention module employs a residual connection topology, which needs to incorporate the semantic interaction features of the injected textual information. With as a query The global sign language visual features are additively fused to obtain intermediate features. This ensures that visual features dominate and avoids excessive textual semantics overshadowing visual details.

[0058] The intermediate features are normalized and mapped to hidden dimensions using a feedforward neural network, followed by nonlinear activation and hidden dimension projection. Residual fusion is then performed with the intermediate features to obtain the enhanced visual sequence, as shown in the following formula: ; in, As an intermediate feature, Representation layer normalization, This represents a feedforward neural network. This indicates an enhanced visual sequence.

[0059] Since scaling dot product attention is primarily responsible for feature aggregation and lacks sufficient non-linear expressive power, it will incorporate intermediate features. The input is a two-layer feedforward neural network (MLP), firstly... Perform the second layer of normalization Then, a linear fully connected layer is used to multiply the feature matrix with a trainable weight matrix of shape [original hidden dimension, higher hidden dimension], projecting it from the original low-dimensional latent space to a higher hidden dimension. This provides a richer feature representation space for subsequent activation functions, thereby enhancing the network's non-linear expressive power. Next, a non-linear mapping is introduced through the GELU activation function. Finally, another linear fully connected layer is used to multiply the feature matrix with a trainable weight matrix of shape [higher hidden dimension, original hidden dimension], projecting it back from the high-dimensional latent space to the original latent space dimension. This ensures that the shape of the output features is completely consistent with the shape of the input features before entering the MLP, facilitating the establishment of residual connections. Then, it is again combined with... Residual fusion involves adding two matrices of the same shape element-wise to obtain an enhanced visual sequence that incorporates contextual information from deep historical dialogue. For example... Figure 6 The diagram shows the network structure of a cross-modal encoder. The network topology, employing dual LayerNorms and dual residual connections, ensures that gradients during backpropagation are stably propagated back to the underlying visual network, regardless of the depth of the subsequent LLM network, thus preventing gradient vanishing.

[0060] S208 concatenates the enhanced visual sequence with the preset cue word feature matrix to obtain a multimodal input long sequence, and inputs it into a trained large language model for autoregressive text generation to obtain the sign language translation result.

[0061] Specifically, the word segmenter and word embedding model of the large language model are used to transform the preset prompt words to obtain a preset prompt word feature matrix. The enhanced visual sequence is concatenated with the preset prompt word feature matrix to obtain a multimodal input long sequence. The multimodal input long sequence is then input into the autoregressive decoder of the large language model, and the sign language translation text is generated autoregressively to obtain the sign language translation result.

[0062] Large Language Models (LLMs), as the foundation for text generation with knowledge from hundreds of billions of pre-trained corpora, require explicit instructions to guide their powerful logical reasoning and language generation capabilities in targeted application to sign language translation tasks. Otherwise, they are prone to producing rambling, illusory text. To provide explicit generation instructions to the LLM, the pre-defined instruction text is converted into a word sequence using the LLM's built-in word segmenter. This sequence is then mapped into a high-dimensional vector matrix through a word embedding layer, forming a feature matrix of cue words. Subsequently, enhanced visual sequences will be provided with rich multimodal semantic support in the time series dimension. With prompt word feature matrix Perform physical dimension splicing This forms a long, multimodal input sequence for use in a large language model. This sequence is then fed into the autoregressive decoder of the large language model, which automatically performs an autoregressive text generation task to produce the final sign language translation result.

[0063] This embodiment uses a one-dimensional temporal convolutional network with alternating stacks of Conv1D and MaxPool1D to perform pooling compression and downsampling on long sequences of sign language videos. It also utilizes a dynamic Top-N mask filling algorithm that depends on the effective length of the current sentence, introduces a uniformly distributed random bias to generate a dynamic truncation threshold, strictly removes low-relevance words, and constructs a network topology based on scaled dot product cross-attention. Furthermore, it combines double-layer normalization (LayerNorm), multilayer perceptron (MLP), and double residual connections during feature injection. It effectively filters out high-frequency static redundant frames in long sequence sign language video streams, exponentially compressing the effective temporal length and avoiding the problems of memory overflow and exponential explosion of computational overhead when LLM processes long sequences; the dynamic Top-N truncation algorithm enables LLM to have strong anti-interference ability and robustness when facing multi-turn dialogue sentences of varying lengths and complex structures; the dual residual fusion topology network can ensure that the prior knowledge of the text is non-linearly and uniformly injected into each frame, ensuring that the gradient during backpropagation can be smoothly and stably propagated back to the underlying visual network at different depths of different LLM networks, effectively overcoming the problems of convergence and gradient vanishing in traditional multimodal end-to-end networks.

[0064] Example 3 Figure 7 This is a schematic diagram of the structure of a sign language translation device based on cross-modal retrieval enhancement according to Embodiment 3 of the present invention. In this embodiment, the sign language translation device based on cross-modal retrieval enhancement includes: The visual encoding module 810 is used to extract sign language spatial features and sign language motion features from sign language videos using a visual encoder, and to perform feature downsampling fusion and dimension mapping to form global sign language visual features. The text encoding module 820 is used to extract word text features and sentence text features from the historical dialogue text corresponding to the sign language video using a text encoder. The cross-modal retrieval fusion module 830 is used to utilize global sign language visual features as query vectors, and perform cross-modal retrieval with word text features and sentence text features respectively to obtain word-level retrieval features and sentence-level retrieval features, and then perform adaptive weighted fusion to form fused features; The cross-modal coding enhancement module 840 is used to inject the textual semantics contained in the fused features into the global sign language visual features through cross attention using a cross-modal encoder to obtain an enhanced visual sequence. The autoregressive text generation module 850 is used to concatenate the enhanced visual sequence with the preset cue word feature matrix to obtain a multimodal input long sequence, and input it into a trained large language model for autoregressive text generation to obtain the sign language translation result.

[0065] This embodiment extracts sign language spatial and motion features from sign language videos using a visual encoding module to form global sign language visual features. It extracts word and sentence text features from historical dialogue text using a text encoding module. A cross-modal retrieval and fusion module performs cross-modal retrieval and adaptive weighted fusion of the global sign language visual features, word text features, and sentence text features to form fused features. A cross-modal encoding enhancement module injects the textual semantics contained in the fused features into the global sign language visual features to obtain an enhanced visual sequence. An autoregressive text generation module constructs a multimodal input long sequence input large language model to obtain the sign language translation result. By accurately denoising the historical context centered on the current sign language action, redundant noise in multi-turn sign language dialogues is effectively eliminated, avoiding the dilution of LLM attention weights. By utilizing a multi-task joint loss function with pre-alignment combined with double alignment, when facing the modal difference gap between high-dimensional continuous video streams and discrete natural language symbols, the visual encoder is forced not only to accurately capture isolated sign language actions, but also to deeply perceive the historical text context. This reduces the cognitive mapping pressure of LLM, effectively suppresses semantic deviation and generative illusions generated by large models in complex contexts, and improves the accuracy of sign language translation.

[0066] The sign language translation device based on cross-modal retrieval enhancement provided in this embodiment of the invention can execute the sign language translation method based on cross-modal retrieval enhancement provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0067] Example 4 Figure 8 This is a structural diagram of an electronic device according to Embodiment 4 of the present invention. Figure 8 A block diagram is shown of an exemplary electronic device 12 suitable for implementing embodiments of the present invention. Figure 8The electronic device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0068] like Figure 8 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0069] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0070] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0071] System memory 28 may include computer system readable media in the form of volatile memory, such as RAM 30 and / or cache 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 Not shown; usually referred to as a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0072] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0073] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the electronic device 12 / server / computer, and / or with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through I / O interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. Figure 8 As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although... Figure 8 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0074] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the sign language translation method based on cross-modal retrieval enhancement provided in the embodiments of the present invention.

[0075] Example 5 Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the sign language translation method based on cross-modal retrieval enhancement as provided in the above embodiments.

[0076] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0077] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0078] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0079] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0080] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A sign language translation method based on cross-modal retrieval enhancement, characterized in that, include: S101 uses a visual encoder to extract sign language spatial features and sign language motion features from sign language videos, and performs feature downsampling fusion and dimension mapping to form global sign language visual features; S102, using a text encoder to extract word text features and sentence text features from the historical dialogue text corresponding to the sign language video; S103 utilizes global sign language visual features as query vectors, performs cross-modal retrieval with word text features and sentence text features respectively, obtains word-level retrieval features and sentence-level retrieval features, and performs adaptive weighted fusion to form fused features; S104 utilizes a cross-modal encoder to inject the textual semantics contained in the fused features into the global sign language visual features through cross-attention, resulting in an enhanced visual sequence; S105, the enhanced visual sequence is concatenated with the preset cue word feature matrix to obtain a multimodal input long sequence, which is then input into a trained large language model for autoregressive text generation to obtain the sign language translation result; The method further includes: A multi-task joint loss function is constructed to jointly train the visual encoder, the large language model adapter, the cross-modal enhanced visual sequence generation module, the cross-modal encoder, and the autoregressive decoder of the large language model. The multi-task joint loss function includes a visual-target text alignment loss to narrow the distance between global sign language visual features and real target translation text features, a visual-historical dialogue context alignment loss to narrow the distance between global sign language visual features and high-purity context features in historical dialogue text, and a large language model autoregressive generation loss to optimize the sign language translation result text generated by the large language model decoder. The visual-historical dialogue context alignment loss includes: calculating the cosine similarity between word text features and target translated text features, extracting Softmax normalized weights to sum the word text features, and then performing L2 normalization to obtain high-purity single-sentence features. Then, calculating the cosine similarity between the high-purity single-sentence features and target translated text features, extracting Softmax normalized weights to sum the high-purity single-sentence features, and then performing L2 normalization to obtain high-purity context features. Finally, the visual-historical dialogue context alignment loss is calculated using the bidirectional symmetric InfoNCE contrastive loss function.

2. The method according to claim 1, characterized in that, S101 includes: Obtain the sign language video data to be translated and the corresponding historical dialogue text; The sign language video data to be translated is input into the trained visual encoder. The image encoder in the visual encoder is used to perform multi-scale checkerboard segmentation and reconstruction of the sign language video data to extract sign language spatial features. The video encoder in the visual encoder is used to extract sign language motion features based on sliding window time slice sampling. The spatial and motor features of sign language are length-padded and concatenated in the time dimension to obtain concatenated features. These features are then downsampled using a large language model adapter and then normalized by a multilayer perceptron to achieve the same dimensionality as the hidden layer of the large language model decoder, resulting in global sign language visual features.

3. The method according to claim 1, characterized in that, S102 includes: Input the historical dialogue text into the trained text encoder, and use the word segmenter in the text encoder to perform word conversion and encoding to obtain word text features; The word text features are processed by masked average pooling and normalization to obtain the sentence text features.

4. The method according to claim 1, characterized in that, S103 includes: Calculate the word-level cosine similarity matrix between word text features and global sign language visual features, and use the dynamic Top-N mask filling algorithm to remove word text features whose word-level cosine similarity does not meet the preset dynamic truncation threshold. The remaining word text features are weighted and summed, and then average pooling and normalization are performed to obtain word-level retrieval features; Calculate the sentence-level cosine similarity matrix between sentence text features and global sign language visual features and convert it into weights. Then perform weighted summation and normalization to obtain sentence-level retrieval features. The cosine similarity between word-level retrieval features, sentence-level retrieval features and global sign language visual features is calculated and concatenated. Then, word-level adaptive weights and sentence-level adaptive weights are calculated to perform weighted summation on word-level retrieval features and sentence-level retrieval features respectively, and the fused features are obtained. The preset dynamic truncation threshold includes: Cut off the lower boundary The calculation formula is: ; Cut off the upper boundary The calculation formula is: ; According to the lower bound and truncation of the upper boundary A uniformly distributed random offset is added to calculate the dynamic truncation threshold. The formula is as follows: ; in, The effective length of the sentence. The offset is a uniformly distributed random offset.

5. The method according to claim 1, characterized in that, S104 includes: The global sign language visual features are used as the attention query vector matrix, and the fused features are expanded in sequence dimension to serve as the key matrix and value matrix. Calculate the dot product similarity between the query vector matrix and the normalized key matrix, perform a scaling activation operation, and then multiply it with the normalized value matrix to obtain the text semantic interaction features. The text semantic interaction features and global sign language visual features are additively fused based on residual connection topology to obtain intermediate features; The intermediate features are normalized and mapped to hidden dimensions using a feedforward neural network, followed by nonlinear activation and hidden dimension projection. Residual fusion is then performed with the intermediate features to obtain the enhanced visual sequence, as shown in the following formula: ； in, As an intermediate feature, Representation layer normalization, This represents a feedforward neural network. This indicates an enhanced visual sequence.

6. The method according to claim 1, characterized in that, S105 includes: The word segmenter and word embedding model of the large language model are used to transform the preset prompt words to obtain the preset prompt word feature matrix; The enhanced visual sequence is concatenated with a preset cue word feature matrix to obtain a multimodal input long sequence; The long sequence of multimodal input is fed into the autoregressive decoder of a large language model, and the sign language translation text is generated autoregressively to obtain the sign language translation result.

7. A sign language translation device based on cross-modal retrieval enhancement, used to implement the sign language translation method based on cross-modal retrieval enhancement as described in any one of claims 1-6, characterized in that, include: The visual encoding module is used to extract sign language spatial features and sign language motion features from sign language videos using a visual encoder, and to perform feature downsampling fusion and dimension mapping to form global sign language visual features; The text encoding module is used to extract word text features and sentence text features from the historical dialogue text corresponding to the sign language video using a text encoder. The cross-modal retrieval fusion module is used to utilize global sign language visual features as query vectors, and perform cross-modal retrieval with word text features and sentence text features respectively to obtain word-level retrieval features and sentence-level retrieval features, and then perform adaptive weighted fusion to form fused features; The cross-modal coding enhancement module is used to inject the textual semantics contained in the fused features into the global sign language visual features through cross-attention, thereby obtaining an enhanced visual sequence; The autoregressive text generation module is used to concatenate the enhanced visual sequence with the preset cue word feature matrix to obtain a multimodal input long sequence, which is then input into a trained large language model for autoregressive text generation to obtain sign language translation results.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the sign language translation method based on cross-modal retrieval enhancement as described in any one of claims 1-6.

9. A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the sign language translation method based on cross-modal retrieval enhancement as described in any one of claims 1-6.

Images