Interactive multi-mode prompt fusion method for continuous learning of sign language translation

Abstract

The invention discloses an interactive multi-modal prompt fusion method for continuous learning of sign language translation, and aims to solve the problems of disastrous forgetting and insufficient multi-modal interaction consideration of an existing sign language translation system in continuous learning. According to the method, a fusion mechanism of task perception prompt and data perception prompt is constructed to realize balance of model stability and plasticity, cross-modal and cross-task interaction loss modules are constructed in training loss, information between modals and between tasks is utilized to prompt model translation accuracy, and meanwhile, low-rank decomposition is introduced to further reduce the number of parameters, so that the model translation accuracy is improved. And relatively light-weight fine tuning and learning are realized. The method improves the stability of continuous learning under the condition of guaranteeing the sign language translation accuracy, is suitable for a barrier-free communication scene of a hearing-impaired group, and is compatible with a mainstream multi-mode sign language translation framework.

Description

Technical Field

[0001] This invention relates to the field of sign language translation technology, and more particularly to an interactive multimodal prompt fusion method for continuous learning in sign language translation. Background Technology

[0002] Sign language, as the primary means of communication for the hearing impaired, possesses unique grammar, vocabulary, and expressions. The goal of sign language translation is to accurately convert sign language into natural language, a process that requires processing multimodal information, including visual and linguistic elements. Existing sign language translation research largely focuses on improving translation accuracy for single tasks and cross-language transferability, rarely considering how to handle sign language translation tasks in a continuous learning environment. In real-world scenarios, sign language is not static; expressions vary between different regions and among different sign language speakers. Traditional sign language translation systems cannot continuously adapt to new performers and are prone to forgetting previously learned knowledge.

[0003] Existing continuous learning methods primarily address the catastrophic forgetting problem through regularization-based, experience replay-based, optimization-based, representation-based, and architecture-based approaches. Regularization-based methods may over-constrain the model, affecting its ability to learn new tasks; replay-based methods face memory limitations and data privacy issues; optimization-based methods have high computational complexity; and architecture-based methods increase model complexity and the difficulty of task identity prediction. Representation-based methods, leveraging the advantages of self-supervised learning and large-scale pre-training, can create and utilize powerful representations to improve continuous learning performance. Among these, representation-based methods, combined with cue learning, have gained increasing attention in recent years due to their efficiency and flexibility in fine-tuning pre-trained models. By introducing learnable cues into the pre-trained model, they can quickly adapt to different tasks without large-scale fine-tuning of the entire model. However, most of these methods only consider unimodal applications, lacking research on intermodal and inter-task interactions in multimodal scenarios. Furthermore, these cues are designed with inter-task generalization in mind, often using the same fixed cues for datasets within the same task, without considering the differences between data within the same task; therefore, there is room for improvement in intra-task generalization. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an interactive multimodal prompt fusion method for continuous learning in sign language translation.

[0005] The objective of this invention is achieved through the following technical solution: an interactive multimodal prompt fusion method for continuous learning in sign language translation, the method comprising:

[0006] S1. Process the arriving task data and extract video features and key point features from the sign language video;

[0007] S2. Freeze the parameters of the pre-trained sign language translation model, and create and initialize the data-aware cue generator and task-aware cue for the current task:

[0008] S3. Based on the prompt generation and fusion mechanism, the data-aware prompt generator generates corresponding modal data-aware prompts based on the input video features and key point features. The data-aware prompts are concatenated with the task-aware prompts. The concatenated prompts and multimodal sign language data are input into the frozen pre-trained sign language translation model, and the translation prediction results are output.

[0009] S4. Calculate the total training loss function for recognition and translation, and optimize the parameters of the data-aware prompt generator and the task-aware prompt using the task-adaptive learning rate;

[0010] S5. After the current task training is completed, the video features of each task are reduced in dimensionality and then clustered to generate cluster centers. The cluster centers of the dataset perception prompt generator, task perception prompt and video features corresponding to the task are saved.

[0011] S6. Continue to accept data from other tasks and repeat the above steps until the model has learned on all tasks.

[0012] S7. After all tasks have been learned, in the inference phase, for the current input to be inferred, the time frame average of the input video features is calculated. The KNN algorithm is used to search for the most similar sample in the cluster centers of all historical tasks and match the corresponding task-aware prompts and data-aware prompt generators. Then, the generator generates data-aware prompts based on the input data, which together with the task-aware prompts form the final prompts. These are then merged with the inputs into the pre-trained model to obtain the final translation prediction result.

[0013] Furthermore, the data-aware cue generator includes a cross-attention mechanism and an MLP module, with video feature modalities and keypoint feature modalities sharing a single generator;

[0014] Data-aware prompt generator f DP After receiving the input video features or keypoint features, a learnable linear projection matrix W is used. k and W v The corresponding attention keys and values ​​are generated and multi-head attention computation is performed with the shared query matrix. The query matrix is ​​trained in the first continuous learning task and shared across subsequent tasks and different modalities within the same task. The attention output is then input into the MLP for feature refinement to generate data-aware cues for the corresponding modality. The specific formula is as follows:

[0015]

[0016] in, and These are data-aware cues generated for video and keypoint inputs, V and B is the batch size, L is the number of video frames, and Attn represents the multi-head attention mechanism. It is a query matrix, W k and W v It is a learnable linear projection matrix used to transform input features into the keys and values ​​required by the attention mechanism.

[0017] Furthermore, during task-aware cues initialization, the matrix is ​​decomposed into a product of a shared matrix and a mode-specific matrix using a low-rank matrix to reduce parameters and improve depth alignment across membrane states. The task-aware cues are expressed by the following formula:

[0018]

[0019] Where r is the decomposition rank and r < min(l / 2, d), d is the embedding dimension, A is a low-rank matrix shared across modalities, and B... m For modality-specific matrices, B is given for both video modality and keypoint modality. V and B K "·" represents the matrix dot product operation. Cross-modal knowledge interaction is achieved through a shared matrix A, and modality-specific matrix B is used for cross-modal knowledge interaction. m Adapt to different modal feature distributions.

[0020] Furthermore, the specific steps of concatenating data-aware prompts and task-aware prompts include:

[0021] The final cue P for the corresponding modality is constructed by combining task-aware cues and data-aware cues. m

[0022]

[0023] Where m∈{Video(V),Keypoint(K)}, and [;] denotes a join operation. It is a task-aware prompt, where l and d represent the length and dimension of the prompt, respectively.

[0024] Furthermore, the total loss function for model training includes:

[0025]

[0026] in It is the CTC task loss for sign language recognition and translation. It is a cross-modal alignment loss. It is the cross-task alignment loss; hyperparameter λ base , λ CMA, λ CTA Used to balance different loss contributions.

[0027] Furthermore, the calculation process for the transmembrane loss consists of a weighted sum of Frobenius norm loss and cosine similarity loss, minimizing B. V and B K The distribution differences ensure that multimodal information is aligned in the shared space:

[0028]

[0029] Among them ||·|| F The Frobenius norm ensures that low-rank matrices in different modalities remain structurally similar, while the cosine similarity term encourages directional alignment, thus promoting shared representation learning. m [i] represents the low-rank matrix B m The relative contribution of the two terms in the i-th row vector is controlled by λ1 and λ2.

[0030] Furthermore, the cross-task loss is calculated using the following formula:

[0031]

[0032] Among them, S m (T i T j ) is the corresponding modality of task T i and T j The cosine similarity between clustering features, where t is the threshold for distinguishing positive and negative samples. If S m (T i ,T j If )>t, then task T i and T j Tips between and Treat them as positive sample pairs and set the label z. ij =1, similarity should be maintained; if S m (T i T j If t < t, then task T i and T j The hints between them are considered negative sample pairs, z ij =0, should be separated; It is the cross-task loss between tasks i and j. τ is the cross-modal loss for task i, Npos and Nneg are the number of positive and negative samples, and τ is the temperature scaling parameter.

[0033] Furthermore, the specific design of the task-adaptive learning rate is as follows:

[0034] ηt =η0+γ(1-S V (T i ,T0)),

[0035] Where η0 is the base learning rate, γ is the condition factor, and S V (T i (T0) is the current task T i Similarity to the baseline task T0 in terms of video modality.

[0036] On the other hand, this invention also provides an interactive multimodal prompt fusion device for continuous learning of sign language translation, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the interactive multimodal prompt fusion method for continuous learning of sign language translation.

[0037] On the other hand, this specification also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the interactive multimodal prompting fusion method for continuous learning of sign language translation.

[0038] The beneficial effects of this invention are as follows: It provides a lightweight, pluggable, continuous learning framework for sign language translation based on cue learning. It proposes a combination of task-aware loss cues and data-aware loss cue generators, balancing intra-task and inter-task generalization. This enhances the quality of continuous sign language translation while mitigating catastrophic forgetting in continuous learning, and improves model stability while maintaining a degree of plasticity. The introduction of low-rank decomposition further reduces the number of architectural parameters, helping to align cues at a deeper level across modalities and tasks. Simultaneously, the designed cross-modal and cross-task interaction loss fully utilizes information from different modalities and tasks, helping the model learn more common knowledge and improve translation performance. Compared to existing traditional sign language translation and continuous learning methods, the method of this invention exhibits better translation performance and a lower forgetting rate in continuous learning scenarios for sign language translation. Attached Figure Description

[0039] Figure 1 This is a flowchart of the training process for the interactive multimodal prompting fusion method for continuous learning of sign language translation according to the present invention.

[0040] Figure 2 This is a flowchart illustrating the reasoning process of the interactive multimodal prompting fusion method for continuous learning of sign language translation according to the present invention.

[0041] Figure 3 This is a schematic diagram of the interactive multimodal prompting fusion device for continuous learning of sign language translation according to the present invention. Detailed Implementation

[0042] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0043] The present invention provides an interactive multimodal prompt fusion method for continuous learning in sign language translation, comprising the following steps:

[0044] Step 1: Process the arriving task data and use a pre-trained data encoder to extract video features and key point features from the sign language video.

[0045] Step 2: Freeze the parameters of the pre-trained sign language translation model, create and initialize the data-aware cue generator and task-aware cue for the current task, and set them as trainable parameters. During task-aware cue initialization, the low-rank matrix is ​​decomposed into the product of a shared matrix and a modality-specific matrix to reduce parameter and cross-membrane depth alignment. The task-aware cue can be expressed by the following formula:

[0046]

[0047] Where r is the decomposition rank and r < min(l / 2, d), d is the embedding dimension, A is a low-rank matrix shared across modalities, and B... m For modality-specific matrices, B is defined for both video modality (V) and keypoint modality (K). V and B K "·" represents the matrix dot product operation. Cross-modal knowledge interaction is achieved through a shared matrix A, and modality-specific matrix B is used for cross-modal knowledge interaction. m Adapt to different modal feature distributions.

[0048] Step 3: Based on the prompt generation and fusion mechanism, the data-aware prompt generator generates corresponding modal data-aware prompts based on the input video features and key point features. The data-aware prompts and task-aware prompts are concatenated to form the prompts for the current data. The fused prompts and multimodal sign language data are input into the frozen pre-trained sign language translation model, and the translation prediction results are output.

[0049] This step is one of the core aspects of this invention, and it consists of the following sub-steps:

[0050] (3.1) Prompt generation

[0051] A data-aware cue generator generates modality-specific data-aware cues based on specific inputs. (Data-aware cue generator f) DP Including a cross-attention mechanism and an MLP module, different modalities share a single generator; the data-aware cue generator receives input video features (V) or keypoint features (K), and then uses a learnable linear projection matrix W. k and W vThe system generates corresponding attention keys and values, and performs multi-head attention computation with a shared query matrix. The query matrix, trained in the first continuous learning task, is shared across subsequent tasks and different modalities within the same task. The attention output is then input into an MLP for feature refinement, generating data-aware cues for the corresponding modality. The specific formula is as follows:

[0052]

[0053] in, and These are data-aware cues generated for video and keypoint inputs, V and B is the batch size, L is the number of video frames, and Attn represents the multi-head attention mechanism. It is a query matrix, W k and W v It is a learnable linear projection matrix used to transform input features into the keys and values ​​required by the attention mechanism.

[0054] (3.2) Hints on fusion

[0055] The final cue P for the corresponding modality is constructed by combining task-aware cues and data-aware cues. m

[0056]

[0057] Where m∈{Video(V), Keypoint(K)}, and [;] denotes a join operation. These are task-aware prompts, where l and d represent the length and dimension of the prompt, respectively.

[0058] (3.3) The fused prompts and the processed input data are concatenated and input into the model for prediction to obtain the predicted translated text.

[0059] Step 4: Calculate the CTC loss, cross-modal loss, and cross-task loss for recognition and translation to obtain the total loss function for model training. Optimize the parameters of the data-aware prompt generator and task-aware prompts using a task-adaptive learning rate.

[0060] This step is one of the core aspects of this invention, and it consists of the following sub-steps:

[0061] (4.1) Calculation of transmembrane loss: Transmodal loss The loss is composed of a weighted average of Frobenius norm loss and cosine similarity loss, minimizing B. V and B K The distribution differences ensure that multimodal information is aligned in the shared space:

[0062]

[0063] Among them ||·|| F The Frobenius norm ensures that low-rank matrices in different modalities remain structurally similar, while the cosine similarity term encourages directional alignment, thereby promoting shared representation learning. m [i] represents the low-rank matrix B m The i-th row vector. The relative contributions of these two terms are controlled by λ1 and λ2;

[0064] (4.2) Cross-task loss calculation: Cross-task loss It is an explicit constraint based on domain feature clustering centers, which improves the model's generalization ability by aligning task-aware cues from different tasks to shared subspaces of similar tasks; assuming S m (T i ,T j ) is the corresponding modality of task T i and T j The cosine similarity between clustering features, where t is the threshold for distinguishing positive and negative samples. If S m (T i T j If ) > t, then task T i and T j Tips between and Treat them as positive sample pairs and set the label z. ij =1, similarity should be maintained; if S m (T i T j If t < t, then task T i and T j The hints between them are considered negative sample pairs, z ij =0, should be separated; specifically, cross-task loss This can be achieved using the following formula:

[0065]

[0066] in, It is the cross-task loss between tasks i and j. τ is the cross-modal loss for task i, Npos and Nneg are the number of positive and negative samples, and τ is the temperature scaling parameter.

[0067] (4.3) Calculation of total training loss: The formula for calculating the total training loss is as follows:

[0068]

[0069] in It is the CTC task loss for sign language recognition and translation. It is a cross-modal alignment loss. It is a cross-task alignment loss designed to mitigate forgetting; hyperparameter λ base , λ CMA , λ CTA Used to balance different loss contributions.

[0070] (4.4) Optimize and update model parameters based on task-adaptive learning rate:

[0071] Considering that the data distribution may vary significantly across different tasks, a task-adaptive learning rate η is also introduced during the training optimization process. t This allows the model to dynamically adjust its update rate when learning new tasks, thereby enhancing its adaptability to new tasks while maintaining the stability of previous tasks. The specific design of the task-adaptive learning rate is as follows:

[0072] η t =η0+γ(1-S V (T i ,T0)),

[0073] Where η0 is the base learning rate, γ is the condition factor, and S V (T i (T0) is the current task T i Similarity to the baseline task T0 in terms of video modality.

[0074] Step 5: After the current task training is completed, the video features of each task are reduced in dimensionality by t-SNE and then clustered by K-means to generate 5 cluster centers. Save the data-aware prompt generator, task-aware prompt, and task cluster centers for that task.

[0075] Step Six: Continue to accept data from other tasks, repeating steps one through six until the model has learned on all tasks;

[0076] Step 7: After all tasks have been learned, in the inference phase, for the current input to be inferred, the time frame average of the input video features is calculated. The KNN algorithm is used to search for the most similar sample in the cluster centers of all historical tasks, and the corresponding task-aware cues and data-aware cues generators are matched. Then, the generator generates data-aware cues based on the input data, which together with the task-aware cues form the final cues. These cues are then merged with the input and fed into the pre-trained model to obtain the final translation prediction result.

[0077] The following describes the experimental process of the method according to the embodiments of the present invention.

[0078] Experiments were conducted on a single NVIDIA 4090 GPU. A sign language translation model with a multimodal Transformer encoder-decoder architecture extended from SLRT was used as the pre-trained model. The cue length was set to 20, and the cue dimension was set to 832. Training was performed using the Adam optimizer with weight decay set to 1.0 × 10⁻⁶. -3 For the task-adaptive learning rate, η0 = 1.0 × 10⁻⁶. -4 γ = 5.0 × 10 -4 In the low-rank decomposition, the rank is set to 4. For the hyperparameters used to control the optimization process, λ1 = λ2 = 0.5, λ base =1.0, λ CMA =0.1, λ CTA =0.1.

[0079] To evaluate the proposed method, a continuous learning benchmark for sign language translation was established, and extensive experiments were conducted on the PHOENIX-2014T and CSL-Daily datasets. PHOENIX-2014T contains continuous, unconstrained sign language sequences from nine different sign language users, covering 1066 unique words. It has 7096, 519, and 642 samples in the training, validation, and test sets, respectively. The corresponding translations are given in spoken German, with a vocabulary of 2887 different words. CSL-Daily is a large-scale Chinese sign language translation dataset characterized by everyday conversational scenarios. It contains sign language videos from ten different sign language users, with corresponding Chinese translations of 2000 and 2343 words, respectively. The training, validation, and test sets have 18401, 1077, and 1176 partitions, respectively. To simulate a continuous learning environment, signers are treated as different domains. Specifically, the model was initially trained on sign language users 1-6 of PHOENIX-2014T and sign language users 0-6 of CSL-Daily to establish a baseline, and then incrementally adapted on the remaining sign language users.

[0080] For translation quality, this invention uses BLEU for evaluation, with n-gram levels ranging from 1 to 4. For continuous learning performance, this invention uses average task accuracy (the model's average BLEU score across all tasks after learning all tasks) and forgetting rate (FM) as evaluation metrics. The forgetting rate quantifies the extent to which the model forgets previously learned tasks when adapting to a new task. The forgetting rate f of the model on the j-th task after learning the k-th task is... jk It can be calculated using the following formula:

[0081]

[0082] Where a ij This represents the model's performance after completing the i-th task out of the first j tasks. The total forgetting rate FM for the k-th task. k It is the average forgetting rate of all old tasks:

[0083]

[0084] The FM evaluation metric used in this invention is the average forgetting rate of BLUE 1-4 across all tasks. The evaluation results are shown in Table 1.

[0085] Table 1 Comparison of results between the method of the present invention and existing methods

[0086]

[0087]

[0088] Corresponding to the aforementioned embodiment of an interactive multimodal prompt fusion method for continuous learning of sign language translation, the present invention also provides an embodiment of an interactive multimodal prompt fusion device for continuous learning of sign language translation.

[0089] See Figure 3 The present invention provides an interactive multimodal prompt fusion device for continuous learning of sign language translation, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements an interactive multimodal prompt fusion method for continuous learning of sign language translation as described in the above embodiment.

[0090] The present invention provides an embodiment of an interactive multimodal prompting fusion device for continuous learning in sign language translation. This device can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is an interactive multimodal prompting fusion device for continuous learning in sign language translation provided by the present invention. Except for... Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0091] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0092] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0093] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements an interactive multimodal prompting fusion method for continuous learning of sign language translation as described in the above embodiments.

[0094] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0095] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned interactive multimodal prompt fusion method for continuous learning in sign language translation.

[0096] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0097] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An interactive multimodal cue fusion method for continuous learning in sign language translation, characterized in that, The method includes: S1. Process the arriving task data and extract video features and key point features from the sign language video; S2. Freeze the parameters of the pre-trained sign language translation model, and create and initialize the data-aware cue generator and task-aware cue for the current task: S3. Based on the prompt generation and fusion mechanism, the data-aware prompt generator generates corresponding modal data-aware prompts based on the input video features and key point features. The data-aware prompts are concatenated with the task-aware prompts. The concatenated prompts and multimodal sign language data are input into the frozen pre-trained sign language translation model, and the translation prediction results are output. S4. Calculate the total training loss function for recognition and translation, and optimize the parameters of the data-aware prompt generator and the task-aware prompt using the task-adaptive learning rate; S5. After the current task training is completed, the video features of each task are reduced in dimensionality and then clustered to generate cluster centers. The cluster centers of the dataset perception prompt generator, task perception prompt and video features corresponding to the task are saved. S6. Continue to accept data from other tasks and repeat the above steps until the model has learned on all tasks. S7. After all tasks have been learned, in the inference phase, for the current input to be inferred, the time frame average of the input video features is calculated. The KNN algorithm is used to search for the most similar sample in the cluster centers of all historical tasks and match the corresponding task-aware prompts and data-aware prompt generators. Then, the generator generates data-aware prompts based on the input data, which together with the task-aware prompts form the final prompts. These are then merged with the inputs into the pre-trained model to obtain the final translation prediction result.

2. The interactive multimodal prompting fusion method for continuous learning in sign language translation according to claim 1, characterized in that, The data-aware cue generator includes a cross-attention mechanism and an MLP module, with video feature modalities and keypoint feature modalities sharing a single generator; Data-aware prompt generator f DP After receiving the input video features or keypoint features, a learnable linear projection matrix W is used. k and W v The corresponding attention keys and values ​​are generated and multi-head attention computation is performed with the shared query matrix. The query matrix is ​​trained in the first continuous learning task and shared across subsequent tasks and different modalities within the same task. The attention output is then input into the MLP for feature refinement to generate data-aware cues for the corresponding modality. The specific formula is as follows: in, and These are data-aware cues generated for video and keypoint inputs, V and B is the batch size, L is the number of video frames, and Attn represents the multi-head attention mechanism. It is a query matrix, W k and W v It is a learnable linear projection matrix used to transform input features into the keys and values ​​required by the attention mechanism.

3. The interactive multimodal prompting fusion method for continuous learning in sign language translation according to claim 1, characterized in that, The task-aware cue is initialized by decomposing a low-rank matrix into the product of a shared matrix and a mode-specific matrix to reduce parameters and improve depth alignment across membrane states. The task-aware cue is expressed by the following formula: where r is the decomposition rank and r < min(l / 2, d), d is the embedding dimension, A is the low-rank matrix shared across modalities, and B m is the modality-specific matrix, which is B V and B K for the video modality and the keypoint modality respectively. "·" represents the matrix dot product operation. Cross-modal knowledge interaction is achieved through the shared matrix A, and different modality feature distributions are adapted through the modality-specific matrix B m ​ 4. The interactive multimodal prompting fusion method for continuous learning in sign language translation according to claim 1, characterized in that, The specific steps of concatenating data-aware prompts and task-aware prompts include: The final cue P for the corresponding modality is constructed by combining task-aware cues and data-aware cues. m : Where m∈{Video(V), Keypoint(K)}, and [;] denotes a join operation. It is a task-aware prompt, where l and d represent the length and dimension of the prompt, respectively.

5. The interactive multimodal prompting fusion method for continuous learning in sign language translation according to claim 1, characterized in that, The total loss function for model training includes: in It is the CTC task loss for sign language recognition and translation. It is cross-modal alignment loss. It is the cross-task alignment loss; hyperparameter λ base , λ CMA , λ CTA Used to balance different loss contributions.

6. The interactive multimodal prompting fusion method for continuous learning in sign language translation according to claim 5, characterized in that, The calculation process for the transmembrane loss consists of a weighted sum of Frobenius norm loss and cosine similarity loss, minimizing B. V and B K The distribution differences ensure that multimodal information is aligned in the shared space: Among them ||·|| F The Frobenius norm ensures that low-rank matrices in different modalities remain structurally similar, while the cosine similarity term encourages directional alignment, thus promoting shared representation learning. m [i] represents the low-rank matrix B m The relative contributions of the two terms in the i-th row vector are controlled by λ1 and λ2.

7. The interactive multimodal prompting fusion method for continuous learning in sign language translation according to claim 5, characterized in that, Cross-task loss is calculated using the following formula: Among them, S m (T i , T j ) is the cosine similarity between the clustering features of task T i and T j . t is the threshold for distinguishing positive and negative samples. If S m (T i , T j ) > t, then the hint between task T i and T j is regarded as a positive sample pair, and the label z and are regarded as positive sample pairs, and the label z ij = 1, and the similarity should be maintained; if S m (T i , T j ) < t, then the hint between task T i and T j is regarded as a negative sample pair, z ij = 0, and separation should be performed; is the cross-task loss between tasks i and j, is the cross-modal loss of task i. Npos and Nneg are the numbers of positive and negative samples, and τ is the temperature scaling parameter.

8. The interactive multimodal prompting fusion method for continuous learning in sign language translation according to claim 1, characterized in that, The specific design of the task-adaptive learning rate is as follows: or t =η0+γ(1-S V (T i ,T0)), Where η0 is the base learning rate, γ is the condition factor, and S V (T i (T0) is the current task T i Similarity to the baseline task T0 in terms of video modality.

9. An interactive multimodal prompting fusion device for continuous learning in sign language translation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an interactive multimodal prompting fusion method for continuous learning of sign language translation as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an interactive multimodal prompting fusion method for continuous learning of sign language translation as described in any one of claims 1-8.

Images