Dynamic decoupling prompt generation and tuning method for modal missing scene

Abstract

The invention relates to a computer vision and natural language processing technology, and provides a dynamic decoupling prompt generation and tuning method for a modal missing scene. Through a multi-channel mechanism, modal feature learning and alignment processes are separated, and the problem that the performance of an existing multi-modal model is remarkably reduced in a modal missing scene is effectively solved. A lightweight multi-modal prompt generator is designed, a missing modal compensation prompt is generated through a lightweight residual network, the calculation efficiency is remarkably improved, meanwhile, efficient integration of modal features is achieved through a dynamic prompt fusion strategy, and the utilization efficiency and contribution ability of a model to multi-modal information are remarkably enhanced. Based on a single-channel freezing-optimization strategy and a non-strong coupling alignment channel combined tuning mechanism, dynamically-decoupled multi-channel prompt tuning is realized, characteristics of different modes are independently optimized, non-strong coupling fusion is performed on information among the modes, excessive dependence of a model on single-mode information is remarkably reduced, and information balance among the modes is realized.

Description

Technical Field

[0001] The present invention relates to computer vision and natural language processing technologies, and in particular to a method for generating and optimizing dynamic decoupling prompts for modality-missing scenarios. Background Art

[0002] In the field of multimodal pre-training, large-scale models such as CLIP have demonstrated strong capabilities in numerous vision-related tasks, including image recognition, object detection, and image segmentation. Initial models often assume that data from all modalities is fully available during training and testing. However, with increasing research and expanding application scenarios, these models have been found to have limitations. In the real world, data unavailability, often referred to as modality loss, can occur due to various factors, such as strict privacy policies that restrict access to certain data or technical limitations that prevent the complete acquisition of data from all modalities. This modality loss has become a significant factor limiting the performance of multimodal models, severely impacting their effectiveness in real-world applications. Furthermore, to enhance the adaptability of multimodal models to specific downstream tasks, prompt tuning techniques, inspired by prompt engineering in natural language processing (NLP), have emerged. These techniques primarily improve model performance by optimizing prompts for specific tasks.

[0003] Following the application of learnable unimodal cues and multimodal coupling cues in multimodal models, a joint cue refinement method (MPVR) has recently been applied to multimodal models, aiming to mitigate the performance degradation caused by modal differences in training or test data samples in modality-missing scenarios. Despite the excellent performance of the MPVR model, subsequent research has not been limited to the existing achievements of the model, but has focused on how to further optimize and improve it. For example, to address the exponential growth of cues in MPVR and its robustness in various modality-missing scenarios, some studies have proposed the design concept of using modality-specific markers as joint cues to simplify the model structure and improve its performance under different modality configurations. In addition, some studies have introduced generative cues, missing signal cues, and missing type cues to effectively generate features for the missing modality and promote the learning of cross-modal information.

[0004] Although these cue tuning techniques have improved model performance to a certain extent and shown great potential, single-modality, coupled, and joint cues may lead to poor performance due to differences between different modalities. Specifically, different modalities usually have different degrees of information density and complexity, and simple coupled or joint cue models have difficulty effectively balancing the contributions of these modalities. In addition, in the presence of modality loss, model performance may further deteriorate because these systems tend to over-rely on feature channels that are particularly effective for the underlying task, resulting in a phenomenon called channel bias. This bias prevents the model from effectively capturing key shared knowledge applicable to new tasks, thereby weakening the model's generalization ability in modality loss scenarios, making it impossible for the model to stably perform well when faced with complex and changing actual application scenarios.

[0005] In summary, the existing technical methods in the field of multimodal pre-training have certain limitations when facing the actual scenario of modality loss. A new method is urgently needed to solve these problems in order to improve the performance and generalization ability of the model under modality loss conditions and meet the needs of practical applications. Summary of the Invention

[0006] The technical problem to be solved by this invention is to propose a method for generating and optimizing dynamic decoupled prompts for modality-missing scenarios. This method aims to address the performance degradation caused by modality heterogeneity and the "channel bias" problem caused by over-reliance on specific features in the underlying task in existing technologies. The method mainly includes the following steps:

[0007] Training process:

[0008] S1. Sample data acquisition and preprocessing: Obtain data containing both text and image modalities, extract high-dimensional text embedding and visual embedding through pre-training models; then input the text embedding and visual embedding into Layer prompt generation network, which specifically includes a multimodal prompt generator, prompt fusion, and a transformer-based multimodal encoder;

[0009] S2, Multimodal Hint Generator: Dynamically generates hint vectors corresponding to missing modalities based on existing modality features;

[0010] S3. Hint fusion: Fusion the hint vectors using a head fusion or cross fusion strategy; the head fusion specifically involves fusing the hint vectors of the text modality and the hint vectors of the image modality, and then combining them with the text embedding and the visual embedding as the head fusion input sequence; the cross fusion specifically involves fusing the hint vectors of the text modality, the hint vectors of the image modality, the text embedding, and the visual embedding, and inserting the hint vectors of the modalities into the corresponding modality embedding to obtain a cross fusion input sequence;

[0011] S4, Transformer-based multimodal encoder: The input sequence obtained by prompt fusion is input into a multi-layer Transformer-based multimodal encoder for processing to generate a joint representation;

[0012] S5. Apply dynamic decoupled multi-channel prompt tuning technology. By building a collaborative mechanism for modality-decoupled single-channel optimization and non-strongly coupled alignment channel joint tuning, dynamic prompt tuning for text / image single channels and cross-modal prompt joint tuning for alignment channels can be achieved.

[0013] Prediction process:

[0014] Y1. Data preprocessing: Preprocess the image and text data to be predicted;

[0015] Y2. The pre-trained model extracts features from the pre-processed image data and text data to obtain text embedding and visual embedding, which are then input into a multi-layer prompt generation network.

[0016] Y3. Generate corresponding prompt vectors for input modality data through the trained multimodal prompt generator;

[0017] Y4. Fuse the text embedding, visual embedding, and generated cue vector, choosing either a head fusion or cross fusion strategy to obtain the fused input sequence, which is fed into a transformer-based multimodal encoder to obtain a joint representation.

[0018] Y5. Use dynamic decoupling multi-channel prompt tuning mechanism to dynamically tune input data;

[0019] First, single-modal prompt tuning is performed based on the dynamic decoupling mechanism, optimizing the prompt vectors of text and image respectively; then, alignment channel prompt tuning is performed;

[0020] Y6. The fused and fine-tuned joint representation is fed into the decoder to generate the final task output.

[0021] As a further optimization, the multimodal prompt generator is specifically a multi-layer perceptron. For a given image modality and text modality, if there is missing data in the modality, the corresponding missing modality prompt is generated: text modality missing prompt vector, image modality missing prompt vector; if the modality is complete, the complete modality prompt is directly used: text complete modality prompt vector, image complete modality prompt vector.

[0022] As a further optimization, the prompt fusion is as follows:

[0023] Head fusion: First, use the element-wise sum operation to fuse the prompt vectors processed by all modalities into head fusion prompts, and then splice them with text embedding and visual embedding to obtain the head fusion input sequence. The formula is as follows:

[0024]

[0025]

[0026] in, For head fusion prompt, represents the element-wise sum operation, is a text modal prompt vector, specifically a text modal missing prompt vector or a text complete modal prompt vector, is an image modality prompt vector, specifically an image modality missing prompt vector or an image complete modality prompt vector; For the The layer prompts the generation network head to fuse the input sequence; Indicates the The text input sequence of the layer, Indicates the The visual input sequence of the layer comes from text embedding and visual embedding respectively;

[0027] Cross-integration: For Layer, the formula for cross-fusion hint is:

[0028] in 、 Respectively expressed in Layer-inserted hint vectors for text and image modalities; For the The layer prompts the generation network to cross-fuse the input sequence.

[0029] As a further optimization, the multi-channel prompt tuning is as follows:

[0030] Single channel tuning: freeze the visual modality hints during text channel training and only optimize the text modality hints; freeze the text modality hints during image channel training and only optimize the visual modality hints; the total loss of the text channel and image channel is calculated by KL divergence and classification loss constitute:

[0031] Total loss of text channel :

[0032] Total loss of image channels :

[0033]

[0034] in 、 is the weight parameter;

[0035] When aligning channel hints for tuning, the loss function uses classification loss.

[0036] As a further optimization, the text embedding and visual embedding are generated as follows:

[0037] Generating text embeddings: First, the input text is segmented and positional encoding is added to each word. Then, a multi-head attention mechanism decoder is used to process the positional encoded text. Finally, each vocabulary unit in the text is mapped to a high-dimensional embedding vector, and these vectors are concatenated to form the text embedding representation.

[0038] Generation of visual embeddings: First, the input image is divided into several fixed-size image blocks, and then an embedding vector is assigned to each image block. Subsequently, a positional encoding is added to the embedding vector of each image block. Finally, the embedding vectors of all image blocks are concatenated into a sequence to form the overall embedding representation of the image.

[0039] As a further optimization, the specific formula of the multimodal prompt generator is as follows:

[0040]

[0041] in, It is used for The weight matrix of the multi-head self-attention in the layer multimodal encoder, Representation layer normalization operation, is the activation function; Output prompts for the previous layer prompt generation network, When the text and image prompts are initialized.

[0042] As a further optimization, during the prediction process, the final task output is as follows:

[0043] In the text generation task, the decoder generates text descriptions through autoregressive decoding. The classification task outputs category labels through the fully connected layer and Softmax. The retrieval task calculates the similarity between text features and image features and retrieves the most relevant matching samples.

[0044] The beneficial effects of the present invention are:

[0045] (1) This paper proposes a dynamic decoupled hint generation and tuning method for modality-missing scenarios. By separating the modal feature learning and alignment processes through a multi-channel mechanism, it effectively solves the problem of significant performance degradation of existing multimodal models in modality-missing scenarios.

[0046] (2) The present invention realizes dynamic decoupled multi-channel prompt tuning based on a single-channel freeze-optimization strategy (alternating training of text / image channels) and a non-strongly coupled aligned channel joint tuning mechanism. It independently optimizes the characteristics of different modalities and performs non-strongly coupled fusion of information between modalities, significantly reducing the model's excessive dependence on single modality information (channel bias problem) and achieving information balance between modalities.

[0047] (3) This paper designs a lightweight multimodal cue generator, which generates missing modality compensation cues through a lightweight residual network, significantly improving computational efficiency. At the same time, it realizes efficient integration of modal features through a dynamic cue fusion strategy (head fusion and cross fusion), significantly enhancing the model's utilization efficiency and contribution to multimodal information.

[0048] (4) The experimental results of the present invention in multimodal data downstream tasks (such as classification, retrieval, generation, etc.) show that compared with the existing technology, the proposed method has achieved significant performance improvement on multiple benchmark datasets, demonstrating its superiority and applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the model training process in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the algorithm for model training in an embodiment of the present invention;

[0051] Figure 3 4 is a flow chart of the model prediction process in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention proposes a dynamic decoupled cue generation and tuning method for modality-missing scenarios, aiming to solve the performance degradation of existing technologies in modality-missing scenarios caused by modal heterogeneity and the "channel bias" problem caused by over-reliance on specific features in basic tasks. The method mainly includes a training process and a prediction process: during the training process, image-text pairs are first acquired and preprocessed from a multimodal dataset to generate high-dimensional text and visual embeddings; then a lightweight multimodal cue generator is designed to dynamically generate cue vectors for the missing modality, and head fusion and cross-fusion strategies are introduced to deeply integrate the cue vectors with the original embeddings, and input them into a Transformer-based multimodal encoder for joint representation learning; finally, a multi-channel cue tuning mechanism is adopted to optimize the model through single-channel tuning and aligned channel tuning to achieve decoupling and optimization of modal feature learning, thereby improving the complementarity of modal features and their adaptability to downstream tasks. During the prediction process, the image and text to be predicted are first preprocessed; then the pre-trained model is used for feature extraction; the hint generator is then used to supplement the missing modal information; finally, the generated hints are combined with the image and text features and input into the decoder through head fusion or cross fusion. The trained multi-channel tuning model is used to extract the complementary information between multimodal features to generate task-related prediction results.

[0053] Example:

[0054] In this embodiment, a dynamic decoupling prompt generation and tuning method for modality-missing scenarios includes two parts: a training process and a prediction process, which are described in detail below with reference to the accompanying drawings:

[0055] See also Figure 1 , the training process includes:

[0056] S1. Sample data acquisition and preprocessing:

[0057] S11. Obtain data containing both text and image modalities from a multimodal dataset, and extract high-dimensional text embeddings and visual embeddings using a pre-trained model. The image modality data is typically natural scene images or images related to specific tasks (such as visual question answering, image classification, etc.). The text modality data corresponding to each image is typically annotations, labels, or other relevant text information in natural language form. The specific steps for generating text embeddings and visual embeddings are as follows:

[0058] Generating Text Embeddings: First, the input text is tokenized and positional encoding is added to each word to preserve the relative position of the word in the sentence. A multi-head attention mechanism decoder is then used to process the positionally encoded text. Finally, each lexical unit in the text is mapped to a high-dimensional embedding vector, and these vectors are concatenated to form the text embedding.

[0059] Generating visual embeddings: First, the input image is divided into several fixed-size patches. Each patch is then assigned an embedding vector using a convolutional neural network and fully connected layers. A positional encoding is then added to the embedding vector of each patch to help the model distinguish the patch's relative position in the original image. Finally, the embedding vectors of all patches are concatenated into a sequence to form the overall embedded representation of the image.

[0060] Then input text embedding and visual embedding into The prompt generation network consists of a multimodal prompt generator, a prompt fusion, and a transformer-based multimodal encoder. The layer prompts the generative network to learn the relationship between modalities and generate joint representations.

[0061] S2. Design a lightweight multimodal cue generator that dynamically generates cue vectors for the missing modalities based on the features of the existing modalities. If the input data is complete, the cue is used to guide model learning. If a modality is missing (image or text), a cue for the missing modality is generated to supplement it. Specifically, for the image modality, a visual cue vector is generated; for the text modality, a text cue vector is generated.

[0062] S21. Definition and initialization of initial prompts: For given image and text modalities, initialize them to text and image prompts respectively, assuming that the initial value is the prompt vector of the complete modality and , Indicates the prompt length, is the vector dimension;

[0063] S22. Build a lightweight multimodal prompt generator , for dynamically generating compensation hints for missing modes: For Layer prompt generation network, which converts the current complete modality prompt vector As a generator The input of is used to obtain the compensation prompt of the missing modality; the specific formula of the multimodal prompt generator is as follows:

[0064]

[0065] in, It is used for The weight matrix of the multi-head self-attention (MSA) in the layer multimodal encoder, Representation layer normalization operation, Is the activation function used to process the normalized input ; In calculation, the original Added to the output as a residual connection to help preserve the transfer of modal information; Specifically, it is the output prompt of the previous layer prompt generation network. When , the initial text and image prompts are used; in order to further supplement the features of the missing modalities, a multi-layer perceptron (MLP) is used to collect additional missing modal information, thereby enhancing the network's learning ability in multimodal tasks;

[0066] S23, prompt selection mechanism: If there is missing data in a certain modality, a corresponding missing modality prompt is generated, such as the text modality missing prompt vector , image modality missing hint vector ; If the modality is complete, then use the complete modal prompt directly, such as the text complete modal prompt vector , image complete modality hint vector In this process, a lightweight multimodal prompt generator is applied Generate the prompt vector corresponding to the missing modality as follows:

[0067]

[0068] Indicates the The missing cue vector for the image modality is generated from the complete cue vector for the text modality in a layer network.

[0069]

[0070] Indicates the Missing hint vectors for the text modality are generated from the complete hint vectors for the image modality in a layer network.

[0071] The resulting missing modality hint vector is directly used as the hint input for the corresponding modality in subsequent fusion operations: When the text modality is missing, the generated text modality missing hint vector is used. As a subsequent fusion step ; When the visual modality is missing, use the generated image modality missing prompt vector As a subsequent fusion step .

[0072] S3. Introducing two cue fusion strategies: head fusion and cross fusion. Head fusion is the most basic fusion strategy, typically appending cue vectors from different modalities to the beginning of the MSA layer input sequence in each multimodal encoder as additional input. Cross fusion is a more complex fusion strategy, typically used to address situations where information from a particular modality is missing or corrupted. By alternating or cross-splicing cue vectors from different modalities, the model's ability to supplement the missing information is enhanced. Depending on the task requirements, the model can choose to adopt different cue fusion strategies, effectively concatenating the text embedding, image embedding, and generated cue vectors to form the initial input.

[0073] S31. The steps of head fusion prompt are as follows:

[0074] S311, in the head fusion stage, prompt processing is performed according to the missing conditions of each modality. When all modalities are complete, the initialized complete modality prompt vector is directly used: the text modality adopts , the image modality adopts For missing modalities, the corresponding missing prompt vector is used to replace them (when the text modality is missing, , used when image modality is missing ). Finally, the element-by-element summation operation is performed All modality processed prompt vectors are merged into a unified , the calculation process takes the complete prompt as an example and can be uniformly expressed as:

[0075]

[0076] in Represents element-wise sum operation;

[0077] S312, add head prompt: Added to the beginning of the input sequence of each layer, concatenated with the input sequence of text and image, for the first layer, the input of the multimodal encoder is:

[0078]

[0079] in For the Layer prompts the generation network head to fuse the input sequence, Indicates the The head fusion hint of the layer, Indicates the The text input sequence of the layer, Indicates the The visual input sequences of the layers come from text embedding and visual embedding respectively.

[0080] S32, the steps of cross-fusion prompt are as follows:

[0081] S321, Cross-fusion prompts: insert the prompts of a specific modality into its corresponding modality input to promote the interaction between the specific modality prompts and the embedding. Layer, the formula for cross-fusion hint is:

[0082]

[0083] in 、 Respectively expressed in hint vectors for text modalities and image modalities inserted by the layer, No. The layer prompts the generation network to cross-fuse the input sequence.

[0084] S322, Layered Fusion Processing: The cross-fusion strategy follows the principle of sequential fusion, incorporating cues into the multi-head self-attention (MSA) layer of each multimodal encoder. Each MSA layer receives cues obtained through cross-fusion, further helping the model learn relationships between modalities.

[0085] S33. This fusion process is recursive: each layer of the multimodal encoder reuses the cues and performs further fusion. Head-fused cues are the default strategy in this paper and have higher robustness in the overall experiment. However, cross-fused cues sometimes outperform head-fused cues in special scenarios (such as when there is severe loss).

[0086] S4: The input sequence obtained by head fusion or cross fusion is fed into a multi-layer Transformer-based multimodal encoder for processing. The multimodal encoder learns the relationship between text and image through a multi-head self-attention mechanism and generates a joint representation.

[0087] S5. Applying dynamic decoupled multi-channel prompt tuning technology, by building a collaborative mechanism between modality-decoupled single-channel optimization and non-strongly coupled aligned channel joint tuning, we achieve dynamic prompt tuning for text / image single channels and promote cross-modal prompt joint tuning for aligned channels. Specifically, single-channel prompt tuning optimizes the prompt vectors for text and image modalities separately to ensure modality-specific feature learning, while aligned channel tuning enhances information exchange between modalities and improves model generalization.

[0088] In this way, the model can flexibly distribute information when processing data from different modalities, thereby improving the ability of cross-modal learning and overall performance.

[0089] S51. Dynamic Decoupling Single-Channel Optimization: Text channel training: freezes visual modal cues and optimizes only text modal cues. At this stage, the model's goal is to capture the independent properties of the text modality; Image channel training: freezes text modal cues and optimizes only visual modal cues. At this stage, the model's goal is to capture the independent properties of the visual modality.

[0090] S52. Loss design for independent modalities: For each channel, a loss function is designed, and the total loss is determined by the Kullback-Leibler (KL) divergence and classification loss constitute:

[0091] The total loss of the text channel is calculated as follows:

[0092]

[0093] The total loss of the image channel is calculated as follows:

[0094]

[0095] parameter and It is used to control the contribution weight of KL divergence in text / image channels to the total loss respectively, effectively retaining modality-specific information.

[0096] In single-channel training, KL divergence is used to measure the distribution difference between text and visual cues while suppressing redundant features between modalities and enhancing the independent representation ability of each channel. The calculation formula is as follows:

[0097]

[0098] in, and Represents text and visual cues, respectively, and Indicates the Layer-based text / visual cues in a Transformer-based multimodal encoder, is the dimension index of the prompt vector in the feature space, is the total number of layers, is the number of samples.

[0099] Classification loss It is used to promote modality fusion and also to process classification tasks, as shown below:

[0100]

[0101] in is the true label of the sample, is the model’s predicted probability, is the index of the category label;

[0102] S53, Alignment Channel Hint Tuning: To further enhance the model's adaptability to multimodal tasks, in this phase, the model focuses on learning and capturing the alignment relationship between modalities, and further optimizes the model through training in the alignment phase. In this channel, the hint vectors of text and image are trainable throughout the training process, and the hints of any one modality are no longer frozen. In the alignment phase, the model only optimizes the classification loss. , the formula is as follows:

[0103]

[0104] Through multi-channel alignment training, the model can effectively learn the relationship between textual and visual modalities, thereby enhancing the perception and understanding of modal interaction features;

[0105] S54. In practical applications, we first use dynamic decoupling single-channel prompt tuning to independently optimize text prompts and image prompts, and use the dynamic decoupling mechanism to ensure the independence of each modality prompt. This process can effectively enhance the representation capabilities of the text and image modalities and dynamically avoid interference between modalities. Then we enter the alignment channel prompt tuning stage. In this stage, the text and image prompts are fused and optimized through multimodal alignment, further promoting mutual understanding and collaborative work between modalities. Through this two-stage optimization strategy, the model can fully explore the correlation between modalities while maintaining the independence of each modality, thereby effectively improving the overall performance and adaptability of the model in multimodal tasks. Repeat steps S2-S5 until a trained model is obtained.

[0106] The algorithm principle of the model training in the above steps S1-S5 is as follows Figure 2 shown.

[0107] The prediction process flow in this embodiment can be found in Figure 3 , which includes:

[0108] S1. Data preprocessing:

[0109] In this step, the image and text data to be predicted must first be preprocessed to ensure that the input data distribution is consistent with the data distribution during training.

[0110] S2. Use the pretrained model to extract features from the preprocessed image and text data to obtain corresponding feature vectors. In this step, image feature extraction uses the visual Transformer to extract image features and obtain visual embeddings. Text feature extraction uses the pretrained language model BERT to embed and encode the text to obtain context-sensitive feature representations, namely text embeddings. The text embeddings and visual embeddings are input into the multi-layer prompt generation network.

[0111] S3. Generate corresponding prompts based on the input modality information (complete or missing) through the trained lightweight multimodal prompt generator:

[0112] In this step, if the input data is complete, a complete modality prompt is generated to enhance the modality information; if a modality is missing, a missing modality prompt is generated to complete the missing information and improve the alignment capability of the model.

[0113] S4. Fuse the text features, image features, and generated hint vectors, and select different fusion strategies:

[0114] In this step, the head fusion strategy or cross fusion strategy is adopted according to the task to obtain the fused sequence and , and feed it into the transformer-based multimodal encoder to further model the interaction relationship between modalities.

[0115] S5. Dynamically tune the input data using a dynamic decoupling multi-channel prompt tuning mechanism to further extract complementary information between modalities:

[0116] In this step, we first perform single-modal prompt tuning based on the dynamic decoupling mechanism, optimizing the prompt vectors for text and images separately to ensure the independence of each modality. Then, we perform alignment channel prompt tuning, introducing KL divergence loss and classification loss to calculate the distribution differences between different modalities. Targeted prompt tuning for modality alignment is then carried out to strengthen cross-modal associations.

[0117] S6. The fused and optimized multimodal feature sequence is fed into the decoder to generate the final task output:

[0118] In this step, in text generation tasks (such as image description generation), the decoder generates text descriptions through autoregressive decoding. Classification tasks (such as image-text matching) output category labels through fully connected layers and Softmax. Retrieval tasks (such as cross-modal retrieval) calculate the similarity between text features and image features (such as cosine similarity) and retrieve the most relevant matching samples.

[0119] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A method for generating and optimizing dynamic decoupling prompts for modality-missing scenarios, characterized by: The following steps are involved: Training process: S1. Sample data acquisition and preprocessing: Obtain data containing both text and image modalities, extract high-dimensional text embedding and visual embedding through pre-training models; then input the text embedding and visual embedding into Layer prompt generation network, which specifically includes a multimodal prompt generator, prompt fusion, and a transformer-based multimodal encoder; S2, Multimodal Hint Generator: Dynamically generates hint vectors corresponding to missing modalities based on existing modality features; S3. Hint fusion: Fusion the hint vectors using a head fusion or cross fusion strategy; the head fusion specifically involves fusing the hint vectors of the text modality and the hint vectors of the image modality, and then combining them with the text embedding and the visual embedding as the head fusion input sequence; the cross fusion specifically involves fusing the hint vectors of the text modality, the hint vectors of the image modality, the text embedding, and the visual embedding, and inserting the hint vectors of the modalities into the corresponding modality embedding to obtain a cross fusion input sequence; S4, Transformer-based multimodal encoder: The input sequence obtained by prompt fusion is input into a multi-layer Transformer-based multimodal encoder for processing to generate a joint representation; S5. Apply dynamic decoupled multi-channel prompt tuning technology. By building a collaborative mechanism for modality-decoupled single-channel optimization and non-strongly coupled alignment channel joint tuning, dynamic prompt tuning for text and image single channels and cross-modal prompt joint tuning for alignment channels can be achieved. Prediction process: Y1. Data preprocessing: Preprocess the image and text data to be predicted; Y2. The pre-trained model extracts features from the pre-processed image data and text data to obtain text embedding and visual embedding, which are then input into a multi-layer prompt generation network. Y3. Generate corresponding prompt vectors for input modality data through the trained multimodal prompt generator; Y4. Fuse the text embedding, visual embedding, and generated cue vector, choosing either a head fusion or cross fusion strategy to obtain the fused input sequence, which is fed into a transformer-based multimodal encoder to obtain a joint representation. Y5. Use dynamic decoupling multi-channel prompt tuning mechanism to dynamically tune the input data. First, single-modal prompt tuning is performed based on the dynamic decoupling mechanism, optimizing the prompt vectors of text and image respectively; then, alignment channel prompt tuning is performed; Y6. The fused and fine-tuned joint representation is fed into the decoder to generate the final task output.

2. A method for generating and optimizing dynamic decoupling prompts for modality-missing scenarios according to claim 1, characterized in that: The multimodal prompt generator is specifically a multi-layer perceptron. For a given image modality and text modality, if there is missing data in the modality, the corresponding missing modality prompt is generated: text modality missing prompt vector, image modality missing prompt vector; if the modality is complete, the complete modality prompt is directly used: text complete modality prompt vector, image complete modality prompt vector.

3. The method for generating and optimizing dynamic decoupling prompts for modality-missing scenarios according to claim 2, characterized in that: The prompt fusion is specifically as follows: Head fusion: First, use the element-wise sum operation to fuse the prompt vectors processed by all modalities into head fusion prompts, and then splice them with text embedding and visual embedding to obtain the head fusion input sequence. The formula is as follows: ； ； in, For head fusion prompt, represents the element-wise sum operation, is a text modal prompt vector, specifically a text modal missing prompt vector or a text complete modal prompt vector, is an image modality prompt vector, specifically an image modality missing prompt vector or an image complete modality prompt vector; Generate the network head fusion input sequence for the i-th layer prompt; Indicates the The text input sequence of the layer, Indicates the The visual input sequence of the layer comes from text embedding and visual embedding respectively; Cross-integration: For Layer, the formula for cross-fusion hint is: ; in 、 Respectively expressed in Hint vectors for text and image modalities inserted by the layer; The i-th layer of the prompt generation network cross-fuses the input sequence.

4. The method for generating and optimizing dynamic decoupling prompts for modality-missing scenarios according to claim 3, characterized in that: The multi-channel prompt tuning is as follows: Single channel tuning: freeze the visual modality hints during text channel training and only optimize the text modality hints; freeze the text modality hints during image channel training and only optimize the visual modality hints; the total loss of the text channel and image channel is calculated by KL divergence and classification loss constitute: Total loss of text channel : ; Total loss of image channels : ； in 、 is the weight parameter; When aligning channel hints for tuning, the loss function uses classification loss.

5. The method for generating and optimizing dynamic decoupling prompts for modality-missing scenarios according to claim 4, characterized in that: The generation of text embedding and visual embedding is as follows: Generating text embeddings: First, the input text is segmented and positional encoding is added to each word. Then, a multi-head attention mechanism decoder is used to process the positional encoded text. Finally, each vocabulary unit in the text is mapped to a high-dimensional embedding vector, and these vectors are concatenated to form the text embedding representation. Generation of visual embeddings: First, the input image is divided into several fixed-size image blocks, and then an embedding vector is assigned to each image block. Subsequently, a positional encoding is added to the embedding vector of each image block. Finally, the embedding vectors of all image blocks are concatenated into a sequence to form the overall embedding representation of the image.

6. The method for generating and optimizing dynamic decoupling prompts for modality-missing scenarios according to claim 5, characterized in that: The specific formula of the multimodal prompt generator is as follows: ； in, It is used for The weight matrix of the multi-head self-attention in the layer multimodal encoder, Representation layer normalization operation, is the activation function; Output prompts for the previous layer prompt generation network, When the text and image prompts are initialized.

7. The method for generating and optimizing dynamic decoupling prompts for modality-missing scenarios according to claim 6, characterized in that: During the prediction process, the final task output is as follows: In the text generation task, the decoder generates text descriptions through autoregressive decoding. The classification task outputs category labels through the fully connected layer and Softmax. The retrieval task calculates the similarity between text features and image features and retrieves the most relevant matching samples.

Images