Incomplete multi-mode learning method based on prompt distillation

Abstract

The invention provides an incomplete multi-modal learning method based on prompt distillation, and aims to improve the robustness and generalization ability of a multi-modal model in a modal missing scene. According to the method, the advantages of knowledge distillation and prompt learning are fused, a collaborative framework composed of a hierarchical prompt generator, a teacher network and an inference network is constructed, and efficient migration of inter-modal shared knowledge and sample-level fine-grained information is realized. The method comprises the following steps: firstly, constructing and preprocessing a multi-modal data set; based on a prompt distillation mechanism, an efficient knowledge transmission mechanism is established between the teacher network and the reasoning network; and finally, inference is performed on the incomplete modal sample by using the trained prompt generator and the inference network, so that the accuracy and the adaptive capacity of downstream tasks are remarkably improved. According to the method, through collaborative optimization of the prompt generator and the double-distillation mechanism, the knowledge migration efficiency is remarkably improved while the light weight of the model is kept, and an efficient solution is provided for multi-modal learning in a complex real environment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an incomplete multimodal learning method based on cue distillation, which aims to improve the robustness and adaptability of the model under modality missing conditions. Background Technology

[0002] Multimodal learning aims to integrate data from different modalities (such as images, text, and audio) to improve model performance in tasks such as perception, understanding, and generation. However, in practical applications, due to limitations in data acquisition, equipment failure, privacy protection, or resource costs, certain modalities are often missing from the input data. This lack of modality leads to incomplete input information, forming a typical incomplete multimodal learning problem. This problem severely affects the robustness and generalization ability of the model and is one of the important challenges that urgently need to be addressed in the field of multimodal learning.

[0003] To address this challenge, various methods have been proposed in previous research. Some works attempt to reconstruct missing modality data through generative models or align multimodal features in the latent space to mitigate the impact of missing information. Furthermore, methods introduce auxiliary tasks, multi-task learning mechanisms, or shared representation structures to enhance the support capabilities of available modalities for downstream tasks. A representative method is ShaSpec, which achieves feature aggregation by sharing task heads (Hu Wang, Yuanhong Chen, et al. Multimodal learning with missing modality via shared-specific featuremodelling. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 15878–15887, 2023.). However, most of these methods assume that the missing modalities are known and fixed, and are typically accompanied by high training overhead and a large number of model parameter updates. In recent years, knowledge distillation-based strategies have also been increasingly applied to the problem of missing modalities. These methods train a "teacher model" with complete modal inputs to guide a "student model" that only receives partial modal inputs to learn key knowledge, thereby improving its independent reasoning ability. For example, CorrKD (M.Li et al., Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities, 2024 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 12458-12468.) and UMDF (Yuxuan Lei, Mingcheng Li, Liuzhen Su, et al. A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities, AAAI 2024, Underline Science Inc.) have shown some flexibility in dealing with incomplete modal combinations, but still have shortcomings such as high distillation costs and limited task transferability.

[0004] With the significant achievements of prompt learning in fields such as natural language processing and computer vision, this lightweight model adaptation paradigm has been gradually introduced into multimodal incomplete learning tasks. Prompt learning guides pre-trained models to adapt to specific tasks by introducing learnable prompt vectors, achieving superior performance with only a small number of parameters needing fine-tuning. Against this backdrop, some studies have attempted to incorporate prompting mechanisms into incomplete multimodal learning frameworks to address the performance degradation and computational burden caused by missing modalities. For example, DCP (Lianyu Hu, Tongkai Shi, Wei Feng, Fanhua Shang, and Liang Wan. Deep Correlated Prompting for Visual Recognition with Missing Modalities. In Advances in Neural Information Processing Systems 38 (NeurIPS), 2024.) proposed deep correlation prompting, utilizing the correlation between prompts at different layers and the relationship between prompts and input features to guide prompt design. DePT (Ji Zhang, Shihan Wu, Lianli Gao, Hengtao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024.) further proposed a knowledge decoupling mechanism during prompt tuning, separating fundamental knowledge in feature channels into an independent space, thereby improving the model's adaptability and expressive power. However, these methods still face many challenges, including insufficient interaction between prompts and model structure, poor prompt extensibility, lack of hierarchical relationship modeling, and limited adaptability under different missing combinations.

[0005] In summary, current multimodal learning methods for modality-deficient scenarios still suffer from poor adaptability, high computational costs, and inflexible prompting mechanisms. Therefore, there is an urgent need for a multimodal learning method that can achieve efficient knowledge transfer and task adaptation without requiring complete modality input, thereby improving the robustness and practicality of models in real-world modality-deficient environments. Summary of the Invention

[0006] This invention proposes an incomplete multimodal learning method based on cue distillation, which combines the advantages of knowledge distillation and cue learning. It aims to solve the core problems of existing multimodal learning, such as insufficient model robustness and low efficiency of cross-modal information utilization in scenarios with missing modalities. In particular, it is difficult to balance inference performance and computational cost when facing complex modal combinations.

[0007] The objective of this invention is achieved through the following technical solution: an incomplete multimodal learning method based on cue distillation, comprising the following steps:

[0008] Step 1: Construct and preprocess the multimodal dataset, simulate the missing cases by randomly masking some modalities, and divide it into training and test sets;

[0009] Step 2: Design a hierarchical prompt generator to generate modality-level prompts and instance-level prompts respectively. The modality-level prompts inform the teacher network of the missing modalities and induce the teacher network to generate specific modality knowledge to guide the inference network. The instance-level prompts help the inference network capture instance-specific knowledge for each input sample. Based on the prompt distillation mechanism, intrinsic modality shared knowledge distillation and task-specific knowledge distillation are used to train the teacher network using complete modality data and the inference network using incomplete modality data.

[0010] Step 3: Use the trained hierarchical prompt generator and inference network to perform inference on incomplete modal samples.

[0011] Furthermore, step 2 is detailed as follows:

[0012] Step 21: First, encode the missing modal data to obtain modal embeddings; for the modal-level cue generator, calculate a corresponding cue P for each missing modal case. t Then, modal-level cues are obtained through affinity projection functions. Right now Where M represents the total number of modes, 2 M -1 covers all possible missing modalities;

[0013] Step 22: For the instance-level hint generator, denote the modal embedding as... Where 1≤n≤M represents the modal index, and the hints for each instance are calculated.

[0014]

[0015] Where G(·) represents an instance-level hint generator, implemented through a multilayer perceptron; [] represents a join operation;

[0016] Then, instance-level prompts are calculated using the cross-attention mechanism.

[0017]

[0018] Among them, W q W k W v Indicates a transition layer; This represents the modal level cue corresponding to the k-th sample; d represents the dimension of the [·] vector;

[0019] Step 23: Obtain the modal-level hints Inserting it into the teacher network will yield instance-level hints. Inserted into the inference network; both the teacher network and the inference network use the ViLT pre-trained model, which includes an input layer and a transformer encoder layer, and the transformer encoder layer includes an attention layer; the cue is inserted into the input layer and attention layer of the ViLT pre-trained model, where both the input layer and attention layer are multi-layered structures;

[0020] Input layer hints are added to the input sequence of each layer:

[0021]

[0022] in, This represents the operation of the first layer of the neural network, Encoder. l This represents the first layer of a teacher network or inference network; [;] represents a join operation; P l This indicates a modal or instance-level hint inserted into level 1; h l This represents the hidden input of layer 1; parameter N represents the number of network layers to insert the cue.

[0023] For attention-level cues, the cue P will be... l The cue vectors are evenly divided into two sub-cues along the dimension of quantity, and then attached to the key and value projection layers of the multi-head self-attention MSA mechanism, respectively:

[0024]

[0025] Among them, ATTENTION l P represents the MSA of the l-th layer; i Divided into two sub-hints P k and P v Attached to key K of MSA respectively l Sum V l Projection layer;

[0026] Step 24: Train the teacher network and the inference network. The formula for the overall loss L is as follows:

[0027] L = L cls +αL acm +λ(L ikd +L tkd )

[0028] Where α and λ are two hyperparameters; L cls For the downstream task loss based on cross-entropy classification loss, L acm Affinity classification loss for modality-level prompts; L ikd The feature alignment loss aims to align sample features in the teacher network with corresponding missing modality sample features in the inference network; L tkd The downstream classification score loss aims to make the downstream classification scores of complete modal data points generated by the teacher model similar to the scores of corresponding incomplete modal samples in the inference model.

[0029] Furthermore, the affinity classification loss is specifically as follows:

[0030]

[0031] Among them, F cls (·) indicates the classification layer that produces the M-dimensional distribution scores; The target affinity label is represented, and 0 indicates that the corresponding modality is missing; BCE(·) represents the binary classification loss; x represents the input sample data, and i is the data index of all possible missing modalities; D represents all data in the dataset;

[0032] The feature alignment loss is as follows:

[0033]

[0034] in, and These represent the last layer of the teacher and the inference network, respectively, generating two hidden features; and Indicates the corresponding modal and instance-level hints; x i Represents a complete modal data point. x represents i The corresponding incomplete modal data points are obtained by randomly masking x. i One of the modes is obtained φ(·) is a measurement function;

[0035] The downstream classification score loss is as follows:

[0036]

[0037] Among them, Encoder t and Encoder iRepresenting the teacher network and inference network respectively, they generate scores for downstream classification tasks; D KL (·) denotes the KL divergence measurement function.

[0038] Furthermore, during training, all parameters of the ViLT pre-trained model in both the teacher and inference networks are frozen, and only the parameters relevant to the downstream task and learnable cues are trained.

[0039] Furthermore, step 1 is detailed as follows:

[0040] Step 11: For image modalities, adjust the image pixels while maintaining the aspect ratio. When handling image missing cases, use virtual input with all pixel values ​​set to 1.

[0041] For text modalities, a tokenizer is used to segment the input text; when handling missing text, an empty string is used as virtual input.

[0042] Step 12: Set the missing rate ξ% for the multimodal dataset. Generate missing modality data by randomly masking the modalities of samples, and divide the dataset into training and testing sets. The missing rate is the proportion of incomplete modality data in the entire dataset, including both training and testing phases. The missing rate ξ% is defined as:

[0043]

[0044] Where M represents the total number of modalities, N represents the total number of samples, and m k Let m represent the number of available modes for the k-th sample, and m k ≧1;

[0045] Step 13: Input the multimodal data, including the missing modal data, into the trained multimodal encoder to obtain the corresponding modal embeddings.

[0046] Furthermore, the reasoning process in step 3 is as follows:

[0047] Step 31: Preprocess the missing modal data of the task to be predicted; input it into the trained multimodal encoder to obtain modal embeddings;

[0048] Step 32: Input the obtained modality embedding into the hierarchical cue generator to obtain the modality-level cue and instance-level cue for the corresponding sample;

[0049] Step 33: Input the missing modality sample embeddings and the generated modality-level and instance-level hints into the inference network to calculate the prediction results for the downstream task.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] 1. This invention proposes an incomplete modality multimodal learning model, which adopts hierarchical cue learning and models modality missing cases by training a set of lightweight cue, rather than training multiple specific modality models for each missing scenario.

[0052] 2. This invention designs two types of prompts: modality-level and instance-level prompts, to capture inherent knowledge across modalities and instance-specific knowledge for each missing modality sample.

[0053] 3. This invention designs an incomplete modal knowledge distillation strategy to transfer the inherent modal shared knowledge and task-specific knowledge in the teacher network to the inference network.

[0054] 4. The present invention achieves better results in downstream tasks with missing modal data by designing a cue-based distillation-based incomplete multimodal learning method. Attached Figure Description

[0055] Figure 1 This is a flowchart of the incomplete multimodal learning method of the present invention.

[0056] Figure 2 This is the model training process flow in the embodiments of the present invention.

[0057] Figure 3 This is a flowchart of the model reasoning process in an embodiment of the present invention.

[0058] Figure 4 This is a flowchart of the model reasoning process in an embodiment of the present invention. Detailed Implementation

[0059] This invention aims to propose a cue distillation-based incomplete multimodal learning method to address the common modality missingness phenomenon in multimodal data, and to solve the problems of low training efficiency and limited inference performance in such scenarios of existing technologies. The proposed method designs a hierarchical cue distillation framework consisting of two core components: a teacher network trained on a small amount of complete modality data, and an inference network trained on a large amount of incomplete data. To effectively guide the model to learn knowledge at different levels, the proposed cue generator simultaneously generates modality-level cueing and instance-level cueing, used to capture cross-modal shared knowledge and instance-specific semantic information, respectively. By introducing a cue-based knowledge distillation mechanism, the transfer of shared knowledge and task-specific knowledge in the teacher network is realized, thereby enhancing the model's adaptability to modality missing conditions.

[0060] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0061] like Figure 1 As shown, the incomplete multimodal learning method based on cue distillation of the present invention includes the following steps:

[0062] S1. Construct and preprocess the multimodal dataset, simulate the missing modalities by randomly masking some modalities, and divide it into training and test sets; specifically including the following steps:

[0063] S11. For image modality, adjust the shorter side of the image to 384 pixels while maintaining the aspect ratio, ensuring that the longer side does not exceed 640 pixels; when handling image missing cases, use virtual input with all pixel values ​​set to 1; for text modality, use the bert-base-uncased tokenizer to segment the input text; when handling text missing cases, use an empty string as virtual input.

[0064] S12. Set the missing rate ξ% of the multimodal dataset to 70%, generate missing modality data by randomly masking the modalities of samples, and divide the dataset into training and testing sets; the missing rate is the proportion of incomplete modality data in the entire dataset, including both training and testing phases, and is defined as:

[0065]

[0066] Where M represents the total number of modalities, N represents the total number of samples, and m k Let m represent the number of available modes for the k-th sample, and m k A value ≥1 indicates that at least one modality is available for each sample;

[0067] S13. Input the multimodal data of the training set, including the missing modal data, into the trained multimodal encoder to obtain meaningful modal embeddings; specifically, use a pre-trained visual transformer (ViT) as an image modal encoder, a pre-trained BERT language model as a text modal encoder, and a pre-trained Whisper speech recognition model as an audio modal encoder.

[0068] S2. Based on the prompt distillation mechanism, a hierarchical prompt generator is designed to generate modality-level prompts and instance-level prompts respectively. Through intrinsic modality-shared knowledge distillation and task-specific knowledge distillation, an efficient knowledge transfer mechanism is established between the teacher network (complete modality data) and the inference network (incomplete modality data). Specifically, the hierarchical prompt generator is responsible for generating two types of adaptive prompts: modality-level prompts and instance-level prompts, targeting the teacher network and the inference network respectively. The modality-level prompts inform the teacher network of missing modalities, inducing the teacher model to generate specific modality knowledge to guide the inference network. The instance-level prompts help the inference network capture instance-specific knowledge for each input sample. The specific steps include:

[0069] S21. For the modal-level cue generator, through the modal embedding described in step S1, a corresponding cue is calculated for each modality missing case to obtain the modal-level cue. Among them, P t The subscript 't' indicates modal-level cues used in the teacher network; M indicates the total number of modalities; 2 M -1 covers all possible missing modalities; during training, modal-level cue P t It is randomly initialized; then P t Through the affinity projection function F composed of multiple linear layers aff Receive optimized modal-level hints Right now Calculate the affinity classification loss L acm :

[0070]

[0071] Among them, F cls (·) indicates the classification layer that produces the M-dimensional distribution scores; Let $\mathbf{x}$ represent the target affinity label, where $0$ indicates the missing modality; $BCE(·)$ represents the binary classification loss; $x$ represents the input sample data, and $i$ is the data index for all possible missing modalities; $D$ represents all data in the dataset; specifically, the dataset $D$ is divided into two subsets: $D = {D_{i}}$. c D s}, where D c Data containing complete modalities, D s Includes data with missing modalities;

[0072] The affinity projection function is used to optimize modal-level cues; specifically, cases with different missing modalities may share the same modality; P t The prompts in the text are related, and this relationship can be learned through affinity projection functions;

[0073] S22. For instance-level prompt generators, the modal embedding is denoted as...

[0074]

[0075] Where 1≤n≤M represents the modal index, M represents the total number of modes; ε(·) represents the encoder operation; x i Represent the input sample; compute unique instance-level hints for each instance.

[0076]

[0077] Where G(·) represents an instance-level hint generator, implemented through a multilayer perceptron; [] represents a join operation; The subscript 's' indicates instance-level hints used in the inference network; specifically, the embedding e of the missing modality. k Unable to obtain, directly replace e with a virtual embedding filled with 0. k The modal-level cue information obtained in step 21 is then processed using a cross-attention mechanism. Inject instance-level hints; specifically, inject instance-level hints. Treat it as a query, modal-level hints Treated as keys and values, allowing and Perform cross-attention calculations to obtain instance-level cues.

[0078]

[0079] Among them, W q W k W v Indicates a transition layer; This represents the modal level cue corresponding to the k-th sample; d represents the dimension of the [·] vector;

[0080] S23, the obtained modal-level hints Inserting it into the teacher network will yield instance-level hints. Inserted into the inference network; specifically, both the teacher network and the inference network are built using a pre-trained ViLT (Kim W, Son B, Kim I. ViLT: Vision-and-Language Transformer Without Convolution or RegionSupervision[J].2021.DOI:10.48550) as the backbone network; the ViLT pre-trained model includes an input layer and a transformer encoder layer, the transformer encoder layer includes an attention layer; cues are inserted into two positions in the ViLT pre-trained model: the input layer and the attention layer, where both the input layer and the attention layer are multi-layered structures. The overall structure of the ViLT pre-trained model is as follows: Figure 2 As shown; input layer cues are cues appended to the input sequence of each layer:

[0081]

[0082] in, This represents the operation of the first layer of the neural network, Encoder. l This represents the first layer of a teacher network or inference network; [;] represents a join operation; P l This indicates a modal or instance-level hint inserted into level 1; h lThis represents the hidden input of layer 1; parameter N represents the number of network layers to insert the cue.

[0083] Similarly, for attention-layer cues, the cue P will be... l The cue vectors are evenly divided into two sub-cues along the dimension of quantity, and then attached to the key and value projection layers of the multi-head self-attention (MSA) mechanism, respectively:

[0084]

[0085] Among them, ATTENTION l P represents the MSA of the l-th layer; i Divided into two sub-hints P k and P v Attached to key K of MSA respectively l Sum V l Projection layer;

[0086] S24. Train the teacher network using a small amount of complete modality data and train the inference network using data containing a large amount of incomplete modality data; through cue distillation, distill the inherent modality sharing and task-specific knowledge in the teacher model into the inference network; specifically, it includes two parts: inherent modality sharing knowledge distillation and task-specific knowledge distillation.

[0087] The goal of intrinsic modality shared knowledge distillation is to align sample features in the teacher network with corresponding missing modality sample features in the inference network, expressed as:

[0088]

[0089] in, and These represent the last layer of the teacher and the inference network, respectively, generating two hidden features; and Indicates the corresponding modal and instance-level hints; x i Represents a complete modal data point. x represents i The corresponding incomplete modal data points are obtained by randomly masking x. i One of the modes is obtained φ(·) is a measurement function implemented using Euclidean distance;

[0090] The goal of task-specific knowledge distillation is to make the downstream classification scores of complete modal data points generated by the teacher model similar to the scores of corresponding incomplete modal samples in the inference model, expressed as:

[0091]

[0092] Among them, Encoder tand Encoder i The teacher network and the inference network are represented respectively, generating scores for downstream classification tasks; and Indicates the corresponding modal and instance-level hints; x i Represents a complete modal data point. x represents i The corresponding incomplete modal data points are obtained by randomly masking x. i One of the modes is obtained D KL (·) denotes the Kullback-Leibler (KL) divergence measure function for calculating the distance between fractional distributions;

[0093] S25. Freeze all parameters of the ViLT backbone network in the teacher and inference networks, and train only the parameters related to downstream tasks and learnable cues. The training process is as follows: Figure 3 As shown; specifically, the training objective of this invention is achieved by minimizing the training loss, wherein the training loss comprises three parts: L acm L ikd and L tkd These are respectively used for learning modal-level cues in step S21, distilling intrinsic modal shared knowledge in step S24, and distilling task-specific knowledge; cross-entropy classification loss is used as the downstream task loss L for inference and the teacher network. cls The overall loss L is obtained:

[0094] L = L cls +αL acm +λ(L ikd +L tkd )

[0095] Here, α and λ are two hyperparameters used to adjust the importance of the corresponding parts during training. Specifically, the length of the learnable cue is set to 16, and the number of layers into which the cue is injected is 5; the hyperparameters α and λ are each set to 10. -1 and 5×10 -4 Using the Adam optimizer, the base learning rate is set to 1×10⁻⁶. -2 The weight decay is set to 2×10. -2 .

[0096] S3. Utilizing the trained prompt generator and inference network to infer from incomplete modal samples significantly improves the accuracy and adaptability of downstream tasks; the inference process flow is as follows: Figure 4 As shown;

[0097] S31. Preprocess the missing modal data of the task to be predicted; input it into the trained multimodal encoder to obtain meaningful modal embeddings.

[0098] S32. The obtained modal embedding is fed into the hierarchical cue generator to obtain modal-level cue and instance-level cue for the corresponding sample;

[0099] S33. Input the missing modality sample embeddings and the generated modality-level and instance-level prompts into the inference network to calculate the prediction results of the downstream task; wherein, the modality-level and instance-level prompts are inserted into the input layer and attention layer of the pre-trained model ViLT.

[0100] The incomplete multimodal learning method based on cue distillation proposed in this application was tested on the public datasets MM-IMDb, UPMC Food-101, and Hateful Memes, with a default modality missing rate of 70%. The experimental results are shown in Table 1. It can be seen that the proposed method exhibits superior performance under all settings, which fully demonstrates the effectiveness of the proposed method.

[0101] Table 1:

[0102]

[0103]

[0104] In this dataset, L, V, and A represent text, visual, and audio modalities, respectively. The MM-IMDb dataset uses Macro-F1 as the evaluation metric, the UPMC Food-101 dataset uses classification accuracy as the evaluation metric, and HatefulMemes uses the Area Under the Receiver Operating Characteristic (AUROC) curve as the evaluation metric. Finetuned VILT is based on the paper: Vilt: Vision-and-language transformer without convolution or region supervision. Finetuned Visual BERT is based on the paper: Visualbert: A simple and performant baseline for vision and language. MPVR is based on the paper: Multimodal prompting with missing modalities for visual recognition. Ma Model is based on the paper: Are multimodal transformers robust to missing modality. MMIN is based on the paper: Missing modality imagination network for emotion recognition with uncertain missing modalities.

[0105] Meanwhile, ablation experiments were conducted on the public datasets MM-IMDb, UPMC Food-101, and Hateful Memes to test each loss function in the cue-based distillation-based incomplete multimodal learning method, with a default modality missing rate of 70%. The experimental results are shown in Table 2, demonstrating that each loss function contributes to the model's performance.

[0106] Table 2:

[0107] Dataset MM-IMDb Food101 Hateful Memes (i)None 42.66 79.08 66.07 (ii)w / o Lacm 37.05 71.64 62.51 (iii)w / o Likd 38.49 77.54 65.84 (iv)w / o Ltkd 42.87 80.53 67.62 (v)Ours 44.78 80.92 68.40

[0108] Among them, (i) only the classification loss of the downstream task is retained, and the affinity classification loss and knowledge distillation loss are removed; (ii) the affinity classification loss is removed; (iii) only the intrinsic modality shared knowledge distillation loss is used; (iv) only the task-specific knowledge distillation loss is used; and (v) all loss functions are fused at the same time.

[0109] This application conducted ablation experiments on the publicly available datasets MM-IMDb, UPMC Food-101, and Hateful Memes to test the modality-level and instance-level hints proposed in the cue-based distillation incomplete multimodal learning method. With an overall missing rate of 30%, three different missing rate scenarios were tested for each modality on each dataset. The experimental results are shown in Tables 3 (modality-level hints) and 4 (instance-level hints). It can be seen that removing either modality-level or instance-level hints degrades the model's performance on the test set, further validating the effectiveness of this approach.

[0110] Table 3:

[0111]

[0112] Table 4:

[0113]

[0114]

[0115] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. An incomplete multimodal learning method based on cue distillation, characterized in that, Includes the following steps: Step 1: Construct and preprocess the multimodal dataset, simulate the missing cases by randomly masking some modalities, and divide it into training and test sets; Step 2: Design a hierarchical prompt generator to generate modal-level prompts and instance-level prompts respectively. The modal-level prompts inform the teacher of the missing modalities in the network and induce the teacher's network to generate knowledge of specific modalities to guide the reasoning network. The instance-level prompts help the inference network capture instance-specific knowledge for each input sample; based on the prompt distillation mechanism, intrinsic modality shared knowledge distillation and task-specific knowledge distillation are used to train the teacher network using complete modality data and the inference network using incomplete modality data. Step 3: Use the trained hierarchical prompt generator and inference network to perform inference on incomplete modal samples.

2. The incomplete multimodal learning method based on cue distillation according to claim 1, characterized in that, Step 2 is described in detail below: Step 21: First, encode the missing modal data to obtain modal embeddings; for the modal-level cue generator, calculate a corresponding cue P for each missing modal case. t Then, modal-level cues are obtained through affinity projection functions. Right now Where M represents the total number of modes, 2 M -1 covers all possible missing modalities; Step 22: For the instance-level hint generator, denote the modal embedding as... Where 1≤n≤M represents the modal index, and the hints for each instance are calculated. Where G(·) represents an instance-level hint generator, implemented through a multilayer perceptron; [] represents a join operation; Then, instance-level prompts are calculated using the cross-attention mechanism. Among them, W q W k W v Indicates a transition layer; This represents the modal level cue corresponding to the k-th sample; d represents the dimension of the [·] vector; Step 23: Obtain the modal-level hints Inserting it into the teacher network will yield instance-level hints. Inserted into the inference network; both the teacher network and the inference network use the ViLT pre-trained model, which includes an input layer and a transformer encoder layer, and the transformer encoder layer includes an attention layer; the cue is inserted into the input layer and attention layer of the ViLT pre-trained model, where both the input layer and attention layer are multi-layered structures; Input layer hints are added to the input sequence of each layer: in, This represents the operation of the first layer of the neural network, Encoder. l This represents the first layer of a teacher network or inference network; [;] represents a join operation; P l This indicates a modal or instance-level hint inserted into level 1; h l This represents the hidden input of layer 1; parameter N represents the number of network layers to insert the cue. For attention-level cues, the cue P will be... l The cue vectors are evenly divided into two sub-cues along the dimension of quantity, and then attached to the key and value projection layers of the multi-head self-attention MSA mechanism, respectively: Among them, ATTENTION l P represents the MSA of the l-th layer; i Divided into two sub-hints P k and P v Attached to key K of MSA respectively l Sum V l Projection layer; Step 24: Train the teacher network and the inference network. The formula for the overall loss L is as follows: L=L cls +αL acm +λ(L ikd +L tkd ) Where α and λ are two hyperparameters; L cls For the downstream task loss based on cross-entropy classification loss, L acm Affinity classification loss for modality-level prompts; L ikd The feature alignment loss aims to align sample features in the teacher network with corresponding missing modality sample features in the inference network; L tkd The downstream classification score loss aims to make the downstream classification scores of complete modal data points generated by the teacher model similar to the scores of corresponding incomplete modal samples in the inference model.

3. The incomplete multimodal learning method based on cue distillation according to claim 2, characterized in that, The specific affinity classification loss is as follows: Among them, F cls (·) indicates the classification layer that produces the M-dimensional distribution scores; The target affinity label is represented, and 0 indicates that the corresponding modality is missing; BCE(·) represents the binary classification loss; x represents the input sample data, and i is the data index of all possible missing modalities; D represents all data in the dataset; The feature alignment loss is as follows: in, and These represent the last layer of the teacher and the inference network, respectively, generating two hidden features; and Indicates the corresponding modal and instance-level hints; x i Represents a complete modal data point. x represents i The corresponding incomplete modal data points are obtained by randomly masking x. i One of the modes is obtained φ(·) is a measurement function; The downstream classification score loss is as follows: Among them, Encoder t and Encoder i Representing the teacher network and inference network respectively, they generate scores for downstream classification tasks; D KL (·) denotes the KL divergence measurement function.

4. The incomplete multimodal learning method based on cue distillation according to claim 3, characterized in that, During training, all parameters of the ViLT pre-trained model in both the teacher and inference networks are frozen, and only the parameters relevant to the downstream task and learnable cues are trained.

5. The incomplete multimodal learning method based on cue distillation according to claim 4, characterized in that, Step 1 is described in detail as follows: Step 11: For image modalities, adjust the image pixels while maintaining the aspect ratio. When handling image missing cases, use virtual input with all pixel values ​​set to 1. For text modalities, a tokenizer is used to segment the input text; when handling missing text, an empty string is used as virtual input. Step 12: Set the missing rate ξ% for the multimodal dataset. Generate missing modality data by randomly masking the modalities of samples, and divide the dataset into training and testing sets. The missing rate is the proportion of incomplete modality data in the entire dataset, including both training and testing phases. The missing rate ξ% is defined as: Where M represents the total number of modalities, N represents the total number of samples, and m k Let m represent the number of available modes for the k-th sample, and m k ≧1; Step 13: Input the multimodal data, including the missing modal data, into the trained multimodal encoder to obtain the corresponding modal embeddings.

6. The incomplete multimodal learning method based on cue distillation according to claim 5, characterized in that, The reasoning process in step 3 is as follows: Step 31: Preprocess the missing modal data of the task to be predicted; input it into the trained multimodal encoder to obtain modal embeddings; Step 32: Input the obtained modality embedding into the hierarchical cue generator to obtain the modality-level cue and instance-level cue for the corresponding sample; Step 33: Input the missing modality sample embeddings and the generated modality-level and instance-level hints into the inference network to calculate the prediction results for the downstream task.

Images