A multimodal prompt learning method and system for modality missing problem

Abstract

The present invention discloses a multimodal prompt learning method and system for the problem of modality missing. The multimodal prompt learning method of the present invention includes: completing pre-training of a multimodal model on a high-resource dataset; restoring missing modal features through a missing modality generation module for incomplete modal data on a low-resource dataset; distinguishing between the true modality and the restored missing modality by merging signal missing prompts; modeling the correlation between multiple missing modalities by adding type missing prompts; inferring the complete modality data in the pre-trained model, and updating the missing modality generation module parameters, signal missing prompt parameters, and type missing prompt parameters. The present invention uses a small amount of learnable prompts to guide the available modalities to restore the features of the missing modality, thereby enhancing the reliability of the multimodal pre-training model when processing incomplete modal data.

Description

Technical Field

[0001] The present invention relates to the field of multimodal learning technology, and in particular to a multimodal prompt learning method and system for modality missing problems. Background Art

[0002] Modeling and mining multimodal data holds significant significance and enormous potential. However, in practical applications, incomplete modal information often occurs due to factors such as device failure, data loss or corruption, and privacy concerns, particularly in resource-constrained environments. Because most multimodal models are trained on complete datasets, this modality gap severely limits the application performance of existing multimodal models, making them difficult to meet practical needs.

[0003] Traditional solutions tend to employ complex structures to address the modality loss problem. These approaches typically do not rely on pre-trained models and therefore consume significant computational resources. In contrast, recent developments in cue learning techniques have provided new approaches to addressing this problem. This approach focuses on maintaining the parameters of the pre-trained model and fine-tuning only small-scale task-specific cues, resulting in significantly improved model performance at a relatively low cost. Summary of the Invention

[0004] Based on the above background technology, the present invention proposes a multimodal prompt learning method and system for the problem of modal missingness, which uses a high-quality dataset with relatively complete modal information for pre-training, and uses a small number of trainable prompts to achieve effective migration of modal data from high-resource fields to low-resource fields (i.e., fields where modal missingness is frequent).

[0005] To this end, the technical solution adopted by the present invention is as follows: a multimodal prompt learning method for the modality missing problem, comprising:

[0006] Step 1) Pre-train the multimodal model on a high-resource dataset to obtain a pre-trained model.

[0007] Step 2) For incomplete modal data on low-resource datasets, the missing modal features are restored through the missing modal generation module;

[0008] Step 3) Incorporating signal missing cues to distinguish the true modality from the recovered missing modality;

[0009] Step 4) Model the correlation of multiple missing modalities by adding additional missing type prompts;

[0010] In step 5, the complete modal data is inferred in the pre-trained model, and the missing modal generation module parameters, signal missing prompt parameters, and type missing prompt parameters are updated.

[0011] This paper aims to address the problem of missing modalities in multimodal learning through efficient prompt fine-tuning. By introducing three unique prompt mechanisms (generative prompts, signal-missing prompts, and type-missing prompts), simply fine-tuning these learnable prompts can guide existing modalities to generate fine-grained representations of the missing modality. This enables multimodal pre-trained models to produce accurate results even when faced with missing modal data, enhancing their ability to handle missing modalities in real-world scenarios.

[0012] Furthermore, the step 1) is specifically as follows: pre-training the multimodal model on a high-resource dataset containing complete modal data to ensure that the multimodal model has rich multimodal knowledge and good generalization ability.

[0013] Furthermore, in step 2), the process of restoring the missing modal features is as follows:

[0014] 2.1) The batch data containing different modal missing cases are fed into the missing modality generation module to obtain the generated features. For multimodal samples , set the corresponding generative prompt parameters ,

[0015] in, Indicates the audio mode, Indicates video mode, Indicates text mode;

[0016] 2.2) If the audio mode is missing, the missing mode generation module is used to recover it as follows: ,

[0017] in, represents the restored audio missing modal features, Indicates the restored audio missing mode;

[0018] 2.3) If both audio and video modalities are missing, the missing modality generation module performs the following recovery: 、 ,

[0019] in, Indicates the missing modal features of the restored video, Indicates the restored video missing mode, represents a convolution block consisting of a one-dimensional convolution layer and an activation function, Indicates a transition from one or two modes to another.

[0020] Furthermore, in step 3), the specific process of merging the signal missing prompt is as follows:

[0021] 3.1) Obtaining samples of the missing original audio modality through the missing modality generation module ;

[0022] 3.2) Merge corresponding signal missing prompts: , , ,in, Indicates that the mode is missing, Indicates that the modality is complete.

[0023] Furthermore, in step 4), the specific process of adding a type missing prompt is as follows:

[0024] 4.1) For samples with missing audio and text modalities , whose type is missing projection matrix ,

[0025] in, 、 、 They represent the audio missing projection matrix, video missing projection matrix, and text missing projection matrix respectively;

[0026] 4.2) Obtain the corresponding type missing hint through the type missing projection matrix ,in, Indicates the type missing prompt parameter, which notifies the multimodal model of the missing status of other modalities.

[0027] Furthermore, the specific process of step 5) is as follows:

[0028] 5.1) Data with missing modalities is generated through the missing modality generation module to obtain complete modality data. This data is then passed to a pre-trained model with signal missing and type missing prompts for inference, and the final result is obtained through the output layer.

[0029] 5.2) Freeze the pre-trained model parameters during backpropagation and optimize a small number of learnable hint parameters, Conv layer and output layer by gradient descent method. The hint parameters are the missing modality generation module parameters. 、 、 , signal missing prompt parameter 、 、 、 、 、 , and type missing hint parameters 、 、 、 .

[0030] The present invention also provides a multimodal prompt learning system for the modality missing problem, which is used to implement the above-mentioned multimodal prompt learning method, and includes:

[0031] Model pre-training unit: completes the pre-training of multimodal models on high-resource datasets to obtain pre-trained models;

[0032] Missing modality generation unit: For incomplete modality data on low-resource datasets, the missing modality features are restored through the missing modality generation module;

[0033] Signal loss hint merging unit: distinguishes the true mode from the restored missing mode by merging signal loss hints;

[0034] Type-missing cue additional unit: by adding type-missing cues to model the correlation of multiple missing modalities;

[0035] Prompt parameter update unit: Infers the complete modal data in the pre-trained model and updates the missing modal generation module parameters, signal missing prompt parameters, and type missing prompt parameters.

[0036] The beneficial effects of the present invention are as follows: The present invention is a multimodal prompt learning method for the problem of missing modalities, which introduces three prompt mechanisms: generative prompts, signal missing prompts, and type missing prompts, wherein the signal missing prompts are modality-specific, and the type missing prompts are modality-shared, which respectively help the model learn the relationship within and between modalities, and solve the problem of missing modalities through prompt learning. The method of the present invention reduces the dependence on complex system architecture, and the number of parameters required for prompts is linearly related to the number of modalities, significantly reducing the number of trainable parameters. Under different model architectures, the method proposed in the present invention can accurately restore the missing modal features, and its performance significantly exceeds that of existing baseline methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of a multimodal prompt learning method for the modality missing problem of the present invention;

[0038] Figure 2 A schematic diagram of a multimodal prompt learning method for the modality missing problem according to the present invention;

[0039] Figure 3 This is a composition diagram of a multimodal prompt learning system for the modality missing problem of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific implementation methods.

[0041] Example 1

[0042] like Figure 1 As shown, the present invention is a multimodal prompt learning method for the modality missing problem, comprising the following steps:

[0043] Step 1) Complete the pre-training of the multimodal model on a high-resource dataset to obtain a pre-trained model.

[0044] Specifically: pre-train the multimodal model on a high-resource dataset containing complete modal data to ensure that the multimodal model has rich multimodal knowledge and good generalization ability.

[0045] Step 2) For incomplete modal data on low-resource datasets, the missing modal features are restored through the missing modal generation module.

[0046] First, batches of data containing missing data of different modalities are fed into the missing modality generation module to obtain generated features, which contain samples of three modalities (video, audio and text). ;

[0047] Then, set the learnable generative hint parameters ;

[0048] Finally, the incomplete modal data are input into the missing modal generation module to recover the missing modal features.

[0049] If a single mode is missing (for example, audio), the missing mode generation module is used to recover the missing mode as follows: If both modalities are missing (for example, video and audio are missing), the missing modality generation module is used to recover them as follows: , ,in represents a convolution block consisting of a one-dimensional convolution layer and an activation function, Indicates a transition from one or two modes to another.

[0050] represents the restored audio missing modal features, Indicates the restored audio missing mode; Indicates the missing modal features of the restored video, Indicates the restored video missing modality.

[0051] Step 3) Incorporate signal missing cues to distinguish the true mode from the recovered missing mode.

[0052] First, obtain the sample after audio modality recovery ;

[0053] Then, merge the corresponding signal loss prompts: , , ,in Indicates that the mode is missing, Indicates that the modality is complete.

[0054] Step 4) Model the correlation of multiple missing modes by adding additional type missing prompts.

[0055] First, for samples with missing audio and text modalities , get the type missing projection matrix ;

[0056] Then, through the type missing projection matrix, the corresponding type missing prompt is obtained ,in, Indicates the type missing prompt parameter, which notifies the multimodal model of the missing status of other modalities.

[0057] Step 5) Infer the complete modal data in the pre-trained model and update the missing modal generation module parameters, signal missing prompt parameters, and type missing prompt parameters.

[0058] First, the complete modality data obtained according to the above steps is passed to the pre-trained model with signal missing prompts and type missing prompts to perform inference, and finally the result is obtained through the output layer;

[0059] Then, the pre-trained model parameters are frozen during the back-propagation process, and a small number of learnable prompt parameters, Conv layers, and output layers are optimized by gradient descent, such as Figure 2 As shown, Figure 2 The Transformer in [1] is a network module based on the self-attention mechanism. It processes sequence data in parallel and models long-distance dependencies. It is widely used in feature extraction and fusion of various modal data. The prompt parameters are the missing modality generation module parameters. 、 、 , signal missing prompt parameter 、 、 、 、 、 , and type missing hint parameters 、 、 、 .

[0060] Compared with existing methods for dealing with modality loss, the present invention can reduce dependence on complex system architectures, and the number of required prompt parameters is linearly related to the number of modalities, which significantly improves the efficiency and stability of the model.

[0061] To simulate real-world scenarios, the training process of this paper uses CMU-MOSEI as a high-resource dataset and CMU-MOSI as a low-resource dataset. The model is pre-trained on CMU-MOSEI, and the method of this paper is evaluated on the CMU-MOSI dataset, verifying its significant effectiveness. CMU-MOSI is a popular multimodal (audio, text, and video) sentiment analysis dataset consisting of 93 English YouTube clips, each manually annotated with a sentiment score ranging from strongly negative to strongly positive (-3 to +3). CMU-MOSEI is an extension of CMU-MOSI, containing over 65 hours of annotated video from over 1,000 speakers and 250 topics, covering a wider range of topics than CMU-MOSI.

[0062] This paper uses an objective evaluation method to evaluate the performance of the model trained by the proposed method. For CMU-MOSI and CMU-MOSEI, 7-level accuracy (ACC7), binary accuracy (ACC), F1 score (F1), mean absolute error (MAE), and Pearson correlation (Corr) are used as evaluation indicators.

[0063] Table 1 shows the quantitative results of our method for six possible modality-missing scenarios. For example, "{a}" indicates that the audio modality is available, while video and text are missing. "Average" refers to the average performance across the six possible scenarios. Bold indicates the best results. Note: All results are averaged across five different random seeds.

[0064] Table 1: Quantitative results of six possible modality missing cases on the CMU-MOSI dataset

[0065]

[0066] Results show that our method significantly outperforms the baseline across all six missing modalities. Furthermore, when the text modality is missing, our method delivers significant gains, improving accuracy by 8-13% compared to the LB baseline. This demonstrates that our three proposed prompt types effectively guide the pre-trained model and yield significant performance improvements.

[0067] Example 2

[0068] This embodiment provides a multimodal prompt learning system for the modality missing problem, which is used to implement the multimodal prompt learning method described in Example 1. Figure 3 As shown in the figure, it consists of a model pre-training unit, a missing modality generation unit, a signal missing prompt merging unit, a type missing prompt appending unit and a prompt parameter updating unit.

[0069] Model pre-training unit: completes the pre-training of multimodal models on high-resource datasets to obtain pre-trained models;

[0070] Missing modality generation unit: For incomplete modality data on low-resource datasets, the missing modality features are restored through the missing modality generation module;

[0071] Signal loss hint merging unit: distinguishes the true mode from the restored missing mode by merging signal loss hints;

[0072] Type-missing cue additional unit: by adding type-missing cues to model the correlation of multiple missing modalities;

[0073] Prompt parameter update unit: Infers the complete modal data in the pre-trained model and updates the missing modal generation module parameters, signal missing prompt parameters, and type missing prompt parameters.

[0074] It should be noted that each unit in the above-mentioned multimodal prompt learning system can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned units. For the specific definition of a multimodal prompt learning system, please refer to the definition of a multimodal prompt learning method (i.e., Example 1) above. The two have the same functions and effects and will not be repeated here.

[0075] It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without requiring creative effort. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multimodal prompt learning method for the modality missing problem, characterized by: include: Step 1) Pre-train the multimodal model on a high-resource dataset to obtain a pre-trained model. Step 2) For incomplete modal data on low-resource datasets, the missing modal features are restored through the missing modal generation module; Step 3) Incorporating signal missing cues to distinguish the true modality from the recovered missing modality; Step 4) Model the correlation of multiple missing modalities by adding additional missing type prompts; Step 5) Infer the complete modal data in the pre-trained model and update the missing modal generation module parameters, signal missing prompt parameters, and type missing prompt parameters; In step 2), the process of restoring the missing modal features is as follows: 2.1) The batch data containing different modal missing cases are fed into the missing modality generation module to obtain the generated features. For multimodal samples , set the corresponding generative prompt parameters ， in, Indicates the audio mode, Indicates video mode, Indicates text mode; 2.2) If the audio mode is missing, the missing mode generation module is used to recover it as follows: ， in, represents the restored audio missing modal features, Indicates the restored audio missing mode; 2.3) If both audio and video modalities are missing, the missing modality generation module performs the following recovery: 、 ， in, Indicates the missing modal features of the restored video, Indicates the restored video missing mode, represents a convolution block consisting of a one-dimensional convolution layer and an activation function, Indicates a transition from one or two modes to another.

2. The multimodal prompt learning method according to claim 1, characterized in that: The step 1) is specifically: pre-training the multimodal model on a high-resource dataset containing complete modal data to ensure that the multimodal model has rich multimodal knowledge and good generalization ability.

3. The multimodal prompt learning method according to claim 1, characterized in that: In step 3), the specific process of merging the signal loss prompt includes: 3.1) Obtaining samples of the missing original audio modality through the missing modality generation module ; 3.2) Merge corresponding signal missing prompts: , , ,in, Indicates that the mode is missing, Indicates that the modality is complete.

4. The multimodal prompt learning method according to claim 3, characterized in that: In step 4), the specific process of adding a type missing prompt includes: 4.1) For samples with missing audio and text modalities , whose type is missing projection matrix ,in, 、 、 They represent the audio missing projection matrix, video missing projection matrix, and text missing projection matrix respectively; 4.2) Obtain the corresponding type missing hint through the type missing projection matrix ,in, Indicates the type missing prompt parameter, which notifies the multimodal model of the missing status of other modalities.

5. The multimodal prompt learning method according to claim 4, characterized in that: The specific process of step 5) is as follows: 5.1) Data with missing modalities is generated through the missing modality generation module to obtain complete modality data. This data is then passed to a pre-trained model with signal missing and type missing prompts for inference, and the final result is obtained through the output layer. 5.2) Freeze the pre-trained model parameters during backpropagation and optimize a small number of learnable hint parameters, Conv layer and output layer by gradient descent method. The hint parameters are the missing modality generation module parameters. 、 、 , signal missing prompt parameter 、 、 、 、 、 , and type missing hint parameters 、 、 、 .

6. A multimodal prompt learning system for modality missing problem, used to implement the multimodal prompt learning method according to any one of claims 1 to 5, characterized in that: include: Model pre-training unit: completes the pre-training of multimodal models on high-resource datasets to obtain pre-trained models; Missing modality generation unit: For incomplete modality data on low-resource datasets, the missing modality features are restored through the missing modality generation module; Signal loss hint merging unit: distinguishes the true mode from the restored missing mode by merging signal loss hints; Type-missing cue additional unit: by adding type-missing cues to model the correlation of multiple missing modalities; Prompt parameter update unit: Infers the complete modal data in the pre-trained model and updates the missing modal generation module parameters, signal missing prompt parameters, and type missing prompt parameters.

Images