An incomplete multi-modal learning method based on hierarchical hints and knowledge distillation

Abstract

This invention proposes an incomplete multimodal learning method based on hierarchical prompts and knowledge distillation, belonging to the field of artificial intelligence technology. The method includes: acquiring and preprocessing a multimodal sample dataset; extracting modal features of different modalities using a multimodal encoder to obtain modality alignment features; constructing and calling a modality prompt generator to generate affinity modality-level prompts, and training a teacher network on a small number of complete modality samples; inputting the modality alignment features into an instance prompt generator to generate instance-level prompts, and training a student network on a large number of missing modality samples; and fine-tuning the teacher and student networks based on a triple knowledge distillation mechanism. This invention possesses stronger adaptability and generalization capabilities. By guiding student network learning through the training of the teacher network on complete data, it can better handle modality missing situations; simultaneously, it eliminates the need for separate modeling of each missing combination, significantly improving the system's scalability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and relates to multimodal learning, knowledge distillation, and cue learning. In particular, it is an incomplete multimodal learning method that improves model performance through cue-driven and hierarchical knowledge distillation under conditions of missing modal information. Specifically, it relates to an incomplete multimodal learning method based on hierarchical cueing and knowledge distillation. Background Technology

[0002] Multimodal learning is a technique that simultaneously utilizes data from different modalities (such as images, text, and audio) for modeling and inference. It effectively integrates multi-source information and is widely applied in artificial intelligence tasks such as action recognition, image and text generation, and privacy protection. In recent years, with the development of deep learning, multimodal learning has achieved significant performance improvements over single-modal methods in many tasks. However, the performance of multimodal learning methods is often highly dependent on well-aligned, noise-free, and fully labeled data across modalities. In practical applications, complete modal data is often difficult to obtain. Due to issues such as sensor failure, bandwidth limitations, or hardware costs, many instances only contain partial modal data, resulting in modality gaps. This modality gap problem has become a key challenge restricting the practical application of multimodal learning. Therefore, to achieve effective deployment of multimodal learning models in real-world scenarios, it is urgent to design learning methods capable of handling incomplete modal inputs.

[0003] Existing research mainly addresses the modality missing problem from two directions. One approach is to infer missing modalities from available modalities, with common methods including modality completion, modality generation, or autoencoder modeling. The other approach attempts to train a unified model on mixed data containing both complete and incomplete samples to adapt to input scenarios with different modality combinations. Although the above methods have made some progress, there are still two main shortcomings: (1) The method of training a generative model separately for each modality missing combination is difficult to extend to the combinatorial explosion problem in real complex scenarios, and the computational cost is high; (2) When jointly training the model, the hierarchical knowledge between different modalities is often ignored, resulting in the model failing to fully capture the potential differences in information under different missing combinations.

[0004] The key challenge lies in how to fully utilize the limited number of complete modality samples available to improve learning performance on incomplete data. To address this, cue learning, as an efficient and transferable learning paradigm, has demonstrated significant performance in visual and language tasks in recent years. Cue learning can achieve rapid adaptation to specific tasks by fine-tuning a small number of parameters, avoiding the cost of overall model fine-tuning required in traditional methods. Summary of the Invention

[0005] The purpose of this invention is to provide an incomplete multimodal learning method based on hierarchical prompts and knowledge distillation, so as to solve the technical problems of performance degradation, low knowledge transfer efficiency and lack of unified processing strategy when dealing with incomplete modal data in the prior art.

[0006] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0007] An incomplete multimodal learning method based on hierarchical prompts and knowledge distillation is proposed. The method includes two parts: a training process and a prediction process. The training process includes the following steps:

[0008] Step A1: Obtain the multimodal sample dataset and perform preprocessing;

[0009] Step A2: Use a multimodal encoder to extract modal features of different modalities, and obtain modal alignment features by embedding the modal features into a unified vector space;

[0010] Step A3: Build and call the modal cue generator, generate affinity modal-level cues through affinity consistency constraint learning, and use them to explicitly label the currently missing / existing modalities. Train the teacher network on a small number of complete modal samples.

[0011] Step A4: Input the modality alignment features into the instance cue generator to generate instance-level cues. Use cross-attention to allow the instance cue to read the modality cue information for missing modalities. Train the student network on a large number of missing modality samples.

[0012] Step A5: Based on the triple knowledge distillation mechanism, perform feature alignment, category classification alignment, and cross-modal alignment on the teacher network and student network, and fine-tune the teacher network and student network.

[0013] Further, step A1 includes the following steps:

[0014] Step A11: Obtain A multimodal sample dataset with multiple modalities;

[0015] Step A12: Divide the multimodal sample dataset into a complete modality data subset and a missing modality data subset according to whether the sample has a missing modality;

[0016] Step A13: For the missing modality data subset The samples are filled with placeholder data to make each sample formally satisfy the condition that the number of actual available modalities equals the total number of modalities, thus obtaining a subset of missing modal fill data and standardizing the input.

[0017] Step A14: Construct a modality missing mask for each sample to prompt generation and loss weighting.

[0018] Further, step A2 includes the following steps:

[0019] Step A21: Using an M-type modal encoder, extract the feature representations of each sample modality from the complete modal data subset and the missing modal completion data subset to obtain the modal features; the modal feature representations are as follows:

[0020]

[0021] in, Indicates the first The first sample Modal features; Indicates the first A modal encoder with multiple modes; Indicates the first The sample at the th Input features under each modality;

[0022] Step A22: Using a cross-modal alignable unified vector space, perform two-term alignment on the modal features of samples in both the complete modal data subset and the missing modal imputation data subset to obtain the modal alignment features, as shown below:

[0023]

[0024]

[0025] in, Indicates the first The first sample Modal alignment features; For mapping functions; Indicates the first Modality alignment features of each sample; Represents the feature aggregation function; Indicates the first The first sample Modality missing mask for each modality;

[0026] Modality alignment features include modality alignment features for complete modality samples and modality alignment features for missing modality samples.

[0027] Further, step A3 includes the following steps:

[0028] Step A31: Randomly initialize a learnable modality-level cue vector set. Each cue vector in the modality-level cue vector set corresponds to a modality absence configuration, used to explicitly encode the presence / absence of each modality in the current sample; the modality-level cue vector set is represented as follows: , Indicates the number of combinations of missing modalities. Indicates the total number of modes; Indicates the first Modal-level cue vectors for combinations of missing modalities;

[0029] Step A32: Project the modal cue vector set into an affinity modal cue vector set with affinity information, and learn the affinity modal cue vector set using multi-label classification loss; the affinity modal cue vector set is represented as follows:

[0030]

[0031] in, This represents the set of affinity modality-level cue vectors. , Indicates the first Affinity modality-level cue vectors for combinations of missing modalities; Represents the affinity mapping function;

[0032] The multi-label classification loss is represented as follows:

[0033]

[0034] in, Indicates loss of affinity. It is a classification layer; Indicates the first The target affinity labels are 0 for missing and 1 for present; BCE is a multi-label binary classification loss.

[0035] Step A33: Input the modality alignment features and affinity modality-level cue vector set of complete modality samples into the teacher network, train the teacher network on a small number of complete modality samples, and achieve the downstream task.

[0036] Further, step A33 includes the following steps:

[0037] Step S331: Set the affinity modality-level cue vector set It is directly attached to the encoder of the teacher network;

[0038] Step S332: Input the modality alignment features of complete modality samples to construct the overall input representation relationship of the teacher network;

[0039] Step S333: Train the teacher network on a small number of complete modal samples to achieve the downstream task.

[0040] Further, step A4 includes the following steps:

[0041] Step A41: Concatenate the modality-aligned feature representations of the samples and input them into the instance cue generator to generate instance-level cue vectors, as shown below:

[0042]

[0043] in, Indicates the first Individual sample instance-level cue vectors; Indicates an instance suggestion generator; This is a vector concatenation operation;

[0044] Step A42: The affinity modal-level cue vector and the instance-level cue vector are fused using a cross-attention mechanism to obtain the fused instance-level cue, as shown below:

[0045]

[0046] in, Indicates the first Instance-level cue vectors after sample fusion These are the learnable query projection matrix, the key projection matrix, and the value projection matrix, respectively. Indicates the scaling factor. Indicates the first Affinity modality-level cue vectors for combinations of missing modalities; This represents the normalized activation function; This represents the matrix transpose operation;

[0047] Step A43: Use the modality alignment features of the missing modality samples and the fused instance-level cue vector as input to the student network, train the student network on a large number of missing modality samples, and achieve the downstream task.

[0048] Further, step A5 includes the following steps:

[0049] Step A51: Construct an intrinsic modality shared knowledge distillation loss to align the feature spaces of the teacher network and the student network;

[0050] Step A52: Construct the task knowledge distillation loss, aiming to make the classification distribution of the teacher network for complete modality samples close to the classification distribution of the student network for missing modality samples corresponding to complete modality samples with one missing modality.

[0051] Step A53: Project the final feature representations of the teacher network and student network into teacher network mapping features and student network mapping features, and calculate the cosine similarity to construct the cross-modal shared knowledge distillation loss;

[0052] Step A54: Construct the classification cross-entropy loss and the total loss function for the downstream task.

[0053] Furthermore, the intrinsic modality shared knowledge distillation loss is represented as follows:

[0054]

[0055] in, This represents the loss from distillation of shared knowledge across intrinsic modalities. These represent the last encoder layer of the teacher network and the student network, respectively; Represents the distance metric function; A sample representing a complete mode; Represents complete modal samples The corresponding missing modality sample with one missing modality;

[0056] The task knowledge distillation loss is represented by the KL divergence metric as follows:

[0057]

[0058] in, This indicates the loss from task knowledge distillation. and These represent the downstream task classification heads for the teacher network and the student network, respectively. It is a complete modal sample The corresponding missing modal-level cue vector; It is a divergence measurement function.

[0059] Furthermore, the cross-modal shared knowledge distillation loss is represented as follows:

[0060]

[0061] in, This represents the loss from cross-modal knowledge distillation. Indicates the batch size of samples during training. This indicates the index of the sample currently being calculated. This represents the index used to iterate through all samples in the current batch; Indicates temperature hyperparameter; Indicates the first The network mapping features of the sample teacher and the first Cosine similarity of network mapping features of each sample student; Indicates the first The network mapping features of the sample teacher and the first Cosine similarity of network mapping features of each sample student.

[0062] Furthermore, the classification cross-entropy loss is expressed as follows:

[0063]

[0064] in, Represents the classification cross-entropy loss. Represents the total number of samples. It is the number of categories. It is the first The sample belongs to the first The true label of the class, It is the first The sample belongs to the first The predicted probability of a class;

[0065] The final total loss function is then used for training.

[0066]

[0067] in, Indicates the total loss. To adjust affinity loss Hyperparameters of relative weights; To adjust for knowledge distillation loss Hyperparameters for relative weights.

[0068] Compared with the prior art, the present invention has the following beneficial technical effects:

[0069] (1) This invention solves the combinatorial explosion problem, significantly improving system scalability and training efficiency: Existing technologies often require separate modeling for each modality missing combination, resulting in extremely high computational costs. This invention introduces a modality cue generator, which explicitly encodes the presence and absence states of modalities by constructing affinity modality-level cue vectors; simultaneously, through affinity consistency constraint learning, it fully explores and utilizes the shared correlations between different missing cases. This mechanism effectively avoids the drawback of needing to repeatedly model when facing exponential missing combinations, enabling a single model to flexibly cover various unknown missing cases, significantly improving the system's scalability.

[0070] (2) This invention achieves fine-grained feature compensation, enhancing the robustness of the model under modality loss: This invention not only intervenes at the modality level, but also constructs an instance prompt generator to extract fine-grained semantic information from samples. Through a cross-attention mechanism, affinity-level prompts containing prior knowledge of missing modes are deeply integrated with instance-level prompts. This hierarchical prompt pattern enables the student network to accurately "read in" missing mode guidance when some modalities are missing, and to use the remaining available modality information for targeted feature compensation, thereby significantly improving the model's expressive power and robustness under incomplete input conditions.

[0071] (3) This invention achieves deep alignment based on triple knowledge distillation, maximizing the utilization efficiency of complete data: Addressing the pain point of scarce complete modal samples in real-world scenarios, this invention designs a triple knowledge distillation mechanism encompassing intrinsic modality sharing, task-specificity, and cross-modal sharing. Even with limited complete modal data, the high-quality deep semantic representation extracted by the teacher network can effectively guide the training of the student network. In particular, the cross-modal shared contrastive learning loss constructed by combining the projection head and cosine similarity effectively avoids feature collapse, ensuring accurate knowledge transfer across modal dimensions in the feature space and classification boundaries, and endowing the student network with strong generalization capabilities comparable to training with complete data.

[0072] (4) The computational paradigm of this invention is highly efficient and unified, and can be widely applied to downstream tasks in real and complex scenarios: This invention achieves standardized input of incomplete multimodal data and cross-modal feature alignment through placeholder data completion and unified vector space mapping. The introduction of cue learning avoids the huge computational cost of overall fine-tuning of large models in traditional methods, and achieves rapid adaptation to specific tasks. As a general and efficient underlying framework, this method can be widely connected to various downstream tasks such as multimodal classification and recognition. In particular, it can directly solve the pain point of high-frequency modal loss caused by sensor failure or environmental interference in real industries such as medical diagnosis, intelligent security, and autonomous driving vehicle systems, and has extremely high practical application and promotion value. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a flowchart of the model training process in an embodiment of the present invention.

[0075] Figure 2 This is a diagram of the overall architecture of the model in an embodiment of the present invention.

[0076] Figure 3 This is a flowchart of the model prediction process in an embodiment of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] This invention aims to propose an incomplete multimodal learning method based on hierarchical prompts and knowledge distillation to address the performance degradation caused by modality missingness in multimodal tasks. The method includes a training process and a prediction process. In the training phase, multimodal samples are first divided into complete modalities and missing modalities, and the missing modalities undergo unified completion processing. Subsequently, multimodal features are extracted using a modality-specific encoder, and modality-level prompts are generated by a modality prompt generator to train the teacher network, learning deep semantic representations on complete data. Simultaneously, an instance prompt generator generates instance-level prompts for the missing modal samples, and a cross-attention mechanism is introduced for prompt fusion to train the student network. A triple knowledge distillation strategy aligns the teacher and student networks, achieving collaborative transfer of feature space and classification output. In the prediction phase, utilizing the trained prompt structure and student network, accurate classification or prediction results can still be output even when the input modality is incomplete, demonstrating good robustness and generalization ability.

[0079] The incomplete multimodal learning method based on hierarchical prompting and knowledge distillation proposed in this invention includes two parts: a training process and a prediction process, such as... Figure 1 As shown, the training process includes the following steps:

[0080] Step A1: Obtain the multimodal sample dataset and perform preprocessing.

[0081] Step A11: Obtain A multimodal sample dataset with [number] modalities. Represented as follows:

[0082]

[0083] in, This represents a multimodal sample dataset; Indicates the total number of samples; Indicates the first A multimodal sample; Indicates the first The task label corresponding to each sample; Indicates the total number of modes; Indicates the first The sample at the th Input features under each modality .

[0084] Multimodal includes three modalities: text, image, and audio; the total number of modalities is... The value is 3.

[0085] Step A12: Divide the multimodal sample dataset into complete modality data subsets according to whether the sample has a missing modality. and missing modal data subset ,Right now A subset of complete modal data Used for online teacher training, missing modal data subset Used for online training for students.

[0086] Specifically, the sample The actual number of available modes is denoted as Complete modal data subset The actual number of usable modes in a sample is equal to the total number of modes, i.e., it satisfies Missing modal data subset The actual number of usable modes in the sample is less than the total number of modes, i.e., it satisfies .

[0087] Percentage of complete samples ;in, This represents the number of samples in a subset of the complete modal data. The missing data subset represents the number of samples; the missing rate. Preferred settings: , .

[0088] Step A13: For the missing modality data subset The samples are filled with placeholder data to make each sample formally satisfy the condition that the number of actual available modalities equals the total number of modalities, thus obtaining a subset of missing modal filler data and standardizing the input.

[0089] Specifically, for the subset of missing modal data The samples in Missing modalities are filled with placeholder data to make each sample formally satisfy the requirements. This yields a subset of data for missing modalities; for example, when an image modality is missing, the image is left blank, and the representation after sample completion is still written as: However, some of them This is placeholder data (without valid semantics).

[0090] Step A14: Construct a modality missing mask for each sample Used to suggest generation and loss weighting.

[0091] Modal Missing Mask ; where 1 indicates the presence of a mode and 0 indicates the absence of a mode.

[0092] No. Modality missing rate per sample for:

[0093]

[0094] in, Indicates the first The first sample Modality missing mask for each modality; .

[0095] Step A2: Use a multimodal encoder to extract modal features of different modalities, and obtain modal alignment features by embedding the modal features into a unified vector space.

[0096] Step A21: Use M modal encoders to extract the feature representations of each sample modality from the complete modal data subset and the missing modal completion data subset, and obtain the modal features.

[0097] Image modality samples used the ViT image encoder to extract image features; text modality samples used the BERT text encoder to extract text features; and audio modality samples used the Whisper audio encoder to extract audio features. The modality features are represented as follows:

[0098]

[0099] in, Indicates the first The first sample Modal features, It is a modal index; Indicates the first A modal encoder with multiple modes.

[0100] Step A22: Using a cross-modal alignable unified vector space, perform two-term alignment on the modal features of samples in the complete modal data subset and the missing modal completion data subset respectively to obtain modal aligned features.

[0101] A unified vector space that is alignable across modalities is used to perform two-term alignment on modal features, as shown below:

[0102]

[0103]

[0104] in, Indicates the first The first sample Modal alignment features; The mapping function can be implemented by a linear layer, a two-layer MLP, or a linear layer with a normalized projection head; it is a unified vector space that can be aligned across modalities. Indicates the first The modal alignment feature of the sample represents the _th ... Multimodal aggregated feature representation of a sample in a unified vector space; This represents a feature aggregation function that aggregates the modal features of a sample to generate modality-aligned features.

[0105] Modality alignment features include modality alignment features for complete modality samples and modality alignment features for missing modality samples.

[0106] Step A3: Build and invoke the modal cue generator, generate affinity modal-level cues through affinity consistency constraint learning, and use them to explicitly label the currently missing / existing modalities. Train the teacher network on a small number of complete modal samples.

[0107] Step A31: Randomly initialize a set of learnable modal-level cue vectors. Each cue vector in the set corresponds to a modality missing configuration, which is used to explicitly encode the presence / missing status of each modality in the current sample.

[0108] Construct a modal-level cue vector set for modal missing combinations, denoted as: , Indicates the number of combinations of missing modalities. Indicates the first A modality-level cue vector for a combination of modal missing features corresponds to a modality missing feature configuration.

[0109] In implementation, modal-level prompts are used to explicitly indicate "which / which modalities are missing", guiding teachers to generate modal sharing and modal-specific knowledge online.

[0110] Step A32: Project the modal cue vector set into an affinity modal cue vector set with affinity information, and learn the affinity modal cue vector set through multi-label classification loss.

[0111] There are sharing relationships between different missing modalities (such as the correlation between "present / missing modal labels"). For example, complete modal data shares at least one modality with any other missing modality. Therefore... Hints should not be constructed independently, but should be associated with other hints. To this end, an affinity consistency mechanism is introduced, utilizing affinity mapping functions. The modal cue vector set is projected into an affinity modal cue vector set with affinity information, as shown below:

[0112]

[0113] in, This represents the set of affinity modality-level cue vectors. , Indicates the first Affinity modality-level cue vectors for combinations of missing modalities; This represents the affinity mapping function, implemented by multiple linear layers.

[0114] In order to learn the affinity modal cue vector set A multi-label classification loss was deployed:

[0115]

[0116] in, Indicates loss of affinity. It is a classification layer; Indicates the first The target affinity labels are 0 for missing and 1 for present; BCE is a multi-label binary classification loss.

[0117] Step A33: Input the modality alignment features and affinity modality-level cue vector set of complete modality samples into the teacher network, train the teacher network on a small number of complete modality samples, and achieve the downstream task.

[0118] The teacher network mainly consists of two substructures: the VILT-backbone and the downstream task classification header.

[0119] VILT-backbone is mainly composed of It consists of stacked identical encoder layers. Each layer adopts the standard Transformer encoder architecture and contains a multi-head self-attention layer (MSA) core computational submodule.

[0120] Step S331: Set the affinity modality-level cue vector set It is directly attached to the encoder of the teacher network.

[0121] Specifically, this invention provides two implementations for embedding affinity modality-level cue vector sets into the encoder of a teacher network: input-level cue embedding and attention-level cue embedding.

[0122] ① Input-level cue embedding. The affinity modality-level cue vector set is appended to the input sequence of each encoder layer of the teacher network to fuse the cue knowledge from previous layers. The specific configuration is as follows:

[0123]

[0124] in, Indicates the teacher network The layer encoder uses the output after the input-level cue is embedded; This represents a vector concatenation operation; Indicates the first A hidden layer, Indicates the first One hidden layer; The first term representing the teacher network Layer encoder; Indicates injection into the teacher network. Layer affinity modal cue vectors.

[0125] ② Attention-level cue embedding. The affinity modality-level cue vector set is appended to the multi-head self-attention (MSA) layer of each encoder layer in the teacher network. Specifically, the affinity modality-level cue vector set... Split into key hints Sum value hint Two sub-tips, and key hints Sum value hint The bonds attached to the multi-head self-attention layer respectively ( ) and value ( In the projection layer, its specific configuration is as follows:

[0126]

[0127]

[0128] in, Indicates the teacher network The multi-head self-attention layer in the layer encoder is combined with the output after attention-level cue embedding; Indicates the teacher network Multi-head self-attention layer in a layer encoder; , , They represent the first The query matrix, key matrix, and value matrix of the layer; This is the scaling factor; Indicates injection into the first Layer affinity modal cue vectors; and They represent the first Layer key hints and value hints; Indicates matrix transpose; This represents the normalized activation function.

[0129] Step S332: Input the modality alignment features of the complete modality samples to construct the overall input representation relationship of the teacher network.

[0130] Specifically, in order for the teacher network to simultaneously capture multimodal data features and modality-deficient prior information, the overall forward mapping process of the teacher network is defined as follows:

[0131]

[0132] Alternatively, the construction of the first-level input vector for input-level prompts can be specifically represented as:

[0133]

[0134] in, This represents the deep semantic features or classification prediction results ultimately output by the teacher network. This represents the nonlinear mapping function of a teacher network consisting of multiple encoders; This represents the concatenated input vector sent to the first layer of the teacher network. This indicates the extraction of the first [item] in step A22. one sample A complete modal feature, Represents the feature aggregation function, Indicates the current sample number Affinity modality-level cue vectors for combinations of missing modalities.

[0135] Step S333: Train the teacher network on a small number of complete modal samples to achieve the downstream task.

[0136] For example, the teacher-forward pair: inputs modal alignment features of complete modal samples and affinity modal-level cue vector sets. Output deep characteristics of teachers Teacher classification logits logits This represents the category prediction score vector output by the teacher network given an input sample, i.e., the unnormalized classification logits, where each dimension corresponds to a predefined category.

[0137] Step A4: Input the modality alignment features into the instance cue generator to generate instance-level cues. Use cross-attention to allow the instance cue to read the modality cue information for the missing modalities. Train the student network on a large number of missing modality samples.

[0138] Step A41: Concatenate the modality-aligned feature representations of the samples and input them into the instance cue generator to generate instance-level cue vectors.

[0139]

[0140] in, Indicates the first Each sample instance-level cue vector is used to characterize the fine-grained semantic information of the sample and its modality missing patterns; This represents an instance suggestion generator, which can be implemented using a fully connected layer (MLP). This is a vector concatenation operation; if a certain mode is missing, then the feature of that mode is replaced by an all-zero tensor.

[0141] Step A42: Combine the affinity modal cue vector and the instance-level cue vector using a cross-attention mechanism to fuse the modal information and obtain the fused instance-level cue.

[0142] Currently, the "missing pattern" is not explicitly encoded; therefore, a cross-attention mechanism is introduced to fuse modal information. During training, affinity modality-level cue vectors and instance-level cue vectors are injected into the input and attention layers of the cross-attention mechanism, respectively. Instance-level cue vectors are represented as query vectors, and affinity modality-level cue vectors are represented as key and value vectors. The cross-attention mechanism fuses information in the following way:

[0143]

[0144] in, Indicates the first Instance-level cue vectors after sample fusion These are the learnable query projection matrix, the key projection matrix, and the value projection matrix, respectively. This represents the scaling factor, where is the dimension of the key vector. Indicates the first Affinity modality-level cue vectors for combinations of missing modalities; This represents the normalized activation function; This represents the matrix transpose operation. Thus, the fused instance-level hints carry both fine-grained sample information and prior knowledge of missing patterns.

[0145] Step A43: Use the modality alignment features of the missing modality samples and the fused instance-level cue vector as input to the student network, train the student network on a large number of missing modality samples, and achieve the downstream task.

[0146] The student network mainly consists of two substructures: the VILT-backbone and the downstream task classification header.

[0147] VILT-backbone is mainly composed of It consists of stacked identical encoder layers. Each layer adopts the standard Transformer encoder architecture and contains a multi-head self-attention layer (MSA) core computational submodule.

[0148] Specifically, in order to enable the student network to perform feature compensation using instance-level cues in the event of modality loss, the overall forward mapping process of the student network is defined as follows:

[0149]

[0150] in, This represents the deep semantic features or classification prediction results ultimately output by the student network. This represents the nonlinear mapping function of a student network consisting of multiple encoders.

[0151] The student network is trained on a large number of missing modal samples to achieve downstream tasks.

[0152] For example, in the student forward pass: input the modality alignment features of missing modality samples and the fused instance-level hints. Output deep features of students Student classification logits logits This represents the class prediction score vector output by the student network given an input sample, i.e., the unnormalized classification logits, where each dimension corresponds to a predefined class.

[0153] Step A5: Based on the triple knowledge distillation mechanism, perform feature alignment, category classification alignment, and cross-modal alignment on the teacher network and student network, and fine-tune the teacher network and student network.

[0154] Step A51: Construct an intrinsic modality shared knowledge distillation loss to align the feature spaces of the teacher network and the student network.

[0155] That is, for a sample of the complete mode Randomly mask complete modality samples One mode yields the corresponding missing mode sample The features of complete modalities from the teacher network should approximate the features of the corresponding missing modalities from the student network. The intrinsic modality shared knowledge distillation loss is represented as follows:

[0156]

[0157] in, The intrinsic modality shared knowledge distillation loss is used to align the feature spaces of the teacher network and the student network. These represent the last encoder layers of the teacher network and the student network, respectively, which generate two hidden features. Indicates the relationship with the first Affinity modality-level cue vectors for combinations of missing modalities; Indicates the first Instance-level cue vectors after sample fusion; This represents the distance metric function, namely Euclidean distance; A sample representing a complete mode; Represents complete modal samples The corresponding missing modality sample is one mode that is missing.

[0158] Step A52: Construct a task knowledge distillation loss, aiming to make the classification distribution of the teacher network for complete modality samples close to the classification distribution of the student network for missing modality samples corresponding to complete modality samples.

[0159] The task knowledge distillation loss is measured using KL divergence, and is expressed as follows:

[0160]

[0161] in, This indicates the loss from task knowledge distillation. and These represent the downstream task classification heads for the teacher network and student network, respectively, used to generate downstream classification scores; It is a sample of the complete modality. It is a complete modal sample The corresponding missing modal-level cue vector; Represents complete modal samples The missing modality sample corresponding to the missing modality is, in practice, solved by randomly masking the complete modality sample. One of the modalities is used to generate missing modality samples. ; Indicates the first Affinity modality-level cue vectors for combinations of missing modalities; Indicates the first Instance-level cue vectors after sample fusion; and It is the Kullback-Leibler (KL) divergence measure function, used to calculate the distance between classification scores.

[0162] Step A53: Project the final feature representations of the teacher network and student network into teacher network mapping features and student network mapping features, and calculate the cosine similarity to construct the cross-modal shared knowledge distillation loss.

[0163] In steps A51 and A52, the distillation loss of intrinsic modal sharing knowledge is given respectively. Distillation loss of task knowledge To enable the model to learn more discriminative high-dimensional semantic representations, a cross-modal shared knowledge distillation loss based on contrastive learning is constructed to avoid the feature collapse problem.

[0164] The final characteristics of the teacher network and the student network are represented as follows:

[0165]

[0166]

[0167] in, , The last encoder layer (the first layer) representing the teacher network and the student network respectively. layer); Indicates the first The final feature representation of a sample teacher network; Indicates the first The final feature representation of a sample student network.

[0168] To avoid compromising the original classification ability in the original feature space, the final feature representations of the teacher and student networks are mapped to a shared contrastive latent space using a projection head:

[0169]

[0170]

[0171] in, Indicates the teacher's network projector head; The student network projector is represented; both the teacher and student network projectors consist of two MLP layers with ReLU activation functions. Indicates the first Individual sample teacher network mapping characteristics; Indicates the first The network mapping features of each sample student.

[0172] Cosine similarity is used to measure the distance between teacher network mapping features and student network mapping features, as shown below:

[0173]

[0174] in, Indicates the first Cosine similarity between network mapping features of teachers and network mapping features of students in a sample. The L2 norm (i.e., the Euclidean length of a vector in space) is used to normalize the dot product result.

[0175] Finally, we define the cross-modal shared knowledge distillation loss. :

[0176]

[0177] in, Indicates the batch size of samples during training. This indicates the index of the sample currently being calculated. This represents the index used to iterate through all samples in the current batch. This represents the temperature hyperparameter, typically set to 0.07; it is used to scale the cosine similarity. The smaller the value, the greater the penalty the model inflicts on hard negative samples (i.e., features that look very similar but are not actually the same sample), and the higher the discrimination. Indicates the first The network mapping features of the sample teacher and the first Cosine similarity of network mapping features of each sample student.

[0178] Step A54: Construct the classification cross-entropy loss and the total loss function for the downstream task.

[0179] The classification cross-entropy loss is expressed as follows:

[0180]

[0181] in, Represents the classification cross-entropy loss. Represents the total number of samples. It is the number of categories. It is the first The sample belongs to the first The true label of the class, It is the first The sample belongs to the first The predicted probability of a class.

[0182] The final total loss function is then used for training.

[0183]

[0184] in, Indicates the total loss. To adjust affinity loss The hyperparameter of relative weights is used to control the proportion of modal affinity constraints in the total loss; To adjust for knowledge distillation loss The relative weight hyperparameters are used to control the proportion of teacher-student feature alignment and task alignment in the total loss. The weight of each loss is adjusted through the hyperparameters.

[0185] The algorithm principle for model training in steps A1-A5 above is as follows: Figure 2 As shown.

[0186] The prediction process flow in this embodiment is shown below. Figure 3 It includes:

[0187] Step B1: Obtain multimodal missing samples and perform preprocessing.

[0188] Step B2: Use a multimodal encoder to extract modal features of different modalities, and obtain modal alignment features by embedding the modal features into a unified vector space.

[0189] Step B3: Invoke the modal cue generator to generate affinity modal cue.

[0190] Step B4: Generate instance-level hints using the instance hint generator.

[0191] Step B5: Input the multimodal missing samples, along with the generated affinity-level and instance-level cueings, into the trained student network, and output the final classification or prediction results.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An incomplete multi-modal learning method based on hierarchical hints and knowledge distillation, characterized in that, The method comprises two parts: a training process and a prediction process. The training process includes the following steps: Step A1: Obtain the multimodal sample dataset and perform preprocessing; Step A2: Use a multimodal encoder to extract modal features of different modalities, and obtain modal alignment features by embedding the modal features into a unified vector space; Step A3: Build and call the modal cue generator, generate affinity modal-level cues through affinity consistency constraint learning, and use them to explicitly label the currently missing / existing modalities. Train the teacher network on a small number of complete modal samples. Step A4: Input the modality alignment features into the instance cue generator to generate instance-level cues. Use cross-attention to allow the instance cue to read the modality cue information for missing modalities. Train the student network on a large number of missing modality samples. Step A5: Based on the triple knowledge distillation mechanism, perform feature alignment, category classification alignment, and cross-modal alignment on the teacher network and student network, and fine-tune the teacher network and student network.

2. The hierarchical prompt and knowledge distillation based incomplete multi-modal learning method according to claim 1, characterized in that, Step A1 includes the following steps: Step A11: obtaining a multi-modal sample dataset of the modalities; Step A12: Divide the multimodal sample dataset into a complete modality data subset and a missing modality data subset according to whether the sample has a missing modality; Step A13: For the missing modality data subset The samples are filled with placeholder data to make each sample formally satisfy the condition that the number of actual available modalities equals the total number of modalities, thus obtaining a subset of missing modal fill data and standardizing the input. Step A14: Construct a modality missing mask for each sample to prompt generation and loss weighting.

3. The incomplete multimodal learning method based on hierarchical prompting and knowledge distillation according to claim 2, characterized in that, Step A2 includes the following steps: Step A21: Using an M-type modal encoder, extract the feature representations of each sample modality from the complete modal data subset and the missing modal completion data subset to obtain the modal features; the modal feature representations are as follows: in, Indicates the first The first sample Modal features; Indicates the first A modal encoder with multiple modes; Indicates the first The sample at the th Input features under each modality; Step A22: Using a cross-modal alignable unified vector space, perform two-term alignment on the modal features of samples in both the complete modal data subset and the missing modal imputation data subset to obtain the modal alignment features, as shown below: in, Indicates the first The first sample Modal alignment features; For mapping functions; Indicates the first Modality alignment features of each sample; Represents the feature aggregation function; Indicates the first The first sample Modal missing mask for each modality; Modality alignment features include modality alignment features for complete modality samples and modality alignment features for missing modality samples.

4. The incomplete multimodal learning method based on hierarchical prompting and knowledge distillation according to claim 3, characterized in that, Step A3 includes the following steps: Step A31: Randomly initialize a learnable modality-level cue vector set. Each cue vector in the modality-level cue vector set corresponds to a modality absence configuration, used to explicitly encode the presence / absence of each modality in the current sample; the modality-level cue vector set is represented as follows: , Indicates the number of combinations of missing modalities. Indicates the total number of modes; Indicates the first Modal-level cue vectors for combinations of missing modalities; Step A32: Project the modal cue vector set into an affinity modal cue vector set with affinity information, and learn the affinity modal cue vector set using multi-label classification loss; the affinity modal cue vector set is represented as follows: in, This represents the set of affinity modality-level cue vectors. , Indicates the first Affinity modality-level cue vectors for combinations of missing modalities; Represents the affinity mapping function; The multi-label classification loss is represented as follows: in, Indicates loss of affinity. It is a classification layer; Indicates the first The target affinity labels are 0 for missing and 1 for present; BCE is a multi-label binary classification loss. Step A33: Input the modality alignment features and affinity modality-level cue vector set of complete modality samples into the teacher network, train the teacher network on a small number of complete modality samples, and achieve the downstream task.

5. The incomplete multimodal learning method based on hierarchical prompting and knowledge distillation according to claim 4, characterized in that, Step A33 includes the following steps: Step S331: Set the affinity modality-level cue vector set It is directly attached to the encoder of the teacher network; Step S332: Input the modality alignment features of complete modality samples to construct the overall input representation relationship of the teacher network; Step S333: Train the teacher network on a small number of complete modal samples to achieve the downstream task.

6. The incomplete multimodal learning method based on hierarchical prompting and knowledge distillation according to claim 5, characterized in that, Step A4 includes the following steps: Step A41: Concatenate the modality-aligned feature representations of the samples and input them into the instance cue generator to generate instance-level cue vectors, as shown below: in, Indicates the first Individual sample instance-level cue vectors; Indicates an instance suggestion generator; This is a vector concatenation operation; Step A42: The affinity modal-level cue vector and the instance-level cue vector are fused using a cross-attention mechanism to obtain the fused instance-level cue, as shown below: in, Indicates the first Instance-level cue vectors resulting from sample fusion These are the learnable query projection matrix, the key projection matrix, and the value projection matrix, respectively. Indicates the scaling factor. Indicates the first Affinity modality-level cue vectors for combinations of missing modalities; This represents the normalized activation function; This represents the matrix transpose operation; Step A43: Use the modality alignment features of the missing modality samples and the fused instance-level cue vector as input to the student network, train the student network on a large number of missing modality samples, and achieve the downstream task.

7. The incomplete multimodal learning method based on hierarchical prompting and knowledge distillation according to claim 6, characterized in that, Step A5 includes the following steps: Step A51: Construct an intrinsic modality shared knowledge distillation loss to align the feature spaces of the teacher network and the student network; Step A52: Construct the task knowledge distillation loss, aiming to make the classification distribution of the teacher network for complete modality samples close to the classification distribution of the student network for missing modality samples corresponding to complete modality samples with one missing modality. Step A53: Project the final feature representations of the teacher network and student network into teacher network mapping features and student network mapping features, and calculate the cosine similarity to construct the cross-modal shared knowledge distillation loss; Step A54: Construct the classification cross-entropy loss and the total loss function for the downstream task.

8. The incomplete multimodal learning method based on hierarchical prompting and knowledge distillation according to claim 7, characterized in that, The intrinsic modality shared knowledge distillation loss is represented as follows: in, This represents the loss from distillation of shared knowledge across intrinsic modalities. These represent the last encoder layer of the teacher network and the student network, respectively; Represents the distance metric function; A sample representing a complete mode; Represents complete modal samples The corresponding missing modality sample with one missing modality; The task knowledge distillation loss is represented by the KL divergence metric as follows: in, This indicates the loss from task knowledge distillation. and These represent the downstream task classification heads for the teacher network and the student network, respectively. It is a complete modal sample The corresponding missing modal-level cue vector; It is a divergence measurement function.

9. The incomplete multimodal learning method based on hierarchical prompting and knowledge distillation according to claim 8, characterized in that, The cross-modal knowledge sharing distillation loss is represented as follows: in, This represents the loss from cross-modal knowledge distillation. Indicates the batch size of samples during training. This indicates the index of the sample currently being calculated. This represents the index used to iterate through all samples in the current batch; This indicates temperature hyperparameters; Indicates the first The network mapping features of the sample teacher and the first Cosine similarity of network mapping features of each sample student; Indicates the first The network mapping features of the sample teacher and the first Cosine similarity of network mapping features of each sample student.

10. The incomplete multimodal learning method based on hierarchical prompting and knowledge distillation according to claim 9, characterized in that, The classification cross-entropy loss is expressed as follows: in, Represents the classification cross-entropy loss. Represents the total number of samples. It is the number of categories. It is the first The sample belongs to the first The true label of the class, It is the first The sample belongs to the first The predicted probability of a class; The final total loss function is then used for training. in, Indicates the total loss. To adjust affinity loss Hyperparameters of relative weights; To adjust for knowledge distillation loss Hyperparameters for relative weights.

Images