Incomplete multi-mode learning method based on graph routing and prompt distillation

Abstract

The invention discloses an incomplete multi-mode learning method based on graph routing and prompt distillation, which comprises the following steps of: firstly, constructing a mode sensing routing and element self-adaptive prompt distillation framework oriented to incomplete multi-mode learning, capturing a dependency relationship between condition modes through a mode sensing graph to obtain fusion representation, and then activating a special path by sparse routing to obtain a fusion representation; according to the method, a low-rank adapter and a meta-adaptive prompt are generated, flexible fusion aiming at various missing modes is realized, finally, uncertainty perception distillation is adopted to carry out final cross-modal reasoning and robust learning, and incomplete multi-modal learning is realized. According to the method, a mode perception element prompt framework is provided, uncertainty perception prompt distillation is designed, the problem that a multi-mode learning model faces mode deficiency uncertainty and the problem that the multi-mode learning model faces mode deficiency performance suboptimum are solved, instance self-adaptive reasoning is achieved, and the method has the advantages of being simple in structure, convenient to operate and high in practicability. Reliable cross-modal knowledge migration and coherent representation learning can be realized in various missing scenes.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an incomplete multimodal learning method based on graph routing and cue distillation. Background Technology

[0002] In recent years, methodologies in Natural Language Processing (NLP) and Computer Vision (CV) have achieved numerous breakthroughs. Besides improvements to unimodal models, large-scale multimodal methods have become a very active research area. Multimodal learning is characterized by parallel analysis of data from different modalities, aiming to leverage complementary information between modalities to improve the overall understanding and performance of artificial intelligence (AI) models, such as in tasks like image segmentation, image captioning generation, and privacy protection. However, multimodal learning faces challenges in practical applications: devices or systems may be resource-constrained, preventing the acquisition of all modalities during training or inference. In such cases, the performance of multimodal learning models can severely degrade. Therefore, incomplete multimodal learning has received increasing attention in recent years, aiming to improve inference under incomplete or even unimodal conditions by utilizing complete modal data from the training phase, thereby enhancing the model's robustness to arbitrary missing modal combinations during testing.

[0003] Multimodal learning models often face the problem of modality missing uncertainty, meaning that one or more modalities may be unavailable during training but suddenly appear during inference. Therefore, effectively handling missing modalities is crucial for building robust multimodal learning systems. Early methods for multimodal learning with missing modalities generally discarded unavailable modalities directly, resulting in significant information loss. Conversely, recent methods have attempted to generate missing modalities or perform joint learning from combinations of available modalities. Although significant progress has been made in incomplete modality learning, there are still significant limitations, which can be mainly categorized into the following three types:

[0004] The first category is generative methods: the core of these methods is to directly reconstruct the data or features of the missing modality using existing modalities. For example, the ActionMAE model proposed in the paper "Sangmin Woo, Sumin Lee, Yeonju Park, Muhammad Adi Nugroho, and Changick Kim, 'Towards good practices for missing modality robust action recognition,' in Proceedings of the AAAI Conference on Artificial Intelligence, 2023, vol. 37, pp. 2776–2784" employs a missing modality predictive encoding scheme, aiming to solve the problem of missing modalities in action recognition. However, the inherent drawback of this type of method is that it usually requires training a specific generative model for every possible combination of missing modalities. As the number of modalities increases, the scheme becomes bloated and impractical due to combinatorial explosion.

[0005] The second category is knowledge distillation-based methods. These methods avoid directly generating missing data and instead enhance the learning of common semantic representations across different missing modalities through knowledge exchange and complementarity. The paper "Hong Liu, DongWei, Donghuan Lu, Jinghan Sun, Liansheng Wang, and Yefeng Zheng, 'M3ae: Multimodal representation learning for brain tumor segmentation with missing modalities,' arXiv preprint arXiv: 2303.05302, 2023" proposes a memory-efficient self-distillation strategy for sharing semantics across heterogeneous missing modalities through consistency loss distillation. However, these methods often indiscriminately aggregate all available information, failing to fully consider the hierarchical and complementary nature of knowledge between modalities, resulting in suboptimal learned representations and susceptibility to training imbalance issues in complex missing modalities.

[0006] The third category is based on prompting-based learning methods. With the widespread adoption of large-scale vision-language pre-trained models (such as ViLT), prompting-based learning has attracted attention due to its ability to efficiently adapt to downstream tasks with a small number of learnable parameters. This involves introducing prompting mechanisms to improve the robustness of multimodal models. The paper "Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee, 'Multimodal prompting with missing modalities for visual recognition,' in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14943–14952" proposes a missing modality-aware prompting method that alleviates the performance degradation caused by missing modalities without requiring large-scale model fine-tuning. However, existing prompting designs are often relatively simple and fail to fully consider the dynamic needs of models for complementary information under different missing modalities, leaving considerable room for improvement in robustness.

[0007] In summary, existing missing modality learning methods lack the ability to dynamically model conditional dependencies between modalities and cannot adapt to various missing modalities, thus limiting their ability to perform robust, uncertainty-aware knowledge transfer in partially observed multimodal environments. Therefore, while existing technologies have made progress, they have not yet effectively solved the core challenge: how to design an efficient method that avoids complex generation processes, dynamically adapts to arbitrary missing modalities, and deeply mines hierarchical complementary information among available modalities. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides an incomplete multimodal learning method based on graph routing and cue distillation, which solves the problems of existing missing modality learning methods lacking the ability to dynamically model the conditional dependencies between modalities and being unable to adapt to various missing modes.

[0009] The technical solution adopted in this invention is: an incomplete multimodal learning method based on graph routing and cue distillation, the specific steps of which are as follows:

[0010] S1. Construct a pattern-aware routing and meta-adaptive prompting distillation framework for incomplete multimodal learning;

[0011] The framework includes: a pattern-aware graph encoding module, a meta-adaptive prompt generation module, and an uncertainty-aware knowledge transfer module.

[0012] Each sample input to the framework is only observed. A portion of the modality, represented by a binary mask. This indicates that if the mode If available, then Otherwise, it is 0; the observed set of modes is denoted as Given a dataset The goal is to minimize Learn a predictor that is robust to multiple missing patterns .

[0013] in, Indicates the first Each sample in the observable modality set The input features are as follows. Indicates the first The label of each sample, where N represents the total number of samples in the dataset; Represents the cross-entropy loss function. This represents the set of parameters that the framework can learn.

[0014] S2. Based on step S1, a pattern-aware graph route is proposed through the pattern-aware graph encoding module. Each modality is projected into a unified space and the missing mode is explicitly encoded. Feature extraction based on the pattern-aware graph is performed to obtain the route context representation.

[0015] First, a statistical embedding vector is generated by combining all observable modal features of the sample with their missing pattern statistics. Then, intermodal relationships are modeled through cross-modal dependencies. Finally, feature extraction based on a pattern-aware graph is performed through modal embedding to obtain the routing context representation.

[0016] S3. Based on the routing context representation obtained in step S2, generate prompts by using the sparse route generation module of the meta-adaptive prompt generation module;

[0017] design 10 candidate paths, each path generates a low-rank adapter and Individual adaptive cue modal features are adjusted. Each path encodes a specific transformation strategy for different missing patterns, and the routing network selects the Top-S paths for each sample.

[0018] S4. Based on step S3, the uncertainty-aware knowledge transfer module performs final cross-modal reasoning and robust learning to achieve incomplete multimodal learning.

[0019] The uncertainty perception knowledge transfer module includes: a full-modal teacher model and a student reasoning model;

[0020] Full-modal teacher model computational fusion representation Then the student reasoning model predicts the probability as follows: , its in The classification loss on each training sample is .

[0021] in, Yes Linear projection of.

[0022] Furthermore, step S2 is specifically as follows:

[0023] S21. Calculate the statistical embedding vector generated by the combined statistical information of all observable modal features of the sample and its missing patterns.

[0024] First, for each sample If the mode is observed, its mode-specific encoder is used. Input for each modality Encoded as Otherwise, use a placeholder instead. Then, aggregate the observed semantics and sample-level statistics to calculate the results from the samples. A statistical embedding vector generated by combining all observable modal features and their missing pattern statistics. The expression is as follows:

[0025] ;

[0026] in, Indicates sample The number of modes observed in the data; and These are used to summarize the proportion and dispersion of missing patterns, respectively; MLP stands for Multilayer Perceptron.

[0027] S22. Based on step S21, model intermodal relationships through cross-modal dependencies;

[0028] Intermodal relationships are represented by a graph. Modeling is performed, where nodes Represents the number of modes, edges This indicates that there are semantic or informational dependencies between modalities. (Figure) adjacency matrix Combining prior knowledge and learnable weights, it is represented as .

[0029] in, Represents the prior matrix, The learning matrix includes learnable weights. Represents the real number field. The modalities learned during training. For modes The intensity of the impact, This indicates element-wise multiplication. Representing modes For modes The impact of incoming edge weights. By analyzing the angle matrix Normalization yields all data from other modalities. Pointing to the current mode A set of weights for the edges.

[0030] S23. Based on step S22, feature extraction based on pattern-aware graph is performed through modal embedding to obtain the routing context representation;

[0031] Each modality is updated via a gated graph, and the samples... In modality Embedded vectors The expression is as follows:

[0032] ;

[0033] in, That is, the angle matrix described in step S22. , and Indicates learnable parameters, Indicates transpose, used to determine the missing pattern. Adjustment right The impact; This represents the Sigmoid function. The final routing context... The expression is as follows:

[0034] ;

[0035] in, , Represents a statistical embedding vector. This represents the dimension of the modal embedding vector. This indicates the dimension of the statistical embedding vector.

[0036] Furthermore, step S3 is specifically as follows:

[0037] S31. Introduce sparsity and diversity regularization loss to construct differentiable sparse routes;

[0038] Given Candidate paths The routing network will context Mapping to the path logits, the expression is as follows:

[0039] ;

[0040] in, This represents the path logits vector.

[0041] Then, the top-S paths are selected through Gumbel-Top-S sampling, which means adding random noise with a Gumbel distribution to the path logit score. Then, all paths are reordered based on the sum of logit and Gumbel noise. Finally, the S paths with the highest scores after sorting are selected, as shown in the following expression:

[0042] ;

[0043] in, Indicates the first The logit score of each path, Controlling the smoothness of sampling. During training, soft weights... To ensure gradient propagation, during the inference phase, only the Top-S paths, i.e., the selected paths, are activated. The rest are 0.

[0044] Then, sparsity and diversity regularization loss are introduced, as shown in the following expression:

[0045] ;

[0046] in, This indicates the current batch size. within, no. The average routing probability of the path; Indicates the first The nth sample pair The soft weight of each path. This represents the sparsity regularization loss. This represents the loss due to diversity regularization.

[0047] S32. Based on step S31, implement meta-adaptive prompt generation;

[0048] Each activated path Includes a prompt network The network is based on the modal embedding and statistical embedding vectors of the samples. Generate a low-rank adapter for each mode. and each mode Individual element adaptive prompt token The expression is as follows:

[0049] ;

[0050] ;

[0051] in, and The parameters represent those of the low-rank adapter. The dimension of the parameter matrix is ​​represented by the number of parameters after low-rank decomposition. Descending to ; Indicates the first The path is modal The generated first Each prompt token; and the low-rank adapter for each modality. Applications include: .

[0052] The resulting cues are then injected into the modal encoder or cross-modal Transformer, using a cue-based cross-attention operator. Guide attention. Along the path Below, modality The update is represented as The expression is as follows:

[0053] ;

[0054] Ultimately, modality The representation is obtained by the weighted average, that is... The corresponding meta-adaptive hint The expression is obtained by aggregating the suggestions from the activated path, as follows:

[0055] .

[0056] Furthermore, step S4 is specifically as follows:

[0057] S41. Perform final cross-modal reasoning and robust learning through uncertainty-aware distillation;

[0058] Full-modal teacher model Training on complete modality samples generates a Dirichlet distribution, expressed as follows:

[0059] ;

[0060] in, This represents the Dirichlet distribution parameter vector of the output of the full-modal teacher model under full-modal input conditions. Representing modes Embedded, Indicates modal aggregator, This indicates a learnable projection. The Dirichlet parameter is guaranteed to be positive; adding 1 ensures that all parameters are strictly greater than 0. Let C represent a vector space of positive real numbers with dimension C. Teachers' categories... The expression for the predicted probability is as follows:

[0061] ;

[0062] in, This represents the predicted probability of the full-modal teacher model for each category; This represents a random probability vector sampled from a Dirichlet distribution. This indicates its expected value. Indicated by The Dirichlet probability distribution is given by parameters. The student inference model obtains the knowledge distillation loss by matching the distribution of the full-modal teacher model with the Dirichlet–Dirichlet KL divergence. The expression is as follows:

[0063] ;

[0064] in, This represents the inference derived from the student's observable modality. The loss for each sample is weighted by the inverse of the teacher's prediction uncertainty, resulting in the uncertainty-perceived distillation loss, expressed as follows:

[0065] ;

[0066] in, This represents the sample-level uncertainty weighting coefficient; This represents a probability vector sampled from the Dirichlet distribution generated by the full-modal teacher model. express The covariance matrix; The trace is a matrix representing the summation of variances over all categories; This represents the final distillation loss after adding uncertainty weights.

[0067] S42. Based on step S41, incomplete multimodal learning is achieved through prompt-guided cross-modal consistency.

[0068] The graph Laplacian regularization term and cross-modal contrastive learning are introduced to encourage intermodal consistency, as shown in the following expression:

[0069] ;

[0070] ;

[0071] in, Indicates a non-linear projection head. Indicates the temperature coefficient. Representing modes For modes The impact. This represents the graph Laplacian regularization loss. This represents the cross-modal contrastive learning loss.

[0072] Finally, by synthesizing the objective functions, we obtain the final multi-task optimization objective, achieving incomplete multimodal learning, as shown in the following expression:

[0073] ;

[0074] in, The coefficients are used to balance the contributions of various losses; This represents the final total loss function.

[0075] The beneficial effects of this invention are as follows: First, the method of this invention constructs a pattern-aware routing and meta-adaptive cue distillation framework for incomplete multimodal learning. It captures the dependencies between conditional modalities through a pattern-aware graph to obtain a fusion representation. Then, sparse routing activates specialized paths to generate low-rank adapters and meta-adaptive cuees, achieving flexible fusion for various missing modes. Finally, uncertainty-aware distillation is used for final cross-modal reasoning and robust learning to achieve incomplete multimodal learning. This invention proposes a pattern-aware meta-hint framework, a novel paradigm for learning incomplete multimodal hints. It models structural dependencies in missing modalities using pattern-aware graphs and employs sparse routing generators for adaptive hints, enabling instance-adaptive reasoning. The method also designs uncertainty-aware hint distillation, combining confidence-weighted teacher supervision with student hint guidance graphs and contrast alignment to achieve reliable cross-modal knowledge transfer and coherent representation learning across various missing modalities. This addresses the challenges of modality-missing uncertainty in multimodal learning models (existing methods often require training multiple modality-specific models for each different missing modality to reconstruct missing features or samples, which is time-consuming and impractical due to the exponential growth of missing modalities) and suboptimal performance issues in multimodal learning models when facing missing modalities (many models tend to aggregate all available information uniformly without considering the hierarchy of knowledge when jointly training with complete and incomplete modal data, leading to suboptimal performance). Attached Figure Description

[0076] Figure 1 This is a flowchart of an incomplete multimodal learning method based on graph routing and cue distillation according to the present invention.

[0077] Figure 2 This is a schematic diagram of a pattern-aware routing and meta-adaptive cue distillation framework for incomplete multimodal learning in an embodiment of the present invention. Detailed Implementation

[0078] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0079] like Figure 1 The flowchart of an incomplete multimodal learning method based on graph routing and cue distillation according to the present invention is shown below. The specific steps are as follows:

[0080] S1. Construct a pattern-aware routing and meta-adaptive prompting distillation framework for incomplete multimodal learning;

[0081] like Figure 2 As shown, the framework includes: a pattern-aware graph encoding module, a meta-adaptive prompt generation module, and an uncertainty-aware knowledge transfer module.

[0082] This embodiment studies a multimodal classification problem with incomplete input: each sample in the frame input only observes a portion of the M modalities, represented by a binary mask. This indicates that if the mode If available, then Otherwise, it is 0; the observed set of modes is denoted as Given a dataset The goal is to minimize Learn a predictor that is robust to multiple missing patterns .

[0083] in, Indicates the first Each sample in the observable modality set The input features are as follows. Indicates the first The label of each sample, where N represents the total number of samples in the dataset; Represents the cross-entropy loss function. This represents the set of parameters that the framework can learn.

[0084] S2. Based on step S1, propose pattern-aware graph routing through the pattern-aware graph encoding module, project each modality into a unified space, explicitly encode missing patterns, perform feature extraction based on pattern-aware graph, and obtain routing context representation.

[0085] To address the heterogeneity of observed modalities and the diversity of missing modes in incomplete multimodal samples, this embodiment proposes a mode-aware graph routing approach. This approach projects each modality into a unified space and explicitly encodes missing modes to achieve adaptive selective fusion.

[0086] First, a statistical embedding vector is generated by combining all observable modal features of the sample with their missing pattern statistics. Then, intermodal relationships are modeled through cross-modal dependencies. Finally, feature extraction based on a pattern-aware graph is performed through modal embedding to obtain the routing context representation.

[0087] S3. Based on the routing context representation obtained in step S2, generate prompts by using the sparse route generation module of the meta-adaptive prompt generation module;

[0088] design There are 10 candidate paths, and each path generates a low-rank adapter and... Meta-adaptive prompts are used to modulate modal features. Each path encodes a specific transformation strategy for different missing patterns, and the routing network selects the Top-S paths for each sample for flexible, pattern-aware fusion, thereby enhancing multimodal robustness.

[0089] S4. Based on step S3, the uncertainty-aware knowledge transfer module performs final cross-modal reasoning and robust learning to achieve incomplete multimodal learning.

[0090] The uncertainty-aware knowledge transfer module includes a full-modal teacher model and a student reasoning model. The full-modal teacher model provides uncertainty-weighted supervision and guides the student reasoning model to learn stable and generalizable representations under partial modal observations through prompt-guided cross-modal consistency.

[0091] Full-modal teacher model computational fusion representation Then the student reasoning model predicts the probability as follows: , its in The classification loss on each training sample is .

[0092] in, Yes Linear projection of.

[0093] In this embodiment, step S2 is specifically as follows:

[0094] S21. Calculate the statistical embedding vector generated by the combined statistical information of all observable modal features and missing modes of the sample.

[0095] First, for each sample If the mode is observed, its mode-specific encoder is used. Input for each modality Encoded as Otherwise, use a placeholder instead. Then, aggregate the observed semantics and sample-level statistics to calculate the results from the samples. A statistical embedding vector generated by combining all observable modal features and their missing pattern statistics. The expression is as follows:

[0096] ;

[0097] in, Indicates sample The number of modes observed in the data; and These are used to summarize the proportion and dispersion of missing patterns, respectively; MLP stands for Multi-Layer Perceptron, a type of feedforward neural network composed of multiple fully connected layers and nonlinear activation functions. Its function is to perform nonlinear mapping and fusion of input features to obtain more discriminative feature representations.

[0098] S22. Based on step S21, model intermodal relationships through cross-modal dependencies;

[0099] Intermodal relationships are represented by a graph. Modeling is performed, where nodes Represents the number of modes, edges This indicates that there are semantic or informational dependencies between modalities. (Figure) adjacency matrix Combining prior knowledge and learnable weights, it is represented as .

[0100] in, This represents the prior matrix, encoding known semantic coupling relationships (e.g., if mode 1 and mode 2 have semantic or informational dependencies, then...). (otherwise it is 0). The learning matrix includes learnable weights. Represents the real number field. The modalities learned during training. For modes The intensity of the impact, This indicates element-wise multiplication. Representing modes For modes The impact of incoming edge weights. By analyzing the angle matrix Normalization yields all data from other modalities. Pointing to the current mode A set of weights for the edges.

[0101] S23. Based on step S22, feature extraction based on the pattern-aware graph is performed through modality embeddings to obtain the routing context representation;

[0102] Each modality is updated via a gated graph, and the samples... In modality Embedded vectors The expression is as follows:

[0103] ;

[0104] in, That is, the angle matrix described in step S22. , and Indicates learnable parameters, Indicates transpose, used to determine the missing pattern. Adjustment right This avoids the influence of noise or missing modes in the update process; This represents the Sigmoid function. The final routing context... The expression is as follows:

[0105] ;

[0106] in, , Represents a statistical embedding vector. This represents the dimension of the modal embedding vector. This indicates the dimension of the statistical embedding vector.

[0107] In this embodiment, step S3 is specifically as follows:

[0108] S31. Introduce sparsity and diversity regularization loss to construct differentiable sparse routing.

[0109] Given Candidate paths The routing network will context Mapping to the path logits, the expression is as follows:

[0110] ;

[0111] in, This represents the path logits vector.

[0112] The top-S paths are selected using Gumbel-Top-S sampling, which involves adding random noise from a Gumbel distribution to the path logit score. Then, all paths are reordered based on the sum of logit and Gumbel noise, and finally, the S paths with the highest scores after sorting are selected, as shown in the following expression:

[0113] ;

[0114] in, Indicates the first The logit score of each path, Controlling the smoothness of sampling. During training, soft weights... To ensure gradient propagation, during the inference phase, only the Top-S paths, i.e., the selected paths, are activated. The rest are 0.

[0115] Then, to further stabilize the routing, sparsity and diversity regularization loss are introduced, as shown in the following expression:

[0116] ;

[0117] in, This indicates the current batch size. within, no. The average routing probability of the path; Indicates the first The nth sample pair The soft weight of each path. The sparsity regularization loss is represented by summing the squares of the different path weights for each sample. Minimizing this loss during model training forces the path weight distribution to be close to one-hot (i.e., only one vector element is 1, and the rest are 0), encouraging sparse selection for each sample. The advantage is that specific samples only activate the path that best suits their missing model, thus learning the optimal adaptation. The diversity regularization loss is essentially the calculation of the i-th element within the current batch. The negative entropy of the path routing probability, minimizing this loss during model training, forces a more uniform probability distribution and encourages diverse choices among different samples within a batch. The benefit is that different samples choose different paths, making all paths "participate in the work" and fully utilizing the model's capabilities.

[0118] S32. Based on step S31, implement meta-adaptive prompt generation.

[0119] Each activated path Includes a prompt network. The network is based on the modal embeddings and statistical embedding vectors of the samples. Generate a low-rank adapter for each mode. and each mode Individual element adaptive prompt token The expression is as follows:

[0120] ;

[0121] ;

[0122] in, and The parameters represent those of the low-rank adapter. The dimension of the parameter matrix is ​​represented by the number of parameters after low-rank decomposition. Descending to It is easier to train; Indicates the first The path is modal The generated first Each prompt token; and the low-rank adapter for each modality. Applications include: .

[0123] The resulting cues are then injected into the modal encoder or cross-modal Transformer, using a cue-based cross-attention operator. Guide attention. Along the path Below, modality The update is represented as The expression is as follows:

[0124] ;

[0125] Ultimately, modality The representation is obtained by the weighted average, that is... The corresponding meta-adaptive hint The expression is obtained by aggregating the suggestions from the activated path, as follows:

[0126] .

[0127] In this embodiment, step S4 is specifically as follows:

[0128] S41. Perform final cross-modal reasoning and robust learning through uncertainty-aware distillation;

[0129] To provide more stable supervision, a full-modal teacher model Training on the complete modality samples generates a Dirichlet distribution to capture prediction uncertainty, as shown in the following expression:

[0130] ;

[0131] in, This represents the Dirichlet distribution parameter vector of the output of the full-modal teacher model under full-modal input conditions. Representing modes Embedded, This represents a modality aggregator (such as mean or attention pooling). This indicates a learnable projection. The Dirichlet parameter is guaranteed to be positive; adding 1 ensures that all parameters are strictly greater than 0. Let C represent a vector space of positive real numbers with dimension C. Teachers' categories... The expression for the predicted probability is as follows:

[0132] ;

[0133] in, This represents the predicted probability of the full-modal teacher model for each category, and it simultaneously encodes category confidence and cognitive uncertainty; This represents a random probability vector sampled from a Dirichlet distribution. This indicates its expected value. Indicated by The Dirichlet probability distribution is given by parameters. The student inference model obtains the knowledge distillation loss by matching the distribution of the full-modal teacher model with the Dirichlet–Dirichlet KL divergence. The expression is as follows:

[0134] ;

[0135] in, This represents the inference derived from the student's observable modality. The loss for each sample is weighted by the inverse of the teacher's prediction uncertainty to emphasize reliable supervision, resulting in the uncertainty-perceived distillation loss, expressed as follows:

[0136] ;

[0137] in, This represents a sample-level uncertainty weighting coefficient, used to measure the teacher model's influence on the sample. The predictive reliability is therefore large. Higher weight will be given to distillation signals that are credible and reliable to teachers; This represents a probability vector sampled from the Dirichlet distribution generated by the full-modal teacher model. express The covariance matrix captures the total prediction uncertainty (cognitive + stochastic). The trace is a matrix representing the summation of variances over all categories; This represents the final distillation loss after adding uncertainty weights.

[0138] S42. Based on step S41, incomplete multimodal learning is achieved through prompt-guided cross-modal consistency.

[0139] To encourage intermodal consistency, a graph Laplacian regularization term and cross-modal contrastive learning are introduced, expressed as follows:

[0140] ;

[0141] ;

[0142] in, Indicates a non-linear projection head. Indicates the temperature coefficient. Representing modes For modes The impact. This represents the graph Laplacian regularization loss, based on the modal graph. adjacency matrix By constraining the distance between modal representations, we can force semantically related modalities in the modal relationship graph to remain close in the feature space, thereby improving the semantic consistency and collaborative representation ability between modalities. This represents cross-modal contrastive learning loss, which enhances instance-level cross-modal consistency, making the different modal features of the same sample more similar, while distinguishing it from other samples.

[0143] Finally, by synthesizing the objective functions, we obtain the final multi-task optimization objective, achieving incomplete multimodal learning, as shown in the following expression:

[0144] ;

[0145] in, The coefficients are used to balance the contributions of various losses; This represents the final total loss function.

[0146] To further verify the effectiveness of the method of the present invention, this embodiment describes the method of the present invention in the context of "multimodal risk identification in urban traffic monitoring scenarios". In this scenario, the system needs to comprehensively analyze heterogeneous data from multiple sensors, including: (1) video data collected by road cameras; (2) trajectory data uploaded by vehicle GPS devices; (3) environmental audio data collected by roadside devices; and (4) structured event data (such as speeding, congestion alerts, etc.) recorded by the traffic management platform. Due to the influence of the actual deployment environment, the above modal data are often missing during the collection and transmission process, such as some cameras being temporarily offline, audio equipment failure causing the environmental sound modality to be unavailable, and GPS signal drift causing the trajectory modality to be missing. In this embodiment, a modal relationship graph is first constructed based on the prior semantic relationship between different modalities (such as video and audio having a complementary relationship for the same road scene, and GPS trajectory and structured events possibly having a causal dependency in time sequence), and the graph routing module dynamically selects the interaction edge path according to the currently available modalities to achieve adaptive modeling of conditional dependencies. Subsequently, through a prompt distillation mechanism, the "complete modality expert prompts" trained in the full modality scenario are transferred to the incomplete modality scenario, so that the system can still use the prompt information to perform compensatory reasoning when some modalities are missing, thereby maintaining high recognition performance.

[0147] This embodiment further validates the method through simulation, evaluating it on the MM-IMDb and CMU-MOSI datasets using F1-Macro and tri-class classification accuracy as evaluation metrics. It is compared with several state-of-the-art models, including MCTN, MMIN, MPVR, EPE-P, and P-EA.

[0148] Images, text, and audio are encoded using ViLT, BERT, and COVAREP, respectively, and mapped to 512, 512, and 128 dimensions. Missing modalities are replaced with placeholders (all-one images, empty strings, or zero vectors). A missing rate is set. (Default 60%): Single-modal missing samples account for %, dual-mode each account for %, higher-order missing samples are handled according to the same rules.

[0149] In the simulation verification of this embodiment, both the full-modal teacher model and the student inference model maintain fixed parameters for the backbone network, training only the task head and the learnable cue module; the learnable cue length is 16, and the cue injection layer number is 5. Dynamic sparse routing uses K = 4 candidate paths, selecting Top-S = 2 paths for each sample. The low-rank adapter is used on a 512-dimensional embedding. = 32. The loss weight is , , , , and The model is trained using the Adam optimizer (learning rate...). Weight decay 0.02, batch size = 32), a total of 50 epochs were trained.

[0150] Table 1 shows the accuracy of the simulation results for identifying known networks. The proposed method consistently outperforms all baseline models: achieving a 2.17 percentage point improvement in F1-Macro on MM-IMDb (2-modality) and a 2.30 percentage point improvement on CMU-MOSI (3-modality). Even with a mismatch between the proportion of missing modalities during training and testing, PAG-MPD maintains strong performance, demonstrating its ability to adaptively fuse cross-modal information and handle missing modalities through pattern-aware graph routing and dynamic hyper-prompt. Notably, its advantages are even more pronounced in complex trimodal scenarios, illustrating the effectiveness of dynamic prompt adaptation in modeling rich multimodal dependencies.

[0151] Table 1

[0152]

[0153] In summary, the method of this invention constructs a novel framework for incomplete modality learning through pattern-aware graph routing and meta-adaptive cue distillation. It captures dependencies between conditional modalities using pattern-aware graphs to derive fusion representations, and sparsely routes to activate specialized paths, generating low-rank adapters and meta-adaptive cueing, thereby achieving flexible fusion for various missing modalities. Finally, it employs uncertainty-aware distillation: a teacher trained on complete modality samples provides uncertainty-weighted supervision, while the student—across all available modalities—learns robustly through cue-guided graph- and contrastive alignment, enabling resilient reasoning and coherent cross-modal knowledge transfer.

[0154] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. An incomplete multimodal learning method based on graph routing and cue distillation, the specific steps of which are as follows: S1. Construct a pattern-aware routing and meta-adaptive prompting distillation framework for incomplete multimodal learning; The framework comprises: The module includes a pattern-aware graph encoding module, a meta-adaptive prompt generation module, and an uncertainty-aware knowledge transfer module. Each sample of the frame input only observes a part of modalities, represented by a binary mask , where 1 if the modality is available, otherwise; the observed set of modalities is denoted as ; Given a dataset The goal is to learn a predictor that is robust to missing data to a variety of missing patterns ; in, Indicates the first Each sample in the observable modality set The input features are as follows. Indicates the first The label of each sample, where N represents the total number of samples in the dataset; Represents the cross-entropy loss function. The framework can learn a set of parameters; S2. Based on step S1, a pattern-aware graph route is proposed through the pattern-aware graph encoding module. Each modality is projected into a unified space and the missing mode is explicitly encoded. Feature extraction based on the pattern-aware graph is performed to obtain the route context representation. First, a statistical embedding vector is generated by the combined statistical information of all observable modal features and their missing modes of the sample. Then, intermodal relationships are modeled through cross-modal dependencies. Finally, feature extraction based on the mode-aware graph is performed through modal embedding to obtain the routing context representation. S3. Based on the routing context representation obtained in step S2, generate prompts by using the sparse route generation module of the meta-adaptive prompt generation module; Design each path generates a low-rank adaptor and each path encodes a specialized transformation strategy for different missing patterns, and the routing network selects Top-S paths for each sample; S4. Based on step S3, the uncertainty-aware knowledge transfer module performs final cross-modal reasoning and robust learning to achieve incomplete multimodal learning. The uncertainty perception knowledge transfer module includes: a full-modal teacher model and a student reasoning model; Full-modal teacher model computes fused representations The student inference model prediction probability is then The classification loss over training samples is ; wherein is a linear projection of .

2. The incomplete multi-modal learning method based on graph routing and hint distillation according to claim 1, characterized in that, Step S2 is as follows: S21. Calculate the statistical embedding vector generated by the combined statistical information of all observable modal features of the sample and its missing patterns. First, for each sample If the mode is observed, its mode-specific encoder is used. Input for each modality Encoded as Otherwise, placeholders are used; then, observed semantics and sample-level statistics are aggregated to calculate the results from the samples. A statistical embedding vector generated by combining all observable modal features and their missing pattern statistics. The expression is as follows: ； wherein, denotes the number of modes observed in the sample and are used to summarize the proportion of missing modes and the degree of dispersion, respectively; MLP denotes a multi-layer perceptron;​ S22. Based on step S21, model intermodal relationships through cross-modal dependencies; Inter-modal relationships are captured through a graph Modeling is performed where nodes represent modalities, edges represent semantic or information dependency relationships between modalities; the graph is represented by an adjacency matrix combining prior knowledge and learnable weights, as ; in, Represents the prior matrix, The learning matrix includes learnable weights. Represents the real number field. The modalities learned during training. For modes The intensity of the impact, This indicates element-wise multiplication; Representing modes For modes Impact; incoming edge weight By analyzing the angle matrix Normalization yields all data from other modalities. Pointing to the current mode A set of weights for the edges; S23. Based on step S22, feature extraction based on pattern-aware graph is performed through modal embedding to obtain the routing context representation; Each modality propagates updates through a gating graph, samples The expression of the embedding vector under modality is as follows: ； wherein, i.e. the angle matrix as described in step S22, , and denote learnable parameters, denotes the transpose, used to adjust the influence of the missing patterns according to the missing patterns ; ; denotes the Sigmoid function; the final routing context is expressed as follows: ； wherein, , denotes a statistical embedding vector, denotes a dimension of the modal embedding vector, denotes a dimension of the statistical embedding vector.

3. The incomplete multi-modal learning method based on graph routing and hint distillation according to claim 2, characterized in that, Step S3 is as follows: S31. Introduce sparsity and diversity regularization loss to construct differentiable sparse routes; Given A candidate path The routing network maps the context to path logits, expressed as follows: ； wherein, represents a path logits vector; Then Top-S paths are selected by Gumbel-Top-S sampling, i.e. adding random noise from Gumbel distribution to the path logit scores and reordering all paths according to the sum of logit and Gumbel noise, and finally taking the S paths with the highest scores after reordering, expressed as follows: ； wherein, represents the logit score of the th path, controls the smoothness of the sampling; during training, the soft weights guarantee the gradient propagation; during inference, only the top-S paths, i.e. the selected paths are activated, the rest are 0; Then, sparsity and diversity regularization loss are introduced, as shown in the following expression: ； where, denotes the average routing probability of the th path within the current batch size ; denotes the soft weight of the th sample on the th path; denotes the sparsity regularization loss; denotes the diversity regularization loss; S32. Based on step S31, implement meta-adaptive prompt generation; each activated path comprises a prompting network based on modal and statistical embedding vectors of the sample generates a low-rank adapter for each modality and meta-adaptive prompt tokens for each modality expressed as follows: ； ； wherein, and denote parameters of the low-rank adapter, denote dimensions of the parameter matrix, the amount of parameters is reduced from to after low-rank decomposition; denote the th path is the th prompt token generated by the modality ; and the low-rank adapter of each modality applies as: ; The resulting cues are then injected into a modal encoder or cross-modal Transformer via a cue-based cross-attention operator Guiding attention; on the path The updated representation of the modal is denoted as The expression is as follows: ； Finally, the representation of the modality is obtained by a weighted average, i.e. The corresponding meta-adaptive cues are aggregated by cues of the activated paths, expressed as follows: 。 4. The incomplete multi-modal learning method based on graph routing and hint distillation according to claim 3, characterized in that, Step S4 is as follows: S41. Perform final cross-modal reasoning and robust learning through uncertainty-aware distillation; Full-modal teacher model Training on full modal samples, generating Dirichlet distribution, expression as follows: ； where, denotes the Dirichlet distribution parameter vector output by the full-modal teacher model under full-modal input condition, denotes the embedding of the modality denotes the modality aggregator, denotes the learnable projection, guarantees Dirichlet parameters are positive numbers, +1 guarantees all parameters are strictly greater than 0, denotes a positive real number vector space of dimension C; the teacher's predicted probability expression for class is as follows:​ ； where, denotes the predicted probability of each class by the full-modal teacher model; denotes a random probability vector sampled from a Dirichlet distribution, denotes its expectation value, denotes a Dirichlet probability distribution parametrized by The student reasoning model matches the distribution of the full-modal teacher model by a Dirichlet-Dirichlet KL divergence, resulting in a knowledge distillation loss , which is expressed as follows: ； where, represents the inference from the student observable modality; the loss for each sample is weighted by the inverse of the teacher prediction uncertainty, resulting in an uncertainty-aware distillation loss, expressed as follows: ； wherein, denotes a sample-level uncertainty weighting coefficient; denotes a probability vector sampled from a Dirichlet distribution generated from the full-modal teacher model, denotes covariance matrix of is the matrix trace, denoting the sum of variances over all classes; denotes the final distillation loss with uncertainty weight added; S42. Based on step S41, incomplete multimodal learning is achieved through prompt-guided cross-modal consistency. The graph Laplacian regularization term and cross-modal contrastive learning are introduced to encourage intermodal consistency, as shown in the following expression: ； ； wherein, denotes a nonlinear projection head, denotes a temperature coefficient, denotes a modality on the modalities ; denotes a graph Laplacian regularisation loss; denotes a cross-modality contrastive learning loss; Finally, by synthesizing the objective functions, we obtain the final multi-task optimization objective, achieving incomplete multimodal learning, as shown in the following expression: ； wherein, coefficients are used to balance the contribution of each loss; represents the final total loss function.

Images