A missing modal multi-modal learning method based on quality perception prompt modulation

Abstract

A missing modal multi-modal learning method based on quality perception prompt modulation. It relates to the technical field of multi-modal learning, aiming at the problem that the existing missing modal multi-modal learning method ignores the reliability difference of modal information, leading to the continuous propagation of low-quality information in the feature extraction and fusion process, affecting the robustness and prediction accuracy of the model, the following scheme is proposed: detecting the missing type of the input sample, generating the missing modal feature and completing the missing modal, combining the task correlation evaluation, decision influence evaluation and intrinsic truth evaluation to obtain the comprehensive quality score, and modulating the prompt vector of the pre-trained visual language encoder according to the comprehensive quality score, then the double modal feature is weighted and fused according to the quality perception, and the task prediction is completed. The present application is suitable for multi-modal learning tasks such as image-text classification, sentiment analysis, visual question answering, medical auxiliary diagnosis and the like under the condition of missing modal.

Description

Technical Field

[0001] This invention belongs to the field of multimodal learning, specifically relating to a method for improving the robustness of multimodal learning by utilizing a quality-aware cue modulation mechanism under modality-deficient conditions. Background Technology

[0002] Existing multimodal learning techniques typically enhance a model's perception, understanding, and decision-making capabilities for complex tasks by jointly modeling information from different sources such as images, text, and speech. In tasks such as sentiment analysis, visual question answering, cross-modal retrieval, medical assisted diagnosis, and social media content recognition, multimodal methods generally achieve higher recognition accuracy and stronger semantic representation capabilities compared to single-modal methods. In particular, with the development of vision-language pre-trained models, the unified representation space formed based on large-scale image-text alignment training has become an important research foundation for current multimodal learning.

[0003] In existing research, a mainstream approach involves constructing multimodal encoders and fusion networks to extract features from different modalities and then concatenating, weighting, applying attention interactions, or performing deep collaborative modeling to accomplish classification, matching, or generation tasks. This type of method typically assumes that data from each modality can be acquired simultaneously and completely, thus leveraging the complementary advantages of multimodal approaches in ideal data environments. However, in real-world applications, modality loss is a common problem. For example, in medical settings, some examination data is missing; in autonomous driving, sensor failure or occlusion leads to incomplete perception information; and in social media, text and images contain one-sided omissions or incomplete information. This causes traditional models that rely on complete multimodal input to experience significant performance degradation during deployment.

[0004] To address the problem of missing modalities, existing research typically employs two approaches. The first is generative completion methods, which utilize available modalities to reconstruct or generate feature representations of the missing modalities. For example, through cross-modal mapping, generative adversarial networks, variational autoencoders, or feature projection networks, text is inferred as image features, or images are inferred as text semantic representations, thus recovering the complete input required by the model. This type of method can formally complete the missing modalities, alleviating the structural problems caused by incomplete input. However, the generated results are essentially estimates, easily introducing noise, semantic bias, and missing details. Especially when the correspondence between different modalities is weak or the training data distribution is complex, there is often a significant quality difference between the completed modal and the true modal.

[0005] The second category is based on cue-based learning or efficient parameter tuning. This involves introducing a small number of learnable cue vectors into the pre-trained visual-language model, guiding the model to adapt to downstream tasks and modality-deficient scenarios. This type of method relies on the existing cross-modal alignment capabilities of the pre-trained model, requiring only a small number of parameters to train. It boasts high computational efficiency and transferability, thus becoming an important research direction in multimodal learning with missing modalities in recent years. For example, when images, text, or some modalities are missing, the corresponding cue template can be switched according to the type of missing data to maintain the model's ability to utilize existing modalities as much as possible. Compared to directly fine-tuning the entire pre-trained model, cue-based learning has advantages such as a small number of parameters, low training cost, and strong adaptability.

[0006] However, current research, whether using completion-based methods or cue-based learning methods, largely revolves around the discrete judgment of "modality existence." This typically involves classifying modalities into two states: "complete and usable" or "missing and unusable," and then performing completion, cue switching, or fixed fusion accordingly. While this approach can address the problem of missing modalities to some extent, it generally lacks further characterization of the differences in modal information quality, failing to effectively reflect the continuous changes in the authenticity, task contribution, and decision influence of different modalities. In other words, existing technologies often assume that all existing modalities have approximately the same reliability during fusion, or only perform coarse-grained processing based on missing labels, without deeply distinguishing between "real but low-quality modalities" and "completed but less reliable modalities."

[0007] For example, while missing modalities obtained through generative models or projection networks structurally recover the bimodal input, the completion result may contain spurious features, semantic drift, or local distortions. Similarly, modalities obtained from real-world acquisitions may suffer from significant quality degradation due to factors such as varying lighting conditions, sensor noise, decreased image resolution, and overly simplistic text descriptions. However, many existing fusion methods still employ a uniform processing strategy for these modalities in both the feature extraction and fusion stages, failing to continuously adjust cue strength, feature encoding processes, and fusion weights based on modal reliability. This results in noise information from low-quality modalities being continuously introduced into the encoding space and decision-making process, thereby weakening the overall model performance and robustness.

[0008] Furthermore, existing methods do not adequately consider the uncertainty of the quality assessment results themselves. When the model's judgment on the authenticity or validity of a certain modality is unstable, direct intervention based on that judgment can easily lead to new misguidance, amplifying erroneous estimates and propagating them to subsequent fusion and classification stages. Therefore, simply processing based on whether a modality is missing, or merely applying static weights to modalities, is no longer sufficient to meet the requirements of robustness, fine-grained control, and practical usability in multimodal learning with missing modalities in complex scenarios.

[0009] In summary, existing technologies suffer from the drawback of neglecting the reliability differences of modal information, failing to accurately distinguish between the completed modal and the true modal, as well as existing modalities of different quality levels, resulting in the continuous propagation of low-quality information during feature extraction and fusion. Summary of the Invention

[0010] To address the shortcomings of existing technologies that neglect the reliability differences of modal information, fail to finely distinguish between completed modalities and true modalities, as well as existing modalities of different quality levels, leading to the continuous propagation of low-quality information during feature extraction and fusion, this invention provides a technical solution: A missing modality multimodal learning method based on quality-aware cue modulation includes the following steps: Step 1: Detect the modality missing type of the input sample. When a modality missing is detected, generate the feature representation of the missing modality to complete the missing modality. Step 2: Input the completed modal features into the task relevance evaluator, calculate the contribution of each modality to the prediction of downstream tasks, and output the task relevance quality score. Step 3: Calculate the marginal impact of each mode on the final prediction result through gradient analysis, and output the decision influence quality score; Step 4: Evaluate the intrinsic authenticity of each modality using an adversarial discriminator, and output an intrinsic authenticity quality score based on the discriminant confidence. Step 5: The three-dimensional quality scores of task relevance, decision influence, and intrinsic authenticity are fused through a learnable weighting mechanism to output the comprehensive quality score of each modality; Step 6: Select the corresponding initial learnable cue vector according to the sample missing type, and generate a modulation factor by mapping the comprehensive quality score through the mapping network. Apply the modulation factor to the cue vectors of each layer of the pre-trained visual-language encoder so that the cue intensity is continuously adjusted with modality reliability. Step 7: Extract the modal features guided by prompts through a pre-trained visual-language encoder, convert the comprehensive quality score into fusion weights using a piecewise smooth mapping function, and then perform weighted fusion of the dual-modal features after confidence gating adjustment, before feeding them into the classification head to complete the task prediction.

[0011] Further, step 1 specifically involves detecting the missing type of the input sample and activating the completion module; the completion module takes the available modal features as input and generates a feature representation consistent with the dimension of the target modality; the two scenarios of image missing and text missing correspond to independent completion module parameters; the completion module is trained on a subset of samples with complete modalities in the training set, and jointly optimizes reconstruction constraints and semantic alignment constraints.

[0012] Further, step 2 specifically involves obtaining the classifier prediction confidence under both complete bimodal input and unimodal input, and using the difference in confidence with and without the modality as a pseudo-label for task contribution; training a lightweight multilayer perceptron predictor with Pearson correlation coefficient loss as the objective function to ensure that the prediction score is statistically correlated with the task contribution; and in the inference stage, the predictor directly estimates the task relevance score from the multimodal features without additional forward propagation.

[0013] Further, step 3 specifically involves calculating the gradient L2 norm of the modal features with respect to the downstream task loss. As a proxy indicator of influence, an exponential moving average is introduced to smooth historical gradients to filter out random fluctuations; the gradient magnitudes of the two modes are normalized to make the influence of the two modes comparable; a lightweight predictor is trained to directly estimate the normalized influence as a pseudo-label during the inference stage, and Pearson correlation coefficient loss is used for supervision.

[0014] Further, step 4 specifically involves the discriminator taking modal features as input and outputting the probability distributions of the true modality and the completed modality, performing adversarial training using binary cross-entropy loss; and using discriminative entropy to measure the uncertainty of the discriminator's output, defining the confidence score and the authenticity score as follows:

[0015] When the confidence level is low, it automatically reverts to a neutral estimate to avoid misjudgment and misleading quality assessment.

[0016] Further, step 5 specifically involves weighted fusion of the three-dimensional quality scores using learnable scalar weight parameters:

[0017] in The Sigmoid function ensures that the weights are non-negative; weight parameters Automatically learns the loss in end-to-end training; image and text modalities independently output a comprehensive quality score. and The range of values ​​is .

[0018] Further, step 6 specifically involves maintaining four independent sets of initial cue parameters, corresponding to four scenarios: modal integrity, image missing, text missing, and general missing; each layer of cue is jointly generated from the previous layer's bimodal cue via a cross-layer projection network, which employs a lightweight structure of linear transformation followed by an activation function; the overall quality score is mapped to a quality embedding vector via a multilayer perceptron, and the quality embedding is modulated by attention to generate a modulation factor. It applies to the cue vectors at each layer. High-quality modal cues retain their full strength, while low-quality modal cues are moderately suppressed.

[0019] Further, step 7 specifically involves obtaining the fusion weights from the quality score using a piecewise smoothed Sigmoid mapping function:

[0020] in For the slope parameter, The midpoint threshold of the mass. and The lower and upper bounds of the weights ensure that low-quality modes retain the minimum fusion weights; confidence-gated secondary adjustment of the fusion weights: When the confidence level is low, the system reverts to a neutral fusion strategy with equal weights; the pre-trained encoder parameters are kept frozen, and only the cue parameters and each auxiliary network are trained.

[0021] The beneficial effects of this invention are as follows: A three-dimensional quality assessment mechanism is proposed, which quantifies the reliability of modal information from three complementary dimensions: task relevance, decision influence, and intrinsic authenticity. Each of the three dimensions plays an independent role and has a synergistic effect, enabling a fine distinction between modalities of different sources and reliability.

[0022] By embedding quality information into the prompt generation stage and using modulation factors to make the prompt strength continuously change with modal reliability, the propagation of low-quality information is suppressed from the source of feature encoding, which is different from the passive strategy of existing methods that only perform quality processing in the fusion stage.

[0023] By introducing a confidence gating mechanism, the system automatically reverts to a neutral estimate when there is uncertainty in the quality assessment, which enhances the system's tolerance to assessment errors and improves its overall robustness.

[0024] Based on the cue-based learning paradigm, the parameters of the pre-trained visual-language encoder are kept frozen, and only a small number of cue parameters and a lightweight auxiliary network are trained. This results in high parameter efficiency and allows for direct reuse of the representation capabilities of existing large-scale pre-trained models.

[0025] By detecting the modality missing type of the input samples and generating feature representations of the missing modalities using an intermodal feature projection network when modality missing is detected, missing modality completion is achieved. The effect of missing modality completion is that samples that were originally difficult to jointly model due to incomplete modalities can regain structurally complete bimodal inputs. This avoids the problem of insufficient information utilization caused by directly discarding missing samples, degenerating into single-modal processing, or using coarse imputation methods in existing technologies. At the same time, it provides a unified input basis for subsequent quality assessment, cue modulation, and feature fusion, thereby mitigating the adverse effects of modality missing on model robustness and prediction accuracy.

[0026] By inputting the completed modal features into the task relevance evaluator, the contribution of each modality to downstream task prediction is calculated, and a task relevance quality score is output. The effect of the task relevance evaluation part is that it enables the system to identify the actual contribution of different modalities to the current prediction results from the task objective itself. Unlike the coarse-grained processing method in the prior art that only relies on the existence of modalities, this feature can further distinguish between modalities that exist but have limited contributions and modalities that exist and are critical to the task. This allows subsequent modulation and fusion to more effectively strengthen high-value modalities and suppress low-contribution modalities, thereby improving the model's efficiency in utilizing task-related information.

[0027] By calculating the marginal impact of each modality on the final prediction result through gradient analysis, a decision influence quality score is output. The effect of the decision influence assessment part is that it enables the system to characterize the actual strength of each modality at the level of the decision-making link within the model. Unlike existing studies that mainly rely on the input surface quality or static feature strength, this feature uses the model's sensitivity to each modality to reflect the real dependency relationship, thereby more accurately identifying the degree of decision dominance of a certain modality in the current sample. It supplements the quality information that is difficult to fully cover by the simple task contribution and improves the consistency between the quality assessment results and the actual prediction process.

[0028] By evaluating the intrinsic authenticity of each modality using an adversarial discriminator and combining the discriminant confidence score to output an intrinsic authenticity quality score, the effect of the intrinsic authenticity evaluation part is that it enables the system to independently measure the reliability of the modality from the perspectives of the modality's naturalness, authenticity, and whether the completion process introduces distortion. This differs from existing completion methods that usually assume that the completed modality can be directly equated with the real modality in subsequent fusion. This feature can identify noise, semantic drift, and missing details that may exist in the completed modality, thereby avoiding low authenticity modalities being given inappropriate weights in subsequent processing and enhancing the system's ability to suppress completion errors.

[0029] By fusing the three-dimensional quality scores of task relevance, decision influence, and intrinsic authenticity through a learnable weighting mechanism, a comprehensive quality score for each modality is output. The effect of this three-dimensional quality fusion is that it integrates the originally scattered multi-dimensional quality information into a unified and learnable comprehensive evaluation result. This allows modal quality to no longer rely on a single standard for judgment, but to simultaneously reflect three complementary dimensions: task contribution, decision-making role, and data authenticity. Compared to the fixed-weight fusion or weighting based on a single heuristic indicator commonly used in existing technologies, this feature can automatically learn the importance of different dimensions during the main task training process, improving the adaptability and refinement of the comprehensive quality assessment, and enabling subsequent cue modulation and feature fusion to be based on a more reliable quality foundation.

[0030] By selecting the corresponding initial learnable cue vector based on the type of missing samples, and generating a modulation factor from the comprehensive quality score through a mapping network, this factor is applied to the cue vectors of each layer of the pre-trained visual language encoder. This allows the cue intensity to be continuously adjusted according to the modality reliability. The effect of the quality-aware cue modulation part is that it moves the quality information to the feature encoding stage. Unlike most existing technologies that only passively weight modalities in the final fusion stage, this feature can apply differentiated guidance to different quality modalities at the source of encoding. This allows high-quality modalities to receive more sufficient cue enhancement, while low-quality modalities are moderately suppressed. This reduces the possibility of low-quality information entering the representation space and propagating backward, thereby improving the noise resistance and encoding effectiveness of the feature extraction stage.

[0031] By setting four independent sets of initial prompt parameters, corresponding to four scenarios: modal integrity, image missing, text missing, and general missing, and by jointly generating prompts for each layer through a cross-layer projection network, the effect of the initial prompt selection and cross-layer prompt generation is that the prompt mechanism can form more targeted initialization and inter-layer transmission logic for different missing modes. This avoids the problem of insufficient adaptability caused by using a unified prompt template to handle different missing scenarios in existing technologies. At the same time, the joint generation of inter-layers maintains the continuity and consistency of prompt information in the deep layers of the network, and improves the coverage and expressiveness of prompt modulation for the encoder's multi-layer representation in missing scenarios.

[0032] By mapping the overall quality score to a quality embedding vector via a multilayer perceptron, and then generating a modulation factor through attention modulation, this factor is applied element-wise to the cue vectors of each layer of the corresponding modality. The effect of the quality embedding and attention modulation is that modal quality information can be continuously and finely integrated into the cue parameters, rather than being coarsely scaled using simple scalar weights. Compared to the uniform gating or fixed-ratio compression methods commonly used in existing methods, this feature can more flexibly express the influence of different quality levels on the components of the cue structure, thereby improving the responsiveness of cue modulation to complex quality differences and making cue suppression or enhancement more precise.

[0033] By extracting modal features guided by prompts through a pre-trained visual language encoder, and converting the overall quality score into fusion weights using a piecewise smooth mapping function, the effect of the quality mapping fusion part is to establish a smooth and constrained correspondence between the modal quality score and the fusion weights. Unlike the existing technology that directly uses linear mapping or hard threshold screening, this feature can effectively enhance the influence of high-quality modalities when the quality is high, and can also retain its minimum information contribution when the quality is low. It avoids the complete discarding of useful information or the excessive amplification of noise information due to excessive weight changes, thereby improving the stability and controllability of the fusion process.

[0034] By introducing confidence-gated adjustment on top of the fusion weights, the effect of the confidence-gated part is that when the quality assessment results are relatively reliable, the system can more actively adjust the fusion strategy according to quality differences; when the quality assessment results are uncertain, the system can automatically fall back to a more neutral fusion state, avoiding being misled by subsequent decisions due to erroneous quality judgments. Compared with the existing technology's approach of directly participating in weighting once a score is obtained, this feature improves the system's tolerance to quality assessment errors and enhances the robustness of the overall method under complex data distributions and unstable completion conditions.

[0035] By weighted fusion of bimodal features and feeding them into the classification head to complete task prediction, the effect of quality-aware feature fusion and task output is that the final prediction result is based on features that have undergone quality screening and reliability adjustment. Unlike existing technologies, which perform approximate equivalence processing on all input modalities, this reduces the interference of low-quality modalities on the classification boundary, improves the quality of the fused representation received by the classification head, and ultimately enhances the prediction accuracy and stability of multimodal learning tasks under missing modalities.

[0036] By freezing the parameters of the pre-trained visual language encoder and training only the cue parameters and auxiliary networks, the efficient parameter updates achieved by freezing the pre-trained encoder allow for direct reuse of the cross-modal representation capabilities already possessed by large-scale pre-trained models. This avoids the high training costs, excessive parameter redundancy, and increased risk of overfitting associated with large-scale fine-tuning of the entire model. Compared to existing solutions that require updating the entire model's parameters, this feature significantly improves training efficiency and deployment feasibility while maintaining performance improvement potential, making it more suitable for widespread application in multimodal learning scenarios lacking specific modalities. Attached Figure Description

[0037] Figure 1 This is an overall flowchart of the method of the present invention.

[0038] Figure 2 Flowchart for completing missing modalities.

[0039] Figure 3 This is a flowchart for three-dimensional quality assessment.

[0040] Figure 4 Flowchart for quality perception prompt modulation.

[0041] Figure 5 Flowchart for quality-perceived feature fusion. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Implementation Method 1: This implementation method is a further detailed description of the technical solution provided by the present invention, specifically: The specific steps for implementing this plan are as follows.

[0044] Step 1: Obtain input samples and determine the missing modal state to form initial modal features to be completed.

[0045] In detail, input interfaces for image and text modalities are first established to receive image and text data from the samples to be trained or inferenced. For image data, uniform size scaling, pixel normalization, and basic enhancement processing can be used; for text data, word segmentation, truncation and padding, and word vector encoding can be used. After preprocessing, modal integrity detection is performed on each sample to determine whether it belongs to the modal integrity, image missing, or text missing state. For modal integrity samples, initial image features and initial text features are directly extracted; for samples with missing features, only the initial features of the available modalities are retained, and the missing feature marker and available modal features are output together as input for the next step of missing modal completion. Through this step, the original heterogeneous inputs can be uniformly organized into standardized feature data with missing feature type identifiers.

[0046] Step 2: Generate missing modal features based on the missing modal states to obtain a structurally complete bimodal feature representation.

[0047] In detail, the missing types and available modal features output from step 1 are processed. When the sample is an image missing feature, the initial text features are input into the image-side intermodal feature projection network, which outputs complete image features with the same dimension as the image modal features. When the sample is a text missing feature, the initial image features are input into the text-side intermodal feature projection network, which outputs complete text features with the same dimension as the text modal features. When the sample modality is complete, the original bimodal features are directly passed through. The projection network can adopt a multilayer perceptron structure, including at least an input mapping layer, an intermediate semantic transformation layer, and an output alignment layer. To ensure the usability of the completed features, simulated missing samples can be constructed from modality-complete samples during the training phase. The target modality is predicted using available modalities, and the results are generated simultaneously using the reconstruction target and semantic alignment target constraints, so that the completed features approximate the real features in both numerical representation and semantic space. The output of this step is the image features and text features corresponding to each sample. Both are structurally complete and can be directly used as input for subsequent quality assessment.

[0048] Step 3: Evaluate task relevance based on bimodal complete features and output the task relevance quality score for each modality.

[0049] In detail, the image and text features output from step 2 are fed into the task relevance evaluation module. The training of this module can be achieved by constructing a supervision signal as follows: Predictions are made for the complete bimodal input and the input after removing one modality, respectively. The magnitude of the change in confidence level of the downstream task output is compared; the greater the change, the more important the modality is to the current task, thus forming a pseudo-label for the task contribution of that modality. A lightweight predictor is then used to learn to directly map the bimodal features to the task relevance score. This predictor can employ a multilayer perceptron or other lightweight structure. Its input is the bimodal features obtained in step 2 or their fused representation, and its output is the image modality task relevance score and the text modality task relevance score. During the inference phase, multiple ablation forward computations are no longer repeated; instead, the estimation results are directly output by the predictor. The output of this step is two task relevance quality scores, which serve as parallel reference information for evaluating the decision influence in the next step.

[0050] Step 4: Evaluate the decision influence based on the sensitivity of the prediction loss to modal features, and output the decision influence quality score for each modality.

[0051] In detail, the bimodal features output from step 2 are input into the current task network to calculate the task loss and analyze the sensitivity of the loss to changes in each modality feature. The dependence of the model on different modalities on the current sample can be characterized by calculating the gradient strength of the task loss with respect to image and text features; the larger the gradient magnitude, the stronger the marginal influence of that modality on the current prediction result. To reduce the impact of random fluctuations during training, historical gradient information can be smoothed, and the influence values ​​of the two modalities can be normalized to a uniform scale for comparison. A dedicated lightweight predictor is then trained to directly predict the image modality decision influence score and the text modality decision influence score based on the bimodal features output from step 2. In actual deployment, the predictor directly outputs the results without explicitly calculating the gradient again. This step outputs two decision influence quality scores, which, along with the task relevance quality score obtained in step 3, are fed into the next step.

[0052] Step 5: Evaluate the intrinsic authenticity and discrimination confidence level of the modal features, and output the intrinsic authenticity quality score of each modality.

[0053] In detail, the image features and text features output from step 2 are input into the corresponding authenticity discrimination modules. For the image modality, an image-side adversarial discriminator can be constructed to distinguish whether the image feature originates from a real captured image or from a missing completion result; for the text modality, a text-side adversarial discriminator can be constructed to distinguish whether the text feature is a real text encoding result or a completed text feature. The discriminator output includes not only the authenticity judgment result but also reflects the discrimination confidence level by considering the concentration of its output distribution. When the discrimination result is relatively clear, the corresponding modality can obtain a higher confidence authenticity score; when the discrimination result has a large degree of uncertainty, the system does not directly adopt extreme judgments but automatically backslides to a neutral estimate to reduce the interference of misjudgments on the overall quality assessment. This step outputs the intrinsic authenticity quality score of the image modality and the intrinsic authenticity quality score of the text modality, and sends them together with the quality scores obtained in steps 3 and 4 into the next step.

[0054] Step 6: Integrate multi-dimensional quality information to form a comprehensive quality score for each modality.

[0055] In detail, the task relevance quality score output in step 3, the decision influence quality score output in step 4, and the intrinsic authenticity quality score output in step 5 are fused together. During fusion, instead of using fixed, manually set proportions, trainable weight parameters are set for different quality dimensions, allowing the system to automatically learn more reasonable dimensional importance under the current task and data distribution during end-to-end training. After fusion, image modality comprehensive quality scores and text modality comprehensive quality scores are obtained separately. This comprehensive quality score no longer only reflects the existence of a modality, but comprehensively reflects the degree of direct help the modality provides to the task, the degree of actual influence on the current decision, and the authenticity and reliability of the features themselves. The output of this step is the two comprehensive quality scores corresponding to each sample, along with the original complete bimodal features, both of which serve as input for the next step of quality-aware cue modulation.

[0056] Step 7: Select initial prompting parameters based on the missing type, and generate quality-modulated prompting vectors for each layer based on the overall quality score.

[0057] In detail, firstly, based on the missing type identifier given in step 1, a corresponding set is selected from multiple pre-maintained initial cue parameter sets as the cue starting point for the current sample. The initial cue parameter set can at least correspond to modal integrity, image missing, text missing, and general missing scenarios, thus allowing different input states to correspond to different cue initialization strategies. Subsequently, the image modality comprehensive quality score and text modality comprehensive quality score output in step 6 are input into a quality mapping network, converting them into quality embedding representations that can participate in cue modulation. Then, combined with an attention modulation mechanism, modulation factors acting on the cue vectors of each layer are generated. For each layer of cue in the pre-trained visual language encoder, the current layer cue can be generated jointly by the bimodal cue from the previous layer through a cross-layer projection network, allowing the cue information to propagate layer by layer along the network depth. After generation, the modulation factors are applied to the cue vector of the corresponding modality in an element-wise manner. For higher-quality modalities, the cue amplitude remains strong; for lower-quality modalities, the cue amplitude is suppressed. The output of this step is an image-side cue sequence and a text-side cue sequence that have undergone quality-aware modulation and match the current sample, serving as the input for feature encoding in the next step.

[0058] Step 8: Use the quality-modulated cue vector to guide the pre-trained visual language encoder to extract features and obtain quality-aware encoded features.

[0059] In detail, the structurally complete bimodal features output from step 2 and the cue vectors from each layer output from step 7 are fed into the pre-trained visual-language encoder. The visual encoder receives image features and corresponding image cue vectors, while the text encoder receives text features and corresponding text cue vectors. During encoding, the main parameters of the pre-trained encoder remain frozen, with only the cue vectors and auxiliary networks handling task adaptation and quality adjustment. This fully utilizes the existing cross-modal representation capabilities of large-scale pre-trained models while avoiding excessive training overhead and overfitting risks caused by full-model fine-tuning. After encoding, image quality-aware encoded features and text quality-aware encoded features are output. The output of this step consists of two high-level semantic representations enhanced by cue enhancement and quality modulation, along with the comprehensive quality score output from step 6, which serve as inputs for the next step of generating fusion weights and feature fusion.

[0060] Step 9: Generate modal fusion weights based on the overall quality score, and combine them with confidence gating to form the final fusion weights.

[0061] In detail, the overall quality score output from step 6 is fed into the fusion weight mapping module. This module does not directly use a simple linear mapping, but rather a smooth and controlled piecewise mapping method to transform quality levels into weight values ​​more suitable for the fusion stage. During the mapping process, the center position of quality change, the steepness of change, and the upper and lower limits of the weights can be set, allowing high-quality modalities to obtain a larger fusion ratio while retaining a minimum information path for low-quality modalities to prevent them from being completely eliminated. Subsequently, combined with the discrimination confidence information obtained in the authenticity assessment step of step 5, the initial fusion weights are subjected to secondary gating correction. When the confidence level is high, the system is allowed to allocate fusion weights more fully based on quality differences; when the confidence level is low, it automatically reverts to a more balanced neutral weight to reduce the risk of quality misjudgment. This step outputs the final fusion weights for the image modality and the final fusion weights for the text modality, and sends them to the next step.

[0062] Step 10: Perform weighted fusion of the quality-aware encoded features and complete the task prediction to obtain the final output result.

[0063] In detail, the image quality-aware encoded features and text quality-aware encoded features output from step 8 are multiplied by the final fusion weights output from step 9 to obtain their respective weighted modal representations. The two weighted representations are then fused according to a preset method, such as concatenation, weighted summation, or other feature integration methods suitable for classification tasks. The fused unified representation is fed into the task classification head, outputting the final prediction result. If applied to a multi-label classification task, the predicted probabilities corresponding to multiple categories are output; if applied to a single-label classification task, the category distribution is output; if applied to a binary classification task, the binary classification result is output. During the training phase, the task prediction error is used as the main supervision signal to jointly update the missing modality completion network, task relevance predictor, decision influence predictor, authenticity discrimination module, quality fusion module, cue parameters, quality mapping network, cross-layer projection network, and classification head parameters. The inference phase is executed in the above order, directly obtaining the final prediction result from the input samples. Thus, the fusion weights and quality-aware encoded features output from the previous step are uniformly utilized in this step, forming a complete closed-loop missing modality multimodal learning process.

[0064] To ensure the full implementation of this plan, it can be further deployed as follows.

[0065] In terms of data preparation, a dataset containing bimodal image and text annotation information can be selected as the training basis. During the training phase, image missing samples, text missing samples, and modal complete samples are randomly constructed, allowing the model to learn the processing logic under three input states simultaneously. This ensures that the model has the ability to adapt to different missing modes in practical applications.

[0066] In terms of network deployment, the intermodal feature projection network, task relevance predictor, decision influence predictor, authenticity discriminator, quality mapping network, cross-layer projection network, and classification head can all be implemented using independent parameter modules, and end-to-end collaborative optimization can be performed through a unified training framework. The visual language pre-trained encoder can be implemented by combining existing mature visual encoders and text encoders, as long as it supports the insertion of learnable cue vectors.

[0067] In terms of the training process, the missing modality completion network can be pre-trained using complete modal samples to enable it to have stable cross-modal mapping capabilities before overall joint training. During joint training, the final task prediction error is the primary objective, while each branch of the quality assessment gradually learns quality judgment criteria consistent with task performance, thereby ensuring that the overall quality score truly serves cue modulation and feature fusion.

[0068] In terms of the reasoning process, after inputting any sample, the system first determines the state of missing modalities, then automatically completes the missing modalities, and subsequently completes task relevance assessment, decision influence assessment, intrinsic authenticity assessment, multidimensional quality fusion, cue modulation, encoding extraction, fusion weighting, and classification prediction. Throughout the entire processing chain, the output of the previous step always serves as the input of the next step, the data flow is clear, the module dependencies are explicit, and engineering implementation can be directly completed based on this.

[0069] Through the above methods, the scheme can form a complete, coherent and implementable technical process from missing modality completion, quality assessment, quality perception cue modulation to quality perception fusion prediction, and still maintain good multimodal learning robustness and prediction performance under the condition of missing modalities.

[0070] Implementation Method 3: This implementation method is described in conjunction with the accompanying drawings. This implementation method demonstrates the implementation process and technical effects of the technical solution provided by the present invention through specific examples.

[0071] Example 1 This embodiment mainly obtains the comprehensive quality score of each modality through missing modality completion and three-dimensional quality assessment. The quality score is embedded into the prompt generation process to form a modulation factor, so that the prompt intensity of each layer of the pre-trained visual-language encoder is continuously adjusted with the modality reliability. In the feature fusion stage, a confidence-gated quality weighting mechanism is introduced to achieve dual-layer suppression of the propagation of low-quality information from feature encoding to classification decision, thereby improving the robustness and prediction accuracy of multimodal learning under missing modality conditions.

[0072] This embodiment provides a missing modality multimodal learning method based on quality-aware cue modulation, specifically including the following steps: Step 1: Detect the modality missing type of the input sample. When a modality missing is detected, generate the feature representation of the missing modality to complete the missing modality.

[0073] Step 2: Input the completed modal features into the task relevance evaluator, calculate the contribution of each modality to the prediction of downstream tasks, and output the task relevance quality score.

[0074] Step 3: Calculate the marginal impact of each mode on the final prediction result through gradient analysis, and output the decision influence quality score.

[0075] Step 4: Evaluate the intrinsic authenticity of each modality using an adversarial discriminator, and output an intrinsic authenticity quality score based on the discriminant confidence score.

[0076] Step 5: The three-dimensional quality scores of task relevance, decision influence and intrinsic authenticity are fused through a learnable weighting mechanism to output the comprehensive quality score of each modality.

[0077] Step 6: Select the corresponding initial learnable cue vector according to the sample missing type, and generate a modulation factor by mapping the comprehensive quality score through the mapping network. Apply the modulation factor to the cue vectors of each layer of the pre-trained visual-language encoder so that the cue intensity is continuously adjusted with modality reliability.

[0078] Step 7: Extract the modal features guided by prompts through a pre-trained visual-language encoder, convert the comprehensive quality score into fusion weights using a piecewise smooth mapping function, and then perform weighted fusion of the dual-modal features after confidence gating adjustment, before feeding them into the classification head to complete the task prediction.

[0079] Step 1 specifically involves: The missing type of the input sample is detected, and the completion module is activated. The completion module adopts an encoder-projector architecture, with the projector using a three-layer multilayer perceptron structure. The first layer maps the input features to a higher-dimensional intermediate representation space, the second layer further extracts modality-related features, and the third layer maps the features back to the dimension of the target modality. Independent projector parameters are maintained for image and text missing scenarios. The completion module is trained only on a subset of samples with complete modalities in the training set, simulating missing features by randomly occluding one modality. The reconstruction loss and contrast loss are jointly optimized, with a total loss of [missing value]. The reconstruction loss includes the mean squared error loss for image reconstruction and the cross-entropy loss for text reconstruction. Contrast loss constrains the alignment of generated features with ground truth features in the semantic space. After completion, all samples have structurally complete bimodal inputs.

[0080] Step 2 specifically involves: Task relevance measures the direct contribution of modal features to downstream tasks. Classifier prediction confidence is obtained under both complete bimodal and monomodal inputs. The difference between the confidence scores with and without the modality is used as a pseudo-label for task contribution; a larger difference indicates a more significant contribution of the modality to task prediction. A lightweight multilayer perceptron predictor is trained using this pseudo-label, supervised by Pearson correlation coefficient loss, ensuring a statistical correlation between prediction scores and task contribution. During the inference phase, the predictor directly estimates the task relevance score from multimodal features without additional forward propagation. Image and text modalities output task relevance scores independently.

[0081] Step 3 specifically involves: Decision influence measures the marginal contribution of a modality under the current fusion configuration. The L2 norm of the gradient of the downstream task loss with respect to the modal features is used as a proxy indicator of influence; a larger gradient magnitude indicates a higher dependence of the model on that modality. An exponential moving average is introduced to smooth historical gradients, filtering out random fluctuations in mini-batch training. The gradient magnitudes of the two modalities are normalized to make the influence scores of the two modalities comparable on the same scale. A lightweight predictor is trained under supervision using the normalized influence as a pseudo-label, and the inference phase directly estimates from the multimodal features, eliminating the need for repeated gradient calculations.

[0082] Step 4 specifically involves: Intrinsic authenticity assesses the completeness and naturalness of the modality itself, focusing on the essential attributes of the data independently of the task objective, and is used to capture the quality degradation introduced during the completion process. Independent adversarial discriminators are trained separately for image and text modalities, using modal features as input and binary cross-entropy loss for adversarial training, enabling the discriminators to distinguish between real and completed modalities. To mitigate the risk of misjudgment caused by early training instability of the discriminator, discriminative entropy is introduced to measure the uncertainty of the discriminator's output, and a confidence level is defined. The authenticity quality score is When the confidence level is low, it automatically reverts to a neutral estimate to avoid misjudgment and misleading quality assessment.

[0083] Step 5 specifically involves: The 3D quality scores are weighted and fused using learnable scalar weight parameters. The weights are activated by a sigmoid function to ensure non-negativity. The fusion formula is as follows: Weight parameters Automatically learned during end-to-end training of the main task loss, reflecting the relative importance of different dimensions under the current task and data distribution. Image and text modalities independently output comprehensive quality scores. and The range of values ​​is . .

[0084] Step 6 specifically involves: Four independent sets of initial cue parameters are maintained, corresponding to four scenarios: modal integrity, image missing, text missing, and general missing. The corresponding initial cue is dynamically selected based on the sample missing type. For the L-layer Transformer encoder, each layer's cue is jointly generated from the previous layer's bimodal cue via a cross-layer projection network. The projection network employs a lightweight structure with a linear transformation followed by an activation function and layer normalization is applied to ensure the transfer and integration of cue information between layers. Quality-aware modulation is then applied: the overall quality score is mapped to a quality embedding vector via a multilayer perceptron, and the quality embedding is then modulated by attention to generate a modulation factor. It acts on the cue vectors of each layer of the corresponding modality. High-quality modal cues maintain their full strength, effectively guiding the encoder's feature extraction; low-quality modal cues are moderately suppressed, reducing the impact of low-quality information entering the representation space from the source.

[0085] Step 7 specifically involves: Pre-trained visual-language encoder parameters are kept frozen to extract cue-guided image features. Text features The fusion weights are derived from the quality scores using a piecewise smoothed Sigmoid mapping function. The slope parameter controls the steepness of the mapping, the midpoint threshold determines the center position of the mapping, and the lower bound of the weights ensures that low-quality modes retain the minimum fusion weight, preventing effective information from being completely discarded. Furthermore, a confidence gating is introduced to perform a secondary adjustment of the fusion weights. When the confidence level is low, a neutral fusion strategy with equal weights is applied. The weighted bimodal features are concatenated and fed into a task-specific classification head to output the prediction result. During training, the task classification loss is used as the main supervision signal, and the cue parameters, quality assessment network, and classification head are updated end-to-end through backpropagation.

[0086] Example 2 from Figure 1 As can be seen, the method provided in this embodiment includes four sequentially executed modules: missing modality completion, three-dimensional quality assessment, quality-aware cue modulation, and quality-aware feature fusion. Taking the image-text bimodal classification task as an example, this embodiment will be described in detail with reference to the accompanying drawings.

[0087] Step 1: Missing modality completion.

[0088] like Figure 1As shown, the system first detects the modality missing type of the input sample. If the sample modality is complete, it proceeds directly to the second step; if a missing modality exists, the completion module is activated. Taking the MM-IMDb movie genre classification task as an example, when the image modality (movie poster) is missing, text features are used as input, and an approximate representation of the image features is generated through a three-layer MLP projector. After completion, the sample has a structurally complete bimodal input. The completion module adopts a two-stage training strategy: the first stage pre-trains the projector on a subset of complete samples; the second stage freezes the projector parameters and performs joint training of the entire framework.

[0089] Step 2: Three-dimensional quality assessment.

[0090] like Figure 2 As shown, the quality assessment module independently scores each completed modality and outputs quality scores in three dimensions.

[0091] In the task relevance assessment, three forward propagations were performed respectively for the full bimodal, image-only, and text-only modes. The difference in classification confidence was recorded as a pseudo-label of task contribution and used to supervise the training of the task relevance predictor.

[0092] In the evaluation of decision influence, a backpropagation is performed after the standard forward propagation to calculate the L2 norm of the task loss with respect to image and text features, respectively. These are then smoothed and normalized using an exponential moving average and used as pseudo-labels to supervise the training of the influence predictor. The two predictors can directly estimate their corresponding scores during the inference phase without repeating the above calculation process.

[0093] In the intrinsic authenticity assessment, the discriminator judges the authenticity of the completed modal features. Completed modalities typically receive a lower authenticity probability due to the introduction of generation error, while genuinely acquired modalities receive a higher score. The confidence score is calculated using discriminant entropy. When the discriminator outputs uncertainty, the authenticity score automatically regresses to a neutral estimate, ensuring the stability of the assessment results.

[0094] The scores from the three dimensions are weighted and fused using learnable weights to ultimately output the overall image quality score and the overall text quality score.

[0095] Step 3: Quality perception prompt modulation.

[0096] like Figure 3As shown, the initial cue set is selected based on the type of missing samples. Taking the CLIP model as an example, the visual encoder uses a ViT-B / 16 structure, with both the visual encoder and text encoder having 12 layers. Each layer maintains cue vectors for both the image and text sides, with the cue length selected between 12 and 16 tokens. A cross-layer projection network jointly maps the bimodal cue from the previous layer to the current layer, allowing cue information to flow between layers. The quality score is then used by the mapping network to generate corresponding modulation factors, which are applied element-wise to the cue vectors of each layer. High-quality modal cues are almost unsuppressed, effectively guiding the encoder to focus on relevant features; low-quality modal cues are compressed accordingly, reducing their interference with the representation space from the encoding stage.

[0097] Step 4: Integration of quality perception features.

[0098] like Figure 4 As shown, after being guided by quality-aware cues, the CLIP encoder outputs image and text features. The combined quality score is used to obtain the fusion weights for each modality through a piecewise smoothed sigmoid mapping. These weights are then adjusted a second time using confidence gating; aggressive adjustment is allowed at high confidence levels, while weights revert to equal weights at low confidence levels. The weighted bimodal features are concatenated and fed into a task-specific classification head to output the prediction result. For the MM-IMDb multi-label classification task, the classification head's final layer is activated by a sigmoid function to output 23-dimensional class probabilities; for the UPMC Food-101 single-label classification task, a softmax function is used to output 101-dimensional probabilities; for the Hateful Memes binary classification task, a single-dimensional logit is output and then processed by a sigmoid function to obtain the judgment probability.

[0099] The method described in this embodiment has been experimentally verified to effectively improve the prediction performance of multimodal classification tasks under different modality missing rates and missing distribution configurations, thus verifying the effective improvement of the robustness of the quality-aware mechanism for multimodal learning with missing modalities.

Claims

1. A missing modality multimodal learning method based on quality-aware cue modulation, characterized in that: Includes the following steps: Step 1: Detect the modality missing type of the input sample. When a modality missing is detected, generate the feature representation of the missing modality to complete the missing modality. Step 2: Input the completed modal features into the task relevance evaluator, calculate the contribution of each modality to the prediction of downstream tasks, and output the task relevance quality score. Step 3: Calculate the marginal impact of each mode on the final prediction result through gradient analysis, and output the decision influence quality score; Step 4: Evaluate the intrinsic authenticity of each modality using an adversarial discriminator, and output an intrinsic authenticity quality score based on the discriminant confidence. Step 5: The three-dimensional quality scores of task relevance, decision influence, and intrinsic authenticity are fused through a learnable weighting mechanism to output the comprehensive quality score of each modality; Step 6: Select the corresponding initial learnable cue vector according to the sample missing type, and generate a modulation factor by mapping the comprehensive quality score through the mapping network. Apply the modulation factor to the cue vectors of each layer of the pre-trained visual-language encoder so that the cue intensity is continuously adjusted with modality reliability. Step 7: Extract the modal features guided by prompts through a pre-trained visual-language encoder, convert the comprehensive quality score into fusion weights using a piecewise smooth mapping function, and then perform weighted fusion of the dual-modal features after confidence gating adjustment, before feeding them into the classification head to complete the task prediction.

2. The missing modality multimodal learning method based on quality-aware cue modulation according to claim 1, characterized in that: Step 1 specifically involves: The missing type of the input sample is detected, and the completion module is started. The completion module takes the available modal features as input and generates a feature representation with the same dimension as the target modality. The two scenarios of image missing and text missing correspond to independent completion module parameters. The completion module is trained on a subset of samples with complete modalities in the training set and jointly optimizes the reconstruction constraints and semantic alignment constraints.

3. The missing modality multimodal learning method based on quality-aware cue modulation according to claim 1, characterized in that: Step 2 specifically involves: Under both complete bimodal input and zero-based unimodal input conditions, the classifier prediction confidence is obtained, and the difference between the two is used as a pseudo-label for the task contribution of that modality. The training of the lightweight multilayer perceptron predictor is supervised by the pseudo-label. During inference, the predictor directly provides an estimate based on the multimodal features without repeating the ablation forward propagation. The task relevance scores of the image modality and the text modality are calculated and output independently.

4. The missing modality multimodal learning method based on quality-aware cue modulation according to claim 1, characterized in that: Step 3 specifically involves: The L2 norm of the gradient of the downstream task loss with respect to each modality feature is used as the proxy for the influence of the decision on that modality; the historical gradient is smoothed by introducing an exponential moving average to eliminate the interference of small-batch random fluctuations. The gradient magnitudes of the two modes are normalized to ensure that the influence scores of the two modes are on the same scale. Similarly, a lightweight predictor is trained to perform supervised learning with the goal of normalizing the influence. During the inference phase, there is no need to recalculate the gradients; the predictor directly provides the estimates.

5. The missing modality multimodal learning method based on quality-aware cue modulation according to claim 1, characterized in that: Step 4 specifically involves: Independent adversarial discriminators are constructed for image and text modalities respectively. Using features from each modality as input, adversarial training is performed using binary cross-entropy loss, enabling the discriminators to distinguish between the real modality and the imputed modality. The uncertainty of the discriminator output is quantified by discriminant entropy, and the confidence level is defined accordingly. The intrinsic authenticity quality score is obtained by a weighted combination of the discrimination probability and the confidence level. When the confidence level is low, the score automatically converges to the neutral value to prevent the discrimination error from propagating backward.

6. The missing modality multimodal learning method based on quality-aware cue modulation according to claim 1, characterized in that: Step 5 specifically involves: The three-dimensional quality scores are weighted and summed using learnable scalar weight parameters: The Sigmoid activation function acts on each weight parameter to ensure its non-negativity; parameters It automatically converges during the end-to-end training process of the main task, without requiring manual setting of the proportions of each dimension; the image modality and the text modality output comprehensive quality scores independently. and Both values ​​are constrained to Within the range.

7. The missing modality multimodal learning method based on quality-aware cue modulation according to claim 1, characterized in that: Step 6 specifically involves: Each layer maintains an independent set of initial cue parameters for four scenarios: modal integrity, image missing, text missing, and general missing. These parameters are dynamically switched based on the type of sample missing. Cues from adjacent layers are jointly generated from bimodal cuees in the previous layer through a cross-layer projection network. This projection network consists of linear transformations, activation functions, and layer normalization, maintaining a lightweight design while enabling information flow between layers. The overall quality score is converted into a quality embedding via a multilayer perceptron and then modulated by attention to obtain a modulation factor. And apply element-wise multiplication to the corresponding layer's cue vector. The cue amplitude corresponding to high-quality modes is preserved, while the cue amplitude corresponding to low-quality modes is suppressed.

8. The missing modality multimodal learning method based on quality-aware cue modulation according to claim 1, characterized in that: Step 7 specifically involves: The fusion weights of each modality are derived from the overall quality score through a piecewise smoothed Sigmoid mapping function: slope parameter Controlling the steepness of the mapping, the midpoint of the mass. Determines the central location and upper and lower bounds of the weight change. and Constrain the weight range to ensure that low-quality modes still retain a certain fusion weight; based on this, confidence gating is introduced for secondary correction. When the confidence level is insufficient, the weights are automatically reduced to equal weights; the parameters of the pre-trained visual-language encoder are frozen throughout the process, and only the prompt parameters and each auxiliary network are updated.

9. A missing modality multimodal learning device based on quality-perceived cue modulation, characterized in that: Includes the following modules: Module 1: Detects the modality missing type of the input sample. When a modality missing is detected, it generates the feature representation of the missing modality to complete the missing modality. Module 2: Input the completed modal features into the task relevance evaluator, calculate the contribution of each modality to the prediction of downstream tasks, and output the task relevance quality score. Module 3: Calculates the marginal impact of each mode on the final prediction result through gradient analysis and outputs the decision influence quality score; Module 4: Evaluate the intrinsic authenticity of each modality using an adversarial discriminator, and output an intrinsic authenticity quality score based on the discriminant confidence. Module 5: The three-dimensional quality scores of task relevance, decision influence, and intrinsic authenticity are fused through a learnable weighting mechanism to output a comprehensive quality score for each modality; Module 6: Select the corresponding initial learnable cue vector according to the sample missing type, generate a modulation factor from the comprehensive quality score through the mapping network, and apply it to the cue vectors of each layer of the pre-trained visual-language encoder so that the cue intensity is continuously adjusted with modality reliability; Module 7: Extracts modal features guided by prompts through a pre-trained visual-language encoder, transforms the comprehensive quality score into fusion weights using a piecewise smooth mapping function, and then performs weighted fusion of the dual-modal features after confidence gating adjustment before feeding them into the classification head to complete the task prediction.

10. A computer, comprising a processor and a storage medium, characterized in that, When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1.

Images