Missing modal multi-modal learning method based on koopman conditional progressive diffusion

Abstract

A missing modality multimodal learning method based on Koopman conditional progressive diffusion is proposed, relating to computer vision, natural language processing, and multimodal information fusion. The method includes: a Koopman structured multimodal encoder that separates each modality representation into a shared subspace and a specific subspace through Koopman feature decoupling, and applies orthogonal constraints to ensure effective decomposition, suppressing modal bias at its source; a conditional fractional diffusion network that constructs multi-scale noise perturbations in the feature space based on variance amplification forward diffusion, combined with a Koopman structured conditioner to maintain stable conditional signals, and a time-varying fractional network performing inverse diffusion reconstruction, while introducing a Koopman guidance mechanism to maintain cross-modal linear consistency, and using residual channel self-attention to further refine the recovery results; and a Koopman-aware progressive training strategy that achieves phased optimization through sample difficulty measurement and dynamic weight scheduling, enabling the model to first establish a robust structure on simple samples and then improve generalization ability on complex samples.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, natural language processing, and multimodal information fusion, specifically to a missing modality multimodal learning method based on Koopman conditional progressive diffusion (KC-ProgDiff). Background Technology

[0002] Multimodal learning aims to jointly utilize information from heterogeneous data sources such as vision, speech, and text to improve the performance of downstream tasks such as object classification, emotion recognition, and decision making. Existing multimodal methods are generally based on an idealized assumption: that complete data for all modalities can be acquired during both the training and testing phases. However, in real-world applications, this assumption is often difficult to uphold. Due to factors such as heterogeneous acquisition devices, difficulties in modal synchronization, environmental occlusion, cost constraints, and quality control failures, multimodal samples generally exhibit varying degrees of missing modality. Directly discarding samples with missing modalities not only wastes a significant amount of valuable information but also causes a sharp decline in the performance of deep models on under-distributed and biased samples.

[0003] To address the problem of missing modalities, current research mainly follows two approaches: (1) Modal reconstruction methods: These methods construct generative models to infer missing modalities from observed modalities. Typical tools include autoencoders (AEs), variational autoencoders (VAEs), cross-modal generative networks, and cascaded residual reconstructors. These methods require separate generators for different modal combinations, resulting in high training costs and difficulty in maintaining scalability as the number of modalities increases. (2) Shared latent space methods: These methods map available modalities to a unified latent representation and achieve cross-modal feature fusion through collaborative regularization, contrastive learning, or shared encoders. However, due to the significant differences in statistical distribution and structural characteristics among modalities, a unified latent space often struggles to balance fine-grained semantic details with cross-modal alignment constraints.

[0004] In recent years, Conditional Diffusion Models (CDDMs) have demonstrated superior performance in modality recovery. Their progressive noise-inverse denoising generative framework enables flexible conditional sampling and naturally expresses the uncertainties in the recovery process. However, existing CDDM methods still have a key bottleneck: the conditional features input to the score network often carry modality-specific biases, making the denoising trajectory susceptible to the influence of the dominant modality and difficult to recover semantically consistent and structurally stable missing modalities. This defect is particularly prominent in the following two aspects: (1) Modality heterogeneity leads to uneven reconstruction quality: visual modalities have high-dimensional continuous spaces and are relatively easy to model; while language modalities, due to the discreteness of symbols and semantic complexity, are more prone to ambiguity, bias, and semantic drift in the recovery results. (2) Uneven distribution of missing modalities exacerbates the accumulation of biases: in actual data where the frequency of missing modalities is significantly higher than that of other modalities, existing diffusion models tend to rely on the shortcut statistical features of the dominant modality rather than following the invariant structure or causal semantic relationship that should be maintained across modalities. Therefore, there is an urgent need for a novel generative framework that can strip away modality-specific biases, strengthen cross-modality invariant structures, and guide progressive uniform recovery. Summary of the Invention

[0005] To address the modality-specific bias issue in existing multimodal learning conditional diffusion generation models for imbalanced scenarios, the technical solution adopted in this invention is as follows:

[0006] This invention proposes a novel Koopman Conditional Progressive Diffusion (KC-ProgDiff) framework. Its core is achieved through three main modules: a Koopman structured multimodal encoder, a conditional fractional diffusion network, and progressive training based on Koopman difficulty awareness. This framework enables high-quality recovery of missing modalities and improves the performance of downstream tasks.

[0007] The Koopman structured multimodal encoder module decomposes the representation of each modality into two parts through Koopman Feature Decoupling (KFD): (1) a modality-shared subspace, used to capture semantic / anatomical invariant features across modalities; and (2) a modality-specific subspace, used to capture appearance / sequence correlation effects of a single modality. Simultaneously, the module applies regularization constraints to ensure the effectiveness of the decomposition. This design separates the shared components before the diffusion process, eliminating modality-specific bias at its source.

[0008] The Conditional Fractional Diffusion (CSD) network directly generates missing modes by performing diffusion operations in the encoder feature space. First, forward noise is added using variance explosion (VE). A Koopman structured conditioner is constructed to ensure the conditional signal remains stable across different noise scales. Finally, inverse diffusion denoising is performed on all noise scales using a time-varying fractional network. Furthermore, the network introduces a Koopman guided term (KGP) to constrain the denoising trajectory, ensuring cross-modal linear consistency; and refines the inverse diffusion output through residual channel self-attention (RCSA), thereby improving the discriminative power of the reconstruction results.

[0009] The proposed Koopman-aware progressive training module (ADTS) achieves phased training through sample difficulty measurement and dynamic weight scheduling: in the early stage, it focuses on simple samples to stabilize the model, and in the later stage, it strengthens complex samples to improve generalization ability.

[0010] The specific steps of the missing modality multimodal learning method based on Koopman conditional progressive diffusion provided by this invention are as follows:

[0011] A. Multimodal feature extraction and decoupling process:

[0012] Acquire multimodal data and extract features, decouple the features of each modality using Koopman features, and decompose them into a shared subspace and a specific subspace;

[0013] Linear prediction of missing modes is obtained based on the cross-modal linear theorem of Koopman shared subspace;

[0014] B. Conditional fraction diffusion process for generating missing modes:

[0015] A fraction-based diffusion model is constructed, in which the forward process performs a positive perturbation on the true feature representation of the missing modes;

[0016] A Koopman structured conditioner is constructed. The missing modes are based on the Koopman feature decoupling results, linear prediction and perturbation embedding information, and the conditional signals are obtained through the Koopman structured conditioner.

[0017] The time-varying fractional network estimates the conditional fractions of missing modes based on the conditional signal, and performs discretization inverse diffusion at the noise scale together with the Koopman guided fraction;

[0018] The original features output by inverse diffusion are sharpened through the residual channel self-attention layer to obtain a refined vector;

[0019] C. Progressive training based on Koopman-aware difficulty: The weights of each loss function are dynamically adjusted using a weight function, and the corresponding weight function or weights are used to weight each loss function to integrate and construct the total training loss.

[0020] D. Model inference process: Based on the trained complete model, the generated missing modalities are used to complete the downstream classification task.

[0021] Furthermore, the multimodal feature extraction and decoupling process is as follows:

[0022] A1. Acquire multimodal data from multiple heterogeneous data sources, including text (L), image (V), and audio (A) modalities;

[0023] A2. For each modality i, a pre-trained modality-specific encoder is used to map the original data into high-dimensional features. ;in, ;

[0024] A3. Koopman feature decoupling is achieved by using a global projection matrix and a modality-specific projection matrix to decouple high-dimensional features. Decomposed into shared subspace and specific subspace ;

[0025] A4. Based on the Koopman shared subspace cross-modal linearity theorem, use observed samples to form a linear prediction of the missing mode j. :

[0026] ;

[0027] in, For the set of observed modes, For a learned linear mapping;

[0028] A5. To avoid degeneracy and ensure decoupling between shared and specific subspaces, an orthogonal regularization loss is applied.

[0029] Furthermore, the process of generating missing modes through conditional fraction diffusion is as follows:

[0030] B1. A fractional diffusion model is introduced, incorporating a continuous-time stochastic differential equation (SDE). In actual training, this equation follows a variance explosion forward process, decomposing it into multi-step discrete noise-adding operations for missing modes. True feature representation Perform a positive disturbance;

[0031] B2. Target Missing Modality Index With the set of observation modes , For the set of missing modes, construct a Koopman structured conditioner:

[0032]

[0033] in, For fixed-width vector conditional signals, To observe the shared components of modes, For the shared predicted values ​​of the missing modes, To observe the specific components of the mode, For the binary mask of the missing mode, For sinusoidal time embedding, For mapping functions;

[0034] B3. Design a time-varying fractional network, using a feature-based MLP or a shallow Transformer, and inject conditional signals using feature-level linear modulation. This is used to estimate the conditional fraction of the noisy features of the missing mode j at diffusion time t. The corresponding inverse SDE is derived through the forward SDE, enabling the samples to propagate backward. The Euler-Maruyama method is used to discretize the inverse SDE into a finite-step iteration at the noise scale.

[0035] B4. Calculate the target missing mode guidance score using the Gaussian potential function and integrate it into the conditional network. Obtain the Koopman guided score Use Koopman to guide the score for backdiffusion updates;

[0036] B5. For the original features of the above inverse diffusion output, input to the L-layer residual channel self-attention layer. This residual channel self-attention layer applies channel attention to the residual connection. The output of the last layer is refined by average pooling to obtain a refined vector. Reconstruction loss is applied to the refined vector.

[0037] B6. To optimize the learnable parameters, conditional score matching loss and Koopman structure loss are introduced, and combined with orthogonal regularization loss and reconstruction loss, the overall objective function is obtained.

[0038] Furthermore, the progressive training based on Koopman's perceived difficulty is as follows:

[0039] C1. Mission Loss The cross-entropy loss function is used to quantify the classification prediction error based on the recovered multimodal features;

[0040] C2, Combining Conditional Score Matching Loss Koopman residual To assess the diffusion difficulty, where P is the global projection matrix. Features of missing modes The Euclidean norm is used to obtain the diffusion difficulty index. :

[0041]

[0042]

[0043] in, For residual weights; For mathematical expectation, For missing features, The time step is the sampling time from a uniform distribution from 0 to T. for Trainable parameters, It is a positive weighting function that varies with time. Let I be a Gaussian noise vector sampled from a standard normal distribution, and let I be the identity matrix.

[0044] C3. Combining Conditional Score Matching Loss With reconstruction losses To evaluate the reconstruction missing mode level of the entire conditional fraction diffusion module, the conditional fraction diffusion loss is obtained. :

[0045]

[0046]

[0047] in, To balance and The weights of the loss function For refined vectors, To observe the true characteristics of the mode;

[0048] C4. Progressive training uses a weight function. To dynamically adjust sample weights:

[0049]

[0050] in, This is the time-step balance coefficient, which increases with the number of training iterations t, i.e. , To normalize the step size, For the corresponding sample difficulty metrics, such as task loss The corresponding difficulty index is itself And diffusion loss The corresponding difficulty indicators are as described above. ;

[0051] C5. Weight each loss function using the corresponding weight function or weights, and then combine them to construct the total training loss:

[0052]

[0053] Among them, the weight function is used. Dynamically adjust task losses Conditional fractional diffusion loss Loss function, weights and Static equilibrium Koopman structural loss and orthogonal regularization loss .

[0054] The orthogonal regularization loss The calculation is as follows:

[0055]

[0056] in, It is a k-dimensional identity matrix. It is the Frobenius norm. For shared subspace and special subspace The cross-correlation matrix of all coordinate pairs in the matrix;

[0057] The Koopman structural loss The calculation is as follows:

[0058]

[0059] in, To observe the single-mode linear mapping from mode i to the missing mode j, This is the shared subspace of the missing mode j.

[0060] Furthermore, the cross-modal linear theorem based on the Koopman shared subspace is as follows:

[0061] In paired modal scenarios, if the operator of observed mode i... If a column has full rank, then there exists a single-modal linear mapping. Predict the shared coordinates of the missing mode j:

[0062]

[0063] in, , yes Moore-Penrose pseudo-inverse;

[0064] In multi-observation modal scenarios, for the set of observation modes If there are r modes in the stacking operator If it has full column rank, then for any missing mode There exists a multimodal linear mapping Predict its shared coordinates:

[0065]

[0066] in, , For operators with missing mode j, yes Moore-Penrose pseudo-reverse.

[0067] Furthermore, the operations of the mapping function include: standardizing each multidimensional feature coordinate to unit variance; and applying a learned lookup table to the binary mask. Embedding is performed to identify modal missing conditions; a single cross-modal attention block is applied to the observed modality to generate a fixed-width conditional signal vector. .

[0068] Furthermore, the discretized inverse diffusion is specifically as follows:

[0069] The corresponding backward-time stochastic differential equation is derived through forward SDE, enabling backpropagation of samples:

[0070]

[0071] in, For Brownian motion, This is the standard Wiener process after time reversal. From the trained time-varying fractional network approximate; For drift term, The diffusion coefficient is denoted as .

[0072] Let the number of diffusion steps be K, and the Euler-Maruyama method be used on the noise scale. The inverse SDE will be discretized into a finite-step iteration:

[0073]

[0074] in, for Step noise standard deviation Indicates that time is moving in reverse. It is random noise;

[0075] For target missing mode Based on the linear prediction values ​​of its shared components ,exist Define the Gaussian potential function on the shared coordinates. Used to measure shared components in known observation modes Under these conditions, the denoising result of this step Does it meet the linear consistency requirement?

[0076]

[0077] in, This is the covariance matrix, used to control the degree of fit between shared components and predicted values;

[0078] Solve for the Gaussian potential function with respect to... The score:

[0079]

[0080] The shared space score and conditional network The predicted data scores are combined to obtain the Koopman guided score. :

[0081]

[0082] in, To guide the intensity; in the finite-step iteration of inverse SDE discretization, Replace with .

[0083] Furthermore, the residual channel self-attention layer is specifically as follows:

[0084] First, we process the input features of the current l-th layer. Channel-level global features are obtained by compressing the sequence dimension through average pooling:

[0085]

[0086] in, For the compression features of the l-th layer, Indicates average pooling;

[0087] The weight matrix can be learned through queries, keys, and values. From the compression features respectively and the original input features Generate query ,key ,value vector:

[0088]

[0089] The attention weights between channels are calculated by querying the inner product of the key and the query result. Based on this, a weighted sum is performed on the values:

[0090]

[0091] in, For the attention enhancement feature of the r-th head;

[0092] The outputs of all attention heads are concatenated and projected through a matrix. Unified Dimension:

[0093]

[0094] Add the attention-enhanced features directly to the input features:

[0095]

[0096] After L layers of RCSA iterations, a refined vector is obtained through average pooling:

[0097]

[0098] in, This is a vectorization operation.

[0099] Furthermore, the complete model reasoning process is as follows:

[0100] Based on the complete model trained above, the inference process is divided into Koopman feature decoupling, conditional score diffusion network generation of missing modalities, and RCSA refinement to obtain the generated missing modalities, which are used to complete the downstream classification task.

[0101] D1. Given an input with a mask b, use a Koopman structured multimodal encoder to encode the observed modes into... It also decouples the computation of the shared subspace and the specific subspace of the observed modes, and then predicts the shared coordinates of the missing modes;

[0102] D2. Initialize missing modes. During backdiffusion, use Koopman guided fractions and conditionals to generate missing modes, and then use the RCSA refiner to refine them to obtain the desired results. ;

[0103] D3. Completing modal data Decode it or input it into the header of a downstream task.

[0104] The beneficial effects of this invention are:

[0105] (1) Compared with the traditional conditional diffusion model, this invention innovatively solves the modality bias problem that is prone to occur in the unbalanced missing data scenario. This invention introduces Koopman feature decoupling (KFD), which decomposes the modality representation into shared and specific subspaces, filters modality-specific noise from the source, and combines Koopman guiding terms to constrain the denoising trajectory, avoiding the denoising process from biasing towards the dominant modality, thereby ensuring the consistency of the missing modality with the original data in terms of distribution and semantics, and improving the robustness in the unbalanced missing data scenario.

[0106] (2) In the Koopman Conditional Progressive Diffusion (KC-ProgDiff) framework, the Conditional Diffusion Network (CSD) is used in conjunction with Adaptive Dynamic Training (ADTS). First, simple samples are used to anchor the shared subspace to stabilize the missing reconstruction model, and then complex samples are used to optimize the quality of diffusion generation. At the same time, the framework is supplemented with Residual Channel Self Attention (RCSA) to achieve refinement of reconstruction results (DRR), which synergistically improves the quality of missing modality recovery and the performance of downstream tasks, and enhances the adaptability of the model to different missing scenarios. Attached Figure Description

[0107] Figure 1 This is a flowchart of the overall model in an embodiment of the present invention.

[0108] Figure 2 This is a flowchart of the Koopman structured multimodal encoder in an embodiment of the present invention.

[0109] Figure 3 This is a flowchart of the conditional fraction diffusion network in an embodiment of the present invention.

[0110] Figure 4 This is a flowchart of the progressive training module based on Koopman-based difficulty perception in an embodiment of the present invention.

[0111] Figure 5 This is a complete model reasoning flowchart in an embodiment of the present invention.

[0112] Figure 6 This paper compares the present invention with state-of-the-art methods in uncertain missing scenarios on four multimodal datasets. Detailed Implementation

[0113] This embodiment proposes a Koopman conditional progressive diffusion (KC-ProgDiff) framework. It addresses the mode generation bias problem in missing mode recovery during multimodal learning. Figure 1 As shown, the method comprises three core modules: a Koopman structured multimodal encoder, a conditional score diffusion network, and progressive training based on Koopman-aware difficulty.

[0114] The Koopman structured multimodal encoder module decomposes the representation of each modality into two parts through Koopman feature decoupling (KFD): (1) a modality-shared subspace for capturing semantic / anatomical invariant features across modalities; and (2) a modality-specific subspace for capturing appearance / sequence correlation effects of a single modality. Simultaneously, the module applies regularization constraints to ensure the effectiveness of the decomposition. This design separates the shared components before the diffusion process, eliminating modality-specific bias at its source.

[0115] Conditional Fractional Diffusion (CSD) networks directly generate missing modes through diffusion operations in the encoder feature space. First, forward denoising is performed using variance explosion (VE). A Koopman structured conditioner is constructed to ensure the conditional signal remains stable across different noise scales. Finally, inverse diffusion denoising is performed on all noise scales using a time-varying fractional network. Furthermore, the network introduces a Koopman guiding term to constrain the denoising trajectory, ensuring cross-modal linear consistency; and refines the inverse diffusion output through residual channel self-attention (RCSA), thereby improving the discriminative power of the reconstruction results.

[0116] Based on the Koopman-aware progressive training module (ADTS), phased training is achieved through sample difficulty measurement and dynamic weight scheduling: the early stage focuses on simple samples to stabilize the model, and the later stage strengthens complex samples to improve generalization ability.

[0117] The missing modality multimodal learning method based on Koopman conditional progressive diffusion in this embodiment consists of three parts, which are described in detail below with reference to the accompanying drawings:

[0118] A. See Figure 2 The multimodal feature extraction and decoupling process includes:

[0119] A1. Acquire multimodal data from multiple heterogeneous data sources, including text (L), image (V), and audio (A) modalities;

[0120] A2. For each mode i, a pre-trained modality-specific encoder is used. , the original data Mapping to high-dimensional features ;

[0121] in, d represents the original high-dimensional feature dimension. Specifically, the Transformer-based bidirectional encoder representation (BERT) can be used to extract text modalities, the Vision-Language Transformer (ViLT) can be used to extract visual modalities, and the Collaborative Speech Analysis Library (COVAREP) can be used to extract audio modalities.

[0122] A3 and Koopman feature decoupling are performed using global projection matrices respectively. With modality-specific projection matrix High-dimensional features Decomposed into shared subspace and specific subspace Where k is the dimension of the shared subspace, then kd is the dimension of the specific subspace;

[0123] A4. Based on the Koopman shared subspace cross-modal linearity theorem, use partial observation samples (x,b) to form a linear prediction of the missing mode j. :

[0124]

[0125] in, For the set of observed modes, For a learned linear mapping; specifically including:

[0126] A41. In a paired modal scenario, if the operator of observed mode i... If a column has full rank, then there exists a single-modal linear mapping. Predict the shared coordinates of the missing mode j:

[0127]

[0128] in, For the shared subspace of the missing mode j, , yes (Regularized) Moore-Penrose pseudoinverse;

[0129] A42. In multi-observation modal scenarios, for the set of observation modes r modes in For an operator of mode i, if the stacking operator If it has full column rank, then for any missing mode , For the set of missing modes, there exists a multimodal linear mapping. Predict its shared coordinates:

[0130]

[0131] in, , For operators with missing mode j, yes (Regularized) Moore-Penrose pseudoinverse;

[0132] A5. To avoid degradation and ensure decoupling between shared and specific subspaces, an orthogonal regularization loss is applied. :

[0133]

[0134] in, It is a k-dimensional identity matrix. The Frobenius norm is used here to make the projection matrix... Orthogonal normalization )for and The cross-correlation matrix of all coordinate pairs in the matrix (i.e., the standardized Pearson correlation coefficient).

[0135] B. See also Figure 3 The conditional fraction diffusion process for generating missing modes includes:

[0136] B1. Segment-based diffusion models (SDMs) introduce a continuous-time stochastic differential equation (SDE). In actual training, they follow the variance explosion (VE) forward process to decompose the equation into multi-step discrete noise addition operations, performing positive perturbations on the true feature representation of the missing mode j; specifically including:

[0137] B11, SDMs introduced a system by Continuous-time diffusion process of the index Where t is the time index, T is the diffusion termination time, and the positive perturbation follows an Itô stochastic differential equation from the sample. Evolved into ,in, For the original data distribution, It follows a standard Gaussian distribution:

[0138]

[0139] in, For Brownian motion, This is the drift term, used to control for deterministic trends. The diffusion coefficient is used to control noise intensity.

[0140] B12. In actual training, the variance explosion (VE) forward process decomposes the continuous-time stochastic differential equation (SDE) into multi-step discrete noise-adding operations.

[0141] B2. Based on the Koopman feature decoupling results and the embedding information after positive noise addition, a Koopman structured conditioner is constructed for missing mode recovery. This is achieved through a mapping function. Perform coordinate normalization, cross-modal attention fusion, and finally output a fixed-width vector conditional signal. Specifically, this includes:

[0142] B21. Target Missing Modality Index With the set of observation modes Construct a Koopman structured conditioner:

[0143]

[0144] in, To observe the shared components of modes, For the shared predicted values ​​of the missing modes, To observe the specific components of the mode, b is the binary mask for the missing mode. Sinusoidal time embedding is used to capture time information at different noise scales;

[0145] B22, Mapping Function The operations include: (1) standardizing each multidimensional feature coordinate to unit variance; (2) embedding the binary mask b using a learned lookup table to identify modality missing conditions; and (3) applying a single cross-modal attention block to the observed modality to generate a fixed-width conditional signal vector. ;

[0146] B3. Based on Time-Varying Fractional Networks We construct stochastic differential equations corresponding to the reverse time and use the Euler-Maruyama method on the noise scale. The following steps involve discretization and inverse diffusion; specifically:

[0147] B31. Design a time-varying fractional network Using feature-based MLPs or shallow Transformers, feature-level linear modulation (FiLM) is employed to inject conditional signals. This is used to estimate the noisy features of the missing mode j at diffusion time t. Conditional score:

[0148]

[0149] in, for The conditional probability distribution, Indicates noisy features Find the gradient of the logarithmic density function. for Trainable parameters;

[0150] B32. Derive the corresponding inverse-time stochastic differential equation through forward SDE, so that from the sample Backpropagation generation :

[0151]

[0152] in, This is the standard Wiener process after time reversal. From the trained score network approximate;

[0153] B33. Using the Euler-Maruyama method at the noise scale The inverse SDE will be discretized into a finite-step iteration:

[0154]

[0155] in, for Step noise standard deviation Indicates that time is moving in reverse. For random noise. Set the number of diffusion steps. Through geometric scheduling control To balance reconstruction accuracy and computational cost in the feature space;

[0156] B4. To prevent the backdiffusion trajectory from deviating from the linear relationship of the shared subspace, a lightweight Koopman guidance mechanism is introduced. The guidance score is calculated using a Gaussian potential function and integrated into the conditional network. Obtain the Koopman guided score Then it is used in reverse updates; specifically including:

[0157] B41. Target Missing Mode Based on the linear prediction values ​​of its shared components ,exist Define the Gaussian potential function on the shared coordinates. Used to measure shared components in known observation modes Under these conditions, the denoising result of this step Does it meet the linear consistency requirement?

[0158]

[0159] in, This is the covariance matrix, used to control the degree of fit between shared components and predicted values;

[0160] B42. Solve for the Gaussian potential function with respect to... The fraction (log density gradient):

[0161]

[0162] B43. This shared space score and conditional network The predicted data scores are combined to obtain the Koopman guided score. :

[0163]

[0164] in, For guiding strength;

[0165] B44. In the reverse diffusion update shown in B33, only the following needs to be done: Replace with This allows you to perform the following steps at each step: Pushing towards affine sets ;

[0166] B5. Even with a moderate number of inverse diffusion steps, feature space diffusion may still produce an overly smoothed representation. Therefore, a lightweight refiner is added to refine the original features output by the inverse diffusion. Sharpening yields refined vectors Specifically, this includes:

[0167] B51. Regarding the original characteristics of the above-mentioned reverse diffusion output This is reshaped into the 0th layer feature matrix used as input to the refiner. ,in If the feature is a vector, then ;

[0168] B52. Next, use multiple RCSA (Residual Channel Self-Attention) layers, for each layer... With each attention head Channel attention and residual connections are applied. First, the current input features of the l-th layer are processed... Channel-level global features are obtained by compressing the sequence dimension through average pooling:

[0169]

[0170] in, For the compression features of the l-th layer, This indicates average pooling.

[0171] B53. Weight matrices can be learned through queries, keys, and values. From the compression features respectively and the original input features Generate a vector of query Q, key K, and value V:

[0172]

[0173] B54. Calculate the attention weights between channels by querying the inner product of Q and key K. Based on this, a weighted sum is performed on the value V:

[0174]

[0175] in, For the attention enhancement feature of the r-th head;

[0176] B55. Concatenate the outputs of all attention heads and apply them through a projection matrix. Unified Dimension:

[0177]

[0178] B56. To avoid the loss of original feature information, residual connections are introduced, directly adding the attention-enhanced features to the input features:

[0179]

[0180] B57. After L layers of RCSA iterations, a refined vector is obtained through average pooling:

[0181]

[0182] in, This is a vectorization operation that restores the features to a one-dimensional vector.

[0183] B58. Refined features obtained Apply reconstruction loss:

[0184]

[0185] in, For the Euclidean norm, To observe the true characteristics of the mode;

[0186] B6. To optimize the learnable parameters, a conditional score matching loss is introduced. and Koopman structural loss Combined with orthogonal regularization loss and reconstruction losses The overall objective function is obtained. Specifically, this includes:

[0187] B61. For missing / observed features Introducing conditional score matching loss :

[0188]

[0189] in, For mathematical expectation, The time step is the sampling time from a uniform distribution from 0 to T. This is a time-varying positive weighting function used to balance the loss at different noise scales. Let I be a Gaussian noise vector sampled from a standard normal distribution, and let I be the identity matrix.

[0190] B62. Introducing Koopman structural loss The objectives include: learning a shared subspace with linear cross-modal relationships, fitting a linear mapping from other shared coordinates to predict a specific shared coordinate, and maintaining consistency between the refined output and the prediction results. The loss function can be expressed as:

[0191]

[0192] B63. Considering the above losses and combining them with orthogonal regularization losses... and reconstruction losses The overall objective function is obtained. :

[0193]

[0194] in, , , To balance the weights of different loss functions.

[0195] C. See also Figure 4 The progressive training process based on Koopman's perceived difficulty includes:

[0196] C1. Mission Loss Cross-entropy loss function is used to quantify the classification prediction error based on recovered multimodal features, and serves as an indicator of the difficulty of samples in downstream tasks.

[0197]

[0198] in, Let n be the total number of categories for the classification task, and n be the category index. This represents the nth element (0 or 1) in the one-hot encoding of the real label. This indicates that the model uses the complete multimodal features reconstructed from the missing modalities to predict the probability value of belonging to the nth class;

[0199] C2, Combining Conditional Score Matching Loss Koopman residual To assess the difficulty of diffusion and obtain a diffusion difficulty index. :

[0200]

[0201] in, For residual weights;

[0202] C3. Combining Conditional Score Matching Loss With reconstruction losses To evaluate the reconstruction missing mode level of the entire conditional fraction diffusion module, the conditional fraction diffusion loss is obtained. :

[0203]

[0204] in, To balance and The weights of the loss function;

[0205] C4. Progressive training uses a weight function. To dynamically adjust sample weights:

[0206]

[0207] in, This is the time-step balance coefficient, which increases with the number of training iterations t, i.e. , To normalize the step size, For the corresponding sample difficulty metrics, such as task loss The corresponding difficulty index is itself And diffusion loss The corresponding difficulty indicators are as described above. ;

[0208] C5. Weight each loss function using the corresponding weight function or weights, and then combine them to construct the total training loss:

[0209]

[0210] Among them, the weight function is used. Dynamic adjustment and Loss function, weights and Static equilibrium and Loss function;

[0211] See D. Figure 5 The complete model inference process includes:

[0212] Based on the complete model trained above, the inference process is divided into Koopman feature decoupling, conditional score diffusion network to generate missing modalities, and RCSA refinement to obtain the generated missing modalities, which are used to complete the downstream classification task.

[0213] D1. Given an input with a mask b, use a Koopman structured multimodal encoder to encode the observed modes into... and decoupled computation and Then predict the shared coordinates of the missing modes. ;

[0214] D2. Initialize missing modes ,exist The process utilizes Koopman guided fractions and conditional backdiffusion to generate missing modes, and then uses the RCSA refiner to refine them to obtain the desired results. ;

[0215] D3. Completing modal data Decode it or input it into the header of a downstream task.

[0216] After the above steps, the Koopman Conditional Progressive Diffusion (KC-ProgDiff) model is constructed. The test data, after being processed by the aforementioned three modules, is used for downstream classification tasks, and the classification accuracy under different scenarios is compared. The modality missing rate setting includes two categories: fixed missing and uncertain missing. Fixed missing means all incomplete samples follow the same missing pattern throughout the evaluation process. Uncertain missing means the missing pattern is uncertain during training and testing; therefore, this invention defines a missing rate. Used to quantify the overall degree of missing modes in a sample:

[0217]

[0218] in, The total number of samples, The total number of modes, For the first The actual number of usable modalities for each sample, in the case of three modalities, is eight selected from the range [0%, 10%, 20%, ..., 70%]. 70% represents the maximum missing rate, while ensuring that at least one usable modality is retained at any given time.

[0219] The experimental datasets and corresponding accuracy metrics were MM-IMDb (F1), Food-101 (Acc), HatefulMemes (AUROC), and IEMOCAP (Acc2 / Acc7), respectively.

[0220] Using the missing modality learning method based on the Koopman conditional progressive diffusion network described above, experiments were conducted with a fixed missing rate of 70%. The ablation results for each component are shown in Table 1. The complete model achieved optimal performance on all metrics across four multimodal datasets, fully validating the effectiveness of the proposed technique. The performance degradation caused by removing the Koopman structure (without Koopman structure) and conditional fractional diffusion (without CSD) was the most significant, indicating that feature decoupling and generation mechanisms constitute the core foundation of this invention. Meanwhile, the absence of other modules, such as the Koopman guidance mechanism (without KGP), residual channel self-attention (without RCSA), and progressive training (without ADTS), all led to varying degrees of performance degradation, confirming their necessity. The synergistic effect of each component significantly improves the robustness of the model in severe missing scenarios.

[0221] Table 1. Validation table of model component effectiveness under 70% missing rate on four multimodal datasets.

[0222]

[0223] With a fixed missing modality setting of 70%, our proposed method is compared with existing high-performance missing modality learning methods, consisting of recovery-based and non-recovery-based methods. Recovery-based methods include Graph Completion Network (GCNet) and Distribution Consistent Modality Recovery (DiCMoR). Non-recovery-based methods include Visual-Language Transformer (VILT), Boundary Aware and Modality Aware Network (MMANet), and Parameter Efficient Adaptation (PEA). As shown in Table 2, our proposed method achieves the best performance across all missing modalities on four datasets. Furthermore, the recovery-based method consistently outperforms the non-recovery method. Modalities include text (L), image (V), and audio (A).

[0224] Table 2 compares the accuracy of each method in all modality missing scenarios on four multimodal datasets.

[0225]

[0226] Under uncertain missing modalities, the performance of our method is compared with that of mainstream methods for handling missing modalities, including the Visual-Language Transformer (VILT), the Visual-Language Model (ViBERT), the Boundary-Aware and Modality-Aware Network (MMANet), the Dynamic Unwrapping Cueing Tuning Model (RobustPT), the Graph Completion Network (GCNet), the Distribution Consistent Modality Recovery (DiCMoR), and the Parameter-Efficient Adaptation (PEA). Figure 6As shown in the experiment, the accuracy of all models gradually decreases with increasing missing data rate, while our method (Ours) consistently maintains the highest accuracy level. However, compared to lower missing data rates, its advantage in the high missing data range (approximately 50%-70%) narrows. This indicates that increasing the missing data rate will affect model performance to some extent, but our method still exhibits relatively strong robustness overall.

Claims

1. A missing modality multimodal learning method based on Koopman conditional progressive diffusion, characterized in that, Includes the following steps: A. Multimodal feature extraction and decoupling process: Multimodal data is acquired and features are extracted. Koopman features are decoupled from each modality feature and decomposed into a shared subspace and a specific subspace. Linear prediction of missing modes is obtained based on the cross-modal linear theorem of Koopman shared subspace; B. Conditional fraction diffusion process generating missing modes: A fraction-based diffusion model is constructed, in which the forward process performs a positive perturbation on the true feature representation of the missing modes; A Koopman structured conditioner is constructed. The missing modes are based on the Koopman feature decoupling results, linear prediction and perturbation embedding information, and the conditional signals are obtained through the Koopman structured conditioner. The time-varying fractional network estimates the conditional fractions of missing modes based on the conditional signals, and performs discretization inverse diffusion at the noise scale together with the Koopman guided fractions; The original features output by inverse diffusion are sharpened through the residual channel self-attention layer to obtain a refined vector; C. Progressive training based on Koopman-aware difficulty: The weights of each loss function are dynamically adjusted using a weight function, and the corresponding weight function or weights are used to weight each loss function to integrate and construct the total training loss. D. Model inference process: Based on the trained complete model, the generated missing modalities are used to complete the downstream classification task; The multimodal feature extraction and decoupling process is as follows: A1. Acquire multimodal data from multiple heterogeneous data sources, including text (L), image (V), and audio (A) modalities; A2. For each modality i, a pre-trained modality-specific encoder is used to map the original data into high-dimensional features. ;in, ; A3. Koopman feature decoupling is achieved by using a global projection matrix and a modality-specific projection matrix to decouple high-dimensional features. Decomposed into shared subspace and specific subspace ; A4. Based on the Koopman shared subspace cross-modal linearity theorem, use observed samples to form a linear prediction of the missing mode j. : ； in, For the set of observed modes, For a learned linear mapping; A5. To avoid degeneracy and ensure decoupling between shared and specific subspaces, an orthogonal regularization loss is applied. The progressive training based on Koopman's perceived difficulty is as follows: C1. Mission Loss The cross-entropy loss function is used to quantify the classification prediction error based on the recovered multimodal features; C2, Combining Conditional Score Matching Loss Koopman residual To assess the diffusion difficulty, where P is the global projection matrix. Features of missing modes The Euclidean norm is used to obtain the diffusion difficulty index. : in, For residual weights; For mathematical expectation, For missing features, The time step is the sampling time from a uniform distribution from 0 to T. for Trainable parameters, It is a positive weighting function that varies with time. Let I be a Gaussian noise vector sampled from a standard normal distribution, and let I be the identity matrix. C3. Combining Conditional Score Matching Loss With reconstruction losses To evaluate the reconstruction missing mode level of the entire conditional fraction diffusion module, the conditional fraction diffusion loss is obtained. : in, To balance and The weights of the loss function, For refined vectors, To observe the true characteristics of the mode; C4. Progressive training uses a weight function. To dynamically adjust sample weights: in, This is the time-step balance coefficient, which increases with the number of training iterations t, i.e. , To normalize the step size, For the corresponding sample difficulty index, the task loss The corresponding difficulty index is itself. And diffusion loss The corresponding difficulty indicators are as described above. ; C5. Weight each loss function using the corresponding weight function or weights, and then combine them to construct the total training loss: Among them, the weight function is used. Dynamically adjust task losses Conditional fractional diffusion loss Loss function, weights and Static equilibrium Koopman structural loss and orthogonal regularization loss .

2. The missing modality multimodal learning method based on Koopman conditional progressive diffusion according to claim 1, characterized in that, The specific process of generating missing modes through conditional score diffusion is as follows: B1. A fractional diffusion model is introduced, incorporating a continuous-time stochastic differential equation (SDE). In actual training, this equation follows a variance explosion forward process, decomposing it into multi-step discrete noise-adding operations for missing modes. True feature representation Perform a positive disturbance; B2. Target Missing Modality Index With the set of observation modes , For the set of missing modes, construct a Koopman structured conditioner: in, For fixed-width vector conditional signals, To observe the shared components of modes, For the shared predicted values ​​of the missing modes, To observe the specific components of the mode, For the binary mask of the missing mode, For sinusoidal time embedding, For mapping functions; B3. Design a time-varying fractional network, using a feature-based MLP or a shallow Transformer, and inject conditional signals using feature-level linear modulation. This is used to estimate the conditional fraction of the noisy features of the missing mode j at diffusion time t. The corresponding inverse SDE is derived through the forward SDE, enabling the samples to propagate backward. The Euler-Maruyama method is used to discretize the inverse SDE into a finite-step iteration at the noise scale. B4. Calculate the target missing mode guidance score using the Gaussian potential function and integrate it into the conditional network. Obtain the Koopman guided score Use Koopman to guide the score for backdiffusion updates; B5. For the original features of the above inverse diffusion output, input to the L-layer residual channel self-attention layer. This residual channel self-attention layer applies channel attention to the residual connection. The output of the last layer is refined by average pooling to obtain a refined vector. Reconstruction loss is applied to the refined vector. B6. To optimize the learnable parameters, conditional score matching loss and Koopman structure loss are introduced, and combined with orthogonal regularization loss and reconstruction loss, the overall objective function is obtained.

3. The missing modality multimodal learning method based on Koopman conditional progressive diffusion according to claim 2, characterized in that, The orthogonal regularization loss The calculation is as follows: in, It is a k-dimensional identity matrix. It is the Frobenius norm. For shared subspace and special subspace The cross-correlation matrix of all coordinate pairs in the matrix; The Koopman structural loss The calculation is as follows: in, To observe the single-mode linear mapping from mode i to the missing mode j, This is the shared subspace of the missing mode j.

4. The missing modality multimodal learning method based on Koopman conditional progressive diffusion according to claim 3, characterized in that, The cross-modal linearity theorem based on the Koopman shared subspace is as follows: In paired modal scenarios, if the operator of observed mode i... If a column has full rank, then there exists a single-modal linear mapping. Predict the shared coordinates of the missing mode j: in, , yes Moore-Penrose pseudo-inverse; In multi-observation modal scenarios, for the set of observation modes If there are r modes in the stacking operator If it has full column rank, then for any missing mode There exists a multimodal linear mapping Predict its shared coordinates: in, , For operators with missing mode j, yes Moore-Penrose pseudo-reverse.

5. The missing modality multimodal learning method based on Koopman conditional progressive diffusion according to claim 4, characterized in that, The operations of the mapping function include: standardizing each multidimensional feature coordinate to unit variance; and applying a learned lookup table to the binary mask. Embedding is performed to identify modal missing conditions; a single cross-modal attention block is applied to the observed modality to generate a fixed-width conditional signal vector. .

6. The missing modality multimodal learning method based on Koopman conditional progressive diffusion according to claim 5, characterized in that, The discretized inverse diffusion is specifically as follows: The corresponding backward-time stochastic differential equation is derived through forward SDE, enabling backpropagation of samples: in, For Brownian motion, This is the standard Wiener process after time reversal. From the trained time-varying fractional network approximate; For drift term, The diffusion coefficient is denoted as . Let the number of diffusion steps be K, and the Euler-Maruyama method be used on the noise scale. The inverse SDE will be discretized into a finite-step iteration: in, for Step noise standard deviation Indicates that time is moving in reverse. It is random noise; For target missing mode Based on the linear prediction values ​​of its shared components ,exist Define the Gaussian potential function on the shared coordinates. Used to measure shared components in known observation modes Under these conditions, the denoising result of this step Does it meet the linear consistency requirement? in, This is the covariance matrix, used to control the degree of fit between shared components and predicted values; Solve for the Gaussian potential function with respect to... The score: Combine the above scores with the conditional network The predicted data scores are combined to obtain the Koopman guided score. : in, To guide the intensity; in the finite-step iteration of inverse SDE discretization, Replace with .

7. The missing modality multimodal learning method based on Koopman conditional progressive diffusion according to claim 6, characterized in that, The residual channel self-attention layer is specifically as follows: First, we process the input features of the current l-th layer. Channel-level global features are obtained by compressing the sequence dimension through average pooling: in, For the compression features of the l-th layer, Indicates average pooling; The weight matrix can be learned through queries, keys, and values. From the compression features respectively and the original input features Generate query ,key ,value vector: The attention weights between channels are calculated by querying the inner product of the key and the query result. Based on this, a weighted sum is performed on the values: in, For the attention enhancement feature of the r-th head; The outputs of all attention heads are concatenated and projected through a matrix. Unified Dimension: Add the attention-enhanced features directly to the input features: After L layers of RCSA iterations, a refined vector is obtained through average pooling: in, This is a vectorization operation.

8. The missing modality multimodal learning method based on Koopman conditional progressive diffusion according to claim 7, characterized in that, The complete model reasoning process is as follows: Based on the complete model trained above, the inference process is divided into Koopman feature decoupling, conditional score diffusion network generation of missing modalities, and RCSA refinement to obtain the generated missing modalities, which are used to complete the downstream classification task. D1. Given an input with a mask b, use a Koopman structured multimodal encoder to encode the observed modes into... It also decouples the computation of the shared subspace and the specific subspace of the observed modes, and then predicts the shared coordinates of the missing modes; D2. Initialize missing modes. During backdiffusion, use Koopman guided fractions and conditionals to generate missing modes, and then use the RCSA refiner to refine them to obtain the desired results. ; D3. Completing modal data Decode it or input it into the header of a downstream task.

Images