Unbiased missing modal learning method based on multi-stage double diffusion network

Abstract

The invention provides an unbiased missing modal learning method based on a multi-stage double diffusion network, and belongs to the field of computer vision, natural language processing and multi-modal information fusion. The method comprises the following steps: a multi-modal feature extraction module maps text modal data, image modal data and audio modal data into a unified potential representation space; the potential space representation of the three modals is used as the original feature of the missing modality and the original feature of the available modality; based on the missing modal original features and the available modal original features, training a multi-stage double-diffusion module through forward diffusion and reverse diffusion to obtain a trained multi-stage double-diffusion module, and based on the trained multi-stage double-diffusion module, performing global structure generation and modal conversion through available modal data to obtain a multi-stage double-diffusion model; and performing local detail optimization reasoning to generate missing modal data. According to the method, the problem of modal generation deviation in the existing multi-modal learning is solved, and the effectiveness of the multi-modal learning effect in the missing scene is enhanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, natural language processing and multi-modal information fusion, and particularly relates to a non-biased missing modality learning method based on a multi-stage double diffusion network. BACKGROUND

[0002] Multi-modal learning exhibits strong capability in modeling complex real-world scenarios by fusing complementary information from heterogeneous data sources (e.g., images, audios, and texts), and is widely applied in fields such as robot perception and control, visual question answering (VQA) and affective computing. The core idea is to jointly model multiple perception modalities, thereby enhancing the expressiveness and robustness of the model. Traditional multi-modal methods usually assume that complete modality combinations are available in both training and inference stages. However, in practical applications, modality missing is ubiquitous due to sensor failure, data loss or limited acquisition conditions. This modality missing phenomenon significantly affects model performance and has become a key bottleneck for multi-modal system deployment.

[0003] To alleviate this problem, early research mainly adopts two types of strategies: one is to discard incomplete samples, and the other is to utilize existing modality information to complete the missing modality. Traditional completion methods mostly rely on shallow models or simple generation mechanisms, which are difficult to handle complex non-linear relationships between modalities. In recent years, deep learning methods such as autoencoders and generative adversarial networks (GANs) have been used for modality recovery, which has improved the quality of completion to some extent. However, in scenarios with a large number of modalities and high missing proportions, these methods are prone to introduce noise or inconsistency, limiting the performance of the model.

[0004] As a new generation of powerful generative model, diffusion model has made remarkable progress in image inpainting, 3D modeling and multi-modal learning in recent years. In the modal recovery task, it gradually shows better modeling ability than traditional generative methods. For example, the IMDER model recovers the missing modal from Gaussian noise based on the diffusion process and achieves excellent performance in the emotion recognition task (Yuanzhi Wang, Yong Li, and Zhen Cui. Incomplete multimodality-diffused emotion recognition. Advances in Neural Information Processing Systems, 36:17117-17128, 2023). The AMM-Diff model proposes an adaptive diffusion mechanism that integrates image and frequency information for medical image modal completion, effectively improving the accuracy of medical image segmentation (Aghiles Kebaili, Jéróme Lapuyade-Lahorgue, Pierre Vera, and Su Ruan. Amm-diff: Adaptive multi-modality diffusion network for missing modality imputation. arXiv preprint arXiv:2501.12840, 2025.3).

[0005] Although the diffusion model significantly improves the modal completion ability, there are still key challenges: modal generation bias. Due to the heterogeneity of multi-modal data, the quality of model generation on different modalities is significantly different. Visual modalities (such as images, videos) are easy to model accurately due to their high-dimensional continuous feature space; while language modalities are more prone to semantic ambiguity and inconsistency due to their discrete symbolic structure. In addition, most existing methods use unified modeling architecture and shared attention mechanism, which fail to fully capture the feature space differences between modalities, making it difficult to effectively coordinate the structural conflicts between the continuity of visual modalities and the compositionality of text modalities. This modal generation bias has become a core problem that restricts the further development of current multi-modal diffusion models. SUMMARY

[0006] The purpose of the present application is to provide a multi-stage double diffusion network-based unbiased missing modal learning method to solve the technical problem of modal generation bias caused by modal heterogeneity in multi-modal learning in the prior art.

[0007] To solve the above technical problems, the specific technical solutions of the present application are as follows:

[0008] An unbiased missing modal learning method based on a multi-stage double diffusion network, the method comprising the following steps:

[0009] Step A: training phase, the multi-modal feature extraction module maps the text modal data, image modal data and audio modal data into a unified latent representation space; the latent space representations of the three modalities are used as missing modal original features and available modal original features.

[0010] Step B: based on the missing modal original features and the available modal original features, a multi-stage double diffusion module is trained through forward diffusion and reverse diffusion to obtain a trained multi-stage double diffusion module;

[0011] The forward diffusion is based on a variance-preserving diffusion model, and through the injection of noise, modal alignment and Gaussian noise degradation, the missing modal original features and the available modal original features are destroyed into noisy missing modal data and noisy available modal data through modal conversion cross connection in the diffusion process of the missing modal data and the available modal data;

[0012] The reverse diffusion is a variational fractional model that constructs a standard stochastic differential equation of reverse time, and generates missing modal recovery features and available modal recovery features from noisy missing modal data and noisy available modal data through three stages of global structure generation, modal conversion and local detail optimization;

[0013] Step C: inference phase, based on the trained multi-stage double diffusion module, the missing modal data is generated by inference based on the available modal data.

[0014] Further, step A comprises the following steps:

[0015] Step A1: obtaining multi-modal data from multiple heterogeneous data sources, the multi-modal data including text modal data, image modal data and audio modal data;

[0016] Step A2: a pre-trained encoder extracts latent space representations from the multi-modal data; the text modal data, image modal data and audio modal data are respectively mapped to latent space representations of a unified dimension through a BERT encoder, a Facet encoder and a COVAREP encoder.

[0017] Step A3: extracting the latent space representations of the three modalities as missing modal original features and available modal original features.

[0018] Further, the forward diffusion is divided into (0, t1], (t1, t2] and (t2, T] three stages, and each stage performs the following steps:

[0019] Step B11: The first stage is the noise injection stage, t∈(0,t1], based on the discretization form of VP-SDE, to inject noise into the original features of the missing modality. Original features of available modalities Noise is injected to obtain noisy missing modal data. and noisy available modal data

[0020] Step B12: The second stage is the modality alignment stage, t∈(t1,t2], containing noisy and missing modality data. and noisy available modal data After transitioning across time steps until t2, mode alignment is performed using the model mode alignment coefficient Φ.

[0021] Step B13: The third stage is the Gaussian noise degradation stage, t∈(t2,T]. Noise is injected based on the discretization form of VP-SDE, ultimately transforming the noisy usable modal data. and noisy missing modal data The destruction is Gaussian noise.

[0022] Furthermore, in the t∈(0,t1] and t∈(t2,T] stages, noisy missing modal data Forward diffusion is:

[0023]

[0024] in, This represents the noisy, missing modal data at time step t. β represents the noisy missing modal data at time step t-1. t ∈(0,1) is the noise intensity coefficient. It is a standard Gaussian noise vector;

[0025] Noisy available modal data Forward diffusion is:

[0026]

[0027] in, This represents the noisy available modal data at time step t. This represents the noisy available modal data at time step t-1;

[0028] Noisy missing modal data in the t∈(t1,t2] stage. Forward diffusion is:

[0029]

[0030] Where Φ is the model modal alignment coefficient;

[0031] Noisy available modal data Forward diffusion is:

[0032]

[0033] Furthermore, the back diffusion is divided into three stages: [T,t2), [t2,t1), and [t1,0). Each stage performs the following steps:

[0034] Step B21: The first stage is the global structure generation stage, t∈[T,t2), which generates the structure from noisy and missing modal data. With noisy available modal data Recover missing modal data and available modal data;

[0035] Step B22: The second stage is the modality transformation stage, t∈[t2,t1), which aligns the missing modality data with the available modality data through cross-transformation between modalities;

[0036] Step B23: The third stage is the local detail optimization stage, t∈[t1,0), from noisy missing modal data With noisy available modal data Recover missing modal data and available modal data.

[0037] Furthermore, in the t∈[T,t2) and t∈[t1,0) stages, noisy missing modal data The reverse diffusion is:

[0038]

[0039] in, To pass through approximation From noisy missing modal data Predicting missing modality recovery features The time-varying fractional model, where y represents the other generated modes, is based on the discretized form of the reverse-time VP-SDE; It is a standard Gaussian noise vector;

[0040] Noisy available modal data The reverse diffusion is:

[0041]

[0042] in, To pass through approximation From noisy available modal data Predictable available modal recovery features Time-varying fractional model;

[0043] In the stage t∈[t2,t1), arrive The conversion process is as follows:

[0044]

[0045] and arrive The conversion process is as follows:

[0046]

[0047] Further, step B3 includes the following steps:

[0048] Step B31: Calculate separately and The score matching loss;

[0049] Step B32: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the and The score matching losses are summed to obtain the score loss;

[0050] Step B33: Use a decoder Reconstruct the generated data and calculate the reconstruction loss;

[0051] Step B34: Combine the fractional loss and reconstruction loss to obtain the total target loss of the multi-stage dual diffusion module;

[0052] Step B35: Optimize the time-varying score model by training the total objective loss to obtain the trained multi-stage double diffusion module.

[0053] Furthermore, in step C, the inference generation is divided into three stages: global structure generation, mode transformation, and local detail optimization.

[0054] Step C1: For available modal data One-dimensional convolution yields the original features of the usable modalities. Available modal primitive features After noise injection and mode alignment, noisy usable mode data is obtained. Sampled Gaussian noise As input for the missing modality;

[0055] Step C2: In the global structure generation stage, t∈[T,t2), the missing modal data are generated by backdiffusion according to the discretization form of the reverse time VP-SDE;

[0056] Step C3: During the mode transition phase t∈[t2,t1), through arrive One-way modal transformation is used to obtain modally aligned missing modal data;

[0057] Step C4: In the local detail optimization stage t∈[t1,0), from the noisy missing modality data Generate missing modal data;

[0058] Step C5: Use a decoder Obtain the decoded missing modal data

[0059] Furthermore, the decoded missing modal data is obtained. It can also be used for multimodal fusion, including the following steps:

[0060] Step D1: Decode the missing modal data and available modal data The data is fused into complete multimodal data using a multimodal Transformer. Used for downstream multimodal fusion and prediction tasks;

[0061] Step D2: Predict the result using fused features via a multilayer perceptron;

[0062] Step D3: Integrate the loss functions to construct the total loss function.

[0063] Compared with the prior art, the present invention has the following beneficial technical effects:

[0064] (1) Compared with traditional diffusion models, this invention innovatively solves the problem of mode generation bias caused by modal heterogeneity in existing multimodal learning. Based on the solution of this invention, the model achieves unbiased modeling and high-quality recovery of missing modes through mutual optimization of available modes and missing modes, thereby enhancing the effectiveness of multimodal learning in missing scenarios.

[0065] (2) In the multi-stage dual-diffusion network module, the generation task is decomposed into three stages: global structure generation, mode transformation, and local detail optimization. This ensures the consistency of the global structure while enhancing the representation of local details. In the critical mode transformation stage, the model can dynamically control noise and learn to fuse mode-invariant representations shared across modes through a bidirectional symmetric transformation mechanism, thereby reducing generation bias and enhancing overall recovery capability. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1This is a flowchart of the overall model in an embodiment of the present invention.

[0068] Figure 2 This is a flowchart of the multimodal feature extraction module in an embodiment of the present invention.

[0069] Figure 3 This is a flowchart illustrating the training process of the multi-stage dual-diffusion network module in an embodiment of the present invention.

[0070] Figure 4 This is a flowchart illustrating the inference process of the multi-stage dual-diffusion network module in an embodiment of the present invention.

[0071] Figure 5 This is a flowchart of the multimodal fusion module in an embodiment of the present invention.

[0072] Figure 6 This is a comparison with state-of-the-art methods on CMU-MOSI and CMU-MOSEI in scenarios with random missing values. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] This invention proposes a multi-stage dual-diffusion network, such as... Figure 1 As shown, its core is to achieve unbiased modeling and high-quality recovery of missing modalities through a structurally symmetric and modal-cooperative generation mechanism. The multi-stage dual-diffusion network includes a multi-modal feature extraction module, a multi-stage dual-diffusion module, and a multi-modal fusion module.

[0075] The multimodal feature extraction module processes text, image, and audio modalities respectively through three independent modality-specific pre-trained encoders. The encoders map the raw modal data to a unified latent representation space. This design ensures the consistency of multimodal features in the initial representation stage, while extracting discriminative low-dimensional information from a single source.

[0076] The multi-stage dual-diffusion module employs a dual-diffusion generation network. It reconstructs each other's data through cross-modal collaboration using a variance-preserving diffusion model (VPDM), and introduces joint optimization of score matching loss and reconstruction loss to ensure that each modality contributes unique information, achieving more accurate and unbiased generation of missing modalities. The variance-preserving diffusion model is a type of diffusion model whose forward process is based on variance preserving (VP). Based on the dual-diffusion network, the training process uses complete multimodal data. In the forward diffusion stage, based on the discretized form of VP-SDE (Variance Preserving-Stochastic Differential Equation), it injects three asymptotic perturbations: noise, modality alignment, and noise destruction. In the backward diffusion stage, it learns the data distribution through a parameterized score network and jointly optimizes the network using score matching loss and reconstruction loss. This module gradually mitigates modality generation bias through three generation stages, achieving balance restoration between modalities. First, the global structure generation stage captures the macroscopic structural features of missing modalities. The modality transition stage avoids premature dominance of any single modality and achieves cross-modal semantic alignment by transferring and exchanging information between available and missing modalities. Finally, the fine-grained features of the generated samples are enhanced in the local detail optimization stage.

[0077] The multimodal fusion module fuses multimodal representations after generation and uses a multimodal transformer for downstream prediction.

[0078] Based on the aforementioned multi-stage dual-diffusion network, this invention proposes an unbiased missing mode learning method based on a multi-stage dual-diffusion network, solving the mode generation bias problem in missing mode recovery during multi-modal learning. For example... Figure 1 As shown, the method includes the following steps:

[0079] Step A: During the training phase, the multimodal feature extraction module maps text modality data (L), image modality data (V), and audio modality data (A) into a unified latent representation space. The latent space representations of the three modalities serve as the original features of the missing modality and the original features of the available modality, such as... Figure 2 As shown.

[0080] Step A1: Obtain multimodal data from multiple heterogeneous data sources, including text modal data (L), image modal data (V), and audio modal data (A).

[0081] Step A2: The pre-trained encoder extracts latent spatial representations from the multimodal data.

[0082] Different encoders were pre-trained for text modal data, image modal data, and audio modal data. The pre-trained BERT encoder extracted text modal data, the Facet encoder extracted image modal data, and the COVAREP encoder extracted audio modal data.

[0083] During training, text modal data, image modal data, and audio modal data are mapped to a unified-dimensional latent space representation χ={x} through the BERT encoder, Facet encoder, and COVAREP encoder, respectively. k}, where k represents the corresponding mode, k∈{L,V,A}; x k This represents the latent space representation of the corresponding modal data after being mapped by the encoder. S is the sequence length, and D is the dimension of the latent space.

[0084] Step A3: Extract the latent space representations of the three modalities as the original features of the missing modalities and the original features of the available modalities.

[0085] During the training phase, the missing modality original features and the available modality original features constitute the complete modality data for three modalities. The missing modality original features are denoted as... The modal primitive features can be denoted as Right now x L x is the latent space representation of text modal data after being mapped by the encoder. V x is the latent space representation of image modal data after being mapped by the encoder. A This is the latent spatial representation of audio modal data after being mapped by the encoder.

[0086] During the inference phase, the original features of the missing modalities are sampled from Gaussian noise. The original features of a modality are the latent space representations of one or two available modal data after being mapped by the encoder.

[0087] like Figure 3 As shown, step B: Based on the original features of the missing modalities and the original features of the available modalities, a multi-stage dual diffusion module is trained through forward diffusion and backward diffusion to obtain a trained multi-stage dual diffusion module.

[0088] Step B1: Based on the variance-preserving diffusion model (VPDM), the original features of the missing modes are extracted using a multi-stage dual diffusion module. and available modal primitive features The disruption occurs in two forms of noise, which cross-connect through mode transitions during the diffusion of missing modal data and the diffusion of available modal data. Forward diffusion is divided into three stages: (0,t1], (t1,t2], and (t2,T].

[0089] Step B11: The first stage is the noise injection stage, t∈(0,t1], based on the discretization form of VP-SDE, to inject noise into the original features of the missing modality. Original features of available modalities Noise is injected to obtain noisy missing modal data. and noisy available modal data

[0090] Noisy missing modal data at this stage Forward diffusion is:

[0091]

[0092] in, This represents the noisy, missing modal data at time step t. β represents the noisy missing modal data at time step t-1. t ∈(0,1) is the noise intensity coefficient. This is a standard Gaussian noise vector used for feature perturbation.

[0093] Noisy modal data can be used at this stage. Forward diffusion is:

[0094]

[0095] in, This represents the noisy available modal data at time step t. This represents the noisy available modal data at time step t-1.

[0096] Step B12: The second stage is the modality alignment stage, t∈(t1,t2], containing noisy and missing modality data. and noisy available modal data After transitioning across time steps until t2, mode alignment is performed using the model mode alignment coefficient Φ.

[0097] Noisy missing modal data at this stage Forward diffusion is:

[0098]

[0099] Where Φ is the model modal alignment coefficient; Represents the original features of the missing mode; Represents the original features of the available modes.

[0100] Noisy modal data can be used at this stage. Forward diffusion is:

[0101]

[0102] Step B13: The third stage is the Gaussian noise degradation stage, t∈(t2,T]. Noise is injected based on the discretization form of VP-SDE, ultimately transforming the noisy usable modal data. and noisy missing modal data The destruction is Gaussian noise.

[0103] Noisy missing modal data at this stage Forward diffusion is:

[0104]

[0105] Noisy modal data can be used at this stage. Forward diffusion is:

[0106]

[0107] Step B2: Based on the Gaussian noise obtained from forward diffusion, use the time-varying fractional model s θ Construct the standard stochastic differential equation (SDE) corresponding to the reverse time, and extract the original features of the missing modes through forward diffusion. Original features of available modalities The probability distribution p0 generates noisy missing modal data. and noisy available modal data p T It follows a standard Gaussian distribution. This is achieved by analyzing noisy, missing modal data. and noisy available modal data Perform backdiffusion to generate missing mode recovery features. Available modal recovery features

[0108] The time-varying score model is implemented using a deep neural network with U-Net as the backbone architecture.

[0109] The reverse diffusion is divided into three stages: [T,t2), [t2,t1), and [t1,0). Each stage includes the following steps:

[0110] Step B21: The first stage is the global structure generation stage, t∈[T,t2), which generates the structure from noisy and missing modal data. With noisy available modal data Recover missing modal data and available modal data.

[0111] Noisy missing modal data at this stage The reverse diffusion is:

[0112]

[0113] in, To pass through approximation From noisy missing modal data Predicting missing modality recovery features The time-varying fractional model, where y represents the other generated modes, is based on the discretized form of the reverse-time VP-SDE; This is a standard Gaussian noise vector.

[0114] Noisy modal data can be used at this stage. The reverse diffusion is:

[0115]

[0116] in, To pass through approximation From noisy available modal data Predictable available modal recovery features The time-varying fractional model.

[0117] Step B22: The second stage is the modality transformation stage, t∈[t2,t1), which aligns the missing modality data with the available modality data through cross-transformation between modalities.

[0118] arrive The conversion process is as follows:

[0119]

[0120] and arrive The conversion process is as follows:

[0121]

[0122] Step B23: The third stage is the local detail optimization stage, t∈[t1,0), from noisy missing modal data With noisy available modal data Recover missing modal data and available modal data.

[0123] Noisy missing modal data at this stage The reverse diffusion is:

[0124]

[0125] Noisy modal data can be used at this stage. The reverse diffusion is:

[0126]

[0127] Step B3: Utilize the transfer kernel p in VP-SDE 0t(x(t)∣x(0)), the time-varying fractional model is trained and optimized through the loss function to obtain the trained multi-stage double diffusion module.

[0128] This transfer kernel approximates the conditional distribution of state x(t) at any time t given an initial state x(0). Specifically, the transfer kernel follows a Gaussian distribution, and its mean and covariance are determined by the initial state and the time-accumulated noise function.

[0129]

[0130] in, Let be the attenuation factor for the diffusion process, and β(s) represent the noise scheduling function, which is a monotonically increasing preset function. This indicates that the signal gradually decays over time, and the covariance... This reflects the noise accumulated during the diffusion process.

[0131] Step B31: Based on the transfer kernel p 0t (x(t)|x(0)) transforms to obtain ∈ represents standard Gaussian noise, and λ(t) represents the weighting function that balances the loss at different time steps. Based on this, the fractional matching loss function is obtained, and the fractional matching loss function is used to calculate... and Score matching loss:

[0132]

[0133] in, express The score matching loss, express The score matching loss, This represents the joint expectation of missing modal data, standard Gaussian noise, and uniform time steps. This represents the joint expectation of available modal data, standard Gaussian noise, and uniform time steps. Let λ(t) be a uniform distribution over the time interval [0, T], and let λ(t) be a weighting function that balances the losses at different time steps:

[0134]

[0135] Where I represents the identity matrix.

[0136] Step B32: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the and The score matching losses are summed to obtain the score loss:

[0137]

[0138] in, This indicates the loss of scores.

[0139] Step B33: Use a decoder The reconstructed data is generated, and its reconstruction loss is defined as:

[0140]

[0141] in, Indicates the reconstruction loss; This represents the missing modal data reconstructed by the decoder; This represents the available modal data reconstructed by the decoder.

[0142] Step B34: Combining the fractional loss and reconstruction loss, the total objective loss of the multi-stage dual-diffusion module is obtained as follows:

[0143]

[0144] in, This represents the total target loss of the multi-stage dual-diffusion module.

[0145] Step B35: Train and optimize the time-varying fractional model using the loss function to obtain the trained multi-stage double diffusion module.

[0146] Step C: Inference phase, based on the trained multi-stage dual-diffusion module, missing modality data is generated through available modality data, such as... Figure 4 As shown.

[0147] The reasoning generation is divided into three stages: global structure generation, modality transformation, and local detail optimization.

[0148] Step C1: For available modal data One-dimensional convolution yields the original features of the usable modalities. Available modal primitive features After noise injection and mode alignment, noisy usable mode data is obtained. Sampled Gaussian noise As a missing modal input.

[0149] Available modal data The first stage of forward diffusion involves injecting noise, and the second stage involves mode alignment to obtain noisy, usable mode data. That is, t∈(0,t1] executes noisy available modal data. Forward diffusion:

[0150]

[0151] t∈(t1,t2] Execute noisy available modal data Forward diffusion:

[0152]

[0153] Step C2: In the global structure generation stage, t∈[T,t2), this process generates missing modal data by backdiffusion based on the discretization form of the reverse time VP-SDE.

[0154] Missing modal data The reverse diffusion is:

[0155]

[0156] Step C3: During the mode transition phase t∈[t2,t1), through arrive One-way modal transformation is performed to obtain modally aligned missing modal data.

[0157] arrive The unidirectional mode transition process is as follows:

[0158]

[0159] Step C4: In the local detail optimization stage t∈[t1,0), from the noisy missing modality data Generate missing modal data.

[0160] Modal data is missing at this stage. The reverse diffusion is:

[0161]

[0162] Step C5: Use a decoder Obtain the decoded missing modal data

[0163] like Figure 5 As shown, step D: multimodal fusion.

[0164] Step D1: Decode the missing modal data and available modal data The data is fused into complete multimodal data using a multimodal Transformer. Used for downstream multimodal fusion and prediction tasks.

[0165] Step D2: Predict the results using fused features via multilayer perceptrons (MLPs).

[0166] Step D3: Construct the total loss function by integrating the loss functions:

[0167]

[0168] in, For cross-entropy loss, Let γ represent the total objective loss of the multi-stage dual-diffusion network, where γ is the weight controlling different loss functions.

[0169] After the above steps, the multi-stage dual-diffusion network is constructed. Test data is then processed through three modules for downstream classification prediction, and the accuracy of sentiment classification regression prediction is compared. The modality missing rate settings include two options: 1) Specific modality always missing; 2) Randomly selected missing modalities. 3) Specific modality always missing: systematically discard one or two modalities throughout the evaluation process. 4) Randomly selected missing modalities: define a missing rate to quantify the overall degree of missing modalities in the sample.

[0170]

[0171] Where, r miss The missing rate is represented by N, where N is the total number of samples, M is the total number of modalities, and m is the number of modalities. i For the i-th sample, the number of modes actually available is determined by selecting eight r values ​​from the range [0%, 10%, 20%, ..., 70%] in the three-mode case. miss , where 70% represents the maximum approximation rate, while ensuring that at least one usable mode is retained at any given time.

[0172] Accuracy metrics used were ACC7, ACC2, and F1. Sentiment classification datasets were CMU-MOSI and CMU-MOSEI, with each sample labeled with a sentiment score ranging from -3 (strongly negative) to +3 (strongly positive). To validate the effectiveness of the core module, a systematic comparative experiment was conducted, comparing the baseline model, a single-layer VP-SDE, a single-layer multi-stage model, and the complete module.

[0173] Table 1. Validation Table of Model Components for Four Comparison Methods

[0174]

[0175]

[0176] Using the unbiased missing modality learning method based on the multi-stage dual-diffusion network described above, and assuming a fixed missing modality rate, this method is compared with existing high-performance missing modality learning methods. The comparison methods consist of recovery-based and non-recovery-based methods. Recovery-based methods include MCTN, TransM, ICDN, MMIN, GCNet, DiCMoR, and IMDer. Non-recovery-based methods include DCCA and DCCAE. As shown in Table 2, the unbiased missing modality learning method based on the multi-stage dual-diffusion network exhibits the best performance among the recovery methods, and the recovery-based methods consistently outperform the non-recovery methods.

[0177] Table 2. Accuracy comparison of various methods on the CMU-MOSEI dataset.

[0178]

[0179] Under random missing rates, the performance of mainstream methods for handling missing modalities and that of our method are compared, such as... Figure 6 As shown in the figure, the experiment demonstrates that the performance degradation of this method is significantly smaller than that of other methods, proving that it has stronger robustness to modal loss.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for unbiased missing modality learning based on a multi-stage dual-diffusion network, characterized in that, The method includes the following steps: Step A: During the training phase, the multimodal feature extraction module maps text modal data, image modal data, and audio modal data into a unified latent representation space; The latent space representations of the three modalities serve as the original features of the missing modalities and the original features of the available modalities. Step B: Based on the original features of the missing modalities and the original features of the available modalities, train the multi-stage dual diffusion module through forward diffusion and backward diffusion to obtain the trained multi-stage dual diffusion module; Forward diffusion is based on a variance-preserving diffusion model. Through three stages—noise injection, mode alignment, and Gaussian noise degradation—it cross-connects mode transformation during the diffusion of missing modal data and available modal data, destroying the original features of missing modal data and available modal data into noisy missing modal data and noisy available modal data. The backward diffusion time-varying fractional model constructs a backward-time standard stochastic differential equation and generates missing mode recovery features and available mode recovery features from noisy missing mode data and noisy available mode data through three stages: global structure generation, mode transformation, and local detail optimization. Step C: Inference phase, based on the trained multi-stage dual diffusion module, missing modal data is generated through available modal data.

2. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 1, characterized in that, Step A includes the following steps: Step A1: Obtain multimodal data from multiple heterogeneous data sources, including text modal data, image modal data, and audio modal data; Step A2: The pre-trained encoder extracts latent spatial representations from the multimodal data; text modal data, image modal data, and audio modal data are mapped to latent spatial representations of a unified dimension through the BERT encoder, Facet encoder, and COVAREP encoder, respectively. Step A3: Extract the latent space representations of the three modalities as the original features of the missing modalities and the original features of the available modalities.

3. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 2, characterized in that, The forward diffusion is divided into three stages: (0, t1], (t1, i2], and (t2, T]. Each stage performs the following steps: Step B11: The first stage is the noise injection stage, t∈(0,t1], based on the discretization form of VP-SDE, to inject noise into the original features of the missing modality. Original features of available modalities Noise is injected to obtain noisy missing modal data. and noisy available modal data Step B12: The second stage is the modality alignment stage, t∈(t1,t2], containing noisy and missing modality data. and noisy available modal data After transitioning across time steps until t2, mode alignment is performed using the model mode alignment coefficient Φ. Step B13: The third stage is the Gaussian noise degradation stage, t∈(t2,T]. Noise is injected based on the discretization form of VP-SDE, ultimately transforming the noisy usable modal data. and noisy missing modal data The destruction is Gaussian noise.

4. The unbiased missing modality learning method based on a multi-stage dual-diffusion network according to claim 3, characterized in that, Noisy missing modal data in the t∈(0,t1] and t∈(t2,T] stages. Forward diffusion is: in, This represents the noisy, missing modal data at time step t. β represents the noisy missing modal data at time step t-1. t ∈(0,1) is the noise intensity coefficient. It is a standard Gaussian noise vector; Noisy available modal data Forward diffusion is: in, This represents the noisy available modal data at time step t. This represents the noisy available modal data at time step t-1; Noisy missing modal data in the t∈(t1,t2] stage. Forward diffusion is: Where Φ is the model modal alignment coefficient; Noisy available modal data Forward diffusion is:

5. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 4, characterized in that, The reverse diffusion is divided into three stages: [T,t2), [t2,t1), and [t1,0). Each stage performs the following steps: Step B21: The first stage is the global structure generation stage, t∈[T,t2), which generates the structure from noisy and missing modal data. With noisy available modal data Recover missing modal data and available modal data; Step B22: The second stage is the modality transformation stage, t∈[t2,t1), which aligns the missing modality data with the available modality data through cross-transformation between modalities; Step B23: The third stage is the local detail optimization stage, t∈[t1,0), from noisy missing modal data With noisy available modal data Recover missing modal data and available modal data.

6. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 5, characterized in that, Noisy missing modal data in the t∈[T,t2) and t∈[t1,0) stages. The reverse diffusion is: in, To pass through approximation From noisy missing modal data Predicting missing modality recovery features The time-varying fractional model, where y represents the other generated modes, is based on the discretized form of the reverse-time VP-SDE; It is a standard Gaussian noise vector; Noisy available modal data The reverse diffusion is: in, To pass through approximation From noisy available modal data Predictable available modal recovery features Time-varying fractional model; In the stage t∈[t2,t1), arrive The conversion process is as follows: and arrive The conversion process is as follows:

7. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 6, characterized in that, Step B3 includes the following steps: Step B31: Calculate separately and The score matching loss; Step B32: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the and The score matching losses are summed to obtain the score loss; Step B33: Use a decoder Reconstruct the generated data and calculate the reconstruction loss; Step B34: Combine the fractional loss and reconstruction loss to obtain the total target loss of the multi-stage dual diffusion module; Step B35: Optimize the time-varying score model by training the total objective loss to obtain the trained multi-stage double diffusion module.

8. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 7, characterized in that, In step C, inference generation is divided into three stages: global structure generation, mode transformation, and local detail optimization. Step C1: For available modal data One-dimensional convolution yields the original features of the usable modalities. Available modal primitive features After noise injection and mode alignment, noisy usable mode data is obtained. Sampled Gaussian noise As input for the missing modality; Step C2: In the global structure generation stage, t∈[T,t2), the missing modal data are generated by backdiffusion according to the discretization form of the reverse time VP-SDE; Step C3: During the mode transition phase t∈[t2,t1), through arrive One-way modal transformation is used to obtain modally aligned missing modal data; Step C4: In the local detail optimization stage t∈[t1,0), from the noisy missing modality data Generate missing modal data; Step C5: Use a decoder Obtain the decoded missing modal data 9. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 8, characterized in that, Obtain the decoded missing modal data It can also be used for multimodal fusion, including the following steps: Step D1: Decode the missing modal data and available modal data The data is fused into complete multimodal data using a multimodal Transformer. Used for downstream multimodal fusion and prediction tasks; Step D2: Predict the result using fused features via a multilayer perceptron; Step D3: Integrate the loss functions to construct the total loss function.

Images