An unbiased missing modal learning method based on a multi-stage double diffusion network

Abstract

The application provides an unbiased missing modal learning method based on a multi-stage double diffusion network, and belongs to the fields of computer vision, natural language processing and multi-modal information fusion. The method comprises the following steps: a multi-modal feature extraction module maps text modal data, image modal data and audio modal data into a unified latent representation space; latent space representations of the three modes are used as missing modal original features and available modal original features; based on the missing modal original features and the available modal original features, a multi-stage double diffusion module is trained through forward diffusion and reverse diffusion, and a trained multi-stage double diffusion module is obtained; based on the trained multi-stage double diffusion module, missing modal data is generated through available modal data, global structure generation, modal conversion and local detail optimization inference. The application solves the modal generation deviation problem in existing multi-modal learning, and enhances the effectiveness of multi-modal learning in a missing scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, natural language processing and multi-modal information fusion, and particularly relates to a non-biased missing modality learning method based on a multi-stage double diffusion network. BACKGROUND

[0002] Multi-modal learning exhibits strong capability in modeling complex real-world scenarios by fusing complementary information from heterogeneous data sources (e.g., images, audios, and texts), and is widely applied in fields such as robot perception and control, visual question answering (VQA) and affective computing. The core idea is to jointly model multiple perception modalities, thereby enhancing the expressiveness and robustness of the model. Traditional multi-modal methods usually assume that complete modality combinations are available in both training and inference stages. However, in practical applications, modality missing is ubiquitous due to sensor failure, data loss or limited acquisition conditions. This modality missing phenomenon significantly affects model performance and has become a key bottleneck for multi-modal system deployment.

[0003] To alleviate this problem, early research mainly adopts two types of strategies: one is to discard incomplete samples, and the other is to utilize existing modality information to complete the missing modality. Traditional completion methods mostly rely on shallow models or simple generation mechanisms, which are difficult to handle complex non-linear relationships between modalities. In recent years, deep learning methods such as autoencoders and generative adversarial networks (GANs) have been used for modality recovery, which has improved the quality of completion to some extent. However, in scenarios with a large number of modalities and high missing proportions, these methods are prone to introduce noise or inconsistency, limiting the performance of the model.

[0004] As a new generation of powerful generative model, diffusion model has made remarkable progress in image inpainting, 3D modeling and multi-modal learning in recent years. In the modal recovery task, it gradually shows better modeling ability than traditional generative methods. For example, the IMDER model recovers the missing modal from Gaussian noise based on the diffusion process and achieves excellent performance in the emotion recognition task (Yuanzhi Wang, Yong Li, and Zhen Cui. Incomplete multimodality-diffused emotion recognition. Advances in Neural Information Processing Systems, 36:17117-17128, 2023). The AMM-Diff model proposes an adaptive diffusion mechanism that integrates image and frequency information for medical image modal completion, effectively improving the accuracy of medical image segmentation (Aghiles Kebaili, Jéróme Lapuyade-Lahorgue, Pierre Vera, and Su Ruan. Amm-diff: Adaptive multi-modality diffusion network for missing modality imputation. arXiv preprint arXiv:2501.12840, 2025.3).

[0005] Although the diffusion model significantly improves the modal completion ability, there are still key challenges: modal generation bias. Due to the heterogeneity of multi-modal data, the quality of model generation on different modalities is significantly different. Visual modalities (such as images, videos) are easy to model accurately due to their high-dimensional continuous feature space; while language modalities are more prone to semantic ambiguity and inconsistency due to their discrete symbolic structure. In addition, existing methods mostly use unified modeling architecture and shared attention mechanism, which fail to fully capture the feature space differences between modalities, making it difficult to effectively coordinate the structural conflicts between the continuity of visual modalities and the compositionality of text modalities. This modal generation bias has become a core problem that restricts the further development of current multi-modal diffusion models. SUMMARY

[0006] The purpose of the present application is to provide a multi-stage double diffusion network-based unbiased missing modal learning method to solve the technical problem of modal generation bias caused by modal heterogeneity in multi-modal learning in the prior art.

[0007] To solve the above technical problems, the specific technical solutions of the present application are as follows:

[0008] An unbiased missing modal learning method based on a multi-stage double diffusion network, the method comprising the following steps:

[0009] Step A: training phase, the multi-modal feature extraction module maps the text modal data, image modal data and audio modal data into a unified latent representation space; the latent space representations of the three modalities are used as missing modal original features and available modal original features.

[0010] Step B: based on the missing modal original features and the available modal original features, a multi-stage double diffusion module is trained through forward diffusion and reverse diffusion to obtain a trained multi-stage double diffusion module;

[0011] The forward diffusion is based on a variance-preserving diffusion model, and through the injection of noise, modal alignment and Gaussian noise degradation, the missing modal original features and the available modal original features are destroyed into noisy missing modal data and noisy available modal data through modal conversion cross connection in the diffusion process of the missing modal data and the available modal data;

[0012] The reverse diffusion is a variational fractional model that constructs a standard stochastic differential equation of reverse time, and generates missing modal recovery features and available modal recovery features from noisy missing modal data and noisy available modal data through three stages of global structure generation, modal conversion and local detail optimization;

[0013] Step C: inference phase, based on the trained multi-stage double diffusion module, the missing modal data is generated by inference based on the available modal data.

[0014] Further, step A comprises the following steps:

[0015] Step A1: obtaining multi-modal data from multiple heterogeneous data sources, the multi-modal data including text modal data, image modal data and audio modal data;

[0016] Step A2: a pre-trained encoder extracts latent space representations from the multi-modal data; the text modal data, image modal data and audio modal data are respectively mapped to latent space representations of a unified dimension through a BERT encoder, a Facet encoder and a COVAREP encoder.

[0017] Step A3: extracting the latent space representations of the three modalities as missing modal original features and available modal original features.

[0018] Further, the forward diffusion is divided into (0, t1], (t1, t2] and (t2, T] three stages, and each stage performs the following steps:

[0019] Step B11: The first stage is a noise injection stage, t ∈ (0, ti], the missing modality raw features with available modality raw features Noise is injected to obtain noisy missing modality data and noisy available modality data

[0020] Step B12: The second stage is a modality alignment stage, t ∈ (ti, t2], the noisy missing modality data and noisy available modality data are converted across time steps until t2, and are modality aligned by the model modality alignment coefficient Φ;

[0021] Step B13: The third stage is a Gaussian noise degradation stage, t ∈ (t2, T], noise is injected based on the discretized form of VP-SDE, finally the noisy available modality data and noisy missing modality data are corrupted as Gaussian noise.

[0022] Further, the forward diffusion of the noisy missing modality data in the stages of t ∈ (0, ti] and t ∈ (t2, T] is:

[0023]

[0024] wherein, denotes the noisy missing modality data at the t-th time step, denotes the noisy missing modality data at the (t-1)-th time step, β t ∈ (0, 1) is a noise intensity coefficient, is a standard Gaussian noise vector.

[0025] The forward diffusion of the noisy available modality data is:

[0026]

[0027] wherein, denotes the noisy available modality data at the t-th time step, denotes the noisy available modality data at the (t-1)-th time step.

[0028] The forward diffusion of the noisy missing modality data in the stage of t ∈ (ti, t2] is:

[0029]

[0030] wherein, Φ is the model modality alignment coefficient.

[0031] Noisy available modal data The forward diffusion of the noisy missing modal data

[0032]

[0033] Further, the backward diffusion is divided into three stages: [T, t2), [t2, t1), [t1, 0), and each stage performs the following steps:

[0034] Step B21: The first stage is a global structure generation stage, t e [T, t2), and the noisy missing modal data and the noisy available modal data are used to recover the missing modal data and the available modal data.

[0035] Step B22: The second stage is a modal conversion stage, t e [t2, t1), and the alignment of the missing modal data and the available modal data is performed through cross-conversion between the modalities.

[0036] Step B23: The third stage is a local detail optimization stage, t e [t1, 0), and the noisy missing modal data and the noisy available modal data are used to recover the missing modal data and the available modal data.

[0037] Further, in the stages of t e [T, t2) and t e [t1, 0), the backward diffusion of the noisy missing modal data is:

[0038]

[0039] where, is a time-varying fractional model that predicts the missing modal recovered feature from the noisy missing modal data by approximating y represents the generated other modal, and the equation is based on the discretization form of the backward time VP-SDE; is a standard Gaussian noise vector;

[0040] The backward diffusion of the noisy available modal data is:

[0041]

[0042] where, is a time-varying fractional model that predicts the available modal recovered feature from the noisy available modal data by approximating

[0043] In the stage of t e [t2, t1), to​ The conversion process is as follows:

[0044]

[0045] and arrive The conversion process is as follows:

[0046]

[0047] Further, step B3 includes the following steps:

[0048] Step B31: Calculate separately and Score matching loss;

[0049] Step B32: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] and The score matching losses are summed to obtain the score loss;

[0050] Step B33: Use a decoder Reconstruct the generated data and calculate the reconstruction loss;

[0051] Step B34: Combine the fractional loss and reconstruction loss to obtain the total target loss of the multi-stage dual diffusion module;

[0052] Step B35: Optimize the time-varying score model by training the total objective loss to obtain the trained multi-stage double diffusion module.

[0053] Furthermore, in step C, the inference generation is divided into three stages: global structure generation, mode transformation, and local detail optimization.

[0054] Step C1: For available modal data One-dimensional convolution yields the original features of the usable modalities. Available modal primitive features After noise injection and mode alignment, noisy usable mode data is obtained. Sampled Gaussian noise As input for the missing modality;

[0055] Step C2: In the global structure generation stage, t∈[T,t2), the missing modal data are generated by backdiffusion according to the discretization form of the reverse time VP-SDE;

[0056] Step C3: During the mode transition phase t∈[t2,t1), through arrive One-way modal transformation is used to obtain modally aligned missing modal data;

[0057] Step C4: In the local detail optimization stage t∈[t1, 0), the missing modality data with noise is obtained from the missing modality data Generate missing modality data;

[0058] Step C5: Adopting the decoder Obtain the decoded missing modality data

[0059] Further, obtain the decoded missing modality data Also used for multi-modal fusion, including the following steps:

[0060] Step D1: The decoded missing modality data And the available modality data Fused into complete multi-modal data by multi-modal Transformer For downstream multi-modal fusion and prediction tasks;

[0061] Step D2: Predict the result by using the fusion features through the multi-layer perception;

[0062] Step D3: Integrate the loss function to build the total loss function.

[0063] Compared with the prior art, the present application has the following beneficial technical effects:

[0064] (1) Compared with the traditional diffusion model, the present application innovatively solves the modal generation bias problem caused by modal heterogeneity in existing multi-modal learning. Based on the scheme of the present application, the model optimizes the available modality and the missing modality, thereby realizing unbiased modeling and high-quality recovery of the missing modality, and enhancing the effectiveness of multi-modal learning under the missing scenario.

[0065] (2) In the multi-stage double diffusion network module, by decomposing the generation task into three stages of global structure generation, modal conversion and local detail optimization, the global structure consistency is ensured, and the local detail representation is enhanced. In the key modal conversion stage, through the bidirectional symmetric conversion mechanism, the model can dynamically control the noise and learn the modal invariant representation shared across modalities, thereby reducing the generation bias and enhancing the comprehensive recovery ability. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0067] Figure 1The model overall flowchart in the embodiment of the present application.

[0068] Figure 2 The multimodal feature extraction module flowchart in the embodiment of the present application.

[0069] Figure 3 The multi-stage double diffusion network module training flowchart in the embodiment of the present application.

[0070] Figure 4 The multi-stage double diffusion network module inference flowchart in the embodiment of the present application.

[0071] Figure 5 The multimodal fusion module flowchart in the embodiment of the present application.

[0072] Figure 6 Comparison with the latest method on CMU-MOSI and CMU-MOSEI under the random missing scenario. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0074] The present application proposes a multi-stage double diffusion network, as shown in Figure 1 The core is to realize unbiased modeling and high-quality recovery of missing modalities through a structure-symmetric and modality-collaborative generation mechanism. The multi-stage double diffusion network includes a multimodal feature extraction module, a multi-stage double diffusion module, and a multimodal fusion module.

[0075] The multimodal feature extraction module processes text, image, and audio modalities through three independent modality-specific pre-training encoders. The encoder maps the original modality data to a unified latent representation space, which ensures the consistency of multimodal features in the preliminary representation stage and extracts discriminative low-dimensional information from a single source.

[0076] The multi-stage double diffusion module adopts a double diffusion generation network, reconstructs the opposite data across modalities through a variance preserving diffusion model (VPDM), and introduces a fractional matching loss combined with a reconstruction loss for optimization, to ensure that each modality contributes unique information and achieve more accurate and unbiased missing modality generation. The variance preserving diffusion model is a diffusion model, and its forward process is based on a variance preserving (VP) process. Based on the double diffusion network, the training process uses complete multi-modal data. In the forward diffusion stage, the data is gradually disturbed by injecting noise, modality alignment and noise destruction according to the VP-SDE (Variance Preserving-Stochastic Differential Equation) discretization form. In the backward diffusion stage, the data distribution is learned through a parameterized fractional network, and the network is optimized using a fractional matching loss and a reconstruction loss. The module gradually relieves the modal generation bias through three generation stages, and realizes the balance recovery between modalities. First, the global structure generation stage captures the macro structure characteristics of the missing modality. The modal conversion stage transfers and mutual guides the information between the available and missing modalities, avoids the dominance of any modality, and realizes the cross-modal semantic alignment. Finally, the local detail optimization stage enhances the fine-grained features of the generated samples.

[0077] The multi-modal fusion module fuses the multi-modal representations after generation, and adopts a multi-modal transformer for downstream prediction.

[0078] Based on the above multi-stage double diffusion network, the present application proposes an unbiased missing modality learning method based on a multi-stage double diffusion network, which solves the modal generation bias problem in the missing modality recovery in multi-modal learning. As shown in Figure 1 The method comprises the following steps:

[0079] Step A: In the training stage, the multi-modal feature extraction module maps the text modality data (L), image modality data (V) and audio modality data (A) into a unified latent representation space, and the latent space representations of the three modalities are used as missing modality original features and available modality original features, as shown in Figure 2 .

[0080] Step A1: Obtain multi-modal data from multiple heterogeneous data sources, and the multi-modal data includes text modality data (L), image modality data (V) and audio modality data (A).

[0081] Step A2: The pre-training encoder extracts the latent space representation from the multi-modal data.

[0082] Different encoders are pre-trained for text modal data, image modal data and audio modal data respectively, the pre-trained BERT encoder extracts text modal data, the Facet encoder extracts image modal data, and the COVAREP encoder extracts audio modal data.

[0083] During the training process, the text modal data, the image modal data and the audio modal data are respectively mapped to the latent space representation χ = {x k} of a unified dimension through the BERT encoder, the Facet encoder and the COVAREP encoder, wherein k represents the corresponding modal, k ∈ {L, V, A}; x k represents the latent space representation of the corresponding modal data after being mapped by the encoder, S is the sequence length, and D is the latent space dimension

[0084] Step A3: Extract the latent space representation of the three modalities as the missing modal original feature and the available modal original feature.

[0085] In the training phase, the missing modal original feature and the available modal original feature are the complete modal data of the three modalities, the missing modal original feature is denoted as The available modal original feature is denoted as That is x L is the latent space representation of the text modal data after being mapped by the encoder, x V is the latent space representation of the image modal data after being mapped by the encoder, x A is the latent space representation of the audio modal data after being mapped by the encoder.

[0086] In the inference phase, the missing modal original feature is sampled from the Gaussian noise The available modal original feature is the latent space representation of one or two modal data after being mapped by the encoder.

[0087] As shown in FIG. 1, Figure 3 Step B: based on the missing modal original feature and the available modal original feature, the multi-stage double diffusion module is trained through forward diffusion and backward diffusion to obtain the trained multi-stage double diffusion module.

[0088] Step B1: based on the variance-preserving diffusion model (VPDM), the missing modal original feature and the available modal original feature are destroyed into two kinds of noise, and are cross-connected through modal conversion in the diffusion process of the missing modal data and the available modal data. The forward diffusion is divided into three stages (0, t1], (t1, t2] and (t2, T]:

[0089] Step B11: The first stage is the noise injection stage, t ∈ (0, ti], the discrete form of VP-SDE is used to inject noise into the missing modality raw features and the available modality raw features Noise is injected to obtain noisy missing modality data and noisy available modality data

[0090] The forward diffusion of noisy missing modality data in this stage is:

[0091]

[0092] where, denotes the noisy missing modality data at the t-th time step, denotes the noisy missing modality data at the (t-1)-th time step, β t ∈ (0, 1) is the noise intensity coefficient, is a standard Gaussian noise vector used for feature perturbation.

[0093] The forward diffusion of noisy available modality data in this stage is:

[0094]

[0095] where, denotes the noisy available modality data at the t-th time step, denotes the noisy available modality data at the (t-1)-th time step.

[0096] Step B12: The second stage is the modality alignment stage, t ∈ (ti, t2], the noisy missing modality data and the noisy available modality data are converted across time steps until t2, and are modality aligned by the model modality alignment coefficient Φ.

[0097] The forward diffusion of noisy missing modality data in this stage is:

[0098]

[0099] where, Φ is the model modality alignment coefficient; denotes the raw features of the missing modality; denotes the raw features of the available modality.

[0100] The forward diffusion of noisy available modality data in this stage is:

[0101]

[0102] Step B13: The third stage is a Gaussian noise degradation stage, t ∈ (t2, T], noise is injected based on the discretized form of VP-SDE, and finally noisy available modal data and noisy missing modal data are destroyed as Gaussian noise.

[0103] The forward diffusion of noisy missing modal data in this stage is:

[0104]

[0105] The forward diffusion of noisy available modal data in this stage is:

[0106]

[0107] Step B2: Based on the Gaussian noise obtained by forward diffusion, the time-varying fractional model s θ is constructed to construct the standard stochastic differential equation (SDE) corresponding to the reverse time, and the noisy missing modal data and the noisy available modal data are generated from the probability distribution p0 of the original features of the missing modal and the available modal p T is a standard Gaussian distribution. By backward diffusion from noisy missing modal data and noisy available modal data , the missing modal recovery feature and the available modal recovery feature are generated.

[0108] Wherein, the time-varying fractional model is realized by a deep neural network with U-Net as the backbone architecture.

[0109] The backward diffusion is divided into three stages: [T, t2), [t2, t1), [t1, 0], and each stage includes the following steps:

[0110] Step B21: The first stage is a global structure generation stage, t ∈ [T, t2), and the missing modal data and the available modal data are recovered from noisy missing modal data and noisy available modal data .

[0111] The backward diffusion of noisy missing modal data in this stage is:

[0112]

[0113] Wherein, is approximated by from noisy missing modal data predicting missing modal recovery features with time-varying fractional model, y represents the generated other modal, the equation is based on the discrete form of inverse time VP-SDE; is a standard Gaussian noise vector.

[0114] inverse diffusion of noisy available modal data at this stage is:

[0115]

[0116] where, is approximated by predicting available modal recovery features from noisy available modal data with time-varying fractional model.

[0117] Step B22: The second stage is a modal conversion stage, t ∈ [t2, t1), the alignment of missing modal data and available modal data is carried out through cross conversion between modes.

[0118] to the conversion process is:

[0119]

[0120] and to the conversion process is:

[0121]

[0122] Step B23: The third stage is a local detail optimization stage, t ∈ [t1, 0), missing modal data and noisy available modal data are recovered from noisy missing modal data and noisy available modal data

[0123] . inverse diffusion of noisy missing modal data

[0124] at this stage is:

[0125] inverse diffusion of noisy available modal data at this stage is:

[0126]

[0127] Step B3: The transition kernel p 0t(x(t)∣x(0)) to obtain the trained multi-stage dual diffusion module.

[0128] The transition kernel approximates the conditional distribution of the state x(t) at any time t given the initial state x(0), specifically, the transition kernel follows a Gaussian distribution, the mean and covariance of which are determined by the initial state and the time-accumulated noise function:

[0129]

[0130] wherein, is the decay factor of the diffusion process, β(s) represents the noise scheduling function, which is a preset function that is monotonically increasing, represents the gradual attenuation of the signal over time, and the covariance reflects the accumulated noise in the diffusion process.

[0131] Step B31: According to the transition kernel p 0t (x(t)∣x(0)) is transformed to ∈ represents standard Gaussian noise, and λ(t) represents a weighting function that balances the loss at different time steps, based on which the fractional matching loss function is obtained, and the fractional matching loss of and is calculated respectively:

[0132]

[0133] wherein, represents the fractional matching loss of , represents the fractional matching loss of , represents the joint expectation of the missing modal data, standard Gaussian noise, and uniform time steps, represents the joint expectation of the available modal data, standard Gaussian noise, and uniform time steps, is a uniform distribution over the time interval [0, T], and λ(t) is a weighting function that balances the loss at different time steps:

[0134] wherein, I represents an identity matrix.

[0135] Step B32: Add the fractional matching losses of

[0136] and to obtain the fractional loss:

[0137]

[0138] wherein, represents the fractional loss.​​

[0139] Step B33: Adopting the decoder The reconstruction loss is defined as:

[0140]

[0141] wherein, represents the reconstruction loss; represents the missing modal data reconstructed by the decoder; represents the available modal data reconstructed by the decoder.

[0142] Step B34: Combining the score loss and the reconstruction loss, the total target loss of the multi-stage double diffusion module is:

[0143]

[0144] wherein, represents the total target loss of the multi-stage double diffusion module.

[0145] Step B35: Training and optimizing the time-varying score model through the loss function to obtain the trained multi-stage double diffusion module.

[0146] Step C: Inference stage, based on the trained multi-stage double diffusion module, the available modal data is used to infer and generate the missing modal data, as shown in Figure 4 .

[0147] The inference generation is divided into three stages: global structure generation, modal conversion, and local detail optimization:

[0148] Step C1: The available modal data is input into the one-dimensional convolution to obtain the available modal original feature . After noise injection and modal alignment, the noisy available modal data is obtained. The Gaussian noise is sampled as the missing modal input.

[0149] The available modal data is input into the first stage of forward diffusion to obtain the noisy available modal data . That is, t∈(0,t1] performs forward diffusion on the noisy available modal data :

[0150]

[0151] t∈(t1,t2] performs forward diffusion on the noisy available modal data :

[0152]

[0153] Step C2: In the global structure generation phase, t ∈ [T, t2), the process generates missing modality data by reverse diffusion according to the discrete form of the reverse time VP-SDE.

[0154] Missing modality data is generated by reverse diffusion as follows:

[0155]

[0156] Step C3: In the modality conversion phase, t ∈ [t2, t1), the missing modality data is obtained by one-way modality conversion from to .

[0157] to . The one-way modality conversion process is as follows:

[0158]

[0159] Step C4: In the local detail optimization phase, t ∈ [t1, 0), the missing modality data is generated from noisy missing modality data .

[0160] The reverse diffusion of the missing modality data in this phase is as follows:

[0161]

[0162] Step C5: The decoder is used to obtain decoded missing modality data .

[0163] As shown in Figure 5 , Step D: Multimodal fusion.

[0164] Step D1: The decoded missing modality data and available modality data are fused into complete multimodal data by a multimodal Transformer for downstream multimodal fusion and prediction tasks.

[0165] Step D2: The fusion features are used to predict the results by a multi-layer perception (MLP).

[0166] Step D3: The total loss function is constructed by integrating the loss function:

[0167]

[0168] wherein, for cross-entropy loss, denotes the total target loss of the multi-stage double diffusion network, and γ is the weight that controls different loss functions.

[0169] After the above steps, the construction of the multi-stage double diffusion network is completed. The test data is used for downstream classification prediction through the three modules, and the accuracy of sentiment classification and regression prediction is compared. Among them, the modal missing rate setting includes specific modal always missing and randomly selected missing modal. The specific modal always missing systematically discards one or two modalities throughout the evaluation process. Randomly selected missing modal defines the missing rate, which is used to quantify the overall degree of missing modal in the sample:

[0170]

[0171] where r miss denotes the missing rate, N is the total number of samples, M is the total number of modalities, m i is the actual number of available modalities for the i-th sample, and in the case of three modalities, eight r miss are selected from the range [0%, 10%, 20%,..., 70%], where 70% represents the maximum approximate rate, while ensuring that at least one available modality is retained at any time.

[0172] The accuracy indicators selected are ACC7, ACC2 and F1, and the sentiment classification data sets selected are CMU-MOSI and CMU-MOSEI. Each sample is labeled with a sentiment score from -3 (strongly negative) to +3 (strongly positive). To verify the effectiveness of the core module, the method conducts a systematic comparative experiment, including the baseline model, single-layer VP-SDE, single-layer multi-stage, and complete module:

[0173] Table 1 Model component effectiveness verification table of four comparison methods

[0174]

[0175]

[0176] Using the above unbiased missing modality learning method based on multi-stage double diffusion network, in the case of fixed modal missing rate, the method is compared with existing high-performance missing modality learning methods. The comparison methods consist of recovery-based methods and non-recovery-based methods. The recovery-based methods include MCTN, TransM, ICDN, MMIN, GCNet, DiCMoR, and IMDer. The non-recovery-based methods include DCCA and DCCAE. As shown in Table 2, the unbiased missing modality learning method based on multi-stage double diffusion network has the best performance in the recovery method, and the recovery-based method always surpasses the non-recovery method.

[0177] Table 2. Accuracy comparison of various methods on the CMU-MOSEI dataset.

[0178]

[0179] Under random missing rates, the performance of mainstream methods for handling missing modalities is compared with that of our method, such as... Figure 6 As shown in the figure, the experiment demonstrates that the performance degradation of this method is significantly smaller than that of other methods, proving that it has stronger robustness to modal loss.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for unbiased missing modality learning based on a multi-stage dual-diffusion network, characterized in that, The method includes the following steps: Step A: During the training phase, the multimodal feature extraction module maps text modal data, image modal data, and audio modal data into a unified latent representation space; The latent space representations of the three modalities serve as the original features of the missing modalities and the original features of the available modalities; Step B: Based on the original features of the missing modalities and the original features of the available modalities, train the multi-stage dual diffusion module through forward diffusion and backward diffusion to obtain the trained multi-stage dual diffusion module; Forward diffusion is based on a variance-preserving diffusion model. Through three stages—noise injection, mode alignment, and Gaussian noise degradation—it cross-connects mode transformation during the diffusion of missing modal data and available modal data, destroying the original features of missing modal data and available modal data into noisy missing modal data and noisy available modal data. The backward diffusion time-varying fractional model constructs a backward-time standard stochastic differential equation and generates missing mode recovery features and available mode recovery features from noisy missing mode data and noisy available mode data through three stages: global structure generation, mode transformation, and local detail optimization. Forward diffusion is divided into ( ]、( , ]、( , The process involves three phases, each with the following steps: Step B11: The first stage is the noise injection stage. Discretization based on VP-SDE to extract original features of missing modalities Original features of available modalities Noise is injected to obtain noisy missing modal data. and noisy available modal data ; Step B12: The second stage is the modal alignment stage. Noisy missing modal data and noisy available modal data After time step transition until And through model modal alignment coefficients Perform modal alignment; Step B13: The third stage is the Gaussian noise degradation stage. Noise is injected based on the discretization form of VP-SDE, ultimately transforming the noisy usable modal data. and noisy missing modal data The destruction is Gaussian noise; Reverse diffusion is divided into , , The process involves three phases, each with the following steps: Step B21: The first stage is the global structure generation stage. From noisy missing modal data With noisy available modal data Recover missing modal data and available modal data; Step B22: The second stage is the mode transition stage. Alignment of missing modal data with available modal data is achieved through cross-transformation between modalities; Step B23: The third stage is the local detail optimization stage. From noisy missing modal data With noisy available modal data Recover missing modal data and available modal data; Step C: Inference phase, based on the trained multi-stage dual diffusion module, missing modality data is generated through available modality data; In step C, inference generation is divided into three stages: global structure generation, mode transformation, and local detail optimization. Step C1: For available modal data One-dimensional convolution yields the original features of the usable modalities. Modal primitive features can be used After noise injection and mode alignment, noisy usable mode data is obtained. Sampling Gaussian noise As input for the missing modality; Step C2: In the global structure generation phase, Based on the discretization form of reverse time VP-SDE, back diffusion is used to generate missing modal data; Step C3: In the mode transition stage ,pass arrive One-way modal transformation is used to obtain modally aligned missing modal data; Step C4: Local detail optimization stage From noisy missing modal data Generate missing modal data; Step C5: Use a decoder Obtain the decoded missing modal data .

2. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 1, characterized in that, Step A includes the following steps: Step A1: Obtain multimodal data from multiple heterogeneous data sources, including text modal data, image modal data, and audio modal data; Step A2: The pre-trained encoder extracts latent spatial representations from the multimodal data; text modal data, image modal data, and audio modal data are mapped to latent spatial representations of a unified dimension through the BERT encoder, Facet encoder, and COVAREP encoder, respectively. Step A3: Extract the latent space representations of the three modalities as the original features of the missing modalities and the original features of the available modalities.

3. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 2, characterized in that, and Phase, noisy missing modal data Forward diffusion is: in, Indicates the first Noisy, missing modal data at time steps Indicates the first Noisy, missing modal data at time steps The noise intensity coefficient, It is a standard Gaussian noise vector; Noisy available modal data Forward diffusion is: in, Indicates the first Noisy time-step modal data can be used. Indicates the first Noisy time-step modal data can be used; Phase, noisy missing modal data Forward diffusion is: in, The model modal alignment coefficient; Noisy available modal data Forward diffusion is: 。 4. The unbiased missing modality learning method based on a multi-stage dual-diffusion network according to claim 3, characterized in that, and Phase, noisy missing modal data The reverse diffusion is: in, To pass through approximation From noisy missing modal data Predicting missing modality recovery features Time-varying fractional model, The equation represents the other generated modes, based on the discretized form of the reverse-time VP-SDE; It is a standard Gaussian noise vector; Noisy available modal data The reverse diffusion is: in, To pass through approximation From noisy available modal data Predictable available modal recovery features Time-varying fractional model; stage, arrive The conversion process is as follows: and arrive The conversion process is as follows: 。 5. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 4, characterized in that, The process of training a multi-stage dual-diffusion module using forward and backward diffusion to obtain a trained multi-stage dual-diffusion module includes the following steps: Step B31: Calculate separately and Score matching loss; Step B32: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] and The score matching losses are summed to obtain the score loss; Step B33: Use a decoder Reconstruct the generated data and calculate the reconstruction loss; Step B34: Combine the fractional loss and reconstruction loss to obtain the total target loss of the multi-stage dual diffusion module; Step B35: Optimize the time-varying score model by training the total objective loss to obtain the trained multi-stage double diffusion module.

6. The unbiased missing mode learning method based on a multi-stage dual-diffusion network according to claim 5, characterized in that, Obtain the decoded missing modal data It can also be used for multimodal fusion, including the following steps: Step D1: Decode the missing modal data and available modal data The data is fused into complete multimodal data using a multimodal Transformer. It is used for downstream multimodal fusion and prediction tasks; Step D2: Predict the result using fused features via a multilayer perceptron; Step D3: Integrate the loss functions to construct the total loss function.

Images