Missing modal data processing method based on mutual information constraint and adversarial learning

Abstract

The invention provides a missing modal data processing method based on mutual information constraint and adversarial learning, and belongs to the technical field of artificial intelligence. The method comprises the following steps: firstly, collecting full-modal data and corresponding missing modal data; then preprocessing the collected full-modal data and the corresponding missing modal data to improve the quality of the full-modal data and the corresponding missing modal data; secondly, after the preprocessed full-modal data and the corresponding missing modal data are obtained, a missing modal processing model based on cross-modal mutual information and adversarial learning is constructed, and meanwhile, a loss function of the missing modal processing model is constructed to constrain the missing modal processing model; continuously training and optimizing the missing modal processing model to obtain a trained missing modal processing model; and finally, inputting the data with the missing modality facing an actual task into the trained missing modality processing model to obtain a segmentation graph.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a missing modal data processing method based on mutual information constraint and adversarial learning. Background Art

[0002] In today's information technology era, multimodal data plays a vital role in various applications, especially in the fields of medical imaging, natural language processing, video analysis, etc. Multimodal data refers to a collection of data from different sources or types, such as visual data, text data, audio data, sensor data, etc. These data can represent the same thing or related events in different forms, structures or perception modes, and can provide information on observing the same phenomenon from different angles, thereby enhancing the system's analysis and decision-making capabilities. By integrating these different types of data, multimodal analysis can improve the accuracy of decision-making and enhance the generalization ability of the model, thereby showing better performance in complex practical applications.

[0003] Although multimodal data has significant advantages, it is often faced with the problem of missing modalities during actual collection and processing. Missing modalities may be caused by a variety of factors, including technical limitations in the data collection process, cost issues, or data privacy policies. Missing modalities not only make the dataset incomplete, but may also greatly reduce the performance of data analysis and problem solving in systems that rely on full modality. The challenge of dealing with missing modalities lies in how to effectively utilize existing modal data, explore the correlation between the remaining modal data, and restore the missing information through the trained network to ensure the accuracy of the system when solving downstream tasks.

[0004] At present, the methods for dealing with missing modalities can be roughly divided into two categories: one is to synthesize the missing modalities to replace them, and the other is to mine the relevance of existing modal data. The method of synthesizing missing modalities usually uses generative models to simulate the data distribution of missing modalities, such as using variational autoencoders (VAE) or generative adversarial networks (GANs). In particular, generative adversarial networks, as a generative model with strong vitality, can generate highly realistic data through the process of adversarial learning. This method can reconstruct modalities in the absence of original modal data, but it may face the problems of large differences between generated data and real data, weak generalization ability, and the need for large-scale annotated data sets for training. Another method is to directly use existing modal data to compensate for the impact of missing modalities through feature extraction and data fusion techniques, such as multi-view learning and collaborative representation. Although this method avoids the uncertainty of synthetic data, it may limit the expressiveness of the model due to the lack of key information. In particular, among the methods of mining the relevance of existing modal data, many methods use the concept of mutual information (MI). Mutual information is a statistical indicator that measures the degree of mutual dependence between variables. By maximizing the mutual information between modes, the correlation and intrinsic connection between different data sources can be captured more accurately. However, existing mutual information-based methods also have some defects in practical applications, such as high computational complexity, unstable estimation, and dependence on prior relationships between modes.

[0005] Therefore, although the existing technology has made a lot of progress in the processing of missing modal data, there are still some problems. How to optimize the system's processing of missing modal input by mining the semantic correlation between modalities without generating missing modal data, so that its performance is as close as possible to the processing results of complete modal data, has become an urgent problem to be solved. If successful, it will greatly improve the processing efficiency and accuracy when the modal data is incomplete, and play an important role in promoting the development of the field of missing modal data processing. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a missing modal data processing method based on mutual information constraints and adversarial learning in response to the above-mentioned deficiencies in the prior art. This method is designed to solve the problems of performance degradation and poor generalization ability caused by the failure to fully utilize the semantic correlation between the modalities in the existing incomplete modal processing technology.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a missing modal data processing method based on mutual information constraint and adversarial learning, comprising the following steps:

[0008] Step 1: Collect full modality data and use certain strategies to generate corresponding missing modality data from the full modality data;

[0009] The full modality data includes image data, text data, audio data, video data and sensor data;

[0010] The specific method of using a certain strategy to generate the corresponding missing modal data from the full modal data is as follows:

[0011] (1) Selective masking: Randomly select the “non-missing” mode in the N-type full-modality data set for masking; each mode includes two cases: “missing” and “non-missing”. The data set containing N full-modality is divided into two N Subsets, remove all the "missing" mode to get the "non-missing" mode, that is, 2 N -1 subset, each subset contains only one mode combination;

[0012] (2) Data corruption: Simulating missing modes by introducing noise into certain modal data or reducing the quality of modal data; when multimodal time series data is involved, simulating missing data of a certain modality by staggering the timestamps of data of different modalities;

[0013] Step 2: Preprocessing the collected full modality data and the corresponding missing modality data to improve the quality of the full modality data and the corresponding missing modality data;

[0014] Step 3: After obtaining the full modality data and the corresponding missing modality data preprocessed in step 2, a missing modality processing model based on cross-modal mutual information and adversarial learning is constructed, and at the same time, a loss function of the missing modality processing model is constructed to constrain the missing modality processing model; the missing modality processing model is continuously trained and optimized to obtain a trained missing modality processing model;

[0015] The missing modality processing model includes a cross-modal mutual information maximization encoder, a knowledge adversarial regularization module, a hybrid attention upsampling module, an entropy regularization module and a decoder;

[0016] The cross-modal mutual information maximization encoder aligns the feature representations of the full modality data and the missing modality data layer by layer, and introduces the mutual information in information theory to measure the difference between the joint distribution of the full modality data and the missing modality data and their respective marginal distributions;

[0017] The input full modal data is recorded as X multi , the input missing modal data is X missing , the output of the cross-modal mutual information maximization encoder is the feature representation F of the full modal data multi and the feature representation F of the missing modal data missing ; The cross-modal mutual information maximization encoder has a total of K layers, which can obtain K pairs of feature representations of full modal data and missing modal data Where k represents the kth layer of the cross-modal mutual information maximization encoder, represents the feature representation of the kth layer of the encoder of the full modal data in the cross-modal mutual information maximization, represents the feature representation of the missing modal data in the k-th layer of the cross-modal mutual information maximization encoder; for ease of writing, m is used 9k) Reference u (k) Reference The entropy of the full modal data of the k-th layer of the encoder that maximizes the cross-modal mutual information is That is, H(m (k) ), let the entropy of the missing modal data of the kth layer of the cross-modal mutual information maximization encoder be That is, H(u (k) ), let the conditional entropy of all modal data under the given missing modal data of the k-th layer of the cross-modal mutual information maximization encoder be That is, H(m (k) |u (k) ); then the feature representation of each pair of full modality data and missing modality data is Mutual information between It can be defined as:

[0018]

[0019] Mutual Information The value of can quantify the amount of information shared between the full modality data and the missing modality data in the kth layer of the cross-modal mutual information maximization encoder; according to the variational information maximization lower bound method, The following conditions must be met:

[0020]

[0021] Among them, p(m (k) ,u (k) ) is m (k) and u (k) The joint probability distribution of (k) |u (k) ) is a given u (k) When, m (k) The conditional probability distribution of q(m (k) |u (k) ) is p(m) estimated by maximizing the cross-modal mutual information encoder (k) |u (k) ), For a given u (k) Under the condition of m (k) The expected value of the log-likelihood of ;

[0022] By minimizing Improved approximate distribution q(m (k) |u (k) ) makes it closer to the actual conditional distribution p(m (k) |u (k) ); then the cross-modal mutual information maximization encoder will be based on the following loss function To optimize:

[0023]

[0024]

[0025] in, is the loss function of the encoder that maximizes the cross-modal mutual information, γ k is a set of hyperparameters, and γ k As the k value increases, C, H, and W are u (k) The number of channels, height and width, μ (k) and σ (k) u (k) The mean and standard deviation of , Z is a formula constant to ensure that the function is a legal probability distribution;

[0026] The knowledge adversarial regularization module is used to represent the feature representation F of the missing modality data. missing Make adjustments, including the generator G kn and the discriminator D kn ;

[0027] The generator G kn Accepts the feature representation F of the missing modality data when the missing modality processing model is run missing As input, the feature representation F of the missing modal data missing After multiple levels of nonlinear transformation and feature extraction, the feature representation F is generated that is consistent with the full modality data. multi Similar false features indicate

[0028] The discriminator D kn Accepting spurious feature representations when running with missing modality handling models and the feature representation of all modal data F multi As input, we use the feature representation F that distinguishes all modal data. multi and false feature representation Prior knowledge of which one is the feature representation of the full modal data F multi , which is generated by the generator G kn Generated spurious feature representation

[0029] The knowledge adversarial regularization module generator G knWith the discriminator D kn The optimization will be performed according to the following loss function:

[0030]

[0031] in, is the loss function of the knowledge adversarial regularization module;

[0032] The hybrid attention upsampling module is used to represent the spurious features and the feature representation of all modal data F multi After a certain upsampling layer to restore details, the hybrid attention upsampling module adopts the upsampling layer of the ResNeXt network and improves it, integrating the hybrid attention mechanism so that the missing modality processing model can integrate local information and global information at the same time to better restore data details; the hybrid attention mechanism includes the channel attention mechanism CAM and the spatial attention mechanism SAM; the channel attention mechanism CAM is mainly used to adjust the weights between different channels in the feature map to highlight the representation of important features; the spatial attention mechanism SAM focuses on adjusting the weight of each pixel in the feature map to suppress information irrelevant to the target area;

[0033] The input of the hybrid attention upsampling module is a fake feature representation and the feature representation of all modal data F multi , and the two feature representations are processed in the same way, and the output is the enhanced feature F' of the missing modal data missing , and the enhanced features F' of the full modality data multi ; Use input feature F to represent false feature representation and the feature representation of all modal data F multi , the enhanced feature F' is used to refer to the enhanced feature F' of the missing modal data missing and the enhanced features F' of the full modality data multi ; Let the dimension of input feature F be H 1 ×W 1 ×C 1 , where H 1 , W 1 , C 1 Respectively represent the height, width and number of channels of the input feature F;

[0034] First, the channel attention weight matrix CAM(F) is calculated according to the following formula:

[0035] F C,max =MLP(maxF(h,w))

[0036]

[0037] CAM(F)=Sigmoid(F C,max +F C,avg )

[0038] Among them, MLP is a multi-layer perceptron consisting of two linear layers and one nonlinear activation layer, F C,max To perform global maximum pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F C,avg To perform global average pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F(h,w) is the C of the input feature F at position (h,w) 1 -dimensional feature vector, CAM(F) is the channel attention weight matrix, which is obtained by dividing the feature map F C,max and F C,avg It is obtained by adding and mapping with the Sigmoid activation function, where Sigmoid is the Sigmoid activation function;

[0039] Secondly, the spatial attention weight matrix SAM(F) is calculated according to the following formula:

[0040] F S,max =maxF(h,w)

[0041]

[0042] SAM(F)=Sigmoid(F S,max +F S,avg )

[0043] Among them, F S,max and F S,avg The two feature maps obtained are global maximum pooling and global average pooling of the input feature F in the channel dimension. SAM(F) is the spatial attention weight matrix, which is obtained by splicing F by channel. S,max and F S,avg And it is obtained by mapping through the Sigmoid activation function;

[0044] Afterwards, the channel attention weight matrix CAM(F) and the spatial attention weight matrix SAM(F) are expanded to the same dimension through an expansion operation and weighted summed to obtain the mixed attention weight matrix M. The specific formula is:

[0045] M C =expand(CAM(F),H,W,C)

[0046] M S =expand(SAM(F),H,W,C)

[0047] M=αM C +βMS

[0048] Among them, expand is the dimension expansion operation, α and β are the hyperparameters used for weighted summation;

[0049] Finally, group convolution is applied to divide the output channels into g groups, with the same convolution kernel size and number of channels in each group, and residual connection is applied to obtain the enhanced feature F'. The specific formula is:

[0050] G i =Conv group (F·M) for each group i

[0051]

[0052] Among them, G i is the output of the grouped convolution, each group processes a part of the channels, Conv group It is a grouped convolution operation, and foreach group i performs the same operation on each group;

[0053] The entropy regularization module uses entropy as an indicator to measure the prediction confidence of the missing modal processing model according to the entropy mapping principle, and is designed for the output of the hybrid attention upsampling layer. en and the discriminator D en Adversarial training between them to improve the confidence of the segmentation map output by the missing modality processing model;

[0054] For full modality data X multi , generator G en Receive enhanced features F' of all modal data multi As input, output the primary segmentation map of all modal data Each element value of this set represents the probability distribution of each pixel point for classifying T types of targets; for missing modal data X missing , generator G en Enhanced features F' that receive missing modal data missing As input, output is a primary segmentation map of the missing modality data Each element value of this set represents the prediction of the probability distribution of each pixel point for classifying T types of targets; the entropy map I of the full modal data is calculated according to the following definition: multi and entropy plots for missing modal data

[0055]

[0056] in, is the enhanced feature F' of the full modal data multiThe probability of category t appearing at the specified position (h,w,c) is, for The logarithm of is the enhanced feature F' for missing modal data missing The probability of category t appearing at the specified position (h,w,c) is, for The logarithm of ;

[0057] When the generator G en After the part is completed, the discriminator D of the entropy regularization module is constructed en ; In adversarial training, the discriminator D en Aims to correctly distinguish which is the entropy map of the full modal data I multi , which is the entropy plot for missing modal data Generator G en With the discriminator D en The optimization is achieved according to the following loss function:

[0058]

[0059] in, is the loss function of the entropy regularization module;

[0060] The decoder is used to generate a primary segmentation map for the missing modality data. Further improve the details to improve the accuracy and quality of segmentation; use the decoder part of U-Net as the decoder of the missing modality processing model to convert the primary segmentation map of the missing modality data Input to the decoder to obtain the final segmentation map;

[0061] The process of training the missing modality processing model based on cross-modal mutual information and adversarial learning is:

[0062] s1: Full modal data X multi and missing modal data X missing Input to the cross-modal mutual information maximization encoder to optimize the loss function of the cross-modal mutual information maximization encoder The output result of the encoder that maximizes the cross-modal mutual information is the feature representation F of the full-modal data. multi and the feature representation of missing modal data F missing ;

[0063] s2: The feature representation F of the missing modal data obtained in step s1 missing and the feature representation of all modal data F multi Input to the knowledge adversarial regularization module, including the following steps:

[0064] s2.1: Using knowledge to counter the generator G in the regularization module kn Feature representation F for missing modal data missing After multi-level nonlinear transformation and feature extraction, the false feature representation is obtained

[0065] s2.2: Using knowledge to counter the discriminator D in the regularization module kn Prior knowledge of which one is the feature representation of the full modal data F multi , which is generated by the generator G kn Generated spurious feature representation

[0066] s2.3: Loss function of the knowledge-based adversarial regularization module Optimizing Generator G kn With the discriminator D kn , thereby aligning the distribution of different modal data in the latent space in a "soft alignment" manner, and obtaining the output of the knowledge adversarial regularization module as a false feature representation

[0067] s3: represents the false features and the feature representation of all modal data F multi Input to the mixed attention upsampling layer module, the output is the enhanced feature F' of the missing modality data missing , and the enhanced features F' of the full modality data multi ;

[0068] s4: Enhance the features F' of the missing modal data obtained in step s3 missing and the enhanced features F' of the full modality data multi Input to the entropy regularization module, including the following steps:

[0069] s4.1: Generator G en Receive enhanced features F' of all modal data multi As input, after multi-level nonlinear transformation and feature extraction, the primary segmentation map of the full modality data is obtained And calculate the entropy map I of the full modal data multi ;

[0070] s4.2: Generator G en Enhanced features F' that accept missing modal data missing As input, after multiple levels of nonlinear transformation and feature extraction, the primary segmentation map of the missing modal data is obtained And calculate the entropy map of the missing modal data

[0071] s4.3: Discriminator Dkn Using prior knowledge, we can determine G en The resulting entropy plot is for the missing modal data. Or the entropy graph of all modal data I multi ;

[0072] s4.4: Optimize G according to the loss function of the entropy regularization module en and D kn , so that the enhanced features F' of the missing modal data missing The entropy distribution of is gradually approaching the enhanced feature F' of the full modal data. multi The entropy distribution of the minimization of the enhanced features F' of the missing modal data missing The entropy of the entropy regularization module is obtained as the primary segmentation map of the missing modal data.

[0073] s5: The primary segmentation map of the missing modality data obtained in step s4.4 Input into the decoder to obtain the final segmentation map;

[0074] s6: The total loss function of the missing modality processing model is:

[0075]

[0076] in, The loss function of the encoder is to maximize the cross-modal mutual information, represents the loss function of the knowledge adversarial regularization module, represents the loss function of the entropy regularization module, and is the segmentation loss function of the missing modality data and the segmentation loss function of the full modality data in the segmentation task;

[0077] Step 4: Input the data with missing modes faced by the actual task into the missing mode processing model trained in step 3 to obtain the segmentation map.

[0078] The beneficial effect of adopting the above technical solution is that: the missing modal data processing method based on mutual information constraint and adversarial learning provided by the present invention has the following advantages:

[0079] (1) More effective integration of shared information: The cross-modal mutual information maximization encoder module effectively integrates the shared information between different modalities by aligning the feature representations of the full modality data and the missing modality data in the encoder layer by layer. This not only improves the performance of the model in the case of missing modalities, but also helps the model learn to ignore irrelevant information, thereby improving the accuracy and efficiency of data processing.

[0080] (2) Better generalization and reliability: The application of the knowledge adversarial regularization module, especially the knowledge adversarial regularization network, effectively improves the generalization ability of the missing modality processing model by adjusting the feature representations of the full modality data and the missing modality data to make their distribution in the latent space more consistent. This ensures that the missing modality processing model can maintain good performance even when faced with new and unseen modality combinations.

[0081] (3) Stronger global information perception: By adding a hybrid attention upsampling layer based on the sampling layer of the ResNeXt network, the model's perception of global and local information is significantly enhanced; this structure enables the network to not only capture local details but also effectively understand the global context, which is particularly important for processing complex multimodal data;

[0082] (4) Higher prediction confidence: The entropy regularization module enhances the reliability and accuracy of the missing modality processing model in performing tasks such as image segmentation by addressing the prediction confidence problem encountered by the missing modality data in downstream tasks;

[0083] (5) Wider application scope: The present invention combines adversarial learning and mutual information maximization strategies, making it not only limited to specific types of multimodal data processing tasks, but also widely applicable to other fields that require high-level information integration and pattern recognition;

[0084] In general, the present invention significantly improves the performance of the model in processing multimodal data by combining advanced network architecture and innovative learning strategies, especially the robustness and accuracy in dealing with modality missing conditions, which makes it have important value and potential in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A flowchart of a method for processing incomplete modal objects based on cross-modal mutual information constraints and adversarial learning provided by an embodiment of the present invention;

[0086] Figure 2 A structural diagram of a missing modality processing model provided by an embodiment of the present invention;

[0087] Figure 3 A structural diagram of a knowledge adversarial regularization module provided by an embodiment of the present invention;

[0088] Figure 4 A structural diagram of an entropy regularization module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0090] This embodiment provides a missing modal data processing method based on mutual information constraint and adversarial learning, such as Figure 1 As shown, the specific steps include:

[0091] Step 1: Collect full modality data and use certain strategies to generate corresponding missing modality data from the full modality data;

[0092] The full modality data should cover all modality types that are planned to be processed by the downstream task, including image data, text data, audio data, video data, and sensor data. At the same time, in order to simulate the situation where the modality may be missing in the real world, a certain strategy is adopted to generate the corresponding missing modality data from the full modality data. The specific method is as follows:

[0093] (1) Selective masking: Randomly select the “non-missing” mode in the N-type full-modality data set for masking; each mode includes two cases: “missing” and “non-missing”. The data set containing N full-modality is divided into two N Subsets, remove all the "missing" mode to get the "non-missing" mode, that is, 2 N -1 subset, each subset contains only one mode combination;

[0094] (2) Data corruption: Simulating missing modalities by introducing noise into certain modal data or reducing the quality of modal data, such as blurring or reducing the resolution of images, adding background noise to audio, or replacing characters in text. When multimodal time series data is involved, simulating missing data in a certain modality by staggering the timestamps of data in different modalities.

[0095] Step 2: Preprocessing the collected full modality data and the corresponding missing modality data to improve the quality of the full modality data and the corresponding missing modality data;

[0096] In order to ensure that the full modality data and the corresponding missing modality data can be efficiently used by the missing modality processing model constructed subsequently, it is necessary to perform preprocessing steps in the preparation stage of the full modality data and the corresponding missing modality data. These steps can improve the quality of the full modality data and the corresponding missing modality data, thereby enhancing the training effect of the missing modality processing model;

[0097] In this embodiment, the specific method of performing preprocessing operations on the collected full modality data and the corresponding missing modality data is as follows: S1: the preprocessing operations on the text data include standardization, removal of stop words, tokenization, stemming, lemmatization and vectorization;

[0098] S2: Preprocessing operations on image data include random cropping, geometric transformation, noise addition, color transformation, and changing image brightness and contrast;

[0099] S3: Preprocessing operations for audio data include sampling rate conversion, slicing, feature extraction, silence removal, and volume normalization;

[0100] S4: Preprocessing of video data Video data can actually be regarded as time series image data, so its preprocessing steps usually include image processing steps, and add some special processing for time series, including frame extraction and selection, frame size adjustment, normalization, rotation, scaling, cropping and other operations on video frames, and optical flow extraction;

[0101] S5: For sensor data including but not limited to temperature, humidity, acceleration, GPS coordinates, etc., the preprocessing steps include data cleaning, feature standardization, window segmentation, feature extraction, and time alignment;

[0102] Step 3: After obtaining the preprocessed full modality data and the corresponding missing modality data in step 2, a missing modality processing model based on cross-modality mutual information and adversarial learning is constructed, such as Figure 2 As shown, a loss function of the missing mode processing model is constructed to constrain the missing mode processing model; the missing mode processing model is continuously trained and optimized to obtain a trained missing mode processing model;

[0103] The missing modality processing model includes a cross-modal mutual information maximization encoder, a knowledge adversarial regularization module, a hybrid attention upsampling module, an entropy regularization module and a decoder;

[0104] The cross-modal mutual information maximization encoder aligns the feature representations of full modal data and missing modal data layer by layer, and introduces mutual information in information theory to measure the difference between the joint distribution of full modal data and missing modal data and their respective marginal distributions; it enables the missing modality processing model to learn shared information between different modalities, while eliminating the interference of irrelevant information in the input on downstream tasks; further, the cross-modal mutual information maximization encoder introduces the concept of mutual information (MI) in information theory; in the discrete case, mutual information represents the degree to which the uncertainty of one random variable is reduced when the value of another random variable is known; this shows that mutual information can measure the degree of association between two random variables, that is, the difference between their joint distribution and their respective marginal distributions;

[0105] The input full modal data is recorded as X multi , the input missing modal data is X missing, the output of the cross-modal mutual information maximization encoder is the feature representation F of the full modal data multi and the feature representation F of the missing modal data missing ; The cross-modal mutual information maximization encoder has a total of K layers, which can obtain K pairs of feature representations of full modal data and missing modal data Where k represents the kth layer of the cross-modal mutual information maximization encoder, represents the feature representation of the kth layer of the encoder of the full modal data in the cross-modal mutual information maximization, represents the feature representation of the missing modal data in the k-th layer of the cross-modal mutual information maximization encoder; for ease of writing, m is used (k) Reference u (k) Reference The entropy of the full modal data of the k-th layer of the encoder that maximizes the cross-modal mutual information is That is, H(m (k) ), let the entropy of the missing modal data of the kth layer of the cross-modal mutual information maximization encoder be That is, H(u (k) ), let the conditional entropy of all modal data under the given missing modal data of the k-th layer of the cross-modal mutual information maximization encoder be That is, H(m (k) |u (k) ); then the feature representation of each pair of full modality data and missing modality data is Mutual information between It can be defined as:

[0106]

[0107] Mutual Information The value of can quantify the amount of information shared between the full modality data and the missing modality data in the kth layer of the cross-modal mutual information maximization encoder; according to the variational information maximization lower bound method, The following conditions must be met:

[0108]

[0109] Among them, p(m (k) ,u (k) ) is m (k) and u (k) The joint probability distribution of (k) |u (k) ) is a given u (k) When, m (k) The conditional probability distribution of q(m (k) |u (k) ) is p(m) estimated by maximizing the cross-modal mutual information encoder (k) |u(k) ), For a given u (k) Under the condition of m (k) The expected value of the log-likelihood of ;

[0110] By minimizing Improved approximate distribution q(m (k) |u (k) ) makes it closer to the actual conditional distribution p(m (k) |u (k) ); then the cross-modal mutual information maximization encoder will be based on the following loss function To optimize:

[0111]

[0112] in, is the loss function of the encoder that maximizes the cross-modal mutual information, γ k is a set of hyperparameters, and γ k As the value of k increases, this is because the higher layers in the encoder contain richer semantic information and should be assigned larger weights for guidance. C, H, and W are u (k) The number of channels, height and width, μ (k) and σ (k) u (k) The mean and standard deviation of , Z is a formula constant to ensure that the function is a legal probability distribution;

[0113] As stated Figure 3 The knowledge adversarial regularization module shown in Figure 2 is used to represent the feature representation F of the missing modality data. missing Make adjustments, including the generator G kn and the discriminator D kn ; Let it be in the latent space with the feature representation F of the full modal data multi The distribution of the missing modality data is more consistent, so that the missing modality processing model can make full use of the shared information contained in the different modal data, and improve the generalization ability and reliability of the missing modality processing model; Existing research is often limited to "hard alignment" when aligning the feature representations of missing modality data and full modality data, that is, minimizing the KL divergence of the two feature representations, but this may cause the feature representation of the missing modality data to be disturbed in the subsequent part of the network learning; In order to solve this problem; Using the idea of ​​generative adversarial network (GAN), by making the feature representation of the missing modality data F missing The feature representation F of the full modality data in the latent space multi More similar, thus achieving “soft alignment”, helping the missing modality processing model to extract the full modality data X multi Learn and improve the feature representation F of the missing modal datamissing ;

[0114] The generator G kn Accepts the feature representation F of the missing modality data when the missing modality processing model is run missing As input, the feature representation F of the missing modal data missing After multiple levels of nonlinear transformation and feature extraction, the feature representation F is generated that is consistent with the full modality data. multi As similar as possible to the false feature representation

[0115] The discriminator D kn Accepting spurious feature representations when running with missing modality handling models and the feature representation of all modal data F multi As input, we use the feature representation F that distinguishes all modal data. multi and false feature representation Prior knowledge of which one is the feature representation of the full modal data F multi , which is generated by the generator G kn Generated spurious feature representation

[0116] The knowledge adversarial regularization module generator G kn With the discriminator D kn The optimization will be performed according to the following loss function:

[0117]

[0118] in, is the loss function of the knowledge adversarial regularization module;

[0119] The hybrid attention upsampling module is used to represent the spurious features and the feature representation of all modal data F multi After a certain upsampling layer to restore details, the hybrid attention upsampling module adopts the upsampling layer of the ResNeXt network and improves it, integrating the hybrid attention mechanism so that the missing modality processing model can simultaneously integrate local information and global information to better restore data details; the hybrid attention mechanism is intended to solve the problem that the existing segmentation method pays the same attention to all regions when processing data, and is prone to ignoring key information; the hybrid attention mechanism includes the channel attention mechanism (CAM) and the spatial attention mechanism (SAM); the channel attention mechanism CAM is mainly used to adjust the weights between different channels in the feature map to highlight the representation of important features; the spatial attention mechanism SAM focuses on adjusting the weight of each pixel in the feature map to suppress information irrelevant to the target area;

[0120] The input of the hybrid attention upsampling module is a fake feature representation and the feature representation of all modal data F multi , and the two feature representations are processed in the same way, and the output is the enhanced feature F' of the missing modal data missing , and the enhanced features F' of the full modality data multi ; For the sake of simplicity, the input feature F is used to represent the false feature representation and the feature representation of all modal data F multi , using enhanced features F' to represent the missing modal data X missing The enhanced feature F' missing and full modality data X multi The enhanced feature F' multi ; Let the dimension of input feature F be H 1 ×W 1 ×C 1 , where H 1 , W 1 , C 1 Respectively represent the height, width and number of channels of the input feature F;

[0121] First, the channel attention weight matrix CAM(F) is calculated according to the following formula:

[0122] F C,max =MLP(maxF(h,w))

[0123]

[0124] CAM(F)=Sigmoid(F C,max +F C,avg )

[0125] Among them, MLP is a multi-layer perceptron consisting of two linear layers and one nonlinear activation layer, F C,max To perform global maximum pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F C,avg To perform global average pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F(h,w) is the C of the input feature F at position (h,w) 1 -dimensional feature vector, CAM(F) represents the channel attention weight matrix, which is obtained by transforming the feature map F C,max and F C,avg It is obtained by adding and mapping with the Sigmoid activation function, where Sigmoid is the Sigmoid activation function;

[0126] Secondly, the spatial attention weight matrix SAM(F) is calculated according to the following formula:

[0127] F S,max =maxF(h,w)

[0128]

[0129] SAM(F)=Sigmoid(F S,max +F S,avg )

[0130] Among them, F S,max and F S,avg The two feature maps obtained are global maximum pooling and global average pooling of the input feature F in the channel dimension. SAM(F) is the spatial attention weight matrix, which is obtained by splicing F by channel. S,max and F S,avg And it is obtained by mapping through the Sigmoid activation function;

[0131] Afterwards, the channel attention weight matrix CAM(F) and the spatial attention weight matrix SAM(F) are expanded to the same dimension through an expansion operation and weighted summed to obtain the mixed attention weight matrix M. The specific formula is:

[0132] M C =expand(CAM(F),H,W,C)

[0133] M S =expand(SAM(F),H,W,C)

[0134] M=αM C +βM S

[0135] Among them, expand is the dimension expansion operation, α and β are the hyperparameters used for weighted summation;

[0136] Finally, group convolution is applied to divide the output channels into g groups, with the same convolution kernel size and number of channels in each group, and residual connection is applied to obtain the enhanced feature F'. The specific formula is:

[0137] G i =Conv group (F·M) for each group i

[0138]

[0139] Among them, G i is the output of the grouped convolution, each group processes a part of the channels, Conv group It is a grouped convolution operation, and foreach groupi performs the same operation on each group;

[0140] The entropy regularization module is used to solve the problem of insufficient prediction confidence in the segmentation task when the existing segmentation method encounters data with missing modes; according to the entropy mapping principle, entropy is used as an indicator to measure the prediction confidence of the missing mode processing model, and an entropy regularization module is designed for the output result of the hybrid attention upsampling layer, such as Figure 4 As shown, through the generator G en and the discriminator D en Adversarial training between them to improve the confidence of the segmentation map output by the missing modality processing model;

[0141] In this embodiment, the missing mode processing model is used for the missing mode data X missing The segmentation of X is usually lacking in confidence and showing high entropy; multi The segmentation of is generally more accurate, showing low entropy; based on this, the training strategy adopted by this module aims to make the enhanced features F' of the missing modal data missing The entropy distribution of is gradually approaching the enhanced feature F' of the full modal data. multi The entropy distribution of , to minimize the enhanced features F' of the missing modal data missing Entropy of

[0142] For full modality data X multi , generator G en Receive enhanced features F' of all modal data multi As input, output the primary segmentation map of all modal data Each element value of this set represents the probability distribution of each pixel point for classifying T types of targets; for missing modal data X missing , generator G en Enhanced features F' that receive missing modal data missing As input, output is a primary segmentation map of the missing modality data Each element value of this set represents the prediction of the probability distribution of each pixel point for classifying T types of targets; the entropy map I of the full modal data is calculated according to the following definition: multi and entropy plots for missing modal data

[0143]

[0144] in, is the enhanced feature F' of the full modal data multi The probability of category t appearing at the specified position (h,w,c) is, for The logarithm of is the enhanced feature F' for missing modal data missingThe probability of category t appearing at the specified position (h,w,c) is, for The logarithm of ;

[0145] When the generator G en After the part is completed, the discriminator D of the entropy regularization module is constructed en ; In adversarial training, the discriminator D en Aims to correctly distinguish which is the entropy map of the full modal data I multi , which is the entropy plot for missing modal data Generator G en With the discriminator D en The optimization is achieved according to the following loss function:

[0146]

[0147] in, represents the loss function of the entropy regularization module;

[0148] The decoder is used to generate a primary segmentation map for the missing modality data. Further improve the details to improve the accuracy and quality of segmentation; use the decoder part of U-Net as the decoder of the missing modality processing model to convert the primary segmentation map of the missing modality data Input to the decoder to obtain the final segmentation map;

[0149] The process of training the missing modality processing model based on cross-modal mutual information and adversarial learning is:

[0150] s1: Full modal data X multi and missing modal data X missing Input to the cross-modal mutual information maximization encoder to optimize the loss function of the cross-modal mutual information maximization encoder The output result of the encoder that maximizes the cross-modal mutual information is the feature representation F of the full-modal data. multi and the feature representation of missing modal data F missing ;

[0151] s2: The feature representation F of the missing modal data obtained in step s1 missing and the feature representation of all modal data F multi Input to the knowledge adversarial regularization module, including the following steps:

[0152] s2.1: Using knowledge to counter the generator G in the regularization module kn Feature representation F for missing modal data missing After multi-level nonlinear transformation and feature extraction, the false feature representation is obtained

[0153] s2.2: Using knowledge to counter the discriminator D in the regularization module kn Prior knowledge of which one is the feature representation of the full modal data F multi , which is generated by the generator G kn Generated spurious feature representation

[0154] s2.3: Loss function of the knowledge-based adversarial regularization module Optimizing Generator G kn With the discriminator D kn , thereby aligning the distribution of different modal data in the latent space in a "soft alignment" manner, and obtaining the output of the knowledge adversarial regularization module as a false feature representation

[0155] s3: represents the false features and the feature representation of all modal data F multi Input to the mixed attention upsampling layer module, the output is the enhanced feature F' of the missing modality data missing , and the enhanced features F' of the full modality data multi ;

[0156] s4: Enhance the features F' of the missing modal data obtained in step s3 missing and the enhanced features F' of the full modality data multi Input to the entropy regularization module, including the following steps:

[0157] s4.1: Generator G en Receive enhanced features F' of all modal data multi As input, after multi-level nonlinear transformation and feature extraction, the primary segmentation map of the full modality data is obtained And calculate the entropy map I of the full modal data multi ;

[0158] s4.2: Generator G en Enhanced features F' that accept missing modal data missing As input, after multiple levels of nonlinear transformation and feature extraction, the primary segmentation map of the missing modal data is obtained And calculate the entropy map of the missing modal data

[0159] s4.3: Discriminator D kn Using prior knowledge, we can determine G en The resulting entropy plot is for the missing modal data. Or the entropy graph of all modal data I multi ;

[0160] s4.4: Optimize G according to the loss function of the entropy regularization module en and D kn , so that the enhanced features F' of the missing modal data missing The entropy distribution of is gradually approaching the enhanced feature F' of the full modal data. multi The entropy distribution of the minimization of the enhanced features F' of the missing modal data missing The entropy of the entropy regularization module is obtained as the primary segmentation map of the missing modal data.

[0161] s5: The primary segmentation map of the missing modality data obtained in step s4.4 Input into the decoder to obtain the final segmentation map;

[0162] s6: The total loss function of the missing modality processing model is:

[0163]

[0164] in, The loss function of the encoder is to maximize the cross-modal mutual information, represents the loss function of the knowledge adversarial regularization module, represents the loss function of the entropy regularization module, and is the segmentation loss function of the missing modality data and the segmentation loss function of the full modality data in the segmentation task, and both use Dice loss as the loss function; λ one , multi , 0 , 1 and 2 These are trade-off parameters, which are set to 0.8, 0.2, 0.001, 0.0002, and 0.5 respectively in model training;

[0165] Step 4: Input the data with missing modes faced by the actual task into the missing mode processing model trained in step 3 to obtain the segmentation map;

[0166] This embodiment proposes a comprehensive method for solving the challenge of incomplete modalities in multimodal data. The core innovation of the present invention is that it integrates cross-modal mutual information maximization, knowledge adversarial regularization, hybrid attention upsampling and entropy regularization in the model training framework; this integration not only aligns the feature representation between full modal data and missing modal data, but also stabilizes the distribution of modal information, ensuring effective segmentation even when some data streams are missing; the present invention is based on the hybrid attention upsampling module, which can effectively restore data details while retaining local and global information through the hybrid attention mechanism and grouped convolution technology; the comprehensive loss function framework adopted by the present invention allows dynamic adjustment during the training process, optimizes the model's response to different data characteristics, and finally trains a more robust and adaptable system; through this method, the present invention effectively solves the processing of missing modalities, paves the way for future work in the field of multimodal learning, especially in real-world scenarios with incomplete data, and provides a feasible solution for maintaining high efficiency and accuracy information processing.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A missing modal data processing method based on mutual information constraint and adversarial learning, characterized in that: The following steps are involved: Step 1: Collect full modality data and use certain strategies to generate corresponding missing modality data from the full modality data; The full modality data includes image data, text data, audio data, video data and sensor data; Step 2: Preprocessing the collected full modality data and the corresponding missing modality data to improve the quality of the full modality data and the corresponding missing modality data; Step 3: After obtaining the full modality data and the corresponding missing modality data preprocessed in step 2, a missing modality processing model based on cross-modal mutual information and adversarial learning is constructed, and at the same time, a loss function of the missing modality processing model is constructed to constrain the missing modality processing model; the missing modality processing model is continuously trained and optimized to obtain a trained missing modality processing model; Step 4: Input the data with missing modes faced by the actual task into the missing mode processing model trained in step 3 to obtain the segmentation map.

2. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 1, characterized in that: The step 1 adopts a certain strategy to generate corresponding missing modal data from full modal data. The specific method is: (1) Selective masking: Randomly select the "non-missing" mode in the N-type full-modality data set for masking; each mode includes two cases: "missing" and "non-missing", and the data set containing N full-modality is divided into two N Subsets, remove all the "missing" mode to get the "non-missing" mode, that is, 2 N -1 subset, each subset contains only one mode combination; (2) Data corruption: The missing mode is simulated by introducing noise into the data of a certain mode or reducing the quality of the modal data. When multimodal time series data is involved, the missing of a certain modal data is simulated by staggering the timestamps of different modal data.

3. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 2, characterized in that: The missing modality processing model based on cross-modal mutual information and adversarial learning constructed in step 3 includes a cross-modal mutual information maximization encoder, a knowledge adversarial regularization module, a hybrid attention upsampling module, an entropy regularization module and a decoder.

4. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 3 is characterized in that: The cross-modal mutual information maximization encoder aligns the feature representations of the full modality data and the missing modality data layer by layer, and introduces the mutual information in information theory to measure the difference between the joint distribution of the full modality data and the missing modality data and their respective marginal distributions; The input full modal data is recorded as X multi , the input missing modal data is X missing , the output of the cross-modal mutual information maximization encoder is the feature representation F of the full modal data multi and the feature representation F of the missing modal data missing ; The cross-modal mutual information maximization encoder has a total of K layers, which can obtain K pairs of feature representations of full modal data and missing modal data Where k represents the kth layer of the cross-modal mutual information maximization encoder, represents the feature representation of the kth layer of the encoder of the full modal data in the cross-modal mutual information maximization, represents the feature representation of the missing modal data in the k-th layer of the cross-modal mutual information maximization encoder; for ease of writing, m is used (k) Reference u (k) Reference The entropy of the full modal data of the k-th layer of the encoder that maximizes the cross-modal mutual information is That is, H(m (k) ), let the entropy of the missing modal data of the kth layer of the cross-modal mutual information maximization encoder be That is, H(u (k) ), let the conditional entropy of all modal data under the given missing modal data of the k-th layer of the cross-modal mutual information maximization encoder be That is, H(m (k) |u (k) ); then the feature representation of each pair of full modality data and missing modality data is Mutual information between It can be defined as: Mutual Information The value of can quantify the amount of information shared between the full modality data and the missing modality data in the k-th layer of the cross-modal mutual information maximization encoder; According to the variational information maximization lower bound method, The following conditions must be met: Among them, p(m (k) ,u (k) ) is m (k) and u (k) The joint probability distribution of (k) |u (k) ) is a given u (k) When, m (k) The conditional probability distribution of q(m (k) |u (k) ) is p(m) estimated by maximizing the cross-modal mutual information encoder (k) |u (k) ), is the expected value of the log-likelihood of m(k) given u(k); By minimizing Improved approximate distribution q(m (k) |u (k) ) makes it closer to the actual conditional distribution p(m (k) |u (k) ); then the cross-modal mutual information maximization encoder will be based on the following loss function To optimize: in, is the loss function of the encoder that maximizes the cross-modal mutual information, γ k is a set of hyperparameters, and γ k As the k value increases, C, H, and W are u (k) The number of channels, height and width, μ (k) and σ (k) u (k) The mean and standard deviation of , Z is the recipe constant to ensure that the function is a valid probability distribution.

5. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 4, characterized in that: The knowledge adversarial regularization module is used to represent the feature representation F of the missing modality data. missing Make adjustments, including the generator G kn and the discriminator D kn .

6. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 5, characterized in that: The generator G kn Accepts the feature representation F of the missing modality data when the missing modality processing model is run missing As input, the feature representation F of the missing modal data missing After multiple levels of nonlinear transformation and feature extraction, the feature representation F is generated that is consistent with the full modality data. multi Similar false features indicate The discriminator D kn Accepting spurious feature representations when running with missing modality handling models and the feature representation of all modal data F multi As input, we use the feature representation F that distinguishes all modal data. multi and false feature representation Prior knowledge of which one is the feature representation of the full modal data F multi , which is generated by the generator G kn Generated spurious feature representation The knowledge adversarial regularization module generator G kn With the discriminator D kn The optimization will be performed according to the following loss function: in, is the loss function of the knowledge adversarial regularization module.

7. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 6, characterized in that: The hybrid attention upsampling module is used to represent the spurious features and the feature representation of all modal data F multi After a certain upsampling layer to restore details, the hybrid attention upsampling module adopts the upsampling layer of the ResNeXt network and improves it, integrating the hybrid attention mechanism so that the missing modality processing model can integrate local information and global information at the same time and better restore data details; The hybrid attention mechanism includes the channel attention mechanism CAM and the spatial attention mechanism SAM; the channel attention mechanism CAM is mainly used to adjust the weights between different channels in the feature map to highlight the representation of important features; The spatial attention mechanism SAM focuses on adjusting the weight of each pixel in the feature map to suppress information irrelevant to the target area; The input of the hybrid attention upsampling module is a fake feature representation and the feature representation of all modal data F multi , and the two feature representations are processed in the same way, and the output is the enhanced feature F′ of the missing modal data missing , and the enhanced features F′ of the full modality data multi ; Use input feature F to represent false feature representation and the feature representation of all modal data F multi , the enhanced feature F′ refers to the enhanced feature F′ of the missing modal data missing And the enhanced features F′ of the full modal data multi ; The dimension of the input feature F is H1×W1×C1, where H1, W1, and C1 represent the height, width, and number of channels of the input feature F, respectively; First, the channel attention weight matrix CAM(F) is calculated according to the following formula: F C,max =MLP(maxF(h,w)) CAM(F)=Sigmoid(F C,max +F C,avg ) Among them, MLP is a multi-layer perceptron consisting of two linear layers and one nonlinear activation layer, F C,max To perform global maximum pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F C,avg is a feature map obtained by global average pooling of the spatial dimension of the input feature F through an MLP layer, F(h, w) is the C1-dimensional feature vector of the input feature F at position (h, w), CAM(F) is the channel attention weight matrix, which is obtained by combining the feature map F C,max and F C,avg It is obtained by adding and mapping with the Sigmoid activation function, where Sigmoid is the Sigmoid activation function; Secondly, the spatial attention weight matrix SAM(F) is calculated according to the following formula: F S,max =maxF(h,w) SAM(F)=Sigmoid(F S，max +F s,avg ) Among them, F S,max and F S,avg The two feature maps obtained are global maximum pooling and global average pooling of the input feature F in the channel dimension. SAM(F) is the spatial attention weight matrix, which is obtained by splicing F by channel. S,max and F S,avg And it is obtained by mapping through the Sigmoid activation function; Afterwards, the channel attention weight matrix CAM(F) and the spatial attention weight matrix SAM(F) are expanded to the same dimension through an expansion operation and weighted summed to obtain the mixed attention weight matrix M. The specific formula is: M C =expand(CAM(F),H,W,C) M S =expand(SAM(F),H,W,C) M=αM C +βM S Among them, expand is the dimension expansion operation, α and β are the hyperparameters used for weighted summation; Finally, group convolution is applied to divide the output channels into g groups, with the same convolution kernel size and number of channels in each group, and residual connection is applied to obtain the enhanced feature F′. The specific formula is: G i =Conv group (F·M)for each group i Among them, G i is the output of the grouped convolution, each group processes a part of the channels, Conv group It is a grouped convolution operation, and for eachgroup i performs the same operation on each group.

8. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 7, characterized in that: The entropy regularization module uses entropy as an indicator to measure the prediction confidence of the missing modal processing model according to the entropy mapping principle, and is designed for the output of the hybrid attention upsampling layer. en and the discriminator D en Adversarial training between them to improve the confidence of the segmentation map output by the missing modality processing model; For full modality data X multi , generator G en Receive enhanced features F′ of all modal data multi As input, output the primary segmentation map of all modal data Each element value of this set represents the probability distribution of each pixel point for classifying T types of targets; for missing modal data X missing , generator G en Enhanced features F′ that receive missing modal data missing As input, output is a primary segmentation map of the missing modality data Each element value of this set represents the prediction of the probability distribution of each pixel point for classifying T types of targets; the entropy map I of the full modal data is calculated according to the following definition: multi and entropy plots for missing modal data in, is the enhanced feature F′ of the full modal data multi The probability of category t appearing at the specified position (h, w, c) of for The logarithm of is the enhanced feature F′ of missing modal data missing The probability of category t appearing at the specified position (h, w, c) of for The logarithm of ; When the generator G en After the part is completed, the discriminator D of the entropy regularization module is constructed en ; In adversarial training, the discriminator D en Aims to correctly distinguish which one is the entropy map of the full modal data I multi , which is the entropy plot for missing modal data Generator G en With the discriminator D en The optimization is achieved according to the following loss function: in, is the loss function of the entropy regularization module.

9. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 8, characterized in that: The decoder is used to generate a primary segmentation map for the missing modality data. Further improve the details to improve the accuracy and quality of segmentation; use the decoder part of U-Net as the decoder of the missing modality processing model to convert the primary segmentation map of the missing modality data Input to the decoder to get the final segmentation map.

10. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 9, characterized in that: The process of training the missing modality processing model based on cross-modal mutual information and adversarial learning is: s1: Full modal data X multi and missing modal data X missing Input to the cross-modal mutual information maximization encoder to optimize the loss function of the cross-modal mutual information maximization encoder The output result of the encoder that maximizes the cross-modal mutual information is the feature representation F of the full-modal data. multi and the feature representation of missing modal data F missing ; s2: The feature representation F of the missing modal data obtained in step s1 missing and the feature representation of all modal data F multi Input to the knowledge adversarial regularization module, including the following steps: s2.1: Using knowledge to counter the generator G in the regularization module kn Feature representation F for missing modal data missing After multi-level nonlinear transformation and feature extraction, the false feature representation is obtained s2.2: Using knowledge to counter the discriminator D in the regularization module kn Prior knowledge of which one is the feature representation of the full modal data F multi , which is generated by the generator G kn Generated spurious feature representation s2.3: Loss function of the knowledge-based adversarial regularization module Optimizing Generator G kn With the discriminator D kn , thereby aligning the distribution of different modal data in the latent space in a "soft alignment" manner, and obtaining the output of the knowledge adversarial regularization module as a false feature representation s3: represents the false features and the feature representation of all modal data F multi Input to the mixed attention upsampling layer module, the output is the enhanced feature F′ of the missing modality data missing , and the enhanced features F′ of the full modality data multi ; s4: Enhanced features F′ of missing modal data obtained in step s3 missing And the enhanced features F′ of the full modal data multi Input to the entropy regularization module, including the following steps: s4.1: Generator G en Receive enhanced features F′ of all modal data multi As input, after multi-level nonlinear transformation and feature extraction, the primary segmentation map of the full modality data is obtained And calculate the entropy map I of the full modal data multi ; s4.2: Generator G en Enhanced features F′ that accept missing modal data missing As input, after multiple levels of nonlinear transformation and feature extraction, the primary segmentation map of the missing modal data is obtained And calculate the entropy map of the missing modal data s4.3: Discriminator D kn Using prior knowledge, we can determine G en The resulting entropy plot is for the missing modal data. Or the entropy diagram of all modal data I multi ; s4.4: Optimize G according to the loss function of the entropy regularization module en and D kn , so that the enhanced features F′ of the missing modal data missing The entropy distribution of is gradually approaching the enhanced feature F′ of the full modal data. multi The entropy distribution of the minimization of the enhanced features F′ of the missing modal data missing The entropy of the entropy regularization module is obtained as the primary segmentation map of the missing modal data. s5: The primary segmentation map of the missing modality data obtained in step s4.4 Input into the decoder to obtain the final segmentation map; s6: The total loss function of the missing modality processing model is: in, The loss function of the encoder is to maximize the cross-modal mutual information, represents the loss function of the knowledge adversarial regularization module, represents the loss function of the entropy regularization module, and It is the segmentation loss function of missing modal data and the segmentation loss function of full modal data in the segmentation task.

Images