A missing modal data processing method based on mutual information constraint and adversarial learning

Abstract

The application provides a missing modal data processing method based on mutual information constraint and adversarial learning, and belongs to the technical field of artificial intelligence.The method first collects full-modal data and corresponding missing modal data;then performs a pretreatment operation on the collected full-modal data and corresponding missing modal data to improve the quality of the full-modal data and corresponding missing modal data;secondly, after obtaining the pretreated full-modal data and corresponding missing modal data, a missing modal processing model based on cross-modal mutual information and adversarial learning is constructed, and a loss function of the missing modal processing model is constructed to constrain the missing modal processing model;the missing modal processing model is continuously trained and optimized to obtain a trained missing modal processing model;finally, data with missing modal faced by an actual task are input into the trained missing modal processing model to obtain a segmentation map.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a missing modal data processing method based on mutual information constraint and adversarial learning. Background Art

[0002] In today's information technology era, multimodal data plays a vital role in various applications, particularly in fields such as medical imaging, natural language processing, and video analysis. Multimodal data refers to a collection of data from different sources or types, such as visual data, text data, audio data, and sensor data. This data can represent the same object or related events in different forms, structures, or perception modes, providing information from different perspectives on the same phenomenon, thereby enhancing the system's analytical and decision-making capabilities. By integrating these different types of data, multimodal analysis can improve decision-making accuracy and enhance the generalization ability of models, thereby demonstrating better performance in complex real-world applications.

[0003] Despite the significant advantages of multimodal data, the actual collection and processing process often faces the problem of missing modalities. Missing modalities can be caused by a variety of factors, including technical limitations in the data collection process, cost issues, or data privacy policies. Missing modalities not only make the dataset incomplete, but can also significantly reduce the performance of data analysis and problem solving in systems that rely on full modality. The challenge of dealing with missing modalities lies in how to effectively utilize existing modal data, explore the correlations between the remaining modal data, and recover the missing information through trained networks to ensure the accuracy of the system when solving downstream tasks.

[0004] Currently, methods for addressing missing modalities can be roughly divided into two categories: synthesizing a replacement for the missing modality, and mining correlations in existing modal data. Methods for synthesizing missing modalities typically employ generative models to simulate the data distribution of the missing modality, such as variational autoencoders (VAEs) or generative adversarial networks (GANs). GANs, in particular, are a powerful generative model capable of generating highly realistic data through an adversarial learning process. This approach can reconstruct modalities in the absence of the original data for the modality, but may suffer from significant discrepancies between the generated data and the real data, weak generalization capabilities, and the need for large labeled datasets for training. Another approach directly utilizes existing modal data, compensating for the effects of the missing modality through feature extraction and data fusion techniques, such as multi-view learning and collaborative representation. While this approach avoids the uncertainty of the synthesized data, it can limit the model's expressiveness due to the lack of key information. In particular, many methods for mining correlations in existing modal data leverage the concept of mutual information (MI). Mutual information is a statistical metric that measures the degree of interdependence between variables. By maximizing mutual information between modalities, it can more accurately capture the correlations and inherent connections between different data sources. However, existing mutual information-based methods have some drawbacks in practical applications, such as high computational complexity, unstable estimation, and dependence on prior relationships between modalities.

[0005] Therefore, despite significant progress in existing technologies for processing missing modal data, some challenges remain. A pressing challenge is how to optimize the system's processing of missing modal inputs by mining semantic relevance between modalities, without generating missing modal data, to achieve performance as close as possible to that of complete modal data. Successful implementation would significantly improve processing efficiency and accuracy when modal data is incomplete, significantly advancing the field of missing modal data processing. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the shortcomings of the above-mentioned existing technologies and provide a missing modal data processing method based on mutual information constraints and adversarial learning. This method solves the problems of performance degradation and poor generalization ability caused by the failure to fully utilize the semantic correlation between the modalities in the existing incomplete modal processing technology.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is: a missing modal data processing method based on mutual information constraint and adversarial learning, comprising the following steps:

[0008] Step 1: Collect full modality data and use a certain strategy to generate the corresponding missing modality data from the full modality data;

[0009] The full modality data includes image data, text data, audio data, video data and sensor data;

[0010] The specific method of using a certain strategy to generate the corresponding missing modal data from the full modal data is as follows:

[0011] (1) Selective shielding: Randomly select the “non-missing” mode in the dataset of N full modalities for shielding; each mode includes two cases: “missing” and “non-missing”, and the dataset containing N full modalities is divided into 2 N Subsets, remove all the "missing" mode to get the "non-missing" mode, that is, 2 N -1 subset, each subset contains only one mode combination;

[0012] (2) Data corruption: This involves simulating missing modalities by introducing noise into the data of a certain modality or reducing the quality of the modal data. When multimodal time series data is involved, the missing data of a certain modality is simulated by staggering the timestamps of data of different modalities.

[0013] Step 2: Preprocess the collected full modality data and the corresponding missing modality data to improve the quality of the full modality data and the corresponding missing modality data;

[0014] Step 3: After obtaining the full modality data and the corresponding missing modality data preprocessed in step 2, a missing modality processing model based on cross-modal mutual information and adversarial learning is constructed. At the same time, a loss function of the missing modality processing model is constructed to constrain the missing modality processing model; the missing modality processing model is continuously trained and optimized to obtain a trained missing modality processing model;

[0015] The missing modality processing model includes a cross-modal mutual information maximization encoder, a knowledge adversarial regularization module, a hybrid attention upsampling module, an entropy regularization module and a decoder;

[0016] The cross-modal mutual information maximization encoder aligns the feature representations of the full modality data and the missing modality data layer by layer, and introduces the mutual information in information theory to measure the difference between the joint distribution of the full modality data and the missing modality data and their respective marginal distributions;

[0017] The full modal input data is recorded as X multi , the input missing modal data is X missing , the output of the cross-modal mutual information maximization encoder is the feature representation F of the full modal data multi and the feature representation F of the missing modal data missing ; The cross-modal mutual information maximization encoder has a total of K layers, which can obtain K pairs of feature representations of full modal data and missing modal data Where k represents the kth layer of the cross-modal mutual information maximization encoder, represents the feature representation of the full modal data in the k-th layer of the cross-modal mutual information maximization encoder, Represents the feature representation of the missing modal data in the k-th layer of the cross-modal mutual information maximization encoder; for simplicity of writing, use m 9k) Reference u (k) Reference The entropy of the full modal data of the k-th layer of the cross-modal mutual information maximization encoder is recorded as That is, H(m (k) ), let the entropy of the missing modal data of the k-th layer of the cross-modal mutual information maximization encoder be That is, H(u (k) ), let the conditional entropy of the full modality data under the given missing modality data of the cross-modality mutual information maximization encoder k layer be That is, H(m (k) |u (k) ); then the feature representation of each pair of full modality data and missing modality data is Mutual information between It can be defined as:

[0018]

[0019] Mutual Information The value of can quantify the amount of information shared between the full modality data and the missing modality data in the k-th layer of the cross-modal mutual information maximization encoder; according to the variational information maximization lower bound method, The following conditions must be met:

[0020]

[0021] Among them, p(m (k) ,u (k) ) is m (k) and u (k) The joint probability distribution of p(m (k) |u (k) ) is a given u (k) When, m (k) The conditional probability distribution of q(m (k) |u (k) ) is p(m) estimated by maximizing the cross-modal mutual information encoder (k) |u (k) ), For a given u (k) Under the condition of m (k) The expected value of the log-likelihood of ;

[0022] By minimizing Improved approximate distribution q(m (k) |u (k) ) parameters to make it closer to the actual conditional distribution p(m (k) |u (k) ); then the cross-modal mutual information maximization encoder will be based on the following loss function To optimize:

[0023]

[0024]

[0025] in, is the loss function of the encoder that maximizes the cross-modal mutual information, γ k is a set of hyperparameters, and γ k As the k value increases, C, H, and W are u (k) The number of channels, height and width, μ (k) and σ (k) u (k) The mean and standard deviation of , Z is a formula constant to ensure that the function is a legal probability distribution;

[0026] The knowledge adversarial regularization module is used to represent the feature representation F of the missing modality data. missing Make adjustments, including the generator G kn and the discriminator D kn ;

[0027] The generator G kn Accept the feature representation F of the missing modality data when the missing modality processing model is run missing As input, the feature representation F of the missing modal data missing After multi-level nonlinear transformation and feature extraction, the feature representation F of the full modal data is generated. multi Similar false features represent

[0028] The discriminator D kn Accepting spurious feature representations when running a missing modality handling model and the feature representation F of full modal data multi As input, we use the feature representation F that distinguishes all modal data. multi and false feature representation Prior knowledge of which one is the feature representation F of full modal data multi , which is generated by the generator G kn Generated false feature representation

[0029] The generator G of the knowledge adversarial regularization module knWith the discriminator D kn The optimization will be performed according to the following loss function:

[0030]

[0031] in, is the loss function of the knowledge adversarial regularization module;

[0032] The hybrid attention upsampling module is used to represent the false features and the feature representation F of full modal data multi After a certain number of upsampling layers to restore details, the hybrid attention upsampling module adopts and improves the upsampling layer of the ResNeXt network, integrating the hybrid attention mechanism, so that the missing modality processing model can simultaneously integrate local information and global information to better restore data details. The hybrid attention mechanism includes the channel attention mechanism CAM and the spatial attention mechanism SAM. The channel attention mechanism CAM is mainly used to adjust the weights between different channels in the feature map to highlight the representation of important features; the spatial attention mechanism SAM focuses on adjusting the weight of each pixel in the feature map to suppress information irrelevant to the target area.

[0033] The input of the hybrid attention upsampling module is a false feature representation and the feature representation F of full modal data multi , and the two feature representations are processed in the same way, and the output is the enhanced feature F' of the missing modal data missing , and the enhanced features F' of full modality data multi ; Use input feature F to represent false feature representation and the feature representation F of full modal data multi , the enhanced feature F' refers to the enhanced feature F' of the missing modal data missing and the enhanced features F' of full modal data multi ;Remember that the dimension of the input feature F is H1×W1×C1, where H1, W1, and C1 represent the height, width, and number of channels of the input feature F respectively;

[0034] First, the channel attention weight matrix CAM(F) is calculated according to the following formula:

[0035] F C,max =MLP(maxF(h,w))

[0036]

[0037] CAM(F)=Sigmoid(F C,max +F C,avg )

[0038] Among them, MLP is a multi-layer perceptron consisting of two linear layers and one nonlinear activation layer, F C,max To perform global maximum pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F C,avg To perform global average pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F(h,w) is the C1-dimensional feature vector of the input feature F at position (h,w), CAM(F) is the channel attention weight matrix, which is obtained by combining the feature map F C,max and F C,avg It is obtained by adding and mapping with the Sigmoid activation function, where Sigmoid is the Sigmoid activation function;

[0039] Secondly, the spatial attention weight matrix SAM(F) is calculated according to the following formula:

[0040] F S,max =maxF(h,w)

[0041]

[0042] SAM(F)=Sigmoid(F S,max +F S,avg )

[0043] Among them, F S,max and F S,avg The two feature maps obtained by performing global maximum pooling and global average pooling on the input feature F in the channel dimension are respectively, and SAM(F) is the spatial attention weight matrix, which is obtained by splicing F by channel. S,max and F S,avg And it is obtained by mapping through the Sigmoid activation function;

[0044] Afterwards, the channel attention weight matrix CAM(F) and the spatial attention weight matrix SAM(F) are expanded to the same dimension through the expansion operation and weighted summed to obtain the mixed attention weight matrix M. The specific formula is:

[0045] M C =expand(CAM(F),H,W,C)

[0046] M S =expand(SAM(F),H,W,C)

[0047] M=αM C +βM S

[0048] Among them, expand is the dimension expansion operation, α and β are the hyperparameters used for weighted summation;

[0049] Finally, group convolution is applied to divide the output channels into g groups. The convolution kernel size and number of channels in each group are the same, and residual connection is applied to obtain the enhanced feature F'. The specific formula is:

[0050] G i =Conv group (F·M) for each group i

[0051]

[0052] Among them, G i The output of the grouped convolution, each group processes a part of the channel, Conv group It is a grouped convolution operation, and foreach group i performs the same operation on each group;

[0053] The entropy regularization module uses entropy as an indicator to measure the prediction confidence of the missing modal processing model according to the entropy mapping principle, and is designed for the output of the hybrid attention upsampling layer. en and the discriminator D en Adversarial training between them to improve the confidence of the segmentation map output by the missing modality processing model;

[0054] For full modality data X multi , generator G en Receive enhanced features F' of full modality data multi As input, output is the primary segmentation map of all modal data Each element value of this set represents the probability distribution of each pixel point for classifying T types of targets; for missing modal data X missing , generator G en Receive the enhanced features F' for missing modal data missing As input, it outputs a primary segmentation map of the missing modality data. Each element value of this set represents the prediction of the probability distribution of each pixel point for classifying T types of targets; the entropy map I of the full modal data is calculated according to the following definition multi and entropy plots for missing modal data

[0055]

[0056] in, is the enhanced feature F' of full modal data multi The probability of category t appearing at the specified position (h,w,c) is, for The logarithm of is the enhanced feature F' for missing modal data missingThe probability of category t appearing at the specified position (h,w,c) is, for The logarithm of

[0057] When the generator G en After the part is completed, the discriminator D of the entropy regularization module is constructed en ; In adversarial training, the discriminator D en Aims to correctly distinguish which one is the entropy map of full modal data I multi , which is the entropy map of the missing modal data Generator G en With the discriminator D en The optimization is achieved according to the following loss function:

[0058]

[0059] in, is the loss function of the entropy regularization module;

[0060] The decoder is used to generate the primary segmentation map for the missing modality data. Further improve the details to improve the accuracy and quality of segmentation; use the decoder part of U-Net as the decoder of the missing modality processing model to convert the primary segmentation map of the missing modality data into Input to the decoder to obtain the final segmentation map;

[0061] The process of training the missing modality processing model based on cross-modal mutual information and adversarial learning is as follows:

[0062] s1: Full modal data X multi and missing modal data X missing Input to the cross-modal mutual information maximization encoder and optimize the loss function of the cross-modal mutual information maximization encoder The output of the cross-modal mutual information maximization encoder is the feature representation F of the full modal data. multi and the feature representation F of missing modal data missing ;

[0063] s2: The feature representation F of the missing modal data obtained in step s1 missing and the feature representation of full modal data F multi Input to the knowledge adversarial regularization module, including the following steps:

[0064] s2.1: Using knowledge to counter the generator G in the regularization module kn Feature representation F for missing modal data missing After multi-level nonlinear transformation and feature extraction, the false feature representation is obtained

[0065] s2.2: Using knowledge to counter the discriminator D in the regularization module kn Prior knowledge of which one is the feature representation F of full modal data multi , which is generated by the generator G kn Generated false feature representation

[0066] s2.3: Loss function of the knowledge-based regularization module Optimize generator G kn With the discriminator D kn , thereby aligning the distribution of different modal data in the latent space in a "soft alignment" manner, and obtaining the output of the knowledge adversarial regularization module as a false feature representation

[0067] s3: represents the false features and the feature representation of full modal data F multi Input to the mixed attention upsampling layer module, the output is the enhanced feature F' of the missing modality data missing , and the enhanced features F' of full modal data multi ;

[0068] s4: Enhance the features F' of the missing modal data obtained in step s3 missing and the enhanced features F' of full modal data multi Input to the entropy regularization module, including the following steps:

[0069] s4.1: Generator G en Receive enhanced features F' of full modality data multi As input, after multi-level nonlinear transformation and feature extraction, the primary segmentation map of the full modal data is obtained. And calculate the entropy map I of the full modal data multi ;

[0070] s4.2: Generator G en Enhanced features F' that accept missing modal data missing As input, after multi-level nonlinear transformation and feature extraction, the primary segmentation map of the missing modal data is obtained. And calculate the entropy map of missing modal data

[0071] s4.3: Discriminator D kn Using prior knowledge, we can judge G en The entropy map of missing modal data is generated Or the entropy diagram of full modal data I multi ;

[0072] s4.4: Optimize G according to the loss function of the entropy regularization module en and D kn , so that the enhanced features F' of the missing modal data missing The entropy distribution gradually approaches the enhanced feature F' of the full modal data multi Entropy distribution of the minimization of the enhanced features F' of the missing modal data missing The entropy of the entropy regularization module is obtained as the primary segmentation map of the missing modal data.

[0073] s5: The primary segmentation map of the missing modality data obtained in step s4.4 Input into the decoder to obtain the final segmentation map;

[0074] s6: The total loss function of the missing modality processing model is:

[0075]

[0076] in, The loss function of the encoder is to maximize the cross-modal mutual information, represents the loss function of the knowledge adversarial regularization module, represents the loss function of the entropy regularization module, and is the segmentation loss function of missing modality data and the segmentation loss function of full modality data in the segmentation task;

[0077] Step 4: Input the data with missing modes faced by the actual task into the missing mode processing model trained in step 3 to obtain the segmentation map.

[0078] The beneficial effects of adopting the above technical solution are: the missing modal data processing method based on mutual information constraint and adversarial learning provided by the present invention has the following advantages:

[0079] (1) More effective integration of shared information: The cross-modal mutual information maximization encoder module effectively integrates the shared information between different modalities by aligning the feature representations of the full modality data and the missing modality data in the encoder layer by layer. This not only improves the performance of the model in the missing modality case, but also helps the model learn to ignore irrelevant information, thereby improving the accuracy and efficiency of data processing;

[0080] (2) Better generalization and reliability: The application of the knowledge adversarial regularization module, especially the knowledge adversarial regularization network, effectively improves the generalization ability of the missing modality processing model by adjusting the feature representations of full modality data and missing modality data to make their distribution in the latent space more consistent. This ensures that the missing modality processing model can maintain good performance even when faced with new and unseen modality combinations.

[0081] (3) Stronger global information perception capability: By adding a hybrid attention upsampling layer based on the sampling layer of the ResNeXt network, the model's ability to perceive global and local information is significantly enhanced; this structure enables the network to not only capture local details but also effectively understand the global context, which is particularly important for processing complex multimodal data;

[0082] (4) Higher prediction confidence: The entropy regularization module addresses the prediction confidence issue encountered by missing modality data in downstream tasks, thereby enhancing the reliability and accuracy of the missing modality processing model in performing tasks such as image segmentation;

[0083] (5) Wider application scope: The present invention combines adversarial learning and mutual information maximization strategies, making it not only limited to specific types of multimodal data processing tasks, but also widely applicable to other fields that require high-level information integration and pattern recognition;

[0084] In general, this invention significantly improves the performance of the model in processing multimodal data by combining advanced network architecture and innovative learning strategies, especially the robustness and accuracy in dealing with modality missing conditions, which makes it have important value and potential in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A flowchart of a method for processing incomplete modal objects based on cross-modal mutual information constraints and adversarial learning provided by an embodiment of the present invention;

[0086] Figure 2 A diagram showing the structure of a missing modality processing model provided by an embodiment of the present invention;

[0087] Figure 3 The structure diagram of the knowledge adversarial regularization module provided by an embodiment of the present invention;

[0088] Figure 4 Structural diagram of the entropy regularization module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0090] This embodiment provides a method for processing missing modal data based on mutual information constraint and adversarial learning, such as Figure 1 As shown, the specific steps include:

[0091] Step 1: Collect full modality data and use a certain strategy to generate the corresponding missing modality data from the full modality data;

[0092] The full-modal data should cover all modal types that are planned to be processed by the downstream task, including image data, text data, audio data, video data, and sensor data. At the same time, in order to simulate the situation where modalities may be missing in the real world, a certain strategy is adopted to generate the corresponding missing modal data from the full-modal data. The specific method is as follows:

[0093] (1) Selective shielding: Randomly select the “non-missing” mode in the dataset of N full modalities for shielding; each mode includes two cases: “missing” and “non-missing”, and the dataset containing N full modalities is divided into 2 N Subsets, remove all the "missing" mode to get the "non-missing" mode, that is, 2 N -1 subset, each subset contains only one mode combination;

[0094] (2) Data corruption: This involves simulating missing modalities by introducing noise into the data of a certain modality or reducing the quality of the modal data. For example, this involves blurring or reducing the resolution of images, adding background noise to audio, or replacing characters in text. When multimodal time series data is involved, the missing data of a certain modality can be simulated by staggering the timestamps of data from different modalities.

[0095] Step 2: Preprocess the collected full modality data and the corresponding missing modality data to improve the quality of the full modality data and the corresponding missing modality data;

[0096] In order to ensure that the full modality data and the corresponding missing modality data can be efficiently used by the missing modality processing model constructed subsequently, preprocessing steps need to be performed in the preparation stage of the full modality data and the corresponding missing modality data. These steps can improve the quality of the full modality data and the corresponding missing modality data, thereby enhancing the training effect of the missing modality processing model;

[0097] In this embodiment, the specific method of performing preprocessing operations on the collected full modality data and the corresponding missing modality data is as follows: S1: the preprocessing operations on the text data include standardization, removal of stop words, tokenization, stemming, lemmatization and vectorization;

[0098] S2: Preprocessing operations on image data include random cropping, geometric transformation, noise addition, color transformation, and changing image brightness and contrast;

[0099] S3: Preprocessing operations for audio data include sampling rate conversion, slicing, feature extraction, silence removal, and volume normalization;

[0100] S4: Preprocessing of video data: Video data can actually be considered as time series image data. Therefore, its preprocessing steps usually include image processing steps, and some special processing for time series is added, including frame extraction and selection, frame size adjustment, normalization, rotation, scaling, cropping and other operations on video frames, and optical flow extraction.

[0101] S5: For sensor data including but not limited to temperature, humidity, acceleration, GPS coordinates, etc., the preprocessing steps include data cleaning, feature normalization, window segmentation, feature extraction and time alignment;

[0102] Step 3: After obtaining the full modality data and the corresponding missing modality data preprocessed in step 2, a missing modality processing model based on cross-modal mutual information and adversarial learning is constructed, such as Figure 2 As shown, a loss function of the missing mode processing model is constructed to constrain the missing mode processing model; the missing mode processing model is continuously trained and optimized to obtain a trained missing mode processing model;

[0103] The missing modality processing model includes a cross-modal mutual information maximization encoder, a knowledge adversarial regularization module, a hybrid attention upsampling module, an entropy regularization module and a decoder;

[0104] The cross-modal mutual information maximization encoder aligns the feature representations of the full modal data and the missing modal data layer by layer, and at the same time introduces mutual information from information theory to measure the difference between the joint distribution of the full modal data and the missing modal data and their respective marginal distributions; it enables the missing modality processing model to learn the shared information between different modalities while eliminating the interference of irrelevant information in the input on downstream tasks; further, the cross-modal mutual information maximization encoder introduces the concept of mutual information (MI) from information theory; in the discrete case, mutual information represents the degree to which the uncertainty of one random variable is reduced when the value of another random variable is known; this shows that mutual information can measure the degree of association between two random variables, that is, the difference between their joint distribution and their respective marginal distributions;

[0105] The full modal input data is recorded as X multi , the input missing modal data is X missing, the output of the cross-modal mutual information maximization encoder is the feature representation F of the full modal data multi and the feature representation F of the missing modal data missing ; The cross-modal mutual information maximization encoder has a total of K layers, which can obtain K pairs of feature representations of full modal data and missing modal data Where k represents the k-th layer of the cross-modal mutual information maximization encoder, represents the feature representation of the full modal data in the k-th layer of the cross-modal mutual information maximization encoder, Represents the feature representation of the missing modal data in the k-th layer of the cross-modal mutual information maximization encoder; for simplicity of writing, use m (k) Reference u (k) Reference The entropy of the full modal data of the k-th layer of the cross-modal mutual information maximization encoder is recorded as That is, H(m (k) ), let the entropy of the missing modal data of the k-th layer of the cross-modal mutual information maximization encoder be That is, H(u (k) ), let the conditional entropy of the full modality data under the given missing modality data of the cross-modality mutual information maximization encoder k layer be That is, H(m (k) |u (k) ); then the feature representation of each pair of full modality data and missing modality data is Mutual information between It can be defined as:

[0106]

[0107] Mutual Information The value of can quantify the amount of information shared between the full modality data and the missing modality data in the k-th layer of the cross-modal mutual information maximization encoder; according to the variational information maximization lower bound method, The following conditions must be met:

[0108]

[0109] Among them, p(m (k) ,u (k) ) is m (k) and u (k) The joint probability distribution of p(m (k) |u (k) ) is a given u (k) When, m (k) The conditional probability distribution of q(m (k) |u (k) ) is p(m) estimated by maximizing the cross-modal mutual information encoder (k) |u(k) ), For a given u (k) Under the condition of m (k) The expected value of the log-likelihood of ;

[0110] By minimizing Improved approximate distribution q(m (k) |u (k) ) parameters to make it closer to the actual conditional distribution p(m (k) |u (k) ); then the cross-modal mutual information maximization encoder will be based on the following loss function To optimize:

[0111]

[0112] in, is the loss function of the encoder that maximizes the cross-modal mutual information, γ k is a set of hyperparameters, and γ k As the value of k increases, this is because the higher layers in the encoder contain richer semantic information and should be assigned greater weights for guidance. C, H, and W are u (k) The number of channels, height and width, μ (k) and σ (k) u (k) The mean and standard deviation of , Z is a formula constant to ensure that the function is a legal probability distribution;

[0113] As stated Figure 3 The knowledge adversarial regularization module shown is used to represent the feature F of the missing modality data. missing Make adjustments, including the generator G kn and the discriminator D kn ; Let it be represented in the latent space with the feature representation F of the full modal data multi The distribution of the missing modality data is more consistent, which enables the missing modality processing model to fully utilize the shared information contained in the different modal data, thereby improving the generalization ability and reliability of the missing modality processing model; Existing research on aligning the feature representations of missing modality data and full modality data is often limited to "hard alignment", that is, minimizing the KL divergence of the two feature representations, but this may cause the feature representation of the missing modality data to be disturbed in the subsequent learning of the network; In order to solve this problem; Using the idea of ​​generative adversarial network (GAN), by making the feature representation F of the missing modality data missing Feature representation F of full modal data in latent space multi More similar, thus achieving "soft alignment", helping the missing modality processing model to extract the full modality data X multi Learn and improve the feature representation F of missing modal datamissing ;

[0114] The generator G kn Accept the feature representation F of the missing modality data when the missing modality processing model is run missing As input, the feature representation F of the missing modal data missing After multi-level nonlinear transformation and feature extraction, the feature representation F of the full modal data is generated. multi As similar as possible to the false feature representation

[0115] The discriminator D kn Accepting spurious feature representations when running a missing modality handling model and the feature representation F of full modal data multi As input, we use the feature representation F that distinguishes all modal data. multi and false feature representation Prior knowledge of which one is the feature representation F of full modal data multi , which is generated by the generator G kn Generated false feature representation

[0116] The generator G of the knowledge adversarial regularization module kn With the discriminator D kn The optimization will be performed according to the following loss function:

[0117]

[0118] in, is the loss function of the knowledge adversarial regularization module;

[0119] The hybrid attention upsampling module is used to represent the false features and the feature representation F of full modal data multi After a certain upsampling layer to restore details, the hybrid attention upsampling module adopts the upsampling layer of the ResNeXt network and improves it, integrating the hybrid attention mechanism, so that the missing modality processing model can simultaneously integrate local information and global information to better restore data details. The hybrid attention mechanism aims to solve the problem that existing segmentation methods pay the same attention to all regions when processing data, which easily ignores key information. The hybrid attention mechanism includes the channel attention mechanism (CAM) and the spatial attention mechanism (SAM). The channel attention mechanism CAM is mainly used to adjust the weights between different channels in the feature map to highlight the representation of important features; the spatial attention mechanism SAM focuses on adjusting the weight of each pixel in the feature map to suppress information irrelevant to the target area.

[0120] The input of the hybrid attention upsampling module is a false feature representation and the feature representation F of full modal data multi , and the two feature representations are processed in the same way, and the output is the enhanced feature F' of the missing modal data missing , and the enhanced features F' of full modality data multi ; For the sake of simplicity, the input feature F is used to represent the false feature representation and the feature representation F of full modal data multi , use enhanced features F' to represent the missing modal data X missing Enhanced feature F' missing and full modal data X multi Enhanced feature F' multi ;Remember that the dimension of the input feature F is H1×W1×C1, where H1, W1, and C1 represent the height, width, and number of channels of the input feature F respectively;

[0121] First, the channel attention weight matrix CAM(F) is calculated according to the following formula:

[0122] F C,max =MLP(maxF(h,w))

[0123]

[0124] CAM(F)=Sigmoid(F C,max +F C,avg )

[0125] Among them, MLP is a multi-layer perceptron consisting of two linear layers and one nonlinear activation layer, F C,max To perform global maximum pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F C,avg To perform global average pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F(h,w) is the C1-dimensional feature vector of the input feature F at position (h,w), CAM(F) represents the channel attention weight matrix, which is obtained by dividing the feature map F C,max and F C,avg It is obtained by adding and mapping with the Sigmoid activation function, where Sigmoid is the Sigmoid activation function;

[0126] Secondly, the spatial attention weight matrix SAM(F) is calculated according to the following formula:

[0127] F S,max =maxF(h,w)

[0128]

[0129] SAM(F)=Sigmoid(F S,max +F S,avg )

[0130] Among them, F S,max and F S,avg The two feature maps obtained by performing global maximum pooling and global average pooling on the input feature F in the channel dimension are respectively, and SAM(F) is the spatial attention weight matrix, which is obtained by splicing F by channel. S,max and F S,avg And it is obtained by mapping through the Sigmoid activation function;

[0131] Afterwards, the channel attention weight matrix CAM(F) and the spatial attention weight matrix SAM(F) are expanded to the same dimension through the expansion operation and weighted summed to obtain the mixed attention weight matrix M. The specific formula is:

[0132] M C =expand(CAM(F),H,W,C)

[0133] M S =expand(SAM(F),H,W,C)

[0134] M=αM C +βM S

[0135] Among them, expand is the dimension expansion operation, α and β are the hyperparameters used for weighted summation;

[0136] Finally, group convolution is applied to divide the output channels into g groups. The convolution kernel size and number of channels in each group are the same, and residual connection is applied to obtain the enhanced feature F'. The specific formula is:

[0137] G i =Conv group (F·M) for each group i

[0138]

[0139] Among them, G i The output of the grouped convolution, each group processes a part of the channel, Conv group It is a grouped convolution operation, and foreach groupi performs the same operation on each group;

[0140] The entropy regularization module is used to solve the problem of insufficient prediction confidence in the segmentation task when the existing segmentation method encounters data with missing modes. According to the entropy mapping principle, entropy is used as an indicator to measure the prediction confidence of the missing mode processing model. An entropy regularization module is designed for the output of the hybrid attention upsampling layer, such as Figure 4 As shown, through the generator G en and the discriminator D en Adversarial training between them to improve the confidence of the segmentation map output by the missing modality processing model;

[0141] In this embodiment, the missing modality processing model is used to process the data X with missing modality. missing The segmentation of X is usually lacking in confidence and showing high entropy; multi The segmentation of is generally more accurate and is characterized by low entropy. Based on this, the training strategy adopted by this module aims to enhance the features F' of the missing modal data. missing The entropy distribution gradually approaches the enhanced feature F' of the full modal data multi Entropy distribution of to minimize the enhanced features F' of missing modal data missing Entropy;

[0142] For full modal data X multi , generator G en Receive enhanced features F' of full modality data multi As input, output is the primary segmentation map of the full modality data Each element value of this set represents the probability distribution of each pixel point for classifying T types of targets; for missing modal data X missing , generator G en Receive the enhanced features F' for missing modal data missing As input, it outputs a primary segmentation map of the missing modality data. Each element value of this set represents the prediction of the probability distribution of each pixel point for classifying T types of targets; the entropy map I of the full modal data is calculated according to the following definition multi and entropy plots for missing modal data

[0143]

[0144] in, is the enhanced feature F' of full modal data multi The probability of category t appearing at the specified position (h,w,c) is, for The logarithm of is the enhanced feature F' for missing modal data missing The probability of category t appearing at the specified position (h,w,c) is, for The logarithm of

[0145] When the generator G en After the part is completed, the discriminator D of the entropy regularization module is constructed en ; In adversarial training, the discriminator D en Aims to correctly distinguish which one is the entropy map of full modal data I multi , which is the entropy map of the missing modal data Generator G en With the discriminator D en The optimization is achieved according to the following loss function:

[0146]

[0147] in, represents the loss function of the entropy regularization module;

[0148] The decoder is used to generate the primary segmentation map for the missing modality data. Further improve the details to improve the accuracy and quality of segmentation; use the decoder part of U-Net as the decoder of the missing modality processing model to convert the primary segmentation map of the missing modality data into Input to the decoder to obtain the final segmentation map;

[0149] The process of training the missing modality processing model based on cross-modal mutual information and adversarial learning is as follows:

[0150] s1: Full modal data X multi and missing modal data X missing Input to the cross-modal mutual information maximization encoder and optimize the loss function of the cross-modal mutual information maximization encoder The output of the cross-modal mutual information maximization encoder is the feature representation F of the full modal data. multi and the feature representation F of missing modal data missing ;

[0151] s2: The feature representation F of the missing modal data obtained in step s1 missing and the feature representation of full modal data F multi Input to the knowledge adversarial regularization module, including the following steps:

[0152] s2.1: Using knowledge to counter the generator G in the regularization module kn Feature representation F for missing modal data missing After multi-level nonlinear transformation and feature extraction, the false feature representation is obtained

[0153] s2.2: Using knowledge to counter the discriminator D in the regularization module kn Prior knowledge of which one is the feature representation F of full modal data multi , which is generated by the generator G kn Generated false feature representation

[0154] s2.3: Loss function of the knowledge-based regularization module Optimize generator G kn With the discriminator D kn , thereby aligning the distribution of different modal data in the latent space in a "soft alignment" manner, and obtaining the output of the knowledge adversarial regularization module as a false feature representation

[0155] s3: represents the false features and the feature representation of full modal data F multi Input to the mixed attention upsampling layer module, the output is the enhanced feature F' of the missing modality data missing , and the enhanced features F' of full modal data multi ;

[0156] s4: Enhance the features F' of the missing modal data obtained in step s3 missing and the enhanced features F' of full modal data multi Input to the entropy regularization module, including the following steps:

[0157] s4.1: Generator G en Receive enhanced features F' of full modality data multi As input, after multi-level nonlinear transformation and feature extraction, the primary segmentation map of the full modal data is obtained. And calculate the entropy map I of the full modal data multi ;

[0158] s4.2: Generator G en Enhanced features F' that accept missing modal data missing As input, after multi-level nonlinear transformation and feature extraction, the primary segmentation map of the missing modal data is obtained. And calculate the entropy map of missing modal data

[0159] s4.3: Discriminator D kn Using prior knowledge, we can judge G en The entropy map of missing modal data is generated Or the entropy diagram of full modal data I multi ;

[0160] s4.4: Optimize G according to the loss function of the entropy regularization module en and D kn , so that the enhanced features F' of the missing modal data missing The entropy distribution gradually approaches the enhanced feature F' of the full modal data multi Entropy distribution of the minimization of the enhanced features F' of the missing modal data missing The entropy of the entropy regularization module is obtained as the primary segmentation map of the missing modal data.

[0161] s5: The primary segmentation map of the missing modality data obtained in step s4.4 Input into the decoder to obtain the final segmentation map;

[0162] s6: The total loss function of the missing modality processing model is:

[0163]

[0164] in, The loss function of the encoder is to maximize the cross-modal mutual information, represents the loss function of the knowledge adversarial regularization module, represents the loss function of the entropy regularization module, and is the segmentation loss function of the missing modal data and the segmentation loss function of the full modal data in the segmentation task, and both use Dice loss as the loss function; λ one ,λ multi , λ0, λ1, and λ2 are all trade-off parameters, which are set to 0.8, 0.2, 0.001, 0.0002, and 0.5, respectively, during model training;

[0165] Step 4: Input the missing modality data of the actual task into the missing modality processing model trained in step 3 to obtain the segmentation map;

[0166] This embodiment proposes a comprehensive method to address the challenge of incomplete modalities in multimodal data. The core innovation of the present invention lies in the integration of cross-modal mutual information maximization, knowledge adversarial regularization, hybrid attention upsampling and entropy regularization in the model training framework; this integration not only aligns the feature representations between full modal data and missing modal data, but also stabilizes the distribution of modal information, ensuring effective segmentation even when some data streams are missing; the present invention is based on a hybrid attention upsampling module, which can effectively restore data details while retaining local and global information through a hybrid attention mechanism and grouped convolution technology; the comprehensive loss function framework adopted by the present invention allows dynamic adjustment during the training process, optimizes the model's response to different data characteristics, and ultimately trains a more robust and adaptable system; through this method, the present invention effectively solves the processing of missing modalities, paves the way for future work in the field of multimodal learning, especially in real-world scenarios with incomplete data, and provides a feasible solution for maintaining high efficiency and accuracy of information processing.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for processing missing modal data based on mutual information constraints and adversarial learning, characterized by: The following steps are involved: Step 1: Collect full modality data and use a certain strategy to generate the corresponding missing modality data from the full modality data; The full modality data includes image data, text data, audio data, video data and sensor data; Step 2: Preprocess the collected full modality data and the corresponding missing modality data to improve the quality of the full modality data and the corresponding missing modality data; Step 3: After obtaining the full modality data and the corresponding missing modality data preprocessed in step 2, a missing modality processing model based on cross-modal mutual information and adversarial learning is constructed. At the same time, a loss function of the missing modality processing model is constructed to constrain the missing modality processing model; the missing modality processing model is continuously trained and optimized to obtain a trained missing modality processing model; Step 4: Input the missing modality data of the actual task into the missing modality processing model trained in step 3 to obtain the segmentation map; The missing modality processing model based on cross-modal mutual information and adversarial learning constructed in step 3 includes a cross-modal mutual information maximization encoder, a knowledge adversarial regularization module, a hybrid attention upsampling module, an entropy regularization module and a decoder; The cross-modal mutual information maximization encoder aligns the feature representations of the full modality data and the missing modality data layer by layer, and introduces the mutual information in information theory to measure the difference between the joint distribution of the full modality data and the missing modality data and their respective marginal distributions; The full modal input data is recorded as X multi , the input missing modal data is X missing , the output of the cross-modal mutual information maximization encoder is the feature representation F of the full modal data multi and the feature representation F of the missing modal data missing ; The cross-modal mutual information maximization encoder has a total of K layers, which can obtain K pairs of feature representations of full modal data and missing modal data Where k represents the k-th layer of the cross-modal mutual information maximization encoder, represents the feature representation of the full modal data in the k-th layer of the cross-modal mutual information maximization encoder, Represents the feature representation of the missing modal data in the k-th layer of the cross-modal mutual information maximization encoder; for simplicity of writing, use m (k) Reference u (k) Reference The entropy of the full modal data of the k-th layer of the cross-modal mutual information maximization encoder is recorded as That is, H(m (k) ), let the entropy of the missing modal data of the k-th layer of the cross-modal mutual information maximization encoder be That is, H(u (k) ), let the conditional entropy of the full modality data under the given missing modality data of the cross-modality mutual information maximization encoder k layer be That is, H(m (k) |u (k) ); then the feature representation of each pair of full modality data and missing modality data is Mutual information between It can be defined as: Mutual Information The value of can quantify the amount of information shared between the full modality data and the missing modality data in the k-th layer of the cross-modal mutual information maximization encoder; According to the variational information maximization lower bound method, The following conditions are met: Among them, p(m (k) ,u (k) ) is m (k) and u (k) The joint probability distribution of p(m (k) |u (k) ) is a given u (k) When, m (k) The conditional probability distribution of q(m (k) |u (k) ) is p(m) estimated by maximizing the cross-modal mutual information encoder (k) |u (k) ), For a given u (k) Under the condition of m (k) The expected value of the log-likelihood of ; By minimizing Improved approximate distribution q(m (k) |u (k) ) parameters to make it closer to the actual conditional distribution p(m (k) |u (k) ); then the cross-modal mutual information maximization encoder will be based on the following loss function To optimize: in, is the loss function of the encoder that maximizes the cross-modal mutual information, γk is a set of hyperparameters, and γk increases with the increase of k value, C, H, and W are u (k) The number of channels, height and width, μ (k) and σ (k) u (k) The mean and standard deviation of , Z is a formula constant to ensure that the function is a legal probability distribution; The knowledge adversarial regularization module is used to represent the feature representation F of the missing modality data. missing Make adjustments, including the generator G kn and the discriminator D kn ; The generator G kn Accept the feature representation F of the missing modality data when the missing modality processing model is run missing As input, the feature representation F of the missing modal data missing After multi-level nonlinear transformation and feature extraction, the feature representation F of the full modal data is generated. multi Similar false features represent The discriminator D kn Accepting spurious feature representations when running a missing modality handling model and the feature representation of full modal data F multi As input, we use the feature representation F that distinguishes all modal data. multi and false feature representation Prior knowledge of which one is the feature representation F of full modal data multi , which is generated by the generator G kn Generated false feature representation The knowledge adversarial regularization module generator G kn With the discriminator D kn The optimization will be performed according to the following loss function: in, is the loss function of the knowledge adversarial regularization module; The process of training the missing modality processing model based on cross-modal mutual information and adversarial learning is as follows: s1: Full modal data X multi and missing modal data X missing Input to the cross-modal mutual information maximization encoder and optimize the loss function of the cross-modal mutual information maximization encoder The output of the cross-modal mutual information maximization encoder is the feature representation F of the full modal data. multi and the feature representation F of missing modal data missing ; s2: The feature representation F of the missing modal data obtained in step s1 missing and the feature representation of full modal data F multi Input to the knowledge adversarial regularization module, including the following steps: s2.1: Using knowledge to counter the generator G in the regularization module kn Feature representation F for missing modal data missing After multi-level nonlinear transformation and feature extraction, the false feature representation is obtained s2.2: Using knowledge to counter the discriminator D in the regularization module kn Prior knowledge of which one is the feature representation F of full modal data multi , which is generated by the generator G kn Generated false feature representation s2.3: Loss function of the knowledge-based regularization module Optimize generator G kn With the discriminator D kn , thereby aligning the distribution of different modal data in the latent space in a "soft alignment" manner, and obtaining the output of the knowledge adversarial regularization module as a false feature representation s3: represents the false features and the feature representation of full modal data F multi Input to the mixed attention upsampling layer module, the output is the enhanced feature F′ of the missing modality data missing , and the enhanced features F′ of full modality data multi ; s4: Enhance the features F′ of the missing modal data obtained in step s3 missing and the enhanced features F′ of full modal data multi Input to the entropy regularization module, including the following steps: s4.1: Generator G en Receive enhanced features F′ of full modality data multi As input, after multi-level nonlinear transformation and feature extraction, the primary segmentation map of the full modal data is obtained. And calculate the entropy map I of the full modal data multi ; s4.2: Generator G en Enhanced features F′ that accept missing modal data missing As input, after multi-level nonlinear transformation and feature extraction, the primary segmentation map of the missing modal data is obtained. And calculate the entropy map of missing modal data s4.3: Discriminator D kn Using prior knowledge, we can judge G en What is generated is the entropy map of the missing modal data Or the entropy diagram of full modal data I multi ; s4.4: Optimize G according to the loss function of the entropy regularization module en and D kn , so that the enhanced features F′ of the missing modal data missing The entropy distribution gradually approaches the enhanced feature F′ of the full modal data multi Entropy distribution of the minimization of the enhanced features F′ of the missing modal data missing The entropy of the entropy regularization module is obtained as the primary segmentation map of the missing modal data. s5: The primary segmentation map of the missing modality data obtained in step s4.4 Input into the decoder to obtain the final segmentation map; s6: The total loss function of the missing modality processing model is: in, The loss function of the encoder is to maximize the cross-modal mutual information, represents the loss function of the knowledge adversarial regularization module, represents the loss function of the entropy regularization module, and is the segmentation loss function of the missing modal data and the segmentation loss function of the full modal data in the segmentation task, λ missing ,λ multi , λ0, λ1, and λ2 are the weight coefficients of each loss function respectively.

2. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 1, characterized in that: The step 1 uses a certain strategy to generate the corresponding missing modal data from the full modal data. The specific method is: (1) Selective shielding: Randomly select the "non-missing" mode in the N-type full-modality data set for shielding; each mode includes two cases: "missing" and "non-missing", and the data set containing N full-modality is divided into 2 N Subsets, remove all the "missing" mode to get the "non-missing" mode, that is, 2 N -1 subset, each subset contains only one mode combination; (2) Data corruption: The missing mode is simulated by introducing noise into the data of a certain modality or reducing the quality of the modal data; when multimodal time series data is involved, the missing mode data is simulated by staggering the timestamps of different modal data.

3. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 2, characterized in that: The hybrid attention upsampling module is used to represent the false features and the feature representation of full modal data F multi After a certain upsampling layer to restore details, the hybrid attention upsampling module adopts the upsampling layer of the ResNeXt network and improves it, integrating the hybrid attention mechanism, so that the missing modality processing model can simultaneously integrate local and global information to better restore data details; The hybrid attention mechanism includes the channel attention mechanism CAM and the spatial attention mechanism SAM; the channel attention mechanism CAM is mainly used to adjust the weights between different channels in the feature map to highlight the representation of important features; The spatial attention mechanism SAM focuses on adjusting the weight of each pixel in the feature map to suppress information irrelevant to the target area; The input of the hybrid attention upsampling module is a false feature representation and the feature representation of full modal data F multi , and the two feature representations are processed in the same way, and the output is the enhanced feature F′ of the missing modal data missing , and the enhanced features F′ of full modality data multi ; Use input feature F to represent false feature representation and the feature representation of full modal data F multi , the enhanced feature F′ refers to the enhanced feature F′ of the missing modal data missing and the enhanced features F′ of full modal data multi ;Remember that the dimension of the input feature F is H1×W1×C1, where H1, W1, and C1 represent the height, width, and number of channels of the input feature F respectively; First, the channel attention weight matrix CAM(F) is calculated according to the following formula: F C,max =MLP(maxF(h,w)) CAM(F)=Sigmoid(F C,max +F C,avg ) Among them, MLP is a multi-layer perceptron consisting of two linear layers and one nonlinear activation layer, F C,max To perform global maximum pooling of the spatial dimension of the input feature F, and obtain the feature map through an MLP layer, F C，avg The feature map obtained by performing global average pooling of the spatial dimension of the input feature F through an MLP layer is F(h, w) the C1-dimensional feature vector of the input feature F at position (h, w), and CAM(F) is the channel attention weight matrix, which is obtained by adding the feature map F C，max and F C，avg It is obtained by adding and mapping with the Sigmoid activation function, where Sigmoid is the Sigmoid activation function; Secondly, the spatial attention weight matrix SAM(F) is calculated according to the following formula: F S,max =maxF(h,w) SAM(F)=Sigmoid(F S,max +F S,avg ) Among them, F S,max and F S,avg The two feature maps obtained by performing global maximum pooling and global average pooling on the input feature F in the channel dimension are respectively, and SAM(F) is the spatial attention weight matrix, which is obtained by splicing F by channel. S,max and F S,avg And it is obtained by mapping through the Sigmoid activation function; Afterwards, the channel attention weight matrix CAM(F) and the spatial attention weight matrix SAM(F) are expanded to the same dimension through the expansion operation and weighted summed to obtain the mixed attention weight matrix M. The specific formula is: M C =expand(CAM(F),H,W,C) M S =expand(SAM(F),H,W,C) M=αM C +βM S Among them, expand is the dimension expansion operation, α and β are the hyperparameters used for weighted summation; Finally, group convolution is applied to divide the output channels into g groups, with the convolution kernel size and number of channels in each group being the same, and residual connection is applied to obtain the enhanced feature F′. The specific formula is: G i =Conv group (F·M) for each groupi Among them, G i The output of the grouped convolution, each group processes a part of the channel, Conv group For grouped convolution operations, for eachgroupi performs the same operation on each group.

4. The method for processing missing modal data based on mutual information constraint and adversarial learning according to claim 3, characterized in that: The entropy regularization module uses entropy as an indicator to measure the prediction confidence of the missing modal processing model according to the entropy mapping principle, and is designed for the output of the hybrid attention upsampling layer. en and the discriminator D en Adversarial training between them to improve the confidence of the segmentation map output by the missing modality processing model; For full modal data X multi , generator G en Receive enhanced features F′ of full modality data multi As input, output is the primary segmentation map of the full modality data Each element value of this set represents the probability distribution of each pixel point for classifying T types of targets; for missing modal data X missing , generator G en Receive the enhanced features F′ for missing modal data missing As input, it outputs a primary segmentation map of the missing modality data. Each element value of this set represents the prediction of the probability distribution of each pixel point for classifying T types of targets; the entropy map I of the full modal data is calculated according to the following definition multi and entropy plots for missing modal data in, is the enhanced feature F′ of full modal data multi The probability of category t appearing at the specified position (h,w,c) is, for The logarithm of is the enhanced feature F′ for missing modal data missing The probability of category t appearing at the specified position (h,w,c) is, for The logarithm of When the generator G en After the part is completed, the discriminator D of the entropy regularization module is constructed en ; In adversarial training, the discriminator D en Aims to correctly distinguish which one is the entropy map of full modal data I multi , which is the entropy map of the missing modal data Generator G en With the discriminator D en The optimization is achieved according to the following loss function: in, is the loss function of the entropy regularization module.

5. The missing modal data processing method based on mutual information constraint and adversarial learning according to claim 4, characterized in that: The decoder is used to generate the primary segmentation map for the missing modality data. Further improve the details to improve the accuracy and quality of segmentation; use the decoder part of U-Net as the decoder of the missing modality processing model to convert the primary segmentation map of the missing modality data into Input to the decoder to obtain the final segmentation map.

Images