Multimodal Medical Missing Data Completion Method and Device Based on Data Relevance Mining

Abstract

The present invention discloses a multi-modal medical missing data completion method and device based on data correlation mining. The method includes: obtaining original multi-modal medical data with data missing problems. According to the obtained original multi-modal data, a generator model and a masked energy discriminator model based on data correlation mining are constructed. The constructed generator model and masked energy discriminator model based on data correlation mining are combined, and the combined model is optimized and trained through game using the original multi-modal medical data. The missing data is completed through the trained generator model to obtain complete data. The present invention aims at the problem of multi-modal data missing, and adopts a method based on mining the correlation between multi-modal data to effectively complete data, with the advantages of high completion accuracy and low model complexity.

Description

Technical Field

[0001] This application relates to the technical field of data completion, and particularly to a multi-modal medical missing data completion method and device based on data correlation mining. Background Art

[0002] Multi-modal data is a common data type in the medical field, including but not limited to modalities such as images, texts, audios, tables, etc. Due to reasons such as medical data collection device failures, unstable external data collection environments, and data privacy protection, there are a large number of missing values in multi-modal medical data, which affects the effect of subsequent multi-modal intelligent analysis. Accurately and reliably filling in the missing values of medical data is one of the ways to solve this problem.

[0003] Currently, regarding the problem of missing data completion, scholars at home and abroad have done some work, but these works still have limitations. For example, the patent with the publication number CN111581189A discloses "a method and device for completing missing air quality detection data", and the completion effect is 30% higher than other disclosed methods. However, it needs to train a modal mapping model for each "source modality - target modality", resulting in high resource consumption and not fully mining the correlation relationship between different modal data. The multi-modal medical missing data completion method based on data correlation mining proposed in the present invention uses a generator model that mines data correlation by designing a joint autoencoder, uses the information of existing data to predict missing values, and avoids training redundant mapping models. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a multi-modal medical missing data completion method and device based on data correlation mining to overcome the problems existing in the above-mentioned prior art.

[0005] According to the first aspect of the embodiments of the present application, a multi-modal medical missing data completion method based on data correlation mining is provided, including:

[0006] S1: According to the original multi-modal medical missing data, use a mask matrix to describe its missing situation and construct a multi-modal data matrix;

[0007] S2: Construct a generator model based on data correlation mining. The generator model consists of a multi-modal joint autoencoder and a target view decoder. The joint autoencoder is used to process data of different modalities simultaneously, and the decoder maps the multi-modal joint features into the feature space of the target modality;

[0008] S3: Use a masked energy divergence adversarial loss function to construct a masked energy discriminator model based on the mask energy;

[0009] S4: Input the multi-modal data matrix into the generator model for calculation to generate target modal data, then calculate the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the target modal data, weight these two loss functions, and update the parameters of the generator model;

[0010] S5: Input the multi-modal data matrix into the discriminator model, recalculate the masked energy divergence adversarial loss function in S4, and update the parameters of the discriminator model;

[0011] S6: Repeat S4 and S5 until the maximum number of iterations is reached to complete the game optimization training process of the generator model and the discriminator model;

[0012] S7: Input the multi-modal medical missing data to be completed into the trained generator model for missing data completion.

[0013] Optionally, both the generator model and the masked energy discriminator model based on data correlation mining are deep neural network structures composed of multiple activation functions.

[0014] Optionally, inputting the multi-modal data matrix into the generator model for calculation to generate target modal data, then calculating the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the target modal data, and weighting these two loss functions to update the parameters of the generator model includes:

[0015] S41. Sample two different batches of samples P and Q and their missing matrices M p and M q ;

[0016] S42. Input sample P and its missing matrix M p , sample Q and its missing matrix M q into the generator model G, and the generator model G mines the correlation relationship between data modalities through the attention mechanism to reconstruct the data of the t-th modality of the sample and

[0017] S43. Calculate the loss function L of the generator model G according to the data and , and the loss function L G , the loss function L G includes: the weighted reconstruction loss function L r and the masked energy divergence adversarial loss function L of the discriminator model D a ;

[0018] S44. The generator model G minimizes its loss function L GPerform model training to obtain the current optimal generator model parameters.

[0019] Optionally, the data and are respectively expressed as:

[0020]

[0021] Optionally, the loss function L G is expressed as:

[0022] L G = L a (P t , Q t ) + φ · [L r (P t ) + L r (Q t )](2)

[0023] where the hyperparameter φ is used to weigh the weight between the weighted reconstruction loss function L r and the masked energy divergence adversarial loss function L a .

[0024] Optionally, the calculation formula of the weighted reconstruction loss function L r is:

[0025]

[0026] where the hyperparameter φ is used to weigh the weight between the weighted reconstruction loss function L r and the masked energy divergence adversarial loss function L a , and similarly, L r (Q t ) is calculated.

[0027] Optionally, the calculation formula of the masked energy divergence adversarial loss function L a is:

[0028]

[0029] where b represents the data batch size, and represent the empirical measures of the reconstructed matrices P t and Q t generated by the generator model G, and represent the empirical measures of the target modality data matrices P t and Q t , δ represents the Dirac distribution, and ∈ m represents the empirical measure The masked energy divergence on it is calculated as follows:

[0030]

[0031] Among them, OT m represents the masked optimal transport metric, and is calculated as follows:

[0032]

[0033] where λ is a hyperparameter, is from the set of the transport plan matrix, represents and of the Frobenius inner product, represents the masked cost matrix, and is calculated as follows:

[0034]

[0035] According to the second aspect of the embodiments of the present application, there is provided a multi-modal medical missing data completion device based on data correlation mining, including:

[0036] The first construction module is configured to use a masked matrix to describe the missing situation according to the original multi-modal medical missing data, and construct a multi-modal data matrix;

[0037] The second construction module is configured to construct a generator model based on data correlation mining, the generator model is composed of a multi-modal joint autoencoder and a target view decoder, and the joint autoencoder is used to process data of different modalities simultaneously, and the decoder maps the multi-modal joint features into the feature space of the target modality;

[0038] The third construction module is configured to construct a masked energy discriminator model based on masked energy by using a masked energy divergence adversarial loss function;

[0039] The first calculation module is configured to input the multi-modal data matrix into the generator model for calculation, generate target modality data, then calculate the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the target modality data, and weight these two loss functions to update the parameters of the generator model;

[0040] The second calculation module is configured to input the multi-modal data matrix into the discriminator model, and calculate the masked energy divergence adversarial loss function in the first calculation module again to update the parameters of the discriminator model;

[0041] A training module, configured to repeatedly execute a first calculation module and a second calculation module until a maximum number of iterations is reached, thereby completing the game optimization training process of the generator model and the discriminator model;

[0042] A completion module, configured to input multi-modal medical missing data to be completed into the trained generator model for missing data completion.

[0043] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including:

[0044] One or more processors;

[0045] A memory, configured to store one or more programs;

[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0047] According to a fourth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method as described in the first aspect are implemented.

[0048] The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

[0049] As can be seen from the above embodiments, compared with the prior art, the present application adopts a generator model based on data correlation mining, and at the same time uses observable data of multiple modalities to complete the target modality, overcoming the computational redundancy problem of the "source modality - target modality" completion mode; adopts a mask energy discriminator model based on mask energy, overcoming the problem of gradient disappearance in game optimization training, and improving the stability of the training process and the model data completion effect. The completion accuracy of the multi-modal medical missing data completion method based on data correlation mining proposed by the present invention is improved by about 6.90% compared with the currently optimal publicly disclosed method, and has a lower model complexity, and has broad application potential in the field of medical multi-modal intelligent analysis.

[0050] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0052] Figure 1 It is a flowchart of a multi-modal medical missing data completion method based on data correlation mining shown according to an exemplary embodiment.

[0053] Figure 2 It is a diagram of the overall model architecture shown according to an exemplary embodiment.

[0054] Figure 3 It is a block diagram of a multi-modal medical missing data completion device based on data correlation mining shown according to an exemplary embodiment. Detailed implementation manners

[0055] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0056] The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "the", and "said" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0057] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0058] Figure 1 It is a flowchart of a multi-modal medical missing data completion method based on data correlation mining shown according to an exemplary embodiment. Figure 2 It is a diagram of the overall model architecture shown according to an exemplary embodiment. The method may include the following steps:

[0059] S1: According to the original multi-modal medical missing data, use a mask matrix to describe its missing situation and construct a multi-modal data matrix;

[0060] S2: Construct a generator model based on data correlation mining. The generator model consists of a multi-modal joint autoencoder and a target view decoder. The joint autoencoder is used to process data of different modalities simultaneously, and the decoder maps the multi-modal joint features to the feature space of the target modality;

[0061] S3: Adopt a masked energy divergence adversarial loss function to construct a masked energy discriminator model based on masked energy;

[0062] S4: Input the multi-modal data matrix into the generator model for calculation to generate target modality data. Then calculate the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the target modality data, weight these two loss functions, and update the parameters of the generator model;

[0063] S5: Input the multi-modal data matrix into the discriminator model, and calculate the masked energy divergence adversarial loss function in S4 again to update the parameters of the discriminator model;

[0064] S6: Repeat S4 and S5 until the maximum number of iterations is reached to complete the game optimization training process of the generator model and the discriminator model;

[0065] S7: Input the multi-modal medical missing data to be completed into the trained generator model to complete the missing data filling.

[0066] As can be seen from the above embodiments, the present application adopts a generator model based on data correlation mining, utilizes the observable data of multiple modalities simultaneously to complete the target modality; adopts a masked energy discriminator model based on masked energy, combines with the generator model, and uses the multi-modal data matrix to perform a game training process to optimize the data filling effect of the generator model, and uses the trained generator model to fill the multi-modal medical missing data.

[0067] S1: According to the original multi-modal medical missing data, use a masked matrix to describe its missing situation and construct a multi-modal medical missing data matrix; specifically including:

[0068] Step S101: Obtain a multi-modal medical data matrix X with data missing problems. Further, X has n samples containing v modalities:

[0069] X = {X 1 , …, X v} = {x1, x2, …, x n} (1)

[0070] Step S102: Based on the obtained multi-modal medical data matrix X, calculate the missing matrix M corresponding to the missing status of the data in X. If a feature of the data matrix X exists, the missing status at the corresponding position in the missing matrix M is 1; if a feature of the data matrix X is missing, the missing status at the corresponding position in the missing matrix M is 0.

[0071] By calculating the missing matrix M, the missing situation of the multi-modal medical missing data matrix can be clearly indicated, so as to clarify the data part that needs to be completed and the data part that the model can observe and utilize.

[0072] Due to factors such as interference of collection devices, storage device failures, and data privacy issues, some modalities or features of multi-modal medical data are missing. For example, X-ray images are missing or some features are missing, electrocardiogram signals are missing or some features are missing, and breath sound signals are missing or some features are missing.

[0073] S2: Construct a generator model based on data correlation mining. The generator model consists of a multi-modal joint autoencoder and a target modality decoder. The joint autoencoder is used to process data of different modalities simultaneously, and the decoder maps the multi-modal joint features to the feature space of the target modality;

[0074] Specifically, the multi-modal joint autoencoder adopted by the generator model uses different types of neural embedding networks to process data of different modalities. For example, a convolutional neural network is used to embed image data;

[0075] The generator model adopts a cross-modal self-attention module to mine the correlation of data of different modalities and obtain the joint expression of data of different modalities in the same feature space;

[0076] The target modality decoder of the generator model uses a corresponding type of neural network. For example, a transposed convolutional network is used to reconstruct image data and map the joint expression to the feature space of the target modality;

[0077] This design scheme can simultaneously utilize all observable multi-modal data for data mining, thus avoiding training different autoencoder models for each pair of "source modality - target modality" and reducing the complexity of the model.

[0078] S3: Adopt a masked energy divergence adversarial loss function to construct a masked energy discriminator model based on masked energy;

[0079] Specifically, the discriminator model calculates the input reconstructed target modality data and the missing matrix to obtain the masked energy divergence ∈ m , for subsequent calculation of the masked energy divergence adversarial loss function;

[0080] The masked energy divergence combines the differentiable masked optimal transport metric and the unbiased estimated energy divergence, thereby obtaining a highly discriminative incomplete data distribution metric, which solves the problem of gradient vanishing in the game optimization training process and improves the game optimization training effect.

[0081] The generator model and the masked energy discriminator model based on data correlation mining are both deep neural network structures composed of multiple activation functions.

[0082] S4: Input the multi-modal medical missing data matrix into the generator model for calculation to generate target modal data, then calculate the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the target modal data, and weight these two loss functions to update the parameters of the generator model; specifically including:

[0083] S41, sample two different batches of samples P and Q and their missing matrices M p and M q ; specifically including:

[0084] S401: Randomly sample b samples from the multi-modal medical missing data matrix to form a batch of samples P and a missing matrix M describing the missing situation of P p ;

[0085] S402: Randomly sample b samples from the multi-modal medical missing data matrix to form a batch of samples Q and a missing matrix M describing the missing situation of Q Q ;

[0086] Among them, b is a hyperparameter controlling the number of batch samples. The missing matrix describes the missing situation of different modal data in the samples. Random sampling ensures that the model does not tend to fit a specific data distribution, thereby improving the generalization ability of the model.

[0087] S42, input the batch of samples P and its missing matrix M p , the batch of samples Q and its missing matrix M q into the generator model G. The generator model G mines the correlation relationship between data modalities through the attention mechanism and reconstructs the data of the t-th modality of the sample and Specifically including:

[0088] S421: Input the samples in the sample batch P into the generator model G;

[0089] S422: The multi-modal joint autoencoder of the generator model G embeds different modal data and uses the cross-modal attention mechanism to mine the correlation relationship between different modal data;

[0090] S423: The generator model G obtains the reconstructed data of each modality of the sample through the target modality decoder;

[0091] S424: Repeat S421, S422, and S423 until all samples in the sample batch P are processed to obtain the reconstructed data of the sample batch P

[0092] S425: Execute the above steps on the sample batch Q to obtain the reconstructed data of the sample batch Q

[0093] where the data and are respectively expressed as:

[0094]

[0095] S43. Calculate the loss function L of the generator model G according to the data and The loss function L G , the loss function L G includes: the weighted reconstruction loss function L r and the masked energy divergence adversarial loss function L of the discriminator model D a ; specifically including:

[0096] S431: Calculate and P t the reconstruction loss function between

[0097] S432: Calculate and Q t the reconstruction loss function between

[0098] S433: Use the discriminator model described in S3 to calculate the masked energy divergence adversarial loss function L a ;

[0099] S433: Calculate the loss function L of the generator model G G ;

[0100] where the loss function L G is expressed as:

[0101] L G = L a (P t , Q t ) + φ · [L r (P t ) + L r (Q t )] (3)

[0102] Among them, the hyperparameter φ is used to weigh the weighted reconstruction loss function L r and the masked energy divergence adversarial loss function L a between the weights.

[0103] The calculation formula of the weighted reconstruction loss function L r is as follows:

[0104]

[0105] Among them, the hyperparameter φ is used to weigh the weighted reconstruction loss function L r and the masked energy divergence adversarial loss function L a between the weights. Similarly, L r (Q t ) is calculated;

[0106] The calculation formula of the masked energy divergence adversarial loss function L a is as follows:

[0107]

[0108] Among them, b represents the data batch size, and represent the empirical measures of the reconstructed matrices P t and Q t generated by the generator model G, and represent the empirical measures of the target modality data matrices P t and Q t . δ represents the Dirac distribution, and ∈ m represents the empirical measure on the masked energy divergence, and the calculation method is as follows:

[0109]

[0110] Among them, OT m represents the masked optimal transport measure, and the calculation method is as follows:

[0111]

[0112] Among them, λ is a hyperparameter, is from the set of the transport plan matrix, represents and of the Frobenius inner product, represents the masked cost matrix, and the calculation method is as follows:

[0113]

[0114] The reconstruction loss function promotes the consistency of the distribution between the reconstructed multimodal data obtained by the generator model and the original multimodal data; the masked energy divergence adversarial loss function is used to discriminate the quality of the reconstructed multimodal data obtained by the generator model; the weighted sum of the two can promote the generator model to mine the correlation relationship between different modal data and generate high-quality reconstructed data.

[0115] S44. The generator model G minimizes its loss function L G to perform model training and obtain the current optimal generator model parameters; specifically including:

[0116] S441: Fix the model parameters of the discriminator model D;

[0117] S442: Update the model parameters of the generator model G with the goal of minimizing the loss function L G

[0118] S5: Input the multimodal data matrix into the discriminator model, and recalculate the masked energy divergence adversarial loss function in S4 to update the parameters of the discriminator model; specifically including:

[0119] S51: Fix the model parameters of the generator model G;

[0120] S52: Input the multimodal data matrix into the discriminator model G, and calculate the masked energy divergence adversarial loss function in S4;

[0121] S53: Update the model parameters of the discriminator model D with the goal of minimizing the masked energy divergence adversarial loss function.

[0122] S6: Repeat S4 and S5 until the maximum number of iterations is reached to complete the game optimization training process of the generator model and the discriminator model; specifically including:

[0123] S61: Set the maximum number of iterations K;

[0124] S62: Execute step S4 to control the discriminator model and optimize the generator model;

[0125] S63: Execute step S5 to control the generator model and optimize the discriminator model;

[0126] S64: Repeat steps S62 and S63 in sequence for K times to complete the game optimization training process of the generator model and the discriminator model.

[0127] ​S7: Input the incomplete multi-modal medical missing data into the trained generator model for missing data completion. Specifically, it includes:

[0128] S71: Prepare the incomplete multi-modal medical missing data X and the missing matrix M describing its missing situation X , where X includes data of v different modalities;

[0129] S72: Input X and M X into the trained generator model G to obtain the reconstructed multi-modal medical data

[0130] S73: Calculate the completed multi-modal medical data based on the multi-modal medical missing data X and the reconstructed multi-modal medical data Specifically as follows: Specifically as follows:

[0131]

[0132] Corresponding to the embodiment of the multi-modal medical missing data completion method based on data correlation mining described above, the present application also provides an embodiment of a multi-modal medical missing data completion device based on data correlation mining.

[0133] Figure 3 is a block diagram of a multi-modal medical missing data completion device based on data correlation mining shown according to an exemplary embodiment. Refer to Figure 3 , this device includes:

[0134] The first construction module 1 is used to describe its missing situation using a mask matrix according to the original multi-modal medical missing data and construct a multi-modal data matrix;

[0135] The second construction module 2 is used to construct a generator model based on data correlation mining. The generator model consists of a multi-modal joint autoencoder and a target view decoder. The joint autoencoder is used to process data of different modalities simultaneously, and the decoder maps the multi-modal joint features to the feature space of the target modality;

[0136] The third construction module 3 is used to construct a mask energy discriminator model based on mask energy using a mask energy divergence adversarial loss function;

[0137] The first calculation module 4 is used to input the multi-modal data matrix into the generator model for calculation, generate target modality data, then calculate the weighted reconstruction loss function and the mask energy divergence adversarial loss function of the target modality data, and weight these two loss functions to update the parameters of the generator model;

[0138] The second calculation module 5 is configured to input the multi-modal data matrix into the discriminator model, recalculate the masked energy divergence adversarial loss function in the first calculation module, and update the parameters of the discriminator model;

[0139] The training module 6 is configured to repeatedly execute the first calculation module and the second calculation module until the maximum number of iterations is reached, and complete the game optimization training process of the generator model and the discriminator model;

[0140] The completion module 7 is configured to input the multi-modal medical missing data to be completed into the trained generator model for missing data completion.

[0141] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0142] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial descriptions of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present application. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0143] Correspondingly, the present application further provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-modal medical missing data completion method based on data correlation mining as described above.

[0144] Correspondingly, the present application further provides a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the multi-modal medical missing data completion method based on data correlation mining as described above is implemented.

[0145] After considering the specification and practicing the content disclosed herein, those skilled in the art will readily think of other implementation manners of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include the common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the claims.

[0146] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A multi-modal medical missing data completion method based on data correlation mining, characterized in that, Including: S1: According to the original multi-modal medical missing data, use a masking matrix to describe its missing situation and construct a multi-modal data matrix; S2: Construct a generator model based on data correlation mining. The generator model consists of a multi-modal joint autoencoder and a target view decoder. The joint autoencoder is used to process data of different modalities simultaneously, and the decoder maps the multi-modal joint features to the feature space of the target modality; S3: Adopt a masked energy divergence adversarial loss function to construct a masked energy discriminator model based on masked energy; S4: Input the multi-modal data matrix into the generator model for calculation to generate target modality data, then calculate the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the target modality data, and weight these two loss functions to update the parameters of the generator model; S5: Input the multi-modal data matrix into the discriminator model, and calculate the masked energy divergence adversarial loss function in S4 again to update the parameters of the discriminator model; S6: Repeat S4 and S5 until the maximum number of iterations is reached to complete the game optimization training process of the generator model and the discriminator model; S7: Input the multi-modal medical missing data to be completed into the trained generator model for missing data completion; Among them, inputting the multi-modal data matrix into the generator model for calculation to generate target modality data, then calculating the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the target modality data, and weighting these two loss functions to update the parameters of the generator model includes: S41, sample two different batches of samples P and Q and their missing matrices and ; S42, input the sample P and its missing matrix , the sample Q and its missing matrix into the generator model G, which mines the correlation relationships between data modalities through the attention mechanism and reconstructs the data of the t-th modality of the sample and ; S43, according to the said data and , calculate the loss function of the generator model G , the said loss function includes: the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the discriminator model D ; S44. The generator model G minimizes its loss function to train the model and obtain the current optimal generator model parameters.

2. The method according to claim 1, characterized in that, Both the generator model based on data correlation mining and the masked energy discriminator model are deep neural network structures composed of multiple activation functions.

3. The method according to claim 1, wherein The said data and are respectively represented as: (1)。 4. The method according to claim 1, characterized in that The loss function is expressed as: (2)； Among them, the hyperparameter is used to weigh the weighted reconstruction loss function and the masked energy divergence adversarial loss function between the weights.

5. The method according to claim 1, wherein The weighted reconstruction loss function has the following calculation formula: (3)； Among them, the hyperparameters are used to weigh the weighted reconstruction loss function and the masked energy divergence adversarial loss function The weights between them are calculated in the same way to obtain .

6. The method according to claim 1, characterized in that, The masked energy divergence adversarial loss function has the following calculation formula: (4)； Among them, represents the data batch size, and represents the reconstruction matrix generated by the generator model G and the empirical measure of, and represents the target modality data matrix and the empirical measure of, represents the Dirac distribution, represents the empirical measure the masked energy divergence on, which is calculated as follows: (5)； Among them, represents the mask optimal transmission metric, and the calculation method is as follows: (6)； where is a hyperparameter, , is a transmission plan matrix from the set , denotes and 's Frobenius inner product, denotes the mask cost matrix, and the calculation method is as follows: (7)。 7. A multimodal medical missing data completion device based on data correlation mining, characterized in that, Including: The first construction module is used to construct a multi-modal data matrix according to the original multi-modal medical missing data by using a masking matrix to describe its missing situation; The second construction module is used to construct a generator model based on data correlation mining. The generator model consists of a multi-modal joint autoencoder and a target view decoder. The joint autoencoder is used to process data of different modalities simultaneously, and the decoder maps the multi-modal joint features to the feature space of the target modality; The third construction module is used to construct a masked energy discriminator model based on masked energy by adopting a masked energy divergence adversarial loss function; The first calculation module is used to input the multi-modal data matrix into the generator model for calculation to generate target modality data, then calculate the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the target modality data, and weight these two loss functions to update the parameters of the generator model; The second calculation module is used to input the multi-modal data matrix into the discriminator model and calculate the masked energy divergence adversarial loss function in the first calculation module again to update the parameters of the discriminator model; A training module, configured to repeatedly execute a first calculation module and a second calculation module until a maximum number of iterations is reached, so as to complete the game optimization training process of the generator model and the discriminator model; A completion module, configured to input the multi-modal medical missing data to be completed into the trained generator model for missing data completion; Wherein, inputting the multi-modal data matrix into the generator model for calculation to generate target modal data, and then calculating the weighted reconstruction loss function and the masked energy divergence adversarial loss function of the target modal data, and weighting these two loss functions to update the parameters of the generator model, including: S41, sample two different batches of samples P and Q and their missing matrices and ; S42, input the sample P and its missing matrix , the sample Q and its missing matrix into the generator model G. The generator model G mines the correlation relationships between data modalities through the attention mechanism and reconstructs the data of the t-th modality of the sample and ; S43. Calculate the loss function of the generator model G based on the said data and , where the loss function includes: a weighted reconstruction loss function and a masked energy divergence adversarial loss function of the discriminator model D ; ​ S44, the generator model G minimizes its loss function to train the model and obtain the current optimal generator model parameters.

8. An electronic device, characterized in that, including: One or more processors; A memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-6.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When the instruction is executed by the processor, the steps of the method according to any one of claims 1-6 are implemented.

Images