Multimodal medical data classification method based on inverse attention and adversarial training

Abstract

The invention relates to a multi-modal medical data classification method based on inverse attention and adversarial training. The method comprises the following steps: acquiring multi-modal medical data and preprocessing the multi-modal medical data; performing preliminary feature extraction on the preprocessed multi-modal medical data by adopting a k-best algorithm; calculating an inverse attention score of each feature by adopting an attention mechanism, and performing multi-modal feature fusion based on the inverse attention scores to obtain multi-modal feature data; and inputting the multi-modal feature data into a classification diagnosis module and a regression prediction module, carrying out adversarial training, and outputting a classification regression result corresponding to the multi-modal feature data. The features obtained under the multi-mode and multi-task condition are fully utilized, so that different modes are better combined, and potential features with statistical significance data, such as the age and the education degree of a subject, are mined; the accuracy, reliability and interpretability of the multi-modal medical data classification problem in the Alzheimer's disease pathological process are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal medical data classification method based on inverse attention and adversarial training. Background Art

[0002] Alzheimer's Disease (AD) is an irreversible neurodegenerative disease characterized by progressive loss of memory and a decline in general cognitive abilities. The incidence of the disease increases significantly with age, so early identification and diagnosis of AD is crucial for new drug development, disease prevention strategy formulation, and disease progression control.

[0003] Thanks to the rapid development of neuroimaging technology, especially functional MRI (fMRI), as an important branch of nuclear magnetic resonance imaging (MRI) technology, it has shown great potential in the early detection of AD. With its unique four-dimensional imaging capability, fMRI not only provides spatial and temporal resolution, but also avoids the harm of radiation, making it a key tool in the field of early diagnosis of AD. This technology can deeply reveal the microstructure and dynamic functional changes of the brain, and is particularly sensitive to capturing subtle pathological and physiological changes in the early stages of AD. More importantly, fMRI has become an indispensable core means for exploring the functional network of the human brain and the field of neural connectomics due to its high precision in capturing brain activity. It not only helps scientists better understand the pathogenesis of AD, but also provides a scientific basis for formulating more effective treatment strategies.

[0004] In the prior art, in the study of using multimodal data to establish a classification model for Alzheimer's disease tasks, most of the research only focuses on the binary classification problem between two pairs in the pathological process. The model is mainly divided into two parts: multimodal data fusion and feature selection, and the model network used for classification. First, since the medical image modality contains many features, selecting appropriate features will enable the model to exert its maximum efficiency, avoid insufficient features and fail to learn important features, and avoid redundant features from interfering with the model. Common feature selection algorithms on the market include: principal component analysis, local preservation projection, norm-based feature selection methods, etc. For incomplete modal data, interpolation is generally used to complete the modality. Common interpolation methods include K-nearest neighbor interpolation (KNN), zero interpolation, missing value analysis and expectation maximum algorithm (EM), and SVD interpolation. After the features are preliminarily processed, the data of different modalities are fused. Common fusion algorithms are divided into early fusion method: fusion at the feature level, and late fusion method: fusion at the decision layer. Second, the fused feature data is input into the classification model, the classification result is output, and the model is evaluated using indicators such as accuracy. The classification algorithm is mainly based on convolutional neural networks. Some work has introduced modules such as representation learning frameworks, multi-source feature learning methods, multi-task learning networks, and hypergraphs. By designing appropriate loss functions, such as multi-task joint loss, the classification and regression losses are combined to train the model and finally obtain the optimal solution. However, this method can only select two pathological processes for classification in the same time period. The developed model cannot solve the multi-classification problem well. The feature-level fusion cannot fully utilize the complementarity between multiple modal data, and the decision-level fusion ignores the correlation between modalities.

[0005] Therefore, in the relevant technology, there is an urgent need for a method that can improve the accuracy, reliability and interpretability of multimodal medical data classification problems in the pathological process of Alzheimer's disease. Summary of the invention

[0006] Based on this, it is necessary to provide a multimodal medical data classification method based on inverse attention and adversarial training, which can improve the accuracy, reliability and interpretability of multimodal medical data classification problems in the pathological process of Alzheimer's disease, in response to the above technical problems.

[0007] In a first aspect, the present application provides a multimodal medical data classification method based on inverse attention and adversarial training. The method comprises:

[0008] Acquire multimodal medical data and perform preprocessing;

[0009] Using k-best algorithm to perform preliminary feature extraction on the preprocessed multimodal medical data;

[0010] An attention mechanism is used to calculate an inverse attention score of each feature, and multimodal feature fusion is performed based on the inverse attention score to obtain multimodal feature data;

[0011] The multimodal feature data is input into a classification diagnosis module and a regression prediction module to perform adversarial training, and a classification regression result corresponding to the multimodal feature data is output.

[0012] Optionally, in one embodiment of the present application, the multimodal medical data includes resting state functional magnetic resonance imaging fMRI, structural magnetic resonance imaging MRI, biomarker data, neuropsychological cognitive scale data and cardiovascular and cerebrovascular risk factor data, and the preprocessing includes:

[0013] Establishing a brain network based on the resting state fMRI to obtain a brain functional connection matrix;

[0014] co-calibrating the biomarker data, removing duplicate data points, and calculating the Z score of the biomarker data;

[0015] The confirmatory factor analysis model was used to jointly calibrate the neuropsychological cognitive scale data and estimate the factor scores;

[0016] Calculating the principal components of the mixture of quantitative and qualitative data for the cardiovascular and cerebrovascular risk factor data, and calculating the score of the cardiovascular and cerebrovascular risk factor based on the principal components;

[0017] The KNN algorithm is used to interpolate missing data.

[0018] Optionally, in one embodiment of the present application, the calculation formula of the inverse attention score is:

[0019]

[0020] Among them, Attention is the attention score, Q is the query matrix Query, K is the key value Key, V is the real value Value, d k is the scaling factor, Feature i is the inverse attention score, n is the total number of features, i is the feature currently calculated, and j is the jth feature among all n features.

[0021] Optionally, in one embodiment of the present application, the adversarial training includes:

[0022] Calculate the loss of the classification task and the loss of the regression prediction task respectively;

[0023] fusing the loss with the multimodal feature data to form a mid-term fusion;

[0024] The loss of the classification task is fused into the feature vector for the regression task, and the loss of the regression prediction task is fused into the feature vector of the classification task.

[0025] Optionally, in one embodiment of the present application, the loss calculation formula of the classification task is as follows:

[0026] L ci =∑y i *log(y i ′ +ε)

[0027] Among them, y i is the value of the original data, y i ′ is the value of the predicted data, and ε is the error term.

[0028] Optionally, in one embodiment of the present application, the loss calculation formula of the regression prediction task is as follows:

[0029]

[0030] Among them, y i is the value of the original data, y i ′ is the value of the predicted data, and n is the total number of samples.

[0031] Optionally, in one embodiment of the present application, the method further includes:

[0032] The OpenMP parallel computing model is used to train the model in parallel.

[0033] In a second aspect, the present application also provides a multimodal medical data classification device based on inverse attention and adversarial training. The device comprises:

[0034] A data acquisition module, used to acquire multimodal medical data and perform preprocessing;

[0035] A feature extraction module, used for performing preliminary feature extraction on the preprocessed multimodal medical data using a k-best algorithm;

[0036] A feature fusion module, used to calculate the inverse attention score of each feature using an attention mechanism, and perform multimodal feature fusion based on the inverse attention score to obtain multimodal feature data;

[0037] The multi-task adversarial training module is used to input the multimodal feature data into the classification diagnosis module and the regression prediction module, perform adversarial training, and output the classification regression results corresponding to the multimodal feature data.

[0038] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the methods described in the above embodiments.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in each of the above embodiments are implemented.

[0040] The above-mentioned multimodal medical data classification method based on inverse attention and adversarial training first obtains multimodal medical data and preprocesses it; then, the k-best algorithm is used to perform preliminary feature extraction on the preprocessed multimodal medical data; then, the attention mechanism is used to calculate the inverse attention score of each feature, and multimodal feature fusion is performed based on the inverse attention score to obtain multimodal feature data; finally, the multimodal feature data is input into the classification diagnosis module and the regression prediction module, adversarial training is performed, and the classification regression results corresponding to the multimodal feature data are output. In other words, an inverse attention mechanism that can be used to explore the correlation between modalities of medical multimodal data is proposed, and the correlation of other features to a single feature is explored to determine whether the feature is important. In addition, the original features are preliminarily fused before the start of training, and the loss calculated from the age, years of education, and neuropsychological cognitive assessment results predicted in each round of training are added to the input data as the features of the next round of training, and fused in the later stage to form a mid-term fusion. At the same time, the loss of classification is added to the regression prediction, and the loss of regression prediction is also added to the classification, so that classification and regression form adversarial training. The features obtained under multi-modal and multi-task conditions were fully utilized to better combine different modalities, and the potential features of statistically significant data such as the age and education level of the subjects were mined, which improved the accuracy, reliability and interpretability of multi-modal medical data classification problems in the pathological process of Alzheimer's disease. At the same time, the support vector machine was used as the basic model for classification and regression, which improved the adaptability of the model to small samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is an application environment diagram of a multimodal medical data classification method based on inverse attention and adversarial training in one embodiment;

[0042] Figure 2 is a flowchart of a multimodal medical data classification method based on inverse attention and adversarial training in one embodiment;

[0043] Figure 3 is a flowchart of specific steps of a multimodal medical data classification method based on inverse attention and adversarial training in one embodiment;

[0044] Figure 4 is a structural block diagram of a multimodal medical data classification device based on inverse attention and adversarial training in one embodiment;

[0045] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The multimodal medical data classification method based on inverse attention and adversarial training provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0048] In one embodiment, Figure 2 As shown in the figure, a multimodal medical data classification method based on inverse attention and adversarial training is provided. Figure 1 The server in the example is used to illustrate the following steps:

[0049] S201: Acquire multimodal medical data and perform preprocessing.

[0050] In the embodiment of the present application, first, multimodal medical data is obtained, and multimodal medical data refers to relevant modal data of normal individuals, early mild cognitive impairment patients (EMCI), late mild cognitive impairment patients (LMCI) and Alzheimer's disease patients (AD) from the ADSP (Alzheimer's Disease Sequencing Project) project, such as images obtained by resting state functional magnetic resonance imaging (including long resting state functional magnetic resonance imaging) (fMRI), structural magnetic resonance imaging (MRI), important biomarkers extracted from cerebrospinal fluid (CSF), results of neuropsychological cognitive assessment, and modal data such as cardiovascular and cerebrovascular risk factors of patients. And pre-process them respectively.

[0051] Specifically, in one embodiment of the present application, the multimodal medical data includes resting state functional magnetic resonance imaging (fMRI), structural magnetic resonance imaging (MRI), biomarker data, neuropsychological cognitive scale data, and cardiovascular and cerebrovascular risk factor data, and the preprocessing includes:

[0052] S301: Establishing a brain network based on the resting state magnetic resonance imaging (fMRI) to obtain a brain functional connection matrix.

[0053] S303: Co-calibrate the biomarker data, delete duplicate data points, and calculate the Z score of the biomarker data.

[0054] S305: The neuropsychological cognitive scale data are co-calibrated using a confirmatory factor analysis model to estimate factor scores.

[0055] S307: Calculate the principal component of the mixture of quantitative and qualitative data for the cardiovascular and cerebrovascular risk factor data, and calculate the score of the cardiovascular and cerebrovascular risk factor based on the principal component.

[0056] S309: Use KNN algorithm to interpolate missing data.

[0057] In one embodiment of the present application, for resting-state magnetic resonance imaging fMRI data, the CONN toolkit is used to perform time correction, head motion correction, spatial alignment, and spatial smoothing steps to minimize errors caused by data acquisition and physiological characteristics, and a first-order linear model is used for fitting to obtain the results of hemodynamic response. A generalized linear model is used for analysis to obtain the properties of the group brain network, and a brain network is established. The same region of interest ROI is selected (i.e., 32 brain networks and brain regions, gray matter under 132 atlas templates, and 2-dimensional QC_timeseries original global signal changes and frame-by-frame displacement time series information, a total of 167 ROI information) to obtain the brain functional connection matrix as the input data of this modality.

[0058] For the important biomarker data extracted from cerebrospinal fluid (CSF), co-calibration was performed and duplicate data points were removed. Biomarker level outliers were calculated using log10-transformed biomarker values, and data outside Q3+1.5IQR and Q1-1.5IQR were marked as outliers (IQR refers to interquartile range, Q1 and Q3 represent the first and third quartile ranges). Afterwards, the Z score of each coordinated fluid biomarker was calculated using the scaling function in R software through all non-outlier data points as the input data for this modality.

[0059] For the results of neuropsychological cognitive assessment, i.e., the neuropsychological cognitive scale data, various assessment items such as the Mini-Mental State Examination were first classified into four indicators according to memory, executive function, language, and visual space, and the confirmatory factor analysis model was used for joint calibration to derive the item parameters of all items. These item parameters were used to estimate the factor scores and the corresponding standard deviations for each field of each study. The obtained factor scores were used as the input data of this modality.

[0060] For the patient's cardiovascular and cerebrovascular risk factor data, the principal components of the quantitative and qualitative data mixture were calculated in the PCAMix software package, and the dimensionality reduction was performed to capture the maximum variance explained by the four cerebrovascular risk factors (hypertension, heart disease, diabetes, and body mass index). The scores of cardiovascular risk factors were calculated based on their principal components as the input data of this modality.

[0061] Except for the fMRI modality, the other modalities all have a small amount of missing data, and the KNN algorithm is used to interpolate the missing values, selecting the top 50% of the data in the same category that are close to it for interpolation.

[0062] S203: Performing preliminary feature extraction on the preprocessed multimodal medical data using a k-best algorithm.

[0063] In the embodiment of the present application, since the brain signal image contains many features, direct fusion will cause modal collapse, so the k-best algorithm is first used for preliminary feature extraction. Specifically, K most relevant features are selected from tens of thousands of data obtained after preprocessing, and a feature matrix is ​​preliminarily constructed.

[0064] S205: Using the attention mechanism to calculate the inverse attention score of each feature, and performing multimodal feature fusion based on the inverse attention score to obtain multimodal feature data.

[0065] In the embodiment of the present application, after obtaining the preliminary feature matrix through preliminary feature extraction, the attention mechanism is used to calculate the attention between a single feature and each feature. Combined with the characteristics of brain signals, when all other signals focus on a certain signal, the signal will show a stronger connection, and the attention, that is, the attention score, is transposed, and the inverse attention score of each feature is calculated, that is, the attention weight score of all other selected features to a single feature, to dig out the correlation between potential modalities. And the obtained inverse attention score is multiplied with the original element to achieve multimodal feature fusion and obtain multimodal feature data.

[0066] In one embodiment of the present application, the calculation formula of the inverse attention score is:

[0067]

[0068] Among them, Attention is the attention score, Q is the query matrix Query, K is the key value Key, V is the real value Value, d k is the scaling factor, Feature i is the inverse attention score, n is the total number of features, i is the feature currently calculated, and j is the jth feature among all n features.

[0069] S207: Input the multimodal feature data into a classification diagnosis module and a regression prediction module, perform adversarial training, and output classification regression results corresponding to the multimodal feature data.

[0070] In an embodiment of the present application, the fused multimodal feature data are respectively input into the classification diagnosis module and the regression prediction module, and the support vector machine algorithm is used as the model of the classification diagnosis module and the regression prediction module to perform adversarial training, and finally the classification regression results corresponding to the multimodal feature data are output.

[0071] Specifically, in one embodiment of the present application, the adversarial training includes:

[0072] S401: Calculate the loss of the classification task and the loss of the regression prediction task respectively.

[0073] S403: Fusing the loss with the multimodal feature data to form a mid-term fusion.

[0074] S405: Fusion the loss of the classification task into the feature vector for the regression task, and fusion the loss of the regression prediction task into the feature vector of the classification task.

[0075] In one embodiment of the present application, the loss of the classification task and the loss of the regression prediction task are calculated separately, the cross entropy algorithm is combined with the support vector machine using the Platt scaling principle to calculate the probability value and calculate the loss of the classification task, and the root mean square error is used as the loss of the regression prediction task such as age, education level, and neuropsychological test score.

[0076] In one embodiment of the present application, the loss calculation formula for the classification task is as follows:

[0077] L ci =∑y i *log(y i ′ +ε)

[0078] Among them, y i is the value of the original data, y i ′ is the value of the predicted data, and ε is the error term.

[0079] In one embodiment of the present application, the loss calculation formula for the regression prediction task is as follows:

[0080]

[0081] Among them, y i is the value of the original data, y i ′ is the value of the predicted data, and n is the total number of samples.

[0082] The above loss is used as the feature of the next round of training and is fused with the original features fused in the early stage before the training starts to form a mid-term fusion. The loss of the classification task is fused into the feature vector used for the regression task, and the loss of the regression task is fused into the feature vector of the classification task, so that it forms adversarial training and makes full use of the features obtained under multi-modal and multi-task conditions.

[0083] In one embodiment of the present application, the method further includes:

[0084] The OpenMP parallel computing model is used to train the model in parallel.

[0085] In an embodiment of the present application, since selecting different numbers of features will affect the prediction and diagnosis effect of the model, training with different numbers of features separately will cause a large time overhead. Considering dividing the training and using the OpenMP parallel computing model to perform parallel training on the model. Specifically, let the total number of tasks be n, and the i-th feature selection task will select i features in the fMRI modality. During the training process, the matrix with fewer features takes less time when calculating the subsequent attention mechanism scores and when performing classification and regression tasks. First, sort these n tasks according to the number of selected features to obtain the task queue T:

[0086] T = {T a1 , T a2 , T a3 … T ai}

[0087] such that a1 < a2 < … ai. Then divide the tasks among k threads to complete. The ai-th feature is given to the (i mod k)-th thread for calculation, that is:

[0088] Ta i → i mod k

[0089] In this embodiment, by specifically dividing the tasks according to the distribution characteristics of the feature selection tasks and the data, each thread has the opportunity to process relatively small tasks and relatively large tasks, ensuring that no thread will wait too long in advance, resulting in a waste of computing resources.

[0090] In the above-mentioned multimodal medical data classification method based on inverse attention and adversarial training, first, multimodal medical data is obtained and preprocessed; then, the k-best algorithm is used to perform preliminary feature extraction on the preprocessed multimodal medical data; then, the attention mechanism is used to calculate the inverse attention score of each feature, and multimodal feature fusion is performed based on the inverse attention score to obtain multimodal feature data; finally, the multimodal feature data is input into the classification diagnosis module and the regression prediction module, adversarial training is performed, and the classification regression results corresponding to the multimodal feature data are output. In other words, an inverse attention mechanism that can be used to explore the correlation between modalities of medical multimodal data is proposed, and the correlation of other features to a single feature is explored to determine whether the feature is important. In addition, the original features are preliminarily fused before the start of training, and the loss calculated from the age, years of education, and neuropsychological cognitive assessment results predicted in each round of training are added to the input data as the features of the next round of training, and fused in the later stage to form a mid-term fusion. At the same time, the loss of classification is added to the regression prediction, and the loss of regression prediction is also added to the classification, so that classification and regression form adversarial training. The features obtained under multi-modal and multi-task conditions were fully utilized to better combine different modalities, and the potential features of statistically significant data such as the age and education level of the subjects were mined, which improved the accuracy, reliability and interpretability of multi-modal medical data classification problems in the pathological process of Alzheimer's disease. At the same time, the support vector machine was used as the basic model for classification and regression, which improved the adaptability of the model to small samples.

[0091] The following is a specific example to illustrate the specific implementation steps of the multimodal medical data classification method based on inverse attention and adversarial training of the present application. Figure 3 As shown, first, S501, multimodal medical data is obtained and preprocessed, and the multimodal medical data includes resting state functional magnetic resonance imaging fMRI, structural magnetic resonance imaging MRI, biomarker data, neuropsychological cognitive scale data and cardiovascular and cerebrovascular risk factor data. Specifically, S503-S511, a brain network is established based on the resting state magnetic resonance imaging fMRI to obtain a brain functional connection matrix; the biomarker data is co-calibrated, duplicate data points are deleted, and the Z score of the biomarker data is calculated; the neuropsychological cognitive scale data is co-calibrated using a confirmatory factor analysis model to estimate factor scores; the principal components in the mixture of quantitative and qualitative data of the cardiovascular and cerebrovascular risk factor data are calculated, and the scores of cardiovascular and cerebrovascular risk factors are calculated based on the principal components; the missing data are interpolated using the KNN algorithm.

[0092] Then, in S513, the k-best algorithm is used to perform preliminary feature extraction on the preprocessed multimodal medical data. Then, in S515, the inverse attention score of each feature is calculated using the attention mechanism, and multimodal feature fusion is performed based on the inverse attention score to obtain multimodal feature data. In S517, the calculation formula of the inverse attention score is:

[0093]

[0094] Among them, Attention is the attention score, Q is the query matrix Query, K is the key value Key, V is the real value Value, d k is the scaling factor, Feature i is the inverse attention score, n is the total number of features, i is the feature currently calculated, and j is the jth feature among all n features.

[0095] Afterwards, S519, the multimodal feature data is input into the classification diagnosis module and the regression prediction module, adversarial training is performed, and the classification regression results corresponding to the multimodal feature data are output. Specifically, S521-S525, respectively calculate the loss of the classification task and the loss of the regression prediction task; fuse the loss with the multimodal feature data to form a mid-term fusion; fuse the loss of the classification task into the feature vector for the regression task, and fuse the loss of the regression prediction task into the feature vector of the classification task. S527, the loss calculation formula of the classification task is as follows:

[0096] L ci =Σy i *log(y i ′ +ε)

[0097] Among them, y i is the value of the original data, y i ′ is the value of the predicted data, and ε is the error term.

[0098] S529, the loss calculation formula of the regression prediction task is as follows:

[0099]

[0100] Among them, y i is the value of the original data, y i ′ is the value of the predicted data, and n is the total number of samples.

[0101] S531, using the OpenMP parallel computing model to perform parallel training on the model.

[0102] Figure 3In the figure, AD represents patients with Alzheimer's disease; CN represents normal individuals; EMCI represents early mild cognitive impairment, and LMCI represents late mild cognitive impairment; age represents the predicted result of the subject's age, edu represents the predicted result of the subject's years of education; mem, exf, lan, vsp represent the predicted results of four neuropsychological assessments: memory, behavior, language, and vision, respectively.

[0103] The accuracy (ACC) of the classification task refers to the probability of classifying a sample into the correct category; the recall rate (Recall) is calculated, which is the probability that the category is correctly predicted; the precision rate (precision) is calculated, which is the proportion of the number of correct samples in the correctly predicted samples; and finally the F1 score (F1-score) is calculated, which is the average of the recall rate and the precision rate. Finally, they are weighted by multiplying them by a weight, and the proportion of the number of samples in each category to the total number of samples in all categories is used as the weight to obtain the weighted recall rate (weight recall), weighted precision rate (weight precision), and weighted F1 score (weight F1-score). The specific calculation formula is as follows:

[0104]

[0105] Among them, n represents the total number of samples, m and M represent categories (AD, CN, EMCI, LMCI), and the meanings of TP, TN, FP, and FN are shown in the following table:

[0106]

[0107] The regression task evaluation uses the mean square error (MSE), which is the mean of the sum of squares of the errors between the predicted data and the original data; the root mean square error (RMSE), which is the square root of the MSE; and the mean absolute error (MAE), which is the average of the absolute errors between the predicted data and the original values. The R2 coefficient of determination measures the degree of fit of the predicted value to the original value. The specific calculation formula is as follows:

[0108]

[0109] Among them, y i Represents the value of the original data, y i ′ Represents the value of the predicted data, and n represents the total number of samples.

[0110] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0111] Based on the same inventive concept, the embodiment of the present application also provides a multimodal medical data classification device based on inverse attention and adversarial training for implementing the multimodal medical data classification method based on inverse attention and adversarial training involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the multimodal medical data classification device based on inverse attention and adversarial training provided below can refer to the limitations of the multimodal medical data classification method based on inverse attention and adversarial training above, and will not be repeated here.

[0112] In one embodiment, Figure 4 As shown, a multimodal medical data classification device 400 based on inverse attention and adversarial training is provided, comprising: a data acquisition module 401, a feature extraction module 403, a feature fusion module 405 and a multi-task adversarial training module 407, wherein:

[0113] The data acquisition module 401 is used to acquire multimodal medical data and perform preprocessing.

[0114] The feature extraction module 403 is used to perform preliminary feature extraction on the preprocessed multimodal medical data using a k-best algorithm.

[0115] The feature fusion module 405 is used to calculate the inverse attention score of each feature using the attention mechanism, and perform multimodal feature fusion based on the inverse attention score to obtain multimodal feature data.

[0116] The multi-task adversarial training module 407 is used to input the multi-modal feature data into the classification diagnosis module and the regression prediction module, perform adversarial training, and output the classification regression results corresponding to the multi-modal feature data.

[0117] In one embodiment of the present application, the multimodal medical data includes resting state functional magnetic resonance imaging fMRI, structural magnetic resonance imaging MRI, biomarker data, neuropsychological cognitive scale data and cardiovascular and cerebrovascular risk factor data, and the data acquisition module is also used to:

[0118] Establishing a brain network based on the resting state fMRI to obtain a brain functional connection matrix;

[0119] co-calibrating the biomarker data, removing duplicate data points, and calculating the Z score of the biomarker data;

[0120] The confirmatory factor analysis model was used to jointly calibrate the neuropsychological cognitive scale data and estimate the factor scores;

[0121] Calculating the principal components of the mixture of quantitative and qualitative data for the cardiovascular and cerebrovascular risk factor data, and calculating the scores of the cardiovascular and cerebrovascular risk factors based on the principal components;

[0122] The KNN algorithm is used to interpolate missing data.

[0123] In one embodiment of the present application, the calculation formula of the inverse attention score is:

[0124]

[0125] Among them, Attention is the attention score, Q is the query matrix Query, K is the key value Key, V is the real value Value, d k is the scaling factor, Feature i is the inverse attention score, n is the total number of features, i is the feature currently calculated, and j is the jth feature among all n features.

[0126] In one embodiment of the present application, the multi-task adversarial training module is also used to:

[0127] Calculate the loss of the classification task and the loss of the regression prediction task respectively;

[0128] fusing the loss with the multimodal feature data to form a mid-term fusion;

[0129] The loss of the classification task is fused into the feature vector for the regression task, and the loss of the regression prediction task is fused into the feature vector of the classification task.

[0130] In one embodiment of the present application, the loss calculation formula for the classification task is as follows:

[0131] L ci =Σy i *log(yi ′ +ε)

[0132] Among them, y i is the value of the original data, y i ′ is the value of the predicted data, and ε is the error term.

[0133] In one embodiment of the present application, the loss calculation formula for the regression prediction task is as follows:

[0134]

[0135] Among them, y i is the value of the original data, y i ′ is the value of the predicted data, and n is the total number of samples.

[0136] In one embodiment of the present application, the multi-task adversarial training module is also used to:

[0137] The OpenMP parallel computing model is used to train the model in parallel.

[0138] Each module in the above-mentioned multimodal medical data classification device based on inverse attention and adversarial training can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0139] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a multimodal medical data classification method based on inverse attention and adversarial training is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0140] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0143] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multimodal medical data classification method based on inverse attention and adversarial training, characterized in that: The method comprises: Acquire multimodal medical data and perform preprocessing; Using k-best algorithm to perform preliminary feature extraction on the preprocessed multimodal medical data; An attention mechanism is used to calculate an inverse attention score of each feature, and multimodal feature fusion is performed based on the inverse attention score to obtain multimodal feature data; The multimodal feature data is input into a classification diagnosis module and a regression prediction module to perform adversarial training, and a classification regression result corresponding to the multimodal feature data is output.

2. The multimodal medical data classification method based on inverse attention and adversarial training according to claim 1, characterized in that: The multimodal medical data includes resting state functional magnetic resonance imaging (fMRI), structural magnetic resonance imaging (MRI), biomarker data, neuropsychological cognitive scale data and cardiovascular and cerebrovascular risk factor data, and the preprocessing includes: Establishing a brain network based on the resting state fMRI to obtain a brain functional connection matrix; co-calibrating the biomarker data, removing duplicate data points, and calculating the Z score of the biomarker data; The confirmatory factor analysis model was used to jointly calibrate the neuropsychological cognitive scale data and estimate the factor scores; Calculating the principal components of the mixture of quantitative and qualitative data for the cardiovascular and cerebrovascular risk factor data, and calculating the score of the cardiovascular and cerebrovascular risk factor based on the principal components; The KNN algorithm is used to interpolate missing data.

3. The multimodal medical data classification method based on inverse attention and adversarial training according to claim 1, characterized in that: The calculation formula of the inverse attention score is: Among them, Attention is the attention score, Q is the query matrix Query, K is the key value Key, V is the real value Value, d k is the scaling factor, Feature i is the inverse attention score, n is the total number of features, i is the feature currently calculated, and j is the jth feature among all n features.

4. The multimodal medical data classification method based on inverse attention and adversarial training according to claim 1, characterized in that: The adversarial training comprises: Calculate the loss of the classification task and the loss of the regression prediction task respectively; fusing the loss with the multimodal feature data to form a mid-term fusion; The loss of the classification task is fused into the feature vector for the regression task, and the loss of the regression prediction task is fused into the feature vector of the classification task.

5. The multimodal medical data classification method based on inverse attention and adversarial training according to claim 4, characterized in that: The loss calculation formula for the classification task is as follows: L ci =∑y i *log(y′ i +e) Among them, y i is the value of the original data, y′ i is the value of the predicted data, and ε is the error term.

6. The multimodal medical data classification method based on inverse attention and adversarial training according to claim 4, characterized in that: The loss calculation formula for the regression prediction task is as follows: Among them, y i is the value of the original data, y′ i is the value of the predicted data, and n is the total number of samples.

7. The multimodal medical data classification method based on inverse attention and adversarial training according to claim 1, characterized in that: The method further comprises: The OpenMP parallel computing model is used to train the model in parallel.

8. A multimodal medical data classification device based on inverse attention and adversarial training, characterized in that: The device comprises: A data acquisition module, used to acquire multimodal medical data and perform preprocessing; A feature extraction module, used for performing preliminary feature extraction on the preprocessed multimodal medical data using a k-best algorithm; A feature fusion module, used to calculate the inverse attention score of each feature using an attention mechanism, and perform multimodal feature fusion based on the inverse attention score to obtain multimodal feature data; The multi-task adversarial training module is used to input the multimodal feature data into the classification diagnosis module and the regression prediction module, perform adversarial training, and output the classification regression results corresponding to the multimodal feature data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images