Multi-modal fusion system and method for Alzheimer's disease

Through multimodal data preprocessing and an attention mechanism-based fusion method, combining clinical, genetic and image data, the problem of limitations of single modal analysis in early prediction of Alzheimer's disease is solved, and more accurate understanding of the disease and Personalized treatment improves the comprehensiveness and accuracy of the test.

CN118942670APending Publication Date: 2024-11-12NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410936645.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing technology lacks unified detection standards in the early prediction of Alzheimer's disease, and the analysis of single modal data is limited, and it is impossible to effectively combine multiple modal information to mine correlations, resulting in inaccurate detection.

Method used

Multimodal data preprocessing, feature extraction and multimodal data fusion methods based on attention mechanisms are adopted, combining clinical, genetic and image data, features are extracted through self-attention and cross-modal attention mechanisms, and in the fully connected layer Carry out fusion and build information fusion characteristics to assist in diagnosis.

Benefits of technology

It improves the prediction accuracy of Alzheimer's disease, provides more personalized treatment measures, enhances comprehensive understanding of the disease, and improves the effectiveness of preventive testing and screening.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942670A_ABST
    Figure CN118942670A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal fusion system and method for Alzheimer's disease, and belongs to the field of artificial intelligence and medicines.The system is composed of a multi-modal data preprocessing layer, a multi-modal data feature extraction layer and a multi-modal data fusion layer based on an attention mechanism; the multi-modal data preprocessing layer is used for preprocessing clinical data, genetic data and image data respectively; a multi-modal data feature extraction layer performs feature extraction on the three pieces of preprocessed single-modal data; and the multi-modal data fusion layer based on the attention mechanism takes the data of the three modalities as input, and extracts single and fusion features of the modalities by using the attention mechanism for fusion. Compared with the prior art, the detection of the Alzheimer's disease better meets the actual clinical diagnosis requirement, the accuracy of prediction of the Alzheimer's disease is improved, the illness state of the patient is known more comprehensively, more personalized treatment measures are provided for the patient, and meanwhile the corresponding effect on prevention, detection and screening of the Alzheimer's disease can be achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and medicine, and specifically relates to a multimodal fusion system and method for Alzheimer's disease. Background Art

[0002] Alzheimer's disease is a neurodegenerative disease and one of the most common causes of dementia in the elderly. Patients with the disease may experience memory loss, repetitive speech or behavior, disorientation and confusion about time. As the disease progresses, the patient's cognitive ability will gradually decline, and language disorders, decreased judgment and problem-solving abilities, as well as behavioral and emotional changes may occur. Ultimately, patients often die due to loss of physical function.

[0003] Early prediction of Alzheimer's disease is crucial for the treatment and care of patients. However, there is no unified detection standard so far. Current detection methods mainly include clinical symptom assessment, neuropsychological testing, imaging analysis, biomarker analysis, and genetic material analysis, etc. These methods have their own limitations. For example, clinical symptom assessment may be affected by subjective factors, and neuropsychological testing can only provide partial information. Structural magnetic resonance imaging is a non-invasive method that obtains information related to Alzheimer's disease through subjective assessment of brain morphological changes, which requires rich experience. And most studies on Alzheimer's disease are based only on single-modality data analysis for early prediction.

[0004] In summary, existing methods all have their limitations, and using multimodal data to detect Alzheimer's disease is more in line with actual clinical diagnosis needs. Therefore, seeking a method that can combine information from multiple modalities and mine the correlation between multimodal data or construct information fusion features is an urgent problem to be solved for the early prediction of Alzheimer's disease. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal fusion method for Alzheimer's disease, which can combine information from multiple modalities, construct information fusion features and mine the correlation between multimodal data to assist in the clinical diagnosis of Alzheimer's disease and perform preventive screening and diagnosis of Alzheimer's disease.

[0006] The technical solution proposed in the present invention includes multimodal data preprocessing, multimodal data feature extraction, multimodal data fusion based on attention mechanism and testing fusion model.

[0007] The specific plan is as follows:

[0008] A multimodal fusion system for Alzheimer's disease, comprising:

[0009] The multimodal data preprocessing layer is used to preprocess the clinical, genetic, and image data in the data and obtain a single dataset for each modality. The data comes from a public dataset of Alzheimer's disease.

[0010] The multimodal data feature extraction layer is used to extract features from clinical, genetic, and image data respectively to obtain more representative feature data for each single modality;

[0011] The multimodal data fusion layer based on the attention mechanism is used to take the data of three modalities as input, extract the features of the three modalities using the self-attention mechanism and the cross-modal attention mechanism, and send the output of self-attention fusion of the data of a single modality, the output of cross-modal attention fusion between two modalities, and the output of cross-modal interaction between the three modalities into a fully connected layer for fusion.

[0012] As a further improvement of the present invention, in the multimodal data preprocessing layer: patient medical data with three modal data of clinical, genetic and image are used, and the missing values ​​and abnormal values ​​in the clinical data are processed, the genetic data is quality controlled and dimensionally reduced according to the list of genetic genes related to Alzheimer's disease, and the image data is compared and corrected.

[0013] As a further improvement of the present invention, in the multimodal data feature extraction layer, a deep neural network is used as a feature representation method for clinical structural data, the genetic data is whole genome sequencing, and a stacked denoising autoencoder is used for feature extraction, and a convolutional neural network is used to extract spatial information features in image data.

[0014] As a further improvement of the present invention, in the multimodal data fusion layer based on the attention mechanism, a unimodal self-attention mechanism and a cross-modal attention mechanism are used for feature fusion, which is divided into a unimodal self-attention layer, a bimodal attention interaction layer, a trimodal joint attention layer and a fusion representation layer.

[0015] As a further improvement of the present invention, the single-modal self-attention layer uses a self-attention mechanism to respectively learn the internal correlation of the single modality of the clinical, genetic and image data after feature extraction, and the formed feature results are spliced ​​and fused to obtain the output of the single-modal self-attention layer.

[0016] As a further improvement of the present invention, the bimodal attention interaction layer uses the data of a single modality that has completed the unimodal self-attention interaction to perform pairwise fusion through cross-modal attention, and concatenates the obtained six cross-modal features in pairs to finally obtain three cross-modal feature representations, which are connected to obtain the output of the bimodal attention interaction layer.

[0017] As a further improvement of the present invention, the trimodal joint attention layer is represented by the three cross-modal features obtained, which are combined in pairs through the cross-modal attention mechanism to obtain six cross-modal features, which are connected to form the output of the trimodal joint attention layer.

[0018] As a further improvement of the present invention, the fusion representation layer sends the output of the unimodal self-attention layer, the output of the bimodal attention interaction layer and the output of the trimodal joint attention layer to a fully connected layer for fusion to obtain the final fusion result.

[0019] A multimodal fusion method for Alzheimer's disease, comprising:

[0020] S1, multimodal data preprocessing stage: preprocess the clinical, genetic and image data in the data and obtain a single data set for each modality;

[0021] The clinical data include neurological examinations, cognitive assessments, and basic patient information, and contain 29 features. Features that are directly indicative of Alzheimer's disease need to be deleted, missing values ​​need to be supplemented, and continuous data need to be normalized.

[0022] The genetic data is the whole genome sequencing (WGS) of the patient. The data is first quality controlled by genotype quality, allele frequency, and genotype value, and then compared with the gene list related to Alzheimer's disease to retain dimensionality reduction to complete the preprocessing of the genetic data.

[0023] The image data were processed through several correction steps, first using the N4 algorithm to correct the bias field, then performing an affine linear alignment of the atlas, and finally stripping the skull from each image to obtain the preprocessed image.

[0024] S2, multimodal data feature extraction stage:

[0025] Deep neural networks are used as the representation method for clinical structural data, genetic data is whole genome sequencing, stacked denoising autoencoders are used for feature extraction, and convolutional neural networks are used to extract spatial information features in image data. Before fusion, the data of the three modalities need to be projected into a common vector space with the same feature dimension.

[0026] S3, multimodal data fusion stage based on attention mechanism:

[0027] Construct a multimodal data fusion mechanism based on attention mechanism, which includes unimodal self-attention layer, bimodal attention interaction layer, trimodal joint attention layer and fusion representation layer.

[0028] First, the data of the three modalities after feature extraction are sent to the unimodal self-attention layer, and the self-attention mechanism is used to better interact with the input of the single modal data to obtain three unimodal interaction features. The three unimodal interaction features are simply connected to obtain the output of the unimodal self-attention layer. Among them, the formula of the self-attention mechanism is:

[0029]

[0030] Among them, SA n is the mode F n The new self-attention interaction features obtained after self-attention interaction, is the i-th linear projection with dimension d, is the jth linear projection with dimension d, where d is the dimension of each projection space.

[0031] Then, the single modal data that has completed the unimodal self-attention interaction is sent to the bimodal attention interaction layer, and the multimodal data is fused pairwise using the cross-modal attention mechanism. The formula of the cross-modal attention mechanism is:

[0032]

[0033] Among them, CA n,m Indicates the modality SA that completes the unimodal self-attention interaction n Modal SA with unimodal self-attention interaction completed m The new cross-modal features generated by fusion through the cross-modal attention mechanism, d represents the dimension of the projection space.

[0034] The six cross-modal features are concatenated in pairs according to the modality to obtain three cross-modal features, and then the three cross-modal features are connected to obtain the output of the bimodal attention interaction layer.

[0035] Next, in order to further extract and capture richer information between data of multiple modalities, the three cross-modal features obtained in the bimodal attention interaction layer are combined in pairs, and six cross-modal features are obtained through cross-modal attention. These six features are connected to obtain the output of the trimodal joint attention layer.

[0036] Finally, the outputs of the unimodal self-attention layer, the bimodal attention interaction layer, and the trimodal joint attention layer are sent to a fully connected layer to obtain the representation of multimodal data after fusion based on the attention mechanism.

[0037] Compared with the existing invention, the beneficial effects of the present invention are:

[0038] (1) The present invention combines data from three modalities: clinical, genetic, and imaging. The detection of Alzheimer's disease is more in line with the actual clinical diagnosis needs, which helps to improve the accuracy of Alzheimer's disease prediction and provide a more comprehensive understanding of the patient's condition, thereby providing patients with more personalized treatment measures. At the same time, it can also play a corresponding role in the prevention, detection and screening of Alzheimer's disease.

[0039] (2) The multimodal data fusion method based on the attention mechanism designed by the present invention first preprocesses and extracts features of the data of the three modalities to enhance the usefulness of the data, and then fuses the data of the three modalities with the data of different modalities respectively. First, the processed data in the single modality is self-fused through the self-attention mechanism. Secondly, the data processed by the single modality self-attention mechanism in the dual modality is fused through the cross-modal attention mechanism. Finally, in the tri-modality, the data processed by the dual modality is fused through the cross-modal attention mechanism. The data of the three modalities are extracted and then fused to form the output of the entire multimodal data fusion method. The construction of this fusion model helps to further explore the correlation between the data of different modalities, so as to better combine the data of different modalities for auxiliary diagnosis, prediction and treatment.

[0040] (3) The fusion architecture designed by the present invention is clear, simple and easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of a flow chart in an embodiment of the present invention.

[0042] Figure 2 It is a structural diagram of a multimodal fusion method based on an attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0044] See also Figure 1 , Figure 2 The present invention provides a multimodal fusion method for Alzheimer's disease, and the detailed steps are as follows:

[0045] Step 1: multimodal data preprocessing.

[0046] The clinical data contains 29 features, including neurological examinations, cognitive assessment scales, and basic information about patients, and the data format is quantitative, categorical, or binary. Data that directly indicate that the patient has Alzheimer's disease will be deleted, and the missing values ​​and outliers will be processed using the mean filling method. The categorical data will be converted into features using one-hot encoding, and the continuous value features will be normalized.

[0047] The genetic data is the whole genome sequencing data of the patient, and the initial VCF file contains 3 million SNP data. First, quality control is performed through methods such as genotype quality, minor allele frequency, and genotype value filtering to delete redundant sites. Secondly, the gene list known to be associated with Alzheimer's disease is further compared and retained. The final data is divided into three types: no allele, only one allele, or both alleles.

[0048] The image data were preprocessed through some correction steps. First, the bias field was corrected using the N4 algorithm, then the MIN152 atlas was aligned linearly, and finally the skull was stripped from each image to obtain the preprocessed image.

[0049] Step 2: Extract features from multimodal data.

[0050] A deep neural network is used as a representation method for clinical structural data. The deep neural network includes an input layer, two hidden layers, and a one-dimensional output layer. An activation function is used between the input layer and the hidden layer, and the output layer uses an activation function and a cross entropy loss function.

[0051] The genetic data is whole genome sequencing, and feature extraction is performed using stacked denoising autoencoders. The stacked denoising autoencoders extract features from the original input by combining multiple denoising autoencoders. The hidden layer of the previous denoising autoencoder is used as the input of the next denoising autoencoder. The stacked denoising autoencoder includes two training processes: unsupervised pre-training and supervised fine-tuning. Unsupervised training uses a zero-masking noise algorithm to add noise to the data to reset certain feature values ​​to zero, and each denoising autoencoder is trained separately. The denoising autoencoder uses an encoder and a decoder to reconstruct the original data. After unsupervised training, all denoising autoencoders are connected and then combined with a classifier to fine-tune the parameters of the cascade network.

[0052] Convolutional neural networks are used to extract spatial information features from image data. The convolution layer first convolves each input image with the learned kernel filter, then adds a bias term and applies a nonlinear activation function, and finally generates a feature vector through the filter. The convolution operation formula is:

[0053]

[0054] Among them, x, y, z represent the given image pixel position, Represents the connection between the k-th feature map of layer l-1 and the j-th 3D kernel weight of layer l. is the kth feature map of the first l-1 layers, δ x ,δ y ,δ z is the kernel size corresponding to the x, y, z coordinates, output is the convolution response. After convolution, each convolution layer is followed by an activation function Sigmoid:

[0055]

[0056] in, is the bias term of the j-th feature map of the l-th layer, and the three-dimensional feature vector It is obtained by adding the feature maps of different convolution kernels.

[0057] Before fusion, since different modalities have different feature dimensions, which will affect the final fusion feature representation, it is necessary to project the data of different modalities into a common vector space with the same feature dimensions.

[0058]

[0059] Among them, W c , W g , W i are the projection matrices of clinical data, genetic data, and image data, respectively. c , F g , F i They are the data after feature extraction corresponding to clinical data, genetic data, and image data respectively. They are the features of clinical data, genetic data, and image data after being projected into the common vector space.

[0060] Step 3: Build Figure 2 The multimodal data fusion mechanism based on the attention mechanism includes a unimodal self-attention layer, a bimodal attention interaction layer, a trimodal joint attention layer and a fusion representation layer.

[0061] First, the data of the three modalities after feature extraction are sent to the single-modal self-attention layer, and the self-attention mechanism is used to better interact with the input of the single-modal data to obtain three single-modal interaction features. Among them, the self-attention mechanism is defined as:

[0062]

[0063] Among them, SA n is the mode F n The new self-attention interaction features obtained after self-attention interaction, is the i-th linear projection with dimension d, where d is the dimension of each projection space.

[0064] The three unimodal interaction features are:

[0065] S c =SA(F c )

[0066] S g =SA(F g )

[0067] S i =SA(F i )

[0068] The output of the unimodal self-attention layer is:

[0069]

[0070] Then, the single modal data that have completed the unimodal self-attention interaction are sent to the bimodal attention interaction layer respectively, and the multimodal data are fused pairwise using the cross-modal attention mechanism, where the cross-modal attention mechanism is defined as:

[0071]

[0072] Among them, CA n,m Indicates the modality SA that completes the unimodal self-attention interaction n Modal SA with unimodal self-attention interaction completed m The new cross-modal features generated by fusion through the cross-modal attention mechanism, d represents the dimension of the projection space.

[0073] The six cross-modal features are:

[0074] CA c,g =CA(SA c ,SA g )

[0075] CA c,i =CA(SA c ,SA i )

[0076] CA g,c =CA9SA g ,SA c )

[0077] CA g,i =CA(SA g ,SA i )

[0078] CA i,c =CA(SA i ,SA c )

[0079] CA i,g =CA(SA i ,SA g )

[0080] After concatenating the six cross-modal features in pairs according to the modality, three cross-modal features are obtained, which are:

[0081]

[0082] The three cross-modal features are then concatenated to obtain the output of the bimodal attention interaction layer.

[0083]

[0084] Next, in order to further extract richer information between data that capture multiple modalities, the three cross-modal features obtained in the bimodal attention interaction layer are combined in pairs. The six cross-modal features obtained through cross-modal attention are:

[0085]

[0086] Concatenating these six features yields the output of the trimodal joint attention layer:

[0087]

[0088] Finally, the outputs of the unimodal self-attention layer, the bimodal attention interaction layer, and the trimodal joint attention layer are fed into a fully connected layer to obtain the representation of multimodal data after fusion based on the attention mechanism:

[0089]

[0090] Step 4: Train the fusion model.

[0091] All the data processed in step 1 are randomly divided into training and test sets in a ratio of 4:1. Medical data features are generally sparse matrices, and the gradient changes greatly during training, so the Adam optimizer is used to better explain the hyperparameter settings. Five-fold cross validation is used for the training set to select the best hyperparameters and the best model for each modality and multimodal fusion to avoid the overfitting problem of limited data sets. Finally, the model is applied to the test set for model evaluation.

[0092] Step 5: When fusing multimodal data of Alzheimer's disease in actual application, the trained fusion model is initialized. And the data is input into the model and fused according to the above steps.

[0093] The above method was simulated and the Alzheimer's disease neuroimaging project dataset was selected. The accuracy evaluation indicators used were accuracy (ACC), sensitivity (SEN), specificity (SPE) and F1-score. The experimental results are as follows:

[0094]

[0095] It is not difficult to find from the table that the method of the present invention that uses the attention mechanism to extract the relationship between multiple modal data for fusion has higher performance in terms of the four evaluation indicators of ACC, SEN, SPE, and F1score than the fusion method that only uses single-modal data or simply splices bimodal data.

[0096] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A multimodal fusion system for Alzheimer's disease, characterized in that: The system comprises: The multimodal data preprocessing layer is used to preprocess the clinical, genetic, and image data in the data and obtain a single dataset for each modality. The data comes from a public dataset of Alzheimer's disease. The multimodal data feature extraction layer is used to extract features from clinical, genetic, and image data respectively to obtain more representative feature data for each single modality; The multimodal data fusion layer based on the attention mechanism is used to take the data of three modalities as input, extract the features of the three modalities using the self-attention mechanism and the cross-modal attention mechanism, and send the output of self-attention fusion of the data of a single modality, the output of cross-modal attention fusion between two modalities, and the output of cross-modal interaction between the three modalities into a fully connected layer for fusion.

2. The multimodal fusion method for Alzheimer's disease according to claim 1, characterized in that: In the multimodal data preprocessing layer: patient medical data with three modal data, namely clinical, genetic and image, are used, and the missing values ​​and outliers in the clinical data are processed, the genetic data is quality controlled and dimensionally reduced according to the list of genetic genes related to Alzheimer's disease, and the image data is compared and corrected.

3. The multimodal fusion method for Alzheimer's disease according to claim 1, characterized in that: In the multimodal data feature extraction layer, a deep neural network is used as the feature representation method for clinical structural data, the genetic data is whole genome sequencing, and a stacked denoising autoencoder is used for feature extraction, and a convolutional neural network is used to extract spatial information features in image data.

4. The multimodal fusion method for Alzheimer's disease according to claim 3, characterized in that: In the convolutional neural network, the convolution layer first convolves each input image with the learned kernel filter, then adds a bias term and applies a nonlinear activation function, and finally generates a feature vector through the filter; the convolution operation formula is: Among them, x, y, z represent the given image pixel position, Represents the connection between the k-th feature map of layer l-1 and the j-th 3D kernel weight of layer l; is the kth feature map of the first l-1 layers, δ x ,δ y ,δ z is the kernel size corresponding to the x, y, z coordinates, output is the convolution response; after convolution, each convolution layer is followed by the activation function Sigmoid: in, is the deviation term of the j-th feature map of the l-th layer, and the three-dimensional feature vector F j l (x, y, z) is obtained by adding the feature maps of different convolution kernels; Before fusion, the data of different modalities are projected into a common vector space with the same feature dimension: Among them, W c , W g , W i are the projection matrices of clinical data, genetic data, and image data respectively; F c , F g , F i They are the data after feature extraction corresponding to clinical data, genetic data, and image data; They are the features of clinical data, genetic data, and image data after being projected into the common vector space.

5. The multimodal fusion method for Alzheimer's disease according to claim 1, characterized in that: In the multimodal data fusion layer based on the attention mechanism, a unimodal self-attention mechanism and a cross-modal attention mechanism are used for feature fusion, which is divided into a unimodal self-attention layer, a bimodal attention interaction layer, a trimodal joint attention layer and a fusion representation layer.

6. The multimodal fusion method for Alzheimer's disease according to claim 5, characterized in that: The unimodal self-attention layer uses the self-attention mechanism to learn the internal correlation of the single modality of clinical, genetic and image data after feature extraction, and the formed feature results are spliced ​​and fused to obtain the output of the unimodal self-attention layer; the bimodal attention interaction layer uses the data of a single modality that has completed the unimodal self-attention interaction to perform pairwise fusion through cross-modal attention, and splices the obtained six cross-modal features in pairs to finally obtain three cross-modal feature representations, which are connected to obtain the output of the bimodal attention interaction layer; the trimodal joint attention layer uses the obtained three cross-modal feature representations to perform pairwise combination through the cross-modal attention mechanism to obtain six cross-modal features, which are connected to form the output of the trimodal joint attention layer.

7. The multimodal fusion method for Alzheimer's disease according to claim 6, characterized in that: The fusion representation layer sends the output of the unimodal self-attention layer, the output of the bimodal attention interaction layer, and the output of the trimodal joint attention layer to a fully connected layer for fusion to obtain the final fusion result.

8. A multimodal fusion method for Alzheimer's disease, characterized in that: A system as claimed in any one of claims 1 to 7, comprising: S1, multimodal data preprocessing stage: preprocess the clinical, genetic and image data in the data and obtain a single data set for each modality; Clinical data included neurological examinations, cognitive assessments, and basic patient information, and contained 29 features, of which those that were directly indicative of Alzheimer's disease were deleted, missing values ​​were supplemented, and continuous data were normalized; The genetic data is the whole genome sequencing of the patient. First, the data quality is controlled by genotype quality, allele frequency and genotype value, and then compared with the gene list related to Alzheimer's disease to retain dimension reduction to complete the preprocessing of genetic data; The image data was processed through some correction steps, first using the N4 algorithm to correct the bias field, then performing affine linear alignment on the atlas, and finally stripping the skull from each image to obtain the preprocessed image; S2, multimodal data feature extraction stage: Deep neural networks were used as the representation method for clinical structural data. Whole genome sequencing was used for genetic data. Stacked denoising autoencoders were used for feature extraction. Convolutional neural networks were used to extract spatial information features in image data. Before fusion, the data of the three modalities were projected into a common vector space with the same feature dimension. S3, multimodal data fusion stage based on attention mechanism: Construct a multimodal data fusion system based on attention mechanism, which includes a unimodal self-attention layer, a bimodal attention interaction layer, a trimodal joint attention layer and a fusion representation layer.

9. The multimodal fusion method for Alzheimer's disease according to claim 8, characterized in that: Specifically: First, the data of the three modalities after feature extraction are sent to the single-modal self-attention layer, and the self-attention mechanism is used to better interact with the input of the single-modal data to obtain three single-modal interaction features; The output of the unimodal self-attention layer is obtained by simply connecting the three unimodal interaction features; the formula of the self-attention mechanism is: Among them, SA n is the mode F n The new self-attention interaction features obtained after self-attention interaction, is the i-th linear projection with dimension d, is the jth linear projection with dimension d, where d is the dimension of the projection space; Then, the single modal data that has completed the unimodal self-attention interaction is sent to the bimodal attention interaction layer, and the multimodal data is fused pairwise using the cross-modal attention mechanism. The formula of the cross-modal attention mechanism is: Among them, CA n,m Indicates the modality SA that completes the unimodal self-attention interaction n Modal SA with unimodal self-attention interaction completed m The new cross-modal features generated by the cross-modal attention mechanism are fused, and d represents the dimension of the projection space; The six cross-modal features are concatenated in pairs according to the modality to obtain three cross-modal features, and then the three cross-modal features are connected to obtain the output of the bimodal attention interaction layer; Next, the three cross-modal features obtained in the bimodal attention interaction layer are combined in pairs, and six cross-modal features are obtained through cross-modal attention. These six features are concatenated to obtain the output of the trimodal joint attention layer. Finally, the outputs of the unimodal self-attention layer, the bimodal attention interaction layer, and the trimodal joint attention layer are sent to a fully connected layer to obtain the representation of multimodal data after fusion based on the attention mechanism.

10. A multimodal fusion method for Alzheimer's disease according to claim 9, characterized in that: The three unimodal interaction features are: IN c =SA(F c ) IN g =SA(F g ) IN i =SA(F i ) Among them, F c , F g , F i They are the data after feature extraction corresponding to clinical data, genetic data, and image data; The output of the unimodal self-attention layer is: The six cross-modal features are: THAT c,g =CA(SA c ,SADDLE g ) THAT c,i =CA(SA c ,SADDLE i ) THAT g,c =CA(SA g ,SADDLE c ) THAT g,i =CA(SA g ,SADDLE i ) THAT i,c =CA(SA i ,SADDLE c ) THAT i,g =CA(SA i ,SADDLE g ) After concatenating the six cross-modal features in pairs according to the modality, three cross-modal features are obtained, which are: Then the three cross-modal features are concatenated to obtain the output of the bimodal attention interaction layer: The three cross-modal features obtained in the bimodal attention interaction layer are combined in pairs, and the six cross-modal features obtained through cross-modal attention are: Concatenating these six features yields the output of the trimodal joint attention layer: Finally, the outputs of the unimodal self-attention layer, the bimodal attention interaction layer, and the trimodal joint attention layer are fed into a fully connected layer to obtain the representation of multimodal data after fusion based on the attention mechanism:

Citation Information

Cited By

  • Alzheimer disease auxiliary accurate identification method and system based on multi-modal marker network

    CN120340837A

  • An alzheimer's disease auxiliary precision identification method and system based on a multi-modal marker network

    CN120340837B

  • Health risk assessment method based on AI multiple modes

    CN120613137A

  • Image recognition method and related device for mycobacterium tuberculosis and nontuberculous mycobacterium

    CN122313075A

  • Cross-modal fusion attention mechanism Alzheimer disease early risk assessment method and system based on PET image, MR image and cognitive scoring data

    CN122599024A