Olzheimer's multi-modal classification method based on MSTNet neural network

Abstract

The invention discloses an MSTNet neural network-based Alzheimer's multi-modal classification method, which comprises the following steps: step 1, acquiring an Alzheimer's ADMC multi-dimensional data set which comprises electroencephalogram, nuclear magnetic resonance imaging and scale data of a unified data source, and dividing the multi-dimensional data set into a training set and a test set; step 2, constructing an MSTNet multi-modal neural network model, wherein the MSTNet multi-modal neural network model comprises a time feature encoder, a cross-modal fusion attention module, a feature marker and a table coding architecture; 3, training the MSTNet multi-modal neural network model by using the training set, and adjusting and optimizing the parameters of the MSTNet multi-modal neural network model in combination with the characteristics of multi-modal data; and step 4, evaluating the trained MSTNet multi-modal neural network model by using a test set, and finally realizing an Olzheimer's multi-modal classification function. The method can realize an automatic and intelligent Olzheimer's multi-mode classification function, and has relatively high classification accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision deep learning and medical image processing, and in particular to an Ozheimer's multimodal classification method based on an MSTNet neural network. Background Art

[0002] At present, the global population aging situation is becoming increasingly serious, and Alzheimer's disease (AD) is a common disease among the elderly. The number of AD patients will continue to rise, posing a severe challenge to the economy, medical system, patients and families. There are many difficulties in the clinical diagnosis of AD, such as unclear early symptoms, diverse symptoms, long detection time, inconsistent diagnostic standards and high dependence on the level of doctors. Mild cognitive impairment (MCI) is an important window period before dementia. From this time, the brain has begun to undergo small and inconspicuous changes, but it is often ignored or misdiagnosed by doctors because of the unclear symptoms. The symptoms of AD patients vary from person to person. The degree and type of cognitive decline, behavioral changes and emotional problems of different individuals may be different, and they are easily interfered by other diseases, thus affecting judgment and diagnosis. Imaging examinations usually take a long time to collect and analyze data, which is not suitable for rapid screening and diagnosis. Different medical institutions may use different standards and methods, and the diagnosis of AD is highly related to the level and experience of doctors, which leads to inconsistent diagnostic results.

[0003] In recent years, deep neural network technologies for AD classification have emerged continuously, and the diagnosis and treatment of AD have gained a strong impetus, which can effectively solve the problems caused by traditional manual diagnosis. These studies can be divided into unimodal classification tasks and multimodal classification tasks according to modality. Unimodal diagnosis usually relies on one type of data (such as imaging, cognitive assessment scales, etc.), which limits the understanding of the overall picture of the disease. Alzheimer's disease is a complex multifactorial disease. A single data source cannot fully capture all relevant pathological characteristics and is easily affected by noise and variation, resulting in a series of problems such as difficulty in diagnosis and high misdiagnosis rate. Compared with unimodality, multimodality can combine different types of data such as clinical information, scale data, neuroimaging, and EEG signals to form complementary advantages and improve the accuracy and robustness of disease prediction models. For example, cognitive scales can comprehensively, accurately and quickly reflect the intellectual status and degree of cognitive impairment of the subjects. In terms of neuroimaging, magnetic resonance imaging (MRI) can provide detailed images of brain structure, helping doctors observe changes in brain volume and cortical atrophy. Electroencephalogram (EEG), as an objective reflection of brain neural activity, is highly objective and difficult to disguise, and contains rich semantic information such as physiological and pathological information.

[0004] Recently, deep neural network technology has overcome the limitations of traditional manual diagnostic methods and greatly promoted the development of attention deficit disorder classification. Research in this field can be roughly divided into two categories: unimodal and multimodal classification tasks. Unimodal diagnosis often fails to capture the full complexity of attention deficit disorder, resulting in diagnostic difficulties and high misdiagnosis rates. In contrast, multimodal methods integrate multiple types of data, including clinical information, cognitive scales, neuroimaging, and electroencephalogram signals, to provide a more comprehensive understanding of the disease. For example, cognitive scales can quickly and accurately assess intellectual status and cognitive deficits. Neuroimaging, especially magnetic resonance imaging (MRI), can provide detailed images of brain structure, thereby observing changes in brain volume and cortical atrophy. Electroencephalogram (EEG) objectively reflects brain neural activity and contains rich physiological and pathological information. The integration of these multimodal data types improves the accuracy and robustness of the model, and has significant advantages over unimodal methods. However, the current multimodal classification method for Ozheimer's disease still has defects such as insufficient accuracy and low efficiency. Therefore, the present invention constructs an Ozheimer's disease multimodal classification method based on MSTNet neural network to address the existing defects. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides an Ozheimer's multimodal classification method based on an MSTNet neural network. The MSTNet neural network model consists of a temporal feature encoder, a cross-modal fusion attention module, a feature marker and a table encoding architecture, which can realize automatic and intelligent Ozheimer's multimodal classification function with high classification accuracy and efficiency.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] An Ozheimer's multimodal classification method based on MSTNet neural network includes the following steps:

[0008] Step 1, obtaining an Ozheimer's ADMC multidimensional data set, wherein the multidimensional data set includes electroencephalogram, magnetic resonance imaging and scale data from a unified data source, and dividing the multidimensional data set into a training set and a test set;

[0009] Step 2: construct an MSTNet multimodal neural network model, wherein the MSTNet multimodal neural network model includes a temporal feature encoder, a cross-modal fusion attention module, a feature marker, and a table encoding architecture;

[0010] The temporal feature encoder includes three consecutive TimesBlocks, and the cross-modal fusion attention module includes a 3×3×3 convolution, a BN+Relu layer, a Residual Dense Connection, a spatial average pooling layer, a cross-modal cross attention and an aggregation module;

[0011] Step 3: Use the training set to train the MSTNet multimodal neural network model, and optimize the parameters of the MSTNet multimodal neural network model based on the characteristics of multimodal data;

[0012] Step 4: Use the test set to evaluate the trained MSTNet multimodal neural network model and finally realize the Ozheimer multimodal classification function.

[0013] Preferably, the multidimensional data set is divided into a training set and a test set in a ratio of 8:2.

[0014] Preferably, the TimesBlock includes an LN layer and a residual connection.

[0015] Preferably, the implementation method of the time feature encoder is:

[0016] First, for a one-dimensional time series with a time length of T and a channel dimension of C The periodicity of the EEG signal is obtained by calculating the fast Fourier transform of the time dimension;

[0017] Based on the selected period, the original one-dimensional time series X 1D Fold and get a set of two-dimensional tensors Corresponding to the period p i dominant 2D temporal changes;

[0018] The 2D tensor is used as input to be converted into 2D features through a 2D convolution kernel, and the internal periodic changes and periodic changes of the EEG signal are captured;

[0019] The 2D features are transformed back to 1D features and weighted aggregated according to the amplitude value of each cycle. The formula is as follows:

[0020]

[0021]

[0022] in, It is the amplitude value of the corresponding frequency obtained by normalizing the EEG data through Softmax;

[0023] is the feature shape reconstruction operation; is a 2D feature; l is the column of the 2D feature, i.e., the periodic change within the EEG signal, and i is the row of the 2D feature, i.e., the periodic change of the EEG signal;

[0024] The weighted aggregated 1D features are residually connected, and finally the features of multiple periods are fused through adaptive aggregation. The fused features use amplitude values ​​to reflect the importance of each period feature, and weighted summation is performed to obtain the final 1D feature representation.

[0025] In the above technical solution, the TimesBlock module of the temporal feature encoder of the MSTNet multimodal neural network model performs a 2D transformation on the multi-periodic features of the time series data, decomposing the complex time change pattern into multiple intra-cycle changes and inter-cycle changes, thereby effectively modeling the complex time series patterns in the EEG data, eliminating noise and improving diagnostic accuracy.

[0026] Preferably, in the cross-modal fusion attention module, the 3D MRI image is first subjected to a 3×3×3 convolution, a BN layer, and a Relu activation function to preliminarily extract features, and then the key information in the feature map is captured through multiple Residual Dense Connection and spatial average pooling operations, and the feature map representation is reduced.

[0027] The 1D features extracted by the temporal feature encoder are transformed into 3D features through scale change.

[0028] The processed MRI and EEG features are subjected to cross-modal cross-attention operations. Through interactive learning between input vectors, different weights of different objects are calculated, so that more focused area information can be found. The calculation formula is as follows:

[0029] Q=XW Q

[0030] K=XW K

[0031] V=XW V

[0032]

[0033] Attention(Q,K,V)=Cos(Q,K)×V

[0034] Where W Q , W K , W V ∈R d×d , Q represents the query information, K represents the vector used to match Q, also known as the index, and V represents the content vector. In addition, Q is the input EEG feature vector sequence X EEG ∈R N×d It is obtained by linear mapping, while K and V are obtained by inputting MRI feature vector sequence X MRI ∈R N×d Obtained through linear mapping.

[0035] The processed MRI and EEG 3D features are concat-fused to obtain simple features, and the simple features and high-level features are sent to the aggregation module together to obtain a comprehensive high-dimensional aggregated feature. The high-level features and low-level features are combined to form a comprehensive feature representation, making the final feature representation richer and more comprehensive. In the aggregation module, the comprehensive features will further capture the deep relationship between features through the residual dense connection module to improve the effect of feature fusion. Through the superposition of multiple convolutional layers, more complex features are gradually extracted, making the final feature representation more refined and effective. In this way, we can finally obtain a high-dimensional feature vector that contains all the key information extracted from different modal data, thereby providing more reliable feature support for subsequent classification tasks.

[0036] In the above technical solution, the cross-modal fusion attention module effectively fuses MRI and EEG features to capture the complex correlation between the two modal data. By focusing on the key features in different modal data, cross-modal cross-attention can better understand and utilize these features, thereby achieving better performance in complex multimodal tasks. In addition, the aggregation module hierarchically aggregates the simple features after the concat of the two modalities with the high-level features after the cross-modal cross-attention processing, and after the residual dense connection module fully captures the complex relationship between the data, the model has stronger robustness and accuracy, maximizing the advantages of each modal data.

[0037] Preferably, the feature tagger converts the input scale data x into an embedding vector Given feature x j The embedding of is calculated as follows:

[0038]

[0039] where b j is the deviation of the jth feature; for numeric features, is through the vector This is achieved by multiplying the elements of ; for categorical features, As a lookup table accomplish.

[0040] In the above technical solution, the main function of the feature marker is to convert numerical features (such as patient age and MMSE score) and categorical features (such as gender and education level) into embedding vectors.

[0041] Preferably, in the table encoding architecture, for the embedding vector output by the feature marker, a multi-head self-attention mechanism and a feedforward network are used to capture the complex relationship between features; finally, the final output is obtained through the Transformer layer; in the output of the Transformer layer, the [CLS] tag is used to represent the information summarizing the entire input sequence, and the [CLS] tag will be passed to the prediction layer, and the 3D features of MRI and EEG and the high-dimensional aggregated features will be concat-operated and then input into the linear layer for classification prediction.

[0042] In the above technical solution, the feature marker and the encoding architecture for tabular domain data are combined to enable the model to effectively extract information and learn features from tabular data. This integration not only enhances the expressive power of specific modalities, but also ensures robust and accurate predictions on diverse data sets, enabling the model to fully utilize the structural characteristics of tabular modal data.

[0043] Preferably, in step 3, the training method of the MSTNet multimodal neural network model is: the MSTNet multimodal neural network model is implemented using the PyTorch framework, and trained on an NVIDIA Tesla V100 GPU. During the training process, the dropout rate is set to 0.1, the batch size is set to 100, the learning rate is set to 0.0001, and the training process lasts for 24 hours.

[0044] Preferably, in step 4, five standard indicators are used to evaluate the classification performance, and the five indicators are accuracy, which is the overall proportion of correct predictions; precision, which is the proportion of true positives in positive predictions; accuracy, which is the proportion of positive predictions and positive instances in positive predictions; recall, which is the proportion of correctly identified positive instances; F1-score, which is the harmonic mean of precision and recall; and MCC is a balanced indicator for measuring the quality of binary classification.

[0045] The present invention has the following characteristics and beneficial effects:

[0046] Using the above technical solutions, an advanced multimodal classification model designed for the diagnosis of attention deficit disorder is introduced. The model overcomes the limitations of traditional unimodal methods by effectively integrating multiple data types. The TimesBlock module is used to process EEG data and is able to capture complex temporal patterns. In addition, feature labeling and table encoding architectures are used to enhance feature extraction of tabular data. Cross-modal attention aggregation is used to integrate spatial MRI data and temporal EEG information. The model of the present invention shows excellent performance in classifying AD, MCI, and NC, which is confirmed in experiments conducted on a private ADMC dataset. By taking advantage of multiple data modalities, the method of the present invention represents a major advancement in the field of neurodegenerative disease classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 This is the overall structure diagram of the MSTNet multimodal neural network model of the present invention.

[0049] Figure 2 Schematic diagram of the feature marker in the present invention converting digital classification features into embedding vectors.

[0050] Figure 3It is a schematic diagram of two-dimensional conversion of multi-period features of time series data by the TimesBlock module of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0053] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0054] The present invention provides an Ozheimer multimodal classification method based on MSTNet neural network, such as Figure 1 As shown, the following steps are included:

[0055] Firstly, an Ozheimer's ADMC multidimensional dataset is obtained, wherein the multidimensional dataset includes electroencephalogram, magnetic resonance imaging and scale data from a unified data source, and the multidimensional dataset is divided into a training set and a test set.

[0056] In this embodiment, the Ozheimer's ADMC multidimensional data set was collected from 100 subjects, including EEG, MRI and scale data of each subject. The scale data refers to the demographic details of the subjects, as follows: average age 72.4 years old, age range 56 to 93 years old, 56 women, 22 married. The data set is divided into 80 training samples and 20 test samples.

[0057] Then, an MSTNet multimodal neural network model is constructed, which includes a temporal feature encoder, a cross-modal fusion attention module, a feature marker and a table encoding architecture.

[0058] Specifically, the temporal feature encoder includes three consecutive TimesBlocks, and TimesBlock is composed of an LN layer and a residual connection.

[0059] The cross-modal fusion attention module consists of 3×3×3 convolution, BN layer and Relu activation function to initially extract features, and then captures the key information in the feature map through multiple Residual Dense Connection and spatial average pooling operations. The high-level features obtained through the cross-modal cross-attention operation and the simple features obtained by concat fusion of the two modal features are input into the aggregation module together.

[0060] The feature marker and table encoding architecture consists of a feature marker and a table encoding architecture. The main function of the feature marker is to convert numerical features (such as patient age and MMSE score) and categorical features (such as gender and education level) into embedding vectors. The table encoding architecture is responsible for processing the embedding vector output by the feature marker, capturing the complex relationship between features through a multi-head self-attention mechanism and a feedforward network.

[0061] Furthermore, the training set is used to train the MSTNet multimodal neural network model, and the parameters of the MSTNet multimodal neural network model are tuned based on the characteristics of multimodal data.

[0062] Specifically, the MSTNet multimodal neural network model was implemented using the PyTorch framework and trained on an NVIDIA Tesla V100 GPU. During the training process, the dropout rate was set to 0.1, the batch size was set to 100, and the learning rate was set to 0.0001. The training process lasted 24 hours.

[0063] The test set is used to evaluate the trained MSTNet multimodal neural network model.

[0064] Specifically, in order to evaluate the model's classification performance on images, MSTNet is compared with other classification models on the ADMC-derived dataset. In this embodiment, five standard indicators are used to evaluate the classification performance. We use five standard indicators to evaluate the classification performance. They are: Accuracy, the overall proportion of correct predictions; Precision, the proportion of true positives in positive predictions; Precision, the proportion of positive predictions and positive instances in positive predictions; Recall, the proportion of correctly identified positive instances; F1-score, the harmonic mean of precision and recall. MCC is a balanced indicator for measuring the quality of binary classification. Their formulas are as follows

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Among them, TP and TN represent the real samples in the data predicted to be true or false, and FP and FN represent the false samples in the data predicted to be true or false.

[0071] Finally, the MSTNet multimodal neural network model that meets the evaluation criteria is used to realize the Ozheimer's multimodal classification function.

[0072] Specifically, the method of implementing Ozheimer's multimodal classification using the MSTNet multimodal neural network model is as follows:

[0073] The TimesBlock module of the time feature encoder performs a 2D transformation on the multi-periodic features of the time series data, decomposing the complex time change pattern into multiple intra-cycle changes and inter-cycle changes, thereby effectively modeling the complex time series pattern in the EEG data, removing noise, and improving the diagnostic accuracy. The specific steps are as follows: First, for a one-dimensional time series with a time length of T and a channel dimension of C The periodicity of the EEG signal is calculated by the fast Fourier transform of the time dimension, namely:

[0074] A=Avg(Amp(FFT(X 1D )))

[0075] f1,…,f k =arg Topk(A)

[0076]

[0077]

[0078] in, Represents X 1D The intensity of each resolved frequency component in the k frequencies with the largest intensity {f1, ..., f k} corresponds to the most significant k period lengths {p1,…,p k}. The above process is briefly recorded as:

[0079] A, {f1,…,f k}, {p1,…,p k}=Period(X 1D )

[0080] The above steps can extract the key periodic features in the EEG signal and effectively filter out noise and irrelevant information.

[0081] Next, if Figure 3 As shown, based on the selected period, the original one-dimensional time series X 1D Folding is performed, and the process can be formalized as:

[0082]

[0083] Among them, Padding (·) is to expand the time series by 0 along the time dimension to make it compatible Specifically, it means adding 0 to the end of the sequence so that the sequence length can be p i Through the above operations, we get a set of two-dimensional tensors Corresponding to the period p i Dominant two-dimensional temporal changes.

[0084] It should be noted that for the above two-dimensional vector, each column and each row corresponds to adjacent moments and adjacent cycles, respectively, and adjacent moments and cycles often contain similar timing changes. Therefore, the above two-dimensional tensor will show two-dimensional locality (2D locality), so that information can be easily captured through 2D convolution. Therefore, in this embodiment, a multi-scale 2D convolution kernel is used to process the transformed 2D features and capture the internal periodic changes (columns) and inter-cycle changes (rows) of the EEG signal. The formula is as follows:

[0085]

[0086] The nception block contains multiple 2D convolution kernels of different scales, which can process multiple 2D transformed tensors at the same time. This multi-scale convolution can better capture different feature patterns in time series data, thereby providing the model with the ability to understand and process EEG signals and improving the effect of feature extraction.

[0087] Afterwards, the processed 2D features are transformed back to 1D features and weighted aggregated according to the amplitude value of each cycle. The formula is as follows:

[0088]

[0089]

[0090] in, It is the amplitude value of the corresponding frequency obtained by Softmax normalization of EEG data; is the feature shape reconstruction operation; is a 2D feature; l is the column of the 2D feature, i.e., the periodic change within the EEG signal, and i is the row of the 2D feature, i.e., the periodic change of the EEG signal;

[0091] The weighted aggregation step ensures that the importance of each cycle feature is reflected in the final feature representation. In addition, in order to ensure the stability and effectiveness of training, the robustness of the model is provided by the residual connection operation:

[0092] X l，1D =TimesBlock(X l-1，1D )+X l-1，1D

[0093] Residual connection technology can help alleviate the gradient vanishing and gradient exploding problems, making the training of deep networks more stable. Finally, the features of multiple cycles are fused through adaptive aggregation. The amplitude value is used to reflect the importance of each cycle feature, and the weighted sum is performed to obtain the final 1D feature representation:

[0094]

[0095] This adaptive aggregation method enables the model to dynamically adjust the weight of each period feature in the final representation, thereby improving the effect of feature fusion.

[0096] In the second step, the cross-modal attention module in the MSTNet multimodal neural network model effectively fuses MRI and EEG features to capture the complex correlation between the two modal data. The cross-modal cross-attention mechanism focuses on the key features in different modal data, which can better understand and utilize these features, thereby achieving better performance in complex multimodal tasks. In addition, the aggregation mechanism hierarchically aggregates the simple features after the concat of the two modalities with the high-level features after the cross-modal cross-attention processing, and after the residual dense connection module fully captures the complex relationship between the data, the model has stronger robustness and accuracy, maximizing the advantages of each modality data.

[0097] Specifically, Figure 2 As shown, the 3D MRI image is first subjected to a 3×3×3 convolution, a BN layer, and a Relu activation function to preliminarily extract features. Then, the key information in the feature map is captured through multiple Residual Dense Connection and spatial average pooling operations, and the feature map representation is reduced. Detailed information on the brain structure is thereby obtained. In this embodiment, the detailed information on the brain structure refers to cortical atrophy and volume changes. At the same time, the 1D features extracted by the temporal feature encoder are converted into 3D features through scale changes to further increase the feature representation of temporal information. The processed MRI and EEG features are subjected to cross-modal cross-attention operations. Through interactive learning between input vectors, different weights of different objects are calculated, so that more concerned regional information can be found. The calculation formula is as follows:

[0098] Q=XW Q

[0099] K=XW K

[0100] v=XW v

[0101]

[0102] Attention(Q,K,V)=Cos(Q,K)×V

[0103] Where W Q , W K , W V ∈R d×d , Q represents the query information, K represents the vector used to match Q, also known as the index, and V represents the content vector. In addition, Q is the input EEG feature vector sequence X EEG ∈R N×d It is obtained by linear mapping, while K and V are obtained by inputting MRI feature vector sequence X MRI ∈R N×d Obtained through linear mapping.

[0104] Subsequently, the processed MRI and EEG 3D features are concat-fused to obtain simple features, and the simple features and high-level features are sent to the aggregation module together to obtain a comprehensive high-dimensional aggregated feature. The high-level features and low-level features are combined to form a comprehensive feature representation, making the final feature representation richer and more comprehensive. In the aggregation module, the comprehensive features will further capture the deep relationship between features through the residual dense connection module to improve the effect of feature fusion. Through the superposition of multiple convolutional layers, more complex features are gradually extracted, making the final feature representation more refined and effective. In this way, a high-dimensional feature vector can be obtained, which contains all the key information extracted from different modal data, thereby providing more reliable feature support for subsequent classification tasks.

[0105] In the further configuration of this embodiment, the MSTNet multimodal neural network model combines feature markers and an encoding architecture for tabular domain data, so that the model can effectively extract information and learn features from tabular data. This integration not only enhances the expressive power of a specific modality, but also ensures robust and accurate predictions on a diverse data set, enabling the model to fully utilize the structural characteristics of tabular modal data.

[0106] As you can understand, the main function of the feature tagger is to convert numerical features (such as patient age and MMSE score) and categorical features (such as gender and education level) into embedding vectors. These embedding vectors are then input into an encoding architecture designed for tabular data for feature extraction.

[0107] Specifically, the feature marker converts the input scale data x into an embedding vector Given feature x j The embedding of is calculated as follows:

[0108]

[0109] where b j is the deviation of the jth feature; for numeric features, is through the vector This is achieved by multiplying the elements of ; for categorical features, As a lookup table The specific formula is as follows:

[0110]

[0111]

[0112]

[0113] in, is the one-hot vector of categorical features.

[0114] Furthermore, in the tabular encoding architecture, the embedding vector output by the feature marker is used to capture the complex relationship between features through a multi-head self-attention mechanism and a feedforward network; the final output is obtained through the Transformer layer; in the output of the Transformer layer, the [CLS] tag is used to represent the information summarizing the entire input sequence, and the [CLS] tag will be passed to the prediction layer, where the 3D features of MRI and EEG are concat-operated with the high-dimensional aggregate features, and then input into the linear layer for classification prediction.

[0115] Finally, the MSTNet model proposed in this embodiment is further compared with several advanced single-modal and multi-modal models.

[0116] Compared with the unimodal model, the MSTNet model equipped with the CMAA module shows superior performance in all evaluation indicators. For example, MSTNet effectively utilizes the spatial information of MRI and the temporal information of EEG, which is superior to TableTransformer in this respect. In addition, MSTNet also solves the difficulties that existing multimodal classification models often face in multimodal feature extraction and fusion. Compared with the best unimodal model, MSTNet's F1-score increased by 0.79%; compared with the best multimodal model, MSTNet's F1-score increased by 4.17%.

[0117] This example conducts a series of elimination experiments to evaluate the impact of the DenseBlock module, TimesBlock module, CMAA module, and feature deviation on the performance of MSTNet. The experiments show that removing each module will lead to varying degrees of performance degradation.

[0118] Specifically, the absence of the DenseBlock module severely affects the model's ability to process and extract complex visual features in the image, resulting in a 15% drop in accuracy. Similarly, removing the TimesBlock module significantly reduces the model's ability to capture temporal features from the EEG signal, resulting in a 10% drop in accuracy. Excluding the CMAA module results in a decrease in multimodal integration capabilities, resulting in a 15% drop in accuracy. In addition, without feature bias, the stability of the model is also affected, resulting in a 5% drop in accuracy. These findings highlight the important role of each module in maintaining the overall effectiveness of the advanced MSTNet model proposed in this example.

[0119] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. An Ozheimer multimodal classification method based on MSTNet neural network, It is characterized in that The steps include: Step 1, obtaining an Ozheimer's ADMC multidimensional data set, wherein the multidimensional data set includes electroencephalogram, magnetic resonance imaging and scale data from a unified data source, and dividing the multidimensional data set into a training set and a test set; Step 2: construct an MSTNet multimodal neural network model, wherein the MSTNet multimodal neural network model includes a temporal feature encoder, a cross-modal fusion attention module, a feature marker, and a table encoding architecture; The temporal feature encoder includes three consecutive TimesBlocks, and the cross-modal fusion attention module includes a 3×3×3 convolution, a BN+Relu layer, a Residual Dense Connection, a spatial average pooling layer, a cross-modal cross attention and an aggregation module; Step 3: Use the training set to train the MSTNet multimodal neural network model, and optimize the parameters of the MSTNet multimodal neural network model based on the characteristics of multimodal data; Step 4: Use the test set to evaluate the trained MSTNet multimodal neural network model and finally realize the Ozheimer multimodal classification function.

2. According to claim 1, an Ozheimer's multimodal classification method based on MSTNet neural network, It is characterized in that The multidimensional dataset is divided into a training set and a test set in a ratio of 8:

2.

3. According to claim 1, an Ozheimer's multimodal classification method based on MSTNet neural network, It is characterized in that The TimesBlock includes an LN layer and a residual connection.

4. The Ozheimer's multimodal classification method based on MSTNet neural network according to claim 1, It is characterized in that The implementation method of the time feature encoder is: First, for a one-dimensional time series with a time length of T and a channel dimension of C The periodicity of the EEG signal is obtained by calculating the fast Fourier transform of the time dimension; Based on the selected period, the original one-dimensional time series X 1D Fold and get a set of two-dimensional tensors Corresponding to the period p i dominant 2D temporal changes; The 2D tensor is used as input to be converted into 2D features through a 2D convolution kernel, and the internal periodic changes and periodic changes of the EEG signal are captured; The 2D features are transformed back to 1D features and weighted aggregated according to the amplitude value of each cycle. The formula is as follows: in, It is the amplitude value of the corresponding frequency obtained by Softmax normalization of EEG data; is the feature shape reconstruction operation; is a 2D feature; l is the column of the 2D feature, i.e., the periodic change within the EEG signal, and i is the row of the 2D feature, i.e., the periodic change of the EEG signal; The weighted aggregated 1D features are residually connected, and finally the features of multiple periods are fused through adaptive aggregation. The fused features use amplitude values ​​to reflect the importance of each period feature, and weighted summation is performed to obtain the final 1D feature representation.

5. According to claim 4, an Ozheimer multimodal classification method based on MSTNet neural network, It is characterized in that In the cross-modal fusion attention module, the 3D MRI image is first subjected to a 3×3×3 convolution, a BN layer, and a Relu activation function to preliminarily extract features. Then, the key information in the feature map is captured through multiple ResidualDense Connection and spatial average pooling operations, and the feature map representation is reduced.

6. The Ozheimer multimodal classification method based on MSTNet neural network according to claim 5, It is characterized in that In the cross-modal fusion attention module, the 1D features extracted by the temporal feature encoder are transformed into 3D features by scaling.

7. The Ozheimer multimodal classification method based on MSTNet neural network according to claim 6, It is characterized in that In the cross-modal fusion attention module, the processed 3D features of MRI and EEG are subjected to cross-modal cross-attention operations. By interactive learning between input vectors, different weights of different objects are calculated, so that high-level features of more concerned regional information can be found. The calculation formula is as follows: Q=XW Q K=XW K V=XW v Attention(Q,K,V)=Cos(Q,K)×V Where W Q , W K , W V ∈R d×d , Q represents the query information, K represents the vector used to match Q, also known as the index, and V represents the content vector. In addition, Q is the input EEG feature vector sequence X EEG ∈R N × d It is obtained by linear mapping, while K and V are obtained by inputting MRI feature vector sequence X MRI ∈R N × d Obtained through linear mapping.

8. According to claim 7, an Ozheimer's multimodal classification method based on MSTNet neural network, It is characterized in that In the cross-modal fusion attention module, the processed 3D features of MRI and EEG are concat-fused to obtain simple features, and the simple features and high-level features are sent to the aggregation module together to obtain a comprehensive high-dimensional aggregation feature.

9. The Ozheimer multimodal classification method based on MSTNet neural network according to claim 8, It is characterized in that The feature tagger converts the input scale data x into an embedding vector Given feature x j The embedding of is calculated as follows: where b j is the deviation of the jth feature; for numeric features, is through the vector This is achieved by multiplying the elements of ; for categorical features, As a lookup table accomplish.

10. The Ozheimer multimodal classification method based on MSTNet neural network according to claim 9, It is characterized in that In the tabular encoding architecture, for the embedding vector output by the feature marker, the complex relationship between features is captured through a multi-head self-attention mechanism and a feedforward network; finally, the final output is obtained through the Transformer layer; in the output of the Transformer layer, the [CLS] tag is used to represent the information summarizing the entire input sequence, and the [CLS] tag will be passed to the prediction layer, and the 3D features of MRI and EEG and the high-dimensional aggregated features will be concat-operated and then input into the linear layer for classification prediction.

11. The Ozheimer multimodal classification method based on MSTNet neural network according to claim 1, It is characterized in that In the step 3, the training method of the MSTNet multimodal neural network model is as follows: the MSTNet multimodal neural network model is implemented using the PyTorch framework and trained on an NVIDIA Tesla V100 GPU. During the training process, the dropout rate is set to 0.1, the batch size is set to 100, the learning rate is set to 0.0001, and the training process lasts for 24 hours.

12. The Ozheimer's multimodal classification method based on MSTNet neural network according to claim 1, It is characterized in that In step 4, five standard indicators are used to evaluate the classification performance, which are accuracy, the overall proportion of correct predictions; precision, the proportion of true positives in positive predictions; accuracy, the proportion of positive predictions and positive instances in positive predictions; recall, the proportion of correctly identified positive instances; F1-score, the harmonic mean of precision and recall; MCC is a balanced indicator for measuring the quality of binary classification.

Images