A Prognostic Classification Method for Anti-NMDAR Encephalitis Based on Multimodal Feature Fusion

Abstract

This invention discloses a prognostic classification method for anti-NMDAR encephalitis based on multimodal feature fusion, mainly relating to the field of medical image processing technology. The method includes the following steps: S1, acquiring the corresponding patient MRI images and clinical features; S2, constructing an MRI feature extractor to extract the patient's image features; S3, constructing a multimodal feature fusion model to fuse image features and clinical features across modalities, generating richer fused features; S4, constructing an anti-NMDAR encephalitis prognostic classifier based on the fused features and training it using a multimodal dataset; S5, inputting the MRI images and clinical features of the anti-NMDAR encephalitis patient to be classified into the trained classifier to obtain the prognostic classification result. This invention significantly reduces the reliance on domain experts for the prognostic diagnosis of anti-NMDAR patients, providing doctors with objective, accurate, consistent, and interpretable diagnostic references.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a prognosis classification method for anti-NMDAR encephalitis based on multi-modal feature fusion. BACKGROUND

[0002] Anti-NMDAR encephalitis is a common type of AE, and its pathological mechanism has been gradually elucidated, but the best treatment plan has not been determined. Accurate early prognosis evaluation has a very significant significance for reversing the symptoms of anti-NMDAR encephalitis. Compared with the expensive manual evaluation cost, the evaluation method based on the machine learning diagnosis model has obvious advantages. At present, there are various anti-NMDAR encephalitis prognosis models. One is a diagnosis model based on deep learning, which extracts magnetic resonance image (MRI) features through a convolutional neural network, and then classifies through a fully connected layer or a global pooling layer to realize the diagnosis and classification of the disease; one is a prediction model based on omics features, which extracts image features of the lesion through a machine learning algorithm, and realizes the classification of the prognosis through a classifier; one is a statistical model based on clinical features, which uses single or multiple factor logistic regression, Lasso and other statistical algorithms to diagnose the supplementary information independent of medical images such as clinical features, and finally forms a diagnosis result. At present, the best prediction method is the multi-model cascade structure, which first establishes an MRI deep learning diagnosis model, an omics feature prediction model and a clinical feature statistical model, and finally fuses the prediction results of the above models to form the final encephalitis prognosis prediction.

[0003] However, the existing anti-NMDAR encephalitis prognosis model has low prediction accuracy, low calculation efficiency, poor interpretability and other shortcomings. The diagnosis model based on deep learning, omics features and clinical features has low prediction accuracy and poor interpretability due to the neglect of other information. The multi-model cascade structure fuses the prediction results of the above several different diagnosis models, realizes the prediction of the prognosis based on the full integration of multi-modal information, has high precision, but due to its complex structure, the model is cascaded by multiple independent models, resulting in large model parameters and large calculation amount, which is not real-time and loses the prerequisite as a diagnosis model. In addition, the medical interpretability of the cascade model is also poor. SUMMARY

[0004] The present application aims to solve the problems in the prior art and provide an anti-NMDAR encephalitis prognosis classification method based on multi-modal feature fusion, which avoids a large model structure, uses an end-to-end structure to predict the prognosis of patients with multi-modal features, has excellent prediction accuracy and generalization ability, greatly reduces the dependence of anti-NMDAR patient prognosis diagnosis on field experts, and can provide objective, accurate, consistent and interpretable diagnosis reference for doctors.

[0005] The application achieves the above-mentioned purpose through the following technical solutions.

[0006] A prognosis classification method for anti-NMDAR encephalitis based on multi-modal feature fusion, comprising the following steps:

[0007] S1. Based on cases diagnosed as anti-NMDAR encephalitis, MRI images and clinical features of the corresponding patients are obtained;

[0008] S2. An MRI feature extractor is constructed to extract image features of the patients;

[0009] S3. A multi-modal feature fusion model is constructed to cross-modally fuse the image features and the clinical features, generate fusion features with more abundant information, and improve the fusion performance through a channel information guiding module;

[0010] S4. An anti-NMDAR encephalitis prognosis classifier is constructed based on the fusion features, and is trained through a multi-modal data set;

[0011] S5. The MRI images and the clinical features of an anti-NMDAR encephalitis patient to be classified are input into the trained classifier to obtain a prognosis classification result.

[0012] Preferably, in step S2, the method for extracting image features from the MRI of a patient comprises the following steps:

[0013] S21. Projecting a three-dimensional MRI into a two-dimensional space to obtain a two-dimensional image sequence as an image input;

[0014] S22. Unifying image sequences of different sizes into 512x512 through a bilinear interpolation method;

[0015] S23. Constructing a convolutional neural network for two-dimensional image sequence feature extraction, which is based on a residual structure and enhances feature extraction performance through a channel information guiding module;

[0016] S24. Inputting the two-dimensional image sequence into the convolutional neural network for MRI feature extraction.

[0017] Preferably, in steps S3 and S23, the channel information guiding module comprises the following steps:

[0018] S311. Extracting two independent context information of the input features through global average pooling and global maximum pooling;

[0019] S312. Processing the context information using a multi-layer perceptron and adding them element by element;

[0020] S313. Obtaining attention weights through a Sigmoid function;

[0021] S314, attention weights are assigned to the input features processed by the multi-layer perceptron to obtain output features.

[0022] Preferably, in step S3, the constructed multi-modal feature fusion model comprises the steps of:

[0023] S321, the MRI image features extracted by the convolutional neural network are reduced in channel number by 1x1 convolution;

[0024] S322, the multi-sequence MRI image features are assigned weights by a dynamic normalization weight layer and are spliced in the channel dimension;

[0025] S323, the spliced multi-sequence MRI fusion features are assigned attention weights by a channel information guiding module and are restored in channel number by 1x1 convolution;

[0026] S324, the clinical features are divided into continuous variables and categorical variables, and semantic information is extracted respectively, and finally all semantic information is spliced in the channel dimension to obtain complete clinical features;

[0027] S325, after the multi-sequence MRI features and the clinical features are adjusted in size by a multi-layer perceptron, the two are spliced by a dynamic normalization weight layer, and the spliced features are assigned attention weights by a channel information guiding module to obtain final fusion features.

[0028] Preferably, in step S324, the extraction method of the clinical features is specifically:

[0029] The clinical features are divided into continuous variables and categorical variables, wherein the continuous variables extract semantic information through a BatchNorm layer; the categorical variables first extract word vectors by a specific dimension Embadding layer, and then extract categorical variable semantic information by a Bi-LSTM layer; these semantic information is then spliced in the channel dimension to form the final clinical semantic features.

[0030] Preferably, in step S4, the constructed anti-NMDAR encephalitis prognosis classifier is: after passing through the multi-modal feature fusion model, the fusion features obtained from the image features and the clinical features of the patient are processed by two fully connected layers to obtain the final prognosis classification result.

[0031] Preferably, in step S5, in the trained anti-NMDAR encephalitis prognosis classification model, the performance evaluation parameters include:

[0032]

[0033] Where AUC refers to the area under the curve during drug administration, which is a pharmacokinetic parameter; Accuracy; Specificity; Sensitivity; M is the number of positive samples; N is the number of negative samples; TP refers to true positive samples; TN refers to true negative samples; FP refers to false positive samples; and FN refers to false negative samples.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] 1. This invention achieves high-precision prognosis for anti-NMDAR encephalitis through deep learning methods. Compared with existing manual prognosis methods, this method greatly reduces prediction time and reduces reliance on domain experts.

[0036] 2. This invention makes full use of the patient's MRI imaging features and clinical features, and performs prognostic classification through richer fusion features. Compared with the existing single-modal prognostic classification model, this method greatly improves the classification accuracy.

[0037] 3. This invention realizes end-to-end model design, achieves multimodal feature fusion for input images and clinical features, and automatically applies them to predict patient prognosis. Compared with existing multi-model cascade and ensemble learning methods, this method greatly reduces computational costs and reflects the timeliness of the prognostic classification model.

[0038] 4. This invention transforms visual and textual information such as medical images and clinical features into deep data features for quantitative research, providing doctors with accurate, objective, and reproducible diagnostic references. Furthermore, it can reproduce the diagnostic causes and regions of interest of the model through methods such as gradient-weighted activation maps, exhibiting extremely high medical interpretability. Attached Figure Description

[0039] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0040] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0041] Example: As attached Figure 1As shown, this invention describes a prognostic classification method for anti-NMDAR encephalitis based on multimodal feature fusion. It is a multimodal feature fusion network that utilizes a channel information guidance module and dynamically normalized weight layers to control the fusion of clinical variables and MRI features, and is used to predict the prognosis of patients with anti-NMDAR encephalitis. The network structure comprises two key substructures. First, a channel information guidance module is designed to constrain the heterogeneity between cross-modal features through an attention mechanism, achieving more effective feature fusion. Second, a backbone network for brain MRI feature extraction is designed. This network is based on a bi-branch residual structure and utilizes the channel information guidance module to preserve the semantic complementarity of multi-scale features. Unlike existing multimodal feature fusion methods, it avoids a large model structure and adopts an end-to-end structure to predict the prognosis of patients with multimodal features. It exhibits excellent prediction accuracy and generalization ability, significantly reducing the reliance on domain experts for the prognostic diagnosis of anti-NMDAR patients, and providing doctors with objective, accurate, consistent, and interpretable diagnostic references.

[0042] The specific steps include:

[0043] S1. Based on cases diagnosed with anti-NMDAR encephalitis, obtain the corresponding patient MRI images and clinical characteristics.

[0044] S2. Construct an MRI feature extractor to extract the patient's image features.

[0045] Preferably, the method for extracting image features from an MRI scan of a patient specifically includes the following steps:

[0046] S21. Project the three-dimensional MRI onto a two-dimensional space, and use the resulting two-dimensional image sequence as the image input;

[0047] S22. Unify image sequences of different sizes to 512×512 using bilinear interpolation.

[0048] S23. Construct a convolutional neural network for feature extraction of two-dimensional image sequences. This neural network is based on a residual structure and its feature extraction performance is enhanced by a channel information guidance module.

[0049] S24. Input the two-dimensional image sequence into the convolutional neural network to extract MRI features.

[0050] S3. Construct a multimodal feature fusion model to fuse imaging features and clinical features across modalities, generate more information-rich fused features, and improve fusion performance through a channel information guidance module.

[0051] Preferably, in steps S3 and S23, the channel information guidance module specifically includes the following steps:

[0052] S311. Extract two independent contextual information of the input features through global average pooling and global max pooling;

[0053] S312. Use a multilayer perceptron to process the context information and add the two elements one by one;

[0054] S313. Obtain attention weights through the Sigmoid function;

[0055] S314. Assign attention weights to the input features processed by the multilayer perceptron to obtain the output features.

[0056] Preferably, the constructed multimodal feature fusion model specifically includes the following steps:

[0057] S321. Reduce the number of channels of the MRI image features extracted by the convolutional neural network by 1×1 convolution;

[0058] S322. Multi-sequence MRI image features are weighted through a dynamic normalization weighting layer and spliced ​​in the channel dimension.

[0059] S323. The spliced ​​multi-sequence MRI fusion features are assigned attention weights through the channel information guidance module and the number of channels is restored through 1×1 convolution.

[0060] S324. Clinical features are divided into continuous variables and categorical variables, and semantic information is extracted from each. Finally, all semantic information is concatenated along the channel dimension to obtain complete clinical features.

[0061] After the S325, multi-sequence MRI features and clinical features are sized by a multilayer perceptron, they are spliced ​​together through a dynamically normalized weighted layer. The spliced ​​features are then given full attention by a channel information guidance module to obtain the final fused features.

[0062] Furthermore, in step S324, the specific method for extracting clinical features is as follows:

[0063] Clinical features are divided into continuous variables and categorical variables. Continuous variables have semantic information extracted through a BatchNorm layer. Categorical variables first have word vectors extracted by a specific dimension of the Embadding layer, and then semantic information of the categorical variables is extracted through a Bi-LSTM layer. These semantic information are then concatenated along the channel dimension to form the final clinical semantic features.

[0064] S4. Construct an anti-NMDAR encephalitis prognostic classifier based on fusion features and train it using a multimodal dataset;

[0065] Preferably, the constructed anti-NMDAR encephalitis prognostic classifier is as follows: after passing through a multimodal feature fusion model, the fused features obtained from the patient's imaging features and clinical features are passed through two fully connected layers to obtain the final prognostic classification result.

[0066] S5. Input the MRI images and clinical characteristics of the patients with anti-NMDAR encephalitis to be classified into the trained classifier to obtain the prognostic classification results.

[0067] In the preferred, well-trained anti-NMDAR encephalitis prognostic classification model, the performance evaluation parameters include:

[0068]

[0069] Where AUC refers to the area under the curve during drug administration, which is a pharmacokinetic parameter; Accuracy; Specificity; Sensitivity; M is the number of positive samples; N is the number of negative samples; TP refers to true positive samples; TN refers to true negative samples; FP refers to false positive samples; and FN refers to false negative samples.

Claims

1. A prognosis classification method for anti-NMDAR encephalitis based on multi-modal feature fusion, The method is characterized by comprising the steps of: S1, based on the confirmed anti-NMDAR encephalitis cases, obtaining the corresponding patient MRI images and clinical features; S2, constructing an MRI feature extractor to extract the image features of the patients; S3, constructing a multi-modal feature fusion model to cross-modally fuse the image features and the clinical features, generate more information-rich fusion features, and improve the fusion performance through a channel information guiding module; The constructed multi-modal feature fusion model comprises the steps of: S321, reducing the channel number of the MRI image features extracted by the convolutional neural network through 1x1 convolution; S322, assigning weights to the multi-sequence MRI image features through a dynamic normalization weight layer, and splicing in the channel dimension; S323, assigning attention weights to the spliced multi-sequence MRI fusion features through the channel information guiding module, and restoring the channel number through 1x1 convolution; S324, dividing the clinical features into continuous variables and classification variables, and extracting semantic information respectively, and finally splicing all semantic information in the channel dimension to obtain complete clinical features; The extraction method of the clinical features is: the clinical features are divided into continuous variables and classification variables, wherein the continuous variables extract semantic information through a BatchNorm layer; the classification variables first extract word vectors through a specific dimension Embadding layer, and then extract classification variable semantic information through a Bi-LSTM layer; these semantic information is then spliced in the channel dimension to form the final clinical semantic features; S325, after adjusting the size of the multi-sequence MRI features and the clinical features through a multi-layer perceptron, the two are spliced through a dynamic normalization weight layer, and the spliced features are assigned attention weights through a channel information guiding module to obtain the final fusion features; S4, constructing an anti-NMDAR encephalitis prognosis classifier based on the fusion features, and training it through a multi-modal data set; S5, inputting the MRI images and clinical features of the anti-NMDAR encephalitis patients to be classified into the trained classifier to obtain the prognosis classification results.

2. The anti-NMDAR encephalitis prognosis classification method based on multi-modal feature fusion according to claim 1, characterized in that, In step S2, the method for extracting image features from the MRI of a patient comprises the steps of: S21, projecting the three-dimensional MRI to a two-dimensional space to obtain a two-dimensional image sequence as an image input; S22, unifying different sizes of image sequences to 512x512 through bilinear interpolation; S23, constructing a convolutional neural network for two-dimensional image sequence feature extraction, which is based on a residual structure and enhances feature extraction performance through a channel information guiding module; S24, inputting the two-dimensional image sequence into the convolutional neural network for MRI feature extraction.

3. The anti-NMDAR encephalitis prognosis classification method based on multi-modal feature fusion according to claim 2, characterized in that, In steps S3 and S23, the channel information guiding module comprises the steps of: S311, extracting two independent context information of the input features through global average pooling and global maximum pooling; S312, processing the context information using a multi-layer perceptron, and adding the two element by element; S313, obtaining attention weights through a Sigmoid function; S314, assigning attention weights to the input features processed by the multi-layer perceptron to obtain output features.

4. The anti-NMDAR encephalitis prognosis classification method based on multi-modal feature fusion according to claim 1, characterized in that, In step S4, the constructed anti-NMDAR encephalitis prognosis classifier is: after the multi-modal feature fusion model, the fusion features obtained from the image features and clinical features of the patient are input into two full connection layers to obtain the final prognosis classification results.

5. The anti-NMDAR encephalitis prognosis classification method based on multi-modal feature fusion according to claim 1, characterized in that, In step S5, the performance evaluation parameters of the trained anti-NMDAR encephalitis prognosis classification model include: Wherein, AUC refers to the area under the curve of the pharmacokinetic parameters; Accurary refers to the accuracy; Specificity refers to the specificity; Sensitivity refers to the sensitivity; M refers to the number of positive samples; N refers to the number of negative samples; TP refers to the true positive sample; TN refers to the true negative sample; FP refers to the false positive sample; FN refers to the false negative sample.

Images