Multimodal information classification method for parkinson diagnosis

Abstract

The present application belongs to the technical field of medical image processing, and more particularly relates to a multi-modal information classification method for Parkinson diagnosis. The method comprises obtaining three-dimensional MRI images and four-dimensional DTI images and pre-processing, and then obtaining a hand-drawn Archimedean spiral diagram; the pre-processed MRI images, DTI images and hand-drawn Archimedean spiral diagram are divided into a training set and a test set; an analysis and processing model is established, which comprises a feature extraction module, a feature fusion module and a classification module; the training set is input into the analysis and processing model for training, and after passing the test set, a final model is obtained, and the to-be-processed data is input into the final model. The present application solves the problems of insufficient fusion, high error rate of diagnosis and the like in the fusion of multi-modal information in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and more specifically, relates to a multimodal information classification method for Parkinson's disease diagnosis. Background Technology

[0002] Parkinson's disease is a common neurological disorder among middle-aged and elderly people, characterized by its long course, high disability rate, and difficulty in detection and diagnosis. Many patients only discover abnormalities when the disease has progressed to the middle or late stages, missing the golden period for early treatment. However, the development of artificial intelligence and deep learning, and their application in the medical field, have led researchers to seek more convenient ways to assist doctors in diagnosing Parkinson's disease, promoting early detection and treatment, and seizing the golden period for early intervention.

[0003] Chinese invention patent CN116072267A discloses an intelligent diagnostic analysis method for Parkinson's disease based on MRI images. First, it reads medical image data, selecting either a DICOM or DaT file using a file selector. After acquiring the selected file, the software starts a file processing thread to obtain information such as voxels, spatial spacing, window width, and window level from the medical image data, and then visualizes this information. Next, a lesion region detection module is used, employing a YOLOv5 model to detect lesions, outputting a disease risk score and the lesion region, and visualizing this information. Finally, the lesion region is segmented, refining DaT image features, and using a U-Net network to segment the striatum region, further locating the lesion area by dividing the feature region into the right substantia nigra, and then visualizing this information.

[0004] There are few existing AI-assisted diagnostic methods for Parkinson's disease, and most of them are single-modal, such as those based on scales, speech, frozen gait, EEG signals, handwriting dynamics, and medical imaging. However, there are bound to be differences between individuals. Some people may only show abnormalities in one aspect but be normal in others. If a diagnosis is made based on a single aspect, the result will inevitably be one-sided. Existing multimodal fusion methods also suffer from insufficient fusion and high error rates in diagnosis. Summary of the Invention

[0005] The present invention aims to overcome at least one of the defects of the prior art and provide a multimodal information classification method for Parkinson's disease diagnosis, so as to solve the problems of misdiagnosis rate due to individual differences and insufficient fusion and high error rate in diagnosis in the multimodal fusion of the prior art.

[0006] The detailed technical solution of this invention is as follows:

[0007] S1. Acquire three-dimensional MRI images and four-dimensional DTI images and preprocess them, then acquire a hand-drawn Archimedes spiral diagram;

[0008] The preprocessing includes skull separation and standard position registration of MRI images to obtain a standard brain map, and then tissue separation of the standard brain map to obtain three-dimensional gray matter, white matter, and cerebrospinal fluid.

[0009] The average value of the volume dimension of the DTI image is calculated to obtain a three-dimensional average DTI image. Then, the skull is separated from the three-dimensional DTI image to obtain the brain structure. The FA and MD values ​​of the brain structure are extracted. Finally, the FA and MD values ​​are registered into the MNI152 standard template.

[0010] S2. The preprocessed MRI images, DTI images, and hand-drawn Archimedes spiral diagrams are divided into training and testing sets.

[0011] S3. Establish an analysis and processing model, which includes a feature extraction module, a feature fusion module, and a classification module;

[0012] S4. Input the training set into the analysis and processing model for training, and then test it on the test set to obtain the final model.

[0013] S5. Input the data to be processed into the final model.

[0014] Furthermore, the feature extraction module uses a ResNet network and has been adjusted as follows:

[0015] Change the batch normalization layer to a group normalization layer;

[0016] Modify the ReLU activation function to the Gelu activation function;

[0017] To reduce the number of downsampling operations, the ResNet network includes three stages: stage1, stage2, and stage3. In stage1, neither of the two residual structures is downsampled. In stage2, one of the two residual structures is downsampled and the other is downsampled. In stage3, there are four residual structures: three are downsampled and one is downsampled.

[0018] Furthermore, the feature fusion module includes N residual structures. Depthwise separable convolutions are used to replace ordinary convolutions in the residual structures. The group convolutions and point convolutions in the depthwise separable structures are separated. The group convolutions are placed inside the residual structures to fuse spatial positions in the feature maps. The point convolutions are placed outside the residual structures to fuse channel positions. After the group convolutions and point convolutions, the Gelu activation function is used to enhance the feature representation ability of the fused features and group normalization to prevent gradient vanishing and gradient exploding.

[0019] Furthermore, the classification module includes a dual-channel pooling layer and a fully connected layer, wherein the dual-channel pooling layer includes a max pooling layer and an average pooling layer.

[0020] The processing steps of the analysis and processing model are as follows:

[0021] S41. The feature extraction module first performs dimensionality reduction and noise reduction on the data, and extracts key information from the data that is identifiable, representative, and used to describe and distinguish the differences or similarities between the data, removes interference information, and obtains the original input.

[0022] The interference information includes noise and redundancy features;

[0023] The original input includes: A1 (gray matter), A2 (white matter), A3 (cerebrospinal fluid), B1 (FA value), B2 (MD value), and C1 (spiral graph), where A1, A2, A3, B1, and B2 have 3 dimensions, and C1 has 2 dimensions. Feature extraction is performed on these six data types using 3D ResNet and 2D ResNet respectively.

[0024]

[0025]

[0026]

[0027] After feature extraction, three types of feature maps, F(A), F(B) and F(C), are obtained. The shapes of F(A) and F(B) are H×W×D, and the shape of F(C) is H′×W′.

[0028] The feature extraction model has three stages, namely the feature extraction process R = {S1, S2, S3}. The feature extraction process of each stage can be expressed by the formula:

[0029] S1=2[F(x)(x,{W i})+x] (4);

[0030] S2=F(x)(x,{W i})+W s x+[F(x)(x,{W i})+x] (5);

[0031] S3=F(x)(x,{W i})+W s x+3[F(x)(x,{W i})+x] (6);

[0032] in,

[0033] F(x)(x,{W i})=GN(W2(Gelu(GN(W1x)))) (7).

[0034] S42. The feature fusion module proposes a feature fusion method, as follows:

[0035] Expanding the six feature maps F(A1), F(A2), F(A3), F(B1), F(B2), and F(C1) yields six feature maps with identical shapes, namely:

[0036] H×W×D=1×(HWD) (8);

[0037] H′×W′=1×(H′W′) (9);

[0038] 1×(HWD)=1×(H′W′)=H×W (10);

[0039] After unfolding, all feature maps have a value of H equal to 1 and the value of W is the same. Therefore, all feature maps can be concatenated along the H dimension to obtain a comprehensive feature map Z containing all modal features. Each row of feature map Z represents a class of feature maps:

[0040] Z=Concat[Flatten(F(A)),Flatten(F(B)),Flatten(F(C))] H (11);

[0041] After all the feature maps are stitched together, N residual structures are used to extract features from the stitched composite features.

[0042] In depthwise separable features, grouped convolutions and point convolutions are separated. Grouped convolutions are placed inside the residual structure to fuse spatial locations in the feature maps, while point convolutions are placed outside the residual structure to fuse channel locations. Gelu activation functions are then applied after both grouped and point convolutions to enhance the expressive power of the fused features and to perform group normalization to prevent gradient vanishing and gradient exploding. Specifically:

[0043] Conv(Z) = {Z D Z P} N (12);

[0044] Z D Z represents the grouped convolution process. P This represents the point convolution process. Where Z... D Located inside the residual structure, Z P Located outside the residual structure, specifically:

[0045] ZD =GN(Gelu(ConvDepthwise(Z) D )))+Z D (13);

[0046] Z P =GN(Gelu(ConvPointwise(Z) P ))) (14).

[0047] S43. After feature extraction and feature fusion are completed, the model performs classification using a dual-channel pooling layer and a fully connected layer, as follows:

[0048] The dual-channel pooling layer refers to passing the extracted composite features after fusion through a max pooling layer and an average pooling layer respectively to obtain the texture information and background information of the features and then adding the two together.

[0049] Finally, the combined features are passed through a fully connected layer to obtain the final classification result.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] (1) The multimodal information classification method for Parkinson's diagnosis provided by the present invention selects three modal data, namely three-dimensional MRI images, four-dimensional DTI images and two-dimensional spiral diagrams, to diagnose Parkinson's disease, thereby reducing the misdiagnosis rate caused by individual differences in traditional single-modal diagnosis.

[0052] (2) The multimodal information classification method for Parkinson's diagnosis provided by the present invention continuously adjusts and extracts features from three different dimensions of data according to their respective characteristics to achieve the fusion of data from different dimensions, and then performs convolution on the fused data again to extract features, thereby ensuring the full fusion of multimodal data.

[0053] (3) The multimodal information classification method for Parkinson's disease diagnosis provided by this invention is based on deep learning. It extracts features from multimodal data to assist in the diagnosis of Parkinson's disease. Compared with traditional clinical experience-based diagnostic methods, deep learning is more objective. Compared with single-modal diagnostic methods, multimodal diagnostic methods can avoid misjudgment caused by individual differences to the greatest extent. Compared with multimodal image diagnostic methods, this method adds modal information of another data structure in addition to multimodal images, making the diagnostic results more comprehensive and reducing the error rate of diagnosis to the greatest extent. Attached Figure Description

[0054] Figure 1 This is the data processing flow described in this invention.

[0055] Figure 2 This is a flowchart of the overall dataset preprocessing process in Embodiment 1 of the present invention.

[0056] Figure 3 These are the MRI images and processed images in Embodiment 1 of the present invention.

[0057] Figure 4 These are the DTI images and processed images in Embodiment 1 of the present invention.

[0058] Figure 5 These are different hand-drawn spiral diagrams in Embodiment 1 of the present invention.

[0059] Figure 6 This is a modified ResNet network structure diagram in Embodiment 1 of the present invention.

[0060] Figure 7 This is the overall flowchart of feature fusion classification in Embodiment 1 of the present invention. Detailed Implementation

[0061] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0062] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0063] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0064] Where there is no conflict, the embodiments and features described herein can be combined with each other.

[0065] Example 1

[0066] This embodiment provides a multimodal information classification method for Parkinson's disease diagnosis, such as... Figure 1 As shown:

[0067] S1. Acquire three-dimensional MRI images and four-dimensional DTI images and preprocess them, then acquire a hand-drawn Archimedes spiral diagram;

[0068] The preprocessing includes skull separation and standard position registration of the MRI images to obtain a standard brain map, followed by tissue separation of the standard brain map to obtain three-dimensional gray matter, white matter, and cerebrospinal fluid, such as... Figure 3As shown, 1 is the MRI image before processing, 2 is the gray matter after processing, 3 is the white matter after processing, and 4 is the cerebrospinal fluid after processing.

[0069] The mean value of the volume dimension of the DTI images is calculated to obtain a 3D averaged DTI image. Then, skull separation is performed on the 3D DTI image to obtain the brain structure. The FA and MD values ​​of the brain structure are extracted, and finally, the FA and MD values ​​are registered into the MNI152 standard template. Figure 4 As shown, 5 is the DTI image before processing, 6 is the FA value after processing, and 7 is the MD value after processing.

[0070] The hand-drawn Archimedes spiral diagram is as follows: Figure 5 As shown, the first image is a spiral diagram of a normal person, the second image is a hand-drawn spiral diagram of an early-stage Parkinson's patient, and the third image is a hand-drawn spiral diagram of a mid-to-late-stage Parkinson's patient.

[0071] The dataset preprocessing process is as follows: Figure 2 As shown:

[0072] Parkinson's disease is often characterized on medical imaging by the loss of normal high signal intensity in the substantia nigra pars compacta, blurred margins, and a reduction or loss of the normal swallowtail shape of the substantia nigra. This section uses 3D MRI and 4D DTI images. The MRI image preprocessing workflow is as follows: First, the BET component in FSL software is used to separate the skull from the MRI images, removing irrelevant tissues such as the eyeball and fat. However, due to factors such as image quality, some parts cannot be completely removed. Therefore, after skull separation, data is filtered to remove data that may affect subsequent experiments. Then, the FLIRT tool in FSL is used to register the MRI images to the MNI152 standard template, ensuring alignment of all brain morphologies and structures. Finally, the FAST tool in FSL is used to segment the brain, obtaining its white matter, gray matter, and cerebrospinal fluid. The DTI image processing workflow is as follows: First, since 4D DTI has one more volume dimension than 3D MRI images, the Maths tool in FSL software is used to average the volume dimension of the DTI images, obtaining a 3D averaged DTI image. Then, the BET component was used to separate the skull from the DTI images at this time to obtain the pure brain structure. The FA and MD values ​​of the brain structure were extracted using the DTIFIT tool in FSL. Finally, the FA and MD values ​​were registered to the MNI152 standard template using FLIRT to obtain the standard FA and MD values.

[0073] S2. The preprocessed MRI images, DTI images, and hand-drawn Archimedes spiral diagrams are divided into training and testing sets.

[0074] S3. Establish an analysis and processing model, which includes a feature extraction module, a feature fusion module, and a classification module;

[0075] Furthermore, the feature extraction module uses a ResNet network and has been adjusted as follows:

[0076] The batch normalization layer was changed to a group normalization layer. Due to the large volume of 3D image data, especially after feature fusion where the data volume increases exponentially, the batch size is limited by GPU size, resulting in a smaller batch size during training. When the batch size is small, the mean and variance of the current batch cannot represent the distribution of the overall input data, leading to a poorer normalization effect of the batch normalization layer on each batch. However, group normalization normalizes each group, is not limited by the batch size, and can better handle small batches of data. Therefore, the batch normalization layer was changed to a group normalization layer.

[0077] The ReLU activation function was modified to the Gelu activation function. While ReLU is widely used due to its computational simplicity, its overly simplistic approach suffers from issues such as output imbalance, vanishing gradients, and neuron death. Since Parkinson's disease images are single-channel images, and the symptoms are not particularly pronounced on these images, using ReLU would exacerbate the vanishing gradient problem. In contrast, Gelu is more suitable. As seen in Gelu's approximation formula, larger x values ​​are more likely to be preserved, while smaller x values ​​are more likely to be approximated to 0, making the function more random and improving the robustness of the trained model. Furthermore, the derivative formula and graph show that Gelu's derivative is smooth and differentiable throughout, effectively avoiding the discontinuous derivative and the inability to update some data in the long term that ReLU encounters when dealing with small inputs. Compared to Sigmoid and Tanh activation functions, Gelu produces a stronger nonlinear response when dealing with larger inputs, and its output falls within a wider range, thus accelerating the model's convergence speed to some extent.

[0078] Gelu approximation formula:

[0079] Gelu's derivative formula:

[0080] To reduce the number of downsampling operations, the original ResNet network consists of four stages: original stage 1, original stage 2, original stage 3, and original stage 4. Original stage 1 has two residual structures, neither of which is downsampled; original stage 2 has two residual structures, one without downsampling and one with downsampling; original stage 3 has two residual structures, one without downsampling and one with downsampling; and original stage 4 has two residual structures, one without downsampling and one with downsampling.

[0081] The modified ResNet network consists of three stages: stage1, stage2, and stage3. Stage1 has two residual structures without downsampling (s=1). Stage2 has one residual structure without downsampling (s=1) and one with downsampling (s=2). Stage3 includes four residual structures: three without downsampling (s=1) and one with downsampling (s=2) [Note: s=1 represents no downsampling, s=2 represents downsampling]. Firstly, this preserves higher-resolution feature maps: removing the downsampling structure in stage 4 maintains a relatively large resolution feature map size, allowing the model to more easily capture image details and better identify tiny lesions invisible to the human eye. Secondly, it preserves more spatial information from 3D images: in 3D images, the model must consider not only semantic information but also spatial information; reducing downsampling by one step preserves more spatial information, leading to better lesion localization.

[0082] Furthermore, the feature fusion module includes N residual structures. Depthwise separable convolutions are used to replace ordinary convolutions in the residual structures. The group convolutions and point convolutions in the depthwise separable structures are separated. The group convolutions are placed inside the residual structures to fuse spatial positions in the feature maps. The point convolutions are placed outside the residual structures to fuse channel positions. After the group convolutions and point convolutions, the Gelu activation function is used to enhance the feature representation ability of the fused features and group normalization to prevent gradient vanishing and gradient exploding.

[0083] Furthermore, the classification module includes a dual-channel pooling layer and a fully connected layer, wherein the dual-channel pooling layer includes a max pooling layer and an average pooling layer.

[0084] S4. Input the training set into the analysis and processing model for training, and then test it on the test set to obtain the final model.

[0085] The processing steps of the analysis and processing model are as follows:

[0086] S41. The feature extraction module first performs dimensionality reduction and noise reduction on the data, and extracts key information from the data that is identifiable, representative, and used to describe and distinguish the differences or similarities between the data, removes interference information, and obtains the original input.

[0087] The interference information includes noise and redundancy features;

[0088] The original input includes: A1, A2, A3, B1, B2, and C1, where A1 represents gray matter, A2 represents white matter, A3 represents cerebrospinal fluid, B1 represents FA value, B2 represents MD value, and C1 represents a spiral diagram, i.e., a hand-drawn Archimedes spiral diagram. A1, A2, A3, B1, and B2 have a dimension of 3, and C1 has a dimension of 2. Feature extraction is performed on these six types of data using 3D ResNet and 2D ResNet respectively.

[0089]

[0090]

[0091]

[0092] After feature extraction, three types of feature maps, F(A), F(B), and F(C), are obtained. F(A) and F(B) have shapes of H×W×D, and F(C) has a shape of H. ′ ×W ′ ;

[0093] The feature extraction model has three stages, namely the feature extraction process R = {S1, S2, S3}. The feature extraction process of each stage can be expressed by the formula:

[0094] S1=2[F(x)(x,{W i})+x] (4);

[0095] S2=F(x)(x,{W i})+W s x+[F(x)(x,{W i})+x] (5);

[0096] S3=F(x)(x,{W i})+W s x+3[F(x)(x,{W i})+x] (6);

[0097] in,

[0098] F(x)(x,{W i})=GN(W2(Gelu(GN(W1x)))) (7).

[0099] S42. The feature fusion module proposes a feature fusion method, as follows:

[0100] Expanding the six feature maps F(A1), F(A2), F(A3), F(B1), F(B2), and F(C1) yields six feature maps with identical shapes, namely:

[0101] H×W×D=1×(HWD) (8);

[0102] H ′ ×W ′ =1×(H ′ W ′ (9);

[0103] 1×(HWD)=1×(H ′ W ′ )=H×W (10);

[0104] After unfolding, the H values ​​of all feature maps become 1 and the W values ​​are the same. Therefore, all feature maps can be concatenated along the H dimension to obtain the comprehensive feature map Z. One row of the comprehensive feature map Z represents a class of feature maps.

[0105] Z=Concat[Flatten(F(A)),Flatten(F(B)),Flatten(F(C))] H (11);

[0106] After all the feature maps are stitched together, N residual structures are used to extract features from the stitched composite features.

[0107] Since the number of parameters is relatively large at this point, the grouped convolution and point convolution in the depthwise separable model are separated. The grouped convolution is placed inside the residual structure to fuse the spatial positions in the feature maps, while the point convolution is placed outside the residual structure to fuse the channel positions. The Gelu activation function is then applied after the grouped and point convolutions to enhance the expressive power of the fused features and to perform group normalization to prevent gradient vanishing and gradient exploding. Specifically:

[0108] Conv(Z) = {Z D Z P} N (12);

[0109] Z D Z represents the grouped convolution process. P This represents the point convolution process. Where Z... D Located inside the residual structure, Z P Located outside the residual structure, specifically:

[0110] Z D=GN(Gelu(ConvDepthwise(Z) D )))+Z D (13);

[0111] Z P =GN(Gelu(ConvPointwise(Z) P ))) (14).

[0112] S43. After feature extraction and feature fusion are completed, the model performs classification using a dual-channel pooling layer and a fully connected layer, as follows:

[0113] The dual-channel pooling layer refers to passing the extracted composite features after fusion through a max pooling layer and an average pooling layer respectively to obtain the texture information and background information of the features and then adding the two together.

[0114] Finally, the combined features are passed through a fully connected layer to obtain the final classification result: normal person or Parkinson's patient.

[0115] Furthermore, after completing feature extraction and feature fusion, the model performs classification using a dual-channel pooling layer and a fully connected layer, as follows:

[0116] The dual-channel pooling layer refers to passing the extracted composite features after fusion through a max pooling layer and an average pooling layer respectively to obtain the texture information and background information of the features and then adding the two together.

[0117] Finally, the combined features are passed through a fully connected layer to obtain the final classification result.

[0118] The criteria for passing the test set can be adjusted by setting relevant thresholds according to actual needs.

[0119] S5. Input the data to be processed into the final model.

[0120] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A multimodal information classification method for Parkinson's disease diagnosis, characterized in that, include; S1: Acquire 3D MRI images and 4D DTI images and preprocess them, then acquire a hand-drawn Archimedes spiral diagram; S2. The preprocessed MRI images, DTI images, and hand-drawn Archimedes spiral diagrams are divided into training and testing sets. S3. Establish an analysis and processing model, which includes a feature extraction module, a feature fusion module, and a classification module; The feature extraction module uses the ResNet network and has been adjusted as follows: Change the batch normalization layer to a group normalization layer; Modify the ReLU activation function to the Gelu activation function; To reduce the number of downsampling operations, the ResNet network comprises three stages: stage1, stage2, and stage3. Neither of the two residual structures in stage1 incorporates downsampling. In stage2, one residual structure does not incorporate downsampling, and the other does. Stage 3 includes four residual structures, three without downsampling and one with downsampling; The feature fusion module includes N residual structures, which use depthwise separable convolutions to replace ordinary convolutions in the residual structures, and separate the group convolutions and point convolutions in the depthwise separable structures. Grouped convolutions are placed inside the residual structure to fuse spatial locations in the feature maps; point convolutions are placed outside the residual structure to fuse channel locations. The Gelu activation function is used after the grouped and point convolutions to enhance the feature representation ability of the fused feature maps and group normalization to prevent gradient vanishing and gradient exploding. The classification module includes a dual-channel pooling layer and a fully connected layer, wherein the dual-channel pooling layer includes a max pooling layer and an average pooling layer; S4. Input the training set into the analysis and processing model for training, and then test it on the test set to obtain the final model. S5. Input the data to be processed into the final model.

2. The multimodal information classification method for Parkinson's disease diagnosis according to claim 1, characterized in that, The preprocessing includes skull separation and standard position registration of MRI images to obtain a standard brain map, and then tissue separation of the standard brain map to obtain three-dimensional gray matter, white matter, and cerebrospinal fluid. The average value of the volume dimension of the DTI image is calculated to obtain a three-dimensional average DTI image. Then, the skull is separated from the three-dimensional DTI image to obtain the brain structure. The FA and MD values ​​of the brain structure are extracted. Finally, the FA and MD values ​​are registered into the MNI152 standard template.

3. The multimodal information classification method for Parkinson's disease diagnosis according to claim 1, characterized in that, The processing steps of the analysis and processing model are as follows: S41. The feature extraction module first performs dimensionality reduction and noise reduction on the data, and extracts key information from the data that is identifiable, representative, and used to describe and distinguish the differences or similarities between the data, removes interference information, and obtains the original input. The original input includes: , , , , and Six categories, Indicates gray matter, Indicates white matter, Indicates cerebrospinal fluid, Indicates FA value, Indicates MD value, Represents a spiral diagram, where, , , , , The dimension is 3. With a dimension of 2, features were extracted from these 6 data types using both 3D ResNet and 2D ResNet: （1）； （2）； （3）； After feature extraction, a total of , , Three types of feature maps, among which... and The shape is , The shape is ; The feature extraction model has three stages, namely the feature extraction process. The feature extraction process for each stage can be expressed by the formula: （4）； （5）； （6）； in, （7）； S42. The feature fusion module proposes a feature fusion method, as follows: Six feature maps , , , , , Unfolding yields six feature maps with identical shapes, namely: （8）； （9）； （10）； All feature maps after unfolding Since W becomes 1 and is the same, all feature maps can be concatenated along the H dimension to obtain a comprehensive feature map. Comprehensive feature map One row represents a type of feature map: （11）； After all the feature maps are stitched together, N residual structures are used to extract features from the stitched composite features. In depthwise separable features, grouped convolutions and point convolutions are separated. Grouped convolutions are placed inside the residual structure to fuse spatial locations in the feature maps, while point convolutions are placed outside the residual structure to fuse channel locations. Gelu activation functions are then applied after both grouped and point convolutions to enhance the expressive power of the fused features and to perform group normalization to prevent gradient vanishing and gradient exploding. Specifically: （12）； This represents the grouped convolution process. This represents the point convolution process; where, Located inside the residual structure, Located outside the residual structure, specifically: （13）； （14）； S43. After feature extraction and feature fusion are completed, the model performs classification using a dual-channel pooling layer and a fully connected layer, as follows: The dual-channel pooling layer refers to passing the extracted composite features after fusion through a max pooling layer and an average pooling layer respectively to obtain the texture information and background information of the features and then adding the two together. Finally, the combined features are passed through a fully connected layer to obtain the final classification result.

4. The multimodal information classification method for Parkinson's disease diagnosis according to claim 3, characterized in that, The interference information includes noise and redundancy features.

Images