A deep learning-based medical image classification and feature visualization method

Abstract

The application provides a medical image classification and feature visualization method based on a deep learning model and a feature visualization tool, and comprises the following steps: A, removing non-key information frames from a brain MRI image; B, performing slice enhancement on the data to obtain a slice data set; C, transversely splicing the slices in sequence to obtain a spliced picture data set; D, putting the spliced picture data set into SE-ResNet50 for training and testing to obtain a classification result; E, putting the trained spliced picture network and a sample spliced picture into Grad-CAM to generate a heat map; F, selecting key frames from the heat map, putting the key frame slice set into SE-ResNet50 to extract features and classify; G, generating a key frame heat map through Grad-CAM; and H, using the obtained classification result and heat map to assist in diagnosis. The application discloses a method for classifying and visualizing features of brain images based on a deep learning method. An optimized SE-ResNet50 network is used to extract features and classify patient brain images. Abnormal regions are marked and visualized through Grad-CAM, so that the efficiency and diagnosis rate are improved.

Description

(I) Technical Field

[0001] This invention belongs to the field of deep learning convolutional neural networks and uses deep learning methods to classify and visualize the features of patient brain images. (II) Background Technology

[0002] The global prevalence of autism is currently on the rise. There is no single, comprehensive diagnostic tool or method for autism, requiring comprehensive evaluation of suspected cases. Cognitive neuroscience in psychology attempts to provide an explanatory model for autism, while neuroscience believes that abnormalities in brain mechanisms are a significant factor in its development. Related research has found that a reduced number of neurons in the brain, and decreased dendrites in the anterior cingulate cortex of the limbic system and the hippocampus, may be related to autism. Currently, there is no single effective intervention to cure autism, but early and comprehensive intervention leads to better treatment outcomes. Therefore, more advanced technologies are needed for the diagnosis of autism.

[0003] With the rapid development of magnetic resonance imaging (MRI) technology, functional magnetic resonance imaging (fMRI) has been widely used in research on brain diseases and cognitive neuroscience. Resting-state functional MRI (rs-fMRI) refers to fMRI images taken when the subject is not performing any tasks, and its purpose is to examine the correlation between different brain regions. Current research using rs-fMRI in the study of autism indicates that abnormalities in the structural connections of the brain in patients, primarily manifested as functional connectivity impairments, lead to cognitive and behavioral abnormalities.

[0004] Deep learning has achieved many innovative results in medical research in recent years, demonstrating excellent performance in the classification, detection, and segmentation of medical images. Its technology is increasingly being applied to medical image processing and analysis, reducing doctors' workload and improving diagnostic efficiency and accuracy. Consequently, deep learning-based fMRI diagnostic models for autism are constantly emerging. The classification and analysis of autistic brain rs-MRI images using deep learning methods plays a crucial role in diagnosis and intervention. It can supplement the physiological explanation of autism, refine and improve diagnostic methods, make autism assessment more comprehensive, assist doctors in judgment, improve the diagnostic accuracy, and increase diagnostic efficiency, enabling patients to receive intervention earlier. Furthermore, proposing an effective method for medical image feature extraction and classification, and providing it to researchers in different fields, can promote the mutual complementarity and development of computer science and modern medicine. (III) Summary of the Invention

[0005] This invention provides a medical rs-fMRI image classification and feature visualization method based on the deep learning model SE-ResNet50 and the feature visualization tool Gradient Weighted Class Activation Map (Grad-CAM). This method can classify corresponding case samples and normal control group samples, and identify areas of abnormal brain structure in patients from both macroscopic and microscopic perspectives.

[0006] The technical problem solved by this invention can be achieved by the following technical solution: This invention provides a medical rs-fMRI image classification and feature visualization method based on SE-ResNet50 and Grad-CAM, comprising the following steps:

[0007] A. Obtain preprocessed three-dimensional brain MRI image samples and remove frames without key information.

[0008] B. Slice each data sample according to the three directions of cross section, sagittal plane and coronal plane, and then overlap every 3 slices in order to form a slice dataset.

[0009] C. Slices in the same direction are horizontally stitched together according to the frame number order, and the stitched images are organized into a stitched image dataset.

[0010] D. Input the stitched image dataset into the optimized SE-ResNet50 for training and testing, and obtain the classification results of the stitched images in three directions of the sample.

[0011] E. Based on different directions, the trained mosaic network and the corresponding sample mosaic are used to generate a heatmap in Grad-CAM.

[0012] F. Select keyframes from the heatmap generated from the stitched image samples, and then put the slice set of keyframes into SE-ResNet50 for feature extraction and classification.

[0013] G. Obtain the feature results of the trained keyframe slice network through Grad-CAM, and generate a keyframe heatmap by combining the corresponding keyframe slice map.

[0014] H. The model output stitched image, keyframe classification results, and corresponding heatmaps serve as auxiliary diagnostic tools.

[0015] Innovation: A method for better classification of patients' brain rs-fMRI images.

[0016] To visualize the identified features

[0017] The processing of the acquired dataset by this invention, such as removing frames without key information, slicing data samples according to direction, overlapping slices, and horizontal stitching, can reduce the number of samples to be classified, enhance the sample data, improve computational performance, and study the features and classification effects of patient brain images in three different directions.

[0018] Meanwhile, to extract features and classify the sample data, the SE-ResNet50 network used in this invention is a fusion of ResNet50 and SE-Block. The SE module enhances important features in the data samples, making ResNet50's classification performance better. Furthermore, to optimize the training effect of SE-ResNet50 and obtain stable and good classification accuracy, this paper also tests and adjusts the optimizer and learning rate of the SE-ResNet50 model pre-trained on ImageNet.

[0019] To visualize the patient brain features extracted by the SE-ResNet50 network, this experiment used the feature visualization tool Grad-CAM. The feature visualization was divided into two types: macroscopic and microscopic. The first type marked relatively macroscopic patient brain features from stitched images of sequence slices from various directions, which could locate the approximate feature regions of the patient's brain and also show the feature relationships between different parts. The second type used microscopic features from keyframe slices selected from macroscopic feature regions, which could display more details and help doctors more accurately locate specific locations in the patient's brain. (iv) Description of the attached drawings

[0020] Figure 1 This is a summary of the instruction manual with accompanying drawings.

[0021] Figure 2 Based on Se - A ResNet50+Grad-CAM brain imaging classification and visualization model for autism.

[0022] Figure 3 This is the overall structure of SE-ResNet50 based on transfer learning in this paper. (V) Specific Implementation Plan

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] See Figure 2 The basic idea of ​​this invention is:

[0025] A. Obtain preprocessed three-dimensional brain MRI image samples and remove frames without key information according to the three directions of transverse, sagittal and coronal planes.

[0026] B. Slice each data sample according to the three directions of cross section, sagittal plane and coronal plane, and then overlap every 3 slices in sequence, i.e. slice enhancement, and organize them into slice datasets in each direction.

[0027] C. Slices in the same direction are horizontally stitched together according to frame number order, and all sample stitched images are organized into a stitched image dataset according to direction.

[0028] D. Input the stitched image dataset into the optimized SE-ResNet50 (as shown in...) Figure 3 (As shown) Training and testing were performed to obtain the classification results of the spliced ​​images of the samples in three directions.

[0029] E. Based on different directions, the trained mosaic network and the corresponding sample mosaic are used to generate a heatmap in Grad-CAM.

[0030] F. Based on the heatmap generated from the stitched image samples, keyframes and corresponding slice datasets are selected according to the positions with high energy. Then, the slice datasets of keyframes are put into SE-ResNet50 for feature extraction and classification, and the keyframe classification results are output.

[0031] G. Obtain the feature results of the trained keyframe slice network through Grad-CAM, and generate a keyframe heatmap by combining the corresponding keyframe slice map.

[0032] H. Finally, the model outputs a stitched image, keyframe classification results, and corresponding heatmaps as auxiliary diagnostic tools.

Claims

1. A method for medical image classification and feature visualization based on the deep learning model SE-ResNet50 and the feature visualization tool Grad-CAM, characterized in that... A method is provided to improve the classification of rs-fMRI images of a patient's brain and to visualize the identified features. The method includes the following steps: A. Obtain preprocessed three-dimensional brain MRI image samples and remove frames without key information. B. Slice each data sample according to the three directions of cross section, sagittal plane and coronal plane, and then overlap every 3 slices in order to form a slice dataset. C. Slices in the same direction are horizontally stitched together according to the frame number order, and then organized into a stitched image dataset. D. Input the stitched image dataset into the optimized SE-ResNet50 for training and testing, and obtain the classification results of the stitched images in three directions of the sample. E. Based on different directions, the trained mosaic network and the corresponding sample mosaic are used to generate a heatmap in Grad-CAM. F. Select keyframes from the heatmap generated from the stitched image samples, and then put the slice set of keyframes into SE-ResNet50 for feature extraction and classification. G. Obtain the feature results of the trained keyframe slice network through Grad-CAM, and generate a keyframe heatmap by combining the corresponding keyframe slice map. H. The stitched image, keyframe classification results, and corresponding heatmaps output by the model are used as auxiliary diagnostic tools.

2. The method according to claim 1, characterized in that: Step A, which involves acquiring preprocessed three-dimensional brain MRI image samples and removing frames without key information, refers to removing frames without key information in three directions: transverse, sagittal, and coronal.

3. The method according to claim 1, characterized in that: Step B involves slicing each data sample according to the three directions of cross section, sagittal plane, and coronal plane, and then overlapping every 3 slices in sequence to form a slice dataset for each sample. The purpose is to reduce the number of samples to be classified, enhance the sample data, and improve computational performance.

4. The method according to claim 1, characterized in that: Step C involves horizontally stitching together slices in the same direction according to frame number order, resulting in a three-directional stitched image dataset.

5. The method according to claim 1, characterized in that: Step F, which involves selecting keyframes from the heatmap generated by the stitched image samples, selects keyframes and corresponding slice datasets based on locations with high energy. Then, the slice datasets of the keyframes are put into SE-ResNet50 for feature extraction and classification.

Images