Multi-time-point based structural magnetic resonance imaging classification method, system and terminal
By combining 3D ResNet18, 3D CLSTM and MSFF networks, spatial and longitudinal features of images are integrated, solving the problems of existing technologies that fail to effectively preserve the three-dimensional spatial information of features and ignore temporal information, thus improving the classification accuracy of structural magnetic resonance imaging.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN UNIV
- Filing Date
- 2024-04-10
- Publication Date
- 2026-07-07
AI Technical Summary
In existing technologies, structural magnetic resonance imaging classification methods fail to effectively preserve the three-dimensional spatial information of features and rely on baseline data without considering temporal information, resulting in inaccurate classification results.
A multi-time-point-based structural magnetic resonance imaging classification method is adopted, which combines a 3D ResNet18 network, a 3D CLSTM network, an MSFF network, and a softmax layer. Through feature extraction, cascading, and fusion, the spatial and longitudinal features of the image are integrated, preserving the three-dimensional spatial information of the features and combining the temporal information of the data.
It improves the accuracy of structural magnetic resonance imaging classification results and enhances the ability to classify images, especially when considering temporal and spatial feature transformations.
Smart Images
Figure CN118230054B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a multi-time-point structural magnetic resonance imaging classification method, system, terminal, and computer-readable storage medium. Background Technology
[0002] Structural magnetic resonance imaging (sMRI) is a non-invasive imaging technique that uses magnetic resonance imaging to perform structural anatomy analysis of the brain. It provides information about the brain's morphology, size, regions, and tissue structure, and is of great significance for studying brain development and anatomical variations. SMRI utilizes the interaction between magnetic fields and radio waves to obtain high-resolution images of the brain by scanning the head. These images can be used to detect abnormal structures and assist doctors in making judgments.
[0003] Currently, deep learning technology has been widely applied to various image processing and classification problems, including structural magnetic resonance imaging (SMR). However, most deep learning methods often use one-dimensional features as input into an RNN (Recurrent Neural Network), without preserving the three-dimensional spatial information of the features. Moreover, the data used still mainly relies on baseline SMR data. Although the baseline SMR data provides key information for classification, this data only reflects the instantaneous state and does not take into account the temporal information in the data, resulting in inaccurate classification results for SMR.
[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention
[0005] The main objective of this invention is to provide a multi-time-point structural magnetic resonance imaging (SMR) classification method, system, terminal, and computer-readable storage medium. This invention aims to address the problems in existing technologies that rely on neural networks for SMR classification, which fail to preserve the three-dimensional spatial information of features and primarily depend on baseline sMR data without considering temporal information, resulting in inaccurate classification results for SMR.
[0006] To achieve the above objectives, the present invention provides a structural magnetic resonance imaging classification method based on multiple time points, the method comprising the following steps:
[0007] Acquire magnetic resonance imaging of the target structure;
[0008] A structural magnetic resonance imaging (SMR) classification model is constructed, and the SMR classification model is trained and tested to obtain a target model. The target model includes a 3D ResNet18 network, a 3D CLSTM network, an MSFF network, a fully connected layer, and a softmax layer.
[0009] The magnetic resonance imaging of the target structure is input into the 3D ResNet18 network for feature extraction to obtain convolutional features. The convolutional features are then input into the 3D CLSTM network for processing to obtain cascaded features. Finally, the cascaded features are input into the MSFF network for feature fusion to obtain fused features.
[0010] The fused features are classified through the fully connected layer and the softmax layer to obtain the classification result.
[0011] Optionally, the multi-time-point-based structural magnetic resonance imaging classification method, wherein training and testing the structural magnetic resonance imaging classification model to obtain the target model specifically includes:
[0012] Acquire historical structural magnetic resonance images of multiple individuals, perform AC-PC correction and skull removal on the multiple historical structural magnetic resonance images to obtain multiple intermediate brain images;
[0013] Each intermediate brain image is segmented and gray matter is extracted to obtain multiple gray matter images. The multiple gray matter images are adjusted to a uniform size to obtain a dataset, wherein the dataset includes gray matter images of each individual at three different time points.
[0014] The dataset is divided into a training set, a validation set, and a test set according to a preset ratio. The training set is used to train the structural magnetic resonance imaging (SMRI) classification model. The validation set is used to verify the performance and adjust the strategy of the trained SMRI classification model. The test set is used to evaluate the performance of the trained SMRI classification model, thereby obtaining the target model that meets the preset requirements.
[0015] Optionally, the multi-time-point structural magnetic resonance imaging classification method, wherein the step of inputting the target structural magnetic resonance image into the 3D ResNet18 network for feature extraction to obtain convolutional features specifically includes:
[0016] The target structure magnetic resonance imaging is preprocessed to obtain the target gray matter image of the target structure magnetic resonance imaging, and the gray matter image is three-dimensional data;
[0017] The grayscale image is input into the convolutional layer of the 3D ResNet18 network to extract vertical features with the same weights, thereby obtaining convolutional features, which are three-dimensional data.
[0018] The convolutional features include a first convolutional feature X1 extracted from the penultimate layer and a second convolutional feature X2 extracted from the penultimate layer.
[0019] Optionally, the multi-time-point-based structural magnetic resonance imaging classification method is characterized in that the 3D CLSTM network includes two improved LSTM models, a batch normalization module, and a modified linear unit activation function.
[0020] The improved LSTM model uses 3D convolution operations to replace the multilayer perceptron transform of the original LSTM model.
[0021] The batch normalization module and the modified linear unit activation function are used to enhance the training stability and convergence speed of the model.
[0022] Optionally, the multi-time-point structural magnetic resonance imaging classification method is characterized in that, the step of inputting the convolutional features into the 3D CLSTM network for processing to obtain cascaded features specifically includes:
[0023] The first convolutional feature X1 is input into an improved LSTM model for concatenation to obtain the first concatenated feature F1;
[0024] The second convolutional feature X2 is input into another improved LSTM model for concatenation to obtain the second concatenated feature F2.
[0025] The scales of the first cascaded feature F1 and the second cascaded feature F2 are different.
[0026] Optionally, the multi-time-point-based structural magnetic resonance imaging classification method is characterized in that the MSFF network includes: a first convolutional kernel, a second convolutional kernel, a global background module, a local background module, and a sigmoid activation function.
[0027] Optionally, the multi-time-point structural magnetic resonance imaging classification method is characterized in that, the step of inputting the cascaded features into the MSFF network for feature fusion to obtain fused features specifically includes:
[0028] The first concatenated feature F1 is input into the first convolution kernel for convolution operation to obtain the first target convolution feature, and the second concatenated feature F2 is input into the second convolution kernel for convolution operation to obtain the second target convolution feature.
[0029] The first target convolutional feature and the second target convolutional feature are added element by element to obtain the summed target convolutional feature;
[0030] The summed target convolutional features are input into the global background module for global average pooling and pointwise convolution to obtain the first low-channel-number convolutional features. The first low-channel-number convolutional features are then subjected to a non-linear transformation using the ReLU activation function. The result of the non-linear transformation is then subjected to pointwise convolution again to obtain the global features that restore the number of channels.
[0031] The summed target convolutional features are input into the local background module for point-by-point convolution to obtain the second low-channel-number convolutional features. The second low-channel-number convolutional features are then subjected to a non-linear transformation using the ReLU activation function. The result of the non-linear transformation is then subjected to point-by-point convolution again to obtain the local features with restored channel numbers.
[0032] The global features and the local features are added together to obtain multi-scale channel features. The multi-scale channel features are then subjected to a non-linear transformation using the sigmoid activation function to obtain target multi-scale channel features. The target multi-scale channel features are then multiplied element-wise with the first target convolutional feature and the second target convolutional feature to obtain target global features and target local features.
[0033] The target global features and the target local features are added element-wise, and the result of the element-wise addition is subjected to global average pooling to obtain the fused features.
[0034] Furthermore, to achieve the above objectives, the present invention also provides a multi-time-point structural magnetic resonance imaging classification system, wherein the multi-time-point structural magnetic resonance imaging classification system includes:
[0035] The target image acquisition module is used to acquire magnetic resonance imaging of the target structure.
[0036] The model building and training module is used to build a structural magnetic resonance imaging (SMRI) classification model, train and test the SMRI classification model to obtain a target model, wherein the target model includes: a 3D ResNet18 network, a 3DCLSTM network, an MSFF network, a fully connected layer and a softmax layer;
[0037] The feature extraction and fusion module is used to input the magnetic resonance imaging of the target structure into the 3D ResNet18 network for feature extraction to obtain convolutional features, input the convolutional features into the 3D CLSTM network for processing to obtain cascaded features, and input the cascaded features into the MSFF network for feature fusion to obtain fused features.
[0038] The classification prediction module is used to classify the fused features through the fully connected layer and the softmax layer to obtain the classification result.
[0039] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multi-time-point-based structural magnetic resonance imaging classification program stored in the memory and executable on the processor, wherein when the multi-time-point-based structural magnetic resonance imaging classification program is executed by the processor, it implements the steps of the multi-time-point-based structural magnetic resonance imaging classification method as described above.
[0040] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multi-time-point-based structural magnetic resonance imaging classification program, which, when executed by a processor, implements the steps of the multi-time-point-based structural magnetic resonance imaging classification method as described above.
[0041] In this invention, target structural magnetic resonance imaging (MRI) images are acquired; a structural MRI classification model is constructed, and the model is trained and tested to obtain a target model. The target model includes a 3D ResNet18 network, a 3D CLSTM network, an MSFF network, fully connected layers, and a softmax layer. The target structural MRI images are input into the 3D ResNet18 network for feature extraction to obtain convolutional features. These convolutional features are then input into the 3D CLSTM network for processing to obtain cascaded features. The cascaded features are then input into the MSFF network for feature fusion to obtain fused features. Finally, the fused features are classified through fully connected layers and a softmax layer to obtain the classification result. This invention integrates the spatial and longitudinal features of the image for image classification. The model retains the three-dimensional spatial information of the features and incorporates the temporal information of the data, thus improving the accuracy of the structural MRI classification results. Attached Figure Description
[0042] Figure 1 This is a flowchart of a preferred embodiment of the structural magnetic resonance imaging classification method based on multiple time points of the present invention;
[0043] Figure 2 This is a diagram of the overall architecture of the structural magnetic resonance imaging classification model in the multi-time-point structural magnetic resonance imaging classification method of this invention.
[0044] Figure 3 This is an architecture diagram of the 3D CLSTM network in the multi-time-point structural magnetic resonance imaging classification method of this invention;
[0045] Figure 4This is an architecture diagram of the MSFF network in the multi-time-point structural magnetic resonance imaging classification method of the present invention;
[0046] Figure 5 This is a flowchart of the process of inputting cascaded features into an MSFF network for feature fusion in the multi-time-point structural magnetic resonance imaging classification method of this invention;
[0047] Figure 6 This is a schematic diagram of a preferred embodiment of the structural magnetic resonance imaging classification system based on multiple time points of the present invention;
[0048] Figure 7 This is a schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. Detailed Implementation
[0049] This application provides a structural magnetic resonance imaging classification method and related equipment based on multiple time points. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.
[0050] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0051] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.
[0052] The preferred embodiment of the structural magnetic resonance imaging classification method based on multiple time points described in this invention, such as... Figure 1 As shown, the multi-time-point structural magnetic resonance imaging classification method includes the following steps:
[0053] Step S10: Obtain magnetic resonance imaging of the target structure.
[0054] Structural magnetic resonance imaging (SMRI) is a non-invasive imaging technique used to visualize the structural anatomy of the brain. It provides information about the brain's morphology, size, compartments, and tissue structure, making it crucial for studying brain development and anatomical variations. Structural MRI utilizes the interaction between magnetic fields and radio waves to acquire high-resolution images of the brain by scanning the head. These images can be used to detect abnormal structures.
[0055] Specifically, magnetic resonance imaging (MRI) technology is used to acquire target structural MRI images of the target user to be detected. The obtained target structural MRI images are then preprocessed, such as AC-PC correction and skull removal. The processed results are then segmented and gray matter extracted to obtain a gray matter image.
[0056] Step S20: Construct a structural magnetic resonance imaging classification model, train and test the structural magnetic resonance imaging classification model to obtain a target model, wherein the target model includes: a 3D ResNet18 network, a 3D CLSTM network, an MSFF network, a fully connected layer and a softmax layer.
[0057] The process of training and testing the structural magnetic resonance imaging classification model to obtain the target model specifically includes:
[0058] Historical structural magnetic resonance imaging (MRI) images of multiple individuals were acquired, and AC-PC correction and skull removal were performed on the multiple historical structural MRI images to obtain multiple mesobrain images.
[0059] Each intermediate brain image is segmented and gray matter is extracted to obtain multiple gray matter images. The multiple gray matter images are adjusted to a uniform size to obtain a dataset, wherein the dataset includes gray matter images of each individual at three different time points.
[0060] Understandably, in order to preprocess the training data, both Statistical Parametric Mapping (SPM) and Computational Anatomy Toolbox (CAT) were used. CAT was used to preprocess the sMRI data as follows: First, AC-PC correction and skull removal were performed on the historical structural MRI images of multiple individuals to obtain multiple intermediate brain images. Then, each brain image was segmented to extract gray matter. The gray matter images were used for classification, and the multiple gray matter images were adjusted to a uniform size, such as 128×128×128 pixels.
[0061] The dataset is divided into a training set, a validation set, and a test set according to a preset ratio. The training set is used to train the structural magnetic resonance imaging (SMRI) classification model. The validation set is used to verify the performance and adjust the strategy of the trained SMRI classification model. The test set is used to evaluate the performance of the trained SMRI classification model, thereby obtaining the target model that meets the preset requirements.
[0062] Understandably, ADAM (Adaptive Moment Estimation) can be chosen as the optimization method during network training. The data is divided into training, validation, and test sets in a 9:1:1 ratio, and then 10-fold cross-validation is used to stabilize the results. Each fold is trained for 60 epochs, with an initial learning rate of 1e-4 and a momentum of 0.99. Accuracy, sensitivity, precision, and F1 score are used as performance metrics to evaluate the model for the classification task.
[0063] Furthermore, such as Figure 2 As shown, this invention constructs a structural magnetic resonance imaging (SMRI) classification model. In this model, the 3D ResNet18 network is a parameter-sharing network. During training, the input data to the 3D ResNet18 network consists of grayscale images at three different time points. At each time point, the 3D ResNet18 uses the same weights to extract longitudinal feature information. The deep architecture of 3DCNN can explicitly utilize the spatial structure of brain images, gradually extracting spatial features useful for classification tasks from low, medium, and high levels.
[0064] The 3D CLSTM network (3D Convolutional Long Short Memory Network) is used to concatenate the convolutional features of the second-to-last and first-to-last layers of the 3DResNet18 network to obtain concatenated features; the MSFF network (Multi-Scale Feature Fusion Network) is used to fuse the concatenated features obtained by the 3D CLSTM network to obtain fused features.
[0065] Step S30: Input the magnetic resonance imaging of the target structure into the 3D ResNet18 network for feature extraction to obtain convolutional features, input the convolutional features into the 3DCLSTM network for processing to obtain cascaded features, and input the cascaded features into the MSFF network for feature fusion to obtain fused features.
[0066] The step of inputting the magnetic resonance imaging of the target structure into the 3D ResNet18 network for feature extraction to obtain convolutional features specifically includes:
[0067] The target structure magnetic resonance imaging is preprocessed to obtain the target gray matter image of the target structure magnetic resonance imaging, and the gray matter image is three-dimensional data;
[0068] The grayscale image is input into the convolutional layer of the 3D ResNet18 network to extract vertical features with the same weights, thereby obtaining convolutional features, which are three-dimensional data.
[0069] The convolutional features include a first convolutional feature X1 extracted from the penultimate layer and a second convolutional feature X2 extracted from the penultimate layer.
[0070] Understandably, to address the computational resource and storage space issues in feature extraction, this invention introduces a parameter-sharing strategy into the model framework. This addresses the challenge of feature extraction for independent branch models at each time point. A parameter-sharing 3D ResNet18 network is introduced throughout the model. This means that data from each time point undergoes feature extraction through the same network structure, significantly reducing the demand for computational and storage resources. This strategy not only improves efficiency but also ensures the consistency of feature extraction, allowing data from different time points to be compared within the same feature space.
[0071] Furthermore, such as Figure 3 As shown, the 3D CLSTM network includes two improved LSTM models, a batch normalization module, and a modified linear unit activation function;
[0072] The improved LSTM model uses 3D convolution operations to replace the multilayer perceptron transform of the original LSTM model.
[0073] The batch normalization module and the modified linear unit activation function are used to enhance the training stability and convergence speed of the model.
[0074] Specifically, this invention introduces a nonlinear convolutional transform into the existing LSTM model to replace the linear transformation of the input data. This modification is made while maintaining the LSTM framework, and the equations are as follows:
[0075] i t =σ(W xi *X t +W hi *H t-1 +b i );
[0076] f t =σ(W xf *X t +W hf*H t-1 +b f );
[0077]
[0078]
[0079] O t =σ(W xo *X t +W ho *H t-1 +b o );
[0080] h t =o t *tanh(C t );
[0081] Among them, i t f t , C t O t and h t These represent the input gate, forget gate, candidate memory cell, memory cell, output gate, and hidden layer, respectively. X t and H t-1 Let W represent the input data and the hidden state at time step t-1, respectively. σ and tanh represent the sigmoid and hyperbolic tangent functions, respectively. xi W hi W xf W hf W xc W hc and W xo This indicates the 3D convolution kernel gated by the corresponding subscript, b i b f b c and b o It is the bias vector corresponding to the subscript gating. It represents the Hadamardi (or Hadama) stack.
[0082] Understandably, traditional LSTM models were originally designed to process one-dimensional feature vectors, performing matrix multiplication on stacked feature vectors using parameters W. However, this invention aims to preserve the three-dimensional structure of features so that the model can learn the spatiotemporal characteristics of the data and capture the spatial and temporal trends of feature changes. To achieve this, 3D convolution operations are used instead of traditional multilayer perceptron transforms on top of the traditional LSTM. 3D convolution operations allow features to undergo sliding window operations across the entire three-dimensional space, thus preserving the internal structure and spatial relationships of the data. This approach enhances the model's ability to capture complex dependencies, showing significant effectiveness, particularly when processing data with spatiotemporal dimensions. Furthermore, this invention introduces batch normalization and modified linear unit activation functions in subsequent layers of the model to enhance training stability and convergence speed.
[0083] These enhancements collectively ensure that the 3D CLSTM network can process 3D data more effectively and retain spatial information to a greater extent. Therefore, the model exhibits better performance and generalization ability, making it particularly suitable for tasks that require consideration of spatiotemporal relationships. The innovation of this framework lies in extending the traditional LSTM model to the 3D domain, enabling it to better handle data with spatiotemporal characteristics.
[0084] Furthermore, the step of inputting the convolutional features into the 3D CLSTM network for processing to obtain cascaded features specifically includes:
[0085] The first convolutional feature X1 is input into an improved LSTM model for concatenation to obtain the first concatenated feature F1;
[0086] The second convolutional feature X2 is input into another improved LSTM model for concatenation to obtain the second concatenated feature F2.
[0087] The scales of the first cascaded feature F1 and the second cascaded feature F2 are different.
[0088] Furthermore, such as Figure 4 As shown, the MSFF network includes: a first convolutional kernel, a second convolutional kernel, a global background module, a local background module, and a sigmoid activation function.
[0089] The purpose of constructing the MSFF network is to combine feature information from different scales. The lightweight network can reduce the number of model parameters to prevent overfitting and reduce training time.
[0090] like Figure 4 and 5As shown, the step of inputting the cascaded features into the MSFF network for feature fusion to obtain fused features specifically includes:
[0091] Step S301: Input the first concatenated feature F1 into the first convolution kernel to perform a convolution operation to obtain the first target convolution feature, and input the second concatenated feature F2 into the second convolution kernel to perform a convolution operation to obtain the second target convolution feature.
[0092] It is understood that, in this embodiment, in order to ensure that the first concatenated feature F1 and the second concatenated feature F2 have the same shape, a convolution operation is performed on the first concatenated feature F1 and the second concatenated feature F2 to obtain the first target convolution feature and the second target convolution feature.
[0093] Step S302: Add the first target convolutional feature and the second target convolutional feature element by element to obtain the summed target convolutional feature.
[0094] Step S303: Input the summed target convolutional features into the global background module for global average pooling and pointwise convolution to obtain the first low-channel-number convolutional features. Perform a nonlinear transformation on the first low-channel-number convolutional features through the ReLU activation function. Perform pointwise convolution again on the result of the nonlinear transformation to obtain the global features with restored channel number.
[0095] It is understandable that the process of inputting the summed target convolutional features into the global background module for global context operation is as follows: first, global average pooling is performed on the feature information with feature channels C; then, the number of channels is changed to C / r through pointwise convolution; after that, the number of feature channels is changed back to C through nonlinear transformation by the ReLU activation function and then through pointwise convolution.
[0096] Step S304: Input the summed target convolutional features into the local background module for point-by-point convolution to obtain the second low-channel-number convolutional features. Perform a nonlinear transformation on the second low-channel-number convolutional features through the ReLU activation function. Perform point-by-point convolution on the result of the nonlinear transformation again to obtain the local features with restored channel number.
[0097] The process of inputting the summed target convolutional features into the local background module for local context operation is similar to the global context operation described above, except that the initial global average pooling operation is omitted, and will not be repeated here.
[0098] Step S305: Add the global features and the local features to obtain multi-scale channel features. Perform a nonlinear transformation on the multi-scale channel features using the sigmoid activation function to obtain target multi-scale channel features. Multiply the target multi-scale channel features element-wise with the first target convolutional feature and the second target convolutional feature to obtain target global features and target local features.
[0099] In other words, the features obtained after global and local context operations are summed to obtain channel features across multiple scales. Then, the sigmoid activation function is applied to the obtained channel features, and then element-wise multiplied with the first target convolutional feature and the second target convolutional feature obtained by convolution to obtain the target global feature and the target local feature.
[0100] Step S306: Add the target global features and the target local features element by element, and perform global average pooling on the result of the element-by-element addition to obtain the fused features.
[0101] It is worth noting that the design of this invention focuses on creating a lightweight network to prevent overfitting and reduce training time. This is particularly evident in the structure of the global and local context components. Therefore, pointwise convolutions are used, and the number of channels is reduced from C to C / r (where C is the number of channels and r is the reduction ratio) through two consecutive convolutional layers, and finally the number of channels is restored to C. In this way, the number of model channels and convolutional kernel parameters are reduced.
[0102] Furthermore, this invention introduces residual connections, using global average pooling to generate a one-dimensional feature vector from the features of F1 and F2. These three one-dimensional vectors are then concatenated as the output of the MSFF module. This method enhances the model's ability to effectively capture features at multiple scales.
[0103] Step S40: Classify the fused features through the fully connected layer and the softmax layer to obtain the classification result.
[0104] Fully connected layers and softmax layers are often used together. Fully connected layers provide the raw output of the model, while softmax layers transform it into a probability distribution, allowing the model to output the predicted probability for each class, and are used for training and prediction.
[0105] As can be seen, this invention employs a three-layer 3D CLSTM method, operating at three different time points. These outputs are fed into an MSFF module for feature fusion, then processed through fully connected layers, and finally classified using softmax. This entire process enhances the model's ability to capture feature information at multiple scales, thereby improving classification performance.
[0106] In summary, this invention provides a multi-time-point structural magnetic resonance imaging (SMR) classification method that combines traditional convolutional neural networks with 3D CLSTM methods to simultaneously learn temporal and spatial feature information and integrate multi-scale information for studying image classification tasks using structural magnetic resonance imaging. Extensive experiments and verifications have demonstrated that the method achieves excellent results. The main advantages of this invention are as follows:
[0107] This invention differs from most current methods that only use baseline data for classification. It utilizes temporal information from longitudinal data to better classify structural magnetic resonance imaging (MRI). Secondly, this invention designs a 3D CLSTM network and combines it with a CNN network to preserve the spatial information of features, enabling it to capture temporal and spatial feature transformations and enhancing the classification capability of longitudinal data. This invention also introduces a multi-scale approach to enhance the model's ability to perceive features at different scales and levels. In summary, this invention integrates the spatial and longitudinal features of images for image classification. The model preserves the three-dimensional spatial information of features and incorporates the temporal information of the data, improving the accuracy of structural MRI classification results.
[0108] Furthermore, such as Figure 6 As shown, based on the above-described multi-time-point structural magnetic resonance imaging classification method, the present invention also provides a multi-time-point structural magnetic resonance imaging classification system, wherein the multi-time-point structural magnetic resonance imaging classification system includes:
[0109] The target image acquisition module 61 is used to acquire magnetic resonance imaging of the target structure.
[0110] The model building and training module 62 is used to build a structural magnetic resonance imaging classification model, train and test the structural magnetic resonance imaging classification model to obtain a target model, wherein the target model includes: a 3D ResNet18 network, a 3D CLSTM network, an MSFF network, a fully connected layer and a softmax layer;
[0111] The feature extraction and fusion module 63 is used to input the magnetic resonance imaging of the target structure into the 3D ResNet18 network for feature extraction to obtain convolutional features, input the convolutional features into the 3D CLSTM network for processing to obtain cascaded features, and input the cascaded features into the MSFF network for feature fusion to obtain fused features.
[0112] The classification prediction module 64 is used to classify the fused features through the fully connected layer and the softmax layer to obtain the classification result.
[0113] Furthermore, such as Figure 7 As shown, based on the above-mentioned multi-time-point structural magnetic resonance imaging classification method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 7 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0114] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a multi-time-point structural magnetic resonance imaging (SMR) classification program 40, which can be executed by the processor 10 to implement the multi-time-point SMR classification method of this application.
[0115] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the multi-time-point structural magnetic resonance imaging classification method.
[0116] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.
[0117] In one embodiment, when the processor 10 executes the multi-time-point-based structural magnetic resonance imaging classification program 40 in the memory 20, the following steps are performed:
[0118] Acquire magnetic resonance imaging of the target structure;
[0119] A structural magnetic resonance imaging (SMR) classification model is constructed, and the SMR classification model is trained and tested to obtain a target model. The target model includes: a 3DResNet18 network, a 3D CLSTM network, an MSFF network, a fully connected layer, and a softmax layer.
[0120] The magnetic resonance imaging of the target structure is input into the 3D ResNet18 network for feature extraction to obtain convolutional features. The convolutional features are then input into the 3D CLSTM network for processing to obtain cascaded features. Finally, the cascaded features are input into the MSFF network for feature fusion to obtain fused features.
[0121] The fused features are classified through the fully connected layer and the softmax layer to obtain the classification result.
[0122] The step of training and testing the structural magnetic resonance imaging classification model to obtain the target model specifically includes:
[0123] Acquire historical structural magnetic resonance images of multiple individuals, perform AC-PC correction and skull removal on the multiple historical structural magnetic resonance images to obtain multiple intermediate brain images;
[0124] Each intermediate brain image is segmented and gray matter is extracted to obtain multiple gray matter images. The multiple gray matter images are adjusted to a uniform size to obtain a dataset, wherein the dataset includes gray matter images of each individual at three different time points.
[0125] The dataset is divided into a training set, a validation set, and a test set according to a preset ratio. The training set is used to train the structural magnetic resonance imaging (SMRI) classification model. The validation set is used to verify the performance and adjust the strategy of the trained SMRI classification model. The test set is used to evaluate the performance of the trained SMRI classification model, thereby obtaining the target model that meets the preset requirements.
[0126] Specifically, the step of inputting the magnetic resonance imaging of the target structure into the 3D ResNet18 network for feature extraction to obtain convolutional features includes:
[0127] The target structure magnetic resonance imaging is preprocessed to obtain the target gray matter image of the target structure magnetic resonance imaging, and the gray matter image is three-dimensional data;
[0128] The grayscale image is input into the convolutional layer of the 3D ResNet18 network to extract vertical features with the same weights, thereby obtaining convolutional features, which are three-dimensional data.
[0129] The convolutional features include a first convolutional feature X1 extracted from the penultimate layer and a second convolutional feature X2 extracted from the penultimate layer.
[0130] The 3D CLSTM network includes two improved LSTM models, a batch normalization module, and a modified linear unit activation function.
[0131] The improved LSTM model uses 3D convolution operations to replace the multilayer perceptron transform of the original LSTM model.
[0132] The batch normalization module and the modified linear unit activation function are used to enhance the training stability and convergence speed of the model.
[0133] Specifically, the step of inputting the convolutional features into the 3D CLSTM network for processing to obtain cascaded features includes:
[0134] The first convolutional feature X1 is input into an improved LSTM model for concatenation to obtain the first concatenated feature F1;
[0135] The second convolutional feature X2 is input into another improved LSTM model for concatenation to obtain the second concatenated feature F2.
[0136] The scales of the first cascaded feature F1 and the second cascaded feature F2 are different.
[0137] The MSFF network includes: a first convolutional kernel, a second convolutional kernel, a global background module, a local background module, and a sigmoid activation function.
[0138] Specifically, the step of inputting the cascaded features into the MSFF network for feature fusion to obtain fused features includes:
[0139] The first concatenated feature F1 is input into the first convolution kernel for convolution operation to obtain the first target convolution feature, and the second concatenated feature F2 is input into the second convolution kernel for convolution operation to obtain the second target convolution feature.
[0140] The first target convolutional feature and the second target convolutional feature are added element by element to obtain the summed target convolutional feature;
[0141] The summed target convolutional features are input into the global background module for global average pooling and pointwise convolution to obtain the first low-channel-number convolutional features. The first low-channel-number convolutional features are then subjected to a non-linear transformation using the ReLU activation function. The result of the non-linear transformation is then subjected to pointwise convolution again to obtain the global features that restore the number of channels.
[0142] The summed target convolutional features are input into the local background module for point-by-point convolution to obtain the second low-channel-number convolutional features. The second low-channel-number convolutional features are then subjected to a non-linear transformation using the ReLU activation function. The result of the non-linear transformation is then subjected to point-by-point convolution again to obtain the local features with restored channel numbers.
[0143] The global features and the local features are added together to obtain multi-scale channel features. The multi-scale channel features are then subjected to a non-linear transformation using the sigmoid activation function to obtain target multi-scale channel features. The target multi-scale channel features are then multiplied element-wise with the first target convolutional feature and the second target convolutional feature to obtain target global features and target local features.
[0144] The target global features and the target local features are added element-wise, and the result of the element-wise addition is subjected to global average pooling to obtain the fused features.
[0145] In summary, this invention provides a multi-time-point structural magnetic resonance imaging (SMRI) classification method, system, and terminal. The method includes: acquiring target structural magnetic resonance images; constructing a structural magnetic resonance imaging classification model; training and testing the SMRI classification model to obtain a target model, wherein the target model includes: a 3DResNet18 network, a 3D CLSTM network, an MSFF network, a fully connected layer, and a softmax layer; inputting the target structural magnetic resonance image into the 3D ResNet18 network for feature extraction to obtain convolutional features; inputting the convolutional features into the 3D CLSTM network for processing to obtain cascaded features; inputting the cascaded features into the MSFF network for feature fusion to obtain fused features; and classifying the fused features through the fully connected layer and the softmax layer to obtain a classification result. This invention integrates the spatial and longitudinal features of the image for image classification. The model retains the three-dimensional spatial information of the features and combines the temporal information of the data, improving the accuracy of the structural magnetic resonance imaging classification results.
[0146] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.
[0147] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0148] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A structural magnetic resonance imaging classification method based on multiple time points, characterized in that, The multi-time-point-based structural magnetic resonance imaging classification method includes: Acquire magnetic resonance imaging of the target structure; A structural magnetic resonance imaging (SMR) classification model is constructed, and the SMR classification model is trained and tested to obtain a target model. The target model includes: a 3DResNet18 network, a 3D CLSTM network, an MSFF network, a fully connected layer, and a softmax layer. The magnetic resonance imaging of the target structure is input into the 3D ResNet18 network for feature extraction to obtain convolutional features. The convolutional features are then input into the 3D CLSTM network for processing to obtain cascaded features. Finally, the cascaded features are input into the MSFF network for feature fusion to obtain fused features. The fused features are classified through the fully connected layer and the softmax layer to obtain the classification result.
2. The structural magnetic resonance imaging classification method based on multiple time points according to claim 1, characterized in that, The process of training and testing the structural magnetic resonance imaging classification model to obtain the target model specifically includes: Acquire historical structural magnetic resonance images of multiple individuals, perform AC-PC correction and skull removal on the multiple historical structural magnetic resonance images to obtain multiple intermediate brain images; Each intermediate brain image is segmented and gray matter is extracted to obtain multiple gray matter images. The multiple gray matter images are adjusted to a uniform size to obtain a dataset, wherein the dataset includes gray matter images of each individual at three different time points. The dataset is divided into a training set, a validation set, and a test set according to a preset ratio. The training set is used to train the structural magnetic resonance imaging (SMRI) classification model. The validation set is used to verify the performance and adjust the strategy of the trained SMRI classification model. The test set is used to evaluate the performance of the trained SMRI classification model, thereby obtaining the target model that meets the preset requirements.
3. The structural magnetic resonance imaging classification method based on multiple time points according to claim 1, characterized in that, The step of inputting the magnetic resonance imaging of the target structure into the 3DResNet18 network for feature extraction to obtain convolutional features specifically includes: The target structure magnetic resonance imaging is preprocessed to obtain the target gray matter image of the target structure magnetic resonance imaging, and the gray matter image is three-dimensional data; The grayscale image is input into the convolutional layer of the 3D ResNet18 network to extract vertical features with the same weights, thereby obtaining convolutional features, which are three-dimensional data. The convolutional features include a first convolutional feature X1 extracted from the penultimate layer and a second convolutional feature X2 extracted from the penultimate layer.
4. The structural magnetic resonance imaging classification method based on multiple time points according to claim 3, characterized in that, The 3D CLSTM network includes two improved LSTM models, a batch normalization module, and a modified linear unit activation function. The improved LSTM model uses 3D convolution operations to replace the multilayer perceptron transform of the original LSTM model. The batch normalization module and the modified linear unit activation function are used to enhance the training stability and convergence speed of the model.
5. The structural magnetic resonance imaging classification method based on multiple time points according to claim 4, characterized in that, The step of inputting the convolutional features into the 3D CLSTM network for processing to obtain cascaded features specifically includes: The first convolutional feature X1 is input into an improved LSTM model for concatenation to obtain the first concatenated feature F1; The second convolutional feature X2 is input into another improved LSTM model for concatenation to obtain the second concatenated feature F2. The scales of the first cascaded feature F1 and the second cascaded feature F2 are different.
6. The structural magnetic resonance imaging classification method based on multiple time points according to claim 5, characterized in that, The MSFF network includes: a first convolutional kernel, a second convolutional kernel, a global background module, a local background module, and a sigmoid activation function.
7. The structural magnetic resonance imaging classification method based on multiple time points according to claim 6, characterized in that, The step of inputting the cascaded features into the MSFF network for feature fusion to obtain fused features specifically includes: The first concatenated feature F1 is input into the first convolution kernel for convolution operation to obtain the first target convolution feature, and the second concatenated feature F2 is input into the second convolution kernel for convolution operation to obtain the second target convolution feature. The first target convolutional feature and the second target convolutional feature are added element by element to obtain the summed target convolutional feature; The summed target convolutional features are input into the global background module for global average pooling and pointwise convolution to obtain the first low-channel-number convolutional features. The first low-channel-number convolutional features are then subjected to a non-linear transformation using the ReLU activation function. The result of the non-linear transformation is then subjected to pointwise convolution again to obtain the global features that restore the number of channels. The summed target convolutional features are input into the local background module for point-by-point convolution to obtain the second low-channel-number convolutional features. The second low-channel-number convolutional features are then subjected to a non-linear transformation using the ReLU activation function. The result of the non-linear transformation is then subjected to point-by-point convolution again to obtain the local features with restored channel numbers. The global features and the local features are added together to obtain multi-scale channel features. The multi-scale channel features are then subjected to a non-linear transformation using the sigmoid activation function to obtain target multi-scale channel features. The target multi-scale channel features are then multiplied element-wise with the first target convolutional feature and the second target convolutional feature to obtain target global features and target local features. The target global features and the target local features are added element-wise, and the result of the element-wise addition is subjected to global average pooling to obtain the fused features.
8. A structural magnetic resonance imaging classification system based on multiple time points, characterized in that, The multi-time-point-based structural magnetic resonance imaging classification system includes: The target image acquisition module is used to acquire magnetic resonance imaging of the target structure. The model building and training module is used to build a structural magnetic resonance imaging (SMRI) classification model, train and test the SMRI classification model to obtain a target model, wherein the target model includes: a 3DResNet18 network, a 3D CLSTM network, an MSFF network, a fully connected layer and a softmax layer; The feature extraction and fusion module is used to input the magnetic resonance imaging of the target structure into the 3DResNet18 network for feature extraction to obtain convolutional features, input the convolutional features into the 3D CLSTM network for processing to obtain cascaded features, and input the cascaded features into the MSFF network for feature fusion to obtain fused features. The classification prediction module is used to classify the fused features through the fully connected layer and the softmax layer to obtain the classification result.
9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a multi-time-point-based structural magnetic resonance imaging classification program stored in the memory and executable on the processor. When the multi-time-point-based structural magnetic resonance imaging classification program is executed by the processor, it implements the steps of the multi-time-point-based structural magnetic resonance imaging classification method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-time-point-based structural magnetic resonance imaging (SMR) classification program, which, when executed by a processor, implements the steps of the multi-time-point-based SMR classification method as described in any one of claims 1-7.