Longitudinal brain grey matter image classification method based on cyclic vision Transform

Abstract

The invention relates to a longitudinal brain grey matter image classification method based on cyclic vision Transform, and belongs to the field of computer vision. According to the spatial-temporal feature modeling method of the longitudinal grey matter image based on the cyclic vision Transformer (RViT), firstly, spatial features in grey matter images at different time points are mined through a shared ResNet, and then global spatial-temporal dependence in the grey matter images at different time points is fused through a shared RViT module, so that the model training time is shortened, and the model training efficiency is improved. And the global space-time relevance in the longitudinal data is mined, so that the accurate classification of the longitudinal grey matter image is realized.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to a longitudinal brain gray matter image classification method based on a cyclic visual Transformer. Background Art

[0002] Neural network-based methods show great advantages in gray matter image classification. Most existing research methods focus on using image data at a single time point to model the spatial features of gray matter images to classify gray matter images. However, it is difficult to model the dynamic changes of gray matter images over time in subjects using only gray matter image data at a single time point. Therefore, how to construct the global spatiotemporal features of longitudinal brain gray matter image data to achieve gray matter image classification is a challenging task.

[0003] At present, there are mainly two methods for classifying longitudinal gray matter images based on the dynamic changes of spatiotemporal features. One method is to use expensive 4D convolution to capture the spatiotemporal features of longitudinal gray matter images. Although this method can make good use of the spatiotemporal features in longitudinal gray matter images, the training process is very time-consuming and requires large computing resources. Another method is to use independent feature extraction and classification models for diagnosis, but the extracted features cannot capture the complete spatiotemporal features that can reflect brain changes, resulting in limited image classification accuracy. Moreover, both methods can only use gray matter images of the same number of time points for classification. If the longitudinal gray matter images of some subjects are lost at certain time points, it will lead to the inability to extract consistent features for longitudinal analysis. Therefore, it is necessary to design an intelligent classification method that can mine the complete spatiotemporal features in longitudinal gray matter images and is applicable to gray matter image data at any time point. Summary of the invention

[0004] The technical problems to be solved by the present invention are:

[0005] In order to solve the problem that the existing methods cannot model the complete spatiotemporal features in the longitudinal gray matter image based on the gray matter image data at any time point and realize the classification of the gray matter image, the present invention provides a longitudinal brain gray matter image classification method based on recurrent visual Transformer.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A longitudinal brain gray matter image classification method based on recurrent visual Transformer, characterized by comprising:

[0008] Acquire structural MRI images at different time points, preprocess the structural MRI images at different time points to obtain longitudinal gray matter images, and divide the longitudinal gray matter images into a training set, a validation set, and a test set;

[0009] A longitudinal brain gray matter image classification model based on RViT is constructed, wherein the longitudinal brain gray matter image classification model based on RViT is composed of ResNet-18 and RViT modules, wherein the depth of the ResNet-18 network is 18 layers, and its backbone network includes 4 stages, each of which includes 2 residual modules; before the 2 residual modules of the fourth stage of the backbone network of the ResNet-18 model, the RViT module is used to aggregate the temporal features of gray matter images at different time points, so that the gray matter image features at the current time point are integrated with the temporal information of gray matter images at other time points;

[0010] The training set is input into the RViT-based longitudinal gray matter image classification model for supervised training, and the validation set is used to verify the quality of the model training and adjust the model's hyperparameters to find the optimal experimental hyperparameters of the model; the model with the optimal parameters is tested on the test set to obtain the effect of the model in classifying longitudinal gray matter images.

[0011] A further technical solution of the present invention: the pretreatment comprises:

[0012] Image correction is performed by the positions of the anterior commissure and the posterior commissure in the sagittal image, and the origin coordinates of the structural magnetic resonance images at different time points are adjusted;

[0013] Perform spatial normalization, skull stripping and image segmentation on the rectified image;

[0014] The segmented gray matter image is corrected for bias, the gray matter image is normalized to the MNI standard space, the inhomogeneity and noise in the gray matter image are removed, and a modulated standardized gray matter image is obtained;

[0015] Resample the normalized gray matter images.

[0016] A further technical solution of the present invention is as follows: the RViT module includes a convolution module, a multi-axis attention module and an LSTM unit, wherein the convolution module is used to combine the prior knowledge of the longitudinal structural magnetic resonance image and endow the model with the ability of conditional position embedding, so that the model does not need to perform absolute or relative position embedding; the multi-axis attention module is used to mine the spatial features in the data; the LSTM unit is used to aggregate the time information in the gray matter image data at different time points and extract the global spatiotemporal features in the longitudinal gray matter image data.

[0017] A further technical solution of the present invention is as follows: the multi-axis attention module comprises a local attention layer and a grid attention layer, the local attention layer is used to extract local features, and the grid attention layer is used to extract global features.

[0018] A further technical solution of the present invention is as follows: the residual module is composed of two 3×3 convolutional layers with the same number of convolution kernels, each convolutional layer is followed by a BN layer and a ReLU activation function, and the original input skips the convolutional layer and is connected to the output of the convolutional layer before passing through the ReLU activation function.

[0019] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0020] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.

[0021] A computer program product, characterized by comprising computer executable instructions, wherein the instructions are used to implement the above method when executed.

[0022] The beneficial effects of the present invention are:

[0023] The present invention provides a longitudinal gray matter image classification method based on recurrent visual transformer (RViT), which successfully implements the longitudinal gray matter image classification method based on RViT, and has the following advantages in the field of brain image classification:

[0024] First, compared with the existing gray matter image classification model, the longitudinal brain gray matter image classification method based on RViT proposed in the present invention uses the shared ResNet-18 as the backbone network of the model to extract spatial features in gray matter image data at different time points, and uses RViT as a time aggregation module, which not only speeds up the training speed of the model, but also can realize accurate diagnosis of gray matter images based on the capture of complete spatiotemporal features related to brain abnormalities. And because the RViT module is used as a time aggregation module, the gray matter image at any time point can be predicted, overcoming the problem that the longitudinal gray matter image of the subject is lost at certain time points, resulting in the inability to extract consistent features for longitudinal analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0026] Figure 1 Schematic diagram of the framework of the method of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] like Figure 1 The figure shows a schematic diagram of the framework of the method of the present invention. Module (a) represents the longitudinal gray matter image data of a subject, wherein T0, T1, and T2 represent the gray matter image data obtained after preprocessing at three time points of the subject. Module (b) represents the RViT module, wherein MLP represents a multi-layer perceptron, which together with the previous attention module constitutes a basic attention layer, and c and h represent the state of the current unit and the state of the hidden layer, respectively. The model is endowed with the ability of conditional position embedding through a 3×3 convolutional layer, and the spatial features in the data are mined through a multi-axis attention mechanism. The multi-axis attention mechanism consists of two stages: local attention and grid attention. Finally, LSTM is used to aggregate the time information in the gray matter image data at different time points, so that the model can extract the global spatiotemporal features in the longitudinal gray matter image data. Module (c) represents a longitudinal brain gray matter image classification model based on RViT, which consists of ResNet-18 and RViT modules. The ResNet-18 backbone network contains 4 stages, each of which consists of two residual modules.

[0030] The present invention extracts the spatial features of gray matter images at different time points through a shared ResNet-18, and aggregates the temporal features of gray matter images at different time points through RViT before the two residual modules in the fourth stage of the ResNet-18 model backbone network, so that the gray matter image features at the current time point are integrated with the temporal information of gray matter images at other time points, thereby realizing gray matter image classification based on the global spatiotemporal features of gray matter images at different time points.

[0031] The present invention provides a longitudinal brain gray matter image classification method based on RViT, comprising the following steps:

[0032] Step 1: Data preprocessing. The gray matter image used in the present invention is obtained by preprocessing the structural magnetic resonance imaging data (sMRI). In order to improve the quality of the gray matter image data collected from the subjects, reduce the interference of noise and artifacts, and provide a reliable basis for subsequent analysis, it is necessary to use SPM software to preprocess the sMRI data. First, the image correction is performed by the position of the anterior commissure and the posterior commissure in the sagittal image, and the origin coordinates of the structural magnetic resonance image are adjusted. Then, the image is spatially normalized, the skull is stripped, and the image is segmented. Then, the segmented gray matter image is corrected for bias, and the gray matter image is normalized to the MNI standard space. The inhomogeneity and noise in the gray matter image are removed to obtain a modulated standardized gray matter image. Finally, the standardized gray matter image is resampled to obtain a gray matter image with an image size of 64*64*64. Finally, each patient obtains a longitudinal gray matter image with a dimension of (t, 1, 64, 64, 64). Where t means that the patient contains gray matter image data at t different time points, 1 is the Channel dimension, and then the sample data is divided into training set, validation set and test set.

[0033] Step 2: Build an RViT module. An RViT module consists of three modules: a convolution module, a multi-axis attention module, and an LSTM unit. The convolution module is used to combine the knowledge priors of the longitudinal sMRI data and give the model the ability to conditionally embed positions, so that the model does not need to perform absolute or relative position embedding. Then the spatial features in the data are mined through the multi-axis attention mechanism, which consists of two stages: local feature interaction and expanded global feature interaction. Specifically, these features are first locally grouped into non-overlapping windows: represents the input feature tensor, and then reshapes the tensor into shape Where P×P is the window size of multi-head attention, which is used to simulate local feature interactions. Next, in order to extract global features, a direct method is to apply self-attention on the entire feature map, but the number of features of global self-attention has quadratic complexity. Therefore, the present invention uses a grid attention mechanism to obtain global features. The grid attention uses a G×G uniform grid to divide the feature map into The resulting window size is Attention is then applied to these windows, which corresponds to sparse global fusion. The interaction of global spatial features is achieved through local and sparse global attention mechanisms, allowing the model to mine global spatial features in the data.

[0034] When self-attention is applied in visual tasks, the image is divided into blocks of equal size and represented as sequence data through position encoding, and the relationship between different parts of the image is focused on through the self-attention mechanism. The formula for self-attention is as follows:

[0035]

[0036] Among them, Q, K, and V represent the three groups of Query, Key, and Value obtained by three linear transformations of the input sequence data, and W q , W k , W v represents the weight matrix of the learnable linear transformation, d k Indicates the dimension of Key, QK T represents the dot product, Indicates the scaling factor. Specifically, Query is used to match with other image blocks and to calculate the association between the current image block and other image blocks. Key is used to match with Query and can be understood as the key information of the image block. Value represents the important features of the current image block.

[0037] Finally, the temporal correlation in the longitudinal sMRI data is mined through the LSTM unit to achieve temporal feature aggregation. Since the temporal and spatial feature aggregation can be completely separated, the present invention uses a common long short-term memory storage unit (LSTM) for temporal feature aggregation, so that the states of the LSTM will not affect each other, avoiding the use of expensive Conv-LSTM units, and greatly reducing the computational complexity and number of parameters. The addition and deletion operations of LSTM on information are mainly realized through gate structures called input gates, forget gates, and output gates.

[0038] Forget gate: a sigmoid function whose input is the output h of the previous unit t-1 and the unit input x t , the value obtained is between 0 and 1, which determines the degree to which the previous unit state is forgotten. The formula is:

[0039] f (t) =σ(w f ·[h (t-1) ,x t ]+b f )

[0040] Among them, f (t) represents the forget gate, w f and b f They represent the weight and bias of the forget gate respectively.

[0041] Input gate: controls the amount of new information that will enter the state. First, it passes through a sigmoid activation function, which determines the information that needs to be updated, followed by a tanh activation function to generate a new candidate vector g t , the new vector value is in the range of [0,1], which controls the amount of new information added. The update control of the state is achieved through the combination of these two parts. The formula is expressed as:

[0042]

[0043] Among them, i t represents the input gate, W i and W c represents the weight, b i and b c represents the bias, c t Represents the status value of the current unit.

[0044] Output gate: controls the output value. First, it passes through a sigmoid activation function, which determines which values ​​of the state unit can be output. Then, the value of the state unit is processed by the tanh function to obtain a value between [-1,1]. The obtained value is then multiplied by the output value of the previous sigmoid function to obtain the part to be output. The formula is expressed as:

[0045]

[0046] Among them, t represents the output gate, W o and b o Respectively represent the weight and bias of the output gate, h t Represents the hidden layer state.

[0047] Step 3: Build a longitudinal gray matter image classification model based on RViT. This model is composed of ResNet-18 and RViT modules. The shared ResNet-18 is used as the backbone network to mine the spatial features between data at different time points. The depth of the ResNet-18 network is 18 layers. Its backbone network contains 4 stages, and each stage contains 2 residual modules. The residual module consists of two 3×3 convolutional layers with the same number of convolution kernels. Each convolutional layer is followed by a BN layer and a ReLU activation function, and the original input skips the convolutional layer and is connected to the output of the convolutional layer before passing through the ReLU activation function.

[0048] Assume that the input of the residual module is x and the output is G(x). At this time, if only a certain layer of the network is referred to, it may cause the degradation of the network. Therefore, it is hoped that the performance of this layer is at least similar to that of its previous layer, that is, G(x) = x, that is, the identity mapping, which is the core idea of ​​ResNet. But in fact, it is difficult to fit a module with input x and output G(x) = x, so the identity mapping is set as part of the network, that is, the output after learning this module is F(x) + x. The task of the residual module is to fit G(x)-x based on the input x. When F(x) = 0, G(x) = x, which constitutes an identity mapping. The use of identity mapping is very useful in the back propagation of the network. It ensures that the problem of gradient disappearance will not occur in the residual module.

[0049] In order to allow the model to aggregate the temporal features of different time points, RViT is used before the two residual modules in the fourth stage of the ResNet-18 model backbone network to aggregate the temporal features of gray matter images at different time points, so that the gray matter image features at the current time point are integrated with the temporal information of gray matter images at other time points. Finally, the analysis and modeling of longitudinal gray matter are achieved through the global spatiotemporal features at different time points.

[0050] Step 4: Input the processed training set into the longitudinal gray matter image classification model based on RViT for supervised training, and use the validation set to verify the quality of the model training and adjust the model's hyperparameters to find the optimal experimental hyperparameters of the model. Test the optimal model on the test set to obtain the effect of the model in classifying longitudinal gray matter images.

[0051] The present invention proposes a spatiotemporal feature modeling method for longitudinal gray matter images based on recurrent visual Transformer (RViT). First, a shared ResNet is used to mine the spatial features in gray matter images at different time points. Then, a shared RViT module is used to fuse the global spatiotemporal dependencies in gray matter images at different time points. This not only speeds up the model training time, but also mines the global spatiotemporal correlation in the longitudinal data, thereby achieving accurate classification of longitudinal gray matter images.

[0052] In order to enable those skilled in the art to better understand the present invention, the present invention is described in detail below in conjunction with specific embodiments.

[0053] This embodiment adopts Python language and Pytorch framework. The operating environment is Ubuntu 14 (LINUX) system, pycharm2022 software, Nvidia GTX 4090 Ti GPU, 24G video memory, and cuda8.0 for GPU graphics card acceleration.

[0054] Step 1: Select a data set. The gray matter image data used in the present invention comes from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database and is obtained by preprocessing sMRI. The sMRI data of 200 subjects were collected, each subject contained sMRI data at three time points, a total of 600 sMRI data, of which 100 were cognitively normal control groups and 100 were gray matter image abnormality groups. The data set was divided into a training set, a validation set, and a test set, with 140 subjects in the training set, and 30 subjects in the validation set and the test set.

[0055] Step 2: Data preprocessing. The present invention uses SPM to preprocess sMRI data. First, image correction is performed through the position of the anterior commissure and the posterior commissure in the sagittal image, and the origin coordinates of the structural magnetic resonance image are adjusted. Then, the image is spatially normalized, skull stripped, and image segmented. The segmented gray matter image is then bias-corrected, and the gray matter image is normalized to the MNI standard space. The inhomogeneity and noise in the gray matter image are removed to obtain a modulated standardized gray matter image. Finally, the standardized gray matter image is resampled to obtain a gray matter image with an image size of 64*64*64. Finally, the dimension of the longitudinal gray matter image obtained for each patient is (t, 1, 64, 64, 64). Where t means that the patient contains gray matter image data at t different time points, and 1 is the Channel dimension.

[0056] Step 3: Model construction. First, build the RViT module. An RViT module consists of three modules: convolution module, multi-axis attention module and LSTM unit. Among them, the convolution module is used to combine the knowledge prior of longitudinal sMRI data and give the model the ability of conditional position embedding, so that the model does not need to perform absolute or relative position embedding. Then, the spatial features in the data are mined through the multi-axis attention mechanism. The multi-axis attention mechanism consists of two stages: local feature interaction and expanded global feature interaction. Finally, the LSTM unit is used to mine the temporal correlation in the longitudinal sMRI data, thereby realizing temporal feature aggregation.

[0057] Then, a longitudinal brain gray matter image classification method model based on RViT was constructed. The model is composed of ResNet-18 and RViT modules. The shared ResNet-18 is used as the backbone network to mine the spatial features between data at different time points. The depth of the ResNet-18 network is 18 layers, and its backbone network contains 4 stages, each of which contains 2 residual modules. In order to allow the model to aggregate the temporal features of gray matter images at different time points, RViT is used to aggregate the temporal features of gray matter images at different time points before the two residual modules in the fourth stage of the backbone network of the ResNet-18 model, so that the gray matter image features at the current time point are integrated with the temporal information of gray matter images at other time points. Finally, the analysis modeling of longitudinal gray matter images is realized through the global spatiotemporal features of gray matter images at different time points.

[0058] Step 3: Model training. The preprocessed training set is input into the RViT-based longitudinal gray matter image classification method model for training, and the validation set is used for hyperparameter tuning. The cross entropy loss function and five-fold cross validation are used to find the optimal model weights.

[0059] Step 4: Model application. The optimal model obtained in step 3 is tested in the test set to obtain the results of the model classification of gray matter images.

[0060] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A longitudinal gray matter image classification method based on recurrent visual Transformer, characterized in that: include: Acquire structural MRI images at different time points, preprocess the structural MRI images at different time points to obtain longitudinal gray matter images, and divide the longitudinal gray matter images into a training set, a validation set, and a test set; A longitudinal brain gray matter image classification model based on RViT is constructed, wherein the longitudinal brain gray matter image classification model based on RViT is composed of ResNet-18 and RViT modules, wherein the depth of the ResNet-18 network is 18 layers, and its backbone network includes 4 stages, each of which includes 2 residual modules; before the 2 residual modules of the fourth stage of the backbone network of the ResNet-18 model, the RViT module is used to aggregate the temporal features of gray matter images at different time points, so that the gray matter image features at the current time point are integrated with the temporal information of gray matter images at other time points; The training set is input into the RViT-based longitudinal gray matter image classification model for supervised training, and the validation set is used to verify the quality of the model training and adjust the model's hyperparameters to find the optimal experimental hyperparameters of the model; the model with the optimal parameters is tested on the test set to obtain the effect of the model in classifying longitudinal gray matter images.

2. According to claim 1, a longitudinal gray matter image classification method based on recurrent visual Transformer is characterized in that: The pre-processing comprises: Image correction is performed by the positions of the anterior commissure and the posterior commissure in the sagittal image, and the origin coordinates of the structural magnetic resonance images at different time points are adjusted; Perform spatial normalization, skull stripping and image segmentation on the rectified image; The segmented gray matter image is corrected for bias, the gray matter image is normalized to the MNI standard space, the inhomogeneity and noise in the gray matter image are removed, and a modulated standardized gray matter image is obtained; Resample the normalized gray matter images.

3. According to claim 1, a longitudinal gray matter image classification method based on recurrent visual Transformer is characterized in that: The RViT module includes a convolution module, a multi-axis attention module and an LSTM unit. The convolution module is used to combine the knowledge prior of the longitudinal structural magnetic resonance image and give the model the ability of conditional position embedding, so that the model does not need to perform absolute or relative position embedding; the multi-axis attention module is used to mine the spatial features in the data; The LSTM unit is used to aggregate the time information in the gray matter image data at different time points and extract the global spatiotemporal features in the longitudinal gray matter image data.

4. According to claim 3, a longitudinal gray matter image classification method based on recurrent visual Transformer is characterized in that: The multi-axis attention module includes a local attention layer and a grid attention layer, the local attention layer is used to extract local features, and the grid attention layer is used to extract global features.

5. According to claim 1, a longitudinal gray matter image classification method based on recurrent visual Transformer, characterized in that: The residual module consists of two 3×3 convolutional layers with the same number of convolution kernels. Each convolutional layer is followed by a BN layer and a ReLU activation function, and the original input skips the convolutional layer and is connected to the output of the convolutional layer before passing through the ReLU activation function.

6. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

7. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

8. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images