A brain lesion region positioning system and method for multi-modal images

Abstract

The present application relates to the field of medical image processing and the field of deep learning, and is a kind of brain lesion region positioning system and method for multi-modal image, the system comprises adaptive multi-modal patch growth and pruning module, multi-modal patch multi-level fusion module based on double attention mechanism and global convolution network classifier module, through adaptive multi-modal patch growth and pruning module, patch is adaptively selected and built in multi-modal image, to identify the patch position with high discriminability;Through a kind of multi-modal patch fusion module based on double attention mechanism, the potential representation form of multi-modal patch is obtained, the multi-level representation form of patch is combined, and they are fused and learned together based on data-driven mode;Through global convolution network classifier module as the final classification model, further obtain the deeper representation of data, extract more potential information.The system proposed in the present application can effectively identify the lesion area highly related to the disease, and can assist in diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing and deep learning, in particular to a brain lesion region positioning system and method for multi-modal images. BACKGROUND

[0002] At present, it is a widely concerned aspect for many researchers to realize stable and accurate diagnosis of AD and its prodromal stage based on multi-modal neuroimaging. The most commonly used feature extraction methods for existing neuroimaging data for AD diagnosis generally include four basic parts: 1) extraction based on brain imaging voxel features; 2) extraction based on cortical surface vertex features; 3) pre-selection based on regions of interest (ROI); 4) selection based on local patch regions. Specifically, the methods based on voxel and vertex features are used to represent the microstructure related to the disease, and such methods usually have a high dimension, thus facing the risk of overfitting, so it is very important to obtain more effective features by dimension reduction. The method based on ROI extracts quantitative features from pre-segmented brain regions for disease classification. The features selected by this method have relatively low dimension and wide distribution, but it only focuses on the experience-defined brain regions, ignoring the influence of the disease on the remaining brain regions. The patch-based method is between the voxel and region-based methods, which can learn from the whole brain, thus effectively identifying and capturing the regions highly related to the disease. Therefore, the features extracted by this kind of method usually can obtain better classification results. However, how to better select the patches related to the disease is a key problem of the patch-based analysis method. In addition, the above-mentioned feature extraction methods have several common shortcomings. First, it is not enough to represent the key information of the brain of all subjects by relying on single-level defined features. Second, the hand-crafted features are usually independent of the subsequent classification task, so they may affect the final diagnosis effect, which is also an important aspect that needs to be considered in the feature selection and classification process.

[0003] In recent years, deep convolutional neural networks (CNN) have been widely used in various medical image classification, detection and segmentation tasks. CNN can effectively discover the potential or hidden representation between different data, so it has good classification performance in the diagnosis of AD and other brain diseases. The original voxel-based or cortical surface vertex feature-based method is usually rarely directly used in classification tasks because of the large feature dimension. Most studies obtain relevant latent space representation by performing sparse selection or learning mapping matrix to obtain disease-related high-level features for subsequent classification tasks. For example, Liu et al. use an autoencoder to map the original multi-modal image to a latent space, then use the latent representation to supplement the kernel matrix of the incomplete view in the kernel space, and apply it to the learned kernel matrix, and finally use it in the classification task. However, considering that the early stage of AD will only cause subtle structural changes in the brain, it is difficult to train a traditional end-to-end CNN model without any guidance of AD classification. Therefore, based on domain knowledge and expert experience, most existing CNN-based methods pre-determine the information area (such as the hippocampus, medial temporal lobe, and other regions) in sMRI to construct a diagnostic model. For example, Liu et al. proposed a multi-modal neuroimaging feature learning algorithm based on SAE, which learns feature representation from ROI-based features for AD diagnosis; Li et al. developed a robust multi-task deep learning framework to improve AD / MCI diagnosis by optimizing the selection of ROI features, and so on. However, this method only selects part of the brain area, and there may be potential influences in other brain areas, which may hinder the effectiveness of deep neural networks in brain disease diagnosis. Therefore, a patch-based method is proposed to more effectively avoid such problems. For example, Payan et al. combined sparse autoencoders and convolutional neural networks to learn feature representation from local patches for AD prediction; Zhu et al. extracted patches by obtaining significant difference information between sMRI images of different types of subjects, and used a dual attention mechanism to obtain the latent representation of each patch to improve the prediction effect of AD. However, these methods may not be the best patch selection method, as they only generate patches based on different large area distributions, which may miss some key patches, thereby having a greater impact on the subsequent classification process.

[0004] Therefore, a deep learning method that can effectively locate patches from multi-modal images and extract more effective brain information from patches is a necessary prerequisite for accurately locating the lesion area of related neurodegenerative diseases and assisting in diagnosis. SUMMARY

[0005] To solve the technical problems existing in the prior art, the present application provides a brain lesion area positioning system and method for multi-modal images, which adaptively selects and constructs patches in multi-modal images through an adaptive multi-modal patch growth and pruning module to identify patch positions with high discriminability, reduces the number of parameters to be learned, and reduces the model size; through a multi-modal patch fusion module based on a double attention mechanism and a global convolution network classifier, a deeper representation of the data is further obtained, and more potential information is extracted, so that the system can effectively identify the lesion area highly related to the disease and can assist in diagnosis.

[0006] The first object of the present application is to provide a brain lesion area positioning system for multi-modal images.

[0007] The second object of the present application is to provide a brain lesion area positioning method for multi-modal images.

[0008] The first object of the present application can be achieved by adopting the following technical solutions:

[0009] A brain lesion area positioning system for multi-modal images, the system comprising:

[0010] An adaptive multi-modal patch growth and pruning module for constructing the corresponding relationship of different modal images of each subject, dividing the different modal images into anatomical regions, randomly generating a patch of a fixed size in each anatomical region, training each patch selection subnetwork to generate n patches related to the disease; the adaptive multi-modal patch growth and pruning module comprises a patch selection subnetwork for training work of multi-modal patches;

[0011] A multi-modal patch multi-level fusion module based on a double attention mechanism, comprising an attention network and an attention pooling layer, for obtaining the potential feature representation of the multi-modal patch through the double attention mechanism, and constructing the multi-level fusion features of the multi-modal patch in combination with the influence score features of each patch; the attention network is used to learn the deep potential feature representation of each patch, and the features that can be used for differentiation in each patch are subjected to enhancement operation, and the attention pooling layer is used to learn the potential features of all patch fusions from different modal images;

[0012] A global convolution network classifier module for learning the deep feature representation of the multi-level fusion global features of the multi-modal patch, flattening the multi-level fusion global features of the multi-modal patch, and generating the score of the multi-level fusion global features of the multi-modal patch.

[0013] The second object of the application can be achieved by adopting the following technical solutions:

[0014] A brain lesion region positioning method for multi-modal images, the method comprising:

[0015] Corresponding relationships of different modal images of each subject are constructed, the different modal images are divided into anatomical regions, a patch with a fixed size is randomly generated in each anatomical region, each patch selects a sub-network for training, and n disease-related patches are generated;

[0016] Potential feature representations of the multi-modal patches are obtained through a double attention mechanism, multi-level fusion features of the multi-modal patches are constructed by combining the influence score features of each patch, deep-level potential feature representations of each patch are learned through an attention network, and the features of the parts that can be used for differentiation in each patch are enhanced, and potential features fused from all patches of different modal images are learned through an attention pooling layer;

[0017] Deep feature representations of multi-level fusion global features of the multi-modal patches are learned, the multi-level fusion global features of the multi-modal patches are flattened, and scores of the multi-level fusion global features of the multi-modal patches are generated.

[0018] Compared with the prior art, the application has the following advantages and beneficial effects:

[0019] The application proposes a brain lesion region positioning system and method for multi-modal images, a self-adaptive multi-modal patch growth and pruning module is used to adaptively select and build patches in multi-modal images to identify patch positions with high discriminability, thereby reducing the number of parameters to be learned and reducing the model size, a multi-modal patch fusion module based on a double attention mechanism is used to obtain potential representations of the multi-modal patches, the multi-level representations of the patches are fused and learned together in a data-driven manner, a global convolution network classifier is used as a final classification model to further obtain deeper representations of the data and extract more potential information, and the application can be used for AD classification and MCI conversion prediction by training and evaluation on two modal datasets, and the system proposed by the application can effectively identify lesion regions highly related to diseases and can assist in diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained based on the drawings shown without any creative effort.

[0021] Figure 1 is a structural block diagram of a brain lesion area positioning system in the embodiments of the present application;

[0022] Figure 2 is a structural schematic diagram of a DA-MPSFDL model in the embodiments of the present application;

[0023] Figure 3 is a structural schematic diagram of a patch selection sub-network in the embodiments of the present application;

[0024] Figure 4 is a structural schematic diagram of an attention network in the embodiments of the present application;

[0025] Figure 5 is a structural schematic diagram of an attention pooling layer in the embodiments of the present application;

[0026] Figure 6 is a flowchart of a brain lesion area positioning method for multi-modal images in the embodiments of the present application. DETAILED DESCRIPTION

[0027] The technical solutions of the present application will be described in further detail below in combination with the accompanying drawings and embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The implementation of the present application is not limited to this. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the scope of protection of the present application.

[0028] Embodiment 1:

[0029] As Figures 1-2As shown, the application proposes a brain lesion area positioning system for multi-modal images, which comprises an adaptive multi-modal patch growth and pruning module (AMPGP), a dual attention mechanism based multi-modal patch multi-level fusion module (DAMPF) and a global convolution network classifier module (GCNC). The adaptive multi-modal patch growth and pruning module is used to establish the correspondence of different modal images of each subject, divide the different modal images into anatomical regions, randomly generate a multi-modal patch of a fixed size in each anatomical region, select a subnetwork for training of each multi-modal patch, and generate n multi-modal patches related to the disease. The adaptive multi-modal patch growth and pruning module comprises a patch selection subnetwork, which is used for the training of multi-modal patches. The dual attention mechanism based multi-modal patch multi-level fusion module is used to obtain the potential feature representation of the multi-modal patch through the dual attention mechanism, and construct the multi-level fusion features of the multi-modal patch by combining the influence score features of each multi-modal patch. The global convolution network classifier module is used to learn the deep feature representation of the multi-level fusion features of the multi-modal patch, extract the high-level potential information of the multi-modal patch, and compress the feature map along the channel.

[0030] The brain lesion area positioning system for multi-modal images described in the application is based on a dual attention mechanism based multi-modal patch automatic selection and multi-level fusion deep learning model (DA-MPSFDL). In this model, the adaptive selection and pruning method in the multi-modal image can be used to identify the patch position with high discrimination. This model reduces the number of parameters to be learned, reduces the model size, and significantly helps the performance of the subsequent model. On this basis, the dual attention mechanism based multi-modal patch fusion module is used to obtain the potential representation form of the multi-modal patch, and the influence score of each patch is combined to perform fusion and common learning in a data-driven manner. A multi-level classification model is constructed by combining the global convolution network classifier, further obtaining the deeper representation of the data, and extracting more potential information.

[0031] Since the data label used by the present application is defined based on multi-modal images, there is no explicit provision of label based on patch level. Therefore, the label definition of the patch is defined according to the image to which it belongs, and the image label is regarded as the only guide for updating the network weight. Therefore, the present application trains the model based on cross-entropy loss, and in this process, the loss function is defined as:

[0032]

[0033] Where N is the number of different modal images, Y n is the label of the nth modality, X n is the feature of the nth modality, W is the weight, and P(Y n |X n ; W) is the probability of correct prediction of X n .

[0034] The DA-MPSFDL model of the present application is not a traditional end-to-end network architecture, and its end-to-end mapping relationship is performed after patch selection. After determining the high correlation patch, the model created by the present application can be regarded as an end-to-end network, and the training loss is propagated from the global convolutional network classifier to the previous module, and then the Adam optimization algorithm is used to help update the network parameters. By minimizing the loss function, the network of the DA-MPSFDL model of the present application can obtain more potential information of the patch level sample and obtain the region highly related to the disease.

[0035] The adaptive multi-modal patch growth and pruning module (AMPGP) is used to build the correspondence of different modal images of each subject, divide the different modal images into anatomical regions, and randomly generate a patch of a fixed size in each anatomical region. Each patch selects a sub-network for training to generate n patches related to the disease.

[0036] The AMPGP module can automatically update the patch with high recognition level and prune the patch with low recognition level in the patch selection process, and finally realize the accurate positioning of the brain lesion patch of the patient. The working steps of the AMPGP module include:

[0037] S11, build the correspondence of different modal images of each subject.

[0038] The magnetic resonance imaging image (MRI image) and the positron emission computed tomography image (PET image) of each subject are preprocessed, the MRI image is registered with a preset standard template, and the PET image of the same subject is registered to the corresponding MRI image, so that the PET image and the MRI image have consistent spatial distribution. In this embodiment, the preset standard template is a Colin27 template.

[0039] S12, divide the different modality images into anatomical regions, and randomly generate path points in the minimum stereoscopic subgraph of each region.

[0040] The different modality images are divided into anatomical regions according to the AAL90 anatomical template, the minimum stereoscopic subgraph containing each region is extracted, and a path point is randomly generated in the stereoscopic graph of each region. The path point of each region must belong to the interior of the region, and a patch of a fixed size (for example, 25x25x25) is generated around the path point. A total of 180 patches are generated, and the value of each path point of each region is recorded each time the path point is updated.

[0041] S13, construct a patch selection subnetwork, and obtain an influence score feature and a diagnosis loss of each patch through the patch selection subnetwork.

[0042] The AMPGP module includes a patch selection subnetwork for training of multi-modality patches. The patch selection subnetwork includes four convolutional layers, two maximum pooling layers, one fully connected layer, and a final sigmoid function. The first convolutional layer has a kernel size of 4x4x4, and the other three convolutional layers have a kernel size of 3x3x3. The channel numbers of the four convolutional layers are 32, 64, 128, and 128, respectively. All the convolutional layers are 0-padded and have a stride of 1, followed by batch normalization (BN) and rectified linear unit (ReLU) activation operations; the maximum pooling layer has a size of 2x2x2 and a stride of 2. The network is used to generate an influence score for each patch, which is used for patch updating and pruning. In addition, it is also used to provide the local feature representation of the current patch after the patch is selected, so as to better fuse the patches and further learn the potential relationship between the patches. The structure diagram of the patch selection subnetwork is shown in Figure 3 .

[0043] S14, training each patch using the patch selection subnetwork, and pruning and updating the generated patch according to the diagnosis loss.

[0044] Each patch enters the patch selection sub-network for training respectively, obtains its own diagnostic loss, retains the top n patches with smaller diagnostic loss, and regenerates path points outside the specific distance of all existing points in the region. The patch is reconstructed and trained, and the training results are compared with the diagnostic loss of the retained patch of the last generation. The top n patches with smaller diagnostic loss are still retained, and the process is repeated until the selection is completed. In addition, in order to improve the efficiency of patch selection, when there are no patches retained in a region for three generations in succession, the region is directly pruned. The patch is generated according to the region to ensure that the network can use the widely distributed locations in the whole brain to train in a data-driven manner, while reducing the redundancy caused by patch overlap. In addition, no explicit assumptions are made about the individual diagnostic ability of the patch in each region. This is different from existing methods based on regions and patches, which usually select and sort regions or patches based on pre-defined domain knowledge, and finally generate n patches most relevant to the disease. Since each region is affected differently by the disease, the number of patches generated by each region may not be exactly the same.

[0045] In addition, this module can also be extended to include prior knowledge methods to reduce computational complexity and improve learning performance. According to the suggestion of prior knowledge, multiple regions related to the disease can be effectively obtained locally, and some obviously useless or ineffective regions can be filtered out. Therefore, the invention can also generate, update and prune the patch region for these high correlation regions. Based on prior knowledge, some important brain regions are selected, and the key patch positioning is obtained through the AMPGP method, and it is involved in the comparison process of different tasks. In the invention, this implementation is called the prior-based adaptive multi-modal patch growth and pruning module (AMPGP).

[0046] The multi-modal patch multi-level fusion module (DAMPF) based on a double attention mechanism includes an attention network (AN) and an attention pooling layer (AP). The module is used after the key multi-modal patch is obtained based on the adaptive multi-modal patch growth and pruning module, and is used to obtain the potential feature representation of the multi-modal patch through the double attention mechanism, and to construct the multi-level fusion feature representation of the multi-modal patch by combining the influence score feature of each patch. The attention network is used to learn the deep-level potential feature representation of each patch, and to perform enhancement operation on the features that can be used for differentiation in each patch. The attention pooling layer is used to learn the potential feature representation of the fusion of all patches from different modal images. The structural diagram of the attention network is shown in Figure 4 The structural diagram of the attention pooling layer is shown in Figure 5 .

[0047] After positioning to the key patch position, in order to further obtain different levels of feature representation, the application adopts a multi-modal patch fusion module based on a double attention mechanism to learn a deeper level of potential representation, and combines the representation data of the influence score of the patch learned by the previous module to combine into a new feature representation fused by multiple levels. This module will help to obtain more perfect brain information from different modal images.

[0048] The working steps of the DAMPF module include:

[0049] S21, the attention network is used to further process the patch block network, to learn the potential feature representation from the original patch, and to reduce the size of the feature mapping. The attention network used to process the patch block representation can adapt to the local structure feature extraction from the three-dimensional image patch. The specific operation is as follows:

[0050] S211, two different channel-based pooling operations of channel maximum pooling and channel average pooling are adopted for each patch to generate a maximum value feature map and an average value feature map.

[0051] Two different channel-based pooling operations (i.e., channel maximum pooling (CMP) and channel average pooling (CAP)) are adopted for the feature map of each retained patch learned by the attention network (each size is Cxwxxwxxw, where C is the number of channels, and wxxwxxw is the dimension of the feature map), respectively to generate two feature maps in the name of maximum feature and average feature.

[0052] S212, connect the maximum value feature map and the average value feature map, perform convolution operation on the connected feature map, and output the attention-based weight feature map.

[0053] The two feature maps are connected to generate a feature map with a size of 2xw x w x w, and then a convolution layer with a stride of 1, a padding of 1, and a kernel size of 3x3x3 is set to perform further convolution operation. The padding is set to 1 to ensure that the size of the feature map does not change. After the output of the convolution layer, the sigmoid function is used to limit the feature value of each position to the range of 0 to 1, and a feature map with the same dimension size as the original input is obtained.

[0054] S213, multiply the attention-based weight feature map with the original feature map, and perform channel maximum pooling on the feature maps of different channels to generate a single-channel patch feature map.

[0055] After obtaining the attention-based weight feature map with the same dimension size as the original input, it is multiplied with the original feature map to generate a local spatial attention perception structure representation. The feature map of each patch after learning by the attention network can show the spatial variation contribution of different parts, further revealing the parts that need to be emphasized or suppressed in the feature representation, and finally the feature maps of different channels are fused into a single-channel feature map to form the final patch feature map.

[0056] Specifically, for the feature map input to the network: F = {F1, F2,..., F C}, where C is the number of channels, The maximum pooling layer F max and the average pooling layer F avgrage are respectively pooled, and two feature maps are respectively generated based on the maximum feature and the average feature; wherein the maximum pooling layer based on each channel axis:

[0057] F max = ChannelMaxPooling(F)

[0058] takes the maximum value of the same position of all channels, that is, The average pooling layer based on each channel axis:

[0059] F avgrage = ChannelAvgragePooling(F)

[0060] takes the average value of the same position of all channels, that is, Then the two feature maps are connected, and an attention-based weight feature map is calculated:

[0061]

[0062] where [;] represents the connection operation, denotes the convolutional layer weight, and σ denotes the sigmoid activation function. The weight feature map is then multiplied element-wise with the original feature map to obtain the patch feature map based on attention weighting:

[0063]

[0064] Finally, F new is obtained by performing channel average pooling along the channel axis. AN .

[0065] S22, the multi-modal patch superimposed feature after channel combination is fused using an attention pooling layer to generate a local spatial attention perception structure representation based on multi-modal patch channel fusion features.

[0066] This module learns the feature map of the attention network training through an attention pooling layer to determine the relative contribution of each feature map and perform fusion operation. The specific operation is:

[0067] S221, the single-channel feature map of each patch is obtained based on the attention network training, and a multi-channel global feature map is obtained by channel superposition.

[0068] The feature map F AN (size 1xw x w x w) representing each patch is connected to generate a multi-channel global feature map F where n represents the number of selected patches (the number of patches is determined by the AMPGP module), denotes the feature map of the i-th patch.

[0069] S222, the multi-channel global feature map F gobal is respectively generated based on different pooling convolution operations to respectively generate two different aspects of feature representation.

[0070] After the connection generates an n-channel feature map, an average pooling layer (AP) and a maximum pooling layer (MP) are constructed in parallel to generate two different aspects of feature representations respectively. The use of the above two different aspects of feature representations can improve the representation ability of the network, which is better than using only one of them. After the pooling operation, two convolution kernel sizes of 1x1x1 convolution layers are set to further learn different feature representations and generate two different levels of patch feature maps; W0 and W1 are used as the weights of the convolution layer, and the convolution layer processing the average feature representation and the convolution layer processing the maximum feature representation share parameters; the two different levels of patch feature maps are combined into a multi-level fusion global patch feature map by element summation, and then a sigmoid function is activated.

[0071] S223, multiply the fused patch feature map with the previous multi-channel global feature map to obtain a global feature representation based on the attention perception area.

[0072] The multi-level fusion global patch feature map activated by the sigmoid function is multiplied element by element with the previous multi-channel global feature map F gobal to obtain a global feature representation based on the attention perception area. In this module, the attention pooling layer (AP) module considers each patch feature, and the different patch channel superimposed feature map F gobal is used to better represent the spatial variation contribution of different parts. Therefore, AP can better obtain deeper feature representations of key patches and reduce noise interference.

[0073] S23, connect the patch impact score feature map after channel superposition and the global feature map based on the attention perception area to construct a multi-level fusion feature of the multi-modal patch.

[0074] Each patch impact score feature map learned by the AMPGP module is superimposed in the channel to obtain the same dimension as the global fusion feature map after AN and AP training. By connecting the impact score feature map and the global fusion feature map, a multi-level feature representation form is obtained, avoiding the loss of potential information while maintaining the connectivity between patches, thereby improving the classification performance.

[0075] S3, a global convolution network classifier module (GCNC) is used to learn the deep feature representation of the multi-level fusion global feature of the multi-modal patch, flatten the multi-level fusion global feature of the multi-modal patch, and generate a score of the multi-level fusion global feature of the multi-modal patch.

[0076] The GCNC is used to consider the correlation between the multi-level fusion global features of the multi-modal patches, and provides a global classifier for these fused representation data, to realize the positioning of the lesion area. Compared with directly using a fully connected layer to explore the correlation between patch-level features, the convolutional layer has superior ability to extract high-level features when learning the features between patches in depth. Therefore, the classification module includes two convolutional networks for further learning of deeper feature representations from the representation data of the DAMPF module, extracting more potential information, and compressing the feature maps along the channel. The result obtained after training the previous module and combining it is a weighted multi-channel feature matrix, also known as a feature map.

[0077] The GCNC includes two convolutional layers, one max-pooling layer, two fully connected layers and a sigmoid layer. The two convolutional layers have 128 and 64 filters respectively, and the convolutional layer size is 2x2x2 and the stride is 1, followed by batch normalization (BN) and ReLU activation. Then, an adaptive three-dimensional max-pooling layer is used to downsample the convolutional results to 1x1x1 size with the same number of channels. Then, the multi-level fusion global features of the multi-modal patches are flattened, and the outputs of the two fully connected layers are 32 and 2 respectively, and finally two score scores are generated, which are normalized by the softmax function, representing the negative probability and positive probability respectively.

[0078] In short, the GCNC further learns the multi-level fusion global features trained by the previous module, represents the overall information of the whole brain results and functions by obtaining a small amount of higher-level potential information, and is used in AD classification or MCI conversion prediction and other tasks.

[0079] In the present embodiment, the brain lesion area positioning system for multi-modal images can be verified in multiple AD-related diagnostic tasks such as AD classification, MCI conversion prediction and MCI classification. In addition, for the true class (TP), the true negative class (TN), the false positive class (FP) and the false negative class (FN) in the classification process, the application applies four indicators to evaluate the classification performance, including accuracy (ACC), sensitivity (SEN), specificity (SPE) and area under the receiver operating characteristic curve (AUC), which are defined as follows.

[0080]

[0081]

[0082]

[0083]

[0084] where ACC, SEN and SPE are calculated with a default 0.5 threshold, and AUC is calculated by varying the threshold on the prediction of the network set by the present application, according to the ratio of all possible true positive and false positive rates.

[0085] First, to verify the effectiveness, the present application is compared with several baseline methods, i.e. traditional voxel-based method (VBM), traditional ROI-based method (ROI), traditional patch-based method (PLM), and some recent advanced deep learning algorithms (e.g. DA-MIDL, H-FCN). In the whole training process, MRI images and PET pictures of different types of subjects are needed, which are divided into four categories: cognitively normal (CN), early mild cognitive impairment (EMRI), late mild cognitive impairment (LMRI) and Alzheimer's disease (AD). The image data comes from the ADNI database. In the whole experiment, the present application is applied to 126 AD patients, 108 MCI patients (60 EMRI, 48 LMRI) and 132 CN, which are subjected to the same preprocessing process. Specifically, the MRI image preprocessing process includes anterior commissure (AC)-posterior commissure (PC) correction, intensity inhomogeneity correction based on N3 algorithm, and skull stripping, cerebellum removal and standard template registration. Then the MRI image is segmented into three different tissues of gray matter (GM), white matter (WM) and cerebrospinal fluid (CSF) using the FAST algorithm in the FSL software package. After extracting the gray matter tissue, each MRI image is divided into 90 regions according to the AAL90 template. Similarly, the same number of PET images are obtained, and each PET image is rigidly registered with the corresponding MRI image. Therefore, 90 regions are extracted from the MRI and PET images respectively, and then the regional minimum spatial subgraph of each region is generated, and a patch is randomly generated in each anatomical region as the original input of the model of the present application. The demographic information of the subjects of the MRI from the ADNI database is shown in Table 1. In Table 1, the gender is divided into male (M) / female (F), and the age and mini-mental state examination (MMSE) values are reported as mean ± standard deviation. The PET images have the same demographic information as the MRI, and in addition, the samples of the ADNI dataset will be divided into a training dataset and a test dataset, of which 90% of the samples are used for model training, and the remaining 10% of the samples are used as the test dataset. The ten-fold cross-validation strategy is adopted to select the hyperparameters and the model training process. Then the trained model with optimized hyperparameters is tested on the reserved ADNI test dataset.

[0086] Table 1

[0087]

[0088] The processor used in the embodiments of the present application is Intel Core i5-4460@3.20GHz, the memory is 32GB, the display memory is 12GB, the display card is GeForce GTX TITAN, and the related experiments are run based on the Pytorch package on Python. In order to alleviate the problem of overfitting, the present application uses batch normalization for activation after each convolutional layer, and all patch networks share weights, reducing the number of training parameters. In the training stage, the present application first trains the patch region by selecting the initial patch position randomly generated by each regional subgraph on the training set, and performs patch updating and pruning work. After selecting the patch highly related to the disease, the patches extracted from the selected positions in the images from different modalities are respectively input into the corresponding attention network. In this process, the present application uses the Adam optimizer to perform 100 iterations of training. In addition, the number of channels at each level of the present application has been described in detail in the foregoing, and in the whole process, the learning rate is set to 0.001, the batch size is 10, the patch size is 25x25x25, and the final selected patch number is 30, and the overall training time is about 10 hours. In the test stage, the present application uses the patch extracted from the same position in the training stage to locate the related lesion area from the unknown type of multi-modal image, and the positioning time is about 1-2 seconds.

[0089] The present application and some other positioning methods achieve the performance of AD disease classification and MCI conversion prediction on the test set, wherein the classification performance of different models on AD and CN is shown in Table 2; the classification performance of different models on EMCI and LMCI is shown in Table 3. From the table, it can be seen that the present model is better than other models in both diagnostic tasks.

[0090] Table 2

[0091]

[0092] Table 3

[0093]

[0094] As shown in Tables 2 and 3, the model used in the application achieved better performance in comparison with other algorithms. In the AD classification task, the prediction positioning accuracy of the application was as high as 93.02%, which was significantly better than the other five models. In addition, in the MCI conversion prediction task, the application obtained a prediction structure of ACC = 0.8101, SEN = 0.7857, SPE = 0.8333 and AUC = 0.796, which was significantly better than the results of the other five methods. At the same time, the performance of the patch-level method (such as PLM, wH-FCN, DA-MIDL and DA-MPSFDL) was better than that of the voxel-level and ROI-level method (such as VBM and ROI). In addition, compared with the traditional patch-level method (i.e. PLM), the deep learning-based method (i.e. wH-FCN, DA-MIDL and DA-MPSFDL) achieved better results in the diagnosis of Alzheimer's disease.

[0095] The application also designs an ablation experiment, by comparing the classification effects of removing single modal image, patch selection module, attention module (attention network plus attention pooling layer) and complete model in the AD prediction process, the classification accuracy of the complete model is as high as 93.02%. When the MRI image is removed, the accuracy of the application for AD decreases to 89.72%, and further removing the attention module, the classification accuracy further decreases by 2.07%; similarly, when the PET image is removed, the accuracy of the application for AD decreases by 1.05%, and further removing the attention module, the classification accuracy further decreases by 1.92%; in addition, only removing the attention module, the classification accuracy is 91.35%, which is 1.67% lower than that of the complete module. Among these modules, the data of different modalities complement each other well, and have better classification effect than using only single modal image for experiment. At the same time, the attention module helps to obtain more complete brain information from different modal images, and when the module is added, the classification effect for AD is obviously improved. In addition, the application also compares the size of the patch dimension, and finds that the 25x25x25 patch block can realize better lesion area representation.

[0096] Removing model structure Accuracy ACC Removing MRI images 0.8972 Removing MRI images and attention module 0.8765 Removing PET images 0.9197 Removing PET images and attention module 0.9005 Removing attention network 0.9135 No 0.9302

[0097] In summary, this invention proposes an adaptive patch selection and construction (AMPGP) module in multimodal images to identify highly discriminative patch locations. This module reduces the number of parameters to be learned, lowers the model size, and significantly speeds up the subsequent diagnostic process. Furthermore, this invention utilizes a multimodal patch fusion module based on a dual-attention mechanism to obtain the latent representations of multimodal patches. Combining the multi-level representations of patches, this invention fuses and co-learns them using a data-driven approach. This invention also uses a global convolutional network classifier as the final classification model to further obtain deeper representations of the data and extract more latent information. In experiments, the proposed method was trained and evaluated on two modal datasets (sMRI and PET) for multiple AD-related diagnostic tasks, including AD classification and MCI conversion prediction. Experimental results show that the proposed method can effectively identify and locate abnormal regions highly correlated with the disease.

[0098] Example 2:

[0099] like Figure 6 The diagram shows a flowchart of a method for locating brain lesions in multimodal images. This invention also provides a method for locating brain lesions in multimodal images, the method comprising:

[0100] The correspondence between different modal images of each subject is constructed, the different modal images are divided into anatomical regions, a fixed-size patch is randomly generated in each anatomical region, and a sub-network is selected for training in each patch to generate n disease-related patches.

[0101] The latent feature representation of multimodal patches is obtained through a dual attention mechanism. Multi-level fusion features of multimodal patches are constructed by combining the impact score features of each patch. Deep latent feature representation of each patch is learned through an attention network. Enhancement operations are performed on the features in each patch that can be used for differentiation. Latent features of all patches from different modal images are learned through attention pooling layers.

[0102] We learn the deep feature representation of the multi-level fusion global features of multimodal patches, flatten the multi-level fusion global features of multimodal patches, and generate a score of the multi-level fusion global features of multimodal patches.

[0103] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. A brain lesion region positioning system for multi-modal images, characterized by, The system comprises: An adaptive multi-modal patch growth and pruning module for establishing a correspondence relationship of different modal images of each subject, dividing the different modal images into anatomical regions, randomly generating a patch of a fixed size in each anatomical region, selecting a subnetwork for training of each patch, and generating n patches related to diseases; the adaptive multi-modal patch growth and pruning module comprises a patch selection subnetwork, and the patch selection subnetwork is used for training of multi-modal patches; A multi-modal patch multi-level fusion module based on a double attention mechanism, comprising an attention network and an attention pooling layer, for obtaining a latent feature representation of multi-modal patches through a double attention mechanism, and constructing a multi-level fusion feature of the multi-modal patches in combination with an influence score feature of each patch; the attention network is used for learning a deep latent feature representation of each patch, and performing an enhancement operation on features of a part that can be used for differentiation in each patch, and the attention pooling layer is used for learning a latent feature fused from all patches of different modal images; A global convolution network classifier module for learning a deep feature representation of a multi-level fusion global feature of multi-modal patches, flattening the multi-level fusion global feature of the multi-modal patches, and generating a score of the multi-level fusion global feature of the multi-modal patches.

2. The brain lesion region positioning system for multi-modal images according to claim 1, wherein, The adaptive multi-modal patch growth and pruning module comprises the following working steps: A correspondence relationship of different modal images of each subject is established; The different modal images are divided into anatomical regions, and path points are randomly generated in a minimum stereoscopic subgraph of each divided anatomical region; A patch selection subnetwork is constructed, and an influence score feature and a diagnosis loss of each patch are obtained through the patch selection subnetwork; Each patch is trained using the patch selection subnetwork, and the generated patch is pruned and updated according to the diagnosis loss.

3. The brain lesion region positioning system for multi-modal images according to claim 2, wherein, The correspondence relationship of different modal images of each subject is established by: pre-processing PET images and MRI images of each subject, registering the MRI images with a preset standard template, and registering the PET images of the subject on the corresponding MRI images, so that the PET images and the MRI images have consistent spatial distribution.

4. The brain lesion region positioning system for multi-modal images according to claim 3, wherein, The patch selection subnetwork comprises four convolution layers, two maximum pooling layers, one fully connected layer, and a final sigmoid function layer, wherein the first convolution kernel size of the four convolution layers is 4x4x4 layers, and the convolution kernel sizes of the other three are all 3x3x3 layers, the channel numbers of the four convolution layers are 32, 64, 128, and 128 respectively, all the convolution layers are 0 padding and stride=1, and the maximum pooling layer is 2x2x2 and stride=2.

5. The brain lesion region positioning system for multi-modal images according to claim 4, wherein, The patch selection subnetwork is used for training of multi-modal patches, and the generated patch is pruned and updated according to the diagnosis loss, comprising: Each patch enters the patch selection sub-network for training respectively, obtains respective diagnostic loss, retains the first n patches with smaller diagnostic loss, regenerates path points outside the specific distance of all existing points in the region for each region, reconstructs the patch and trains, compares the training result with the diagnostic loss of the retained patch of the last generation, retains the first n patches with smaller diagnostic loss, and the like is repeated until the selection is completed; when no patch is retained in a region for three generations in succession, the region is pruned.

6. The brain lesion region positioning system for multi-modal images according to claim 1, wherein, The working steps of the multi-modal patch multi-level fusion module based on the double attention mechanism include: The network using an attention network further processes the patch block, learns potential feature representation from the original patch, and reduces the size of the feature mapping; An attention pooling layer is used to fuse the multi-modal patch superimposed features after channel combination, to generate a local spatial attention perception structure representation based on multi-modal patch channel fusion features; The patch influence score feature map after channel superposition and the global feature map based on attention perception regions are connected to construct a multi-level fusion feature representation of the multi-modal patch.

7. The brain lesion region positioning system for multi-modal images according to claim 6, wherein, The network using an attention network further processes the patch block, learns potential feature representation from the original patch, and reduces the size of the feature mapping, includes: Channel maximum pooling and channel average pooling are used for each patch to generate a maximum value feature map and an average value feature map; The maximum value feature map and the average value feature map are connected, and a convolution operation is performed on the connected feature map to output a weight feature map based on attention; The weight feature map based on attention is multiplied by the original feature map, and a single-channel patch feature map is generated by channel maximum pooling of the feature map in different channels.

8. The brain lesion region positioning system for multi-modal images according to claim 6, wherein, The network using an attention network further processes the patch block, learns potential feature representation from the original patch, and reduces the size of the feature mapping, includes: The single-channel feature map of each patch is obtained based on the attention network training, and a multi-channel global feature map is obtained by channel superposition; The multi-channel global feature map Fgobal is respectively generated into two different aspects of feature representation based on different pooling convolution operations; The fused patch feature map is multiplied by the previous multi-channel global feature map to obtain a global feature representation based on attention perception regions.

9. The brain lesion region positioning system for multi-modal images according to claim 1, wherein, The global convolution network classifier module includes two convolution layers, one maximum pooling layer, two fully connected layers and one sigmoid function layer, the two convolution layers have 128 and 64 filters respectively, the convolution layer size is 2*2*2 and the stride is 1, and the outputs of the two fully connected layers are 32 and 2 respectively.

10. A method for locating brain lesion region based on multi-modal images, implemented by the system for locating brain lesion region based on multi-modal images according to any one of claims 1-9, characterized in that, The method includes: Corresponding relationship of different modal images of each subject is constructed, different modal images are divided into anatomical regions, a patch with a fixed size is randomly generated in each anatomical region, each patch selects a sub-network for training, and n disease-related patches are generated; The latent feature representation of the multi-modal patch is obtained through the double attention mechanism, and the multi-level fusion feature of the multi-modal patch is constructed by combining the influence score feature of each patch; the deep-level latent feature representation of each patch is learned through the attention network, and the features that can be used for differentiation in each patch are enhanced, and the latent features of all patch fusions from different modal images are learned through the attention pooling layer; The deep feature representation of the multi-level fusion global feature of the multi-modal patch is learned, the multi-level fusion global feature of the multi-modal patch is flattened, and the score of the multi-level fusion global feature of the multi-modal patch is generated.

Images