An image classification method, device, equipment and storage medium

Abstract

The present application relates to the technical field of brain disease determination, in particular to an image classification method, device and equipment and storage medium. Magnetic resonance images can accurately reflect the changes in brain structure caused by lesions, and positron emission tomography images can accurately reflect the changes in brain function caused by lesions. The present application first uses a large kernel attention (LKA) module to learn an attention map. Then, advanced feature learning is performed through a backbone network composed of a CNN and a Transformer dual branch. The present application also designs a modal feature fusion block (MFFB) to interactively fuse MRI and PET features in the backbone network. Finally, after extracting the advanced features, a spatial channel attention module (SCA) is used to process the obtained advanced features in space and channels. A large number of experimental results on the ADNI dataset show that the model of the present application is superior to the most advanced method, indicating the effectiveness of the method proposed by the present application.

Description

Technical Field

[0001] This invention relates to the field of brain disease diagnosis technology, specifically to an image classification method, apparatus, device, and storage medium. Background Technology

[0002] Brain images can be used to diagnose many brain diseases, such as Alzheimer's disease (AD). Brain images can be used to determine if a patient has AD and the severity of their condition. AD is an irreversible, progressive neurodegenerative disease, primarily characterized by memory loss, cognitive decline, and inability to care for oneself. Mild cognitive impairment (MCI) is an early stage of AD, further subdivided into pMCI and sMCI depending on whether it progresses to AD. Timely detection and intervention in pMCI or sMCI can potentially delay the progression to AD. Currently, early AD diagnosis often relies on doctors' subjective experience, which is time-consuming and can delay optimal treatment. Therefore, research into early AD diagnostic models using computer-aided technology can help explore the pathogenesis of AD.

[0003] Currently, the main imaging diagnostic methods for early Alzheimer's disease (AD) include structural magnetic resonance imaging (sMRI), functional magnetic resonance imaging (fMRI), diffusion tensor imaging (DTI), and positron emission tomography (PET). sMRI can provide a direct visual representation of changes in brain structure, and many studies on the early diagnosis of AD have been developed using sMRI-based analyses. However, the brain information provided by sMRI images is relatively limited and insufficient to accurately determine the extent of the patient's lesions.

[0004] In summary, existing image classification methods have low accuracy.

[0005] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an image classification method, apparatus, device, and storage medium, which solves the problem of low accuracy in existing image classification methods.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides an image classification method, comprising:

[0009] Generate magnetic resonance imaging and positron emission tomography (PET) images of the brain;

[0010] Extract the fusion feature map of the magnetic resonance image and the positron emission tomography image, wherein the fusion feature map includes features of the magnetic resonance image and features of the positron emission tomography image;

[0011] The fused feature maps are classified to obtain classification results used to characterize brain diseases.

[0012] In one implementation, generating magnetic resonance imaging (MRI) images and positron emission tomography (PET) images of the brain includes:

[0013] Acquire raw MRI and positron emission tomography images of the brain;

[0014] The original magnetic resonance image is subjected to bias field correction to obtain the corrected original magnetic resonance image.

[0015] The original MRI image after removal correction is processed by applying a craniotomy algorithm to remove non-brain tissue pixels, resulting in the original MRI image after craniotomy.

[0016] The images corresponding to the gray matter of the brain are extracted from the original magnetic resonance images after skull dissection to obtain magnetic resonance images;

[0017] The original positron emission tomography (PET) image was subjected to a skull peeling algorithm to remove non-brain tissue pixels, resulting in the original PET image after skull peeling.

[0018] The original positron emission tomography (PET) images after skull dissection are smoothed to obtain PET images.

[0019] In one implementation, the extraction of a fused feature map of the magnetic resonance image and the positron emission tomography (PET) image, wherein the fused feature map includes features of the magnetic resonance image and features of the PET image, includes:

[0020] The deep convolution algorithm is applied to the magnetic resonance image and the positron emission tomography image respectively to obtain a first result and a second result output by the deep convolution algorithm. The first result corresponds to the magnetic resonance image and the second result corresponds to the positron emission tomography image.

[0021] The first result and the second result are respectively applied to the depthwise dilated convolution algorithm to obtain the third result and the fourth result output by the depthwise dilated convolution algorithm. The third result corresponds to the magnetic resonance image and the fourth result corresponds to the positron emission tomography image.

[0022] The third and fourth results are convolved to obtain the attention map of the magnetic resonance image and the attention map of the positron emission tomography image. The attention map is used to characterize the weights corresponding to each feature.

[0023] Multiply the attention map of the magnetic resonance image with the magnetic resonance image to obtain the primary feature map of the magnetic resonance image;

[0024] Multiply the attention map of the positron emission tomography (PET) image with the PET image to obtain the primary feature map of the PET image.

[0025] A fused feature map is obtained based on the primary feature map of the magnetic resonance image and the primary feature map of the positron emission tomography image.

[0026] In one implementation, obtaining the fused feature map based on the primary feature map of the magnetic resonance image and the primary feature map of the positron emission tomography image includes:

[0027] The primary feature map of the magnetic resonance image is input into a convolutional neural network and a ViT network, respectively, to obtain a first local detail feature map output by the convolutional neural network and a first global feature map output by the ViT network.

[0028] The first local detail feature map and the first global feature map are fused to obtain the first feature map of the magnetic resonance image;

[0029] The primary feature map of the positron emission tomography image is input into a convolutional neural network and a ViT network, respectively, to obtain a second local detail feature map output by the convolutional neural network and a second global feature map output by the ViT network.

[0030] The second local detail feature map and the second global feature map are fused to obtain the second feature map of the positron emission tomography image;

[0031] The first feature map and the second feature map are concatenated to obtain a concatenated feature map.

[0032] Perform a convolution operation on the spliced ​​feature map to obtain a similar feature map that characterizes the similarity between the first feature map and the second feature map;

[0033] The similarity feature map and the original features of the magnetic resonance image are connected to obtain a first connection map. The original features of the magnetic resonance image are used to characterize the features of each brain region corresponding to the magnetic resonance image.

[0034] The similarity feature map is connected to the original features of the positron emission tomography (PET) image to obtain a second connection map. The original features of the PET image are used to characterize the features of each brain region corresponding to the PET image.

[0035] Perform a convolution operation on the first connection graph to obtain the high-level feature map of the magnetic resonance image;

[0036] Perform a convolution operation on the second connectivity graph to obtain a high-level feature map of the positron emission tomography image;

[0037] The high-level feature maps of the magnetic resonance imaging and the positron emission tomography (PET) images are fused to obtain a fused feature map.

[0038] In one implementation, performing a convolution operation on the first connectivity graph to obtain a high-level feature map of the magnetic resonance image includes:

[0039] Perform a 1x1 convolution operation on the first connection graph to obtain the specific feature map of the magnetic resonance image;

[0040] The specific feature map is transformed in dimension to obtain the specific feature map of each cross-section;

[0041] Convolution operations are performed on the specific feature maps of each cross section to obtain the convolution results of the specific feature maps of each cross section;

[0042] The high-level feature map of the magnetic resonance image is obtained by convolving the specific feature maps of each cross section.

[0043] In one implementation, classifying the fused feature map to obtain a classification result for characterizing brain diseases includes:

[0044] Calculate the weights of each channel in the fused feature map;

[0045] Multiply the fused feature map by the weights of each channel to obtain the first channel feature;

[0046] Calculate the features of each channel of the fused feature map;

[0047] Calculate the average value of the features of each channel to obtain the channel mean;

[0048] Multiply the fused feature map by the channel mean to obtain the second channel feature;

[0049] The first channel feature and the second channel feature are added together to obtain the final feature map;

[0050] The final feature maps are classified to obtain classification results used to characterize brain diseases.

[0051] In one implementation, the multiple final feature maps are classified to obtain classification results for characterizing brain diseases, including:

[0052] Perform global average pooling on the final feature map to obtain the pooled feature map;

[0053] Apply a classifier to the pooled feature map to obtain the classifier's output.

[0054] Based on the results of the classifier, brain diseases are classified.

[0055] Secondly, embodiments of the present invention also provide a brain disease classification device, wherein the device comprises the following components:

[0056] The image acquisition module is used to generate magnetic resonance imaging and positron emission tomography (PET) images of the brain;

[0057] The feature extraction module is used to extract a fusion feature map of the magnetic resonance image and the positron emission tomography image, wherein the fusion feature map includes features of the magnetic resonance image and features of the positron emission tomography image;

[0058] The classification module is used to classify the fused feature map to obtain classification results for characterizing brain diseases.

[0059] Thirdly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and a brain disease classification program stored in the memory and executable on the processor, wherein when the processor executes the brain disease classification program, it implements the steps of the image classification method described above.

[0060] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a brain disease classification program, wherein when the brain disease classification program is executed by a processor, it implements the steps of the image classification method described above.

[0061] Beneficial effects: Magnetic resonance imaging (MRI) can accurately reflect changes in brain structure caused by lesions, while positron emission tomography (PET) can accurately reflect changes in brain function caused by lesions. This invention uses MRI and PET to classify brain diseases. The combination of these two methods can demonstrate morphological changes in the brain caused by brain atrophy, thus avoiding the limited information resulting from using only one type of image and improving the accuracy of brain disease classification. Attached Figure Description

[0062] Figure 1This is an overall flowchart of the present invention;

[0063] Figure 2 This is a schematic diagram illustrating the generation of a primary feature map in an embodiment of the present invention;

[0064] Figure 3 This is a structural diagram of the residual module in an embodiment of the present invention;

[0065] Figure 4 This is a structural diagram of the MHSA (Multi-Head Attention) module in an embodiment of the present invention;

[0066] Figure 5 This is a flowchart of the preprocessing, feature extraction, and classification decision-making processes in an embodiment of the present invention.

[0067] Figure 6 This is a diagram of the RC structure in an embodiment of the present invention;

[0068] Figure 7 This is a structural diagram of the NC structure in an embodiment of the present invention;

[0069] Figure 8 This is a flowchart illustrating the generation of high-level feature maps in an embodiment of the present invention;

[0070] Figure 9 This is a structural diagram of the spatial channel attention module in an embodiment of the present invention;

[0071] Figure 10 This is a diagram of the system interface initialization page in an embodiment of the present invention;

[0072] Figure 11 This is a slice diagram from an embodiment of the present invention;

[0073] Figure 12 This is a diagnostic image of an AD sample in an embodiment of the present invention;

[0074] Figure 13 This is a diagnostic image of an MCI sample in an embodiment of the present invention;

[0075] Figure 14 This is a diagnostic diagram of an NC sample in an embodiment of the present invention;

[0076] Figure 15 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0077] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0078] Research has shown that collected brain images can be used to diagnose many brain diseases, such as Alzheimer's disease (AD). Brain images can be used to determine whether a patient has AD and the severity of their condition. AD is an irreversible, progressive neurodegenerative disease, primarily characterized by memory loss, cognitive decline, and inability to care for oneself. Mild cognitive impairment (MCI) is a pre-AD stage, further subdivided into pMCI and sMCI depending on whether it progresses to AD. Timely detection and intervention in pMCI or sMCI can potentially slow the progression to AD.

[0079] Table 1 shows the proportions of NC patients, sMCI, pMCI, and AD patients in the population.

[0080] Table 1

[0081]

[0082] Currently, early diagnosis of Alzheimer's disease (AD) often relies on doctors' subjective experience, which is time-consuming and can easily delay optimal treatment. Therefore, research into early AD diagnostic models using computer-aided technology can help explore the pathogenesis of AD. Current imaging methods for early AD diagnosis mainly include structural magnetic resonance imaging (sMRI), functional magnetic resonance imaging (fMRI), diffusion tensor imaging (DTI), and positron emission tomography (PET). sMRI provides a direct visual representation of changes in brain structure, and many early AD diagnostic studies have been developed using sMRI-based analysis. However, the brain information provided by sMRI images is relatively limited and insufficient to accurately determine the extent of the disease.

[0083] To address the aforementioned technical problems, this invention provides an image classification method, apparatus, device, and storage medium, solving the problem of low accuracy in existing image classification methods. Specifically, it first generates magnetic resonance imaging (MRI) images and positron emission tomography (PET) images of the brain, then extracts a fusion feature map from the MRI and PET images, and finally classifies brain diseases based on the fusion feature map. The classification method of this embodiment can significantly improve the accuracy of brain disease classification.

[0084] Exemplary methods

[0085] The image classification method of this embodiment can be applied to terminal devices, which can be terminal products with image acquisition functions, such as computers. In this embodiment, as... Figure 1 As shown, the image classification method specifically includes the following steps:

[0086] S100 generates magnetic resonance imaging and positron emission tomography (PET) images of the brain.

[0087] The magnetic resonance imaging (MRI) and positron emission tomography (PET) images used in this embodiment are preprocessed images. The preprocessing tools used include SPM and CAT. SPM is a free, open-source toolkit developed by Professor Karl Friston and others at the Functional Imaging Laboratory, University College London, based on the MATLAB platform, specifically designed for analyzing brain image sequence data. CAT is an extension of SPM developed by Professors Christian Gaser and Robert Dahnke at Jena University Hospital, designed to compute anatomical structures, including voxel-based morphometry (VBM), deformation-based morphometry (DBM), and surface-based morphometry (SBM).

[0088] In one embodiment, the preprocessing of the original MRI image to acquire magnetic resonance imaging includes the following steps in sequence: noise filtering; resampling; bias field correction; skull dissection; linear registration; tissue segmentation; spatial normalization; and clipping boundaries.

[0089] In one embodiment, the preprocessing of the original PET image for obtaining positron emission tomography (PET) images includes the following steps: registration; spatial normalization; skull dissection; and smoothing.

[0090] Among them, noise filtering involves removing non-brain images (i.e., noise) from the original image to reduce the impact of noise on subsequent brain disease classification.

[0091] Resampling removes identical original images, reducing data redundancy and computational load.

[0092] Bias field refers to the brightness difference of the same tissue in an MRI image caused by external factors. Bias field correction is to correct this difference to reduce error. This embodiment uses a nonparametric non-uniform normalization algorithm for bias field correction.

[0093] Skull dissection refers to the removal of non-brain tissues such as the skull and neck from MRI images.

[0094] Linear registration refers to aligning all MRI images from their original spaces to a common space in order to compare the corresponding anatomical locations for each subject. This embodiment uses mutual information affine registration to align the MRI images to an ICBM 152 template, which corresponds to the standard space – the MNI space.

[0095] Tissue segmentation involves separating the gray matter, white matter, and cerebrospinal fluid of the brain tissue (separating gray matter from white matter and cerebrospinal fluid to distinguish gray matter), because gray matter is considered to be the brain tissue most associated with AD.

[0096] Spatial normalization refers to the use of Geodesic Shooting registration to normalize segmented tissue images into a common space.

[0097] Cropping boundaries involves removing portions of the original image that do not contain brain data, thereby reducing useless data in subsequent classification.

[0098] S200, extract the fusion feature map of the magnetic resonance image and the positron emission tomography image, the fusion feature map including the features of the magnetic resonance image and the features of the positron emission tomography image.

[0099] The fusion feature map incorporates features from both magnetic resonance imaging (MRI) and positron emission tomography (PET) images. These two features interact to improve the accuracy of subsequent brain disease classification. Step S200 includes the following steps S201 to S2016:

[0100] S201, apply a depth convolution algorithm to the magnetic resonance image and the positron emission tomography image respectively to obtain a first result and a second result output by the depth convolution algorithm. The first result corresponds to the magnetic resonance image and the second result corresponds to the positron emission tomography image.

[0101] In one embodiment, a deep convolution algorithm is applied to the original features F of the magnetic resonance imaging (the original features are those obtained by converting the original MRI and PET images into computable tensors using the SimpleITK function library in Python, where each value in the tensor represents the original feature of each brain region) and the original features F of the positron emission tomography (PET) images to obtain a first result and a second result, respectively.

[0102] S202, apply the depthwise dilated convolution algorithm to the first result and the second result respectively to obtain the third result and the fourth result output by the depthwise dilated convolution algorithm. The third result corresponds to the magnetic resonance image and the fourth result corresponds to the positron emission tomography image.

[0103] S203, Convolve the third result and the fourth result respectively to obtain the attention map of the magnetic resonance image and the attention map of the positron emission tomography image. The attention map is used to characterize the weights corresponding to each feature.

[0104] In this embodiment, the attention map of each image is used to record the weight of each original feature of each image.

[0105] S204, Multiply the attention map of the magnetic resonance image with the magnetic resonance image to obtain the primary feature map of the magnetic resonance image.

[0106] S205, Multiply the attention map of the positron emission tomography (PET) image with the PET image to obtain the primary feature map of the PET image.

[0107] Steps S201 to S204 obtain the initial feature map Output using the following formula:

[0108] Attention = Conv 1x1 (DW-D-Conv 1x1 (DW-Conv(F)))

[0109]

[0110] Attention mechanisms can select discriminative features based on input features and automatically ignore noisy responses, generating attention maps that play a crucial role in subsequent feature extraction. Currently, the most common attention mechanism for establishing relationships between different parts is self-attention (SA). However, when applied to computer vision, SA has significant drawbacks, such as only achieving spatial adaptability while neglecting channel adaptability. Generating attention maps often involves using large-kernel convolutions, which incur substantial computational overhead. Therefore, this embodiment combines attention mechanisms with large-kernel convolutions to generate attention maps, reducing computational cost while improving channel adaptability.

[0111] DW-Conv(·) represents the depthwise convolution algorithm, DW-D-Conv 1x1 (·) denotes the depthwise dilated convolution algorithm. When F is the original feature of the magnetic resonance image, DW-Conv(F) is the first result. 1x1 (F) is the third result, where Attention is the attention map of the magnetic resonance imaging (MRI) image, and Output is the primary feature map of the MRI image. When F is the original feature map of the positron emission tomography (PET) image, DW-Conv(F) is the second result, DW-D-Conv. 1x1 (F) represents the fourth result. Attention refers to the attention map of the positron emission tomography (PET) image, and Output refers to the primary feature map of the PET image. F∈R C×H×W These are the raw features of the input brain imaging data, Attention ∈ R C×H×W This represents the attention map obtained after passing through a large kernel convolutional layer, indicating the importance of each input feature, where Output ∈ R.C×H×W This represents the features of attention guidance. The large kernel convolutional attention module in this embodiment can be simply described as follows: (e.g.) Figure 2 As shown, shallow features are first extracted using a 1×1 convolutional layer and a deep convolutional layer. Then, local contextual information and dynamic processes are further learned using a depth-dilated convolutional layer and two 1×1 convolutional layers. Finally, an attention map is generated and multiplied element-wise with the input MRI or PET data to obtain a feature map. Channel information is then processed by an MLP block consisting of fully connected layers and deep convolutional layers.

[0112] S206, the primary feature map of the magnetic resonance image is input into the convolutional neural network and the ViT network respectively to obtain the first local detail feature map output by the convolutional neural network and the first global feature map output by the ViT network.

[0113] ViT stands for Vision Transformer, which is an extended application of Transformer in the field of computer vision.

[0114] S207, the first local detail feature map and the first global feature map are fused to obtain the first feature map of the magnetic resonance image.

[0115] S208, the primary feature map of the positron emission tomography image is input into the convolutional neural network and the ViT network respectively to obtain the second local detail feature map output by the convolutional neural network and the second global feature map output by the ViT network.

[0116] S209, the second local detail feature map and the second global feature map are fused to obtain the second feature map of the positron emission tomography image.

[0117] In this embodiment, both magnetic resonance imaging (MRI) and positron emission tomography (PET) images employ convolutional neural networks (CNNs) and ViT networks, respectively, to obtain local detail feature maps and total feature maps. The CNNs and ViT networks used in this embodiment are pre-trained networks. The reason for using CNNs and ViT networks to generate feature maps is as follows:

[0118] CNNs are commonly used for 2D slices of brain MRI and PET scans in AD diagnosis. CNNs focus on learning and extracting local information, while global information is crucial for MRI and PET brain imaging data. The attention mechanism (SA) in ViT networks can capture global information and plays an important role in MCI diagnostic studies using MRI and PET brain imaging data. Therefore, incorporating an attention mechanism into CNNs can significantly improve performance.

[0119] The CNN model in this embodiment is designed using convolutional layers and residual modules. It consists of one convolutional layer and three residual blocks, where DSC is used to replace the convolutional design of the CNN branches. The residual modules are as follows: Figure 3 As shown. In order to capture global information from brain images, this embodiment adds a feature such as [example missing] after the residual module. Figure 4 The MHSA shown.

[0120] S2010, the first feature map and the second feature map are spliced ​​together to obtain a spliced ​​feature map.

[0121] S2011, Perform convolution operation on the spliced ​​feature map to obtain a similar feature map used to characterize the similarity between the first feature map and the second feature map.

[0122] S2012, connect the similar feature map and the original features of the magnetic resonance image to obtain the first connection map.

[0123] S2013, connect the similar feature map with the original features of the positron emission tomography image to obtain a second connection map.

[0124] S2014, perform convolution operation on the first connection graph to obtain the high-level feature map of the magnetic resonance image.

[0125] S2015, perform convolution operation on the second connection map to obtain the high-level feature map of the positron emission tomography image.

[0126] S2016, the high-level feature map of the magnetic resonance image and the high-level feature map of the positron emission tomography image are fused to obtain a fused feature map.

[0127] In this embodiment, in order to obtain the fused feature map, Figure 5 In the feature extraction part, each stage repeats steps S201 to S2016, and the output of the previous stage is used as the input of the next stage, until the fused feature map output by stage 4 is used as the final fused feature map.

[0128] In this embodiment, the first three stages all involve using CNN and ViT to calculate local detail feature maps and global feature maps, respectively. The CNN branch design for the first three stages is as follows:

[0129] f h1 =ReLu(BN(DSC(f h1 )))

[0130] f h2 =ReLu(BN(DSC(f h1 )))

[0131] Output = ReLU(BN(DSC(f)) h2 ))+f i )

[0132] When f h1 For the primary feature map of magnetic resonance imaging, f i When the features are those of a magnetic resonance image, Output is the first local detail feature map; when f h1 Primary feature map of positron emission tomography (PET) image, f i When the features are those of positron emission tomography (PET) images, Output is the second local detail feature map.

[0133] In the fourth stage, the CNN branches are designed as follows:

[0134] q, k, v = Conv 1×1 (f i )

[0135]

[0136]

[0137]

[0138] Among them, f i ∈R C×H×W Let R represent the input features, where q, k, and v represent the query, key, and value, respectively. h ∈R C×H×W and R w ∈R C×H×W It is a relative positional encoding consistent with the dimension of the input features.

[0139] ViT, an extension of the Transformer in computer vision, has achieved great success in various visual tasks. Specifically, ViT segments an image into several small blocks, each recorded as a token. After rearranging all the tokens into a sequence, it is directly input into a Transformer network to extract features. However, ViT requires a large amount of training data and a long training time, making it difficult to apply to MRI or PET. A significant reason is that ViT lacks specific inductive biases (IBs) when modeling visual tasks, such as modeling local correlations and scale invariance in brain imaging data. This prevents the model from effectively utilizing MRI or PET data and affects model performance. Since convolutional operations in CNNs extract local features from neighboring pixels, CNNs possess an inherent IB in the local model. Furthermore, inter-layer or intra-layer fusion gives CNNs an inherent IB in modeling scale invariance. Therefore, utilizing the inherent IBs of CNNs can facilitate the training of ViT and help improve its performance.

[0140] To address the issues encountered by ViT in MCI diagnosis, this embodiment uses reduction cells (RC) and normal cells (NC) to construct the model. RC and NC fully utilize the inherent IB in CNNs. RC has two parallel branches, responsible for modeling local information and long-range dependencies respectively, before being processed by an FFN for feature transformation. Figure 6 As shown, Figure 5 In the four stages, the RC structure is as follows:

[0141]

[0142]

[0143]

[0144]

[0145] PRM is a block used to perform downsampling operations on the input features. ij It's a parameter setting (dilation rate) for convolution operations in PRM, r i It's downsampling rate. Let represent the input feature of the i-th RC, Conv(.) be the j-th convolutional layer in the Pyramid Reduction Module (PRM), and Img2Seq(·) be the dimension reshaping operation that flattens the feature map into a one-dimensional sequence. PCM iThe (·) is a parallel convolutional module (PCM), consisting of three stacked convolutional layers and an Img2Seq(.) operation. SeqImg(·) is the inverse operation of Img2Seq(.), reshaping the sequence back to the dimension of the feature map. NC is similar to RC without PRM, such as... Figure 7 As shown. Since the feature map after RC processing already has a relatively small spatial size, there is no need to use PRM in NC.

[0146] In one embodiment, steps S2010 to S2016 involve Figure 5 The process of fusing features at each stage is as follows:

[0147] like Figure 8 As shown, input two modal features X MRI ∈R C×H×W (First feature map) and X PET ∈R C×H×W (Second feature map) after X MRI ∈R C×H×W and X PET ∈R C×H×W The features of two modalities are concatenated to obtain a concatenated feature map. The concatenated feature map is then passed through a 1×1 convolutional layer (which performs convolution operations on the concatenated feature map) to learn their similarity features f. fusion ∈R C×H×W (Similarity feature map). Then, the learned similar features are concatenated with the original features (the similarity feature map and the original features of the magnetic resonance image are concatenated to obtain the first concatenation map; or the similarity feature map is concatenated with the original features of the positron emission tomography image to obtain the second concatenation map). Next, two 1×1 convolution operations are used to obtain the specific feature f. MRI_fusion ∈R C×H×W (High-level feature maps of magnetic resonance imaging) and f PET_fusion ∈R C×H×W (High-level feature map of positron emission tomography).

[0148] Both steps S2014 and S2015 adopt the following approach: Figure 8 The principle in the above describes the process of obtaining high-level feature maps. Taking the first connectivity map of a magnetic resonance image as an example, the detailed process of obtaining high-level feature maps from the first connectivity map of a magnetic resonance image is explained as follows:

[0149] Since the MRI and PET data used in this embodiment have three sections—sagittal, coronal, and transverse—with the sagittal section being the primary section for learning, and each section playing a crucial role in the early diagnosis of Alzheimer's disease (AD), feature learning is performed on the three sections by exchanging dimensions. These three features are then concatenated, with the features from the sagittal and coronal sections used to enhance the feature representation of the transverse section.

[0150] Therefore, the first connection graph first performs the following... Figure 8 The 1x1 convolution operation shown yields the specific feature maps of the magnetic resonance image. Then, the dimensions are swapped (the specific feature maps undergo dimensionality transformation to obtain the specific feature maps for each slice), and three branches are designed, each branch corresponding to the feature learning of a slice. Along the channel dimension, i.e., the sagittal plane C, a convolution kernel k is used. C ∈R 1×n×n , obtain feature f C ∈R C×H×W Along the height dimension, i.e., the coronal plane H, use convolution kernel K. H ∈R n×1×n Obtain feature f H ∈R C×H×W Along the width dimension, i.e., the cross-section W, use convolution kernel K. W ∈R n×1×n Obtain feature f W ∈R C×H×W , then f C f H f W This forms the convolution result of the specific feature maps of each cross-section, combining the three features f C f H f W Connect them together to obtain the final feature f dim ={f C ,f H ,f w}∈R 3C×H×W (High-level feature map). Finally, the number of output channels is reshaped to the number of input channels through a 1×1 convolutional layer.

[0151] S300, the fused feature map is classified to obtain a classification result for characterizing brain diseases.

[0152] In this embodiment, the fused feature map is a fusion of high-level feature maps from magnetic resonance imaging and positron emission tomography (PET) images. Based on these two high-level feature maps, the degree of AD in the patient is determined. Step S300 includes the following steps S301 to S309:

[0153] S301, Calculate the weights of each channel of the fused feature map.

[0154] In this embodiment, the channels refer to the C channel (image channel of the fused feature map), the H channel (height of the fused feature map), and the W channel (width of the fused feature map).

[0155] S302, multiply the fused feature map by the weights of each channel to obtain the first channel feature.

[0156] S303, calculate the features of each channel of the fused feature map.

[0157] S304, calculate the average value of the characteristics of each channel to obtain the channel mean.

[0158] S305, Multiply the fused feature map by the channel mean to obtain the second channel feature.

[0159] S306, add the first channel feature and the second channel feature to obtain the final feature map.

[0160] S307 performs a global average pooling operation on the final feature map to obtain the pooled feature map.

[0161] S308 applies a classifier to the pooled feature map to obtain the classifier's output.

[0162] S309, classify brain diseases based on the results of the classifier output.

[0163] Steps S301 to S309 classify brain diseases based on the following principles:

[0164] To accurately diagnose Alzheimer's disease (AD), learning the spatial and channel features of brain imaging data is crucial. Therefore, this embodiment designs a Spatial Channel Attention (SCA) module to further learn the spatial and channel information of high-level features after extraction, thereby improving the performance of the method in this embodiment. Figure 9 As shown, SCA consists of a channel attention branch and a spatial attention branch. In the channel attention branch, the feature map (the fused feature map obtained in step S200) is first converted into a vector through a global average pooling layer. Then, the weights of each channel are calculated through two 1×1 convolutional layers (the weights of each channel in the fused feature map are calculated in step S301). Finally, the input feature map is multiplied by the channel weights to obtain the output features of the channel attention branch (i.e., multiplying the fused feature map by the weights of each channel in step S302 to obtain the first channel feature). In the spatial attention branch, the channel features of the feature map are first averaged (the average value of the features of each channel is calculated to obtain the channel mean). The weights in each pixel are determined by the sigmoid activation function. Finally, the feature map is the spatial weights multiplied pixel by pixel (the fused feature map is multiplied by the channel mean to obtain the second channel feature). Finally, the outputs of the channel and spatial attention branches are added element-wise to form a new feature map (the first channel feature and the second channel feature are added to obtain the final feature map).

[0165] After obtaining the final feature map, global average pooling is used to prevent overfitting. Next, a fully connected layer and a softmax classifier are used to output the predicted labels. Finally, the final diagnosis result is obtained based on the predicted labels. A label of 0 indicates no disease, while a label of 1 indicates disease.

[0166] Figure 5 This demonstrates the overall process of classifying brain diseases in steps S100 to S300:

[0167] Figure 5 The model consists of three parts: attention weight learning, feature extraction, and classification decision. To improve the accuracy of learned features, an attention module composed of large kernel convolutions is first used to generate an attention map to assist the high-level feature learning of the backbone network. Next, the learned network is sent to the feature extraction backbone. The backbone network of this embodiment consists of four stages. Each stage consists of two branches: the CNN branch consists of ResNet50 and multi-head attention, and the Transformer branch consists of restoration units and normal units. The unique ability of CNN to learn local information combined with the global modeling ability of Transformer allows for better learning of feature representations. Simultaneously, to fully utilize the similarity and specificity between the two modalities, a modality feature fusion block is designed to strengthen the inter-modality connections and improve the comprehensiveness of learned features. After learning high-level features, spatial information—channel attention—is further used to learn the spatial and channel information of high-level features. Finally, a Softmax classifier is used for classification.

[0168] In one embodiment, the above-described image classification method is integrated into a brain disease classification system. The system in this embodiment primarily performs the following functions: First, it selects the brain imaging data (MRI or PET) of the patient to be diagnosed; then, it visualizes the initial data, using a slider to switch between different slices and sections due to the different slices and sections used; after inputting information such as name, age, and gender, clicking "output" will output the diagnostic results and a brain region heatmap. The brain region heatmap reflects the patient's lesion condition; the darker the color of the relevant brain region, the more severe the lesion.

[0169] See system interface initialization Figure 10 After selecting the raw data, adjusting different knob values ​​will display different slices. Figure 11 The images shown are slices displayed when the knob values ​​are 10 and 40. Next, input the data and click "Output" to obtain the diagnostic results. The results for AD, pMCI, and NC data are shown in the following order. Figures 12-14 As shown, this demonstrates the practicality of the system in this embodiment.

[0170] In summary, magnetic resonance imaging (MRI) can accurately reflect changes in brain structure caused by lesions, while positron emission tomography (PET) can accurately reflect changes in brain function caused by lesions. This invention uses both MRI and PET to classify brain diseases. The combination of these two methods can reveal morphological changes in the brain caused by brain atrophy, thus avoiding the limited information resulting from using only one type of image and improving the accuracy of brain disease classification.

[0171] In addition, this invention proposes a novel end-to-end hybrid framework that utilizes large convolutional kernels and takes into account inductive bias to learn feature representations.

[0172] A modal feature fusion block was designed, which fully considers the similarity and specificity between structural features and functional features.

[0173] A dimension fusion block was designed and integrated into the feature fusion block, which fully considers the impact of the three sections of MRI and PET brain imaging data on diagnostic performance.

[0174] A diagnostic framework for MCI based on multimodal data was designed to fully utilize structural and functional features to improve diagnostic accuracy.

[0175] Exemplary device

[0176] This embodiment also provides a brain disease classification device, the device comprising the following components:

[0177] The image acquisition module is used to generate magnetic resonance imaging and positron emission tomography (PET) images of the brain;

[0178] The feature extraction module is used to extract a fusion feature map of the magnetic resonance image and the positron emission tomography image, wherein the fusion feature map includes features of the magnetic resonance image and features of the positron emission tomography image;

[0179] The classification module is used to classify the fused feature map to obtain classification results for characterizing brain diseases.

[0180] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 15As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image classification method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0181] Those skilled in the art will understand that Figure 15 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0182] In one embodiment, a terminal device is provided, comprising a memory, a processor, and a brain disease classification program stored in the memory and executable on the processor. When the processor executes the brain disease classification program, it implements the following operation instructions:

[0183] Generate magnetic resonance imaging and positron emission tomography (PET) images of the brain;

[0184] Extract the fusion feature map of the magnetic resonance image and the positron emission tomography image, wherein the fusion feature map includes features of the magnetic resonance image and features of the positron emission tomography image;

[0185] The fused feature maps are classified to obtain classification results used to characterize brain diseases.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image classification method, characterized in that, include: Generate magnetic resonance imaging and positron emission tomography (PET) images of the brain; Extract the fusion feature map of the magnetic resonance image and the positron emission tomography image, wherein the fusion feature map includes features of the magnetic resonance image and features of the positron emission tomography image; The fused feature maps are classified to obtain classification results used to characterize brain diseases; The extraction of the fusion feature map of the magnetic resonance image and the positron emission tomography (PET) image, wherein the fusion feature map includes features of the magnetic resonance image and features of the PET image, including: The deep convolution algorithm is applied to the magnetic resonance image and the positron emission tomography image respectively to obtain a first result and a second result output by the deep convolution algorithm. The first result corresponds to the magnetic resonance image and the second result corresponds to the positron emission tomography image. The first result and the second result are respectively applied to the depthwise dilated convolution algorithm to obtain the third result and the fourth result output by the depthwise dilated convolution algorithm. The third result corresponds to the magnetic resonance image and the fourth result corresponds to the positron emission tomography image. The third and fourth results are convolved to obtain the attention map of the magnetic resonance image and the attention map of the positron emission tomography image. The attention map is used to characterize the weights corresponding to each feature. Multiply the attention map of the magnetic resonance image with the magnetic resonance image to obtain the primary feature map of the magnetic resonance image; Multiply the attention map of the positron emission tomography (PET) image with the PET image to obtain the primary feature map of the PET image. A fused feature map is obtained based on the primary feature map of the magnetic resonance image and the primary feature map of the positron emission tomography image.

2. The image classification method as described in claim 1, characterized in that, The generation of magnetic resonance imaging and positron emission tomography (PET) images of the brain includes: Acquire raw MRI and positron emission tomography images of the brain; The original magnetic resonance image is subjected to bias field correction to obtain the corrected original magnetic resonance image. The original MRI image after removal correction is processed by applying a craniotomy algorithm to remove non-brain tissue pixels, resulting in the original MRI image after craniotomy. The images corresponding to the gray matter of the brain are extracted from the original magnetic resonance images after skull dissection to obtain magnetic resonance images; The original positron emission tomography (PET) image was subjected to a skull peeling algorithm to remove non-brain tissue pixels, resulting in the original PET image after skull peeling. The original positron emission tomography (PET) images after skull dissection are smoothed to obtain PET images.

3. The image classification method as described in claim 1, characterized in that, The process of obtaining a fused feature map based on the primary feature map of the magnetic resonance imaging and the primary feature map of the positron emission tomography image includes: The primary feature map of the magnetic resonance image is input into a convolutional neural network and a ViT network, respectively, to obtain a first local detail feature map output by the convolutional neural network and a first global feature map output by the ViT network. The first local detail feature map and the first global feature map are fused to obtain the first feature map of the magnetic resonance image; The primary feature map of the positron emission tomography image is input into a convolutional neural network and a ViT network, respectively, to obtain a second local detail feature map output by the convolutional neural network and a second global feature map output by the ViT network. The second local detail feature map and the second global feature map are fused to obtain the second feature map of the positron emission tomography image; The first feature map and the second feature map are concatenated to obtain a concatenated feature map. Perform a convolution operation on the spliced ​​feature map to obtain a similar feature map that characterizes the similarity between the first feature map and the second feature map; The similarity feature map and the original features of the magnetic resonance image are connected to obtain a first connection map. The original features of the magnetic resonance image are used to characterize the features of each brain region corresponding to the magnetic resonance image. The similarity feature map is connected to the original features of the positron emission tomography (PET) image to obtain a second connection map. The original features of the PET image are used to characterize the features of each brain region corresponding to the PET image. Perform a convolution operation on the first connection graph to obtain the high-level feature map of the magnetic resonance image; Perform a convolution operation on the second connectivity graph to obtain a high-level feature map of the positron emission tomography image; The high-level feature maps of the magnetic resonance imaging and the positron emission tomography (PET) images are fused to obtain a fused feature map.

4. The image classification method as described in claim 3, characterized in that, The step of performing a convolution operation on the first connectivity graph to obtain the high-level feature map of the magnetic resonance image includes: Perform a 1x1 convolution operation on the first connection graph to obtain the specific feature map of the magnetic resonance image; The specific feature map is transformed in dimension to obtain the specific feature map of each cross-section; Convolution operations are performed on the specific feature maps of each cross section to obtain the convolution results of the specific feature maps of each cross section; The high-level feature map of the magnetic resonance image is obtained by convolving the specific feature maps of each cross section.

5. The image classification method as described in claim 1, characterized in that, The classification of the fused feature map to obtain classification results for characterizing brain diseases includes: Calculate the weights of each channel in the fused feature map; Multiply the fused feature map by the weights of each channel to obtain the first channel feature; Calculate the features of each channel of the fused feature map; Calculate the average value of the features of each channel to obtain the channel mean; Multiply the fused feature map by the channel mean to obtain the second channel feature; The first channel feature and the second channel feature are added together to obtain the final feature map; The final feature maps are classified to obtain classification results used to characterize brain diseases.

6. The image classification method as described in claim 5, characterized in that, The classification of the final feature map to obtain classification results for characterizing brain diseases includes: Perform global average pooling on the final feature map to obtain the pooled feature map; Apply a classifier to the pooled feature map to obtain the classifier's output.

7. A brain disease classification device, characterized in that, The device comprises the following components: The image acquisition module is used to generate magnetic resonance imaging and positron emission tomography (PET) images of the brain; The feature extraction module is used to extract a fusion feature map of the magnetic resonance image and the positron emission tomography image, wherein the fusion feature map includes features of the magnetic resonance image and features of the positron emission tomography image; A classification module is used to classify the fused feature map to obtain classification results for characterizing brain diseases; The extraction of the fusion feature map of the magnetic resonance image and the positron emission tomography (PET) image, wherein the fusion feature map includes features of the magnetic resonance image and features of the PET image, including: The deep convolution algorithm is applied to the magnetic resonance image and the positron emission tomography image respectively to obtain a first result and a second result output by the deep convolution algorithm. The first result corresponds to the magnetic resonance image and the second result corresponds to the positron emission tomography image. The first result and the second result are respectively applied to the depthwise dilated convolution algorithm to obtain the third result and the fourth result output by the depthwise dilated convolution algorithm. The third result corresponds to the magnetic resonance image and the fourth result corresponds to the positron emission tomography image. The third and fourth results are convolved to obtain the attention map of the magnetic resonance image and the attention map of the positron emission tomography image. The attention map is used to characterize the weights corresponding to each feature. Multiply the attention map of the magnetic resonance image with the magnetic resonance image to obtain the primary feature map of the magnetic resonance image; Multiply the attention map of the positron emission tomography (PET) image with the PET image to obtain the primary feature map of the PET image. A fused feature map is obtained based on the primary feature map of the magnetic resonance image and the primary feature map of the positron emission tomography image.

8. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a brain disease classification program stored in the memory and executable on the processor. When the processor executes the brain disease classification program, it implements the steps of the image classification method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a brain disease classification program, which, when executed by a processor, implements the steps of the image classification method as described in any one of claims 1-6.

Images