Brain tumor classification method based on class activation mapping guidance

Abstract

The invention provides a brain tumor classification method based on class activation mapping guidance. The method aims at improving classification accuracy and interpretability of brain tumors in MRI images. The brain tumor classification model constructed by the method is composed of an attention branch, a perception branch and a loss function of the classification model. The attention branch firstly uses class activation mapping to determine a key area in an image so as to guide a model focus to a significant feature of a brain tumor. The sensing branch uses the key regions to extract features and construct positive and negative samples, so that the learning ability and discrimination ability of the model in a specific region are enhanced. A loss function of the classification model is combined with direct supervision loss and positive and negative sample prediction loss generated by sensing branches, model parameters are adjusted by optimizing the loss, and the accuracy and robustness of a classification result are ensured. According to the method, the efficiency and accuracy of brain tumor classification are improved, the interpretability of the model is enhanced through visual class activation mapping, and doctors can better understand and trust the decision process of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and in particular to a brain tumor classification method in MRI images based on class activation mapping guidance. Background Art

[0002] Brain tumors are a serious threat to human health, including primary and secondary brain tumors. Although medical imaging technology, especially magnetic resonance imaging (MRI), plays a key role in the diagnosis of brain tumors, existing diagnostic methods have certain limitations. Traditional brain tumor classification methods rely on manual feature extraction and complex preprocessing steps, which are not only inefficient but also often lack sufficient flexibility to adapt to new cases or changing diagnostic conditions.

[0003] The main challenges faced by current brain tumor classification techniques include the similarity of intensity distribution between modalities, the high variability of tumor morphology, and the fuzziness of boundaries. These factors greatly increase the difficulty of the classification task and also affect the accuracy and reliability of the classification. In addition, existing classification models often lack sufficient interpretability, which is an important consideration in clinical applications.

[0004] To overcome these limitations, a new brain tumor classification method needs to be developed that can automatically extract features directly from medical images using advanced computing techniques, especially deep learning, to provide higher classification accuracy and better interpretability. This method can not only improve the efficiency and accuracy of classification, but also provide doctors with an intuitive understanding of classification decisions, thereby formulating more effective treatment plans. Summary of the invention

[0005] The present invention provides a brain tumor classification method based on class activation mapping guidance, aiming to improve the classification accuracy and interpretability of brain tumors in MRI images. The system includes the following steps:

[0006] The object of the present invention is achieved through the following technical solution, and the method comprises the following steps:

[0007] (1) Constructing a brain tumor classification model: The tumor classification model is mainly composed of three parts: an attention branch, a perception branch, and a loss function of a classification model.

[0008] 1-1) The construction method of the attention branch is as follows:

[0009] The attention branch first receives the feature map F output from the last convolutional layer of the feature extractor. last ∈R D×H′×W′ , then F lastIt is superimposed with the downsampled original grayscale image and then transmitted together as input to the Saliency Learning Module (SLM). The SLM learns the attention weights corresponding to the feature map through the supervision signal.

[0010] Input data X∈R C×H×W , first extract features through the ConvNext network, as shown below:

[0011] F last =ConvNext(X_#(1_

[0012] The final convolutional layer outputs the feature map Where C1 represents the number of channels of the feature map. Based on the idea of ​​Score-CAM, the original image needs to be converted into a single-channel grayscale image, and then downsampled through bilinear interpolation to match the size of the feature map.

[0013] X down =Down(Rgb2Gray(X))#(2)

[0014] get Then F last Each channel feature map is normalized and mapped to [0,1].

[0015]

[0016] Norm means the maximum and minimum value normalization. Then, the downsampled image X down To weight,

[0017]

[0018] The purpose of this step is to obtain the salient areas corresponding to each of the C1 channels on the original image, which will be subsequently represented by F' last It is fed into the SLM as new input to learn the importance of each channel.

[0019] F' last After multiple consecutive SLMs, the weight W of each channel is obtained through the global average pooling (GAP) layer and the Softmax function.

[0020] W = Softmax(GAP(SLM(F′ last )))#(5)

[0021] at this time, Represents the original feature map F lastThe contribution of each channel to the input image. The attention map (AM) corresponding to the original feature map is obtained by weighted summation, and the negative area, that is, the non-response area, is removed by ReLU.

[0022]

[0023] in The attention map is then fed into the perception branch to generate positive and negative samples.

[0024] 1-2) The construction method of the perception branch is as follows:

[0025] The perception branch uses the attention map (AM) generated by the attention branch to construct positive samples. and negative samples The positive sample indicates the area enhanced by the attention map, which helps the model identify the correct category; the negative sample indicates the area suppressed by the attention map, which enhances the model's ability to distinguish non-positive categories. The process is expressed as follows:

[0026]

[0027] The constructed positive and negative samples are passed through the normalization (Layer Normalization, LN) layer and the fully connected (LayerNormalization, FC) layer, and finally through the Softmax function to obtain the predicted category probability. pos The model is required to predict the probability of positive samples as large as possible, Score neg The probability of predicting a positive sample is required to be as small as possible.

[0028]

[0029] The loss function in the perception branch is used to increase the prediction value of positive and negative samples. The positive sample uses the cross-entropy (CE) to calculate the loss, and the prediction result of the negative sample is regarded as a binary classification task, and the obtained score is directly neg Back propagated as loss. Score neg The smaller the value, the lower the probability of being identified as a positive class. neg When it is close to 1, it means that the model is not capable of identifying negative samples, and a greater penalty is given to the model.

[0030] loss pos =CE(Score pos ,label)#(11)

[0031] loss neg=Score neg #(12)

[0032] 1-3) The method for constructing the loss function of the classification model is as follows:

[0033] The loss function of the brain tumor classification model is composed of the direct supervision loss loss sup , positive sample prediction loss loss pos And negative sample prediction loss loss neg Weighted composition, the loss of negative samples directly uses the Score in the perception branch neg As the loss, the direct supervision loss loss sup And the positive sample prediction loss loss pos The cross entropy loss is used for loss calculation. The reason why this paper chooses to use cross entropy loss is that cross entropy loss is a common and effective loss function, especially suitable for classification tasks. It can quantify the difference between the probability distribution predicted by the model and the true label, which is particularly effective for multi-class classification problems. In the brain tumor classification model, the use of cross entropy loss can more accurately measure the deviation between the probability distribution output by the model and the true label, which helps the model learn more accurate classification boundaries and feature representations.

[0034] The loss function formula of the classification model in this paper is expressed as follows.

[0035]

[0036] Among them, C is the number of categories and N is the number of samples.

[0037] loss = αloss pos +βloss neg +loss sup #(14_

[0038] Among them, α and β are hyperparameters, and α+β=1, which are used to adjust the model's ability to distinguish positive and negative samples, ensuring that the model focuses on positive and negative samples more balanced during training, thereby improving the overall performance of the model.

[0039] (2) Model training and testing process: The brain tumor classification model was built and optimized by training using the brain tumor classification dataset. The obtained classification model was then used in the test set to verify the model performance. In order to comprehensively evaluate the trained classification model, four key evaluation indicators were used: accuracy, precision, recall, and F1 score.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The method shown in (1) proposes a brain tumor classification model guided by class activation maps to classify five types of brain tumor images. The model body automatically and accurately classifies brain tumors through two parts: the perception branch and the attention branch, and enhances the interpretability of the model through class activation maps.

[0042] (2) The method designs a saliency learning module (SLM) in the attention branch to focus on learning the importance of feature maps in the channel dimension, and obtains more reliable attention weights through direct supervision signals, thereby improving the decision accuracy of the model.

[0043] The method shown in (3) designs a sample selection module (SSM) in the perception branch, and constructs positive and negative samples through the attention mechanism using the obtained attention map. The positive sample uses the attention map to enhance the relevant area, and the negative sample uses the attention map to suppress the relevant area, providing additional weak supervision for the model during training, so that the model can learn to distinguish positive and negative samples at the same time.

[0044] (4) The method can be trained and tested on the dataset jointly constructed by Figshare and BraTS2023. The experimental results show that the method proposed in the present invention is more competitive than other state-of-the-art brain tumor classification methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is an example diagram of MRI images in the brain tumor classification dataset jointly constructed by Figshare and BraTS2023 provided in an embodiment of the present invention.

[0046] Figure 2 This is a framework diagram of a brain tumor classification method based on class activation mapping guidance provided by an embodiment of the present invention.

[0047] Figure 3 It is a schematic diagram of a visualized class activation area provided by an embodiment of the present invention. Specific implementation methods

[0049] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0050] Figure 1 1 is an example of an MRI image in a brain tumor classification dataset provided by an embodiment of the present invention. From left to right in the image are a normal human brain, a brain metastasis, a glioma, a brain tumor, and a pituitary tumor.

[0051] like Figure 1Shown is a brain tumor classification dataset jointly constructed by Figshare and BraTS2023 in the present invention. Figshare contains a four-category dataset of brain tumors, a total of 233 patients' brain MRI data, including 3064 images at different anatomical levels (cross-section, coronal and sagittal), each image is 512×512 in size, and it is necessary to distinguish normal images, meningiomas, gliomas and pituitary tumor images. Since there are many duplicate samples in this dataset, this paper first cleans the data, including data deduplication, deletion of incorrectly labeled samples, and scaling the image to 224×224 size to adapt to the network input. In view of the fact that brain metastases also have a high incidence rate in clinical practice, accurate identification of brain metastases is also of great clinical significance. Therefore, the present invention collected MRI data of 165 patients from the training set part of the BraTS2023 brain metastasis segmentation task and constructed a brain metastasis classification dataset. For each case, the largest slice of brain metastasis was first selected, then the slice was resized to 224×224 and finally saved as a JPG image.

[0052] Figure 2 This is the method framework of the present invention. The perception branch and the attention branch are used to infer the result from the modality.

[0053] like Figure 2 Shown is a brain tumor classification model based on class activation mapping guidance proposed in the present invention, which includes two branches: an attention branch and a perception branch. First, a feature extractor extracts feature maps from an input image. Then, these feature maps are passed to the attention branch. In the attention branch, the feature map and the grayscale map generated by the original image are superimposed and input into the Saliency Learning Module (SLM) to learn the attention weights of the feature map, thereby generating an attention map for use by the perception branch. After receiving the attention map, the perception branch combines it with the feature map obtained by feature extraction to generate positive and negative samples. Next, after global pooling and fully connected layer processing, the network outputs the probability of the category. The model uses a cross entropy loss function to process positive samples, while negative samples directly use the prediction score to calculate the loss.

[0054] like Figure 2 The attention branch shown first receives the last convolutional layer from the feature extractor.

[0055] Output feature map F last ∈R D×H′×W′ , then F lastIt is superimposed with the downsampled original grayscale image and then transmitted together as input to the Saliency Learning Module (SLM). The SLM learns the attention weights corresponding to the feature map through the supervision signal.

[0056] Input data X∈R 3×224×224 , first extract features through the ConvNext network, as shown below:

[0057] F last =ConvNext(X)#(1)

[0058] The final convolutional layer outputs the feature map F last ∈R 128×14×14 , where 128 represents the number of channels of the feature map. Based on the idea of ​​Score-CAM, the original image needs to be converted into a single-channel grayscale image, and then downsampled by bilinear interpolation to match the size of the feature map.

[0059] X down =Down(Rgb2Gray(X))#(2)

[0060] Get X down ∈R 1×14×14 , and then F last Each channel feature map is normalized and mapped to [0,1].

[0061]

[0062] Norm means the maximum and minimum value normalization. Then, the downsampled image X down To weight,

[0063]

[0064] The purpose of this step is to obtain the salient areas corresponding to each of the 128 channels on the original image, which will be subsequently represented by F' last It is fed into the SLM as new input to learn the importance of each channel.

[0065] F' last After multiple consecutive SLMs, the weight W of each channel is obtained through the global average pooling (GAP) layer and the Softmax function.

[0066] W = Softmax(GAP(SLM(F′ last )))#(5)

[0067] At this time, W∈R 128×1×1 Represents the original feature map F lastThe contribution of each channel to the input image. The attention map (AM) corresponding to the original feature map is obtained by weighted summation, and the negative area, that is, the non-response area, is removed by ReLU.

[0068]

[0069] where AM∈R 1×14×14 ,The attention map is then provided to the perception branch to generate positive and negative samples.

[0070] like Figure 2 The perception branch shown uses the attention map (AM) generated from the attention branch to construct positive samples. and negative samples The positive sample indicates the area enhanced by the attention map, which helps the model identify the correct category; the negative sample indicates the area suppressed by the attention map, which enhances the model's ability to distinguish non-positive categories. The process is expressed as follows:

[0071]

[0072] The constructed positive and negative samples are passed through the normalization (Layer Normalization, LN) layer and the fully connected (LayerNormalization, FC) layer, and finally through the Softmax function to obtain the predicted category probability. pos The model is required to predict the probability of positive samples as large as possible, Score neg The probability of predicting a positive sample is required to be as small as possible.

[0073]

[0074] The loss function in the perception branch is used to increase the prediction value of positive and negative samples. The positive sample uses the cross-entropy (CE) to calculate the loss, and the prediction result of the negative sample is regarded as a binary classification task, and the obtained score is directly neg Back propagated as loss. Score neg The smaller the value, the lower the probability of being identified as a positive class. neg When it is close to 1, it means that the model is not capable of identifying negative samples, and a greater penalty is given to the model.

[0075] loss pos =CE(Score pos ,label)#(11)

[0076] loss neg =Score neg #(12)

[0077] like Figure 2 The loss function of the brain tumor classification model guided by class activation mapping is composed of direct supervision loss loss sup , positive sample prediction loss loss pos And negative sample prediction loss loss neg Weighted composition, the loss of negative samples directly uses the Score in the perception branch neg As the loss, the direct supervision loss loss sup And the positive sample prediction loss loss pos The cross entropy loss is used for loss calculation. The reason why this paper chooses to use cross entropy loss is that cross entropy loss is a common and effective loss function, especially suitable for classification tasks. It can quantify the difference between the probability distribution predicted by the model and the true label, which is particularly effective for multi-class classification problems. In the brain tumor classification model, the use of cross entropy loss can more accurately measure the deviation between the probability distribution output by the model and the true label, which helps the model learn more accurate classification boundaries and feature representations.

[0078] The loss function formula of the classification model in this paper is expressed as follows.

[0079]

[0080] Among them, C is the number of categories and N is the number of samples.

[0081] loss = αloss pos +βloss neg +loss sup #(14_

[0082] Among them, α and β are hyperparameters, and α+β=1, which are used to adjust the model's ability to distinguish positive and negative samples, ensuring that the model focuses on positive and negative samples more balanced during training, thereby improving the overall performance of the model. In this paper, the values ​​of α and β are both set to 0.5.

[0083] An embodiment of the present invention includes a model architecture implemented on an NVIDIA GeForce RTX2070 GPU using the Pytorch 1.7.0 software framework. In this embodiment, the model is trained with a batch size of 64, and iterative optimization is performed using the Adamw optimizer, with an initial learning rate set to 0.0005 and a weight decay parameter set to 0.05. In addition, the feature extractor used in this embodiment is a ConvNext model, which has a four-layer structure, and each layer contains two ConvNext blocks. In order to ensure that the model is not affected by overfitting, the present invention adopts an early stopping strategy for training control. Specifically, if the loss of the model on the validation set does not decrease further within 20 consecutive training cycles, the training process is automatically stopped. This strategy ensures the efficiency and effect of model training and prevents the problem of overfitting training data.

[0084] Model training and testing process: The brain tumor classification model and the comparison model proposed in the present invention were trained to build and optimize using the dataset jointly constructed by Figshare and BraTS2023. The obtained classification model was then used in the test set to verify the model performance. In order to comprehensively evaluate the trained classification model, four key evaluation indicators were used: accuracy, precision, recall, and F1 value. The results obtained by the test are shown in Table 1.

[0085] Table 1 Classification results of different models

[0086]

[0087] Note: The bold indicates the best value in each column

[0088] Figure 3 Schematic diagram of class activation regions provided by an embodiment of the present invention. It visualizes the class activation regions when each baseline model and the model in this paper make decisions.

[0089] like Figure 3As shown in the figure, the activation areas of different models have their own characteristics. DenseNet, RegNet, and MobileNet have wider ranges, indicating that the model focuses on the tumor and its surrounding low-correlation areas when making decisions. Although ResNet and VGG will also pay attention to the lesion area, they will still pay attention to the low-correlation area. The way LFICAM and ABN obtain weights is similar. They are all obtained through continuous convolutional layer training, and the activation areas show a certain range of diffusion. ShuffleNet pays attention to the area more accurately, but does not pay special attention to the lesion area. In contrast, this model can pay attention to the tumor area directly and accurately in a relatively stable manner, and focus more on the core area of ​​the tumor when making decisions. The reason may be that this paper introduces the SLM module and direct supervision signal in the attention branch. The SLM module focuses on enhancing the attention weight of the channel dimension, and the direct supervision signal is directly associated with the classification label after backpropagation, so the attention area is more relevant and stable.

[0090] The experimental results of the method on the dataset jointly constructed by Figshare and BraTS2023 show that the classification accuracy, precision, recall rate and F1 value of the method are 0.9626, 0.9642, 0.9666 and 0.9653. And the tumor area is more accurately focused in the visualized attention map. Specifically, the invention proposes a brain tumor classification model guided by class activation maps. The model body classifies brain tumors through two parts: the perception branch and the attention branch, and enhances the interpretability of the model through the class activation map.

Claims

1. A brain tumor classification method guided by class activation mapping, Features: include: Step 1: Establish a brain tumor classification model guided by class activation mapping; Step 2: Train and test the model; The steps of the established brain tumor classification model based on class activation mapping guidance include: The attention branch is used to adjust the focus area of ​​the feature map to emphasize the significant features of brain tumors. The perception branch is used to construct positive and negative samples. The positive samples strengthen the features of the brain tumor area, while the negative samples strengthen the features of the non-tumor area. The loss function of the classification model is designed, combining the direct supervision loss with the positive and negative sample prediction loss to train and optimize the model. 1) The construction method of the attention branch is as follows: The attention branch first receives the feature map F output from the last convolutional layer of the feature extractor. last ∈R D×H′×W′ , then F last It is superimposed with the downsampled original grayscale image and then transmitted together as input to the Saliency Learning Module (SLM). The SLM learns the attention weights corresponding to the feature map through the supervision signal. Input data X∈R C×H×W , first extract features through the ConvNext network, as shown below: F last =ConvNext(X)#(1) The final convolutional layer outputs the feature map Among them C 1 Indicates the number of channels of the feature map. Based on the idea of ​​Score-CAM, the original image needs to be converted into a single-channel grayscale image, and then downsampled through bilinear interpolation to match the size of the feature map. X down =Down(Rgb2Gray(X))#(2) get Then F last Each channel feature map is normalized and mapped to [0,1]. Norm means the maximum and minimum value normalization, and then the downsampled image X down To weight, The purpose of this step is to obtain C 1 The highlighted area corresponding to each channel on the original image is subsequently referred to as F′ last It is fed into SLM as new input to learn the importance of each channel; F′ last After multiple consecutive SLMs, the weight W of each channel is obtained through the global average pooling (GAP) layer and the Softmax function. W=Softmax(GAP(SLM(F′ last )))#(5) at this time, Represents the original feature map F last The contribution of each channel to the input image is obtained by weighted summation to obtain the attention map (AM) corresponding to the original feature map, and the negative area, that is, the non-response area, is removed by ReLU. in The attention map is then fed to the perception branch to generate positive and negative samples; 2) The construction method of the perception branch is as follows: The perception branch uses the attention map (AM) generated by the attention branch to construct positive samples. and negative samples The positive sample represents the area enhanced by the attention map, which helps the model identify the correct category; the negative sample represents the area suppressed by the attention map, which enhances the model's ability to distinguish non-positive categories. The process is expressed as follows using the formula: The constructed positive and negative samples are passed through the normalization (Layer Normalization, LN) layer and the fully connected (Layer Normalization, FC) layer, and finally through the Softmax function to obtain the predicted category probability, Score pos The model is required to predict the probability of positive samples as large as possible, Score neg The probability of predicting a positive sample is required to be as small as possible; The loss function in the perception branch is used to increase the prediction value of positive and negative samples. The positive sample uses the cross-entropy (CE) to calculate the loss, and the prediction result of the negative sample is regarded as a binary classification task. neg As the loss is back-propagated, Score neg The smaller the value, the lower the probability of being identified as a positive class. neg When it is close to 1, it means that the model is not capable of identifying negative samples, and a greater penalty is given to the model. loss pos =CE(Score pos ,label)#(11) loss neg =Score neg #(12) 3) The loss function of the classification model is constructed as follows: The loss function of the brain tumor classification model is composed of the direct supervision loss loss sup , positive sample prediction loss loss pos And negative sample prediction loss loss neg Weighted composition, the loss of negative samples directly uses the Score in the perception branch neg As the loss, the direct supervision loss loss sup And the positive sample prediction loss loss pos Cross entropy loss is used for loss calculation. The reason why this paper chooses to use cross entropy loss is that cross entropy loss is a commonly used and effective loss function, especially suitable for classification tasks. It can quantify the difference between the probability distribution predicted by the model and the true label, which is particularly effective for multi-category classification problems. In the brain tumor classification model, the use of cross entropy loss can more accurately measure the deviation between the probability distribution output by the model and the true label, which helps the model learn more accurate classification boundaries and feature representations. The loss function formula of the classification model in this article is expressed as follows: Among them, C is the number of categories and N is the number of samples; loss=αloss pos +βloss neg +loss sup #(14) Among them, α and β are hyperparameters, and α+β=1, which are used to adjust the model's ability to distinguish positive and negative samples, ensuring that the model focuses on positive and negative samples more balanced during training, thereby improving the overall performance of the model.

2. The brain tumor classification model guided by class activation mapping according to claim 1, Features: The model training and testing steps include: The brain tumor classification dataset was used for training to build and optimize, and the resulting classification model was then used on the test set to verify the model performance.

Images