Medical image classification method based on feature fusion and attention mechanism

Abstract

The application discloses a medical image classification method based on feature fusion and attention mechanism, comprising the following steps: image preprocessing; feature extraction based on a convolutional neural network backbone network; multi-scale feature fusion; interactive attention pyramid, screening out discriminative information and focusing on the important information; and outputting a classification result through an average pooling layer. The classification method adopts a convolution mode to extract features, has fewer parameters, runs faster, is suitable for the characteristics of a small amount of medical image data samples, does not need to update too many weight parameters, and can be well adapted to some difficult medical image data sets with large class differences and small class differences through experiments. The method is not only suitable for binary classification tasks, but also suitable for multi-classification tasks, and the type of the backbone network can be flexibly adjusted according to the task requirements to adapt to different classification tasks.

Description

Technical Field

[0001] This invention belongs to the field of image classification technology, specifically relating to a medical image classification method based on feature fusion and attention mechanisms. Background Technology

[0002] Medical imaging is a research field at the intersection of biomedicine and computer science. Medical images include various modalities of imaging information such as USI images, CXR images, CT images, NMRI images, and ECT images.

[0003] In recent years, deep learning has seen rapid development in the medical field, especially in medical imaging, such as image recognition of skin cancer, classification and segmentation of lung nodules, and reconstruction of MRI images. With the rapid advancement of computer science and technology, researchers have placed higher demands on the accuracy of medical image classification networks, making it a key research area for clinical disease diagnosis based on medical images and artificial intelligence.

[0004] In existing technologies, when classifying a large number of images, a separate model is often built for classification based on different medical tasks. Furthermore, a large amount of raw data needs to be pre-screened before classification, which consumes a lot of human and material resources.

[0005] Therefore, based on the aforementioned shortcomings, it is necessary to design a classification method applicable to large-scale medical images under broad-domain conditions to solve the problems of existing technologies. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide a medical image classification method based on feature fusion and attention mechanisms, applicable to image classification in the medical field, with low cost and wide application.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A medical image classification method based on feature fusion and attention mechanisms, the key technology of which lies in the following steps:

[0009] Step 1: Obtain a set of medical images of any size from the medical image database, and obtain a medical image training set and a medical image test set after image preprocessing;

[0010] Step 2: Select a convolutional neural network backbone based on the medical task;

[0011] Step 3: Introduce a feature pyramid to perform multi-scale feature fusion of medical images;

[0012] Step 4: Introduce the interactive attention pyramid to generate the weight system of the backbone network;

[0013] Step 5: After combining the backbone network, feature pyramid, and interactive attention pyramid, conduct ablation experiments to obtain the effect of each module combination.

[0014] Step Six: Using the medical image training set, train the optimal classification model based on the convolutional neural network to obtain a classification model based on the medical task;

[0015] Step 7: Input the medical image test set into the medical task-based classification model, and output the medical image classification results after average pooling.

[0016] The above method extracts features entirely through convolution, resulting in fewer parameters, faster processing speed, and suitability for small sample sizes of medical image data. It does not require updating excessive weight parameters. Furthermore, experiments have verified that this method is well-suited for some difficult-to-class medical image datasets characterized by large inter-class differences but small inter-class differences. It is applicable not only to binary classification tasks but also to multi-class tasks, and the backbone network type can be flexibly adjusted to adapt to different classification tasks.

[0017] Furthermore, in step one, the image preprocessing step is as follows:

[0018] Set the image size to a×a to unify the size of all images in the medical image set;

[0019] All medical images with uniform size are divided into a training set to be enhanced and a test set of medical images;

[0020] After performing image augmentation operations on the training set to be augmented, a medical image training set is obtained.

[0021] The image enhancement operation includes the following:

[0022] Random cropping operation: Randomly crop out a portion of the medical images in the training set to be enhanced and fill it with 0 pixel values;

[0023] Randomly flip the image horizontally; randomly adjust the image's brightness, contrast, and saturation.

[0024] Transform it into a tensor form;

[0025] Standardize the data;

[0026] The standardized formula is as follows:

[0027]

[0028] In equation (1), u represents the image mean, and X represents the image matrix:

[0029]

[0030] In equation (2), σ represents the standard deviation and N represents the number of pixels in the image.

[0031] Furthermore, in step two, the convolutional neural network backbone includes at least VGG, ResNet, GoogleLeNet, and Inception.

[0032] Furthermore, in step two, after confirming the convolutional neural network backbone, four levels of feature information at different scales are designed in the backbone for extraction; the matrix size of the feature information is a multiple of each other.

[0033] Furthermore, in step four, the interactive attention pyramid consists of four parallel layers, referred to as the first to the fourth layer. Each layer contains a channel attention module and a spatial attention module, which are connected in parallel and have two interactive paths: the first interactive path and the second interactive path.

[0034] The first interaction path involves superimposing the channel attention mask output by the second-layer channel attention module onto the input of the third-layer channel attention module through a broadcast addition operation.

[0035] The second interaction path is to superimpose the channel attention mask output by the third-layer channel attention module onto the input of the fourth-layer channel attention module through a broadcast addition operation.

[0036] In each layer, the channel attention mask output by the channel attention module and the spatial attention mask output by the spatial attention module are first added together in the form of broadcast addition to obtain the comprehensive attention weight of the layer. The comprehensive attention weight is then multiplied element-wise with the initial feature map input to the layer in the form of Had code product to generate the enhanced feature output of the layer.

[0037] Furthermore, in step seven, the result of average pooling is the output of the results from the four layers in the form of an average.

[0038] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0039] 1) The interactive attention pyramid proposed in this invention is based on channel attention and spatial attention. It interacts with channel attention in two hierarchical structures and recycles feature information. This can fuse high-level discriminative information with low-level discriminative information, ensuring that the network learns more useful information.

[0040] 2) This invention proposes a medical image classification method based on feature fusion and attention mechanisms, combining a CNN backbone, a feature pyramid, and an invented interactive attention pyramid. Utilizing the high resolution of low-level features and the high semantic information of high-level features, this method achieves excellent classification results by fusing features from different layers. It significantly improves network performance without substantially increasing the computational load of the original model, reduces the impact of redundant information from the original image on the classification results, and further enhances the classification of medical images. Attached Figure Description

[0041] Figure 1 This is a diagram illustrating the model structure of the medical task-based classification model of this invention.

[0042] Figure 2 This is a flowchart of the medical image classification method of the present invention;

[0043] Figure 3 This is a diagram of the spatial attention structure.

[0044] Figure 4 This is a diagram of the channel attention structure.

[0045] Figure 5 This is a classification diagram of skin lesions from the first embodiment. Detailed Implementation

[0046] The present invention will now be described in further detail with reference to the accompanying drawings.

[0047] In practical implementation: such as Figures 1 to 5 As shown,

[0048] A medical image classification method based on feature fusion and attention mechanisms, see [link to relevant documentation]. Figure 1 and Figure 2 The classification method is carried out according to the following steps:

[0049] Step 1: Obtain a set of medical images of any size from the medical image database, and obtain a medical image training set and a medical image test set after image preprocessing;

[0050] In this embodiment, two-dimensional medical images of arbitrary size are preprocessed to convert their size to 224×224. Simultaneously, the original medical images are resized to 299×299 and then randomly cropped to 224×224. The preprocessing module identifies the current network model's usage status. If the network model is in the training phase, image data augmentation operations are performed, such as Cutout operations: randomly cutting out parts of the sample and filling them with 0-pixel values; random horizontal flipping; and random modification of image brightness, contrast, and saturation. If the network model is in the testing phase, these operations are not performed. Finally, all data is converted to tensor form and normalized. The normalization formula used in this invention is as follows:

[0051]

[0052] In equation (1), u represents the image mean, and X represents the image matrix:

[0053]

[0054] In equation (2), σ represents the standard deviation and N represents the number of pixels in the image.

[0055] Step 2: Select a convolutional neural network backbone based on the medical task;

[0056] Combination Figure 1 As can be seen, the network model consists of four parts, mainly divided into the backbone network module, the feature pyramid module (a), the interactive attention pyramid module (b), and the pooling classification module (c).

[0057] The modules are connected using max pooling layers, and the kernel size and stride of each layer can be set according to the feature map size.

[0058] In this embodiment, the default input data dimension of the network is N×3×224×224, where N is the size of each batch of data loaded, the height and width of the input image are both 224, and the color channels are RGB, where 3 represents the color channels.

[0059] Among them, see Figure 1 :

[0060] (1) Module 1: Backbone Network Module

[0061] Feature extraction from the backbone network is the first stage of image processing. It's important to note that the backbone network in this first stage is not fixed; different deep learning networks can be selected depending on the task, such as common backbones like VGG, ResNet, GoogleLeNet, and Inception. Four levels of feature information at different scales are designed for extraction within the backbone network. The matrix sizes of the extracted feature information at these four different levels are proportional. Different sizes can be chosen depending on the selected backbone network; for example, the dimensions of the four-level feature matrices obtained in ResNet50 can be: 256×56×56, 512×28×28, 1024×14×14, and 2048×7×7, respectively.

[0062] Generally, LSTM models are suitable for time series prediction tasks; deeper models such as ResNet101 are chosen for fine-grained classification tasks; and smaller models such as MobileNet are chosen for classification tasks requiring lightweight performance.

[0063] In this embodiment, the method proposed in this invention is used, with ResNet101 selected as the backbone network. Seven types of skin lesions are classified: benign (B), seborrheic keratosis (SK), basal cell carcinoma (BCC), actinic keratosis (AK), dermatofibroma (DF), vascular lesions (VL), and melanoma (M). The classification is nclass=7. Example images are shown below. Figure 5 As shown.

[0064] In this embodiment, the dataset provides labels for 10,015 images. Therefore, we divide the data into 80% for training (8,017 images) and 20% for testing (1,998 images).

[0065] Step 3: Introduce a feature pyramid to perform multi-scale feature fusion of medical images;

[0066] In this embodiment, the purpose of using a feature pyramid is to extract features at different scales. The residual network extracts four convolutional blocks of different scales that have the same spatial dimensions, such as... Figure 1 (a) consists of four feature levels {B0, B1, B2, B3,}. Then, features that are spatially coarser but semantically stronger are sampled from the upper pyramid levels down to the lower pyramid levels, such as... Figure 1 (a) In order to preserve the backbone information, the information is broadcast and added directly from the feature map Bn by the horizontal connection with the upsampled information. This operation is then repeated downwards to form an information-rich feature pyramid {F0, F1, F2, F3}.

[0067] Step 4: Introduce the interactive attention pyramid to generate the weight system of the backbone network;

[0068] (3) Module 3: Interactive Attention Pyramid Module

[0069] The invented interactive attention pyramid is a four-layer attention pyramid composed of a spatial attention pyramid and a channel attention pyramid.

[0070] 1) Channel Attention Pyramid

[0071] The channel attention structure used is as follows Figure 3 As shown, it consists of an average pooling layer and two fully connected layers, for each level of the feature map F from the feature pyramid. n (n = 0, 1, 2, 3) will all generate corresponding weight coefficients A (c) n (n=0,1,2,3) can be represented as:

[0072]

[0073] Here, σ and ReLU represent the sigmoid and ReLU activation functions, respectively; · represents element-wise multiplication; and W1 and W2 are the weight matrices of the two fully connected (FC) layers. Figure 1 (b) The channel attention pyramid is a system that integrates lower-level information from the attention pyramid with higher-level information. It begins with the weight coefficient A. (c) 4. The propagation begins upwards, reaching the third-level weight coefficient A. (c) At 3 o'clock, the channel attention information is passed up from the lower level to perform an averaging operation, such as... Figure 1 (b) The subsequent upward transmission constitutes the channel attention pyramid A. (c) n (n = 0, 1, 2, 3).

[0074] 2) Spatial Attention Pyramid

[0075] The spatial attention pyramid used is as follows: Figure 4 As shown, the structure can be derived from... Figure 4 We see that max pooling and average pooling are performed separately along one channel dimension to obtain channel descriptions. These two descriptions are then concatenated along the channel dimension and passed through a 3×3 convolutional layer with the sigmoid activation function to obtain the weight coefficients Ms, or A(s)n, which can be expressed as:

[0076]

[0077] In the formula, σ represents the sigmoid activation function, f 3×3This represents a 3×3 convolution operation. AvgPool and MaxPool represent max pooling and average pooling, respectively. Enclosing them in square brackets [] concatenates their results by channel. For the feature map F transmitted from the feature pyramid... n Each of these will generate a corresponding spatial attention weight coefficient A(s)n, thus forming a four-layer spatial attention pyramid A. (s) n (n = 0, 1, 2, 3).

[0078] Before the CS attention pyramid is constructed, the spatial attention weights A of the channel attention at the corresponding feature level are... (s) n (n = 0, 1, 2, 3) and the weight A of the channel attention (c) n (n = 0, 1, 2, 3) will be added before being combined with the feature map F. n Multiplying them together yields the scaled new feature F. (a) n (n=0,1,2,3) can be represented as:

[0079]

[0080] In the formula, the dot represents element multiplication.

[0081] 3) Interaction path

[0082] like Figure 1 (b) There are two interaction paths: Path 1, which interacts with the channel attention mask values ​​of the second layer (counting backwards from the input) and the channel attention mask values ​​of the third layer via broadcast addition; and Path 2, which also interacts with the channel attention of the third layer via the fourth layer. The channel attention mask generated by each layer is added to the spatial attention mask via broadcast addition, and the result is multiplied by the corresponding initial feature map of each layer via a Had code product before being output.

[0083] Step 5: After combining the backbone network, feature pyramid, and interactive attention pyramid, conduct ablation experiments to obtain the effect of each module combination.

[0084] Step Six: Using the medical image training set, train the optimal classification model based on the convolutional neural network to obtain a classification model based on the medical task;

[0085] Step 7: Input the medical image test set into the medical task-based classification model, and output the medical image classification results after average pooling.

[0086] F1 of each output feature matrix (a) F2 (a)F3 (a) F4 (a) The data is directly fed into a classifier for classification to obtain the output vector predk{k=1,2,3,4}, where the classifier is as follows: Figure 1 (ii) It includes a global pooling (GAP) layer and two fully connected (FC) layers. Finally, the results of the four layers are summed and averaged as follows: Figure 1 (iii) After that, the dimensionality reduction outputs N×nclass (N is the number of classes) classification results.

[0087] The softmax function outputs the predicted probability of each category, thus yielding our classification result. In this embodiment, seven types of skin lesion images are classified, and the classification results are detailed in Table 1.

[0088] Table 1 Evaluation Indicators for Skin Lesions

[0089] index Accuracy % Accuracy % Recall rate % Specificity % F1 score % This invention 96.60 91.43 93.26 99.34 92.27

[0090] The above are merely preferred embodiments of the present invention. It should be noted that any modifications and improvements made by those skilled in the art without departing from the present technical solution should also be considered to fall within the scope of protection claimed in this claim.

Claims

1. A medical image classification method based on feature fusion and attention mechanism, characterized in that According to the following steps: Step one: obtain a set of medical images of any size from a medical image database, and obtain a medical image training set and a medical image test set after image preprocessing; Step two: select a convolutional neural network backbone network based on the medical task; Step three: introduce a feature pyramid for multi-scale feature fusion of medical images; Step four: introduce an interactive attention pyramid to generate a weight system for the backbone network; In step four, The interactive attention pyramid is composed of four parallel structures, which are respectively referred to as the first layer to the fourth layer, each layer includes a channel attention module and a spatial attention module, and the two are connected in parallel and provided with two interaction paths: the first interaction path and the second interaction path; The first interaction path: the channel attention mask output by the second layer channel attention module is superimposed on the input end of the third layer channel attention module through broadcast addition operation; The second interaction path: the channel attention mask output by the third layer channel attention module is superimposed on the input end of the fourth layer channel attention module through broadcast addition operation; In each layer, the channel attention mask output by the channel attention module and the spatial attention mask output by the spatial attention module are first added in the form of broadcast addition to obtain the comprehensive attention weight of the layer; The comprehensive attention weight is then multiplied with the initial feature map input into the layer in the form of Hadamard product to generate the enhanced feature output of the layer; Step five: perform an ablation experiment on the combination of the backbone network, the feature pyramid and the interactive attention pyramid to obtain the effect of each module combination; Step six: train the optimal classification model based on the convolutional neural network using the medical image training set to obtain a classification model based on the medical task; Step seven: input the medical image test set into the classification model based on the medical task, combine the average pooling, and output the medical image classification result. 2.The medical image classification method based on feature fusion and attention mechanism according to claim 1, characterized in that In step one, the image preprocessing step is: Set the image size to a x a, and unify the size of all images in the medical image set; Divide all the size-unified medical images into an enhanced training set and a medical image test set; Obtain the medical image training set after performing image enhancement operation on the enhanced training set; The image enhancement operation includes: Randomly crop part of the area of the medical image in the enhanced training set and fill it with 0 pixel value; Randomly flip horizontally, randomly modify the brightness, contrast and saturation of the picture; Convert to tensor form; Perform data standardization; The standardization formula is as follows: （1） In formula (1), u is the image mean value, and X represents the image matrix: （2） In formula (2), sigma represents the standard deviation, and N represents the pixel of the image. 3.The medical image classification method based on feature fusion and attention mechanism according to claim 2, characterized in that: In step two, the convolutional neural network backbone network at least includes VGG, ResNet, GoogleLeNet and Inception. 4.The method of claim 3, wherein: In step two, after confirming the convolutional neural network backbone network, four levels of feature information of different scales are designed in the backbone network for extraction; The matrix size of the feature information is in a multiple relationship. 5.The method of claim 4, wherein: In step seven, the average pooling result is the output after the four layers of results are processed in the form of an average.

Images