Pathological image classification method based on semantic spatial attention
By introducing a semantic spatial attention mechanism into pathological image classification, the problem of information interaction difficulties caused by the small proportion of cellular regions in pathological images is solved, which improves classification accuracy and reduces the dependence on data annotation.
Patent Information
- Application Number
- CN202310364700.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-07
- Publication Date
- 2025-12-30
- Estimated Expiration
- 2043-04-07
AI Technical Summary
Existing pathological image classification methods ignore the fact that cellular regions account for a small proportion of pathological images, leading to difficulties in information exchange and affecting the learning of discriminative information by classification network models.
A semantic spatial attention mechanism is adopted to extract semantic information in an unsupervised manner and perform information interaction within the same semantic region. Combined with the ResNet50 classification network, a semantic-level spatial attention mechanism is constructed to enhance attention to cellular regions.
It improves the performance of pathological image classification, overcomes the problem of insufficient data annotation, and only requires the introduction of a small number of parameters to improve classification accuracy.
Smart Images
Figure CN116824204B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, to pathological image classification technology, and particularly to a pathological image classification method based on semantic spatial attention. Background Technology
[0002] The background technology involves three main areas: unsupervised segmentation, attention mechanisms, and pathological image classification.
[0003] 1) Unsupervised segmentation
[0004] Image segmentation algorithms aim to separate specific object regions from an image. After segmentation, pixels within each region are generally similar and possess similar semantics. With the development of deep learning, neural networks have demonstrated superior performance in image segmentation, which can be viewed as a pixel-level classification. Based on the granularity of the segmentation labels, segmentation can be categorized into supervised segmentation, weakly supervised segmentation, semi-supervised segmentation, and unsupervised segmentation. Supervised segmentation is the most basic method, requiring pixel-level labels, and its label creation is the most costly. Weakly supervised segmentation relies on point-level, bounding box, or image category labels. In semi-supervised segmentation, only labels for some samples are available. Unsupervised segmentation does not require image labels or object information; it automatically finds separable objects in the image. This requires preprocessing image segmentation algorithms, such as Slice and Felz algorithms, followed by further optimization of these coarse segmentation results.
[0005] Unsupervised segmentation primarily relies on unsupervised methods, commonly including K-means clustering, mean shift, and region-based segmentation. K-means clusters pixels into k clusters, which typically represent different objects or regions. Mean shift, also a clustering technique, finds cluster centers by estimating density; it's a non-parametric method. Mean shift starts by selecting a random point and then moves along directions of higher density, eventually finding a local density maximum. Region-based segmentation divides the image into multiple distinct regions, each sharing common attributes such as color or texture. Thresholding or edge detection methods are typically used, and multiple regions can be further merged into a larger region.
[0006] Unsupervised segmentation networks include methods based on autoencoders and sparse coding. Autoencoder-based methods encode the input image into a low-dimensional feature vector using an encoder, and the decoder reconstructs the original data for segmentation prediction. Sparse coding-based methods use sparse coding to represent sparse features and assign pixels to specified regions. Sparse coding can extract important features from the original data, making pixels easier to classify.
[0007] 2) Attention mechanism
[0008] Attention mechanisms are common techniques in deep learning, and their core purpose is to make the network model focus more on discriminative regions. The successful design of self-attention has sparked much research, and besides self-attention, there are other mainstream attention methods, such as channel attention and spatial attention. Channel attention primarily focuses on learning attention weights related to channels, causing the network model to focus on important channels and suppress unimportant ones. SENet is a classic channel attention-based method; Squeeze and Excitation are two key operations for channels, modeling the interdependencies between them. Spatial attention focuses on learning the spatial relationships of feature maps, aiming to learn spatially related attention weights. In the feature map, calculating the relationship between a point and other locations yields attention weights; alternatively, max pooling and average pooling can be used to calculate spatial attention weights. CBAM is a classic method that combines channel and spatial attention, where both are obtained through pooling. This simple method, which doesn't introduce additional learning parameters, effectively improves network model performance, but this design still inherently lacks the ability to interact with pixel information. The weights of spatial attention can intuitively show the regions that the network model focuses on, and this spatial attention can, to some extent, serve as an interpretability feature of the network model. Current research explores the consistency of spatial attention, aiming to maintain a consistent attention distribution across feature maps of different depths.
[0009] 3) Classification of pathological images
[0010] Image classification is a fundamental machine learning task used to categorize images. It involves extracting feature representations from an input image and predicting its category. With the development of convolutional neural networks, the predictive performance of classification network models has approached human-level capabilities, and even surpassed them in some areas. These network models often require large amounts of manually labeled image data for training, and during the testing phase, the network model makes predictions on new images.
[0011] Pathological image classification is a special type of image classification task that uses computer vision technology to classify medical images. Pathological images are typically digitized images of tissue or cell sections. Pathological image classification automatically categorizes pathological images into different classes, such as normal tissue, malignant tumors, or benign tumors. Pathological image classification can assist pathologists in diagnosis and significantly reduce their workload.
[0012] Currently, a series of methods for classifying pathological images have been proposed, mainly including multi-view, multi-scale, and multi-instance learning methods. Multi-view based methods learn consistency while incorporating complementary information from multiple views, such as combining traditional texture information with deep feature maps from neural networks. Multi-scale based methods combine images at different magnifications to avoid visual interference caused by magnification. Multi-instance learning addresses the whole-slice diagnosis problem in pathological images, where ultra-large pixel whole slices are treated as bags, and patches are treated as instances within those bags. A series of methods for extracting features from pathological images have also been proposed, focusing more on shallow texture information or global contextual information. However, these methods ignore the characteristic that cellular regions account for a small proportion of pathological images, which makes information interaction difficult and hinders the learning of discriminative information by classification network models. Summary of the Invention
[0013] The purpose of this invention is to provide a semantic spatial attention-based method for classifying pathological images. This method achieves classification of pathological images by extracting semantic information and incorporating an attention mechanism. Considering the small proportion of cellular regions in pathological images, this method enhances focus on cellular regions by allowing sufficient information interaction within the same semantic region.
[0014] The specific technical solution for achieving the objective of this invention is as follows:
[0015] A method for pathological image classification based on semantic spatial attention, comprising the following steps:
[0016] Step 1: Establish a pathological image dataset
[0017] The pathological image dataset is in the format of: images With tags Where M represents the number of data, X i Y represents the i-th pathological image. i This represents the true label corresponding to the i-th pathological image;
[0018] Step 2: Determine the semantic information extraction network model
[0019] The Felz algorithm is used to obtain preliminary semantic information, and then a convolutional neural network is used to optimize the semantic information. The convolutional neural network consists of four basic convolutional blocks, each of which includes a 2D convolutional layer, batch normalization, and ReLU activation function.
[0020] Step 3: Constructing semantic information
[0021] Step a1: X-ray the pathological image i The input is fed into the Felz algorithm to obtain preliminary semantic information. Where K represents the number of semantic information region categories, Ck This indicates that the semantic information belongs to the k-th class;
[0022] Step a2: Calculate the output probability of each pixel using a convolutional neural network, i.e. Where N represents the number of pixels in the image, and Net() represents a convolutional neural network;
[0023] Step a3: Maximize the output probability, i.e., yn = argmax(p n ), argmax() represents maximizing the output probability function;
[0024] Step a4: The most frequent y within the same semantic information region n As the semantic information category label for each pixel, that is
[0025]
[0026] Step a5: Use cross-entropy loss as the loss function, i.e., Loss = Cross entropy(p n , t n ), where Cross entropy represents the cross entropy function;
[0027] Step a6: Optimize the semantic information network model using stochastic gradient descent;
[0028] Step a7: Repeat steps a1 to a6 above until the network model converges, and obtain the semantic information category label t corresponding to each pixel. n ;
[0029] Step 4: Determine the network model for pathological image classification
[0030] ResNet50 was used as the classification network model; this network model consists of an input layer, a group of convolutional layers, and a classification output layer; the input is a pathological image X. i First, it goes through a convolutional layer with a 7*7 kernel, then through four convolutional blocks. Each convolutional block consists of multiple basic convolutional layers, with a 1*1 kernel for scaling channels. The input and output of each convolutional block are connected via residuals. In the ResNet50 network model, the feature map F of the last convolutional block is obtained.
[0031] Step 5: Determine the spatial attention mechanism and incorporate semantic information to construct a semantic-level spatial attention mechanism.
[0032] Pixel-level spatial attention is obtained by using the feature map F through flat pooling (AvgPool) and max pooling (MaxPool), i.e., F avg =AvgPool(F), F max=MaxPool(F); Based on pixel-level spatial attention, an attention graph F is constructed through interactions within the same semantic information region. avgavg and F maxmax Then F avg F max F avgavg and F maxmax After fusion and activation, the final attention S is obtained; a new feature map F′ is obtained by element-wise multiplying the feature map F of the original image with the attention S, i.e., F′=F⊙<; the new feature map F′ is processed by global pooling to obtain the feature vector, and then input into the fully connected layer to obtain the final output probability;
[0033] Step Six: Train the classification network model and classify the pathological images.
[0034] The classification network model is optimized using learning rate decay and stochastic gradient descent; the specific training includes the following steps:
[0035] Step b1: Crop the trained pathological images to a uniform size of 448*448, flip them horizontally with a 50% probability, and then normalize them;
[0036] Step b2: Input the image into the classification network model and calculate the feature map F of the last convolutional block;
[0037] Step b3: Obtain the semantic space attention of feature map F, and multiply feature map F and semantic space attention element by element to obtain a new feature map F';
[0038] Step b4: Obtain the feature vector from the feature map F' through global pooling, and then obtain the final output probability through a fully connected layer;
[0039] Step b5: Calculate the cross-entropy loss between the output probability and the true label, optimize the objective using stochastic gradient descent, and update the learning rate; repeat steps b2 to b5 above until the network model converges, and terminate training.
[0040] Step b6: Based on the classification network model, obtain the probability of the final output, and after passing through the Softmax activation function, obtain the final output classification.
[0041] Step five describes the interaction of attention within the same semantic information region based on pixel-level spatial attention: Within the same semantic information region, max pooling and average pooling operations are applied in the spatial dimension based on pixel-level spatial attention, as described by the following formula:
[0042] F avg =AvPoo1(F),
[0043] Fmax =MPoo1(F),
[0044]
[0045]
[0046] S=σ(fAIF avg ;F max ;F avg ⊙F avgavg ;F max ⊙F maxmax ])),
[0047]
[0048] Where σ(·) represents the sigmoid activation function, f(·) represents a standard convolution operation, ⊙ represents element-wise multiplication, and I represents a 0-1 matrix; semantic-level spatial attention <not only through F avg and F max Incorporating information from the channel dimension, and also through F avgavg and F maxmax It incorporates semantic information; t n The semantic information category label represents each pixel, where u represents the semantic information category.
[0049] The beneficial effects of this invention include:
[0050] 1) This invention incorporates semantic information and guides attention from the perspective of information interaction, and reconstructs a spatial attention mechanism based on semantic information level, breaking the regional limitations of the original spatial attention information interaction based on pixel level.
[0051] 2) The semantic information of this invention is obtained in an unsupervised manner, overcoming the problem of lack of data annotation in pathological images. The proposed semantic attention, based on the original spatial attention, only requires the introduction of very few parameters to improve the performance of pathological image classification. Attached Figure Description
[0052] Figure 1 This is a flowchart of the present invention;
[0053] Figure 2 This is a schematic diagram of the pathological image classification method based on semantic-level spatial attention according to the present invention. Detailed Implementation
[0054] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.
[0055] This invention includes the following steps:
[0056] Step 1: Establish a pathological image dataset
[0057] The pathological image dataset is in the format of: images With tags Where M represents the number of data, X i Y represents the i-th pathological image. i This represents the true label corresponding to the i-th pathological image;
[0058] Step 2: Determine the network model for extracting semantic information
[0059] The Felz algorithm is used to obtain preliminary semantic information, and then a convolutional neural network is used to optimize the semantic information. The convolutional neural network consists of four basic convolutional blocks, each of which includes a 2D convolutional layer, batch normalization, and ReLU activation function.
[0060] Step 3: Constructing semantic information
[0061] Step a1: X-ray the pathological image i The input is fed into the Felz algorithm to obtain preliminary semantic information. Where K represents the number of semantic information region categories. This indicates that the semantic information belongs to the k-th class;
[0062] Step a2: Calculate the output probability of each pixel using a convolutional neural network, i.e. Where N represents the number of pixels in the image, and Net() represents a convolutional neural network;
[0063] Step a3: Maximize the output probability, i.e., y n =argmax(p n ), argmax() represents maximizing the output probability function;
[0064] Step a4: The most frequent yn within the same semantic information region is used as the semantic information category label for each pixel, i.e.
[0065]
[0066] Step a5: Use cross-entropy loss as the loss function, i.e., Loss = Cross entropy(p n , t n ), where Cross entropy represents the cross entropy function;
[0067] Step a6: Optimize the semantic information network model using stochastic gradient descent;
[0068] Step a7: Repeat the above operations a1 to a6 until the network converges, and obtain the semantic information category label tn corresponding to each pixel;
[0069] Step 4: Determine the network model for pathological image classification
[0070] ResNet50 was used as the classification network; this network consists of an input layer, a group of convolutional layers, and a classification output layer; the input pathological image X... i First, it goes through a convolutional layer with a 7*7 kernel, then through four convolutional blocks. Each convolutional block consists of multiple basic convolutional layers, with a 1*1 kernel for scaling channels. The input and output of each block are connected via residuals. In the ResNet50 network model, the feature map F of the last convolutional block is obtained.
[0071] Step 5: Determine the spatial attention mechanism and incorporate semantic information to construct a semantic-level spatial attention mechanism. Pixel-level spatial attention is obtained by using the feature map F through average pooling (AvgPool) and max pooling (MaxPool), i.e., F... avg =AvgPool(F), F max =MaxPool(F); Based on pixel-level spatial attention, an attention graph F is constructed through interactions within the same semantic information region. avgavg and F maxmax Then F avg F max F avgavg and F maxmax After fusion and activation, the final attention S is obtained;
[0072] F avg =AvgPool(F),
[0073] F max =MaxPool(F),
[0074]
[0075]
[0076] S=σ(f([F avg ;F max ;F avg ⊙F avgavg ;F max ⊙F maxmax ])),
[0077]
[0078] Where σ(·) represents the sigmoid activation function, f(·) represents a standard convolution operation, ⊙ represents element-wise multiplication, and I represents a 0-1 matrix; the semantic-level spatial attention S not only through F avg and F max Incorporating information from the channel dimension, and also through F avgavg and F maxmax It incorporates semantic information about space.
[0079] A new feature map F′ is obtained by element-wise multiplying the feature map F of the original image with the attention S, i.e., F′=F⊙S; the new feature map F′ is then processed by global pooling to obtain a feature vector, which is then input into a fully connected layer to obtain the final output probability.
[0080] Step Six: Train the classification network model and classify the pathological images.
[0081] The classification network model is optimized using learning rate decay and stochastic gradient descent; the specific training includes the following steps:
[0082] Step b1: Crop the trained pathological images to a uniform size of 448*448, flip them horizontally with a 50% probability, and then normalize them;
[0083] Step b2: Input the image into the classification network model and calculate the feature map F of the last convolutional block;
[0084] Step b3: Obtain the semantic space attention of feature map F, and multiply feature map F and semantic space attention element by element to obtain a new feature map F′;
[0085] Step b4: Obtain the feature vector from the feature map F′ through global pooling, and then obtain the final output probability through a fully connected layer;
[0086] Step b5: Calculate the cross-entropy loss between the output probability and the true label, optimize the objective using stochastic gradient descent, and update the learning rate; repeat steps b2 to b5 above until the network model converges, and terminate training.
[0087] Step b6: Based on the classification network model, obtain the probability of the final output, and after passing through the Softmax activation function, obtain the final output classification.
[0088] Example
[0089] The following is the implementation process of this embodiment:
[0090] 1, such as Figure 1 As shown, the image With tags Where N represents the number of data points, X iY represents the i-th pathological image. i This represents the true label corresponding to the i-th pathological image;
[0091] 2. Preliminary semantic information is obtained using the Felz algorithm. Where K represents the number of semantic information region categories, C k This indicates that the semantic information belongs to the k-th class. Then, a convolutional neural network is used to calculate the output probability of each pixel, i.e. Maximize the output probability, i.e., y n =argmax(p n The maximum number of occurrences (yn) within the same semantic information region is used as the semantic information category label for each pixel.
[0092] 3, such as Figure 2 As shown, ResNet50 is used as the classification network model; the feature map F of the pathological image is extracted from the input image through ResNet50, and pixel-level spatial attention, i.e., F, is obtained through average pooling (AvgPool) and max pooling (MaxPool). avg =AvgPool(F), F max =MaxPool(F); Based on pixel-level spatial attention, an attention graph F is constructed through interactions within the same semantic information region. avgavg and F maxmax Then F avg F max F avgavg and F maxmax After fusion and activation, the final attention S is obtained; by element-wise multiplying the feature map F of the original image with the attention S, a new feature map F′ can be obtained, i.e. Finally, the data is fed into the classification layer for category prediction.
[0093] The experimental datasets used in this invention are the BreakHist dataset and the gastric cancer dataset. The BreakHist dataset contains 7909 pathological images, cropped at magnifications of 40x, 100x, 200x, and 400x. It includes 2480 benign samples and 5429 malignant samples. Approximately 70% of the data was randomly selected as the training set, and the remainder was used to test classification performance, ensuring that the patients in the training and test sets did not overlap. The gastric cancer dataset is a publicly accessible dataset of H&E-stained gastric histopathological images. This dataset contains 700 pathological images, of which 140 are normal pathological images and 560 are cancer images, with the original pathological image size being 2048×2048. In the experiments, the training, validation, and test datasets of this dataset were partitioned in a 1:1:2 ratio.
[0094] Table 1. Accuracy score (%) on the BreakHist test dataset.
[0095]
[0096] Table 2 shows the Accuracy score on the gastric cancer validation dataset and the performance scores (%) of Accuracy, Precision, Recall, and F1 score on the test dataset.
[0097]
[0098] Table 1 shows the accuracy scores of the proposed method and some other methods on the BreakHist dataset. VGG16 (Very deep convolutional networks for large-scale image recognition), Inception_v3 (Inception-v3 for flower classification, in the International Conference on Image, Vision and Computing), ResNet18 (Deep residual learning for image recognition), and ResNet50 are classic convolutional neural networks; SA (SA-Net: Shuffle attention for deep convolutional neural networks), CBAM (CBAM: Convolutional block attention module), and BAM (BAM: Bottleneck attention module) are attention-based methods. The semantic attention method of this invention outperforms all the compared methods on the BreakHist dataset in terms of accuracy. Experimental results also show that general spatial attention methods are ineffective and may even suppress the performance of the original classification network model, but the semantic attention method of this invention incorporates semantic information, thus effectively solving this problem.
[0099] Table 2 shows the Accuracy, Precision, Recall, and F1 scores of the proposed method and some other methods on the gastric cancer dataset. The proposed method outperforms most of the compared methods in terms of Accuracy, Precision, Recall, and F1 score on the gastric cancer dataset.
[0100] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.
Claims
1. A pathological image classification method based on semantic space attention, characterized in that, The method comprises the following steps: Step one: establishing a pathological image dataset The pathological image dataset format is: image With label Wherein M represents the number of data, X i represents the i-th pathological image, Y i represents the real label corresponding to the i-th pathological image; Step two: determining a semantic information extraction network model The Felz algorithm is used to obtain preliminary semantic information, and then a convolutional neural network is used for semantic information optimization; the convolutional neural network is composed of four basic convolutional blocks, and each convolutional block includes a 2D convolutional layer, batch normalization and a Relu activation function; Step three: constructing semantic information Step a1 : inputting the pathological image X i into the Felz algorithm to obtain preliminary semantic information where K represents the number of semantic information region categories, C k represents that the semantic information belongs to the kth category; Step a2: calculate the output probability of each pixel point by the convolutional neural network, i.e. where N represents the number of pixel points in the image, and Net() represents the convolutional neural network. Step a3: maximize the output probability, i.e. y n = argmax(p n ), argmax() represents the maximum output probability function; Step a4: the maximum number of y in the same semantic information area n As the semantic information category label of each pixel point, that is Step a5: Cross-entropy loss is used as the loss function, i.e. Loss = Cross entropy(p n ,t n ), where Cross entropy represents the cross-entropy function; Step a6: optimizing the semantic information network model by random gradient descent; Step a7: repeat the above a1 to a6 operations until the network model converges, and obtain the semantic information category label t corresponding to each pixel point n ; Step four: determining a pathological image classification network model ResNet50 is adopted as the classification network model; the network model is composed of an input layer, a convolutional layer group and a classification output layer; the input pathological image X i firstly passing through a convolutional layer with a convolution kernel of 7*7, then passing through four convolutional blocks, each of which is composed of multiple basic convolutional layers with a convolution kernel of 1*1 for scaling channels; the input and output of each convolutional block are connected in a residual manner; in the ResNet50 network model, a feature map F of the last convolutional block is obtained; Step five: determining a spatial attention mechanism and integrating semantic information to construct a semantic-level spatial attention mechanism The pixel-level spatial attention is to obtain the feature map F through the average pooling AvgPool and the maximum pooling MaxPool, that is, F avg = AvgPool(F), F max = MaxPool(F); on the pixel-level spatial attention, the attention graph F is constructed through the interaction within the same semantic information region avgavg and F maxmax , and then F avg , F max , F avgavg and F maxmax are fused and activated to obtain the final attention S; A new feature map F' is obtained by element-by-element multiplication of the feature map F of the original image and the attention S, i.e., F' = F <; A feature vector is obtained by global pooling operation on the new feature map F', and the final output probability is obtained by inputting the feature vector into the fully connected layer; Step six: training the classification network model and classifying the pathological image The classification network model is optimized by learning rate decay and random gradient descent; the specific training comprises the following steps: Step b1: the training pathological image is cropped to a uniform size of 448*448, and is horizontally flipped with a probability of 50%, and then is normalized; Step b2: the image is input into the classification network model, and the feature map F of the last convolutional block is calculated; Step b3: the semantic spatial attention of the feature map F is obtained, and the feature map F is multiplied with the semantic spatial attention element by element to obtain a new feature map F'; Step b4: a feature vector is obtained by global pooling operation on the feature map F', and the final output probability is obtained by inputting the feature vector into the fully connected layer; Step b5: the cross-entropy loss between the output probability and the real label is calculated, the objective is optimized by random gradient descent, and the learning rate is updated; the above steps b2 to b5 are repeated until the network model converges, and the training is terminated; Step b6: according to the classification network model, the final output probability is obtained, and after the Softmax activation function, the final output classification is obtained.
2. The pathological image classification method based on semantic spatial attention according to claim 1, characterized in that, In the pixel-level spatial attention described in step five, the attention in the same semantic information region is interacted: in the pixel-level spatial attention, the maximum pooling and average pooling operations are used in the spatial dimension in the same semantic information region, which is described as follows: F avg = AvgPool(F), F max = MaxPool(F), S = σ(f([F avg ; F max ; F avg ⊙ F avgavg ; F max ⊙ F maxmax ])), wherein, σ(·) represents a sigmoid activation function, f(·) represents a standard convolution operation, represents element-by-element multiplication, I represents a 01 matrix; the spatial attention S based on the semantic level not only combines the channel dimension information through F avg and F max Combine the semantic information through F avgavg and F maxmax t n represents the semantic information category label of each pixel point, and u represents the semantic information category.