A cervical cell classification method based on multi-scale attention feature enhancement

Abstract

The application discloses a cervical cell classification method based on multi-scale attention feature enhancement, comprising the following steps: S1, extracting multi-scale features by using a deep convolutional neural network; S2, constructing a multi-scale feature pyramid according to a feature fusion method in a FPN; S3, respectively calculating spatial attention and channel attention for each layer of features in the pyramid features, and generating a spatial attention pyramid and a channel attention vector; S4, enhancing spatial attention by using masks obtained by threshold segmentation of the spatial attention pyramid; S5, performing attention weighting on the multi-scale pyramid features in S2 by using the enhanced spatial attention pyramid and the channel attention vector of each layer, so as to obtain a multi-scale attention feature pyramid; S6, respectively constructing a classifier for each layer of features in the multi-scale attention feature pyramid; and S7, performing gradient descent optimization training on the whole network and classifying and predicting cervical cells. The application provides a cervical cell classification model with higher classification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and medicine, in particular to a cervical cell classification method based on multi-scale attention feature enhancement, and belongs to the application of deep learning in the medical field. BACKGROUND

[0002] Cervical cancer is one of the most common gynecological malignancies, with a very high incidence and mortality rate. The high-incidence age of carcinoma in situ is 30-35 years old, and the high-incidence age of invasive carcinoma is 45-55 years old. In recent years, the incidence of cervical cancer has shown a trend of younger age. Clinical practice has found that effective cervical cell screening methods can make cervical cancer and precancerous lesions be discovered and treated earlier, and significantly reduce the incidence and mortality of cervical cancer.

[0003] According to the different stages of precancerous lesions of cervical cancer, cervical cells can be divided into the following 5 categories: NORMAL (normal cells), ASC-US (atypical squamous cells of undetermined significance, atypical squamous cells of undetermined significance), ASC-H (atypical squamous cells but cannot rule out high-grade squamous intra-epithelial lesion, atypical squamous cells that cannot rule out high-grade squamous intra-epithelial lesion), LSIL (low-grade squamous intra-epithelial lesion, low-grade squamous intra-epithelial lesion), and HSIL (high-grade squamous intra-epithelial lesion, high-grade squamous intra-epithelial lesion). The five types of cells are generally similar in appearance, but as the disease progresses, the differences between them mainly manifest in different degrees of changes in their pathological characteristics, for example, abnormal cells often have dyskaryotic nuclei due to uncontrolled division. Therefore, in clinical diagnosis, cervical cell screening is usually performed by cytologists observing the pathological characteristics of cell nuclei in cervical cell smears under a microscope to determine the degree of cell lesions. However, manual cervical cytology screening requires a high level of expertise and experience from cytologists, and is also very time-consuming and labor-intensive. Fortunately, with the rapid development of machine learning methods, especially deep learning, more and more researchers have begun to use computer-aided automated diagnosis (CAD) to make up for the shortcomings of manual cytology screening methods.

[0004] Many cervical cell automatic classification tasks have emerged, but these methods still do not directly solve the two difficulties faced by cervical cell classification. First, the distinction between different categories of cervical cells is mainly reflected on the cell nucleus, so the classification network needs to pay more attention to the cell nucleus area. Second, cervical cell classification is a fine-grained image classification task, and the similarity between different categories of cells is large, and due to the inconsistency of staining conditions or the difference in personal preferences of cervical smear makers, there may be great differences in cells of the same category, which makes the cervical cell classification task more challenging. SUMMARY

[0005] In view of the above technical problems, the application discloses a cervical cell classification method based on multi-scale attention feature enhancement, which extracts a multi-scale feature pyramid containing low-level detail features and high-level semantic features by building a feature pyramid network with bidirectional feature transmission, and uses this more abundant feature to build multiple joint classifiers to more effectively mine the discriminative features between various types of cervical cells of fine granularity, and then enhances the multi-scale attention by using the attention masks generated by threshold segmentation, and finally realizes more accurate cervical cell classification.

[0006] The technical scheme provided by the application is as follows:

[0007] A cervical cell classification method based on multi-scale attention feature enhancement, the steps are as follows:

[0008] S1: using a deep convolutional neural network containing multiple blocks to extract multi-scale features from cervical cell images;

[0009] S2: constructing a multi-scale feature pyramid according to the feature fusion method in the FPN;

[0010] S3: using the APN to calculate the spatial attention and channel attention of each layer of features in the multi-scale pyramid features respectively, and generating a spatial attention pyramid and channel attention vectors of each layer;

[0011] S4: using the masks obtained by threshold segmentation of the spatial attention pyramid to enhance the spatial attention;

[0012] S5: using the enhanced spatial attention pyramid and the channel attention vectors of each layer to weight the multi-scale pyramid features in S2, to obtain a multi-scale attention feature pyramid for classification;

[0013] S6: constructing a classifier for each layer of features in the multi-scale attention feature pyramid respectively;

[0014] S7: Gradient descent optimization training of the whole network is performed using the three loss functions, and the trained classification network is used to classify and predict cervical cells.

[0015] Further, the deep convolutional neural network in step S1 is the backbone of any general classification network, which is divided into n blocks, and the output feature map of each block has a different scale. Preferred classification networks include ResNet, DenseNet, VGG, etc.

[0016] Further, in step S2, the features of the deep block output are sampled to the same size as the shallow block output feature map, and then added to the shallow block output feature map to realize the fusion of different scale features from top to bottom, and the features of each block output feature and the fused deep features are used to construct a multi-scale feature pyramid.

[0017] Further, step S3 includes the following sub-steps:

[0018] S3.1: For each layer of features in the multi-scale pyramid feature Perform deconvolution and sigmoid normalization to generate a spatial attention map corresponding to each layer of features ;

[0019] S3.2: For each layer of features in the multi-scale pyramid feature After sequentially passing through global average pooling and two fully connected layers, an initial channel attention is generated ;

[0020] S3.3: Use the initial channel attention of the current layer and the channel attention of the shallow layer to add and fuse to obtain the final channel attention of the current layer , realizing the fusion of different scale attention features from bottom to top.

[0021] Further, step S4 includes the following sub-steps:

[0022] S4.1: Calculate the mean of each layer of spatial attention map ;

[0023] S4.2: Use as a threshold to segment the spatial attention map , to obtain the corresponding segmentation mask ;

[0024] S4.3: Use to segment the spatial attention map The attention scores at points with small attention weights are removed to obtain enhanced multi-scale spatial attention features.

[0025] Further, in step S4.2, the attention masks are obtained by using the spatial attention map as the threshold. The segmentation method is as follows: values greater than or equal to are assigned a value of 1, values less than are assigned a value of 0.

[0026] Further, in step S5, each value in the channel attention vector corresponds to a channel in the current layer feature map, and each attention value is weighted to all feature values in the corresponding channel by multiplication.

[0027] Further, in step S6, the classifier is composed of a global average pooling layer and several fully connected layers.

[0028] Further, in step S7, the three loss functions are composed of the weighted sum of the joint cross-entropy loss of all classifiers, the smoothing loss of masks and the channel feature diversity loss , that is, λ where μ and are constants for balancing the losses.

[0029] Further, in step S7, the smoothing loss is used to constrain the boundary smoothing of the attention mask, and the first-order gradient of each scale attention mask is the sum, and its formula is:

[0030]

[0031] wherein is the total number of attention masks, and are the length and width of the i-th attention mask, respectively, and m and n represent the coordinate index of a point in the attention mask. represents the gradient of the point with coordinates (m, n) in the i-th attention mask, which is obtained using the first-order gradient operator in the present application.

[0032] Further, in step S7, the channel feature diversity loss loss The steps for forcing the feature channel output by the last convolutional layer in the classifier to be discriminative to a specific class, thereby increasing the diversity of each channel feature, include:

[0033] (1) Each layer of classification features is divided into groups according to the channel, Each group contains channel features, is the number of channel features of the current layer of classification features, is the number of cervical cell categories;

[0034] (2) The channel features in each group are aggregated into a single channel feature using cross-channel average pooling, and a class response value is calculated for each group of features using global average pooling. The softmax function is used to normalize all response values;

[0035] (3) The cross-entropy loss value of the vector composed of the normalized response values and the one-hot encoding of the current image category is calculated, and the channel diversity loss of the current layer of features is calculated. The average of the channel diversity loss of each layer is calculated as the channel diversity loss of the current network.

[0036] Further, in step S7, the result of the classification prediction is the average of all classifier prediction results.

[0037] Further, in step S7, the cervical cells are divided into 5 categories, including NORMAL, ASC-US, ASC-H, LSIL and HSIL.

[0038] The beneficial effects of the present application are as follows:

[0039] (1) The present application builds a feature pyramid network with bidirectional feature transmission to extract a multi-scale feature pyramid containing low-level detailed features and high-level semantic features. Using this more abundant feature to build multiple joint classifiers can more effectively mine the discriminative features between various types of cervical cells.

[0040] (2) The feature enhancement method based on the attention mask corrects the attention weight to further improve the attention to the key area, so as to increase more attention to the nucleus area. In the implementation process, first, the multi-scale features and multi-scale attention of the cervical cell image are extracted through the FPN (Feature Pyramid Network) and the APN (Attention Pyramid Network), then the multi-scale attention is enhanced by using the attention masks generated by threshold segmentation, and finally, the features generated by the attention weighting of the multi-scale features using the enhanced attention are used to construct a classifier, so as to improve the accuracy of cervical cell classification from the perspective of improving the explainability. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0042] Figure 1 The various cervical cell image example diagrams used for the embodiments of the present application are shown in the following table:

[0043] Figure 2 The overall flow structure diagram of the embodiments of the present application is shown in the following table:

[0044] Figure 3 The specific implementation details of each module in the embodiments of the present application are shown in the following table: DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0046] EMBODIMENT

[0047] The embodiments of the present application provide a cervical cell classification method based on multi-scale attention feature enhancement (based on ResNet50), and the overall flow structure diagram is shown in the following table: Figure 2 , including the following steps:

[0048] S1: Use the ResNet50 backbone as a multi-scale feature extractor, and use the output feature maps of 4 of its blocks. This constitutes multi-scale features;

[0049] S2: Multi-scale features based on the feature fusion method in FPN. A multi-scale feature pyramid is obtained by performing top-down feature fusion. For specific implementation details, please see Figure 3 In Modules, it is important to note Directly by Perform 1 1. Convolution yields, It was not included in the consideration because its feature map scale is large and requires a lot of computing resources. Therefore, in the implementation process, features of partial scale can be selectively used to construct the feature pyramid.

[0050] S3: Use APN to calculate spatial attention and channel attention for each layer of features in the multi-scale pyramid feature set, generating a spatial attention pyramid and channel attention vectors for each layer. See details for implementation. Figure 3 middle The module includes the following steps:

[0051] S3.1: Features of each layer in the multi-scale pyramid conduct Deconvolution and sigmoid normalization generate spatial attention maps corresponding to the features of each layer. ;

[0052] S3.2: Features of each layer in the multi-scale pyramid The initial channel attention is generated after passing through global average pooling and two fully connected layers. ;

[0053] S3.3: Use the initial channel attention of the current layer With shallow channel attention The summation and fusion yields the final channel attention for the current layer. To achieve bottom-up fusion of attention features at different scales, it's important to note that the final channel attention of the shallowest features is the same as the initial channel attention. = ;

[0054] S4: Enhance spatial attention by using masks obtained from spatial attention pyramid threshold segmentation. See details for implementation. Figure 3 middle The module includes the following steps:

[0055] S4.1: Calculate the spatial attention map of each layer ; ;

[0056] S4.2: Segment the spatial attention map using as the threshold, i.e., the value in greater than or equal to is assigned a value of 1, i.e., , and the value less than is assigned a value of 0, resulting in the corresponding segmentation mask . In the implementation process, in order to facilitate the update of when backpropagation, the sigmoid function can be used to make derivable, and the specific calculation formula is as follows:

[0057]

[0058] where is set to a very large constant to ensure that the elements in tend to 1 or 0; generally, = .

[0059] S4.3: Use to weight the spatial attention map , i.e., by eliminate the attention score at the point with small attention weight, resulting in enhanced multi-scale spatial attention features ;

[0060] S5: Use the enhanced spatial attention pyramid and the channel attention vector of each layer to weight the multi-scale pyramid features in S2, resulting in a multi-scale attention feature pyramid for classification , see Figure 3 for specific implementation details module, which can be represented by the formula: where is broadcast addition, is bitwise multiplication;

[0061] S6: Construct a classifier for each layer of features in the multi-scale attention feature pyramid, and each classifier consists of a global average pooling layer and two fully connected layers;

[0062] S7: Combine the outputs of the classifiers in Figure 1The cervical cell image dataset shown is divided into training, validation, and test sets in a 7:1:2 ratio. The entire network is trained using gradient descent with three loss functions. After 200 training epochs, the network with the highest classification accuracy on the validation set is used to predict cervical cell classification on the test set. The final prediction result is the mean of the predictions from all classifiers. The three loss functions consist of the joint cross-entropy loss of all classifiers. Smoothing loss of masks and loss of channel feature diversity The weighted sum composition, i.e. ,in λ and μ It is a constant used to balance the various losses.

[0063] In the specific implementation process, smoothing out losses The boundary smoothing used to constrain the attention mask is applicable to attention masks of various scales. The sum of the first-order gradients is given by the following formula:

[0064]

[0065] in, The total number of attention masks. and They are the first The length and width of an attention mask, where m and n represent the coordinate indices of a point in the attention mask; Let represent the gradient at coordinates (m,n) in the i-th attention mask, obtained using the first-order gradient operator in this invention. Channel feature diversity loss. This method forces the feature channels output by the last convolutional layer in the classifier to be discriminative for a specific class, thereby increasing the diversity of features in each channel. The calculation process includes:

[0066] First, the classification features of each layer are divided into 5 groups according to channels (number of cervical cell types), and each group contains Channel characteristics, This represents the number of classification feature channels in the current layer.

[0067] Next, cross-channel average pooling is used to divide the data within each group. The features of each channel are aggregated into a single channel feature, and then global average pooling is used to calculate a class response value for each group of features. The softmax function is used to normalize all five response values.

[0068] Finally, the cross-entropy loss value of the normalized 5 response value vector and the current image category one-hot encoding is calculated, and the channel diversity loss of the current layer feature is calculated, and the average of the channel diversity loss of each layer is calculated as the channel diversity loss of the current network.

[0069] In today's cervical cancer patients tend to be younger, effective and automated cervical cell classification technology is essential to reduce the incidence of cervical cancer and mortality. In view of the two difficulties faced by the current cervical cell automatic classification task, the application proposes a cervical cell classification method based on multi-scale attention feature enhancement, which uses multi-scale attention features enhanced by attention mask for cervical cell classification, realizes the simultaneous use of high-level semantic features and more detailed low-level features paid attention to by general classification network, and uses attention mask to make the network pay more attention to the nucleus region which is very important for cervical cell classification, thereby further improving the accuracy of cervical cell classification from the perspective of improving interpretability.

[0070] The above examples are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cervical cell classification method based on multi-scale attention feature enhancement, characterized in that, The steps are as follows: S1: extracting multi-scale features from cervical cell images using a deep convolutional neural network comprising multiple blocks; the deep convolutional neural network is the backbone of any general classification network, which is divided into n blocks, and the output feature map of each block has a different scale; S2: constructing a multi-scale feature pyramid according to the feature fusion method in FPN; in S2, the features of the deep block output are sampled to the same size as the shallow block output feature map, then added to the shallow block output feature map to realize the fusion of features with different scales from top to bottom, and the multi-scale feature pyramid is constructed using the features after the fusion of each block output feature and deep feature; S3: using APN to calculate the spatial attention and channel attention of each layer of the multi-scale pyramid feature respectively, and generating a spatial attention pyramid and a channel attention vector of each layer; S3 includes: S3.1: For each layer feature in the multi-scale pyramid feature perform deconvolution and sigmoid normalization, generate the spatial attention map corresponding to each layer feature ; S3.2: For each layer feature in the multi-scale pyramid After passing through global average pooling and two fully connected layers in turn, the initial channel attention is generated ; S3.3: use the initial channel attention of the current layer with the channel attention of the shallow layer additive fusion to obtain the final channel attention of the current layer to realize bottom-up different scale attention feature fusion; S4: using the masks obtained by threshold segmentation of the spatial attention pyramid to enhance the spatial attention; S5: using the enhanced spatial attention pyramid and the channel attention vector of each layer to weight the multi-scale pyramid feature in S2, to obtain a multi-scale attention feature pyramid for classification; S6: constructing a classifier for each layer of feature in the multi-scale attention feature pyramid respectively; S7: using three loss functions to perform end-to-end gradient descent optimization training on the entire network, and using the trained classification network to classify and predict cervical cells.

2. The method of claim 1, wherein: Step S4 includes the following sub-steps: S4.1: Compute spatial attention map for each layer the mean of ; S4.2: using as a threshold to the spatial attention map segmenting, to obtain a corresponding segmentation mask ; S4.3: Utilize Spatial attention map By weighting and removing attention scores at points with smaller attention weights, the enhanced multi-scale spatial attention features are obtained.

3. The method of claim 2, wherein: In step S4.2, the spatial attention map is used as a threshold The method for segmentation is: values greater than or equal to are assigned a value of 1, values less than are assigned a value of 0.

4. The method of claim 1, wherein: In step S5, each value in the channel attention vector corresponds to a channel in the current layer feature map, and each attention value is weighted to all feature values in the corresponding channel through multiplication.

5. The method of claim 1, wherein: The classifier in step S6 is composed of a global average pooling layer and several fully connected layers.

6. The method of claim 1, wherein: In step S7, the three loss functions consist of a weighted sum of the joint cross-entropy loss of all classifiers the smoothness loss of masks and the channel feature diversity loss i.e. where λ and μ are constants for balancing the losses.

7. The method of claim 1, wherein: In step S7, the result of the classification prediction is the mean of all classifier prediction results.

Images