Cervical cell pathology classification system based on mixed multiple attention

Abstract

The invention discloses a cervical cell pathology classification system based on mixed multiple attention, on the basis of Faster RCNN, a Resnet101 module in a network model is improved, a mixed multiple attention pipeline module is provided, the mixed multiple attention pipeline module comprises a class enhanced attention (CEA) module branch implanted with a class channel attention (CCA) module and a regional shuffling attention (RSA) module branch, and the CCA module branch and the RSA module branch are connected with the network model. And fusing the output features of the two branches and the local representation. According to the method, the relationship between different ROI regions is obtained through three attention mechanism modules, and the ROI features are enhanced through the context of the relationship between cells, so that the sensitivity of the model to details and the accuracy of classification are enhanced, and the MMAP model is more efficient and accurate when processing complex images.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and pathology classification, and in particular to a cervical cell pathology classification system based on hybrid multiple attention. Background Art

[0002] In the current medical field, the use of liquid-based cytology images for cervical cancer screening is particularly critical, and it plays an important role in the prevention and control of cervical cancer. When conducting cervical cancer screening in my country, it is generally relied on doctors to manually interpret pathological images. This process not only consumes a lot of doctors' time, but may also affect the accuracy of screening due to various factors such as doctors' experience and subjective judgment, thereby increasing the risk of misscreening or missed screening. However, with the rapid advancement of deep learning technology, computer-assisted cervical cancer screening has become a reality. Despite this, the current automated screening methods for cervical cancer still have shortcomings in terms of detection accuracy, mainly in the following two aspects:

[0003] First, most of the current target detection algorithms usually rely on local features extracted from the region of interest (ROI) or anchor box to identify and locate the target when performing detection tasks. When processing natural images, such as the cat and dog classification task, due to the obvious differences between objects of different categories, local features are often sufficient to achieve effective classification. However, in the application scenario of cervical cell detection, cells of different categories show a high degree of intra-class diversity and inter-class similarity, making classification based solely on the features of a single region inaccurate.

[0004] Current research on cervical cell detection based on the target detection framework mainly focuses on analyzing cell features from a single perspective, lacking consideration of cell-to-cell interactions and feature associations. This approach ignores the complex relationships that may exist between cells, thus limiting the effectiveness of the detection system in distinguishing between subtly different cell categories. Especially in the case of cervical cells, the shortcomings of this approach are more prominent because they cannot fully utilize the correlation between cells to enhance classification accuracy.

[0005] Second, in clinical screening practice, when pathologists identify abnormal cells, their judgment is not based solely on the characteristics of the target cells themselves, but on the characteristics of possible normal cells around them and the overall information of the image. This method helps to improve the accuracy of screening. For example, Figure 6 As shown, the classification of the same target cell may be different when referring to different neighboring cells; if referring to cell A, the target cell in the red frame may be classified as an "ASC-US" type of lesion, while when referring to cell B, the same cell may be identified as normal.

[0006] Therefore, how to provide a cervical cell pathology classification system that can comprehensively consider the characteristics of cell-to-cell interactions and process complex images more efficiently and accurately is a technical problem that needs to be urgently solved by those skilled in the art. Summary of the invention

[0007] In view of the above research status and existing problems, the present invention provides a cervical cell pathology classification system based on hybrid multiple attention. On the basis of FasterRCNN, the Resnet101 module in the network model is improved, and a mixed multiple attention pipeline (Mixed Multiple Attention Pipeline, MMAP) module is proposed. The relationship between different ROI areas is obtained through three attention mechanism modules, and the ROI features are enhanced through the inter-cell relationship context, thereby enhancing the model's sensitivity to details and classification accuracy, making the MMAP model more efficient and accurate in processing complex images.

[0008] The present invention provides a cervical cell pathology classification system based on hybrid multiple attention, comprising: based on the Faster RCNN network framework, including: a Resnet101 module, and a hybrid multiple attention module cascaded after the Resnet101 module; wherein,

[0009] The Resnet101 module includes:

[0010] The backbone network module is used to receive the input cervical cell image and extract the convolution feature map;

[0011] The region proposal network module is used to obtain a region candidate frame according to the convolutional features;

[0012] The ROI Align module is used to map the region candidate box to the convolution feature map to obtain the ROI feature map corresponding to the region candidate box;

[0013] The hybrid multi-attention module includes:

[0014] The CEA module is used to receive the ROI feature map, calculate the weighted class affinity matrix generated in a dense manner through the class affinity mapping, and the CCA module is used to receive the class affinity matrix, adaptively adjust the category level through a linear transformation function, generate an adaptively weighted class affinity matrix, and generate a feature map Y accordingly;

[0015] The RSA module is used to receive the ROI feature map, and use a shuffle operation to convert a dense point-to-point affinity matrix into a sparse affinity matrix based on global information, thereby generating a feature map Z;

[0016] The feature map fusion module is used to fuse the feature map Y, the feature map Z and the ROI feature map to obtain the final detection image.

[0017] Preferably, the CEA module integrated with the CCA module follows the self-attention mechanism to extract the correlation information of the class channel and adaptively aggregate the long-distance context information in the category perspective, and finally generates a feature map Y with the same size as the input ROI feature map after dimension reduction.

[0018] Preferably, the CEA module performs the following steps:

[0019] The ROI feature map output by the Resnet101 module generates two feature maps through two convolutional layers: X'∈R C'×H×W , P∈R N×H×W , the number of channels of the input feature map is C, the height is H, and the width is W. C' is the number of channels corresponding to the reduction of local features in the channel, X' is the local feature after the reduction under the number of channels C', and P is the category attention map with true value labels;

[0020] After aggregating all positions i of X' and P in the spatial dimension, the softmax function is performed on the class dimension to generate the class affinity graph A∈R C ' ×N , N is the total number of categories; the class affinity operation is as follows:

[0021]

[0022] In the formula, s u,k Represents feature X' u With P k The category correlation between them, u ranges from [1,2,...,C'], k ranges from [1,2,...,N], j ranges from [1,2,...,C'], and S belongs to dimension R C×N space, in P∈R N×H×W In each category k, P k,i ∈R H×W It is used to record the probability that all pixels at position i belong to category k, P j,i Used to record the probability that the pixel at position i belongs to channel number j, X' u,i is the feature of all positions i under the u-th channel of X' along the channel dimension;

[0023]

[0024] Where Y u To output the feature map Y∈R C×H×W The u-th feature layer, a u,kis s obtained after calculation by the softmax layer u,k The scalar value, P k represents the probability of category k after batch normalization and ReLU activation function processing, P j represents the probability of the number of channels j after batch normalization and ReLU activation function processing, X represents the original local features, and X u Represents the feature map X∈R C×H×W For the u-th feature layer of , the functions δ(·) and ρ(·) are transformation functions consisting of batch normalization, ReLU activation function, and 1×1 convolution.

[0025] Preferably, the CCA module performs the following steps:

[0026] The category attention map P∈R output by the CEA module N×H×W Adaptive channel statistics weighting factor W k :

[0027]

[0028] In the formula, is the channel-wide average pooling operation that generates class-dependent statistics; σ is the Sigmoid activation function;

[0029] make The key adaptive feature recalibration function is:

[0030]

[0031] Where W 2 ∈R N×αN , W 1 ∈R αN×N are two linear fully connected transformation layers as dimension scaling layers with ratio α; η is the ReLU function;

[0032] The final output of the CCA module is based on the weighting factor W k And the class affinity graph A, adding residual connections to recalibrate A to obtain:

[0033] Where γ is a learnable parameter initialized to 0, W k is the adaptive channel statistical weighting factor, A k Represents the class affinity subgraph A after each single calibration by the weighted factor 1 , A 2 , ..., A k , ..., A N .

[0034] Preferably, the RSA module divides the ROI feature map into regions through a permutation operation, each region is fed into an adaptive global average pooling layer to obtain a point-by-point representation of the region; and the point-by-point representations of all regions are merged to generate a sparse representation of the entire ROI feature map.

[0035] Preferably, the RSA module performs the following steps:

[0036] The ROI feature map X output by the Resnet101 module is divided into G partitions, each partition X G Contains Q positions, q∈R C×Q Yes X G A subset of

[0037] Each partition is fed into an adaptive global average pooling layer, which merges the point statistics after global average pooling to obtain a sparse representation X m ∈R C×G ;

[0038] Apply the self-attention mechanism to X m , following the non-local operation:

[0039]

[0040] Z m =w A m g(X m )+Xm;

[0041] In the formula, A m ∈R G×G is a sparse affinity matrix based on global information; Z m ∈R C×G is the weighted output feature; θ(·) and φ(·) are transformation functions consisting of 1×1 convolution, batch normalization and ReLU activation function; g(·) is 1×1conv; w is a learnable parameter initialized to 0; d is a constant indicating the total number of partition blocks.

[0042] Preferably, the loss function is as follows:

[0043]

[0044] In the formula, λ 1 ,λ 2 and λ 3 is the balance coefficient; is the multi-class cross entropy loss; Cascade the convolutional network classification loss of the Resnet101 module for the CEA module; is the class attention loss in the CEA module; B is the image size of the current batch; N is the number of pixels in each batch; K is the number of categories; is the prediction of the kth class for the jth pixel in the ith patch after passing through the ResNet101 module; is the indicator function. When the jth position in the i-th patch, that is, When the true value label of belongs to the kth class, the value is 1, otherwise the value is 0; is the response value of the class attention map of the j-th pixel in the ith patch to the k-th class.

[0045] Preferably, a cell classification head and a bounding box regression head are further included, and the detection image is input into the cell classification head and the bounding box regression head, and two different head networks are used for classification and regression respectively.

[0046] The cervical cell pathology classification system based on hybrid multiple attention proposed in the present invention has the following beneficial effects compared with the prior art:

[0047] The dual-stage target detection framework is adopted, and the Resnet101 module is improved on the basis of Faster RCNN. On Resnet101, by cascading a feature fusion module MMAP based on the attention mechanism, the intra-regional relationship between cervical cells and the contextual information of the overall image can be fused into the local ROI features, thereby enhancing the performance of cell detection, improving the expression ability of the original ROI features, and optimizing the classification and positioning accuracy of cells.

[0048] The proposed MMAP effectively combines category correlation and regional information to facilitate more detailed and comprehensive feature analysis. This architecture not only considers the subtle differences between categories, but also adds a global perspective to identify complex patterns within the region, proving its effectiveness in improving detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0050] Figure 1 It is a block diagram of a cervical cell pathology classification system based on hybrid multiple attention provided by an embodiment of the present invention;

[0051] Figure 2 It is a network structure diagram of a hybrid multiple attention module MMAP provided by an embodiment of the present invention;

[0052] Figure 3 is a network structure diagram of a class enhanced attention module provided by an embodiment of the present invention;

[0053] Figure 4 is a network structure diagram of a channel-like attention module provided by an embodiment of the present invention;

[0054] Figure 5 is a network structure diagram of a regional shuffling attention module provided by an embodiment of the present invention;

[0055] Figure 6 is a schematic diagram of classification results of the same target cell under different neighboring cell references provided by an embodiment of the present invention;

[0056] Figure 7 is a schematic diagram of successfully identifying HSIL cells missed by the baseline method using the method of the present invention provided in an embodiment of the present invention;

[0057] Figure 8 It is a schematic diagram of successfully correcting the false positive classification error in the baseline method using the method of the present invention provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In view of the special needs of cervical cell detection, a detection method that can comprehensively consider the interaction characteristics between cells is needed. When classifying cells, in addition to considering the local characteristics of a single cell, it is also necessary to pay attention to the characteristics of surrounding cells and the global information of the entire image. This comprehensive consideration can simulate the thinking mode of pathologists in the actual screening process, and enhance the accuracy of cell classification judgment by analyzing the relationship between cells in the image. This requires not only in-depth analysis of the target cells themselves, but also comprehensive consideration of surrounding cells and the overall image context to achieve more accurate cell classification.

[0060] In view of the limitations of the current cervical cell detection technology in the classification process of single cell area features, the weak discrimination ability of single features, the low information interoperability between single small area images, and the limited generalization ability of network models, the present invention improves the Resnet101 module in the network model on the basis of Faster RCNN, and proposes a Mixed Multiple Attention Pipeline (MMAP) module. MMAP is optimized on the basis of ResNet 101 and connected to the Faster R-CNN network as a backbone network. This structure not only retains the high performance of Faster R-CNN as a two-stage target detection network, but also enhances the model's sensitivity to details and classification accuracy by introducing these three attention mechanism modules, making the MMAP model more efficient and accurate when processing complex images.

[0061] The cervical cell pathology classification system based on hybrid multi-attention provided by the embodiment of the present invention adopts a two-stage target detection framework. On the basis of Faster RCNN, the Resnet101 module is improved. On Resnet101, the ROI feature is enhanced by cascading a feature fusion module MMAP based on the attention mechanism. The overall process is as follows Figure 1 As shown in the figure. First, the image is input and the convolutional features are extracted through the backbone network; then the region proposal network (PRN) and ROI Align are used to obtain the region candidate box and the corresponding ROI feature map. Subsequently, a feature fusion module, the hybrid multi-attention module MMAP, is cascaded after the Resnet101 module to enhance the original ROI features. MMAP enhances the ROI features by exploring the contextual relationship between different ROIs.

[0062] Specifically, Figure 1-2 As shown, the cervical cell pathology classification system includes: based on the Faster RCNN network framework, including: Resnet101 module, and a hybrid multi-attention module cascaded after the Resnet101 module; wherein,

[0063] The Resnet101 module includes:

[0064] The backbone network module is used to receive the input cervical cell image and extract the convolution feature map;

[0065] The region proposal network module is used to obtain region candidate boxes based on convolutional features;

[0066] The ROI Align module is used to map the region candidate box to the convolution feature map and obtain the ROI feature map corresponding to the region candidate box [R 1 ...R N ];

[0067] The hybrid multi-attention module includes:

[0068] The CEA module is used to receive the ROI feature map, calculate the weighted class affinity matrix generated in a dense manner through the class affinity mapping, and the CCA module is used to receive the class affinity matrix, adaptively adjust the category level through the linear transformation function, generate an adaptively weighted class affinity matrix, and generate the feature map Y accordingly;

[0069] The RSA module is used to receive the ROI feature map, and use the shuffle operation to transform the dense point-to-point affinity matrix into a sparse affinity matrix based on global information, and generate the feature map Z accordingly;

[0070] The feature map fusion module is used to fuse the feature map Y, feature map Z and ROI feature map to obtain the final detection image [R' 1 ...R' N ].

[0071] In this embodiment, Figure 2 As shown, the main structure of MMAP is two branches working in parallel:

[0072] One branch includes a Class Enhance Attention (CEA) module embedded with a Class Channel Attention (CCA) module. For the input features, the CEA module first calculates the class-based correlation and then generates weighted class representations in a dense manner through class affinity mapping. In addition, the introduction of the CCA module helps to adaptively adjust the class-level information through two linear transformation functions, significantly enhancing the ability to distinguish classes with fewer parameters.

[0073] The other branch is the Regional Shuffle Attention (RSA) module, which aims to capture global information within the region using the shuffle operation to achieve a robust representation of the relationship between objects. In the final step, this embodiment fuses the output features of the two branches and the local representation and feeds them into the convolutional layer to produce a more accurate detection image.

[0074] Through this design, MMAP effectively combines category correlation and regional information to facilitate more detailed and comprehensive feature analysis, thereby improving the performance of object detection. This architecture not only considers the subtle differences between categories, but also adds a global perspective to identify complex patterns within the region, proving its effectiveness in improving detection accuracy.

[0075] In one embodiment, the CEA module integrated with the CCA module follows the self-attention mechanism to extract the correlation information of the class channel and adaptively aggregate the long-distance context information in the category perspective, and finally generates a feature map Y with the same size as the input ROI feature map after the feature map size is reduced by the reduction operation.

[0076] In this embodiment, in MMAP, the CEA and CCA modules are combined into the top-level branch of the network. The role of the CEA module is to capture the interdependence between specific categories and corresponding features after dimensionality reduction processing. It explores the similarity between categories and channels of input features through matrix operations. This process helps to generate more detailed feature representations, which are more sensitive to the category information of the object, thereby improving the classification accuracy of the network. The CCA module is designed as an adaptive feature reshaping tool to optimize the feature expression of category information (as shown in formula (4)). It is worth noting that the CCA module receives the class affinity matrix (as shown in formula (1)) and the class attention map generated by the CEA module as input to generate an adaptively weighted class affinity matrix. Under ideal conditions, given the input feature map X∈R C×H×W , where C, H and W represent the number of channels, height and width of the feature map respectively. CEA integrated with CCA module can effectively extract the correlation of class channels and adaptively aggregate long-distance context information in the category perspective, and finally generate a feature map Y∈R of the same size as the input C×H×W , which follows the self-attention mechanism. In this way, the model can refine the feature representation and enhance the sensitivity and recognition of different categories.

[0077] In one embodiment, the calculation principle of the self-attention mechanism can be viewed as a mathematical matrix multiplication operation, where the two key dimensions involved are the number of channels C of the input feature map and the product of the height H and the width W.

[0078] The standard channel affinity matrix, of size C×C, can be obtained by multiplying two input matrices of size C×HW and HW×C respectively. Intuitively, the definition of non-local operation restricts the channel scaling in this channel attention module, that is, the query, key and value transformation functions are omitted in the process. However, if a channel in channel C is replaced by a region mapping with ground-truth annotations while keeping the query, key and value transformation functions, this will effectively incorporate the category information. The class-enhanced attention mechanism captures the long-range context information from the perspective of category information, which involves explicitly modeling the interaction between each category in the dataset and each channel of the input feature cube.

[0079] The following is a detailed description of the specific process of obtaining class-level context information. Figure 3 Shows the local features X∈R C×H×W Here X represents the 3×3 convolutional layer output of the backbone network ResNet101. The CEA module performs the following steps:

[0080] The ROI feature map output by the Resnet101 module is passed through two convolutional layers to generate two feature maps: X'∈R C ' ×H×W , P∈R N×H×W , the number of channels of the input feature map is C, the height is H, and the width is W. C' is the number of channels corresponding to the reduction of local features in the channel, X' is the local feature after the reduction under the number of channels C', and P is the category attention map with true value labels;

[0081] After aggregating all positions i of X' and P in the spatial dimension, the softmax function is performed on the class dimension to generate the class affinity graph A∈R C ' ×N , N is the total number of categories; the class affinity operation is as follows:

[0082]

[0083] In the formula, s u,k Represents feature X' u With P k The category correlation between them, u ranges from [1,2,...,C'], k ranges from [1,2,...,N], j ranges from [1,2,...,C'], and S belongs to dimension R C×N space, in P∈R N×H×W In each category k, P k,i ∈R H×W It is used to record the probability that all pixels at position i belong to category k, P j,i Used to record the probability that the pixel at position i belongs to channel number j, X'u,i is the feature of all positions i under the u-th channel of X' along the channel dimension;

[0084]

[0085] Where Y u is the output feature map Y∈R C×H×W The u-th feature layer, a u,k is s obtained after calculation by the softmax layer u,k The functions δ(·) and ρ(·) are transformation functions consisting of 1×1 convolution, batch normalization, and ReLU activation functions.

[0086] In order to improve the expressiveness of the features, this embodiment incorporates the original local features X. As shown in formula (2), the ultimate representation of each channel is the result of the class-based weighted sum of all channels in the class attention map, which reflects the class-based interdependence between feature maps. The CEA module enhances the perception and recognition of class-level information in an intuitive way.

[0087] In one embodiment, high-level semantic information in CNN is mainly embedded in the channel dimension, and the channel mapping of each deep feature is regarded as a response related to a specific category. In addition, it is important to model channel correlation in classification and segmentation tasks. Therefore, a class channel attention (CCA) module is introduced, which creates a class correlation graph rich in adaptive contextual information based on the dependency relationship of class channels, and then effectively integrates this graph into the CEA module. This method utilizes the intrinsic semantic relationship of the channel to enhance the network's ability to represent category information. Figure 4 As shown, the CCA module performs the following steps:

[0088] The category attention map P∈R output by the CEA module N×H×W Adaptive channel statistics weighting factor W k :

[0089]

[0090] In the formula, is the channel-wide average pooling operation that generates class-dependent statistics; σ is the Sigmoid activation function;

[0091] make The key adaptive feature recalibration function is:

[0092]

[0093] Where W 2 ∈RN×αN , W 1 ∈R αN×N are two linear fully connected transformation layers, which serve as dimension scaling layers with a ratio of α to enhance and compress the category information expression in the channel dimension respectively; η is the ReLU function, which aims to optimize the information representation in the channel dimension while maintaining the simplicity of the model structure;

[0094] The final output of the CCA module is based on the weighting factor W k And the class affinity graph A, add residual connections to recalibrate A to obtain: (5);

[0096] Here, γ is a learnable parameter initialized to 0. By adding residual connections and keeping the original representation (see “+1” in formula (5)), it can be integrated into the above standard CEA module without destroying its initial behavior, effectively enhancing the feature adaptive recalibration of class information.

[0097] like Figure 3 As shown, A' (with a size of C'×N) is used to perform matrix multiplication operation with the transposed and reshaped P' (with a size of N×H×W) to obtain X" (with a size of C'×H×W) to achieve CEA enhanced calculation.

[0098] In one embodiment, the RSA module divides the ROI feature map into regions through a permutation operation, each of which is fed into an adaptive global average pooling layer to obtain a point-by-point representation of the region; the point-by-point representations of all regions are merged to generate a sparse representation of the entire ROI feature map. Therefore, for convenience, the self-attention to the original input features can be effectively replaced with the same attention to the merged corresponding features.

[0099] In this embodiment, the regional shuffle attention (RSA) module converts the dense point-to-point affinity matrix into two sparse corresponding matrices based on regions. These sparse matrices can efficiently capture the sparse global context through adaptive average pooling. The RSA module combines these two affinity matrices to fully obtain the detailed spatial context information of the local input feature X and generate the output feature Z∈R accordingly. C×H×W Subsequently, we integrate the output features of the two branches {Y, Z} with the local feature representation X to form an enhanced feature representation. Then, this fused feature is fed into the convolutional layer to produce a refined detection map.

[0100] In this way, the RSA module effectively uses regional information to enhance the understanding of the global context, thereby providing the model with richer and more comprehensive context information. This process ensures that the network can comprehensively consider various feature information and ultimately generate detection results with high accuracy and rich details.

[0101] It should be noted that attention-based neural networks, in terms of spatial point-wise correlation representation, mainly capture long-range context dependencies through the self-attention mechanism or its variants, and finally generate a dense affinity matrix. Even for smaller feature maps, the obtained point-by-point correlation matrix will occupy a lot of (GPU) memory. Therefore, the key point of regional shuffling attention is to obtain the reorganized regional dependencies and their correspondences in a sparse and efficient way.

[0102] In one embodiment, Figure 5 As shown, it is an example of regional shuffle attention when the number of partitions G and the position of each partition P are both 2. Although the self-attention on the merged features can empirically capture long-range contextual information from all positions, the pixel-to-pixel connection is still vague. In order to mine clearer contextual dependencies from a regional perspective, this embodiment applies shuffle attention to alternately pool the corresponding sub-regions and calculates their self-attention representations separately, thereby achieving complementary representation of spatial information. Further experiments show that the cascade of the attention-weighted representations of the two sub-regions can effectively enhance contextual dependencies, which is better than the pixel-by-pixel non-local operator. The RSA module executes the following steps:

[0103] The ROI feature map X output by the Resnet101 module is divided into G partitions, each partition X G Contains Q positions, q∈R C×Q Yes X G A subset of

[0104] Each partition is fed into an adaptive global average pooling layer, which merges the point statistics after global average pooling to obtain a sparse representation X m ∈R C×G ;

[0105] Apply the self-attention mechanism to X m , following the non-local operation:

[0106]

[0107] Z m =w A m g(X m )+X m (7);

[0108] In the formula, A m ∈R G×G is a sparse affinity matrix based on global information; Z m ∈R C×G is the weighted output feature; θ(·) and is a conversion function consisting of 1×1 convolution, batch normalization and ReLU activation function; g(·) is 1×1conv; w is a learnable parameter initialized to 0; d is a constant representing the total number of partitioned blocks, such as Figure 5 As described above, the total number of partitioned small blocks d=the number of partitions when the partition number G has a stride of 2 (4 partitions)×the number of sub-blocks when the stride P in each small partition has a stride of 2 (4 sub-blocks)=4×4=16.

[0109] Area weighted representation X' G Can be done through Z m and X G The specific steps are:

[0110] Step 1: First, X G Perform partition and reorganization operations to obtain X m , and then X m Compute S using self-attention A , and obtain its self-attention representation Z m The purpose of this step is to use the complementary representation of spatial information to obtain a "weight" Z m ;

[0111] Step 2: Z m With X G The original region can be expressed as a weighted region by matrix operation, namely X' G The purpose of this step is to preserve the contextual dependencies of the original pixels instead of directly G Perform a regional shuffle.

[0112] Step 3: Regional weighted representation X' G Get X by region shuffling attention s . Specifically, for X' G After shuffling, the cards are reassembled. The specific process of shuffling is a shuffle operation. This embodiment adopts a fixed rule shuffle, that is, shuffling is performed based on a predefined order or pattern. Such an operation can be more stable during training. Figure 5 , except for the first small block in the first sub-region, the small blocks in each sub-region are extended by one stride unit and then re-embedded into the original block. Feature X S will be input into the same regional attention block for X s Repeat the above steps 1 and 2 to finally generate the final representation Z.

[0113] Compared with the standard self-attention mechanism, our method greatly reduces the time and space complexity from O((H×W) 2 C) to Among them G h and Gw is the number of partitions along the height and width dimensions, and each partition contains P h and P w Pixels, where P h represents G obtained along the height partition h The pixel size in a partition, P w represents G obtained by partitioning along the width w The specific calculation method is to assume that the pixel value of the input image is 512×512, denoted as S, then P h =[S / (G h ×G w )]×G h ,}=S / G w ,P w =[S / (G h ×G w )]×G w ,}=S / G h .

[0114] Overall, the proposed region-shuffling attention module makes up for the huge memory footprint of non-local blocks. Moreover, it can be inserted into any existing architecture at any stage without destroying its initial performance and optimized in an end-to-end manner.

[0115] In one embodiment, in addition to using the conventional multi-class cross entropy loss Two additional auxiliary losses are also used to improve the performance. One is the auxiliary supervision loss Second, auxiliary supervision loss Specifically, the loss function is as follows:

[0116]

[0117] In the formula, λ 1 ,λ 2 and λ 3 is the balance coefficient, the values ​​can be set to 1, 0.5 and 0.4 respectively to balance the loss; is the multi-class cross entropy loss; The convolutional network classification loss of the Resnet101 module is cascaded to the CEA module, such as Figure 3 As shown, it is the classification loss of the 3×3 convolutional network module; is the class attention loss in the CEA module; B is the image size of the current batch; N is the number of pixels in each batch; K is the number of categories; is the prediction of the kth class for the jth pixel in the ith patch after passing through the ResNet101 module; is the indicator function. When the jth position in the i-th patch, that is, When the true value label of belongs to the kth class, the value is 1, otherwise the value is 0; is the response value of the class attention map of the j-th pixel in the ith patch to the k-th class.

[0118] In one embodiment, a cell classification head and a bounding box regression head are further included. The detection image is input into the cell classification head and the bounding box regression head, and two different head networks are used for classification and regression respectively.

[0119] Methods A series of model comparison experiments were carried out on the cervical cell detection dataset collected in this paper. The experimental results showed that MMAP effectively improved the recognition accuracy. Figure 7 and Figure 8 The detection results of cervical cells by the baseline method and the MMAP method proposed in this study are demonstrated. Figure 7 and Figure 8 The left image is the true value image, and the right image is the benchmark model detection image and the detection image of the method of the present invention from top to bottom. Figure 7 In the example 2, the baseline method missed the cells in the dotted box marked as "HSIL" cells, resulting in missed detection. However, the detector using the CCA, CEA and RSA modules of the present invention successfully identified the "HSIL" cells. Figure 8 In the experiment, the baseline method had a classification error (false positive). The false positive phenomenon in the baseline method was corrected by the method of cascading CEA and RSA modules of the present invention. It was confirmed that the performance of the method of the present invention surpassed the most advanced standard target detection algorithm and the algorithm dedicated to cervical cell detection, showing its significant superiority.

[0120] The above is a detailed introduction to a cervical cell pathology classification system based on hybrid multiple attention provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0121] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A cervical cell pathology classification system based on hybrid multiple attention, characterized in that: Based on the Faster RCNN network framework, it includes: a Resnet101 module, and a hybrid multi-attention module cascaded after the Resnet101 module; wherein, The Resnet101 module includes: The backbone network module is used to receive the input cervical cell image and extract the convolution feature map; The region proposal network module is used to obtain a region candidate frame according to the convolutional features; The ROI Align module is used to map the region candidate box to the convolution feature map to obtain the ROI feature map corresponding to the region candidate box; The hybrid multi-attention module includes: The CEA module is used to receive the ROI feature map, calculate the weighted class affinity matrix generated in a dense manner through the class affinity mapping, and the CCA module is used to receive the class affinity matrix, adaptively adjust the category level through a linear transformation function, generate an adaptively weighted class affinity matrix, and generate a feature map Y accordingly; The RSA module is used to receive the ROI feature map, and use a shuffle operation to convert a dense point-to-point affinity matrix into a sparse affinity matrix based on global information, thereby generating a feature map Z; The feature map fusion module is used to fuse the feature map Y, the feature map Z and the ROI feature map to obtain the final detection image.

2. A cervical cell pathology classification system based on hybrid multiple attention according to claim 1, characterized in that: The CEA module integrated with the CCA module follows the self-attention mechanism to extract the correlation information of the class channels and adaptively aggregate the long-distance context information in the category perspective, and finally generates a feature map Y with the same size as the input ROI feature map after dimension reduction.

3. A cervical cell pathology classification system based on hybrid multiple attention according to claim 1, characterized in that: The CEA module performs the following steps: The ROI feature map output by the Resnet101 module generates two feature maps through two convolutional layers: X'∈R C ' ×H×W , P∈R N×H×W , the number of channels of the input feature map is C, the height is H, and the width is W. C' is the number of channels corresponding to the reduction of local features in the channel, X' is the local feature after the reduction under the number of channels C', and P is the category attention map with true value labels; After aggregating all positions i of X' and P in the spatial dimension, the softmax function is performed on the class dimension to generate the class affinity graph A∈R C ' ×N , N is the total number of categories; the class affinity operation is as follows: In the formula, s u,k Represents feature X' u With P k The category correlation between them, u ranges from [1,2,...,C'], k ranges from [1,2,...,N], j ranges from [1,2,...,C'], and S belongs to dimension R C×N space, in P∈R N×H×W In each category k, P k,i ∈R H×W It is used to record the probability that the pixel at position i belongs to category k, P j,i Used to record the probability that the pixel at position i belongs to channel number j, X' u,i is the feature of all positions i under the u-th channel of X' along the channel dimension; Where Y u To output the feature map Y∈R C×H×W The u-th feature layer, a u,k is s obtained after calculation by the softmax layer u,k The scalar value, P k represents the probability of category k after batch normalization and ReLU activation function processing, P j represents the probability of the number of channels j after batch normalization and ReLU activation function processing, X represents the original local features, and X u Represents the feature map X∈R C×H×W For the u-th feature layer of , the functions δ(·) and ρ(·) are transformation functions consisting of batch normalization, ReLU activation function, and 1×1 convolution.

4. A cervical cell pathology classification system based on hybrid multiple attention according to claim 3, characterized in that: The CCA module performs the following steps: The category attention map P∈R output by the CEA module N×H×W Adaptive channel statistics weighting factor W k : In the formula, is the channel-wide average pooling operation that generates class-dependent statistics; σ is the Sigmoid activation function; make The key adaptive feature recalibration function is: Where W2∈R N×αN , W1∈R αN×N are two linear fully connected transformation layers as dimension scaling layers with ratio α; η is the ReLU function; The final output of the CCA module is based on the weighting factor W k And the class affinity graph A, adding residual connections to recalibrate A to obtain: Where γ is a learnable parameter initialized to 0, A k Represents the class affinity subgraphs A1, A2, ..., A after each single calibration by the weighted factor k , ..., A N .

5. The cervical cell pathology classification system based on hybrid multiple attention according to claim 1, characterized in that: The RSA module divides the ROI feature map into regions through a permutation operation, and each region is fed into an adaptive global average pooling layer to obtain a point-by-point representation of the region; the point-by-point representations of all regions are merged to generate a sparse representation of the entire ROI feature map.

6. The cervical cell pathology classification system based on hybrid multiple attention according to claim 1, characterized in that: The RSA module performs the following steps: The ROI feature map X output by the Resnet101 module is divided into G partitions, each partition X G Contains Q positions, q∈R C×Q Yes X G A subset of Each partition is fed into an adaptive global average pooling layer, which merges the point statistics after global average pooling to obtain a sparse representation X m ∈R C×G ; Apply the self-attention mechanism to X m , following the non-local operation: Z m =w A m g(X m )+X m ; In the formula, A m ∈R G×G is a sparse affinity matrix based on global information; Z m ∈R C×G is the weighted output feature; θ(·) and φ(·) are transformation functions consisting of 1×1 convolution, batch normalization, and ReLU activation functions; g(·) is a 1×1 conv; w is a learnable parameter initialized to 0; d is a constant representing the total number of partition blocks.

7. The cervical cell pathology classification system based on hybrid multiple attention according to claim 1, characterized in that: The loss function is as follows: Where λ1, λ2 and λ3 are balance coefficients; is the multi-class cross entropy loss; Cascade the convolutional network classification loss of the Resnet101 module for the CEA module; is the class attention loss in the CEA module; B is the image size of the current batch; N is the number of pixels in each batch; K is the number of categories; is the prediction of the kth class for the jth pixel in the ith patch after passing through the ResNet101 module; is the indicator function. When the jth position in the i-th patch, that is, When the true value label of belongs to the kth class, the value is 1, otherwise the value is 0; is the response value of the class attention map of the j-th pixel in the ith patch to the k-th class.

8. The cervical cell pathology classification system based on hybrid multiple attention according to claim 1, characterized in that: It also includes a cell classification head and a bounding box regression head, the detection image is input into the cell classification head and the bounding box regression head, and two different head networks are used for classification and regression respectively.

Images