Mammary gland immunohistochemical Ki-67 image segmentation method based on attention mechanism

Abstract

The invention discloses a mammary gland immunohistochemical Ki-67 image segmentation method based on an attention mechanism, and the method comprises the following steps: obtaining a mammary gland immunohistochemical Ki-67 image, and carrying out the segmentation processing of the mammary gland immunohistochemical Ki-67 image, and obtaining mammary gland immunohistochemical Ki-67 image blocks; taking the breast immunohistochemical Ki-67 image block as input, taking the corresponding cell type classification result as output, and training a pre-constructed breast cell segmentation model; a mammary gland cell segmentation model is constructed based on a VGG16 network and a cooperative attention mechanism of a space and a channel; segmenting a mammary gland immunohistochemical Ki-67 image to be predicted into a plurality of image blocks, and inputting the image blocks into the trained mammary gland cell segmentation model for segmentation; and obtaining a distribution density map of classification results of different types of cells. The invention relates to the technical field of image segmentation, and by coordinating the relationship between spatial features and channel features and enhancing the guiding capability of the features, semantic feature differences are effectively relieved, and the precision, flexibility and adaptability of tumor cell segmentation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and more specifically to a breast immunohistochemistry Ki-67 image segmentation method based on an attention mechanism. Background Art

[0002] Ki-67 protein is an important marker of cell proliferation and plays a key role in the pathological analysis of tumors, especially in the diagnosis and prognosis assessment of tumors such as breast cancer. The traditional Ki-67 index method relies on manual counting by pathologists, which has problems such as poor accuracy, lack of universality and cumbersome operation. It is urgent to develop automated technology to improve efficiency and accuracy.

[0003] At present, traditional methods mainly rely on morphological analysis, texture feature extraction and classification-based threshold processing of images. Morphological methods can capture the basic morphological characteristics of cells, texture analysis helps describe the texture information of images, and classification and threshold processing distinguish different types of cells or regions through preset rules. These traditional techniques usually require manual feature design and are easily affected by image noise and cell overlap, resulting in insufficient accuracy. Especially when dealing with complex morphology or mutual occlusion between cells, the performance of traditional methods is often unsatisfactory. In addition, these methods lack an understanding of the deep semantics of the image and have difficulty coping with the variability and complexity of complex pathological images, limiting their scope of application in clinical practice.

[0004] With the development of deep learning technology, automated image analysis methods based on convolutional neural networks (CNNs) have made significant progress in tumor cell detection and Ki-67 estimation. Deep learning methods can automatically learn features from images, especially in tasks such as cell detection, cell nucleus segmentation, and tumor classification. Compared with traditional methods, deep learning can avoid the tedious process of artificial feature design and has stronger robustness and generalization ability when processing complex images.

[0005] Although deep learning has made some progress in the field of tumor cell detection, the current models still have certain pain points and difficulties. Especially in complex scenes, the traditional decoder network structure cannot fully capture the correlation between space and channels, which makes the model easy to lose detail information when processing images with multi-level semantics, affecting segmentation accuracy and adaptability. Especially in complex tumor cell segmentation tasks, the decoder often cannot effectively coordinate local and global features, resulting in poor performance in complex backgrounds or areas with cell clusters.

[0006] Therefore, coordinating the characteristic relationship between space and channels, enhancing the feature guidance capability, and improving the accuracy of tumor cell segmentation are issues that technical personnel in this field need to solve urgently. Summary of the invention

[0007] In view of this, the present invention provides a breast immunohistochemistry Ki-67 image segmentation method based on the attention mechanism, which coordinates the relationship between spatial and channel features, enhances the feature guidance capability, effectively alleviates the semantic feature differences, and significantly improves the accuracy, flexibility and adaptability of tumor cell segmentation.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] The present invention provides a breast immunohistochemistry Ki-67 image segmentation method based on an attention mechanism, comprising the following steps:

[0010] S1, obtaining a breast immunohistochemistry Ki-67 image, performing segmentation processing, and obtaining a breast immunohistochemistry Ki-67 image block; the image block has a corresponding cell type label;

[0011] S2. Using breast immunohistochemistry Ki-67 image blocks as input and corresponding cell type classification results as output, a pre-constructed breast cell segmentation model is trained; wherein the breast cell segmentation model is constructed based on a VGG16 network and a spatial and channel collaborative attention mechanism;

[0012] S3, dividing the breast immunohistochemistry Ki-67 image to be predicted into multiple image blocks, and inputting the image blocks into the trained breast cell segmentation model for segmentation; obtaining a distribution density map of the classification results of different types of cells;

[0013] S4. Apply threshold method, distance transformation and watershed algorithm to the distribution density map for post-processing to obtain segmented cell regions; calculate the Ki-67 index and reverse display the cell positions.

[0014] Furthermore, the cell types in step S1 include immune-positive cells, immune-negative cells and lymphocytes, and the marked positions are at the coordinate positions of the center points of the cells.

[0015] Furthermore, the breast cell segmentation model in step S2 includes a feature extraction module and a reconstruction module;

[0016] The feature extraction module uses the VGG16 network to extract features from the breast immunohistochemistry Ki-67 image block to obtain a preliminary feature map;

[0017] The reconstruction module performs weighted processing on the preliminary feature map in spatial and channel dimensions through a spatial and channel collaborative attention mechanism to enhance the expression of overall features.

[0018] Furthermore, the feature extraction module includes five convolution blocks, each of which includes multiple convolution layers and pooling layers;

[0019] Among them, the first convolution block includes two convolution layers, which perform convolution operations on the breast immunohistochemistry Ki-67 image block; the second convolution block includes a maximum pooling layer and two convolution layers; the third convolution block, the fourth convolution block and the fifth convolution block include a pooling layer and three convolution layers respectively; after sequential convolution, the fifth convolution block outputs a preliminary feature map.

[0020] Furthermore, the reconstruction module includes four spatial and channel collaborative attention upsampling blocks;

[0021] Among them, each upsampling block performs spatial and channel collaborative attention mechanism processing and bilinear interpolation operation on the output of a convolution block of the feature extraction module from bottom to top; the bilinear interpolation operation result is spliced ​​with the output of the previous convolution block of the corresponding convolution block; and a double convolution operation is performed on the splicing operation result to obtain the sampling result of the corresponding upsampling block.

[0022] Furthermore, the spatial and channel collaborative attention mechanism is composed of a series connection of shareable multi-semantic spatial attention SMSA and progressive channel self-attention PCSA.

[0023] Furthermore, the shared multi-semantic spatial attention decomposes the input feature X along the height and width dimensions, and applies global average pooling to the two dimensions respectively to obtain the corresponding output feature set;

[0024] The output feature set is divided into Z independent sub-features of the same size; convolution kernels of different sizes are used to perform convolution operations on the sub-features, and lightweight shared convolution is used to align, concatenate and normalize the convolution operation results corresponding to the two dimensions to obtain semantic space information of the two dimensions;

[0025] The semantic space information of the two dimensions is activated by Sigmoid function to obtain the output feature X that can share multi-semantic space attention s .

[0026] Furthermore, the progressive channel self-attention is applied to the output feature X of the sharable multi-semantic spatial attention. s Perform pooling and normalization to obtain the pooled and normalized feature map X p ;

[0027] Use depthwise separable convolution on the X p Processing is performed and queries, keys and values ​​are generated. The channel multi-head self-attention mechanism is adopted. By performing dot product operations and normalization on multiple queries and keys, the weights are obtained and then multiplied by the values ​​to obtain the self-attention output.

[0028] Furthermore, the step S4 specifically includes:

[0029] S41, using a threshold method to remove pixel points whose density values ​​in the distribution density map are lower than a specified threshold, and setting points whose density values ​​are higher than the threshold to non-zero values, thereby obtaining a tumor cell region;

[0030] S42, applying distance transformation to each channel, using the obtained distance to represent the tumor cell area and generate a grayscale image;

[0031] S43, performing inversion processing on the overlapping tumor cell regions, and then filling the local minimum region with a watershed algorithm, identifying the local maximum, and separating the overlapping tumor cell regions;

[0032] S44, calculating the center position of each tumor cell region, calculating the Ki-67 index according to the center position and inverting the cell position.

[0033] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a breast immunohistochemistry Ki-67 image segmentation method based on an attention mechanism, and constructs a breast cell segmentation model based on a VGG16 network and a spatial and channel collaborative attention mechanism; the VGG16 network gradually extracts low-level to high-level features in image blocks through hierarchical convolution and pooling operations; the spatial and channel collaborative attention mechanism coordinates the relationship between global semantic information and local detail features by weighting features in both spatial and channel dimensions, thereby effectively alleviating the differences between different semantic features and improving the adaptability and generalization ability of the model.

[0034] The spatial and channel collaborative attention mechanism of the present invention can efficiently extract information from different semantic spaces by sharing multi-semantic spatial attention, thereby enhancing the semantic understanding ability of the network at multiple levels; and calculate the self-attention relationship between channels through progressive channel self-attention; accurately adjust the influence of each channel, enhance the interdependence between channels, and ultimately improve the overall feature representation capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0036] Figure 1 A flow chart of a breast immunohistochemistry Ki-67 image segmentation method based on an attention mechanism is provided in an embodiment of the present invention.

[0037] Figure 2A schematic diagram of the segmentation and screening of full-field slice images of breast immunohistochemistry Ki-67 images provided in an embodiment of the present invention.

[0038] Figure 3 This is a breast cell segmentation model provided by an embodiment of the present invention.

[0039] Figure 4 A structural diagram of the spatial and channel collaborative attention mechanism provided for an embodiment of the present invention.

[0040] Figure 5 A diagram of a sharable multi-semantic spatial attention structure provided for an embodiment of the present invention.

[0041] Figure 6 A diagram of the progressive channel self-attention structure provided in an embodiment of the present invention.

[0042] Figure 7 A flowchart for predicting the center point of Ki-67 cells in breast immunohistochemistry provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The embodiment of the present invention discloses a breast immunohistochemistry Ki-67 image segmentation method based on an attention mechanism, referring to Figure 1 Said, comprising the following steps:

[0045] S1, obtaining a breast immunohistochemistry Ki-67 image, performing segmentation processing, and obtaining a breast immunohistochemistry Ki-67 image block; the image block has a corresponding cell type label;

[0046] S2. Using breast immunohistochemistry Ki-67 image blocks as input and corresponding cell type classification results as output, a pre-constructed breast cell segmentation model is trained; wherein the breast cell segmentation model is constructed based on a VGG16 network and a spatial and channel collaborative attention mechanism;

[0047] S3, dividing the breast immunohistochemistry Ki-67 image to be predicted into multiple image blocks, and inputting the image blocks into the trained breast cell segmentation model for segmentation; obtaining a distribution density map of the classification results of different types of cells;

[0048] S4. Apply threshold method, distance transformation and watershed algorithm to the distribution density map for post-processing to obtain segmented cell regions; calculate the Ki-67 index and reverse display the cell positions.

[0049] The embodiment of the present invention constructs a breast cell segmentation model based on the VGG16 network and the spatial and channel collaborative attention mechanism; in this embodiment, a female patient is diagnosed with breast cancer, and the Ki-67 expression level in the tumor tissue is detected by using immunohistochemistry technology to help determine the best treatment plan.

[0050] First, the breast immunohistochemistry Ki-67 image block data set is used to train and verify the breast cell segmentation model; then the trained breast cell segmentation model is used to predict and segment the breast immunohistochemistry Ki-67 image block to be tested, and the density map of immune positive cells, immune negative cells and lymphocytes is obtained; the density map is subjected to post-processing operations such as threshold method, distance transformation, and watershed algorithm to obtain the tumor cell area; in this embodiment, the cell area of ​​immune positive cells and immune negative cells is used to determine the tumor cell area; finally, the center point coordinates of each tumor cell are calculated by the tumor cell area, and the Ki-67 index and the anti-display cell position are calculated. The present invention coordinates the relationship between space and channel features, enhances the feature guidance capability, effectively alleviates the semantic feature differences, and significantly improves the accuracy, flexibility and adaptability of tumor cell segmentation.

[0051] The following are detailed descriptions of each of the above steps:

[0052] Step 1: Slice the breast immunohistochemistry Ki-67 full-field slice image to obtain breast immunohistochemistry Ki-67 image blocks;

[0053] Step 2: annotating the obtained breast immunohistochemistry Ki-67 image blocks to obtain a breast immunohistochemistry Ki-67 image block data set;

[0054] Step 3: Construct a breast cell segmentation model based on spatial and channel collaborative attention mechanism;

[0055] Step 4: Use the breast immunohistochemistry Ki-67 image patch dataset to train and validate the breast cell segmentation model;

[0056] Step 5: Use the trained breast cell segmentation model to segment the breast immunohistochemistry Ki-67 image to obtain the density map of immune-positive cells, immune-negative cells and lymphocytes.

[0057] The density map was subjected to a threshold method to generate a binary image, and a distance transform was performed to generate a grayscale image. The watershed algorithm was used to segment the cell region, and the coordinates of the minimum circumscribed circle center of each cell region were calculated. Finally, the Ki-67 index was calculated based on the coordinates and the cell position was reversed.

[0058] Among them, step one, refer to Figure 2 As shown, specifically including:

[0059] First, the breast immunohistochemistry Ki-67 whole slide image (WSI) was transcoded into PNG format for subsequent processing. Figure 2 A medium to large circular area; Figure 2 The horizontal axis represents the pixel position of the image in the horizontal direction; the value increases from left to right. The vertical axis represents the pixel position of the image in the vertical direction; the value increases from top to bottom.

[0060] Then, the slice image was divided into 256×256 pixel blocks using the sliding window method to ensure that each image block contained sufficient detail information for cell segmentation; see Figure 2 The small square in the.

[0061] During the segmentation process, by calculating the ratio of the number of pixels in the cell area to the total pixels in each image block, image blocks with too high a background area ratio are screened out and removed. This screening process effectively removes large areas of blank or sample-free background areas in the slices and retains image blocks containing valid cell information.

[0062] Finally, each qualified 256×256 image block was used as an independent sample for subsequent annotation and construction of the breast immunohistochemistry Ki-67 image dataset.

[0063] Step 2 specifically includes:

[0064] First, all the breast immunohistochemistry Ki-67 image blocks obtained in step 1 were annotated using the Pari annotation tool.

[0065] This embodiment uses a senior pathologist to mark the center point coordinates of different types of cells in the image block, specifically including immune-positive cells, immune-negative cells and lymphocytes; and saves the annotation results in the form of a JSON file. Each JSON file contains a cell type and its corresponding center point coordinates. Through the annotated JSON file, the center point coordinates of each cell are extracted. For each cell, a Gaussian distribution kernel is generated at its center position as a mask for the cell. The Gaussian distribution kernel is used to simulate the shape and size of the cell, and by blurring the cell area, accurate annotation is provided for subsequent cell segmentation tasks.

[0066] Finally, the generated mask is saved as a Numpy array for subsequent model training and validation. All breast immunohistochemistry Ki-67 image blocks and their corresponding masks are divided into training set, validation set and test set in a ratio of 7:2:1.

[0067] Step 3, refer to Figure 3 As shown, specifically, the constructed breast cell segmentation model includes a feature extraction module and a reconstruction module.

[0068] in,

[0069] The feature extraction module uses the VGG16 network to extract features from breast immunohistochemistry Ki-67 image blocks. VGG16 consists of five convolutional blocks. Figure 3 Conv means convolution layer, Maxpooling means maximum pooling layer; each block contains multiple convolution layers and pooling layers. The convolution kernel size of each convolution layer is 3×3. The first layer includes two convolution layers, the second layer includes one maximum pooling layer and two convolution layers, and the third, fourth and fifth layers each have one pooling layer and three convolution layers. Through these hierarchical convolution and pooling operations, VGG16 can gradually extract low-level to high-level features in image blocks. The convolution layer is responsible for extracting local features such as edges and textures in image blocks, while the pooling layer helps reduce the size of the feature map and enhance the abstraction ability of the features. Figure 3 As shown in the figure, the input image block size of the first convolution block is 3×256×256, where 3 represents the number of channels and the two 256s represent the image height and width pixel values ​​respectively.

[0070] The feature extraction process is described by the formula:

[0071]

[0072]

[0073] Among them, B1(‧) represents the first convolutional block, B i (‧) represents the i-th convolutional block, I0 represents the input image block, F i represents the output features of the i-th convolutional block, m represents the number of convolutional blocks, and in this example, m=5.

[0074] The reconstruction module consists of four spatial and channel co-attention upsampling blocks. Each upsampling block first processes the input feature map through the spatial and channel co-attention mechanism to enhance the expression of important features. Then, the size of the feature map is enlarged by bilinear interpolation to improve its spatial resolution. Subsequently, a jump connection is made with the output of the encoder convolution block to fuse the low-level detail features with the upper-level semantic information to preserve the detail information in the image block. Finally, through two consecutive convolution operations, the number of channels of the spliced ​​feature map is reduced to effectively restore the spatial structure of the image.

[0075] The reconstruction process can be described as:

[0076] The first upsampling block performs spatial and channel collaborative attention mechanism processing and bilinear interpolation operation on the output of the fifth convolution block; the bilinear interpolation operation result is spliced ​​with the output of the fourth convolution block; and a double convolution operation is performed on the splicing operation result to obtain a first sampling result.

[0077] The second upsampling block performs spatial and channel collaborative attention mechanism processing and bilinear interpolation operation on the output of the fourth convolution block; the bilinear interpolation operation result is spliced ​​with the output of the third convolution block; and a double convolution operation is performed on the splicing operation result to obtain a second sampling result.

[0078] The third upsampling block performs spatial and channel collaborative attention mechanism processing and bilinear interpolation operation on the output of the third convolution block; the bilinear interpolation operation result is spliced ​​with the output of the second convolution block; and a double convolution operation is performed on the splicing operation result to obtain a third sampling result.

[0079] The fourth upsampling block performs spatial and channel collaborative attention mechanism processing and bilinear interpolation operation on the output of the second convolution block; the bilinear interpolation operation result is spliced ​​with the output of the first convolution block; and a double convolution operation is performed on the splicing operation result to obtain a fourth sampling result.

[0080] The formula is:

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089] Among them, SCSA(‧) represents the spatial and channel coordinated attention mechanism, Bilinear(‧) represents the bilinear interpolation operation, Concat(‧) represents the concatenation operation, DoubleConv(‧) represents the double convolution operation, F4 and F5 are the outputs of the feature extraction module, represents the output features of the first spatial and channel-wise joint attention, represents the output feature of the first bilinear interpolation operation, represents the output features of the first skip connection operation, Represents the output features of the first spatial and channel co-attention upsampling block; represents the output features of the i-1th spatial and channel collaborative attention upsampling block, represents the output features of the i-th spatial and channel co-attention upsampling block, represents the output feature of the i-th spatial and channel collaborative attention, represents the output feature of the i-th bilinear interpolation operation, is the output of the feature extraction module, represents the output feature of the i-th skip connection operation, n represents the number of spatial and channel co-attention upsampling blocks, in this example, n=4.

[0090] Due to the diverse morphological changes of Ki-67-labeled cells, including differences in size, shape, and color, the presentation of cells in the image is extremely complex. Secondly, the boundaries between cells are often blurred, and the image background may contain interference information, which further increases the difficulty of segmentation. In addition, the cell density, irregular morphology of cells, and overlap between cells in the image also make accurate segmentation difficult. In order to further improve the ability of the VGG16-UNet structure in detail recovery, the present invention introduces the spatial and channel collaborative attention mechanism (SCSA) into the VGG16-UNet network to enhance the network's ability to reconstruct detail features in the decoding stage and optimize the information flow between different semantic levels. Specifically, the spatial and channel collaborative attention mechanism (SCSA) coordinates the relationship between global semantic information and local detail features by weighting features in both spatial and channel dimensions, thereby effectively alleviating the differences between different semantic features and improving the adaptability and generalization of the model.

[0091] The following is a detailed introduction to the spatial and channel collaborative attention mechanism (SCSA) in the present invention. Figure 4As shown, it consists of the shared multi-semantic spatial attention (SMSA) and the progressive channel self-attention (PCSA) in series.

[0092] 1. Shared Multi-Semantic Spatial Attention (SMSA), see Figure 5 shown.

[0093] First, the input feature X is decomposed along the height (H) and width (W) dimensions, and global average pooling is applied to each dimension. Figure 5 In the above formula, X Avgpool and Y Avgpool represent the average pooling operations in the height and width dimensions, respectively, thus obtaining two unidirectional 1D sequence structures. These two sequences represent the global feature information of the image in the height and width dimensions, respectively, and provide compressed semantic representation for subsequent processing.

[0094] The process is described by the formula:

[0095]

[0096]

[0097] Among them, X represents the input feature, , Respectively represent the global average pooling operation in the height and width dimensions, X H , X W Respectively represent the output feature sets after global average pooling in the height (H) and width (W) dimensions.

[0098] Next, the output feature set is divided into Z independent sub-features of equal size, and the number of channels of each sub-feature is (The default is Z=4). This decomposition process can efficiently extract information from different semantic spaces, thereby enhancing the network's semantic understanding ability at multiple levels. Each sub-feature is independent, which helps the network focus on semantic information at different levels and improves the diversity and robustness of the model. By dividing the input features, the sub-features can be processed independently in their respective spatial dimensions, making the network more flexible in processing different semantic information.

[0099] The decomposition process is described by the formula:

[0100]

[0101]

[0102] in, , Represents the i-th sub-feature in the height (H) and width (W) dimensions, X H , X WThey represent the output feature set after global average pooling in the height (H) and width (W) dimensions, respectively. C represents the number of channels of the output feature set. Z represents the number of sub-features divided by the output feature set. Indicates the number of channels of the sub-feature. In this example, Z=4.

[0103] After the input features are divided, convolution operations are performed on the sub-features using convolution kernels of different sizes (3, 5, 7, and 9, respectively) to effectively capture the multi-scale spatial information in the image. In this way, the network can more efficiently extract spatial features of different scales in the image. In order to address the limitations of traditional convolution operations in terms of receptive field, lightweight shared convolution is used to align the height and width dimensions, implicitly modeling the dependency between the two dimensions. The process of extracting semantic spatial information is described as:

[0104]

[0105]

[0106] in, represents a lightweight shared convolution operation, Z i represents the convolution kernel size applied to the i-th sub-feature, represents the number of channels of the sub-features, Indicates that the number of channels of each sub-feature remains unchanged after the lightweight shared convolution operation. , Represents the i-th sub-feature in the height (H) and width (W) dimensions, respectively. , They represent the semantic spatial information of the i-th sub-feature in the height and width dimensions after the lightweight shared convolution operation, respectively. In this example, Z=4.

[0107] In order to accurately calculate the spatial attention map of each feature, sub-features with different semantics are aggregated and normalized using Group Normalization (GN). GN effectively avoids the interference of batch statistical noise, reduces the semantic differences between sub-features, and prevents attention dilution by group normalizing each sub-feature.

[0108] Finally, the spatial attention in the height and width dimensions is generated by a simple Sigmoid activation function, which activates or suppresses features in a specific spatial region. The output feature X that can share multiple semantic spatial attention S The calculation process is as follows:

[0109]

[0110]

[0111]

[0112] in represents the Sigmoid activation function, and and denotes a group normalization operation with Z groups in the height (H) and width (W) dimensions, respectively, Represents a splicing operation, Represents the semantic spatial information of the first sub-feature of the height dimension after the lightweight shared convolution operation. Represents the semantic spatial information of the second sub-feature of the height dimension after the lightweight shared convolution operation. Represents the semantic spatial information of the Zth sub-feature of the height dimension after the lightweight shared convolution operation, is the spatial attention output applied to the height dimension, Represents the semantic spatial information of the first sub-feature of the width dimension after the lightweight shared convolution operation. Represents the semantic spatial information of the second sub-feature of the width dimension after the lightweight shared convolution operation. Represents the semantic spatial information of the Zth sub-feature in the width dimension after the lightweight shared convolution operation. is the spatial attention output applied to the width dimension, X is the input feature that can share multiple semantic spatial attention, and X S Represents the output features that can share multi-semantic spatial attention. In this example, Z=4.

[0113] 2. Progressive Channel Self-Attention (PCSA), see Figure 6 shown.

[0114] First, the spatial attention X that can share multiple semantic spatial attention outputs S Through a 7×7 size Avgpool pooling operation, the main purpose of the pooling process is to reduce the resolution of the feature map and reduce the computational burden, while retaining enough spatial information to ensure the efficiency of subsequent operations. After the pooling operation, the feature map is further processed by group normalization (GN). GN normalizes all channels as a group, which helps stabilize the training process, avoid gradient explosion or disappearance, and improve the convergence speed of the network.

[0115] The process is described by the formula:

[0116]

[0117] Among them, X S represents the output features that can share multi-semantic spatial attention, Represents a pooling operation with a convolution kernel size of 7×7, which reduces the size of the feature map from Adjust to , Indicates that all channels are normalized as a group, X P Represents the feature map after pooling and normalization.

[0118] Next, the pooled and normalized feature map is processed using a depthwise separable convolution (DWConv) to generate query (Q), key (K), and value (V). These representations will be used to calculate the self-attention relationship between channels. Through the convolution operation, the input feature map is mapped into three different representations of query (Q), key (K), and value (V), which correspond to the similarity measure between channels.

[0119] The process is described by the formula:

[0120]

[0121]

[0122] in, represents the mapping function, represents a depth-wise separable convolution with a kernel size of 1 and a constant number of channels, X P represents the pooled and normalized feature map, , , Respectively represent the application of mapping functions to generate queries (Q), keys (K), and values ​​(V).

[0123] The self-attention calculation uses the channel multi-head self-attention (MHSA) mechanism to evaluate the correlation between channels by performing dot product operations on multiple queries (Q) and keys (K). Specifically, the dot product result of each query (Q) and key (K) is normalized by Softmax, and the weight is obtained and then multiplied by the value (V) to finally obtain the self-attention output. The advantage of multi-head self-attention is that it can capture the channel dependencies of different subspaces, thereby enhancing the model's ability to model the interaction between channels.

[0124] The process is described by the formula:

[0125]

[0126] in, represents the self-attention output, represents the multi-head self-attention mechanism in the channel dimension, represents the normalization function, Represents the transpose of the key (K).

[0127] Finally, the calculated self-attention output will be combined with the output feature X that can share multi-semantic spatial attention S Through further pooling operations, the self-attention output is adjusted to a (1,1) size, and then nonlinearly transformed through the Sigmoid activation function. Next, the adjusted self-attention output is element-by-element weighted multiplication with the original feature map. In this way, the model can accurately adjust the influence of each channel, enhance the interdependence between channels, and ultimately improve the overall feature representation capability.

[0128] The calculation process of the output features is as follows:

[0129]

[0130] in, represents the self-attention output, X S represents the shared multi-semantic spatial attention output, X C represents the progressive channel self-attention output, represents the Sigmoid activation function, Indicates that the convolution kernel size is The pooling operation of self-attention output size Adjust to .

[0131] According to step 4, the breast immunohistochemistry Ki-67 image dataset was used to train and verify the breast cell segmentation model; the mean square error (MSE) loss function was used in the training.

[0132] Mean Squared Error (MSE) is a commonly used loss function used in the image segmentation task of this embodiment to measure the difference between the model prediction value and the true segmentation label. It evaluates the accuracy of the segmentation result by calculating the square of the error between the predicted value and the true value and averaging all pixels. The smaller the MSE value, the closer the segmentation result of the model is to the true label; conversely, the larger the MSE value, the larger the segmentation error and the poorer the segmentation effect.

[0133] The loss calculation formula is expressed as:

[0134]

[0135] Among them, num is the total number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample.

[0136] In terms of optimizing model parameters, the Adam optimizer is used. The Adam optimizer combines the characteristics of momentum and adaptive learning rate adjustment. It can adaptively adjust the learning rate of each parameter during training, so that the model can converge faster and more stably.

[0137] According to step five, the trained breast cell segmentation model is used to segment the breast immunohistochemistry Ki-67 image block to be tested, and the density map of immune-positive cells, immune-negative cells and lymphocytes is obtained.

[0138] In order to solve the ambiguity and false positive problems in the density map and optimize the location of the cell center point, this embodiment introduces a post-processing stage. Figure 7 As shown in the figure, the generated cell density map is processed to remove pixels with density values ​​below the specified threshold, and the points with density values ​​above the threshold are set to 255. Subsequently, the distance transform is applied to each channel to generate a grayscale image, in which the pixel value in the continuous area represents the distance from the pixel to the boundary of the area. The distance transform result ensures that there is only one local maximum point in each area, but in the case of non-circular areas or overlapping cells, there may still be multiple local maximum points.

[0139] To further segment the complex overlapping areas, the watershed algorithm is used. The watershed algorithm is based on the principle of topology, which treats the image as a terrain and divides the area by filling the "valley bottom". Before applying the algorithm, the input image is inverted to ensure that the algorithm can correctly identify the local maximum of the area for segmentation.

[0140] Finally, the center point of each cell was determined by calculating the center coordinates of the minimum circumscribed circle of the connected area. These center points not only accurately marked the location of the cells, but also provided key data support for the subsequent calculation of the immunohistochemical Ki-67 index, thus significantly improving the practicality and reliability of the method.

[0141] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0142] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A breast immunohistochemistry Ki-67 image segmentation method based on attention mechanism, characterized in that: The following steps are involved: S1, obtaining a breast immunohistochemistry Ki-67 image, performing segmentation processing, and obtaining a breast immunohistochemistry Ki-67 image block; the image block has a corresponding cell type label; S2. Using breast immunohistochemistry Ki-67 image blocks as input and corresponding cell type classification results as output, a pre-constructed breast cell segmentation model is trained; wherein the breast cell segmentation model is constructed based on a VGG16 network and a spatial and channel collaborative attention mechanism; S3, dividing the breast immunohistochemistry Ki-67 image to be predicted into multiple image blocks, and inputting the image blocks into the trained breast cell segmentation model for segmentation; obtaining a distribution density map of the classification results of different types of cells; S4. Apply threshold method, distance transformation and watershed algorithm to the distribution density map for post-processing to obtain segmented cell regions; calculate the Ki-67 index and reverse display the cell positions.

2. The method for breast immunohistochemistry Ki-67 image segmentation based on attention mechanism as claimed in claim 1, characterized in that: The cell types described in step S1 include immune-positive cells, immune-negative cells and lymphocytes, and the marked positions are at the coordinate positions of the center points of the cells.

3. The method for breast immunohistochemistry Ki-67 image segmentation based on attention mechanism as claimed in claim 2, characterized in that: In step S2, the breast cell segmentation model includes a feature extraction module and a reconstruction module; The feature extraction module uses the VGG16 network to extract features from the breast immunohistochemistry Ki-67 image block to obtain a preliminary feature map; The reconstruction module performs weighted processing on the preliminary feature map in spatial and channel dimensions through a spatial and channel collaborative attention mechanism to enhance the expression of overall features.

4. The method for breast immunohistochemistry Ki-67 image segmentation based on attention mechanism as claimed in claim 3, characterized in that: The feature extraction module includes five convolution blocks, each of which includes multiple convolution layers and pooling layers; Among them, the first convolution block includes two convolution layers, which perform convolution operations on the breast immunohistochemistry Ki-67 image block; the second convolution block includes a maximum pooling layer and two convolution layers; the third convolution block, the fourth convolution block and the fifth convolution block include a pooling layer and three convolution layers respectively; after sequential convolution, the fifth convolution block outputs a preliminary feature map.

5. The method for breast immunohistochemistry Ki-67 image segmentation based on attention mechanism as claimed in claim 3, characterized in that: The reconstruction module, comprises four spatial and channel co-attention upsampling blocks; Among them, each upsampling block performs spatial and channel collaborative attention mechanism processing and bilinear interpolation operation on the output of a convolution block of the feature extraction module from bottom to top; the bilinear interpolation operation result is spliced ​​with the output of the previous convolution block of the corresponding convolution block; and a double convolution operation is performed on the splicing operation result to obtain the sampling result of the corresponding upsampling block.

6. The method for breast immunohistochemistry Ki-67 image segmentation based on attention mechanism as claimed in claim 5, characterized in that: The proposed spatial and channel-wise collaborative attention mechanism consists of a cascade of shareable multi-semantic spatial attention and progressive channel-wise self-attention.

7. The method for breast immunohistochemistry Ki-67 image segmentation based on attention mechanism as claimed in claim 6, characterized in that: The shared multi-semantic spatial attention decomposes the input feature X along the height and width dimensions, and applies global average pooling to the two dimensions respectively to obtain the corresponding output feature set; The output feature set is divided into Z independent sub-features of the same size; convolution kernels of different sizes are used to perform convolution operations on the sub-features, and lightweight shared convolution is used to align, concatenate and normalize the convolution operation results corresponding to the two dimensions to obtain semantic space information of the two dimensions; The semantic space information of the two dimensions is activated by Sigmoid function to obtain the output feature X that can share multi-semantic space attention s .

8. The method for breast immunohistochemistry Ki-67 image segmentation based on attention mechanism as claimed in claim 6, characterized in that: The progressive channel self-attention, the output feature X of the shared multi-semantic spatial attention s Perform pooling and normalization to obtain the pooled and normalized feature map X p ; Use depthwise separable convolution on the X p Processing is performed and queries, keys and values ​​are generated. The channel multi-head self-attention mechanism is adopted. By performing dot product operations and normalization on multiple queries and keys, the weights are obtained and then multiplied by the values ​​to obtain the self-attention output.

9. The method for breast immunohistochemistry Ki-67 image segmentation based on attention mechanism as claimed in claim 1, characterized in that: The step S4 specifically includes: S41, using a threshold method to remove pixel points in the distribution density map whose density values ​​are lower than a specified threshold, and setting points whose density values ​​are higher than the threshold to non-zero values, thereby obtaining a tumor cell region; S42, applying distance transformation to each channel, using the obtained distance to represent the tumor cell area and generate a grayscale image; S43, performing inversion processing on the overlapping tumor cell regions, and then filling the local minimum region with a watershed algorithm, identifying the local maximum, and separating the overlapping tumor cell regions; S44, calculating the center position of each tumor cell region, calculating the Ki-67 index according to the center position and inverting the cell position.

Images