Multi-histological pathological cell nucleus instance segmentation and classification method and system based on key point guided Transform

Abstract

The invention relates to the technical field of image segmentation, in particular to a multi-histological pathological cell nucleus instance segmentation and classification method and system based on a key point guided Transform, and the method comprises the steps: inputting a preprocessed pathological image into a PointFormer, and generating three output predictions, namely a key point heat map, a nucleus prediction and an edge prediction; positioning a central point of a cell nucleus by filtering local peak values in the key point heat map, and generating a central point map; integrating the kernel prediction and the edge prediction to obtain a class-independent segmentation prediction mask; and mapping the category of the corresponding central point in the central point diagram to a category-independent segmentation prediction mask, and generating a final cell nucleus instance segmentation classification mask. According to the method, detection and classification of each kernel are used as a semantic key point estimation problem to determine the center point of each kernel, and benefits can be obtained from category-perceived detection and category-independent segmentation by decoupling a segmentation classification task and utilizing the advantage that Transform has modeling long-distance dependence, so that the segmentation classification performance of aggregated and overlapped kernels is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a multi-histological pathological cell nucleus instance segmentation classification method and system based on key point guided Transformer. Background Art

[0002] Cancer poses a serious threat to global public health, with its mortality rate second only to cardiovascular disease. Biopsy-based histopathological examination is the gold standard for cancer diagnosis in current clinical practice. Analysis of the morphological characteristics and spatial distribution patterns of cell nuclei can assist pathologists in formulating a comprehensive treatment plan and accurately predict the patient's prognosis. However, comprehensive cellular level analysis not only requires a lot of time investment, but also has significant differences between and within observers. With the emergence of digital whole-slide image (WSI), computer-aided technology has shown great potential in automatic cell nucleus analysis, among which convolutional neural networks (CNNs) dominate due to their superior generalization and simplicity. Most methods use multi-task learning to simultaneously learn cell nucleus and edge information through multiple decoders to improve segmentation performance.

[0003] Existing CNN-based tissue nucleus segmentation and classification (NSC) methods can be divided into two categories: bottom-up and top-down methods. Bottom-up methods first generate high-resolution semantic segmentation masks and then group pixels into any number of target instances. Due to the reliance on complex pixel grouping post-processing techniques to extract target instances, their performance is highly dependent on the segmentation results and grouping methods. Although some studies directly predict the nucleus contours and show good results, their advantages are limited due to the neglect of the correlation between learning branches and prediction consistency. In contrast, top-down methods (such as Mask-RCNN) first locate class-independent target objects in a priori bounding boxes, generate candidate regions, and then segment and classify target instances within the region of interest. Although Mask-RCNN can well separate overlapping nuclei, it can only segment instances within the bounding box at a very low resolution (i.e., 28×28). If the bounding box prediction is inaccurate and smaller than the actual instance, it will lead to poor segmentation results. In addition, the convolution operation inherent to CNNs cannot effectively capture global spatial information or the correlation between nuclei. Clinical practice shows that there are long-range correlations in actual cell nucleus segmentation tasks, which provides a key basis for domain experts to accurately segment cell nuclei. Summary of the invention

[0004] The purpose of the present invention is to solve the above-mentioned problems existing in the prior art, and propose a multi-histological pathological cell nucleus instance segmentation and classification method and system based on a keypoint-guided Transformer (PointFormer for short), which regards the detection and classification of each nucleus as a semantic keypoint estimation problem to determine the center point of each nucleus. By decoupling the segmentation and classification tasks and utilizing the advantage of Transformer in modeling long-distance dependencies, it can benefit from category-aware detection and category-independent segmentation, thereby improving the segmentation and classification performance of clustered overlapping nuclei.

[0005] To achieve the above purpose, the technical solution adopted is:

[0006] A multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer, comprising the following steps:

[0007] Step 1: Preprocess the pathological image;

[0008] Step 2: Input the preprocessed pathological image to PointFormer to generate three output predictions, namely, key point heat map, core prediction, and edge prediction;

[0009] Step 3: locate the center point of the cell nucleus by filtering the local peaks in the key point heat map and generate a center point map;

[0010] Step 4: Integrate the kernel prediction and edge prediction to obtain a class-independent segmentation prediction mask;

[0011] Step 5: Map the categories of the corresponding center points in the center point map to the category-independent segmentation prediction mask to generate the final cell nucleus instance segmentation classification mask.

[0012] According to the multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer of the present invention, further, in the step 1, the preprocessing of the pathological image is to perform a closing process on the real edge label.

[0013] According to the multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer of the present invention, further, the PointFormer is composed of an encoder and three decoders, and an attention filtering enhancement module is embedded between the decoders.

[0014] According to the multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer of the present invention, further, the core component of the encoder is a multi-local perception Transformer module MLPFormer composed of two Transformer modules, and the two Transformer modules are respectively a local feature extraction module composed of local window attention LWA and depthwise separable convolution in parallel, and a global feature extraction module composed of local vertical attention LVA and local horizontal attention LHA in parallel; the specific calculation process is:

[0015] a. Given an input feature map X, divide the feature map X into multiple local regions X = [X 1 ,X 2 ,…,X n ], where n represents the number of local regions;

[0016] b. Calculate the multi-head attention within the segmented local area and obtain the output Z of the i-th local area i ;

[0017] c. Concatenate the outputs of n local regions to obtain the local attention output LA(·);

[0018] LA(X)=Concat(Z 1 ,Z 2 ,…,Z n )

[0019] d. The calculation formula of the entire MLPFormer is as follows:

[0020]

[0021]

[0022] in, represents the input of LWA, represents the input of DWConv, represents the input of LVA, represents the input of LHA, Z l and Z l+1 They represent the outputs of the two Transformer modules in MLPFormer respectively. LWA(·), LVA(·) and LHA(·) are calculated in the same way as LA(·); LN(·) represents LayerNorm normalization, and FNN(·) represents a feedforward neural network.

[0023] According to the multi-histological pathological cell nucleus instance segmentation and classification method based on the key point guided Transformer of the present invention, further, the core component of the nuclear branch decoder and the edge branch decoder is the cross MLPFormer. Except for the query vector Q, the calculation method is consistent with the MLPFormer; in the cross MLPFormer calculation, the query vector Q is the Refined_Q after the current branch Q and the Cross_Q from the key point estimation branch are refined by the AFE module.

[0024] According to the multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer of the present invention, further, the core of the AFE module includes channel attention and spatial attention, which are formally expressed as:

[0025]

[0026] Among them, Q comes from the core prediction branch or the edge prediction branch, Cross_Q comes from the key point estimation branch, Att c and Att sp They represent channel attention and spatial attention respectively.

[0027] According to the multi-histological pathological cell nucleus instance segmentation classification method based on key point guided Transformer of the present invention, further, the core component of the key point branch decoder is consistent with the MLPFormer of the encoder; the goal of the key point branch decoder is to regress the low resolution key point heat map The training process of the key point branch decoder is:

[0028] a. Given an input image X with height H and width W and the corresponding ground truth heat map Y, for each ground truth key point p of class c, calculate a low-resolution equivalent

[0029] b. Using an unnormalized Gaussian kernel, all ground truth key points are rendered in the heat map Y. The real key point heat map is shown in the following formula; where σ p represents the standard deviation of an adaptive size of a target, x and y represent the horizontal and vertical coordinates of the real key points, respectively. and They represent the low-resolution equivalent of the key point x coordinate and the low-resolution equivalent of the key point y coordinate, respectively. xyc Represents the true key point heat map; if two Gaussian distributions of the same category overlap, take the element-wise maximum value;

[0030]

[0031] c. Key point estimation branch loss function L pt The definition is as follows:

[0032]

[0033] in, represents the predicted key point heat map, α and β are the hyper parameters of focal loss, N k Represents the number of key points of image X.

[0034] According to the multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer of the present invention, further, the step 3 specifically includes: detecting all key points with a neighborhood response value greater than or equal to 8, and filtering out low confidence predictions using a confidence threshold of 0.4.

[0035] According to the multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer of the present invention, further, the step 4 specifically comprises: removing noise from the predicted cell nucleus mask; performing boundary refinement on the predicted edge mask; aggregating the results of the nucleus prediction branch and the edge prediction branch, and constructing a category-independent segmentation prediction mask by performing an expansion operation;

[0036] The step 5 specifically includes: for each cell nucleus in the category-independent segmentation prediction mask, calculating the distance between the centroid of the cell nucleus and the center point of the corresponding area in the center point map, and taking the type represented by the nearest center point as the category of the cell nucleus.

[0037] Furthermore, the present invention also provides a multi-histological pathological cell nucleus instance segmentation and classification system based on key point-guided Transformer, comprising:

[0038] A preprocessing module, used for preprocessing pathological images;

[0039] PointFormer module, used to input the preprocessed pathological image into PointFormer and generate three output predictions, namely key point heat map, core prediction and edge prediction;

[0040] A center point map generation module is used to locate the center point of the cell nucleus by filtering the local peak value in the key point heat map to generate a center point map;

[0041] An integration module for integrating kernel prediction and edge prediction to obtain a class-independent segmentation prediction mask;

[0042] The segmentation classification mask generation module is used to map the categories of the corresponding center points in the center point map to the category-independent segmentation prediction mask to generate the final cell nucleus instance segmentation classification mask.

[0043] The beneficial effects achieved by adopting the above technical solution are:

[0044] 1. In view of the problem that the bottom-up method is sensitive to pixel grouping technology, the present invention regards the detection and classification of each nucleus as a semantic key point estimation problem to determine the center point of each nucleus, thereby efficiently detecting and classifying cell nuclei without complex pixel grouping post-processing.

[0045] 2. In view of the problem that the multi-task learning method ignores the correlation and prediction consistency between branches, the present invention proposes an attention guidance strategy based on Attention Filtering Enhancement (AFE) to share the features of key point estimation, thereby guiding the cell nucleus and edge segmentation and constraining the prediction consistency between different branches.

[0046] 3. To address the problem that CNNs cannot effectively capture the long-range correlations between cell nuclei, the present invention introduces a multi-local perception Transformer module (Multi-Local Perception Transformer, MLPFormer for short), which improves the model's global dependency modeling capability by parallel calculation of local horizontal and vertical attention, while maintaining a balance between high-resolution features and global dependencies.

[0047] The present invention decouples the segmentation and classification tasks and utilizes the advantage of Transformer in modeling long-distance dependencies, so as to benefit from category-aware detection and category-independent segmentation, thereby improving the segmentation and classification performance of clustered overlapping kernels. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings of the embodiments of the present invention, wherein the drawings are only used to illustrate some embodiments of the present invention, but not to limit all embodiments of the present invention thereto.

[0049] Figure 1 1 is a flow chart of a method for segmenting and classifying multiple histological pathological cell nuclei based on a key point-guided Transformer according to an embodiment of the present invention;

[0050] Figure 2 is a framework diagram of a method for segmenting and classifying multiple histological pathological cell nuclei instances based on key point-guided Transformer according to an embodiment of the present invention;

[0051] Figure 3 is a schematic structural diagram of an AFE module according to an embodiment of the present invention;

[0052] Figure 4This is a visualization diagram of the results of the multi-histological pathological cell nucleus instance segmentation and classification method based on key point-guided Transformer in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings of specific embodiments of the present invention to clearly and completely describe the exemplary scheme of the embodiment of the present invention. Unless otherwise defined, the technical terms or scientific terms used in the present invention should be the common meanings understood by people with ordinary skills in the field.

[0054] like Figure 1 As shown, this embodiment discloses a multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer, comprising the following steps:

[0055] Step S101: pre-process the pathological image.

[0056] like Figure 2 As shown in (a), data preprocessing is performed on the PanNuke dataset. PanNuke is currently the largest public dataset for cell nucleus segmentation and classification, covering 19 different tissue types and 5 types of cell nuclei: tumor, inflammatory, connective tissue, dead cells, and non-tumor epithelial cells. The main focus is on closing the true edge labels in PanNuke.

[0057] Step S102: input the preprocessed pathological image to PointFormer to generate three output predictions, namely, key point heat map, core prediction and edge prediction.

[0058] Input the preprocessed pathological image and use PointFormer to generate key point heat map, core prediction and edge prediction, such as Figure 2 (b) PointFormer consists of an encoder and three decoders, and an attention filter enhancement module AFE is embedded between the decoders to effectively pass the guidance signal from the key point estimation branch to the core prediction and edge prediction branches, thereby alleviating the inconsistency in multi-decoder prediction.

[0059] The core component of the encoder is the multi-local perception Transformer module MLPFormer, which consists of two Transformer modules. The two Transformer modules are a local feature extraction module composed of local window attention (LocalWindowAttention, LWA) and depthwise separable convolution in parallel, and a global feature extraction module composed of local vertical attention (LocalVerticalAttention, LVA) and local horizontal attention (LocalHorizontalAttention, LHA) in parallel. The specific calculation process is:

[0060] Step S1021: Given an input feature map Through two predefined parameters l w ,l h Divide the feature map X into multiple local regions X = [X 1 ,X 2 ,…,X n ], where n = (HW) / (l w ·l h ) represents the number of local regions.

[0061] Step S1022: Calculate the multi-head attention in the segmented local area. The number of heads is n_head, and the calculation formula is:

[0062]

[0063] in, represents the size of each local area, i=1,2,…,n; h represents the number of multi-head attention heads, h=1,2,…,n_head; The projection matrices of the query vector, key vector, and value vector of the h-th head, d k =C / h, represents the output after the hth head calculates the attention in the i-th local area, Z i represents the output of the i-th local area, and W° represents the output projection matrix.

[0064] Step S1023: Concatenate the outputs of n local regions to obtain the local attention output LA(·).

[0065] LA(X)=Concat(Z 1 ,Z 2 ,…,Z n ) (4)

[0066] Step S1024, the calculation formula of the entire MLPFormer is as follows:

[0067]

[0068] in, represents the input of LWA, represents the input of DWConv, represents the input of LVA, represents the input of LHA, Z l and Z l+1 They represent the outputs of the two Transformer modules in MLPFormer respectively. LWA(·), LVA(·) and LHA(·) are calculated in the same way as LA(·); LN(·) represents LayerNorm normalization, and FNN(·) represents a feedforward neural network.

[0069] The core component of the core branch decoder and edge branch decoder is the cross-MLPFormer, which is calculated in the same way as the MLPFormer except for the query vector Q. In the cross-MLPFormer calculation, the query vector Q is the Refined_Q of the current branch Q and the Cross_Q from the key point estimation branch after being refined by the AFE module.

[0070] The core of the AFE module includes channel attention and spatial attention, such as Figure 3 As shown, the formal expression is:

[0071]

[0072]

[0073] Among them, Q comes from the core prediction branch or the edge prediction branch, Cross_Q comes from the key point estimation branch, Att c and Att sp Represent channel attention and spatial attention respectively; MaxPool and AvgPool represent maximum pooling and average pooling, f 7 ×7 represents a convolution operation with a convolution kernel of 7×7, σ represents a sigmoid function, and shared MLP represents a multilayer perceptron with hidden layers. The weights are

[0074] The core component of the keypoint branch decoder is the same as the MLPFormer of the encoder; the goal of the keypoint branch decoder is to regress the low-resolution keypoint heatmap The training process of the key point branch decoder is:

[0075] Step a: Given an input image X with a height of H and a width of W and the corresponding ground truth heat map Y, for each ground truth key point p∈R of class c2 , compute a low-resolution equivalent In this embodiment, c=5 represents the number of key point types, and R=4 represents the downsampling factor.

[0076] Step b: Use the unnormalized Gaussian kernel to render all ground truth key points in the heat map Y. The true key point heat map expression is shown in Equation 13; where σ p Represents the standard deviation of an object's adaptive size. If two Gaussian distributions of the same category overlap, the element-wise maximum value is taken. x and y represent the horizontal and vertical coordinates of the real key points, respectively. and They represent the low-resolution equivalent of the key point x coordinate and the low-resolution equivalent of the key point y coordinate, respectively. xyc Represents the true keypoint heatmap.

[0077]

[0078] Step c: Key point estimation branch loss function The definition is as follows, and the training target is penalty-reduced pixel-wise logistic regression with focal loss:

[0079]

[0080] in, represents the predicted key point heat map, α and β are the hyper parameters of focal loss, N k represents the number of key points of the image X. In this embodiment, α=2, β=4.

[0081] Step S103: locate the center point of the cell nucleus by filtering the local peak value in the key point heat map to generate a center point map.

[0082] Specifically, all keypoints with a response value greater than or equal to 8 neighborhoods are detected, and a confidence threshold of 0.4 is used to filter out low-confidence predictions.

[0083] Step S104: adopt post-processing techniques such as denoising and boundary refinement to integrate the kernel prediction and the edge prediction to obtain a segmentation prediction mask that is independent of the category.

[0084] Specifically, noise (i.e., nucleus regions in the background or background regions in the nucleus) is removed from the predicted nucleus mask; a boundary thinning algorithm is applied on the predicted edge mask to reduce the thickness to 1 pixel; the results of the nucleus prediction branch and the edge prediction branch are aggregated and a class-independent segmentation prediction mask is constructed by performing a dilation operation using a 3×3 kernel, as shown in Figure 2 (c) as shown.

[0085] Step S105 : Map the category of the corresponding center point in the center point map to a category-independent segmentation prediction mask to generate a final cell nucleus instance segmentation classification mask.

[0086] Specifically, for each cell nucleus in the category-independent segmentation prediction mask, the distance between the centroid of the cell nucleus and the center point of the corresponding area in the center point map is calculated, and the type represented by the closest center point is taken as the category of the cell nucleus, such as Figure 2 (c) as shown.

[0087] Furthermore, a multi-task learning paradigm is adopted to train the three decoder networks, aiming to improve the overall performance by utilizing additional supervisory signals from key points and edges, and the total optimization function is the weighted sum of the loss functions on the three branches. In addition to the key point estimation branch loss function mentioned in step S102 above In this embodiment, both the core prediction branch and the edge prediction branch use the weighted sum of binary cross entropy (BCE) and Dice loss, where the BCE weight is 0.4 and the Dice weight is 0.6. The total weight coefficients of the three branches are all 1. The training uses the Adam optimizer, with a weight decay of 0.0001, an initial learning rate of 0.001, and a batch size of 4.

[0088] Corresponding to the above method, this embodiment also proposes a multi-histological pathological cell nucleus instance segmentation and classification system based on key point-guided Transformer, comprising:

[0089] The preprocessing module is used to preprocess the pathological images.

[0090] The PointFormer module is used to input the preprocessed pathological image into PointFormer and generate three output predictions, namely key point heat map, core prediction and edge prediction.

[0091] The center point map generation module is used to locate the center point of the cell nucleus by filtering the local peak value in the key point heat map to generate the center point map.

[0092] An integration module is used to integrate the kernel prediction and edge prediction to obtain a class-independent segmentation prediction mask.

[0093] The segmentation classification mask generation module is used to map the categories of the corresponding center points in the center point map to the category-independent segmentation prediction mask to generate the final cell nucleus instance segmentation classification mask.

[0094] To verify the effect of the present invention, training and evaluation were performed on the PanNuke dataset, and mPQ and bPQ were used as evaluation indicators. Compared with the classic cell nucleus instance segmentation classification method HoverNet, the mPQ and bPQ of the present invention were improved by 4.66% and 4.61% respectively; compared with the benchmark SOTA cell segmentation classification method PointNu-Net, the mPQ and bPQ of the present invention were improved by 1.40% and 2.49% respectively, which shows that the present invention can effectively improve the segmentation and classification performance of clustered overlapping nuclei. Figure 4 The visualization results on PanNuke are shown. PointNu-Net often misidentifies other tissue regions as cell nuclei, while HoVer-Net tends to segment a cell nucleus into multiple cell nuclei. In contrast, the present invention has edge perception capabilities and generates high-confidence instances, thus showing stronger robustness in actual clinical applications.

[0095] In summary, the present invention regards the detection and classification of each nucleus as a semantic key point estimation problem to determine the center point of each nucleus without complex pixel grouping post-processing, and proposes an attention guidance strategy based on AFE to share the features of key point estimation, thereby guiding the segmentation of cell nuclei and edges and alleviating the prediction inconsistency between different branches. By decoupling the segmentation and classification tasks and leveraging the advantage of Transformer in modeling long-distance dependencies, it can benefit from category-aware detection and category-independent segmentation, thereby improving the segmentation and classification performance of clustered overlapping nuclei.

[0096] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0097] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0098] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0099] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0100] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer, characterized in that: The following steps are involved: Step 1: Preprocess the pathological image; Step 2: Input the preprocessed pathological image to PointFormer to generate three output predictions, namely, key point heat map, core prediction, and edge prediction; Step 3: locate the center point of the cell nucleus by filtering the local peaks in the key point heat map and generate a center point map; Step 4: Integrate the kernel prediction and edge prediction to obtain a class-independent segmentation prediction mask; Step 5: Map the categories of the corresponding center points in the center point map to the category-independent segmentation prediction mask to generate the final cell nucleus instance segmentation classification mask.

2. The multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer according to claim 1 is characterized in that: In the step 1, the preprocessing of the pathological image is to perform a closing process on the real edge label.

3. The multi-histological pathological cell nucleus instance segmentation and classification method based on key point guided Transformer according to claim 1 is characterized in that: The PointFormer consists of an encoder and three decoders, and an attention filtering enhancement module is embedded between the decoders.

4. The method for segmenting and classifying multiple histological pathological cell nuclei based on key point-guided Transformer according to claim 3, characterized in that: The core component of the encoder is a multi-local perception Transformer module MLPFormer composed of two Transformer modules, the two Transformer modules are a local feature extraction module composed of local window attention LWA and depthwise separable convolution in parallel, and a global feature extraction module composed of local vertical attention LVA and local horizontal attention LHA in parallel; The specific calculation process is: a. Given an input feature map X, divide the feature map X into multiple local regions X = [X 1 ,X 2 ,…,X n ], where n represents the number of local regions; b. Calculate the multi-head attention within the segmented local area and obtain the output Z of the i-th local area i ; c. Concatenate the outputs of n local regions to obtain the local attention output LA(·); LA(X)=Concat(Z 1 ,Z 2 ,…,Z n ) d. The calculation formula of the entire MLPFormer is as follows: in, represents the input of LWA, represents the input of DWConv, represents the input of LVA, represents the input of LHA, Z l and Z l+1 They represent the outputs of the two Transformer modules in MLPFormer respectively. LWA(·), LVA(·) and LHA(·) are calculated in the same way as LA(·); LN(·) represents LayerNorm normalization, and FNN(·) represents a feedforward neural network.

5. The method for segmenting and classifying multiple histological pathological cell nuclei based on key point-guided Transformer according to claim 4, characterized in that: The core component of the core branch decoder and the edge branch decoder is the cross-MLPFormer. The calculation method is consistent with that of MLPFormer, except for the query vector Q. In the cross-MLPFormer calculation, the query vector Q is the Refined_Q of the current branch Q and the Cross_Q from the key point estimation branch after being refined by the AFE module.

6. The method for segmenting and classifying multiple histological pathological cell nuclei based on key point-guided Transformer according to claim 5, characterized in that: The core of the AFE module includes channel attention and spatial attention, which can be formally expressed as: Among them, Q comes from the core prediction branch or the edge prediction branch, Cross_Q comes from the key point estimation branch, Att c and Att sp They represent channel attention and spatial attention respectively.

7. The method for segmenting and classifying multiple histological pathological cell nuclei based on key point-guided Transformer according to claim 4, characterized in that: The core components of the keypoint branch decoder are consistent with the MLPFormer of the encoder; The goal of the keypoint branch decoder is to regress the low-resolution keypoint heatmap The training process of the key point branch decoder is: a. Given an input image X with height H and width W and the corresponding ground truth heat map Y, for each groundtruth key point p of class c, calculate a low-resolution equivalent b. Using an unnormalized Gaussian kernel, all ground truth key points are rendered in the heat map Y. The real key point heat map is shown in the following formula; where σ p represents the standard deviation of a target adaptive size, x and y represent the horizontal and vertical coordinates of the real key points, respectively. and They represent the low-resolution equivalent of the key point x coordinate and the low-resolution equivalent of the key point y coordinate, respectively. xyc Represents the true key point heat map; if two Gaussian distributions of the same category overlap, take the element-wise maximum value; c. Keypoint estimation branch loss function The definition is as follows: in, represents the predicted key point heat map, α and β are the hyper parameters of focal loss, N k Represents the number of key points of image X.

8. The method for segmenting and classifying multiple histological pathological cell nuclei based on key point-guided Transformer according to claim 1, characterized in that: The step 3 specifically includes: detecting all key points with a response value greater than or equal to 8 neighborhoods, and filtering out low-confidence predictions using a confidence threshold of 0.

4.

9. The method for segmenting and classifying multiple histological pathological cell nuclei based on key point-guided Transformer according to claim 1, characterized in that: The step 4 specifically includes: removing noise from the predicted cell nucleus mask; performing boundary refinement on the predicted edge mask; aggregating the results of the nucleus prediction branch and the edge prediction branch, and constructing a category-independent segmentation prediction mask by performing a dilation operation; The step 5 specifically includes: for each cell nucleus in the category-independent segmentation prediction mask, calculating the distance between the centroid of the cell nucleus and the center point of the corresponding area in the center point map, and taking the type represented by the nearest center point as the category of the cell nucleus.

10. A multi-histological pathological cell nucleus instance segmentation and classification system based on key point guided Transformer, characterized in that: Include: A preprocessing module, used for preprocessing pathological images; PointFormer module, used to input the preprocessed pathological image into PointFormer and generate three output predictions, namely key point heat map, core prediction and edge prediction; A center point map generation module is used to locate the center point of the cell nucleus by filtering the local peak value in the key point heat map to generate a center point map; An integration module for integrating kernel prediction and edge prediction to obtain a class-independent segmentation prediction mask; The segmentation classification mask generation module is used to map the categories of the corresponding center points in the center point map to the category-independent segmentation prediction mask to generate the final cell nucleus instance segmentation classification mask.

Images