Deep learning-based pathological image full-automatic segmentation and classification method and system

Abstract

The invention discloses a pathological image full-automatic segmentation and classification method and system based on deep learning, and the method comprises the steps: selecting a to-be-processed tissue slice image, extracting the feature information of a cell nucleus in the to-be-processed tissue slice image, and carrying out the initialization processing; the unified feature decoder is introduced to process the image, and different feature information is integrated; carrying out instance segmentation on the cell nucleus by adopting a convolutional neural network, and optimizing a feature extraction process in combination with a self-attention mechanism and a residual network; meanwhile, a classification module is used for carrying out category prediction on cell nucleuses, and tissue region guide information is fused; and carrying out comprehensive optimization on the segmentation result and the classification result through multi-task joint training until a preset stop condition is met, and obtaining a high-quality cell nucleus segmentation and classification result. The method can automatically and efficiently perform cell nucleus segmentation and classification in the digital pathological image, has high accuracy and stability, is suitable for various complicated tissue slice analysis tasks, and does not need manual intervention.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and specifically relates to a method and system for fully automatic segmentation and classification of pathological images based on deep learning. Background Art

[0002] The advent of whole-slide scanning imaging technology has enabled digital pathology to play a vital role in modern clinical practice. Nucleus segmentation and classification are fundamental tasks in digital pathology. As a key step in the analysis process, accurate nucleus segmentation results provide a reliable foundation for subsequent histological analysis and clinical prognosis prediction, helping to assess disease outcomes. Furthermore, nuclear feature extraction and analysis are widely used in survival prediction, pathological grading, and cancer diagnosis. Different types of nuclei may also have potential biological significance. For example, the tumor microenvironment can influence cancer progression by recruiting specific types of neural progenitor cells (NPCs).

[0003] However, manual nuclear analysis is often affected by inter- and intra-observer variability and is often limited by processing efficiency and classification accuracy, especially in the analysis of large-scale tissue sections. Therefore, automated segmentation methods, especially deep learning-based cell nucleus segmentation and classification techniques, have gradually become the focus of pathology image analysis research.

[0004] Despite some progress, nucleus segmentation remains a challenging task, especially in certain tissues, such as tumor tissue, where nuclei often cluster or overlap, significantly increasing the difficulty of automated segmentation. Furthermore, color inconsistencies and blurred nuclear boundaries caused by uneven manual manipulation also affect the quality of nucleus segmentation. For nucleus classification tasks, the problem of class imbalance in datasets persists. Furthermore, the misclassification, overlap, and blurred boundaries of nuclei remain significant challenges in research.

[0005] In addition to the aforementioned challenges, some methods treat nucleus segmentation and classification as a two-stage task: first, nucleus segmentation or detection, followed by classification based on nucleus or surrounding features. This second stage fails to fully utilize the spatial relationships between nuclei and the overall tissue structure, affecting classification accuracy. Furthermore, errors in the first-stage segmentation results (such as missed detections or incorrect contours) directly affect the second-stage classification, and inevitably increase training time. Summary of the Invention

[0006] The purpose of the present invention is to provide a method and system for fully automatic segmentation and classification of pathological images based on deep learning, so as to solve the problems of misclassification caused by insufficient extraction of environmental information, large number of training parameters, slow speed and other problems in the existing cell nucleus segmentation and classification methods on pathological images, and achieve the purpose of accurately segmenting and classifying cell nuclei.

[0007] In order to achieve the above object of the invention, the present invention adopts the following technical solutions: In the first aspect, the present invention provides a method for fully automatic segmentation and classification of pathological images based on deep learning, comprising the following steps:

[0008] (1) Select the tissue section image to be processed and perform preliminary image preprocessing, including pre-segmentation, content screening, and color normalization;

[0009] (2) Feature extraction is performed using an encoder composed of dilated-SE residual units and a unified feature decoder composed of self-attention dense units;

[0010] (3) Based on the extracted features, a kernel pixel map, horizontal and vertical distance maps, and a classification map are generated in the segmentation module and the classification module. A tissue region guidance branch is introduced in the classification module to improve the classification accuracy.

[0011] (4) The classification of each instance is completed by performing pixel-by-pixel voting within each predicted cell nucleus region; finally, the cell nucleus segmentation classification result is obtained.

[0012] Furthermore, in step (1), an image of a size suitable for deep learning is obtained by pre-segmentation, and the difference caused by color is eliminated by a color normalization method, and the processed image is used as the input image.

[0013] Furthermore, step (2) specifically includes: extracting features by an encoder composed of dilated-SE residual units; obtaining shared features by a unified feature decoder composed of self-attention dense units; the dilated part in the dilated-SE residual unit can be expressed as:

[0014]

[0015] in, is the energy of the geometric active contour model, is the output feature map, is the input feature map, is the convolution kernel weight, is the expansion rate, which controls the size of the receptive field, is the height and width of the convolution kernel;

[0016] The SE layer can be expressed as:

[0017]

[0018] in is the output feature map, X is the input feature map, is the global average pooling operation, is the ReLU activation function, is the Sigmoid activation function, , are the weight matrices of the two linear layers, ⊙ represents element-wise multiplication;

[0019] For the SE layer, the global information of each channel is obtained by global average pooling:

[0020]

[0021] in, It is The sample in The global average pooling result on the channels is: is the input feature map, where is the batch size, is the number of channels, and are the height and width of the feature map respectively;

[0022] The fully connected layer is represented as:

[0023]

[0024] in, is the desired channel attention weight vector, and are the weight matrices of the two fully connected layers, It is The global average pooling result vector of samples;

[0025] The calculation formula for feature recalibration is:

[0026]

[0027] in, is the final output feature map.

[0028] Furthermore, the encoder composed of the dilated-SE residual unit is composed of the following parts: (1) conv0 module, which consists of a , a convolutional layer with a stride of 1, a batch normalization layer, and a ReLU activation layer as the starting module; (2) an encoder backbone module consisting of four feature extraction layers composed of different numbers of dilated-SE residual units, which is used to extract local and global features; (3) the convbot layer adjusts the feature dimension and improves the feature expression ability.

[0029] Furthermore, the unified feature decoder composed of the self-attention intensive units can be expressed as:

[0030]

[0031] in is the output of the self-attention module, which integrates global information from all spatial locations and is learned through a learnable parameter After weighting, add back to the input feature map . The calculation formula is as follows:

[0032]

[0033] in, , , are the query, key, and value matrices obtained through convolution transformation, is a learnable weight matrix.

[0034] The unified feature encoder includes two layers of feature transfer layers consisting of self-attention dense units and two independent but internally shared feature modules, classification and segmentation modules, which further process the feature maps to output results.

[0035] Furthermore, the segmentation module and classification module are constructed as follows: (1) The shared part is represented by the u1 module, which consists of two layers. The pad layer consists of a convolution kernel of size The convolution layer is composed of a convolution kernel of size , which fills the input to ensure that the output size is consistent with the input size in subsequent convolution operations. The convolution layer performs preliminary feature extraction and conversion on the input features, providing shared feature representation for subsequent task branches; (2) The output of the u1 module is connected to two heads, each of which consists of a batch normalization layer, a ReLU activation function layer and a conv layer, responsible for giving the final output.

[0036] Furthermore, the classification module includes an auxiliary task of tissue region segmentation to make the cell nucleus segmentation more concerned with the surrounding environment. The annotation of this task is completely derived from the existing cell nucleus annotation, and its pseudo region mask is generated as follows: given the annotation type map , each kernel class ,in For the total number of categories, an influence map is generated by aggregating the distance decay effects from foreground pixels. , the final pseudo region mask By Influence diagram The calculation formula is obtained by double assignment:

[0037]

[0038] Through this dual assignment method, the original annotation information of the foreground pixels and the influence of different categories on the background pixels are comprehensively considered to generate the final pseudo region mask. , achieving accurate approximation of tissue regions with minimal human intervention.

[0039] In a second aspect, the present invention provides a system for fully automatic segmentation and classification of pathological images based on deep learning, which is used to implement the method described in the first aspect, including:

[0040] The first module is used to select the tissue section image to be processed and perform preliminary image preprocessing, including pre-segmentation, content screening and color normalization;

[0041] In the second module, an encoder composed of dilated-SE residual units and a unified feature decoder composed of self-attention dense units are used for feature extraction;

[0042] The third module uses the extracted features to generate kernel pixel maps, horizontal and vertical distance maps, and classification maps in the segmentation module and classification module. The tissue region guidance branch is introduced in the classification module to improve the classification accuracy.

[0043] The fourth module completes the classification of each instance by performing pixel-by-pixel voting within each predicted cell nucleus area; finally, the cell nucleus segmentation classification result is obtained.

[0044] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the program.

[0045] Compared with existing technologies, the present invention offers the following advantages: It introduces a novel deep learning framework designed to improve the accuracy of cell nucleus segmentation and classification by combining unified feature decoding with tissue region guidance. This framework leverages shared feature extraction and context-awareness to address significant challenges, including misclassification due to insufficient extraction of contextual information and the large number of training parameters and slow training speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a method for fully automatic segmentation and classification of pathological images based on deep learning according to an embodiment of the present invention;

[0047] Figure 2is a schematic diagram of straight line detection of an outer catheter on a longitudinal section according to an embodiment of the present invention;

[0048] Figure 3 is a flow chart of generating an initialized standard cylindrical surface on a three-dimensional OCT image according to an embodiment of the present invention;

[0049] Figure 4 is a schematic diagram of an initialized standard cylindrical surface generated on a three-dimensional OCT image according to an embodiment of the present invention;

[0050] Figure 5 is a comparison of segmentation results of different models on images with less interference according to an embodiment of the present invention;

[0051] Figure 6 This is a comparison of segmentation results of different models on images with large interference according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] This paper proposes a deep learning-based method for fully automatic pathology image segmentation and classification, suitable for tissue section image analysis in digital pathology. The method involves selecting a tissue section image to be processed, extracting nucleus feature information, and performing initialization processing. A unified feature decoder is introduced to process the image, integrating different feature information to improve segmentation and classification accuracy. A convolutional neural network (CNN) is used for instance-based segmentation of nuclei, and a self-attention mechanism and residual network are combined to optimize the feature extraction process. Furthermore, a classification module is used to predict the nucleus category, incorporating tissue region guidance information to improve classification accuracy. To overcome misclassification issues caused by insufficient extraction of contextual information and the large number of training parameters and slow training speed, the method utilizes multiple innovative modules, including a dilated-SE residual unit and a self-attention intensive unit, to enhance the network's contextual awareness and improve model robustness. Finally, through multi-task joint training, the segmentation and classification results are comprehensively optimized until a predetermined stopping condition is met, resulting in high-quality nucleus segmentation and classification results. The present invention can automatically and efficiently perform cell nucleus segmentation and classification in digital pathology images with high accuracy and stability, and is suitable for various complex tissue section analysis tasks without the need for human intervention.

[0053] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0054] Reference Figure 1 The present invention proposes a method for fully automatic segmentation and classification of pathological images based on deep learning, comprising the following steps:

[0055] Step (1) selects the tissue slice image to be processed and performs preliminary image preprocessing, including pre-segmentation, content screening, and color normalization.

[0056] In step (1), an image of a size suitable for deep learning is obtained by pre-segmentation, and the difference caused by color is eliminated by the Stain-to-Stain Translation (STST) color normalization method, and the processed image is used as the input image.

[0057] Pre-segmentation is performed on the pathological slide to be processed, breaking it into multiple sub-images of 256 pixels x 256 pixels. The average pixel value of each image is calculated, and images with an average pixel value below 20 or above 200 are eliminated. This achieves the purpose of preliminary image screening, filtering background, and improving speed. Color normalization is also used to eliminate color interference. Figure 2 It is a schematic diagram of the entire image processing process according to an embodiment of the present invention.

[0058] In step (2), an encoder composed of dilated-SE residual units and a unified feature decoder composed of self-attention dense units are used for feature extraction and feature decoding.

[0059] The dilation part in the dilated-SE residual unit is expressed as:

[0060]

[0061] in, is the energy of the geometric active contour model, is the output feature map, is the input feature map, is the convolution kernel weight, is the expansion rate, which controls the size of the receptive field, is the height and width of the convolution kernel;

[0062] For the SE (Squeeze-and-Excitation) layer, global information of each channel is obtained through global average pooling (Squeeze operation):

[0063]

[0064] in, It is The sample in The global average pooling result on the channels is: is the input feature map, where is the batch size, is the number of channels, and are the height and width of the feature map respectively;

[0065] The fully connected layer (Excitation operation) can be expressed as:

[0066]

[0067] in, is the desired channel attention weight vector, and are the weight matrices of the two fully connected layers, is the Sigmoid function, is the ReLU activation function, It is The global average pooling result vector of samples;

[0068] The calculation formula for the feature recalibration (Scale operation) is:

[0069]

[0070] in, is the final output feature map.

[0071] The encoder is composed of dilated-SE residual units for feature extraction, which includes: (1) conv0 module, which consists of a , a convolutional layer with a stride of 1, a batch normalization layer, and a ReLU activation layer as the starting module; (2) an encoder backbone module consisting of four feature extraction layers composed of different numbers of dilated-SE residual units, which is used to extract local and global features; (3) the convbot layer adjusts the feature dimension and improves the feature expression ability.

[0072] The extracted features are decoded by a unified feature decoder composed of self-attention dense units, which includes: (1) the self.u3 submodule includes a conva layer with a convolution kernel size of , a convolutional layer with a step size of 1 and padding of 0 processes the input feature map; the self-attention dense unit layer receives the feature map output by the conva layer, and uses convolution kernels of different sizes of 1x1 and 5x5 internally, containing 8 self-attention dense units, and divides the feature map into 4 parts for processing, thereby enhancing feature propagation, alleviating the gradient disappearance problem and increasing feature reusability; finally, the convf layer, as another convolutional layer, processes the feature map output by the self-attention dense unit layer, and completes the channel dimension adjustment and feature fusion through a 1x1 convolution kernel with a step size of 1, padding of 0 and no bias term; (2) the self.u2 submodule receives the feature map output by self.u3 and continues processing. The conva layer is a two-dimensional convolution layer that further reduces the number of channels of the input feature map. The convolution kernel size is 5, the stride is 1, and the padding is 0. The self-attention dense unit layer receives the output feature map of conva, uses 1x1 and 5x5 convolution kernels, contains 4 self-attention dense units, and divides the feature map into 4 parts for processing, which plays a role in enhancing feature propagation and reuse. Finally, the convf layer is a two-dimensional convolution layer of 1x1 convolution, which adjusts the channel dimension and fuses the feature map output by the self-attention dense unit layer. (3) Two independent but internally shared feature modules, the classification and segmentation modules, further process the feature map to output the result. Specifically, the feature map is input into the segmentation module and the classification module, and the result map is output through operations such as convolution, pooling, activation, etc., and the result is subjected to the post-processing process of the fourth module to output the final result, such as Figure 3 As shown in A.

[0073] In step (3), the extracted features are used to generate a nuclear pixel map (NP map) and a horizontal and vertical distance map (HoVer map), as well as a classification map (TP map) in two independent modules. The tissue region guidance branch (AR branch) is introduced into the classification module to improve the classification accuracy.

[0074] First, we need to obtain the annotation of the auxiliary organization area task, given the annotation type diagram , each kernel class ,in For the total number of categories, an influence map is generated by aggregating the distance decay effects from foreground pixels. For each nuclear category , first extract its binary mask:

[0075]

[0076] in is the indicator function, Indicated in coordinates The category to which the pixel at belongs, label each foreground pixel The spatial impact of propagation within the region is calculated as follows:

[0077]

[0078] in is the Euclidean distance calculation formula, is the attenuation coefficient, is the foreground weight parameter, which can be used to adjust the influence of each foreground pixel. Foreground pixels , which is applied to all pixels Impact Add to influence diagram It can also be expressed as:

[0079]

[0080] After all the categories The calculation and accumulation of foreground pixels of the category Influence diagram , which reflects the category The comprehensive influence distribution of all foreground pixels in the entire image area. The final pseudo region mask By Influence diagram The calculation formula is obtained by double assignment:

[0081]

[0082] T(x,y) represents the category to which the pixel at coordinate (x,y) belongs, and BG represents the background, that is, the pixels that do not belong to the cell nucleus;

[0083] Through this dual assignment method, the original annotation information of the foreground pixels and the influence of different categories on the background pixels are comprehensively considered to generate the final pseudo region mask. , achieving accurate approximation of tissue regions with minimal human intervention. Figure 4 As shown, (a) is the H&E stained pathological section, (b) is the distribution of cell nuclei on the pathological section, (c) is the tissue pseudo-region mask generated by this method based on the distribution of cell nuclei, and (d) is the real region mask of the tissue region. It can be seen that our tissue region annotation is already quite close to the real annotation.

[0084] The structure of the segmentation and classification module is as follows: (1) The shared part is represented by the u1 module, which consists of two layers. The pad layer consists of a convolution kernel of size The convolution layer is composed of a convolution kernel of size , which fills the input to ensure that the output size is consistent with the input size in subsequent convolution operations. The convolution layer performs preliminary feature extraction and conversion on the input features, providing shared feature representation for subsequent task branches; (2) The output of the u1 module is connected to two heads, each of which consists of a batch normalization layer, a ReLU activation function layer, and a conv layer, responsible for giving the final output. Figure 3 As shown in Figures B and C, the nuclear pixel map (NP map) and horizontal and vertical distance maps (HoVer maps) can be obtained, as well as the classification map (TP map) and tissue area segmentation map (only in the training stage).

[0085] In step (4), each instance is classified by performing pixel-by-pixel voting within each predicted cell nucleus region, and an accurate cell nucleus segmentation classification result is finally obtained.

[0086] The implementation effect of the method of the present invention is verified by a specific example below.

[0087] For three data sets, totaling 15,916 images, containing a total of 468,743 cell nuclei, the method for fully automatic segmentation and classification of pathological images based on deep learning of the present invention is compared with five other methods (see "H. Zhao, J. Shi, X.Qi, X. Wang, J. Jia, Pyramid scene parsing network, in Proceedings of the IEEE conference on computer vision and pattern recognition (2017), pp. 2881–2890", "W. Zhang, J. Pang, K. Chen, CC Loy, K-Net: Towards Unified Image Segmentation, in NeurIPS (2021)", "B. Cheng, I. Misra, AG Schwing, A.Kirillov, R. Girdhar, Masked-attention mask transformer for universal images segmentation, in Proceedings of the IEEE / CVF conference on computer visionand pattern recognition (2022), pp. 1290–1299.", "E. Xie, et al. al., SegFormer:Simple and Efficient Design for Semantic Segmentation with Transformers.arXiv preprint arXiv:2105.15203 (2021).","S. ​​Graham, et al., Hover-net:Simultaneous segmentation and classification of nuclei in multi-tissuehistology images. Medical image analysis 58, 101563 (2019)."), and the Dice coefficient is used to compare the segmentation and classification performance. The calculation formula is:

[0088]

[0089] Here, A represents the segmentation result, and G represents the true target region manually annotated by the doctor. Dice measures the degree of overlap between the segmentation result and the true target region. A high Dice value indicates that the experimental method is effective. Table 1 compares the Dice values ​​obtained by applying different segmentation models to the three datasets.

[0090] Table 1 Dice values ​​corresponding to each model segmentation in the example

[0091] Model PanMix CoNSeP PanNuke PSPNet 0.719 0.797 0.707 KNet 0.759 0.757 0.748 Mask2Former 0.684 0.823 0.609 Segformer 0.762 0.813 0.637 Hovernet 0.714 0.742 0.660 Model of the present invention 0.802 0.823 0.751

[0092] Experiments show that the model of the present invention is better than the other five model methods in segmenting and classifying pathological images.

[0093] Figure 5 It is a comparison of the segmentation results of different images of the three data sets according to different models. Figure 6It is the model performance when the data is unbalanced, where the first column is the initial image, the second column is the manually annotated real target boundary, and the third to seventh columns are other advanced models (see "H. Zhao, J. Shi, X. Qi, X. Wang, J. Jia, Pyramid sceneparsing network, in Proceedings of the IEEE conference on computer vision and pattern recognition (2017), pp. 2881–2890", "W. Zhang, J. Pang, K. Chen, CC Loy, K-Net: Towards Unified Image Segmentation, in NeurIPS (2021)", "B.Cheng, I. Misra, AG Schwing, A. Kirillov, R. Girdhar, Masked-attentionmask transformer for universal image segmentation, in Proceedings of theIEEE / CVF conference on computer vision and pattern recognition (2022), pp.1290–1299.", "E. Xie, et al., SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. arXiv preprint arXiv:2105.15203(2021).”, “S. Graham, et al., Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical image analysis 58, 101563 (2019).” The eighth column is the segmentation and classification results of the model of the present invention. It can be seen that the results of the segmentation and classification of the model of the present invention are significantly better than those of other methods, especially in Figure 6 The images shown are affected by data imbalance.

Claims

1. A method for fully automatic segmentation and classification of pathological images based on deep learning, characterized in that: The following steps are involved: (1) Select the tissue section image to be processed and perform preliminary image preprocessing, including pre-segmentation, content screening, and color normalization; (2) Feature extraction is performed using an encoder composed of dilated-SE residual units and a unified feature decoder composed of self-attention dense units; (3) Based on the extracted features, a kernel pixel map, horizontal and vertical distance maps, and a classification map are generated in the segmentation module and the classification module. A tissue region guidance branch is introduced in the classification module to improve the classification accuracy. (4) The classification of each instance is completed by performing pixel-by-pixel voting within each predicted cell nucleus region; finally, the cell nucleus segmentation classification result is obtained.

2. The method for fully automatic segmentation and classification of pathological images based on deep learning according to claim 1 is characterized in that: In step (1), an image of a size suitable for deep learning is obtained by pre-segmentation, and the differences caused by color are eliminated by the color normalization method, and the processed image is used as the input image.

3. The method for fully automatic segmentation and classification of pathological images based on deep learning according to claim 1 is characterized in that The dilation part in the dilated-SE residual unit is expressed as: ； in, is the energy of the geometric active contour model, is the output feature map, is the input feature map, is the convolution kernel weight, is the expansion rate, which controls the size of the receptive field, is the height and width of the convolution kernel; For the SE layer, the global information of each channel is obtained by global average pooling: ； in, It is The sample in The global average pooling result on the channels is: is the input feature map, where is the batch size, is the number of channels, and are the height and width of the feature map respectively; The fully connected layer is represented as: ； in, is the desired channel attention weight vector, and are the weight matrices of the two fully connected layers, is the Sigmoid function, is the ReLU activation function, It is The global average pooling result vector of samples; The calculation formula for feature recalibration is: ； in, is the final output feature map.

4. The method for fully automatic segmentation and classification of pathological images based on deep learning according to claim 3 is characterized in that: The encoder is composed of dilated-SE residual units, which consists of the following parts: (1) conv0 module, which consists of a , a convolutional layer with a stride of 1, a batch normalization layer, and a ReLU activation layer as the starting module; (2) an encoder backbone module consisting of four feature extraction layers composed of different numbers of dilated-SE residual units, which is used to extract local and global features; (3) a convbot layer adjusts the feature dimension to improve the feature expression ability.

5. The method for fully automatic segmentation and classification of pathological images based on deep learning according to claim 3 is characterized in that: The unified feature decoder consists of self-attention dense units, and the self-attention dense unit is expressed as: ； in is the output of the self-attention module, which integrates global information from all spatial locations and is learned through a learnable parameter After weighting, add back to the input feature map , The calculation formula is as follows: ； in, , , are the query, key, and value matrices obtained through convolution transformation, is a learnable weight matrix.

6. The method for fully automatic segmentation and classification of pathological images based on deep learning according to claim 5, characterized in that: The unified feature decoder includes: (1) self.u3 submodule, including a conva layer with a convolution kernel size of , the convolution layer with a step size of 1 and padding of 0 processes the input feature map; the self-attention dense unit layer receives the feature map output by the conva layer, and uses convolution kernels of different sizes of 1x1 and 5x5 internally, contains 8 self-attention dense units, and divides the feature map into 4 parts for processing; finally, the convf layer serves as another convolution layer to process the feature map output by the self-attention dense unit layer, and completes the channel dimension adjustment and feature fusion through a 1x1 convolution kernel with a step size of 1, padding of 0 and no bias term; (2) the self.u2 submodule receives the feature map output by the self.u3 submodule for further processing; the conva layer serves as a two-dimensional convolution layer to reduce the number of channels of the input feature map, with a convolution kernel size of 5, a step size of 1, and padding of 0; the self-attention dense unit layer receives the output feature map of the conva layer, uses 1x1 and 5x5 convolution kernels, contains 4 self-attention dense units, and divides the feature map into 4 parts for processing, which plays a role in enhancing feature propagation and reuse; finally, the convf The layer is a two-dimensional convolution layer with 1x1 convolution, which adjusts the channel dimension and fuses the feature maps output by the self-attention dense unit layer; (3) Two independent but internally shared feature modules: the segmentation module and the classification module, process the feature maps to output results.

7. The method for fully automatic segmentation and classification of pathological images based on deep learning according to claim 6, characterized in that: The segmentation module and classification module are composed as follows: (1) The shared part is represented by the u1 module, which consists of two layers. The pad layer consists of a convolution kernel of size The convolution layer is composed of a convolution kernel of size , and the input is padded to ensure that the output size is consistent with the input size in subsequent convolution operations; the conva layer is composed of a convolution kernel of size The convolution layer performs preliminary feature extraction and conversion on the input features, providing shared feature representation for subsequent task branches; (2) The output of the u1 module is connected to two heads, each of which consists of a batch normalization layer, a ReLU activation function layer and a conv layer, responsible for giving the final output.

8. The method for fully automatic segmentation and classification of pathological images based on deep learning according to claim 7, characterized in that: The classification module includes an auxiliary task of tissue region segmentation to make the cell nucleus segmentation more concerned with the surrounding environment. The annotation of this task is completely derived from the existing cell nucleus annotation. The pseudo region mask is generated as follows: given the annotation type map , each nuclear category ,in For the total number of categories, an influence map is generated by aggregating the distance decay effects from foreground pixels. ; For each kernel category , first extract its binary mask ； in is the indicator function, Indicated in coordinates The category to which the pixel at belongs, label each foreground pixel The spatial impact of propagation within the region is calculated as follows: ； in is the Euclidean distance calculation formula, is the attenuation coefficient, is the foreground weight parameter, which is used to adjust the influence of each foreground pixel; For each category Foreground pixels , which is applied to all pixels Impact Add to influence diagram In the expression: ； After all the categories The calculation and accumulation of foreground pixels to obtain the category Influence diagram , which reflects the category The comprehensive influence distribution of all foreground pixels in the entire image area; the final pseudo region mask By Influence diagram The calculation formula is obtained by double assignment: ； Among them, T(x,y) represents the category of the pixel at coordinate (x,y), and BG represents background; Through the dual assignment method, the original annotation information of the foreground pixels and the influence of different categories on the background pixels are comprehensively considered to generate the final pseudo region mask. .

9. A system for fully automatic segmentation and classification of pathological images based on deep learning, characterized in that: The method for implementing any one of claims 1 to 8 comprises: The first module is used to select the tissue section image to be processed and perform preliminary image preprocessing, including pre-segmentation, content screening and color normalization; In the second module, an encoder composed of dilated-SE residual units and a unified feature decoder composed of self-attention dense units are used for feature extraction; The third module uses the extracted features to generate kernel pixel maps, horizontal and vertical distance maps, and classification maps in the segmentation module and classification module. The tissue region guidance branch is introduced in the classification module to improve the classification accuracy. The fourth module completes the classification of each instance by performing pixel-by-pixel voting within each predicted cell nucleus area; finally, the cell nucleus segmentation classification result is obtained.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images