A full-slice pathological image segmentation and auxiliary annotation method

Abstract

The present invention relates to the field of image processing and medical technology, and specifically to a full-slice pathological image segmentation and auxiliary annotation method. First, the full-slice pathological image WSI is divided into a model training data set and a data set to be annotated, the cancer area in the model training data set is annotated, and a pathological image segmentation data set and a model prediction data set are constructed; the pathological image segmentation data set is used to train the U‑Net segmentation model; secondly, the trained U‑Net segmentation model is used to predict the small image blocks in the model prediction data set, and the segmentation results of each small image block are generated and pieced together into the result of the whole pathological slice; finally, the contour of the segmentation result is extracted, and the coordinate point values ​​of the contour are written into an XML annotation file, and the pathologist can review the annotation quality on the pathological image viewing software ASAP. The full-slice pathological image segmentation and auxiliary annotation method of the present invention can effectively segment the cancer area, and the auxiliary annotation can further reduce the annotation burden of the pathologist.

Description

Technical Field

[0001] The invention belongs to the field of image processing and medical technology, and in particular relates to a full-slice pathological image segmentation and auxiliary annotation method. Background Art

[0002] Histopathological images contain rich morphological characteristics and phenotypic information of cancer tissues. They are the gold standard for clinical diagnosis of cancer and provide an important reference for patients' preoperative diagnosis, postoperative prognosis, and targeted treatment. With the development of digital pathology technology, pathological sections are converted into digital images through high-resolution scanning, which is convenient for storage, sharing, and automated analysis. Whole-slice pathological image segmentation aims to automatically identify and separate areas of interest, such as cancer lesions, tumor areas, or other tissue structures, from large-size digital pathology images. In clinical practice, pathological image segmentation helps doctors diagnose diseases more accurately, especially in the detection and grading of major diseases such as cancer. The segmented areas can provide pathologists with quantitative indicators, such as tumor size, distribution, and invasiveness.

[0003] Deep learning-based methods have achieved great success in the whole-slice pathology image segmentation task. However, this method also faces many challenges. First, most deep learning models with good performance adopt a fully supervised approach, that is, the training process of the pathology image segmentation task requires a large amount of annotated data. Secondly, the annotation of pathology images requires professional pathologists, and due to the complexity of pathology image data, it is very time-consuming and labor-intensive to perform detailed annotation of WSI. Therefore, the study of accurate and efficient segmentation methods and auxiliary annotation methods is of great significance to reduce the burden on pathologists and improve the accuracy and efficiency of cancer diagnosis. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a full-slice pathological image segmentation and auxiliary annotation method. To achieve this purpose, the present invention adopts the following technical solutions:

[0005] A full-slice pathology image segmentation and auxiliary annotation method, the image segmentation and auxiliary annotation method comprises the following steps:

[0006] S1: Divide the pathological image data WSI into a model training data set and a data set to be labeled;

[0007] S2: Annotate the ROI region of the cancer area sliced ​​in the model training dataset, cut the WSI into several image blocks of 512×512 pixels at 10× magnification, and perform image preprocessing to construct a pathology image segmentation dataset;

[0008] S3: Cut the slices in the dataset to be labeled into several image blocks of 512×512 pixels at a magnification of 10× to construct a model prediction dataset;

[0009] S4: Using the pathological image segmentation dataset and the U-Net segmentation model as the backbone network, a pathological image segmentation model was established;

[0010] S5: The weighted sum of the cross entropy loss and the Dice loss is used as the total loss of the network to train the segmentation model;

[0011] S6: Use the segmentation model to predict the image blocks in the model prediction data set, generate the segmentation results of the image blocks, and piece together the segmentation results of the entire large image to facilitate visualization of the segmentation effect on the entire slice;

[0012] S7: extracting the contour of the whole slice segmentation result, and generating an XML annotation file according to the label format supported by the pathology image viewing software ASAP according to the coordinate point values ​​of the contour;

[0013] S8: The pathologist opens the pathology slide on ASAP and loads the XML annotation file to review the quality of the annotation and make adjustments for subsequent use.

[0014] Furthermore, in S2, the image preprocessing step in the model training data set includes:

[0015] S21: discard image blocks with white background exceeding 50%;

[0016] S22: Image blocks with cancer areas exceeding 25% are retained, and image blocks cut out from the ROI area of ​​lung squamous cell carcinoma slices are defined as LUSC, and image blocks cut out from the ROI area of ​​lung adenocarcinoma slices are defined as LUAD;

[0017] S23: The image block with the cancer area of ​​0% is retained and defined as Normal.

[0018] Furthermore, in S3, the slices in the dataset to be labeled are cut into several image blocks of 512×512 pixels at a magnification of 10×, and all the image blocks are retained to ensure the integrity of the prediction of the entire slice. The cut image blocks are used to construct the model prediction dataset.

[0019] Furthermore, in S4, the U-Net network model is composed of an encoding network and a decoding network. The encoding network includes 4 layers of convolution and maximum pooling operations, the decoding network includes 4 layers of convolution and deconvolution operations, a jump connection is added between the encoding network and the decoding network to fuse the semantic features of the corresponding layers of the encoding network and the decoding network, and the output channel of the last convolution layer of the decoding network is set to 3, the number of segmentation categories of the pathological image dataset.

[0020] Furthermore, in S5, the cross entropy loss and the Dice loss are weightedly summed as the total loss of the network to train the segmentation model, wherein the cross entropy loss L CE The expression is:

[0021] (1);

[0022] in, Indicates the size of the image. Indicates the label, Represents the prediction results of the model.

[0023] The Dice loss L Dice The expression is:

[0024] (2) ;

[0025] in, is a constant used to prevent division by zero.

[0026] The total loss of the segmentation model L total The expression is:

[0027] (3);

[0028] Furthermore, in S6, the trained segmentation model is used to predict the image blocks in the model prediction data set to generate the segmentation results of the image blocks. In the segmentation results of the image blocks, white is used to represent LUSC, green is used to represent LUAD, and black is used to represent Normal. The segmentation results of the small image blocks are pieced together into the segmentation results of the entire slice, so as to facilitate more intuitive observation of the segmentation visualization effect.

[0029] Furthermore, in S7, the step of extracting the contour of the segmentation result of the entire slice and generating an XML annotation file includes:

[0030] S71: extracting the contour of the segmentation result of the entire slice, and obtaining the coordinate points of the segmentation contour by taking values ​​at intervals of 15 pixels;

[0031] S72: according to the slice magnification factor during data preprocessing, the coordinate point value is magnified to the maximum level size of the whole slice, so as to ensure that the segmentation contour can correspond to the actual size of the whole slice;

[0032] S73: Generate an XML annotation file according to the label format supported by the pathology image viewing software ASAP according to the values ​​of all coordinate points;

[0033] Furthermore, in S8, the pathologist opens the pathology slide on ASAP and loads the XML annotation file, reviews the quality of the annotation, and makes adjustments for subsequent use.

[0034] Beneficial effects of the present invention: The present invention proposes a full-slice pathology image segmentation and auxiliary annotation method, which can perform fine segmentation on full-slice pathology images and assist pathologists in diagnosis. In addition, the auxiliary annotation method also enables a large amount of unlabeled data to be efficiently utilized, and the physician only needs to modify the annotations in a few areas with inaccurate segmentation or areas with complex edges. The modified annotations can be used as new training data to further improve the learning ability of the model and enhance the accuracy of the pathology image segmentation model. This method of human-computer collaborative annotation helps to continuously optimize the segmentation performance of the model, significantly speed up the annotation speed, and reduce the annotation burden of pathologists. It is particularly suitable for full-slice pathology image environments with large data volumes and complex annotations, and brings significant convenience to clinical research, pathology analysis, and diagnosis processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A flow chart of the method of this application;

[0037] Figure 2 For the segmentation model in 1 st Segmentation visualization effect diagram on the HSUMC dataset;

[0038] Figure 3 Comparison chart of manually refined labels and labels generated with the help of models;

[0039] Figure 4 Comparison chart of the original slice image and the label generated with the help of the model; DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0041] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0042] Embodiment 1

[0043] like Figure 1 As shown, this embodiment provides a full-slice pathology image segmentation and auxiliary annotation method, and the image segmentation and auxiliary annotation method includes the following steps:

[0044] S1: Divide the pathological image data WSI into a model training data set and a data set to be labeled;

[0045] S2: Annotate the ROI region of the cancer area sliced ​​in the model training dataset, cut the WSI into several image blocks of 512×512 pixels at 10× magnification, and perform image preprocessing to construct a pathology image segmentation dataset;

[0046] S3: Cut the slices in the dataset to be labeled into several image blocks of 512×512 pixels at a magnification of 10× to construct a model prediction dataset;

[0047] S4: Using the pathological image segmentation dataset and the U-Net segmentation model as the backbone network, a pathological image segmentation model was established;

[0048] S5: The weighted sum of the cross entropy loss and the Dice loss is used as the total loss of the network to train the segmentation model;

[0049] S6: Use the segmentation model to predict the image blocks in the model prediction data set, generate the segmentation results of the image blocks, and piece together the segmentation results of the entire large image to facilitate visualization of the segmentation effect on the entire slice;

[0050] S7: extracting the contour of the whole slice segmentation result, and generating an XML annotation file according to the label format supported by the pathology image viewing software ASAP according to the coordinate point values ​​of the contour;

[0051] S8: Pathologists can open pathology slides on ASAP and load XML annotation files to review the quality of the annotations and make adjustments for use in subsequent projects.

[0052] Furthermore, in S2, the image preprocessing step in the model training data set includes:

[0053] S21: discard image blocks with white background exceeding 50%;

[0054] S22: Image blocks with cancer areas exceeding 25% are retained, and image blocks cut out from the ROI area of ​​lung squamous cell carcinoma slices are defined as LUSC, and image blocks cut out from the ROI area of ​​lung adenocarcinoma slices are defined as LUAD;

[0055] S23: The image block with the cancer area of ​​0% is retained and defined as Normal.

[0056] Furthermore, in S3, the slices in the dataset to be labeled are cut into several image blocks of 512×512 pixels at a magnification of 10′, and all the image blocks are retained to ensure the integrity of the prediction of the entire slice. The cut image blocks are used to construct the model prediction dataset.

[0057] Furthermore, in S4, the U-Net network model is composed of an encoding network and a decoding network. The encoding network includes 4 layers of convolution and maximum pooling operations, the decoding network includes 4 layers of convolution and deconvolution operations, a jump connection is added between the encoding network and the decoding network to fuse the semantic features of the corresponding layers of the encoding network and the decoding network, and the output channel of the last convolution layer of the decoding network is set to 3, the number of segmentation categories of the pathological image dataset.

[0058] Furthermore, in S5, the cross entropy loss and the Dice loss are weighted and summed as the total loss of the network to train the segmentation model, wherein the expression of the cross entropy loss is:

[0059] (1) ;

[0060] in, Indicates the size of the image. Indicates the label, Represents the prediction result of the model. The expression of Dice loss is:

[0061] (2) ;

[0062] in, is a small constant used to prevent division by zero. The total loss of the segmentation model is expressed as:

[0063] (3);

[0064] Furthermore, in S6, the trained segmentation model is used to predict the image blocks in the model prediction data set to generate the segmentation results of the image blocks. In the segmentation results of the image blocks, white is used to represent LUSC, green is used to represent LUAD, and black is used to represent Normal. The segmentation results of the small image blocks are pieced together into the segmentation results of the entire slice, so as to more intuitively observe the segmentation visualization effect.

[0065] Furthermore, in S7, the step of extracting the contour of the segmentation result of the entire slice and generating an XML annotation file includes:

[0066] S71: extracting the contour of the segmentation result of the entire slice, and obtaining the coordinate points of the segmentation contour by taking values ​​at intervals of 15 pixels;

[0067] S72: according to the slice magnification factor during data preprocessing, the coordinate point value is magnified to the maximum level size of the whole slice, so as to ensure that the segmentation contour can correspond to the actual size of the whole slice;

[0068] S73: Generate an XML annotation file according to the label format supported by the pathology image viewing software ASAP according to the values ​​of all coordinate points;

[0069] Furthermore, in S8, the pathologist can open the pathology slide on ASAP and load the XML annotation file, review the quality of the annotation, and make adjustments to facilitate use in subsequent projects.

[0070] Embodiment 2

[0071] This embodiment should be understood to include at least all the features of the aforementioned embodiments and be further implemented on the basis of the aforementioned embodiments.

[0072] Experimental data:

[0073] (1) 1 st HSUMC Dataset

[0074] This dataset contains 25 HE-stained non-small cell lung cancer sections, including 16 lung squamous cell carcinoma sections and 9 lung adenocarcinoma sections. All sections were scanned by a 40x digital slide scanner, the slice format is MRXS, and the size of each slice is approximately 18000×37000 pixels. The cancer area was annotated on the ASAP software to generate an XML label file. All WSIs were divided into two groups, 18 WSIs were used for training, of which 12 were lung squamous cell carcinomas and 6 were lung adenocarcinomas; 7 were used for testing, of which 4 were lung squamous cell carcinomas and 3 were lung adenocarcinomas. First, each WSI was cut into 512×512 pixel blocks at 10× magnification and 50% overlap, and the blurred, dirty, over-stained and white background areas exceeding 50% were removed; the image blocks with more than 25% cancer area were retained, and the image blocks cut out from the ROI area of ​​lung squamous cell carcinoma slices were defined as LUSC, and the image blocks cut out from the ROI area of ​​lung adenocarcinoma slices were defined as LUAD; the image blocks with 0% cancer area were retained and defined as Normal.

[0075] Experimental configuration and parameter setting

[0076] The network model used in this embodiment is U-Net, including an encoding network and a decoding network. The encoding network contains 4 layers of convolution and maximum pooling operations, and the decoding network contains 4 layers of convolution and deconvolution operations. Different from the original U-Net, the number of channels of each layer of the U-Net network is halved. During training, the batch size is 16 and the epoch is 80; the SGD optimizer is used, the learning rate is 0.01, and the momentum is 0.9.

[0077] Experimental results analysis

[0078] The evaluation indicators of the model training in this embodiment are IoU, Dice, Recall, and Precision. The experimental results are shown in Table 1. Table 1 shows the evaluation indicators of the segmentation model in 1 st Performance comparison on the HSUMC dataset. As shown in Table 1, the segmentation model of this embodiment has good performance, with IoU reaching 0.7484 and Dice reaching 0.8404. In order to more intuitively show the effectiveness of the method proposed in the present invention, Figure 2 For the segmentation model in 1 st Segmentation visualization on the HSUMC dataset, Figure 2 (a) is a thumbnail of the whole section of lung squamous cell carcinoma. Figure 2 (b) is the segmentation result of the whole slice of lung squamous cell carcinoma. Figure 2 (c) is a thumbnail of a whole section of lung adenocarcinoma. Figure 2 (d) is the segmentation result of the whole slice of lung adenocarcinoma, where the blue line indicates the cancer area, white indicates lung squamous cell carcinoma, green indicates lung adenocarcinoma, and black indicates non-tumor area. As can be seen from the figure, the overall segmentation of the whole slice by the proposed method is good, and the visual similarity with the true label is high.

[0079] Table 1 Segmentation model in 1 st Performance on the HSUMC dataset

[0080]

[0081] In order to demonstrate the effectiveness of the auxiliary annotation method of the present invention, the outline of the segmentation result is extracted according to steps S71-S73 to generate an XML file. Figure 3 Shown on a slice with manual labeling, Figure 3 (a) is a manually labeled image. Figure 3 (b) is a label map generated with the assistance of the model, comparing the labels generated by model prediction with the manual labels, where the blue line area indicates cancer. Figure 4 Shown on slices without fine labeling, Figure 4 (a) is the original image without manual labeling. Figure 4(b) is the label map generated by model prediction, which is a comparison between the labels generated by model prediction and the original slice image. As can be seen from the figure, the segmentation effect of the model on the whole slice is good, the boundary segmentation of the cancer area is smooth, and the labels generated by model prediction are highly consistent with the manual fine labels. The intervals between the annotation points of the manual labels are different, while the annotation point intervals of the XML labels generated by the auxiliary annotation method are consistent, the edge smoothness of the cancer area is good, and the quality is high. It can be seen that the model-assisted annotation method can further reduce the annotation burden of pathologists.

[0082] It should be noted that the above specific implementations are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art should understand that various modifications, equivalent substitutions, changes, etc. can be made to the present invention. However, as long as these changes do not deviate from the spirit of the present invention, they should be within the scope of protection of the present invention. In addition, some terms used in the specification and claims of this application are not restrictive, but are only for the convenience of description.

Claims

1. A full-slice pathological image segmentation and auxiliary annotation method, characterized in that: The image segmentation and auxiliary annotation method comprises the following steps: S1: Divide the pathological image data WSI into a model training data set and a data set to be labeled; S2: annotate the ROI region of the cancer area sliced ​​in the model training data set, cut the WSI into several image blocks of 512×512 pixels at 10× magnification, perform image preprocessing, and construct a pathology image segmentation data set; S3: Cut the slices in the dataset to be labeled into several image blocks of 512×512 pixels at a magnification of 10× to construct a model prediction dataset; S4: Using the pathological image segmentation dataset and the U-Net segmentation model as the backbone network, a U-Net pathological image segmentation model was established; S5: The cross entropy loss and the Dice loss are weightedly summed as the total loss of the network to train the U-Net pathological image segmentation model; S6: using the U-Net pathological image segmentation model to predict the image blocks in the model prediction data set, generating segmentation results of the image blocks, and stitching the segmentation results of the entire large image together to facilitate visualization of the segmentation effect on the entire slice; S7: extracting the contour of the whole slice segmentation result, and generating an XML annotation file according to the label format supported by the pathology image viewing software ASAP according to the coordinate point values ​​of the contour; S8: The pathologist opens the pathology slide on ASAP and loads the XML annotation file, reviews the quality of the annotation, and makes adjustments for subsequent use; In S4, the U-Net pathological image segmentation model is composed of an encoding network and a decoding network, the encoding network includes 4 layers of convolution and maximum pooling operations, the decoding network includes 4 layers of convolution and deconvolution operations, a jump connection is added between the encoding network and the decoding network to fuse the semantic features of the corresponding levels of the encoding network and the decoding network, and the output channel of the last convolution layer of the decoding network is set to 3, the number of segmentation categories of the pathological image dataset; In S5, the cross entropy loss and the Dice loss are weightedly summed as the total loss of the network to train the U-Net pathological image segmentation model, wherein the cross entropy loss L CE The expression is: ； Among them, HW represents the size of the image, y represents the label, Represents the prediction results of the model; The Dice loss L Dice The expression is: ； Where ε is a constant used to prevent division by zero; The total loss of the segmentation model is L total The expression is: ； In S7, the step of extracting the contour of the segmentation result of the entire slice and generating an XML annotation file includes: S71: extracting the contour of the segmentation result of the entire slice, and obtaining the coordinate points of the segmentation contour by taking values ​​at intervals of 15 pixels; S72: according to the slice magnification factor during data preprocessing, the coordinate point value is magnified to the maximum level size of the whole slice, so as to ensure that the segmentation contour can correspond to the actual size of the whole slice; S73: Generate an XML annotation file according to the label format supported by the pathology image viewing software ASAP according to the numerical values ​​of all coordinate points.

2. The whole-slice pathological image segmentation and auxiliary annotation method according to claim 1, characterized in that: In S2, the image preprocessing step in the model training data set includes: S21: discard image blocks with white background exceeding 50%; S22: retain the image blocks with cancer area exceeding 25%, and define the image blocks cut out from the ROI area of ​​lung squamous cell carcinoma slices as LUSC, and define the image blocks cut out from the ROI area of ​​lung adenocarcinoma slices as LUAD; S23: The image patch with cancer area of ​​0% is retained and defined as Normal.

3. The whole-slice pathological image segmentation and auxiliary annotation method according to claim 2, characterized in that: In S3, the slices in the dataset to be labeled are cut into several image blocks of 512×512 pixels at a magnification of 10×, and all the image blocks are retained to ensure the integrity of the prediction of the entire slice, and the cut image blocks are used to construct the model prediction dataset.

4. The whole-slice pathological image segmentation and auxiliary annotation method according to claim 1, characterized in that: In S6, the trained segmentation model is used to predict the image blocks in the model prediction data set to generate the segmentation results of the image blocks. In the segmentation results of the image blocks, white is used to represent LUSC, green is used to represent LUAD, and black is used to represent Normal. The segmentation results of the small image blocks are pieced together into the segmentation results of the entire slice to facilitate more intuitive observation of the segmentation visualization effect.

Images