A method for evaluating segmentation models of digital full-frame histopathology images

Abstract

A method for evaluating a digital full-frame histopathology image segmentation model comprises the following steps: S100: proposing a high-density false positive rate MAFaR indicator for measuring the affected false positive FP area generated by the digital full-frame histopathology image segmentation algorithm; S200: using the MAFaR to complete the performance evaluation of the digital full-frame histopathology image segmentation model. The method introduces the concept of MAFaR and proposes an easy-to-interpret intuitive indicator for evaluating the performance of the segmentation model, so that the segmentation performance of the actual training model can be correctly reflected.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing and artificial intelligence technology, and in particular relates to a method for evaluating a digital full-frame tissue pathology image segmentation model. Background Art

[0002] In recent years, with the rapid development of information technology and the widespread popularity of artificial intelligence, computer vision has also developed rapidly. Image segmentation is a crucial preprocessing step for image recognition and computer vision. As a hot topic in the field of computer vision, it is widely used in various fields in real life, such as face recognition, autonomous driving systems, pedestrian monitoring, and content-based image retrieval systems.

[0003] Image segmentation refers to dividing an image into several non-overlapping regions based on features such as grayscale, color, texture, and geometric shape, so that these features remain consistent or similar within the same segmented region, but show obvious differences between different segmented regions. A large number of computer vision tasks usually rely on correct segmentation results. However, segmentation results are often affected by changes in object size, image contrast, and image resolution. To date, researchers have made a lot of efforts to develop segmentation algorithms for different types of images in different scenarios. In the process of developing segmentation algorithms, it is crucial to use appropriate evaluation metrics because these evaluation metrics evaluate and compare the performance of different algorithms in an objective way.

[0004] After years of research, researchers have proposed many metrics to evaluate the performance of image segmentation algorithms. The following content is based on the premise of binary segmentation. In the context of binary segmentation, we can construct a confusion matrix to calculate the number of True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) pixels contained in the segmentation result (MGSR) image generated by the model. In an MGSR image, we call the object area (foreground) the positive area and the corresponding background area the negative area. The basic concepts in the confusion matrix are defined as follows: TP and TN represent the correctly segmented object / positive area and background / negative area in MGSR, respectively, and FP and FN represent the incorrectly segmented object / positive area and background / negative area in MGSR, respectively.

[0005] First, we briefly describe popular segmentation evaluation metrics.

[0006] Overlap-based metrics include:

[0007] Accuracy: Accuracy is the ratio of the number of correctly segmented pixels in MGSR to the total image. The value range of accuracy is [0, 1], where 1 indicates perfect segmentation and 0 indicates that no pixel in MGSR is correctly segmented. The definition of accuracy is as follows:

[0008]

[0009] When the amount of data of positive and negative classes in an image is unbalanced, the accuracy metric will have a great impact on the performance evaluation of the segmentation model. For example, if the number of negative pixels in the image is dominant, that is, the number of negative pixels is much higher than the positive pixels, even if the number of correctly classified positive pixels in MGSR is zero, that is, TP = 0, the value of accuracy is very high. In this case, the metric does not correctly reflect the true performance of the segmentation model. Unfortunately, in practice, data with unbalanced classes is very common, especially in the context of medical images.

[0010] Specificity: Specificity measures the ratio of negative pixels in the annotated image (ground truth, GT) that are also identified as negative pixels in MGSR to all negative pixels in GT. The value range of this indicator is [0, 1], where 1 indicates perfect segmentation. The definition of specificity is as follows:

[0011]

[0012] When MGSR contains a large number of TN pixels, it means that most of the background areas are correctly segmented, so this indicator will get a very high score, but it ignores whether the positive areas in the MGSR image are correctly segmented. When MGSR contains a large number of FN areas, it indicates that the segmentation performance of the model is not good. Therefore, in this case, specificity does not really reflect the performance of the model.

[0013] False Positive Rate (FPR): The false positive rate measures the ratio of the number of negative pixels (background) in the GT image that are identified as positive pixels in the MGSR to all negative pixels in the GT. This indicator measures the positive pixels that are segmented incorrectly, and its value range is [0, 1], where 0 indicates perfect segmentation and 1 indicates that no negative pixels are correctly segmented. The definition of the false positive rate is as follows:

[0014]

[0015] When MGSR contains a large number of TN pixels, it means that most of the background areas are correctly segmented. At this time, the FPR value is very small. However, FPR ignores whether MGSR contains a lot of FN areas, that is, it ignores the segmentation of the positive class. Therefore, in this case, FPR does not really reflect the performance of the model.

[0016] Recall: Recall is also called sensitivity, which measures the ratio of the number of positive pixels in the GT image that are also identified as positive pixels in the MGSR to all positive pixels in the GT. The value range of this indicator is [0, 1], where 0 means that the MGSR and the corresponding GT image have no intersection (FP = 0), and 1 means that the segmentation model has perfectly segmented the object / foreground pixels. The definition of recall is as follows:

[0017]

[0018] When MGSR contains a large number of TP pixels, it means that most of the foreground areas are correctly segmented, and this indicator will get a very high score. However, it ignores whether MGSR contains a large number of FP areas, that is, the segmentation effect of background pixels may not be good. In this case, the recall rate does not really reflect the performance of the model.

[0019] Precision: It measures the ratio of the number of positive pixels in the GT image that are also identified as positive pixels in the MGSR to all positive pixels in the MGSR. The value range of this indicator is [0, 1], where 0 means that the MGSR and the corresponding GT image have no intersection (TP=0), and 1 means that the segmentation model has perfectly segmented the object / foreground pixels. The definition of precision is as follows:

[0020]

[0021] As long as MGSR contains a large number of TP pixels, this indicator will achieve a high value, but it ignores whether the number of positive pixels in the GT image that are also identified as positive pixels in MGSR accounts for the majority of the true positive pixels in GT.

[0022] Dice Similarity Coefficient (DSC): This is an evaluation indicator often used in medical image segmentation algorithms. The definition of this indicator is as follows:

[0023]

[0024] When the structure of the segmented region is small, DSC is not a suitable evaluation metric because the single-pixel difference between the two segmentation models may have a large impact on the value of DSC. DSC is also not suitable for evaluating segmentation models for multi-target detection tasks.

[0025] Jaccard Index / Jaccard: Jaccard Index is also called Intersection Over Union (IOU). It describes the relationship between the intersection of MGSR and GT and their union. As the overlap between MGSR and GT increases, the number of elements in the intersection will gradually increase, and the cardinality of the union will gradually decrease. When the overlap between MGSR and GT reaches the maximum, the Jaccard value is 1, and vice versa, when their overlap is 0, the Jaccard value is 0. The definition of Jaccard is as follows:

[0026]

[0027] DSC and Jaccard are used to evaluate and compare the results of medical image segmentation, but these two indicators are unbalanced because they penalize under-segmentation and over-segmentation to the same extent. For binary segmentation problems, errors in labeling negative pixels (i.e., under-segmentation) have a greater impact on the results than errors in labeling positive pixels (i.e., over-segmentation).

[0028] The value range of both DSC and Jaccard is [0, 1], and there is redundancy between them. The definition of the redundancy relationship is as follows:

[0029]

[0030] This means that these two metrics measure the same aspect of the segmentation model, and there is no additional information provided to select them as evaluation metrics at the same time, otherwise there will be redundancy.

[0031] Voxel Overlap Error (VOE): This metric focuses on the measurement of pixels that are incorrectly segmented. The value range of this metric is [0, 1], where 0 is a perfect segmentation and 1 means that there is no overlap between MGSR and GT. It is defined as follows:

[0032]

[0033] When positive pixels dominate the image, that is, the number of positive pixels is much greater than that of negative pixels, and when most of the positive pixels are correctly segmented, TP is large and FN is small. Even if FP is large (particularly small compared to TP), the value of VOE is still small. In this case, VOE does not really reflect the performance of the model.

[0034] VOE and Jaccard index are a pair of complementary indicators. When they need to be selected for segmentation model evaluation, they should not appear at the same time, otherwise redundancy will occur.

[0035] Size-based metrics include:

[0036] Relative Volume Difference (RVD): Relative volume difference is a measure of the difference in size between MGSR and GT images. The definition of this indicator is as follows:

[0037]

[0038] A value of 0 means that the area of ​​foreground pixels contained in MGSR and GT is the same. Note that this does not mean that MGSR and GT are identical, or actually overlap each other. In other words, when their areas are the same, even if they do not have any overlapping parts, the value of RVD is still 0.

[0039] In fact, size-based metrics only focus on the size difference between MGSR and GT images, but ignore their spatial position relationship.

[0040] Surface distance-based metrics include:

[0041] Hausdorff distance (HD): Hausdorff distance is also called the maximum symmetric surface distance, and its definition is as follows:

[0042]

[0043] Among them B MGSR and B GT denote the boundaries of the object regions in MGSR and GT, respectively, and d(·) denotes the Euclidean distance.

[0044] Average symmetric surface distance (Average symmetric surface distance / ASD): The average symmetric surface distance is based on the surface pixels of the MGSR and GT images. It is the accumulation of the minimum distances from all points on the object boundary in the MGSR image to the object boundary in the GT image, and the accumulation of the minimum distances from all points on the object boundary in the GT image to the object boundary in the MGSR image, and then the average of the two is calculated. It is defined as follows:

[0045]

[0046] The value range of HD and ASD is [0, fixed value], where 0 indicates perfect segmentation and the fixed value is related to the size and position of the segmented area in the image. For WSI, the fixed value is very large. When using these two indicators to evaluate the performance of the segmentation model, compared with recall, which has a value range of [0, 1], we cannot directly analyze the performance of the segmentation model, and may even make mistakes.

[0047] Surface distance based methods only consider the minimum distance between MGSR and GT boundaries, but do not reflect the difference in the real area between MGSR and GT images. At the same time, they do not accurately express the segmentation quality of objects with complex shapes.

[0048] Currently, evaluation metrics used to evaluate natural image segmentation models are still used to evaluate segmentation models for digital WSI. However, due to the uniqueness of digital WSI itself (i.e., large size and extremely complex structure), existing metrics may not correctly reflect the true performance of the trained model. For example, in the segmentation task of small structures (e.g., brain lesions), a single pixel difference may affect the Dyss Similarity Coefficient (DSC) by 33%. In the context of pathological image analysis, the images processed by researchers are usually high-resolution images containing more than 400 million pixels. In addition, the regions / objects of interest in pathological tissues may be sparsely distributed and vary from organ to organ, for example, from small mitotic images to large tumor area images. Therefore, existing segmentation metrics may no longer meet the needs for reasonable evaluation of segmentation models developed for digital full-frame tissue pathology slide images (WSI). How to adapt or compensate for existing segmentation metrics to better evaluate the performance of segmentation algorithms designed for WSI has become an urgent problem to be solved. Summary of the invention

[0049] In order to solve the above problems, the present disclosure provides a method for evaluating a digital full-frame tissue pathology image segmentation model, which comprises the following steps:

[0050] S100: Propose the MAFaR metric, which is used to measure the affected false positive FP area generated by the segmentation algorithm of digital full-frame histopathology images;

[0051] S200: Using the MAFaR to evaluate the performance of the digital full-frame tissue pathology image segmentation model.

[0052] Through the above technical solution, an easy-to-interpret intuitive evaluation metric is proposed to measure the impact of the area containing dense false positive pixels generated by the segmentation model. Compared with traditional segmentation metrics that usually only consider pixel-level differences, this method proposes a measurement method based on the affected area of ​​FP area / pixel distribution, which can correctly reflect the segmentation performance of the actual training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of a method for evaluating a digital full-frame tissue pathology image segmentation model provided in one embodiment of the present disclosure;

[0054] Figure 2 (a) to Figure 2 (d) is an example diagram of epithelial segmentation of oral squamous cell carcinoma (OCSCC) in one embodiment of the present disclosure;

[0055] Figure 3 (a) to Figure 3 (i) is a flowchart of MAFaR calculation in one embodiment of the present disclosure;

[0056] Figure 4 (a) to Figure 4 (d) shows three representative pathological images evaluated and compared using MAFaR and existing indicators in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] In one embodiment, Figure 1 As shown, a method for evaluating a digital full-frame tissue pathology image segmentation model is disclosed, which includes the following steps:

[0058] S100: Propose the MAFaR metric, which is used to measure the affected false positive FP area generated by the segmentation algorithm of digital full-frame histopathology images;

[0059] S200: Using the MAFaR to evaluate the performance of the digital full-frame tissue pathology image segmentation model.

[0060] Before introducing MAFaR, let’s first describe the reasons for developing the MAFaR indicator. Figure 2 (a) to Figure 2 (d) shows an example of epithelial segmentation for oral squamous cell carcinoma (OCSCC), where Figure 2 (a) is the GT binary image, Figure 2 (b) is the corresponding MGSR image generated by the pre-trained segmentation model. It can be observed that the segmentation result (MGSR) contains most of the epithelial tissue shown in the GT image. Therefore, it is expected that the recall rate of MGSR will be high, Recall = 0.9738, as shown in Table 1. In addition, MGSR also obtains high values ​​in other indicators, such as Accuracy and Precision. However, please note that in the context of histological images, we are dealing with high-resolution images that may be 20 to 100 times larger than natural images. If we include a small part of the FP region in MGSR (such as Figure 2(b)) is enlarged to obtain the enlarged image (as shown in Figure 2 (c)) and its corresponding histological images ( Figure 2 (d)), we can see that these small areas scattered on the left side of the MGSR image are FP areas generated by the pre-trained model. These FP areas are muscles and other tissues that the model incorrectly segmented from the image. In addition, these FP components are densely distributed in a certain area, indicating that the pre-trained model also made similar mistakes in the image. However, several existing pixel-level indicators shown in Table 1 cannot reflect this defect of the pre-trained model. In other words, the values ​​of accuracy, precision, and recall are very high, all greater than 0.9 and close to 1, and the HD indicator cannot intuitively reflect the segmentation performance. Therefore, it is necessary to develop an indicator to evaluate the affected areas where FP pixels are densely distributed.

[0061]

[0062] Table 1

[0063] In another embodiment, the MAFaR calculation comprises the following steps:

[0064] S101: Estimate the affected false positive FP area in the model segmentation result MGSR;

[0065] S102: Calculate the penalty for the affected false positive FP region.

[0066] For this embodiment, the schematic diagram of MAFaR calculation is as follows Figure 3 (a) to Figure 3 As shown in (i), it consists of two modules: 1-estimation of affected false positive FP regions in MGSR, 2-calculation of penalties for affected false positive FP regions. First, the density of accurate FP regions is estimated by Gaussian kernel density estimation. Then the Mean Shift clustering algorithm is applied to these dense regions to further identify the affected regions of dense FP pixels. Finally, a penalty function is used to penalize these affected regions for the calculation of MAFaR.

[0067] In another embodiment, the step S101 further includes:

[0068] S1011: Remove large false positive FP regions;

[0069] S1012: Estimating the density of FP pixels;

[0070] S1013: thresholding the FP density map to obtain a high-density FP map;

[0071] S1014: clustering pixel regions of adjacent high-density FP images;

[0072] S1015: Convex hull construction, using different convex hulls to represent the area of ​​the affected FP in the MGSR.

[0073] For this embodiment, given a MGSR image, denoted as B MG , and the corresponding GT image, denoted as B GT The main purpose of this module is to identify the affected FP regions of densely FP pixels.

[0074] In another embodiment, the step S102 further includes:

[0075] A modified logistic function is used as the penalty function to penalize the area of ​​the affected region.

[0076] For this embodiment, the main goal of this module is to use a penalty function to penalize the affected area determined in S101 and then perform MAFaR calculation. The larger the area of ​​the affected area, the greater the impact on the model segmentation performance, and the greater the corresponding penalty. Therefore, we design a modified logistic function as a penalty function to penalize the area of ​​the affected area.

[0077] In another embodiment, the step S1011 further includes:

[0078] S10111: Get the FP region in MGSR, denoted as B FP ;

[0079] S10112: From the B FP The threshold method is used to remove the large FP area, and the image with the large FP area removed is obtained, which is represented by B fFP .

[0080] In this embodiment, the FP region in the MGSR is first obtained, denoted as B FP According to the input GT and MGSR images, the FP image is obtained using the following formula:

[0081] B FP =(B MG ∪B GT )-B GT

[0082] It is worth noting that existing metrics have already considered the impact of FP regions with a large number of pixels (e.g. Figure 3 Therefore, before MAFaR calculation, we first calculate the FPRemove these large FP regions. Use Z = (z1, z2, ... z N ) means B FP The binary small components (connected components in binary images) scattered in B, where N is the total number of small components. We select a threshold for B FP Thresholding is performed, where the threshold is denoted by T size , the specific size is 2000 pixels, thus obtaining a set of small component sets Z S ={z i ∈Z|z i <T size}, Z S The number of FP pixels contained in each element in is less than 2000. This step obtains an image with large FP areas removed, denoted as B fFP .

[0083] In another embodiment, the step S1012 further includes:

[0084] Use kernel density estimation to estimate B fFP The density of FP pixels in .

[0085] In this embodiment, unlike the existing indicators that only consider the absolute number of FP pixels, we believe that high-density FP pixels will also affect the segmentation performance of the model. In this step, we use the kernel density estimation method to estimate B fFP The density of FP pixels in . We use kernel density estimation because it does not require prior knowledge of the data distribution.

[0086] We define T = (t1, t2, ... t S ) to represent B fFP FP pixels in B, and assume that these pixels are independent and identically distributed, where S is the FP pixel in B fFP The total number of FP pixels in T. T obeys a density distribution function f(t), but we don’t know what kind of distribution it is, so we need to estimate the distribution function of the entire FP pixel from T The density at any given FP pixel t can be calculated as follows:

[0087]

[0088] Where K is a probability kernel function (a non-negative function, such as a Gaussian kernel function). h is a smoothing parameter called bandwidth, which determines the smoothness of the kernel density estimation curve. For a smaller h, the estimated curve will be rough, and for a larger h, a smooth estimate will be obtained (based on experience, we set h = 103). The kernel with the subscript h is called a scaling kernel, which is defined as follows:

[0089]

[0090] In this paper, we choose the Gaussian kernel function as the probability kernel function to estimate the density of FP pixels, which is defined as follows:

[0091]

[0092] where σ is the standard deviation (σ = 17 in this paper). Finally, the FP density map is as follows Figure 3 As shown in (e), it is a grayscale image, where the higher the FP pixel density, the larger the value.

[0093] In another embodiment, the step S1013 further includes:

[0094] Select a threshold of 50.

[0095] In this embodiment, a threshold of 50 is selected to threshold the FP density map, thereby obtaining a high-density FP pixel area. Figure 3 (f) is shown as B hFP .

[0096] In another embodiment, the step S1014 further includes:

[0097] The Mean Shift clustering algorithm is used to cluster the pixels of adjacent high-density images.

[0098] In this embodiment, once a high-density FP image is obtained, we will use the Mean Shift clustering algorithm to cluster adjacent high-density pixels. The Mean Shift algorithm is a non-parametric feature space analysis technique for locating the maximum value of a density function.

[0099] In this paper, we define P as B hFP The total number of FP pixels in , and each pixel only considers the characteristics of the 2D spatial coordinates. Therefore, we get a feature vector set W = (w1, w2, ..., w P ), and w i =(x i ,y i ). For each eigenvector w i ∈W, corresponds to a cluster center point y i Mean shift will help us identify the cluster center point y corresponding to each feature vector in the feature space defined above. i At the beginning, y i Initialized to a certain feature vector w i ,Right now Then Use the iterative formula to continuously iterate and update. The iterative formula is as follows:

[0100]

[0101] in yes An updated version of It is called the Mean Shift vector and the basic form of Mean Shift is defined as follows:

[0102]

[0103] where V is a high-dimensional ball with radius r (in this paper, V is two-dimensional) and B is the number of FP pixels in V. w is the center of the high-dimensional ball and is one of the eigenvectors mentioned above.

[0104] The Mean Shift vector always moves in the direction of the maximum increase in the potential density function. And when the kernel function is introduced into the Mean Shift vector, which determines the weights of neighboring pixels, the Mean Shift vector is redefined as follows:

[0105]

[0106] where k(·) is the kernel contour, w s is the spatial coordinate, D is the normalization constant, h s is the kernel bandwidth used to control the kernel size. s The larger the value, the more adjacent pixels are used to estimate the density. We use h s =50 to identify the cluster centers of all feature vectors.

[0107] To summarize briefly, the spatial coordinates of the FP pixels are input into the Mean Shift algorithm. First, a high-density FP pixel is randomly selected from W as the initial cluster center, and then it is iteratively updated using the iterative formula until convergence (e.g., For example, 0.001), and then is the cluster center, h s The FP pixels contained in the bandwidth area are removed from W to obtain W′, and then a FP pixel is randomly selected from W′ as the new initial cluster center until each high-density FP pixel finds its corresponding cluster center. Figure 3 As shown in (g), each high-density FP pixel is connected to the corresponding cluster center. Figure 3 In (g), we zoom in on one of the clusters, such as Figure 3 As shown in the lower right corner of (g), the circle indicates the cluster center.

[0108] In another embodiment, the step S1015 further includes:

[0109] All FP pixels contained in each cluster are used as a point set to construct the convex hull, thereby obtaining the affected FP area in MGSR.

[0110] For this embodiment, once the clustering relationship of high-density FP pixels is determined, the convex hull is constructed (e.g. Figure 3 (h)). These convex hulls are represented by C = c1, c2, ..., c K , which is used to explain the affected area densely filled with FP pixels. More specifically, we use all FP pixels contained in each cluster as a point set to construct the convex hull.

[0111] In another embodiment, the modified logistic function is:

[0112]

[0113] For this example, the logistic function was originally proposed by Pierre Verhulst introduced it as a common S-shaped curve (an example is Figure 3 (i)), is defined as follows:

[0114]

[0115] Where x0 is the midpoint of the S-shaped curve, L is the maximum value of the curve, and k is the logistic growth rate or the steepness of the curve. The standard logistic function is a logical function with parameters k=1, x0=0, and L=1, and its expression is as follows:

[0116]

[0117] In the standard logistic function, f(x) increases gradually as x increases, and it grows exponentially at the beginning. When f(x) gradually approaches 1, the slope of the function gradually decreases. The characteristics of the standard logistic function are consistent with our original intention that the larger the convex hull area, the greater the penalty for it.

[0118] However, it is worth noting that there are negative values ​​in the independent variables of the standard logistic function, but the area of ​​the convex hull in this study is non-negative. Therefore, we propose to modify the standard logistic function and then use it as the penalty function for the final calculation of MAFaR, which is defined as follows:

[0119]

[0120] Where x∈[0, 20], f(x)∈[0, 1]. Because we are studying the full-frame tissue pathology image, the range of the convex hull area obtained by module 1 varies greatly, ranging from 1 pixel to tens of thousands of pixels. In our dataset, although the range of values ​​of the convex hull area contained in an MGSR varies greatly, the number of convex hulls with an area greater than 5000 only accounts for a small proportion. Therefore, when the area of ​​the convex hull is greater than 5000, we give the convex hull the maximum penalty, that is, the maximum value of f(x) is 1. Therefore, we set the convex hull area A i We scale it down by a factor of 250 and map it to [0, 20] to ensure that when A i =5000 when f(A i )=1. Therefore, for a segmented image MAFaR is defined as follows:

[0121]

[0122] Where A is the area of ​​the convex hull and K is the total number of convex hulls in a MGSR image.

[0123] In another embodiment, the step S200 further includes:

[0124] The MAFaR is used to evaluate the performance of the digital full-frame tissue pathology image segmentation model.

[0125] For this embodiment, the experimental data used are digital full-width histopathological images of oral squamous epithelial cells, and the experimental data are from two data centers (institutions): 1) San Francisco Veterans Administration Medical Center (San FranciscoVA Medical Center, SFVA); 2) Vanderbilt University Medical Center (Vanderbilt University Medical Center, VUMC). Patients with oral squamous cell carcinoma were identified from the radiation oncology and otolaryngology head and neck surgery clinician database approved by the Office of Human Research Protection.

[0126] Two independent and well-characterized OCSCC datasets consisting of formalin-fixed paraffin-embedded (FFPE) H&E (hematoxylin-eosin) stained whole glass slide images (WSI) were used in this study, with n = 22 images. The WSI images used for segmentation were digitally scanned using an AperioScanscope XT digital scanner at 40x magnification with a resolution of 0.25 μm / pixel per pixel. The image quality of the WSI images used for segmentation was checked by HistoQC to ensure that the images used had few bubbles, no overlap of staining with tissue, no more than 10% of the contaminated area and not within the region of interest; no more than 20% of the blurred area affected by the scan; the contrast, brightness, color and tone of the image were normal and not much different from other images; the thickness of the slices was normal, the over-thick area did not exceed 10% of the whole image and was not in the center of the region of interest. The epithelial area was segmented from the WSI using a UNet-based deep learning framework to obtain the MGSR image, and the pathologists annotated the WSI images using annotation software (such as QuPath) to obtain the labels, i.e., the GT image. It is worth noting that since the original images (40x magnification) are too large, for ease of observation, both MGSR and GT images were analyzed at 2.5x magnification.

[0127] Since the MAFaR indicator targets the affected FP area in MGSR (denoted as Affected_FP), and as mentioned in S1011 above, the existing indicators have already taken B into account. FP The FP region (denoted as Large_FP) containing a large number of pixels in the image has an impact on the segmentation model. Therefore, before calculating MAFaR, we first calculate the FP region from B FP Large_FP is removed from MGSR, that is, MAFaR only measures the impact of Affected_FP on model performance, but the impact of Large_FP on the model is not reflected in MAFaR. In other words, Affected_FP and Large_FP in MGSR both affect the performance of the segmentation model. Therefore, to evaluate the impact of the FP area in an MGSR image on the segmentation performance of the model, not only the Affected_FP contained therein should be considered, but also Large_FP. In other words, MAFaR needs to be coordinated with existing indicators to jointly evaluate the performance of model segmentation, that is, use MAFaR to measure Affected_FP, and then use existing indicators to measure Large_FP.

[0128] like Figure 4 (a) to Figure 4(d) shows three representative pathological images evaluated and compared using MAFaR and existing indicators. The first column represents the annotated image GT ( Figure 4 (a)), the second column shows the corresponding model segmentation result image MGSR ( Figure 4 (b)), the third column represents the Affected_FP ( Figure 4 (c)), the fourth column shows the MGSR image after removing Affected_FP from MGSR ( Figure 4 (d), denoted as MGSR'), which includes Large_FP, in other words, Figure 4 (c) and (d) together constitute Figure 4 (b). To evaluate the model segmentation results, we can directly evaluate the MGSR image, or evaluate Affected_FP and MGSR' separately. Comparing the GT and MGSR images, we can see that MGSR has segmented most of the foreground areas in GT, and also segmented many more FP areas. In this case, existing indicators often ignore the impact of Affected_FP on model performance, and mistakenly believe that the segmentation results are particularly good.

[0129] Here we use two existing FP-related indicators to compare with MAFaR to illustrate the correctness and rationality of the MAFaR indicator. The evaluation results of the indicators are shown in Table 2, where each pathological image is analyzed at a magnification of 2.5x. The numbers in the brackets in the first column of Table 2, such as (7046*5142), represent the size of Image 1 at a magnification of 2.5x. The following columns in Table 2 represent the results of evaluating MGSR image segmentation using different indicators or combinations of indicators. The two existing indicators selected are: FPR and VOE.

[0130] Taking the comparison between FPR and MAFaR as an example, we first use the FPR indicator to directly evaluate MGSR, and the result is FPR. Then we use FPR and MAFaR to evaluate the segmentation model together (MAFaR evaluates Affected_FP, and the result is MAFaR, and the FPR indicator evaluates MGSR', and the result is FPR'), and the result is MAFaR+FPR'. Comparing FPR with MAFaR+FPR', VOE is similar.

[0131] As can be seen from Table 2, the difference between using MAFaR and FPR or VOE to evaluate the segmented image and using FPR and VOE alone is still very obvious. For example, in Image 1, the difference between FPR (0.0096) and MAFaR+FPR' (0.1809) is still very large. Figure 4 (a) to Figure 4 (d) From the third column, we can see that Affected_FP is relatively scattered, but relatively dense locally. And because the WSI image itself is large in size, Affected_FP actually has a great impact on the performance of the segmentation model. However, the separate FPR and VOE indicators do not accurately reflect the impact of Affected_FP on the segmentation model. The specific explanation is as follows:

[0132] 1) Comparison with FPR: FPR only measures the impact of FP regions (including Affected_FP and Large_FP) on the segmentation model. However, when FPR is used to directly measure MGSR, it almost ignores the penalty for FP regions (e.g., the FPR value of Image 1 is only 0.0096). Even in Image 3, which has the worst segmentation result, the FPR value is very small (0.0647), which shows that FPR ignores the impact of Affected_FP. However, when MAFaR and FPR are used to measure MGSR images together, MAFaR measures the Affected_FP part reasonably, thus correctly reflecting the segmentation performance of the model.

[0133] 2) Comparison with VOE indicator: The VOE indicator can simultaneously measure the impact of FP and FN (false negative) on the segmentation model, but it only considers the error rate of the segmented image at the pixel level. When VOE measures FP, it does not consider the impact of the FP distribution area on the model performance.

[0134] When VOE is used directly to evaluate MGSR, such as Image 1, VOE(MGSR) = 0.2545, VOE(MGSR') = 0.2211, which shows that the penalty of VOE on Affected_FP (denoted as VOEA) is only 0.0334 (0.2545-0.2211). Similarly, in Images 2 and 3, VOEA is 0.0605 (0.1026-0.0421) and 0.0441 (0.2848-0.2407), respectively. It can be seen that the penalty of VOE on Affected_FP is very small. And from Figure 4 (a) to Figure 4 (d) As can be seen from the third column, the segmentation result of Image 3 contains the most Affected_FP, and the distribution is more dispersed, but the VOEA value of Image 3 is smaller than that of Image 2. Therefore, when the VOE indicator is directly used for evaluation, the penalty of the VOE indicator on Affected_FP is generally small, and it does not correctly reflect the impact of Affected_FP.

[0135] When MAFaR and VOE are used to measure MGSR together, MAFaR measures the Affected_FP part reasonably, thus correctly reflecting the segmentation performance of the model. It is worth noting that compared with Image 2, the MAFaR+VOE' of Image 1 is larger. This is because the VOE indicator measures both FP and FN. Figure 4 (a) to Figure 4 As can be seen in (d), the MGSR of Image 1 contains more FN, so MAFaR+VOE' is larger, but this does not affect the correct and reasonable measurement of Affected_FP by MAFaR.

[0136] In summary, for the evaluation of WST image segmentation models, MAFaR's metric of Affected_FP is better, more intuitive, and more realistic than FPR and VOE.

[0137] In summary, we proposed the MAFaR indicator, which uses the WST image as a basis to measure the dense false positive pixel area generated by the segmentation model. Compared with the traditional segmentation indicators that only consider pixel-level differences, we pay more attention to the impact of the FP distribution area on the segmentation model, which can more realistically reflect the segmentation performance of the training model.

[0138]

[0139] Table 2

[0140] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A method for evaluating a digital full-frame tissue pathology image segmentation model, comprising the following steps: S100: Propose the MAFaR metric, which is used to measure the affected false positive FP area generated by the segmentation algorithm of digital full-frame histopathology images; S200: using the MAFaR to complete the performance evaluation of the digital full-frame tissue pathology image segmentation model; in, MAFaR is defined as follows: ， in, is the area of ​​the convex hull, , is the total number of convex hulls in a MGSR image; when When it is greater than 5000, is equal to 1; The MAFaR calculation comprises the following steps: S101: Estimate the affected false positive FP area in the model segmentation result MGSR; S102: Calculate the penalty of the affected false positive FP region; Wherein, the step S101 further includes: S1011: Remove large false positive FP regions; S1012: Estimating the density of FP pixels; S1013: thresholding the FP density map to obtain a high-density FP map; S1014: clustering pixel regions of adjacent high-density FP images; S1015: Convex hull construction, using different convex hulls to represent the affected FP area in MGSR; Wherein, the step S102 further includes: A modified logistic function is used as the penalty function to penalize the area of ​​the affected FP region; Wherein, the step S1011 further includes: S10111: Get the FP region in MGSR, denoted as B FP ; S10112: From the B FP The threshold method is used to remove the large FP area, and the image with the large FP area removed is obtained, which is represented by B fFP ; Wherein, the step S1015 further includes: All FP pixels contained in each cluster are used as a point set to construct the convex hull, thereby obtaining the affected FP area in MGSR; Wherein, the modified logistic function is: ， in, , .

2. According to the method of claim 1, the step S1012 further comprises: Use kernel density estimation to estimate B fFP The density of FP pixels in the image.

3. According to the method of claim 1, the step S1013 further comprises: Select a threshold of 50.

4. According to the method of claim 1, the step S1014 further comprises: The Mean Shift clustering algorithm is used to cluster the adjacent pixel areas of the high-density FP map.

Images