A method for detecting cells in pathological images based on crowd-sourced annotation and active learning

Abstract

The application discloses a kind of based on crowdsourcing annotation and active learning's pathological image cell detection method, obtain crowdsourcing annotation data;Construct target detection model;Based on crowdsourcing annotation data, image sample information amount, the reliability of labeler and the cost of labeling are comprehensively measured, select several comprehensive measurement optimal labeler-image sample pair, to corresponding image is labeled by corresponding labeler to the labeled image as training set, the target detection model is trained, and the trained model is obtained;The image to be detected is input into the trained model, and the pathological image cell detection result is obtained.The reliability of labeler to image annotation and the cost of labeling are considered comprehensively, to avoid the problem of pursuing low cost sacrificing model accuracy and pursuing high model accuracy cost, increase the number of high-quality sample data, effectively improve model accuracy under lower cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of bioinformatics, in particular to a pathological image cell detection method based on crowd-sourced annotation and active learning. BACKGROUND

[0002] Detecting cells or nuclei from histopathology images is a prerequisite for in-depth analysis of human cancer progression. Due to the differences in shape and appearance of different types of cells, accurate detection of cells from pathological images is still a challenge. Deep learning technology has achieved great success in the field of computer vision, which has also promoted great interest in detecting cells from H&E stained pathological analysis. However, designing an effective deep learning-based cell detection algorithm requires a large amount of labeled data, which increases the labeling burden of experts. In order to reduce the labeling cost, active learning algorithms are applied in this field.

[0003] Although the cell detection algorithm based on active learning can make the process of labeling new data more efficient, it is generally believed that the labeling ability of different annotators is the same, which ignores the different expertise of different annotators. In fact, for the cell detection task, pathologists are more likely to make correct decisions than non-pathologists. However, the number of professional pathologists is small, the annotation cost is high, and the labeled data samples for cell detection by professional pathologists are small, although the sample data quality is high, but the insufficient data quantity of sample data will still affect the accuracy of the cell detection algorithm model constructed; However, it is not feasible to assign all unlabeled images to pathologists for annotation, which brings great cost and cost. Non-pathologists are more numerous and have lower annotation costs, so there are more data samples based on non-pathologist annotations, but it is difficult for non-pathologists to label cells with complex morphology, so even if the data quantity of sample data is greatly increased, the accuracy of the cell detection algorithm model constructed cannot be guaranteed. SUMMARY

[0004] The present application provides a pathological image cell detection method based on crowd-sourced annotation and active learning with high accuracy and low cost.

[0005] Technical scheme: In order to solve the above problems, the present application adopts a pathological image cell detection method based on crowd-sourced annotation and active learning, comprising the following steps:

[0006] (1) Obtain crowd-sourced annotation data; the crowd-sourced annotation data includes image samples, annotators and labels given by the annotators to the image samples;

[0007] (2) Construct a target detection model, the input of the target detection model is a labeled pathological image, and the output is a cell detection result of the pathological image;

[0008] (3) training a target detection model based on the crowd-sourced annotation data to obtain a trained model; the selecting a training set based on the crowd-sourced annotation data specifically comprises:

[0009] (3.1) performing preliminary processing on the image samples to obtain the information amount in the images;

[0010] (3.2) calculating the reliability of each annotator in annotating each image based on the crowd-sourced annotation data;

[0011] (3.3) obtaining the annotation cost of each annotator based on the reliability of each annotator in annotating each image, the higher the reliability of the annotation, the higher the annotation cost;

[0012] (3.4) obtaining the comprehensive measurement of each annotator in annotating each image based on the obtained information amount, reliability and annotation cost;

[0013] (3.5) selecting a number of annotator-image sample pairs with optimal comprehensive measurement, obtaining the annotation images of the corresponding images annotated by the corresponding annotators, and taking the obtained annotation images as the training set;

[0014] (4) inputting the to-be-detected image into the trained model to obtain the cell detection result of the pathological image.

[0015] Further, the step (3.5) of selecting the annotator-image sample pair with optimal comprehensive measurement is performed before the annotator reliability judgment: setting an expected threshold θ; the annotators are classified into experts and non-experts according to social identity; the average reliability of the image samples annotated by the experts is calculated based on the crowd-sourced annotation data and the average reliability of the image samples annotated by the non-experts is calculated based on the crowd-sourced annotation data

[0016] the judgment parameter γ r is calculated:

[0017]

[0018] When γ r > θ, only the non-experts are selected as the annotators to select the annotator-image sample pairs, and the annotation images annotated by the non-experts are obtained;

[0019] When γ r < θ, part of the image samples are randomly selected to select the annotator-image sample pairs with the experts as the annotators, and the annotation images annotated by the experts are obtained.

[0020] Further, the step (3.1) inputs the image sample into the trained target detection model to obtain a detection frame and a detection category of the image sample, and a calculation formula of the information amount I(u) of the image sample u is:

[0021]

[0022] wherein B represents all detection frames of the image sample u; G represents a union of all detection categories; represents a probability that the hth detection frame of the image sample u is a detection category g.

[0023] A reliable degree R of each annotator for each image i (u j ) is calculated according to a calculation formula:

[0024] R i (u j )=q i (u j )·(1-m i )·o i

[0025] wherein q i (u j ) represents an annotation accuracy of each annotator p i annotating the image u j , o i represents an annotation average IoU capability of each annotator p i , and m i represents an average cell loss rate of each annotator p i .

[0026] A calculation formula of the annotation accuracy q i (u j ) of each annotator p i annotating the image u j is:

[0027]

[0028] wherein S(·) represents a cosine similarity function; represents a cell composition vector of the image u j ; represents a cell composition vector of the nth, n=1,…,N labeled image ; represents an annotation accuracy of each annotator p i annotating the nth, n=1,…,N labeled image .

[0029] A calculation formula of the annotation accuracy q iThe annotation accuracy of the annotated image is calculated according to each annotator p i The annotation accuracy of each cell type q i (g t ) is calculated, and the accuracy q i (g t ) is calculated by the following formula:

[0030]

[0031] Wherein, I[·] represents the indicator function, y i (g t ) represents the annotation of the annotator p i For the type annotation of the cell whose true class is g t , g t represents the tth cell type, M t represents the total number of the tth cell type g t .

[0032] Further, the annotation cost C i of the annotator in step (3.3) is calculated by the following formula:

[0033]

[0034] Wherein, T is the total number of cell types.

[0035] Further, the comprehensive measure Z(u i , p j ) of the image u j annotated by the annotator p i in step (3.4) is calculated by the following formula:

[0036]

[0037] The annotator-image sample pair (u * , p * ) with the optimal comprehensive measure is argmax Z(u j , p i ).

[0038] Beneficial effects: Compared with the prior art, the present application has the significant advantages that the reliability of the annotator in image annotation and the annotation cost are comprehensively considered, the problems of sacrificing model accuracy for low cost and spending high cost for high model accuracy are avoided, the number of high-quality sample data is increased, and the model accuracy is effectively improved at a lower cost. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The figure is a framework diagram of the pathological image cell detection method of the present application. DETAILED DESCRIPTION

[0040] As Figure 1 shown, a kind of pathological image cell detection method based on crowd annotation and active learning in the embodiment includes the following steps:

[0041] (1) obtain crowd annotation data;Crowd annotation data includes image samples, labelers and labels given by labelers to image samples;

[0042] (2) based on Mask R-CNN, construct target detection model, the input of target detection model is labeled pathological image, and the output is the cell detection result of pathological image;According to the existing pathological image with real label, a preliminary target detection model Mask R-CNN is trained as a base model, which is used for the processing of image samples in the first iteration;The construction of model refers to prior art: He K, Gkioxari G, P Dollar, et al. Mask R-CNN [J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017.,The specific content will not be repeated here.

[0043] (3) based on crowd annotation data selection training set, the target detection model is trained, and the trained model is obtained;The training set based on crowd annotation data selection specifically includes:

[0044] (3.1) according to the trained target detection model, the image sample is preliminarily processed, and in the first iteration, the trained base model is used to preliminarily process the image sample to obtain the information amount in the image.

[0045] Specifically, the target detection model is divided into two parts of detection frame regression and detection class identification. For the calculation of image information amount, it can be considered that the more unknown categories contained in an image, the greater the inaccuracy of the model prediction, and the more information amount contained in the image. The information amount of image sample u is calculated using information entropy:

[0046]

[0047] Wherein, B represents all detection frames predicted by the detection model on the image, G represents the union of all detection categories, Represent the probability of the hth detection frame detected by the detection model as category g. Information entropy can be referred to in prior art: Lee J. Maximum Entropy Sampling [M]. John Wiley & Sons, Ltd, 2006.,Will not be repeated here.

[0048] (3.2) Calculate the reliability of each annotator's annotation of each image based on the crowdsourced annotation data; specifically including:

[0049] (3.2.1) Calculate p for each annotator based on the crowdsourced annotation data. i Average IoU capability of annotation i and average cell loss rate m i ;

[0050] (3.2.2) Calculate p for each annotator i Label each category g t accuracy q i (g t ):

[0051]

[0052] Where I[·] represents the indicator function, y i (g t ) represents the annotator p i For the true category g t Cell type labeling, M t Indicates category g t The total number.

[0053] (3.2.3) According to q i (g t )Calculate p for each annotator i Image in the annotation pool k The accuracy of the annotation q i (l k ).

[0054] (3.2.4) Using the base model, predict the cell composition of unlabeled images. For each unlabeled image u j Let its cellular composition be denoted as a vector. in Representative image u j Medium cell type g t It consists of 7 cell types.

[0055] (3.2.5) For each image of the labeled pool species l k Calculate its cell composition vector

[0056] (3.2.6) Calculate u for each unlabeled image j and each labeled image l k The cosine similarity between them, sum to u j The five most similar images k , recorded as The corresponding cellular composition is denoted as

[0057] (3.2.7) Calculate p for each annotator i Annotated image u j The accuracy of the annotation q i (u j ):

[0058]

[0059] Where S(·) represents the cosine similarity function. Cosine similarity can be found in existing technology: Xia P, Zhang L, Li F. Learning similarity with cosine similarity ensemble[J]. Information Sciences, 2015, 307:39-52.

[0060] (3.2.8) Calculate p for each annotator i Mark u j The reliability of the labeling R i (u j ):

[0061] R i (u j )=q i (u j )·(1-m i )·o i

[0062] (3.3) The annotation cost for each annotator is obtained based on the reliability of each annotator's annotation of each image. Specifically, it is assumed that if annotator p i Higher quality annotations require higher annotation costs; calculate the annotator p. i The annotation cost C i :

[0063]

[0064] (3.4) Based on the amount of information obtained, the reliability, and the annotation cost, a comprehensive metric is obtained for each annotator's annotation of each image; annotator p i Annotated image u j The comprehensive metric Z(u) j ,p i The calculation formula is:

[0065]

[0066] Determine the reliability of the annotators:

[0067] The annotators are classified into experts and non-experts according to social identity. Non-experts, such as medical students, are very willing to perform image annotation, and the cost of these non-experts is very small, close to 0, and their comprehensive metrics will tend to infinity, so these non-experts will be preferred when selecting the optimal annotator-sample pair. However, the annotation reliability of these non-experts is often low. In order to avoid the cost C i occupying the dominant factor of the comprehensive metric, a desired threshold θ is set, and the average reliability of the experts in each round is denoted as the average reliability of the non-experts is denoted as The ratio γ of the average ability of the non-experts to the average ability of the experts in each round is calculated r :

[0068]

[0069] wherein r represents the number of model training rounds.

[0070] When γ r > θ, only non-experts are asked to annotate images to further reduce the annotation cost. When γ r < θ, it means that the average level of the non-experts does not meet the expectation, and at this time, part of the images (randomly selected 10% of the total number of images added to the model training) are asked to be annotated by experts to ensure the detection accuracy of the model.

[0071] (3.5) Select a number of annotator-image sample pairs (u * , p * ) = argmax Z(u j , p i ) with the optimal comprehensive metric, obtain the annotation images corresponding to the corresponding annotators for the corresponding images, and use the obtained annotation images as the training set;

[0072] (4) Input the to-be-detected image into the trained model to obtain the cell detection result of the pathological image.

Claims

1. A method for cell detection in pathological images based on crowdsourced annotation and active learning, characterized in that, Includes the following steps: (1) Obtain crowdsourced annotation data; crowdsourced annotation data includes image samples, annotations, and the labels given by annotationers to the image samples; (2) Construct a target detection model. The input of the target detection model is the labeled pathological image, and the output is the cell detection result of the pathological image. (3) The target detection model is trained using a training set selected based on crowdsourced annotation data to obtain a trained model; the selection of the training set based on crowdsourced annotation data specifically includes: (3.1) Perform preliminary processing on the image samples to obtain the amount of information in the images; input the image samples into the trained target detection model to obtain the detection boxes and detection categories of the image samples. Information content The calculation formula is: in, Represents image samples All detection boxes; This represents the collection of all detection categories; Represents image samples The Middle Each detection box represents a detection category. The probability of; (3.2) Calculate the reliability of each annotator's annotation of each image based on the crowdsourced annotation data; the reliability of each annotator's annotation of each image. The calculation formula is: in, Indicate each annotator Annotated images The accuracy of the annotation, Indicate each annotator The average IoU capability of the annotation Indicate each annotator The average cell loss rate; Each annotator Annotated images labeling accuracy The calculation formula is: in, Represents the cosine similarity function; Representing an image Cell composition vector; Indicates the first , Zhang's labeled image Cell composition vector; Indicate each annotator Mark the first , Zhang's labeled image The accuracy of the annotation; (3.3) The annotation cost for each annotator is obtained based on the reliability of each annotator's annotation of each image. The higher the reliability of the annotator's annotation, the higher the annotation cost. The annotation cost for the annotator The calculation formula is: in, The total number of cell types; Each annotator The accuracy of annotations on labeled images depends on each annotator. Accuracy of labeling each cell type The accuracy was calculated. The calculation formula is: in, Represents an indicator function. Representative of the annotator For the true category is Cell type labeling, Indicates the first Cell types, Indicates belonging to the first Cell types The total number of cells; (3.4) Based on the amount of information obtained, the reliability, and the annotation cost, obtain a comprehensive metric for each annotator's annotation of each image; Annotator Annotated images Comprehensive measurement The calculation formula is: (3.5) Select several annotator-image sample pairs with the best comprehensive metric, obtain the annotated images of the corresponding annotators for the corresponding images, and use the obtained annotated images as the training set; (4) Input the image to be detected into the trained model to obtain the cell detection results of the pathological image.

2. The method for detecting cells in pathological images according to claim 1, characterized in that, Step (3.5) selects the annotator-image sample with the best comprehensive metric to assess the reliability of the annotators previously used: set the expected threshold. ; The annotators are categorized based on their social identity, including experts and non-experts; the average reliability of expert-annotated image samples is calculated based on crowdsourced annotation data. Average reliability of non-expert-annotated image samples ; Calculate the judgment parameters : when At that time, only non-experts were used as annotators to select annotator-image sample pairs, and annotated images were obtained by non-experts. when At that time, a portion of the images in the image sample are randomly selected, and experts are used as annotators to select annotator-image sample pairs to obtain annotated images annotated by experts.

3. The method for detecting cells in pathological images according to claim 1, characterized in that, In step (2), an object detection model is constructed based on Mask R-CNN.

4. The method for detecting cells in pathological images according to claim 3, characterized in that, The best annotator-image sample pair based on comprehensive metrics .

Images