An active learning sample labeling method based on latent layer representation

Abstract

The application relates to an active learning sample labeling method based on a latent layer representation, and the sample labeling method comprises the following steps: S1, a feature extraction step: simulating the extraction of the representation of the latent layer from a pathological image through an automatic encoder; S2, a discrimination step: taking the pathological image as the input of a discriminator D, training a two-classification model, distinguishing the sample data set with labels and without labels through the discriminator D, and obtaining a probability value; S3, a sample labeling step: setting a sample selection strategy ALHS, selecting unlabeled samples for labeling through the sample selection strategy ALHS according to the output probability value distribution of the discriminator, and realizing the automatic labeling of the pathological data set. The application can greatly reduce the heavy burden of labeling the data set by pathologists, can relieve the sample redundancy problem, improve the quality of sample selection, greatly save the sample labeling cost, has strong universality, and can be applied to the occasions of class imbalance and noise samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a method for labeling samples based on latent layer representation in active learning. BACKGROUND

[0002] Accurate and timely cancer diagnosis is the key to treat each cancer patient, and the cancer diagnosis is generally determined by human pathologists through observing the stained specimens on patient pathological slides under a microscope. Each pathological slide image (WSI) contains rich phenotypic information, which is crucial for the diagnosis and classification of cancer. However, a slide image can contain tens of thousands of image blocks, and it will take a lot of time and effort to label the images in a large WSI dataset only by visual inspection of pathologists, which is extremely costly. And most countries in the world are facing the problem of serious shortage of pathologists, which greatly hinders the timely diagnosis and effective treatment of cancer patients. Further development of artificial intelligence technology can improve the current plight of the lack of pathologists. Machine learning, especially deep learning technology, has made extraordinary achievements in many fields, but its success largely depends on large-scale labeled training datasets.

[0003] For pathological diagnosis, the first problem to be solved is the serious shortage of labeled samples in the WSI dataset, i.e., the number of samples labeled by experts in the WSI dataset is limited. Because the labeling work of WSI dataset image samples is a labor-intensive manual process, it requires a large amount of time and effort of field experts, i.e., pathologists, and the number of pathologists is limited and the working time is limited, so the cost of completely labeling the samples in the WSI dataset is very high. As a branch of machine learning, active learning attempts to achieve the best performance of the model with as few high-quality sample labels as possible, i.e., for the WSI dataset, due to the redundancy and poor quality of the data samples, it is not necessary to label all the samples, but to select some high-quality samples to train the model through active learning technology, so as to obtain the performance comparable to the model trained by the fully labeled sample dataset. The prior art proposes an active learning method using gene expression data, which uses a rough fuzzy classifier to classify cancer samples, and also proposes an active learning framework that uses conditional generative adversarial networks (cGANs) to select samples with the most information, and generates real chest X-ray images with different disease characteristics. However, there are very few active learning methods used for pathological image labeling.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art, and provides a latent layer representation-based active learning sample labeling method, which effectively avoids the sample redundancy problem by using active learning for pathological image labeling, and improves the quality of the training sample set.

[0006] The object of the present application is achieved by the following technical solutions: a latent layer representation-based active learning sample labeling method, the sample labeling method comprising:

[0007] S1, a feature extraction step: a latent layer representation is extracted from a pathological image by an autoencoder;

[0008] S2, a discrimination step: a pathological image is taken as an input of a discriminator D, a binary classification model is trained, and a probability value is obtained by distinguishing the labeled and unlabeled sample data sets through the discriminator D;

[0009] S3, a sample labeling step: a sample selection strategy ALHS is set, unlabeled samples are selected for labeling through the sample selection strategy ALHS according to the output probability value distribution of the discriminator, and automatic labeling of the pathological data set is realized.

[0010] The step S1 specifically comprises the following contents:

[0011] The latent layer representation learning is performed by using the autoencoder, the medical data set sample is embedded into a low-dimensional space, and is the mapping from the original input space to the learned latent layer representation, that is, z=Ψ(x), the decoder reconstructs the input image according to the low-dimensional representation z, and is the mapping from the latent layer feature representation to the reconstructed image, that is, The objective function of the autoencoder is the MAE loss function, which is represented as x represents the input of the autoencoder, that is, the original image, represents the output of the autoencoder, that is, the reconstructed image, and N represents the number of images;

[0012] The autoencoder model uses the stochastic gradient descent method to optimize the objective function during the training process and saves the model with the minimum loss, and after the autoencoder model is trained, the latent layer representation information is provided to the discriminator D to distinguish the labeled and unlabeled data sets while realizing image reconstruction.

[0013] The step S2 specifically comprises the following contents:

[0014] The labeled sample set is denoted as The unlabeled sample set is denoted as All pathological images are taken as inputs of the discriminator D, the labels of the labeled samples are set to 1, and the labels of the unlabeled samples are set to 0;

[0015] Let D be the discriminator is the input space, is the latent representation of the image learned by the encoder, is the label space, where l is the label of the labeled sample set, i.e., the set of all 1s, and u is the label of the unlabeled sample set, i.e., the set of all 0s, so the objective function of the discriminator D is the cross-entropy loss function: where t i is the label of the sample, 1 or 0, p i is the predicted probability that the model predicts the label of the sample to be t i , in each iteration of the active learning process, the objective function is optimized using the stochastic gradient descent method, and the model with the minimum training loss is saved to obtain the best model

[0016] The step S3 specifically includes the following contents:

[0017] According to the output probability value of the discrimination step, the data set is sorted and divided into K parts in order, and a sample is selected from each part of the data set by random sampling method or minimum confidence method or marginal sampling method or entropy method to be labeled by a pathologist.

[0018] The minimum confidence method specifically includes:

[0019] Select the sample with the smallest maximum probability for labeling, where is the class with the maximum probability predicted by the model for input x, and θ represents the parameter set of the trained machine learning model.

[0020] The marginal sampling method specifically includes: selecting the sample with the smallest difference between the maximum probability and the second maximum probability by the formula where and respectively represent the maximum possible class and the second largest possible class predicted by the model for the sample point x.

[0021] The entropy method specifically includes: using the entropy of the predicted probability as the uncertainty measure as an uncertainty sampling strategy to select the sample by the formula where S is the number of classes of the sample set, and p i (x) represents the probability that the sample x is predicted to be the i-th class.

[0022] ​​The application has the advantages that the active learning sample labeling method based on the latent layer representation can greatly reduce the heavy burden of pathologists labeling the data set, can alleviate the sample redundancy problem, improve the quality of sample selection, greatly save the sample labeling cost, and has strong universality and can be applied to the occasions of class imbalance and noise samples. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 An active learning sample labeling process diagram of the application;

[0024] Figure 2 A network structure diagram of an autoencoder;

[0025] Figure 3 A network structure diagram of a discriminator;

[0026] Figure 4 An ALHS sample selection strategy diagram;

[0027] Figure 5 An average accuracy diagram on the Kather's data set;

[0028] Figure 6 A confusion matrix diagram of 25% sample labeling and 100% sample labeling on the Kather's data set;

[0029] Figure 7 An average accuracy diagram on the BreakHis data set;

[0030] Figure 8 A confusion matrix diagram of 45% sample labeling and 100% sample labeling on the BreakHis data set;

[0031] Figure 9 A latent layer representation diagram of the Kather's data set and the BreakHis data set;

[0032] Figure 10 A sample selection visualization diagram of four methods on the Kather's data set;

[0033] Figure 11 A sample selection visualization diagram of four methods on the BreakHis data set. DETAILED DESCRIPTION

[0034] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in combination with the drawings of the present application is not intended to limit the protection scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application. The present application will be further described below in combination with the drawings.

[0035] As shown in Figure 1 , the present application specifically relates to a learning sample labeling method based on latent layer representation. The method first learns the latent layer representation of a pathology image through an automatic encoder, and trains a binary classification model to distinguish the latent layer representation of labeled and unlabeled samples, which serves as the basis for sample selection. Subsequently, based on the ALHS active learning sample selection mechanism, unlabeled samples are selected for labeling, thereby realizing sample selection and labeling of the pathology dataset, and alleviating the heavy burden of manual sample labeling by pathologists. The method specifically includes three stages of feature extraction, discrimination and sample labeling.

[0036] 1. Feature extraction stage: the latent layer representation is extracted from the pathology image using an automatic encoder (Auto-Encoder) model. The automatic encoder is an unsupervised deep learning algorithm that can learn from unlabeled data. The automatic encoder is used for representation learning, where the encoder learns to embed the medical dataset samples into a low-dimensional space, denoted as is the mapping from the original input space to the learned latent layer representation, i.e., z = Ψ(x), and the decoder reconstructs the input image from the low-dimensional representation z, denoted as is the mapping from the latent layer feature representation to the reconstructed image, i.e.,

[0037] The Auto-Encoder selects the Resnet50 network structure as the encoder to extract image features, and the network structure of the decoder is a transformation corresponding to the encoder, as shown in Figure 2 . The objective function of the automatic encoder is the MAE loss function, i.e., L1 loss:

[0038]

[0039] where x represents the input of the automatic encoder, i.e., the original image, where N denotes the number of images. During the training process, the objective function is optimized using stochastic gradient descent method and the model with the minimum loss is saved. For the trained auto-encoder model, while achieving excellent image reconstruction task, its latent representation information can provide good representation input for the discriminant model to distinguish between labeled and unlabeled datasets.

[0040] 2. Discrimination stage: After learning the latent representation and dimension reduction through the auto-encoder network, enter the discrimination stage, in which the labeled sample set is denoted as and the unlabeled sample set is denoted as All pathological images are input to the discriminator D, and the labels of the labeled samples are set to 1 and the labels of the unlabeled samples are set to 0. Let the discriminator be is the input space, and is the latent representation of the image learned by the encoder, is the label space, where l is the label of the labeled sample set, i.e. a set of all 1s, and u is the label of the unlabeled sample set, i.e. a set of all 0s. Thus, the objective function of the discriminator D is the cross-entropy loss function: where t i is the label of the sample, 1 or 0, and p i is the predicted probability that the model predicts the label of the sample to be t i .

[0041] In each iteration of the active learning process, the objective function is optimized using stochastic gradient descent method on , and the model with the minimum training loss is saved to obtain the best model

[0042] As shown in Figure 3 , it is the network structure diagram of the discriminator used in the present application. The discriminator is used to distinguish between labeled and unlabeled datasets, and a probability value output is obtained.

[0043] 3. Sample annotation stage: based on the latent representation information obtained by the auto-encoder, the discriminator is used to distinguish between labeled and unlabeled sample datasets, and a probability value output is obtained, and then the sample annotation stage is entered. By observing the output distribution of the discriminator, it is found that the input of the neural network approximation point has similar probability output, which may make the selected samples redundant, thereby leading to unsatisfactory performance of the final model. In order to overcome the problem of sample redundancy, representative samples are selected, and the present application considers the output distribution characteristics of the discriminator and designs a new sample selection strategy ALHS, which can avoid selecting samples with similar output under the discriminant model.

[0044] Further, the ALHS sample selection process is as shown in Figure 4 .

[0045] First, the data set is sorted according to the output probability value of the discrimination stage, and then it is divided into K parts in order. A sample is selected from each part of the data set and labeled by a pathologist. In order to better select valuable samples, four methods are adopted to select the sample of each part, namely random sampling (Random), least confidence method (Least Confidence), margin sampling (Margin Sampling) and entropy method (Entropy).

[0046] Among them, the least confidence method: select the sample with the smallest maximum probability for labeling, the formula is:

[0047]

[0048] Among them, is the class with the maximum probability predicted by the model for input x, and represents the parameter set of the trained machine learning model.

[0049] Margin sampling method: select sample data that is easy to be judged as one class or two classes, or sample data whose probability difference of being considered as two classes is extremely small. Margin sampling is to select the sample with the smallest difference between the maximum probability and the second largest probability predicted by the model, and the formula is:

[0050]

[0051] Among them, and respectively represent the maximum possible class and the second largest possible class predicted by the model for the sample point x.

[0052] Entropy method: The general uncertainty sampling strategy uses the entropy of the predicted probability as the uncertainty measure. Entropy is a measure in information theory, which represents the amount of information required to "encode" a distribution. Therefore, it is generally considered as a measure of uncertainty or impurity in machine learning, and the formula is:

[0053]

[0054] Among them, S is the number of classes of the sample set, and p i (x) represents the probability of sample x being predicted as the i-th class.

[0055] Each time a sample is selected from a class, one of the above methods is randomly selected with the same probability. According to such a sample selection method, the representativeness and uncertainty of the sample are considered when selecting the sample, and the redundancy problem of the sample can also be improved.

[0056] Experiments of the present application on different data sets are as follows:

[0057] Experiment 1: Multi-classification task experiment on Kather’s dataset

[0058] Kather’s dataset is a multi-class histological image dataset of colorectal tissue, containing eight classes of histological images: fat, debris, lymphoma, mucosa, complex, stroma, tumor and void, with a total of 5000 images, 625 images for each class. The multi-classification experiment is divided into two groups: one group has 10% of the initial sample data pool labeled, and the budget for each iteration is 5% of the training dataset; the other group has only 5% of the initial sample data pool labeled, and the budget for each iteration is 2.5% of the training dataset. The unlabeled data pool is the remaining part of the labeled training samples, and the samples are selected from it by experts for labeling. Once labeled, they will be added to the initial training set, and the model will be retrained on the new training set.

[0059] As shown in Figure 5 , the multi-classification performance of the method proposed in the present application compared with other methods on Kather’s dataset. Figure 5 (a) shows the experimental results of 10% initial labeled training set and 5% budget per iteration. The results show that the method proposed in the present application can obtain an average accuracy of 90.6% using only 25% of the labeled sample data, which is better than other methods. Figure 5 (b) shows the experimental results of 5% initial sample labeled training set and 2.5% budget per iteration. Similarly, the method proposed in the present application first exceeds 90% average accuracy, and can obtain an average accuracy of 91.2% by using a 25% sample labeled dataset, and an average accuracy of 93.1% by using a 60% sample labeled dataset. The accuracy of the dataset trained using 100% labeled samples is 93.3%, which shows that the method proposed in the present application only needs 60% of the labeled samples to achieve the performance obtained by the 100% labeled sample dataset, greatly reducing the labeling budget and reducing the workload of pathologists.

[0060] As shown in Figure 6 , the confusion matrix obtained after training the task model using 25% and 100% of the sample labeled dataset respectively, setting an accuracy of 90% as the target performance of the model on Kather’s dataset. As can be seen from the figure, the model trained using only 25% of the sample labeled dataset can achieve the target performance, and is very close to the final model performance trained using 100% of the sample labeled dataset. Compared with the average accuracy of more than 15% data, the method proposed in the present application is obviously better than the random sampling, entropy and discriminative active learning (DAL) methods.

[0061] Experiment 2: Binary classification task experiment on BreakHis dataset

[0062] The BreakHis dataset contains microscopic images of biopsies of benign and malignant breast tumors, totaling 7909 images. This dataset includes eight types of benign and malignant tumors. Figure 7 The figure shows the binary classification performance of the proposed method on the BreakHis dataset compared to other methods. Figure 7 (a) Experimental results with a training set of 10% initial sample annotations and 5% budget per iteration. The results show that the method proposed in this invention is the first to exceed an average accuracy of 95%, and that the method proposed in this invention, by selecting 45% of the samples for annotation, achieves an accuracy of 95.2%, which is higher than other methods; Figure 7 (b) Experimental results with a training set labeled with 5% of the initial samples and a budget of 2.5% for each iteration. Figure 8 As shown, the method proposed in this invention achieves an average accuracy of 95.5% using a dataset with 45% sample annotation and 96.3% using a dataset with 60% sample annotation. Even with a dataset using 100% sample annotation, the maximum achievable average accuracy is only 97.2%. Furthermore, the average accuracy on datasets with more than 15% annotation demonstrates that the method proposed in this invention significantly outperforms random sampling, entropy, and DAL methods.

[0063] Experiment 3: Visualization of Latent Layer Characterization and Sample Selection

[0064] To better visualize the generated results and subsequent selection results, this invention uses the t-SNE method to further reduce the feature dimensionality. t-SNE (t-distributed stochastic neighbor embedding) is an unsupervised nonlinear dimensionality reduction technique. Its main purpose is to construct a probability distribution from paired distances, such that larger distances correspond to smaller probabilities. This invention uses t-SNE to reduce the latent layer representation dimensionality to 2 and plots the results. Figure 9 The two-dimensional scatter plot shown. Figure 9 (a) and Figure 9 (b) represents the results of visualizing the latent representations of the Kather's and BreakHis datasets, respectively. From Figure 9 The results show that pathological images exhibit a certain degree of clustering under the latent representation of the Auto-Encoder model. That is, although the Auto-Encoder is a self-supervised model that can be trained without labels, the representation of the feature space it extracts is interpretable.

[0065] Figure 10 and Figure 11The sample selection results of applying the four methods on the Kather's dataset and the BreakHis dataset are shown respectively. From the selection results, the performance of the DAL method is not ideal. It almost selects all the samples in the lower right of the Kather dataset and the lower data points of the BreakHis dataset in one selection, and the selected samples have obvious redundancy phenomenon, resulting in poor results of the final target model. The second is the entropy method, which selects the upper samples in the Kather dataset and the left middle samples in the BreakHis dataset. The sample selection strategy ALHS of the method proposed in the application performs very well, and the selected samples almost cover all aspects of the feature map in the Kather dataset. In the BreakHis dataset, it can be found that the method proposed in the application selects a few sample points on the right in the sample selection, and almost all the sample points on the right represent the same category, so only a small amount of samples need to be selected. This shows that the method proposed in the application can use the expensive label for other more valuable sample instances. In addition, the method proposed in the application does not produce sample redundancy, which is one of the main reasons why the method proposed in the application performs well on the task model.

[0066] The method proposed in the application can greatly reduce the heavy burden of pathologists labeling the dataset, and on the two datasets of the experiment, it saves 75% and 55% of the sample labeling budget respectively while achieving the target performance of the two datasets. In addition, the method proposed in the application can be applied to datasets with class imbalance and noise samples, and can achieve the target performance with a small sample labeling cost.

[0067] The above only describes the preferred embodiments of the application, and it should be understood that the application is not limited to the forms disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above-mentioned teaching or related art or knowledge. The modifications and changes made by those skilled in the art without departing from the spirit and scope of the application shall be within the protection scope of the claims of the application.

Claims

1. A method for active learning sample labeling based on latent layer representation, characterized in that: The sample labeling method comprises: S1, a feature extraction step: extracting the representation of the latent layer from the pathological image through an autoencoder; S2, a discrimination step: taking the pathological image as the input of a discriminator D, training a two-classification model, and distinguishing the labeled and unlabeled sample datasets through the discriminator D to obtain a probability value; S3, a sample labeling step: setting a sample selection strategy ALHS, selecting unlabeled samples for labeling through the sample selection strategy ALHS according to the output probability value distribution of the discriminator, and realizing automatic labeling of the pathological dataset; The step S1 specifically comprises the following contents: The medical data set samples are embedded in a low-dimensional space by using automatic encoder for latent layer representation learning is a mapping from the original input space to the learned latent layer representation, i.e. , the decoder reconstructs the input image according to the low-dimensional representation z, and let is a mapping from the latent feature representation to the reconstructed image, i.e. , the objective function of the automatic encoder is the MAE loss function, which is represented as , represents the input of the automatic encoder, i.e. the original image, represents the output of the automatic encoder, i.e. the reconstructed image, and N represents the number of images. In the training process, the autoencoder model uses the stochastic gradient descent method to optimize the objective function and save the model with the minimum loss. After the autoencoder model is trained, the latent layer representation information provides good representation input for the discriminator D to distinguish the labeled and unlabeled datasets while realizing image reconstruction. The step S3 specifically comprises the following contents: According to the output probability value of the discrimination step, the dataset is sorted and divided into K parts in order. In each part of the dataset, a sample is selected by a pathologist for labeling through a random sampling method or a minimum confidence method or a marginal sampling method or an entropy method. 2.The method of claim 1, wherein: The step S2 specifically comprises the following contents: Let the labeled sample set be denoted as Let the unlabeled sample set be denoted as All the pathological images are input into the discriminator D, and the label of the labeled sample is set to 1, and the label of the unlabeled sample is set to 0. Let the discriminator , be the input space, the latent representation of the image learned by the encoder, be the label space, where be the labels of the labeled sample set, i.e., a set of all 1s, be the labels of the unlabeled sample set, i.e., a set of all 0s, then the objective function of the discriminator D is the cross-entropy loss function: where is the label of the sample, 1 or 0, is the predicted probability that the model predicts the label of the sample to be In each iteration of the active learning process, the objective function is optimized on using stochastic gradient descent, and the model with the smallest training loss is saved to obtain the best model .

3. The method of claim 1, wherein the method further comprises: The minimum confidence method specifically includes: marking the sample with the minimum maximum probability by selecting the sample with the minimum maximum probability, wherein, is the class with the maximum predicted probability, is the class with the maximum predicted probability, denotes a parameter set of the trained machine learning model.

4. The method of claim 1, wherein the method further comprises: The marginal sampling method specifically includes: selecting a sample with the smallest difference between the maximum probability and the second maximum probability by the formula wherein, and respectively represent the maximum possible category and the second largest possible category predicted by the model for the sample point .

5. The method of claim 1, wherein: The entropy method specifically includes: through formula As an uncertainty sampling strategy, the entropy of the prediction probability is used as an uncertainty measure to select samples, where S is the number of categories of the sample set, Indicates the probability that the sample x is predicted as the i-th category.

Images