White blood cell classification method based on deep self-supervised transformation learning

Abstract

This invention relates to a white blood cell classification method based on deep self-supervised transform learning. First, the white blood cell dataset is augmented using k data augmentation methods. Then, a neural network is trained to predict the data augmentation methods used in white blood cell images, thus acquiring prior knowledge of the white blood cell images. This invention improves the accuracy of white blood cell classification.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a white blood cell classification method based on deep self-supervised transform learning. Background Technology

[0002] White blood cells (WBCs) are the main components of immune cells and play a crucial role in the diagnosis and prognosis of blood diseases through visual examination of white blood cell images. Visual examination of white blood cell images is essential for the diagnosis and prognosis of hematological disorders. Most early detection and classification methods are not only labor-intensive and time-consuming, but the classification results can also be affected by the subjective bias of hematologists. Therefore, automated white blood cell classification has been a focus of research for the past 20 years.

[0003] Over the past few decades, the development of automated white blood cell diagnostic systems has achieved considerable success in image analysis, surpassing manual analysis. Traditional machine learning algorithms were initially applied to this field. However, these methods rely on preprocessing for cell segmentation, errors in which negatively impact cell classification. Recently, researchers have turned to deep learning methods, using models called neural networks for learning. Convolutional Neural Networks (CNNs), in particular, as a rapidly developing deep learning model, have been widely applied to white blood cell classification. While CNN models achieve high success rates, their high requirement for annotation and computational resources becomes a significant issue. In most cases, due to the limited number of medical cell datasets and low image resolution, training CNNs from scratch is difficult. Therefore, self-supervised learning has been used to reduce the training data and computational costs required to build deep learning models. Initially, the model is pre-trained on large and diverse general image datasets before being applied to classification tasks.

[0004] In the development of automatic white blood cell classification models, existing methods still require a large amount of data annotation and processing. The large amount of data annotation and processing cannot alleviate the predicament of heavy workload. Therefore, how to learn useful representations from unlabeled data has become the key to reducing the dependence on expensive manual annotation. One solution is to define an unlabeled task and use the visual information of the input data itself as a supervision signal. As a result, different mechanisms have emerged in this field to solve the representation learning problem, achieving encouraging results on multiple visual tasks, such as patch position prediction [4], puzzle solving [5], and rotation prediction [6]. Their basic goal is to learn invariant representations under different distortions or image transformations. Or image transformations, where different explanatory tasks are performed by the model in a self-supervised manner to learn useful representations directly on unlabeled data. However, few methods explore self-supervised visual representations of white blood cell images because white blood cell images are very different from images in natural scenes.

[0005] Self-supervised learning is a branch of unsupervised learning that aims to learn effective representations of unlabeled data. In particular, contrastive learning-based methods [1,2] have become the main branch due to their good interpretability. The goal of contrastive learning is to increase the distance between negative samples and decrease the distance between positive samples. However, contrastive learning requires consideration of the trade-off between uniformity features and the arrangement of positive features. In addition, clustering-based methods have also played an important role in self-supervised learning. These methods require samples to move closer to the center of the cluster in order to learn visual representations. For example, DeepCluster [3] uses the k-means clustering algorithm to iteratively group features and then uses them as pseudo-labels to update the weights based on the clustering results. Recently, studies have shown that pseudo-labels often use degenerate solutions to assign the same label and are not competitive in the same domain. Unlike the above methods, a unique type of self-supervised learning is to construct an effective excuse task to introduce some useful prior information for downstream tasks. Examples of effective excuse tasks include predicting the offset of relative patches [4], completing a jigsaw puzzle [5], and predicting image rotation [6]. The self-supervised method proposed in this invention belongs to the auxiliary task-based branch. The most relevant one to us is RotationNet[6]. It learns the object categories and their semantic information in an image by predicting the image rotation angle. However, for the white blood cell dataset, a single rotated image misses the morphological features and texture of the cells. To address this issue, we designed a broader task to predict the type of data augmentation. Summary of the Invention

[0006] The purpose of this invention is to provide a white blood cell classification method based on deep self-supervised transformation learning, which improves the accuracy of white blood cell classification.

[0007] To achieve the above objectives, the technical solution of the present invention is: a white blood cell classification method based on deep self-supervised transform learning, comprising:

[0008] Image enhancement stage:

[0009] Establish a classification task based on image transformations, defining k image transformations applicable to the input image X. The image generated after image transformation with label y is X. y = g(X|y);

[0010] Feature extraction stage:

[0011] Given an unlabeled dataset And a nonlinear function f parameterized by a set of learnable weight vectors. θ (·) That is, a convolutional neural network is used as the model; using k predefined image transformations. G(X n ) is used to represent the image X for each applied image transformation. n The resulting probability distribution, where The sample distribution representing the i-th data augmentation is the specific augmented version of the input instance; the data-augmented image. Forward propagation into the model produces a feature vector. Collected in a set

[0012] Classification stage:

[0013] k classification heads are selected to replace the original single classification head to output the probability distribution generated for the data-augmented image; for each feature vector The input is fed into k classification heads, and the output probability values ​​are then concatenated to form the probability distribution for the current image enhancement; the concatenated probability distribution is described as follows:

[0014]

[0015]

[0016] Where a cat (·) indicates a splicing operation. θ represents the probability value of the nth image output from the ith classifier, k is the number of classifiers, and θ is the parameter of the network model.

[0017] Compared to existing technologies, this invention offers the following advantages: This invention uses a self-supervised algorithm to learn from unlabeled white blood cell images. Because white blood cell images are very similar, it is difficult to distinguish the categories of white blood cells using conventional self-supervised algorithms. The algorithm of this invention first augments the white blood cell dataset using k data augmentation methods. Then, it trains the neural network by having it predict the data augmentation methods used for the white blood cell images, thus acquiring prior knowledge about the white blood cell images. We conducted linear evaluation, domain transfer, and fine-tuning tests on two white blood cell datasets, and the results confirm that the algorithm of this invention improves the classification accuracy for both datasets. Attached Figure Description

[0018] Figure 1 This is a diagram of the network model architecture of the method of the present invention.

[0019] Figure 2 This is a visualization of the t-SNE results. Detailed Implementation

[0020] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] This invention discloses a white blood cell classification method based on deep self-supervised transform learning, comprising:

[0022] Image enhancement stage:

[0023] Establish a classification task based on image transformations, defining k image transformations applicable to the input image X. The image generated after image transformation with label y is X. y = g(X|y);

[0024] Feature extraction stage:

[0025] Given an unlabeled dataset And a nonlinear function f parameterized by a set of learnable weight vectors. θ (·) That is, a convolutional neural network is used as the model; using k predefined image transformations. G(X n ) is used to represent the image X for each applied image transformation. n The resulting probability distribution, where The sample distribution representing the i-th data augmentation is the specific augmented version of the input instance; the data-augmented image. Forward propagation into the model produces a feature vector. Collected in a set

[0026] Classification stage:

[0027] k classification heads are selected to replace the original single classification head to output the probability distribution generated for the data-augmented image; for each feature vector The input is fed into k classification heads, and the output probability values ​​are then concatenated to form the probability distribution for the current image enhancement; the concatenated probability distribution is described as follows:

[0028]

[0029]

[0030] Where a cat (·) indicates a splicing operation. θ represents the probability value of the nth image output from the ith classifier, k is the number of classifiers, and θ is the parameter of the network model.

[0031] The following is a detailed implementation process of the present invention.

[0032] like Figure 1 The diagram shows the overall framework of the method of the present invention, which includes:

[0033] 1. Image enhancement module

[0034] Given the striking similarity of white blood cell images, choosing a suitable image transformation method can not only generate visual differences but also add more perspectives to the network's learning. First, we establish an image transformation-based classification task, forcing the ConvNet model to learn high-quality representations that can be used for downstream tasks (e.g., image classification). Using multiple views of things allows for better differentiation of their differences. Therefore, we define k image transformations applied to the input image X. The image generated after image transformation with label y is X. y = g(X|y). Through experiments, we selected five data augmentation methods to enhance white blood cell images as input. More formally, if H(X,y) is an operator that transforms an image X through an image transformation, then the image transformation consists of k (i.e., k=5) image transformations X. y = g(X|y).

[0035] 2. Feature Extraction Module

[0036] Our work aims to learn ConvNet-based representations in a self-supervised manner. In other words, we propose a self-supervised method based on multiple image transformations and apply it to white blood cell images. Given an unlabeled dataset... And a nonlinear function f parameterized by a set of learnable weight vectors. θ(·), meaning a neural network is used as the model. Specifically, predefined k images are used for enhancement. G(X n ) is used to represent each image X that has been enhanced using an image enhancement method. n The resulting probability distribution, where This represents the sample distribution for the i-th data augmentation (a specific augmentation version of the input instance). The data-augmented image. Forward propagation into the model produces a feature vector. This will be collected in a set

[0037] In self-supervised learning, we use a convolutional network as the backbone of the model to extract features. Specifically, given an unlabeled dataset... Let G be a set of image augmentation operations. Each unlabeled data point is augmented by the image augmentation operation g(·|y) to obtain the corresponding augmented image X. y = g(X|y).

[0038] 3. Classifier Module

[0039] We select k classifiers to replace the original single classifier to output the probability distribution for the data-augmented image. For each feature vector The inputs are fed into k classification heads, and the output probability values ​​are then concatenated to form the probability distribution for the current image enhancement. The concatenated probability distribution can be described as follows:

[0040]

[0041]

[0042] Where a cat (·) indicates a splicing operation. θ represents the probability value of the nth image output from the ith classifier, k is the number of classifiers, and θ is the parameter of the network model.

[0043] 4. Loss Function

[0044] The data augmentation method used to train a convolutional model f for a given input image as a predictor. θ This model f θ Get the input image X y (where y is the label the model wants to predict) and generate the probability distributions related to all possible data augmentation methods, as follows:

[0045]

[0046] Where f(X) y|θ) is the probability distribution for image prediction.

[0047] Therefore, for a given training image of length N The goal of self-supervised algorithms is that the convolutional model must learn to minimize the crossover loss, as follows:

[0048]

[0049] The loss function loss(·) can be expressed as:

[0050]

[0051] In summary, the algorithm of this invention consists of a data transformation module (i.e., an image enhancement module), a feature extraction module, and a classifier module. The data transformation module renders the image into k images and assigns corresponding labels. The feature extraction module consists of a backbone network used to extract features from the transformed image. This represents the features extracted by the network, and its probability distribution is obtained from the output of a classifier module.

[0052] To evaluate the performance of the self-supervised white blood cell classification algorithm, we tested its results on two white blood cell datasets, PBC and BCISC1K, using linear evaluation, domain transfer, and fine-tuning. Linear evaluation refers to training the self-supervised algorithm on the unlabeled dataset, freezing the trained backbone network, and then testing the results using labeled white blood cell samples. Domain transfer refers to training on the PBC dataset and performing linear evaluation on the BCISC1K dataset. Fine-tuning refers to training the self-supervised algorithm on the unlabeled dataset and testing it on labeled data without freezing the backbone network. In this invention, we use Raw, Colorjitter, Grayscale, Blend Alpha, and Gamma Contrast as data augmentation methods within the algorithm framework.

[0053] (1) Qualitative comparison

[0054] To further validate the effectiveness of our method, we used a pre-trained backbone network to extract features and visualized them in two-dimensional space using t-SNE. Figure 2 The visualization results are shown on the PBC dataset, using embeddings from our algorithm and DeepCluster. Compared to DeepCluster, our method exhibits lower scattering and more pronounced clustering performance.

[0055] Table 1. Quantitative comparison of linear evaluation, domain transfer, and fine-tuning of the algorithm of the present invention on the PBC and BCISC1K datasets.

[0056]

[0057] (2) Quantitative comparison

[0058] To quantitatively compare the classification accuracy of five algorithms (DeepCluster[1], RotationNet[6], Jigsaw Puzzles[4], Context[5], and the algorithm of this invention), we conducted linear evaluation, domain transfer, and fine-tuning experiments on the white blood cell datasets PBC and BCISC1K, respectively, and used accuracy as the standard to quantitatively evaluate the classification results. We also show the classification results of the white blood cell data under fully supervised algorithms and random parameters in Table 1, as the upper and lower bounds of the algorithms. Table 1 shows the linear evaluation, domain transfer, and fine-tuning results of the five algorithms on the white blood cell dataset on the PBC and BCISC1K datasets. The best metric value in each column is bolded. The linear evaluation results of white blood cell classification show that the feature representation of the backbone trained by the algorithm of this invention is the best, with corresponding accuracies of 88.88% and 71.82% on the PBC and BCISC1K datasets, respectively. Domain transfer is measured by testing a backbone network trained on one dataset on another dataset. As shown in the table, our results remain the highest. Even after fine-tuning, our results still maintain an optimal level. Therefore, it can be concluded that the self-supervised method designed in this invention produces the best feature quality on the white blood cell dataset.

[0059] References:

[0060] [1]He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE / CVF conference on computervision and pattern recognition. 2020.

[0061] [2] Gutmann, Michael, and Aapo "Noise-contrastive estimation:Anew estimation principle for unnormalized statistical models."Proceedings ofthe thirteenth international conference on artificial intelligence andstatistics.JMLR Workshop and Conference Proceedings,2010.

[0062] [3]Caron,Mathilde,et al."Deep clustering for unsupervised learning ofvisual features."Proceedings ofthe European conference on computer vision(ECCV).2018.

[0063] [4]Doersch,Carl,Abhinav Gupta,and Alexei A.Efros."Unsupervised visualrepresentation learning by context prediction."Proceedings ofthe IEEEinternational conference on computer vision.2015.

[0064] [5]Noroozi,Mehdi,and Paolo Favaro."Unsupervised learning of visualrepresentations by solvingjigsaw puzzles."European conference on computervision.Springer,Cham,2016.

[0065] [6] Gidaris, Spyros, Praveer Singh, and Nikos Komodakis. "Unsupervisedrepresentation learning by predicting image rotations." arXiv preprint arXiv:1803.07728(2018).

[0066] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A white blood cell classification method based on deep self-supervised transformation learning, characterized in that, Comprising: An image enhancement stage: establishing a classification task based on image transformations, defining k image transformations applied to the input image ; defining a set of k image transformations applied to the input image ; The image produced after the image transformation with the label y is ; A feature extraction stage: Given an unlabeled dataset A nonlinear function parameterized by a set of learnable weight vectors. That is, a convolutional neural network is used as the model; k predefined image transformations are used. ; It is used to represent the image for each applied image transformation. The resulting probability distribution, where Representing the The data augmentation sample distribution is a specific augmented version of the input instance; the data-augmented image. Forward propagation into the model produces a feature vector. Collected in a set ; A classification stage: The k classification heads are selected to replace the original single classification head to output probability distributions generated for the data-enhanced image; for each feature vector The input is input into the k classification heads, and then the output probability values are spliced together as the probability distribution of the current image enhancement; the spliced probability distribution is described as: wherein denotes a concatenation operation, represents the first image from the first probability value output by the classification head, k is the number of classification heads, is a parameter of the network model.

2. The white blood cell classification method based on deep self-supervised transformation learning according to claim 1, characterized in that, In the image enhancement stage, five data augmentation methods are selected to enhance the white blood cell images as input, i.e. if g(X, y) is an operator that transforms an image X by an image transformation, then the image transformation includes k (i.e., k = 5) image transformations .

3. The white blood cell classification method based on deep self-supervised transformation learning according to claim 1, characterized in that, Each unlabeled data in the unlabeled dataset in the feature extraction stage is enhanced by an image transformation operation to obtain a corresponding data augmentation, i.e., an image transformed image .

4. The white blood cell classification method based on deep self-supervised transformation learning according to claim 1, characterized in that, In the feature extraction stage, Obtain the input image And generate the probability distribution associated with all data augmentation methods as follows: where is the probability distribution of the image prediction.

5. The white blood cell classification method based on deep self-supervised transformation learning according to claim 4, characterized in that, For a given one length of training images The goal of the method is One must go to learn to minimize cross-loss, as follows: where the loss function is expressed as: 。

Images