A label-free leukocyte three-classification method based on bright-field microscopy
Through the labelless leukocyte triclassification method based on bright-field microscopy images, the batch-free normalized classification network and adaptive gradient cutting strategy are used to solve the problem of fluorescence staining and high-cost flow cytometry in the existing technology, and a more efficient, accurate and popular leukocyte classification is achieved.
Patent Information
- Application Number
- CN202111599585.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-24
- Publication Date
- 2025-04-25
- Estimated Expiration
- 2041-12-24
AI Technical Summary
The existing leukocyte classification methods require fluorescent staining of blood samples, resulting in complex sample pretreatment and the staining agent affects cell activity and physiological status. The flow cytometry is costly and large in size, making it difficult to popularize.
The three-classification method of labelless leukocytes based on bright-field microscopy images was adopted to construct a batch-normalized labelless leukocyte classification network, and the adaptive gradient cutting strategy and data enhancement technology were used to achieve accurate classification of labelless leukocytes.
The pretreatment requirements for the detection samples are reduced, the damage to white blood cells by fluorescence staining is avoided, the classification accuracy and convergence speed of the classification model are improved, and the method is cheaper and more popular.
Smart Images
Figure CN114332855B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical image processing and deep learning cell classification, and in particular to a label-free leukocyte three-classification method based on bright field microscopic images. Background Art
[0002] Cells are the basic units of the human body. The types and content of white blood cells in the blood reflect the health status of an individual. In clinical diagnosis, white blood cells are usually divided into three categories, namely granulocytes, lymphocytes and monocytes. The changes in the morphology and content of various subtypes of white blood cells are closely related to the diagnosis of diseases, such as the clinical diagnosis of leukemia and anemia. Therefore, studying the white blood cell classification method and accurately classifying white blood cells can effectively improve the efficiency of disease detection and screening.
[0003] The traditional method of cell detection is manual microscopic examination or detection using a cell detector. The former method requires professionals to spend a lot of time on detection, and the accuracy of this method is heavily dependent on the professional knowledge and experience of the detector, which is prone to errors. The latter detection method, which is mainly based on flow cytometry, uses the photoelectric signals of cells after fluorescent staining for classification. Although this method has the characteristics of high throughput and high accuracy, the high cost and large size of flow cytometers limit its popularity. Both methods require fluorescent staining of blood samples before detection. However, fluorescence has a certain degree of phototoxicity, and the dye will change the physiological state of white blood cells, such as neutrophils from an inactive state to an activated state, so this method is not the best cell detection method.
[0004] In order to overcome the disadvantage that the aforementioned leukocyte classification method requires staining of the leukocytes to be detected, the present invention proposes a label-free leukocyte three-classification method based on bright field microscopic imaging. This method can classify bright field leukocyte microscopic images, reduce the pretreatment requirements for the samples to be detected, and effectively avoid the damage of fluorescent staining to the leukocytes to be detected. At the same time, this method abandons the batch normalization operation widely used in previous convolutional neural networks and adds an adaptive gradient clipping strategy. Compared with the existing model, it effectively improves the classification accuracy and convergence speed of the classification model. Summary of the invention
[0005] The purpose of the present invention is to provide a method for classifying unlabeled leukocytes, aiming to solve the problem of three-classification of unlabeled leukocytes using bright field microscopic images.
[0006] The specific steps of this classification method are as follows:
[0007] Step 1: Build a white blood cell dataset with cell category labels.
[0008] Step 2: Construct a label-free leukocyte classification network without batch normalization.
[0009] The label-free leukocyte count classification network without batch normalization includes an input layer, a convolution stack layer, a hidden layer and an output layer connected in sequence. The convolution stack layer includes four convolution parts. The convolution stack layer is used to extract features from the input image to form a feature map for classification. The feature map is input to the hidden layer. The residual block in the convolution part includes an adjustment layer, a convolution layer and an activation layer connected in sequence. The feature of the input adjustment layer is recorded as feature x l Adjust the layer to feature x l Calculate the variance σ of all elements in the three dimensions of channel, height and width; l Update to feature The convolution kernel parameters in the convolution layer are updated; the updated convolution kernel parameters Among them, W is the original convolution kernel parameter, μ W and σ W are the mean and variance of W in the fan-in dimension, and N is the number of parameters of W in the fan-in dimension. The output x of a single residual block o =x l +f l (x l ), where f l (·) is the feature x of the input residual block l The features obtained after being processed in sequence through all hierarchical structures in the residual block.
[0010] Step 3: Use the white blood cell dataset constructed in step 1 to train the classification network constructed in step 2.
[0011] Step 4: Use the trained classification network to identify the type of white blood cells.
[0012] Preferably, the types of white blood cells include granulocytes, lymphocytes and monocytes.
[0013] Preferably, the numbers of residual blocks of the four convolution parts are 3, 8, 36, and 3 respectively.
[0014] Preferably, the number of hidden neurons in the hidden layer is 2048.
[0015] Preferably, the activation function in the activation layer adopts a rectified linear unit.
[0016] Preferably, the process of predicting the category probability of the classification network is as follows:
[0017] (1) Set the original output of the output layer to x i , where i=0,1,2.
[0018] (2) Output x i Perform softmax calculation to obtain the predicted probability of each category
[0019] (3) Calculate the predicted category
[0020] Preferably, the specific steps of step one are as follows:
[0021] Unlabeled leukocyte images are collected and divided into training sets, validation sets, and test sets for training, validation, and testing of the classification network. The specific operation process of collecting microscopic images and performing segmentation and preprocessing is as follows:
[0022] (1) Use a multidimensional double-helix microfluidic chip to separate white blood cells from whole blood samples.
[0023] (2) The separated pure leukocyte solution is dripped onto a thin coverslip, and after the cells reach a stable state, optical microscopic imaging is performed using an oil immersion objective. At the same time, bright field and fluorescent field images of each field of view are captured. Multiple sets of image data are obtained by moving the field of view and changing the sample.
[0024] (3) Binarize the image to locate all individual cells and crop the corresponding image.
[0025] (4) Eliminate incomplete cell images and overlapping cell images to ensure that there is only one complete white blood cell in each image.
[0026] (5) Determine the cell type in the image based on the fluorescent field image, and use the cell type as the label of the corresponding bright field image.
[0027] (6) Data enhancement operations are performed on each image in the original dataset using data enhancement methods including horizontal flipping, vertical flipping, and rotation. After the data enhancement operation, the number of granulocyte, lymphocyte, and monocyte images are 10266, 10392, and 10400, respectively.
[0028] Preferably, the steps in step three are as follows:
[0029] (1) In the training phase, the cross entropy loss function is used as the target optimization function; the target optimization function L = -∑logy s ; Among them, y s is the predicted probability value corresponding to the true category s.
[0030] (2) During the training phase, the Adam optimizer is used to minimize the objective function, and the learning rate is 1×10 -4The running average coefficients for calculating the gradient and the square of the gradient are 0.9 and 0.999 respectively. No weight decay strategy is used, the probability of random dropout is set to 0.5, the batch size is set to 32, and the training is repeated for 100 rounds.
[0031] (3) Adaptive gradient clipping method is used during training. The specific gradient clipping strategy is designed as follows: Among them, ||·|| F is the Frobenius norm, is the i-th parameter to be trained in the l-th layer of the network, The objective optimization function L is about The gradient of λ is a scalar hyperparameter representing the threshold for gradient clipping.
[0032] (4) Save the model with the best verification accuracy for model performance testing.
[0033] The beneficial effects of the present invention are:
[0034] 1. The present invention uses an independently designed adjustment layer in the residual block, using the feature x l The input features are updated by taking the variance of all elements in the three dimensions of channel, height and width, which can effectively suppress the variance growth within the layer. In addition, the residual block uses a self-designed convolution kernel, the parameters of which are based on the original convolution kernel parameters, combined with the mean, variance and number of parameters of W in the fan-in dimension, which can effectively suppress the mean growth within the layer.
[0035] 2. The present invention proposes a method for classifying white blood cell bright field microscopic images. Compared with flow cytometers, bright field microscopes are smaller, cheaper, and more widely distributed. This solves the problem that flow cytometers are bulky, expensive, and difficult to popularize.
[0036] 3. The present invention classifies unlabeled leukocytes. Compared with the existing classification method based on staining, the sample pretreatment step is simpler and the influence of the stain on the cell activity and physiological state is overcome.
[0037] 4. The present invention uses a self-designed network structure to achieve the three-classification of unlabeled leukocytes. Compared with the existing network structure, the method proposed by the present invention can improve the convergence speed of the model, obtain better test accuracy, and have a better classification effect on unlabeled leukocytes. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is the overall flow chart of the present invention.
[0039] Figure 2It is a processing flow chart of step one of the present invention.
[0040] Figure 3 It is a design structure diagram of the label-free leukocyte classification network without batch normalization provided by the present invention.
[0041] Figure 4 It is a design structure diagram of the residual block in the present invention.
[0042] Figure 5 It is a design structure diagram of the adjustment layer in the present invention. DETAILED DESCRIPTION
[0043] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are described below with reference to the accompanying drawings.
[0044] like Figure 1 As shown, a label-free leukocyte three-classification method based on bright field microscopy imaging, the specific steps are as follows:
[0045] Step 1: Construction of unlabeled leukocyte dataset.
[0046] like Figure 2 As shown, unlabeled white blood cell images are collected and divided into training sets, validation sets and test sets for training, validation and testing of the classification network. The specific operation process of collecting microscopic images and performing segmentation and preprocessing is as follows:
[0047] (1) Separation of leukocytes from whole blood samples using a multidimensional double-helix microfluidic chip. The microfluidic chip can effectively separate leukocytes from diluted blood samples in a fully automated and label-free manner.
[0048] (2) The separated pure leukocyte solution was dripped onto a 24×50 mm thin coverslip. After the cells reached a stable state, optical microscopy was performed using a 100x oil immersion objective. The field of view of the optical microscopy was 960×720 pixels. At the same time, the bright field and fluorescent field images of each field of view were captured. Multiple sets of image data were obtained by moving the field of view and changing the sample.
[0049] (3) Use the adaptive threshold segmentation algorithm based on the Otsu method to perform a binarization operation on the image to locate all individual cells and crop the corresponding image to a size of 200 × 200 pixels. The Otsu method segmentation threshold k = σ b ×σ f ×(μ b -μ f ) 2 , parameter σ b and σ f Respectively represent the proportion of background pixels and foreground pixels, μ b and μf Represent the average gray value of the background and the average gray value of the foreground, respectively.
[0050] (4) Incomplete cell images and overlapping cell images were manually removed to ensure that each 200 × 200 pixel image contained only one complete white blood cell.
[0051] (5) Determine the cell category in the image based on the fluorescent field image, and use the cell category as the label of the corresponding bright field image, that is, determine which category the cells in the bright field image belong to: granulocytes, lymphocytes, or monocytes.
[0052] (6) Divide each labeled bright field image into three parts: original training set, original validation set, and original test set, with each part accounting for 70%, 10%, and 20% respectively.
[0053] (7) Data enhancement operations were performed on each image in the original dataset using data enhancement methods including horizontal flipping, vertical flipping, and rotation. After the data enhancement operation, the number of granulocyte, lymphocyte, and monocyte images were 10266, 10392, and 10400, respectively.
[0054] Step 2: Build a label-free leukocyte classification network without batch normalization
[0055] like Figure 3 As shown, the unlabeled white blood cell count classification network without batch normalization includes an input layer 1, a convolution stack layer 2, a hidden layer 4 and an output layer 5 connected in sequence.
[0056] The input layer 1 receives the bright field unlabeled leukocyte microscopic image as input; the convolution stack layer 2 includes four convolution parts, each of which is composed of a different number of residual blocks. The number of residual blocks in the four convolution parts is 3, 8, 36, and 3 respectively. The convolution stack layer 2 is used to extract features from the input image to form features for classification. Figure 3 .feature Figure 3 The input is sent to hidden layer 4. The number of hidden neurons in hidden layer 4 is 2048, which is used to further process the feature map. The output layer 5 outputs the probabilities of granulocytes, lymphocytes, and monocytes in the image respectively, and the category corresponding to the maximum probability is the predicted category of the classification network.
[0057] The process of predicting category probabilities by the classification network is as follows:
[0058] (1) Assume that the original output of the output layer is x i , where i=0,1,2.
[0059] (2) Output x i Perform softmax calculation to obtain the predicted probability of each category The calculation formula is
[0060] (3) Calculate the predicted category s is the i value corresponding to the maximum predicted probability; s = 0, 1, 2 correspond to granulocytes, lymphocytes, and monocytes, respectively.
[0061] like Figure 4 As shown in , the residual block in the convolution part includes an adjustment layer, a convolution layer, and an activation layer connected in sequence. The adjustment layer structure is shown in Figure 5 As shown, its structure is:
[0062] (1) The feature of the input adjustment layer is recorded as feature x l .
[0063] (2) For feature x l Calculate the variance σ of all elements in the three dimensions of channel, height and width; l Update to feature x t , the calculation formula is Thus, the correlation between different samples in the same batch is eliminated and the variance growth within the layer is suppressed.
[0064] In order to suppress the growth of the mean value within the layer, the convolution kernel in the convolution layer needs to be redesigned. The calculation formula is: Among them, W is the original convolution kernel parameter, W' is the updated convolution kernel parameter, μ W and σ W are the mean and variance of W in the fan-in dimension, and N is the number of parameters of W in the fan-in dimension.
[0065] The activation function in the activation layer uses the rectified linear unit, and the output feature of the activation layer is ReLU(x)=max(0,x).
[0066] The output x of a single residual block o =x l +f l (x l ), where f l (·) is the calculation function of the residual branch, that is, the feature x of the input residual block l The features obtained after being processed in sequence through all hierarchical structures in the residual block.
[0067] Step 3: Model training and testing.
[0068] (1) In the training phase, the cross entropy loss function is used as the target optimization function. The specific calculation method is the target optimization function. Among them, y s is the predicted probability value corresponding to the true category s.
[0069] (2) During the training phase, the Adam optimizer is used to minimize the objective function, and the learning rate is 1×10 -4 The running average coefficients for calculating the gradient and the square of the gradient are 0.9 and 0.999 respectively. No weight decay strategy is used, the probability of random dropout is set to 0.5, the batch size is set to 32, and the training is repeated for 100 rounds.
[0070] (3) Adaptive gradient clipping method is used during training. The specific gradient clipping strategy is designed as follows: Among them, ||·|| F is the Frobenius norm, is the i-th parameter to be trained in the l-th layer of the network, The objective optimization function L is about The gradient of To prevent the gradient of parameters initialized to 0 from being clipped to 0, λ is a scalar hyperparameter representing the threshold of gradient clipping.
[0071] (4) Save the model with the best verification accuracy for model performance testing.
[0072] Step 4: Use the trained model to identify the bright field unlabeled image with cells to obtain the category of the cells in the image.
Claims
1. A label-free leukocyte three-classification method based on bright field microscopy, characterized by: Step 1: Construct a white blood cell dataset with cell category labels; Step 2: Construct a label-free leukocyte classification network without batch normalization; The label-free leukocyte count classification network without batch normalization includes an input layer, a convolution stack layer, a hidden layer and an output layer connected in sequence; the convolution stack layer includes four convolution parts; the convolution stack layer is used to extract features of the input image to form a feature map for classification; the feature map is input to the hidden layer; the residual block in the convolution part includes an adjustment layer, a convolution layer and an activation layer; the feature of the input adjustment layer is recorded as feature x l ; Adjust the layer to feature x l Calculate the variance σ of all elements in the three dimensions of channel, height and width; l Update to feature The convolution kernel parameters in the convolution layer are updated; the updated convolution kernel parameters Among them, W is the original convolution kernel parameter, μ W and σ W are the mean and variance of W in the fan-in dimension, N is the number of parameters of W in the fan-in dimension; the output x of a single residual block o =x l +f l (x l ), where f l (·) is the feature x of the input residual block l The features obtained after being processed by all hierarchical structures in the residual block in sequence; Step 3: Use the white blood cell dataset constructed in step 1 to train the classification network constructed in step 2. The training process is as follows: (1) In the training phase, the cross entropy loss function is used as the target optimization function; the target optimization function L = -Σlogy s ; Among them, y s is the predicted probability value corresponding to the true category s; (2) During the training phase, the Adam optimizer is used to minimize the objective function, and the learning rate is 1×10 -4 , the running average coefficients for calculating the gradient and the square of the gradient are 0.9 and 0.999 respectively; no weight decay strategy is used, the probability of random dropout is set to 0.5, the batch size is set to 32, and the training is performed for 100 rounds; (3) Adaptive gradient clipping method is used during training; the specific gradient clipping strategy is designed as follows: Among them, ||·|| F is the Frobenius norm, is the i-th parameter to be trained in the l-th layer of the network, The objective optimization function L is about The gradient of ∈=10 -3 , λ is a scalar hyperparameter representing the threshold of gradient clipping; (4) Save the model with the best verification accuracy for model performance testing; Step 4: Use the trained classification network to identify the type of white blood cells.
2. The label-free leukocyte three-classification method based on bright field microscopy according to claim 1, characterized in that: The types of white blood cells include granulocytes, lymphocytes and monocytes.
3. The label-free leukocyte three-classification method based on bright field microscopy according to claim 1, characterized in that: The number of residual blocks in the four convolutional parts are 3, 8, 36, and 3 respectively.
4. The label-free leukocyte three-classification method based on bright field microscopy according to claim 1, characterized in that: The number of hidden neurons in the hidden layer is 2048.
5. The label-free leukocyte three-classification method based on bright field microscopy according to claim 1, characterized in that: The activation function in the activation layer adopts a rectified linear unit.
6. The label-free leukocyte three-classification method based on bright field microscopy according to claim 1, characterized in that: The process of predicting category probabilities by the classification network is as follows: (1) Set the original output of the output layer to x i , where i = 0, 1, 2; (2) Output x i Perform softmax calculation to obtain the predicted probability of each category Calculate predicted category 7. The label-free leukocyte three-classification method based on bright field microscopy according to claim 1, characterized in that: The specific steps of step one are as follows: The unlabeled leukocyte images were collected and divided into training set, validation set and test set for the training, validation and testing of the classification network; the specific operation process of collecting microscopic images and performing segmentation and preprocessing is as follows: (1) Use a multidimensional double-helix microfluidic chip to separate leukocytes from whole blood samples; (2) dropping the separated pure leukocyte solution onto a thin coverslip, and after the cells reach a stable state, performing optical microscopic imaging using an oil immersion objective lens; at the same time, capturing bright field and fluorescent field images of each field of view; and obtaining multiple sets of image data by moving the field of view and changing the sample; (3) Binarize the image to locate all individual cells and crop the corresponding image; (4) Eliminate incomplete cell images and overlapping cell images to ensure that there is only one complete white blood cell in each image; (5) determining the cell type in the image based on the fluorescent field image, and using the cell type as the label of the corresponding bright field image; Perform data augmentation operations on each image in the original dataset using data augmentation methods including horizontal flipping, vertical flipping, and rotation; After the data augmentation operation, the number of granulocyte, lymphocyte, and monocyte images are 10266, 10392, and 10400, respectively.