A white blood cell recognition method based on deep learning image enhancement
By building the UR-Net network, the combination of image enhancement and classification networks is used to solve the problem of stained leukocyte images, and efficient classification and accurate identification of label-free leukocytes are achieved, which simplifies the processing process and improves the recognition accuracy and speed.
Patent Information
- Application Number
- CN202310305575.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-27
- Publication Date
- 2025-09-02
- Estimated Expiration
- 2043-03-27
AI Technical Summary
In the prior art, stained leukocyte images have problems with long preparation time and irreversible effects on cells. At the same time, traditional image enhancement technology is not effective in computer classification recognition, and the quality of labelless leukocyte images is low.
The label-free leukocyte data set is used to construct the UR-Net network, including the image enhancement network and the classification network, and image enhancement is performed through 4 downsampling blocks and 4 upsampling blocks, combining pre-training and transfer learning to improve the performance of the classification network.
It improves the accuracy and convergence speed of leukocyte classification, simplifies data processing steps, avoids the impact of staining on cells, and can extract shallow and deep features, improving image quality and classification accuracy.
Smart Images
Figure CN116503854B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of white blood cell image recognition, and in particular to a white blood cell recognition method based on deep learning image enhancement. Background Art
[0002] Blood testing plays a vital role in the diagnosis and treatment of diseases. White blood cells, as a vital component of blood, fight bacteria and viruses in the human body. Based on their morphology and structure, white blood cells can be divided into three types: granulocytes, monocytes, and lymphocytes. The white blood cell count in the blood is closely correlated with various blood disorders and can be used as a diagnostic tool to diagnose the type and severity of diseases such as leukemia and cancer. Therefore, studying the classification and count of white blood cells is of great value to medical diagnosis.
[0003] Traditionally, white blood cell classification is typically performed by professionals using blood smears, and its accuracy relies heavily on their knowledge and experience. Currently, with the widespread application of deep learning technology, computers are better able to assist humans in completing complex tasks. Using convolutional neural networks to classify white blood cells not only reduces the workload for professionals but also achieves higher accuracy than human identification. Currently, most deep learning-based white blood cell classification techniques use stained white blood cells as datasets. However, stained white blood cells have significant disadvantages: the preparation of staining reagents is time-consuming, staining can cause irreversible damage to cells, and the staining process requires specialized personnel. Therefore, it is extremely important to use unlabeled white blood cells as datasets for identification and classification. Furthermore, microscopes are the most convenient instruments for acquiring label-free white blood cell images. However, microscopes are susceptible to external factors such as light sources, resulting in low image quality. Image enhancement technology has a good effect in improving image quality. Traditional image enhancement technologies such as histogram equalization and wavelet transform require professional mathematical statistics knowledge and heavy parameter adjustment tasks. Image enhancement technology based on convolutional neural networks can achieve higher performance and efficiency to a certain extent. However, the above methods are designed to improve human visual sensory effects and may not perform well in computer classification and recognition tasks.
[0004] To overcome the shortcomings of the aforementioned white blood cell classification methods, this paper proposes a white blood cell recognition method based on deep learning image enhancement. This method uses unlabeled white blood cells as a dataset and builds a novel network architecture, UR-Net, consisting of an image enhancement network and an image classification network. The image enhancement network aims to improve the classification performance of the classification network, rather than necessarily the human visual perception. Furthermore, the proposed pre-training method for the image enhancement network effectively improves the classification accuracy and convergence speed of the classification model. Summary of the Invention
[0005] The purpose of the present invention is to provide a leukocyte identification method based on deep learning image enhancement to solve the staining problem and the shortcomings of image enhancement technology in automatic leukocyte identification.
[0006] The white blood cell recognition method based on deep learning image enhancement includes the following steps:
[0007] Step 1: Build a white blood cell dataset with cell class labels.
[0008] Step 2: Build a white blood cell recognition network. The white blood cell recognition network consists of an image enhancement network and an image classification network. The output of the image enhancement network serves as the input to the image classification network. The image enhancement network is driven by the image classification network, enhancing the image in a way that improves the performance of the classification network.
[0009] The image enhancement network consists of four downsampling blocks and four upsampling blocks. Each downsampling block consists of three convolutional layers with strides of 2, 1, and 1, respectively. Each upsampling block consists of one deconvolutional layer and two convolutional layers. The deconvolutional layer has a scaling factor of 2, and the stride of both convolutional layers is 1.
[0010] The feature maps output by the deconvolution layer in each upsampling block are concatenated with the feature maps of the same size obtained by the corresponding downsampling block, and then the convolution operation in the upsampling block is performed.
[0011] Step 3: Perform pre-training on the image enhancement network to obtain the optimal initial weight parameters of the image enhancement network; perform transfer learning on the image classification network to obtain the optimal initial weight parameters of the image classification network.
[0012] Step 4: Use the white blood cell recognition network obtained after pre-training in step 3 to perform model training.
[0013] Step 5: Use the leukocyte recognition network obtained in step 4 to identify the type of the tested leukocytes.
[0014] Preferably, the convolution kernel sizes of the three convolution layers in the downsampling block and the two convolution layers in the upsampling block are all 3, and the padding is all 1.
[0015] Preferably, the upsampling algorithm of the upsampling block adopts a nearest neighbor sampling algorithm.
[0016] Preferably, in the image enhancement network, the activation function after each convolutional layer uses the LeakyReLU function.
[0017] Preferably, the image classification network adopts the ResNet101 network.
[0018] Preferably, the specific steps of step 3 are as follows:
[0019] (1) In the pre-training stage of the image enhancement network, the input image is used as the target image of the image enhancement network so that the image output by the image enhancement network is close to the input image.
[0020] (2) Use the cross entropy loss function as the objective function to pre-train the image enhancement network.
[0021] (3) Use the Adam optimizer to minimize the objective function in the image enhancement network pre-training, with a learning rate of 1×10 -3 , the running average coefficients used to calculate the gradient and the gradient square are 0.9 and 0.999 respectively, and the weight decay strategy is not used;
[0022] (4) During the pre-training of the image enhancement network, the loss value and weight parameters of each round of training are recorded, and the weight parameters corresponding to the minimum loss value are selected as the optimal initial weight parameters of the image enhancement network.
[0023] (5) Perform transfer learning on the image classification network to obtain the optimal initial weight parameters of the classification network.
[0024] Preferably, the types of white blood cells include granulocytes, monocytes and lymphocytes.
[0025] Preferably, the acquisition process of the white blood cell dataset described in step 1 is as follows: data is amplified based on the original dataset using an amplification method including horizontal flipping, vertical flipping, and rotating at different angles; an amplified white blood cell dataset is obtained; the original dataset contains labeled white blood cell images of all identified types.
[0026] Preferably, the white blood cell dataset is divided into a training set and a test set in a ratio of 8:2.
[0027] Preferably, the specific steps of step 4 are as follows:
[0028] (1) Connect the output of the image enhancement network to the input of the image classification network, and set the output of the last layer of the image classification network to 3 categories.
[0029] (2) In the training phase, the cross entropy loss function is used as the objective function of the leukocyte recognition network.
[0030] (3) The Adam optimizer is used to minimize the objective function of the leukocyte recognition network with a learning rate of 1×10 -4 The running average coefficients for calculating the gradient and the square of the gradient are 0.9 and 0.999 respectively. No weight decay strategy is used. The training batch size is set to 32 and the training is repeated for 200 rounds.
[0031] (4) Use the leukocyte recognition network with the best verification accuracy to test the model performance.
[0032] The beneficial effects of the present invention are:
[0033] 1. This invention abandons traditional enhancement methods and uses four downsampling blocks and four upsampling blocks to enhance images. The purpose of image enhancement is to improve the classification performance of the classification network, rather than the human visual perception effect. In this invention, each downsampling block reduces the image dimension by half; each upsampling block doubles the image dimension. The feature map output by the deconvolution layer in each upsampling block is concatenated with the feature map of the same size obtained by the corresponding downsampling block, and then the convolution operation in the upsampling block is performed. This helps to extract features that are easily recognized by the classification network, thereby improving the recognition accuracy of the white blood cell classification network.
[0034] 2. The present invention classifies unlabeled leukocytes. Compared with existing classification methods based on stained leukocytes, the present invention simplifies the data set processing steps and avoids the irreversible effects of staining reagents on cell activity and physiological status.
[0035] 3. The leukocyte recognition network structure provided by the present invention is more conducive to processing leukocyte images with simple structures, and can simultaneously extract shallow and deep features in leukocyte images, ensuring that their features are not lost during the image enhancement process.
[0036] 4. The pre-training process proposed in the present invention enables the network to have a better initial state, improves the convergence speed of the model, and obtains a higher test accuracy within a limited number of training times. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is an overall flow chart of the present invention.
[0038] Figure 2 It is the design structure diagram of the UR-Net network in the present invention.
[0039] Figure 3 It is a design structure diagram of the image enhancement network in the present invention.
[0040] Figure 4 This is a flowchart of pre-training and training the UR-Net network in steps 3 and 4 of the present invention. DETAILED DESCRIPTION
[0041] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are described below with reference to the accompanying drawings.
[0042] like Figure 1As shown in the figure, a white blood cell recognition method based on deep learning image enhancement is shown in the figure. The specific steps are as follows:
[0043] Step 1: Build a white blood cell dataset with cell class labels.
[0044] The sample types in the original white blood cell dataset include granulocytes, monocytes, and lymphocytes. The number of granulocyte, monocyte, and lymphocyte images is 1540, 211, and 444, respectively. Data augmentation methods, including horizontal flipping, vertical flipping, and rotating at different angles, were used to amplify the original white blood cell dataset. The resulting amplified images are 10266, 10400, and 10392, respectively. These images are then divided into training and test sets in an 8:2 ratio.
[0045] Step 2: Build a UR-Net network consisting of an image enhancement network and an image classification network.
[0046] like Figure 2 As shown in the figure, the UR-Net network uses the UNet network as the basic architecture of the image enhancement network and the ResNet101 network as the image classification network. The output of the image enhancement network serves as the input of the image classification network, thus achieving end-to-end three-level classification of white blood cells. The image enhancement network is driven by the image classification network, enhancing the image in a direction that is conducive to improving the performance of the classification network.
[0047] like Figure 3 As shown in Figure 2, the specific structure of the image enhancement network is as follows:
[0048] (1) The image enhancement network consists of 4 downsampling blocks and 4 upsampling blocks;
[0049] (2) Each downsampling block consists of three convolutional layers with kernel sizes of 3, 3, and 3, strides of 2, 1, and 1, respectively, and padding of 1, 1, and 1. The first convolutional layer reduces the width and height of the feature map to half of the original size, and the following two convolutional layers keep the size of the feature map unchanged so that it can be concatenated with the corresponding feature map in the upsampling process.
[0050] (3) Each upsampling block consists of a deconvolution layer and two convolution layers. The scaling factor of the deconvolution layer is 2, and the upsampling algorithm uses nearest neighbor sampling. The convolution kernel size of the two convolution layers is 3, the stride is 1, and the padding is 1. The deconvolution layer magnifies the width and height of the feature map to twice the original size, and the subsequent two convolution layers keep the size of the feature map unchanged to facilitate splicing with the corresponding feature map in the downsampling process.
[0051] (4) The deconvolution layer performs a concatenation operation with the feature map of the same size obtained by the downsampling block and then performs a convolution operation to simultaneously extract the shallow and deep features of the image;
[0052] (5) The activation function after each convolutional layer uses the LeakyReLU function: Among them, γ is a very small constant, which makes the network neurons less likely to be inactivated during the learning process, thereby improving the robustness of the network.
[0053] Step 3: Perform pre-training on the image enhancement network constructed in step 2 to obtain the optimal initial weight parameters of the image enhancement network; perform transfer learning on the image classification network to obtain the optimal initial weight parameters of the image classification network.
[0054] like Figure 4 As shown, the specific steps of this step are as follows:
[0055] (1) In the image enhancement network pre-training stage, the input image is used as the target image of the image enhancement network, so that the image output by the image enhancement network is close to the input image;
[0056] (2) In the image enhancement network pre-training phase, the cross entropy loss function is used as the objective function: L = -Σlogy s ; Among them, y s is the predicted probability value corresponding to the true category s.
[0057] (3) In the image enhancement network pre-training phase, the Adam optimizer is used to minimize the objective function and the learning rate is set to 1×10 -3 The running average coefficients used to calculate the gradient and the square of the gradient are 0.9 and 0.999 respectively, and the weight decay strategy is not used.
[0058] (4) During the pre-training of the image enhancement network, the loss value and weight parameters of each round of training are recorded, and the weight parameter corresponding to the minimum loss value is selected as the optimal initial weight parameter of the image enhancement network;
[0059] (5) Perform transfer learning on the image classification network on the ImageNet dataset to obtain the optimal initial weight parameters of the image classification network.
[0060] Step 4: If Figure 3 As shown in Figure 1, the UR-Net network composed of the image enhancement network pre-trained in step 3 and the image classification network obtained through transfer learning is used for training; the specific steps are as follows:
[0061] (1) The output of the image enhancement network is connected to the input of the image classification network, and after fine-tuning, the 1000 categories in the last layer of the image classification network are changed to 3 categories, resulting in the end-to-end network model UR-Net.
[0062] (2) During the training phase, the cross entropy loss function is used as the objective function: L = -Σlogy s , where y s is the predicted probability value corresponding to the true category s.
[0063] (3) Use Adam optimizer to minimize the objective function with a learning rate of 1×10 -4 The running average coefficients for calculating the gradient and the square of the gradient are 0.9 and 0.999 respectively. No weight decay strategy is used. The training batch size is set to 32 and the training is repeated for 200 rounds.
[0064] (4) Save the model with the best verification accuracy for model performance testing.
[0065] Step 5: Use the trained classification network to identify the type of the tested white blood cells whose types are unknown.
Claims
1. A method for white blood cell recognition based on deep learning image enhancement, characterized by: The following steps are involved: Step 1: Construct a white blood cell dataset with cell category labels; Step 2: Construct a leukocyte recognition network. The leukocyte recognition network includes an image enhancement network and an image classification network. The output of the image enhancement network serves as the input of the image classification network. The image enhancement network is driven by the image classification network to enhance the image in a direction that is conducive to improving the performance of the classification network. The image enhancement network includes 4 downsampling blocks and 4 upsampling blocks; each downsampling block consists of 3 convolutional layers; the step sizes of the 3 convolutional layers are 2, 1, and 1 respectively; each upsampling block consists of a deconvolutional layer and 2 convolutional layers; the scaling factor of the deconvolutional layer is 2; the step sizes of the 2 convolutional layers are both 1; The feature maps output by the deconvolution layer in each upsampling block are concatenated with the feature maps of the same size obtained by the corresponding downsampling block, and then the convolution operation in the upsampling block is performed; Step 3: Perform pre-training on the image enhancement network to obtain the optimal initial weight parameters of the image enhancement network; perform transfer learning on the image classification network to obtain the optimal initial weight parameters of the image classification network; Step 4: Use the leukocyte recognition network obtained after pre-training in step 3 to perform model training; Step 5: Use the leukocyte recognition network obtained in step 4 to identify the type of the tested leukocytes.
2. The method for white blood cell recognition based on deep learning image enhancement according to claim 1, characterized in that: The convolution kernel size of the three convolution layers in the downsampling block and the two convolution layers in the upsampling block are all 3, and the padding is all 1.
3. The method for white blood cell recognition based on deep learning image enhancement according to claim 1, characterized in that: The upsampling algorithm of the upsampling block adopts the nearest neighbor sampling algorithm.
4. The method for white blood cell recognition based on deep learning image enhancement according to claim 1, characterized in that: In the image enhancement network, the activation function after each convolutional layer uses the LeakyReLU function.
5. The method for white blood cell recognition based on deep learning image enhancement according to claim 1, characterized in that: The image classification network adopts the ResNet101 network.
6. The method for white blood cell recognition based on deep learning image enhancement according to claim 1, characterized in that: The specific steps of step 3 are as follows: (1) In the pre-training stage of the image enhancement network, the input image is used as the target image of the image enhancement network, so that the image output by the image enhancement network is close to the input image; (2) Use the cross entropy loss function as the objective function to pre-train the image enhancement network; (3) Use the Adam optimizer to minimize the objective function in the image enhancement network pre-training, with a learning rate of 1×10 -3 , the running average coefficients used to calculate the gradient and the gradient square are 0.9 and 0.999 respectively, and the weight decay strategy is not used; (4) During the pre-training of the image enhancement network, the loss value and weight parameters of each round of training are recorded, and the weight parameter corresponding to the minimum loss value is selected as the optimal initial weight parameter of the image enhancement network; (5) Perform transfer learning on the image classification network to obtain the optimal initial weight parameters of the classification network.
7. The method for white blood cell recognition based on deep learning image enhancement according to claim 1, characterized in that: The types of white blood cells include granulocytes, monocytes and lymphocytes.
8. The method for white blood cell recognition based on deep learning image enhancement according to claim 1, characterized in that: The process of acquiring the white blood cell dataset described in step 1 is as follows: based on the original dataset, data is amplified using an amplification method including horizontal flipping, vertical flipping, and rotating at different angles; an amplified white blood cell dataset is obtained; the original dataset contains labeled white blood cell images of all identified types.
9. The method for white blood cell recognition based on deep learning image enhancement according to claim 1, characterized in that: The white blood cell dataset is divided into a training set and a test set in a ratio of 8:
2.
10. The method for white blood cell recognition based on deep learning image enhancement according to claim 1, characterized in that: The specific steps of step 4 are as follows: (1) Connect the output of the image enhancement network to the input of the image classification network, and set the output of the last layer of the image classification network to 3 categories; (2) In the training phase, the cross entropy loss function is used as the objective function of the leukocyte recognition network; (3) The Adam optimizer is used to minimize the objective function of the leukocyte recognition network with a learning rate of 1×10 -4 The running average coefficients for calculating the gradient and the square of the gradient are 0.9 and 0.999 respectively. No weight decay strategy is used. The training batch size is set to 32 and the training is repeated for 200 rounds. (4) Use the leukocyte recognition network with the best verification accuracy to test the model performance.