White blood cell fine-grained classification method based on attention mechanism

Abstract

The application discloses a white blood cell fine-grained classification method based on an attention mechanism, and an end-to-end trainable model combining a Transformer and a convolution, which combines the advantages of the Transformer in capturing long-distance dependent relationships and extracting global features and the advantages of a CNN in extracting low-level image local features, can better construct a feature map of a white blood cell image, enrich feature information of the white blood cell, and improve the recognition accuracy of the cell image. Moreover, the model has certain generalization and stability, and optimal solutions can be obtained by using two optimizers, namely, an SGD and an AdamW.

Description

Technical Field

[0001] This invention belongs to the field of medical image recognition and deep learning technology, specifically involving a fine-grained white blood cell classification method based on Global-Local attention. Background Technology

[0002] The incidence of serious and dangerous diseases such as acute leukemia has been increasing globally in recent years, with a more pronounced trend towards affecting younger people. Diagnosis of these diseases relies on blood tests or the identification and classification of blood smears using microscopic imaging. Based on this, a count is performed to determine the proportion of different white blood cell types (white blood cell count, or complete blood count), thus enabling a diagnosis of various serious and dangerous diseases. Currently, in clinical hospitals, this mainly relies on experienced physicians manually identifying and counting the cells. This process is time-consuming, labor-intensive, and prone to errors, leading to misclassification, misdiagnosis, or missed diagnosis, which can threaten the patient's life.

[0003] Therefore, to address this phenomenon, scholars have introduced computer vision and machine learning techniques to automatically identify microscopic images of blood. Traditional machine learning techniques, such as edge detection, threshold segmentation, support vector machines, grayscale contrast analysis, and K-Means classification, are available, but their accuracy is not very high due to limitations of traditional computer vision and machine learning technologies.

[0004] With the advancement of computing power, deep learning has experienced tremendous growth and has been successfully applied in multiple vision fields. It primarily utilizes deep neural networks such as convolutional neural networks to extract local features from images and gradually constructs abstract feature maps through convolution and pooling methods, ultimately outputting object recognition results through fully connected layers. Compared to traditional machine learning methods, it significantly improves classification accuracy.

[0005] However, white blood cell recognition for the auxiliary diagnosis of serious diseases such as leukemia faces the following problems: 1) Due to the influence of different hospitals, different machines and equipment, and different imaging environments, the generated white blood cell images have different color differences, making automatic machine recognition difficult. 2) Due to the characteristics of the medical field of serious and dangerous diseases, many medical datasets have a long tail phenomenon, with very few samples for rare disease categories and a large number for common disease categories. Therefore, the category distribution is extremely unbalanced. 3) Since white blood cell images only contain white blood cell images, their features are very simple, which makes image recognition by the model difficult. 4) Since it is for the auxiliary diagnosis of leukemia, bone marrow smear cytology examination is required, that is, cell extraction from the bone marrow. The differences between certain cell groups and categories of white blood cells may not be particularly obvious, or the differences are too small. For example, the differences between early erythroblasts, intermediate erythroblasts and late erythroblasts in red blood cells are too small. 5) Diagnosing white blood cells requires fine-grained and accurate identification of up to 40 types of white blood cells.

[0006] Current CNN convolutional neural network methods cannot effectively solve the above problems. Summary of the Invention

[0007] This invention addresses the shortcomings of existing technologies by proposing a white blood cell classification method based on a global-local attention mechanism.

[0008] This invention proposes a white blood cell classification method based on a global-local attention mechanism. This method is an end-to-end trainable model combining Transformer and convolution, integrating the advantages of Transformer in capturing long-range dependencies and extracting global features with the advantages of CNN in extracting local image features at a low level. This allows for better construction of feature maps of white blood cell images, enriching the feature information of white blood cells and improving the recognition accuracy of cell images. Furthermore, this model exhibits certain generalization and stability, achieving optimal solutions using both SGD and AdamW optimizers.

[0009] The method of this invention specifically includes the following three steps: data collection, construction and training of the WBCLformer model, and performance evaluation of the WBCLformer model.

[0010] Step 1: Obtain white blood cell images as the basic dataset, and divide them into training and test sets;

[0011] Step 2: Building and training the WBCLformer model

[0012] The training of the WBCLformer model consists of three steps: building the neural network model, pre-training, and training on the white blood cell dataset.

[0013] Step 2.1: Building the Neural Network Model

[0014] The neural network consists of three parts: a feature extractor, an encoder, and a region filtering.

[0015] Step 2.1.1: Feature Extractor

[0016] Because white blood cell images lack rich features, the model struggles to extract accurate white blood cell image information. Unlike previous ViT and its improved models, which directly slice the image without processing, making it difficult to extract local information from the sliced ​​tokens, this step first transforms the image data into local feature data and uses a 3x3 convolution operation to obtain the image's feature data.

[0017] Step 2.1.2: Encoder

[0018] There are two types of encoders in WBCLformer: one is the encoder in Transformer, and the other is a modified local encoder.

[0019] Step 2.1.2.1: Transformer Encoder

[0020] The encoder is divided into two modules: one is the Multi-head Self Attention layers, i.e., MSA, and the other is the Feed Forward Network, i.e., FFN.

[0021] MSA: Learns from the interactions between input vectors and calculates different weights for different objects to find information about regions of interest.

[0022] FFN: After obtaining the weighted vector, the vector needs to be input into FFN for further processing.

[0023] Step 2.1.2.2: Local Encoder

[0024] The local encoder combines global attention from the self-attention mechanism with CNN to achieve a fusion of local and global information, enabling the extraction of effective white blood cell feature information. Since the self-attention mechanism learns the correlation between different patches and focuses more on global feature information, while locality is crucial for white blood cell images, convolution operations are incorporated into the encoder to extract local information.

[0025] Step 2.1.3: Identify and filter regions

[0026] Since the differences between similar images are extremely small, a region filtering module was added to enable the network to focus on the differences between different images. This module filters out tokens that are relevant.

[0027] Step 2.2: Training the white blood cell dataset

[0028] Because WBCLformer can overfit on small datasets, this step is divided into two processes: pre-training on the ImageNet-2012 dataset and training on the white blood cell dataset.

[0029] Step 2.2.1: Pre-training

[0030] The model training uses the AdamW optimization algorithm to adjust parameters, with a learning rate of 5e-4, a weight decay of 1e-3, a first-order exponential decay rate of 0.9, and a second-order exponential decay rate of 0.999. The model is trained for a total of 150-250 rounds.

[0031] Step 2.2.2: Training the white blood cell dataset

[0032] The last layer of the model is replaced, and the pre-trained model parameters are used as the initial values ​​for training, after which the model is trained.

[0033] The model training uses the AdamW optimization algorithm to adjust parameters, and the training parameters are the same as those used in pre-training. The learning rate adjustment strategy employs warm-up and cosine annealing to ensure the model's training stability.

[0034] Step 3: Verify the performance of the WBCLformer model

[0035] In order to quantitatively analyze the generalization ability of the model, the trained model needs to be tested on a test set, the predicted results are compared with the actual values, and evaluation metrics are used for analysis.

[0036] Preferably, the acquisition of white blood cell images is used as the basic dataset, and it is divided into a training set and a test set. The specific operation is as follows: sampling of the white blood cell image dataset is performed, and the dataset is divided into a training set and a test set according to the category in a ratio of 8:2.

[0037] Preferably, the evaluation metrics used are accuracy, precision, recall, and F1 score. Before introducing the evaluation metrics, let's first introduce the confusion matrix:

[0038] • TP: Positive samples predicted as positive by the model

[0039] •TN: Negative samples predicted as negative by the model

[0040] •FP: Negative samples predicted as positive by the model

[0041] • FN: Positive samples predicted as negative by the model

[0042] Accuracy: The percentage of correctly predicted results out of the total sample, calculated using the following formula.

[0043]

[0044] Precision: It represents the probability that a sample is actually positive out of all samples predicted as positive. The formula is as follows:

[0045]

[0046]

[0047] Recall: It represents the probability that a sample that is actually positive will be predicted as positive. The formula is as follows:

[0048]

[0049]

[0050] F1 score: To balance precision and recall, the F1 score was introduced. The formula is as follows:

[0051]

[0052] Since this is a multi-class classification problem, the calculation method for precision and recall is changed. First, the precision and recall of each class are calculated, and then they are added together according to their weights to calculate the average precision and recall.

[0053] After the model derives four metrics, it is compared with current mainstream models, proving that its performance is superior.

[0054] The method of this invention can combine local and global features without increasing the number of model parameters, thus enriching the image features of white blood cells. Optimal performance can be achieved based on a pre-trained WBCLformer. This invention has the following characteristics:

[0055] 1) This technology proposes a feature extraction method that combines local and global features, enriching the image features of white blood cells and improving the model's recognition accuracy. The feature extractor and local encoder use CNN to extract local features of the image, while the encoder uses a self-attention mechanism to extract global features of the image. The local and global features are combined in a cascaded manner.

[0056] 2) This technology proposes a region-selection module to improve the classification accuracy of white blood cells by filtering discriminative regions in the image. Since different types of bone marrow white blood cells exhibit morphological similarities, a self-attention mechanism is used to filter discriminative regions, thereby improving classification accuracy. Attached Figure Description

[0057] Figure 1 This is the overall flowchart of the present invention;

[0058] Figure 2 This is a network structure diagram of the present invention;

[0059] Figure 3 It is the feature extractor in the network structure of this invention;

[0060] Figure 4 This is a comparison chart of results within this dataset;

[0061] Figure 5 Comparison chart of results in publicly available datasets. Detailed Implementation

[0062] The method of this invention specifically includes the following three steps: data collection, construction and training of the WBCLformer model, and performance evaluation of the WBCLformer model.

[0063] Step 1: Obtain white blood cell images as the basic dataset, and divide them into training and test sets. The specific operations are as follows:

[0064] We collected a dataset of white blood cell images from several local hospitals, totaling 92,335 images across 40 white blood cell categories. We divided the dataset into categories in an 8:2 ratio, with the training set containing 73,877 samples and the test set containing 18,458 samples.

[0065] Step 2: Building and training the WBCLformer model ( Figure 2 )

[0066] The training of the WBCLformer model consists of three steps: building the neural network model, pre-training, and training on the white blood cell dataset.

[0067] Step 2.1: Building the Neural Network Model

[0068] The neural network consists of three parts: a feature extractor, an encoder, and a region filtering.

[0069] Step 2.1.1: Feature Extractor ( Figure 3 )

[0070] The input image data is converted into feature data, and the specific steps are as follows:

[0071] The input image data is processed through convolution with 64, 128, 256, and 512 channels to extract image features. The calculation formula is shown below:

[0072] z = Conv(BN(ReLU(x)))

[0073] Where x is the input image. Where C represents the number of output channels, S is the stride of the input image, ReLU is its activation function, BN (batch normalization) is batch normalization, and Conv is a 3x3 convolution operation.

[0074] Step 2.1.2: Encoder

[0075] WBCLformer has two types of encoders: one is the encoder in Transformer, and the other is a modified local encoder.

[0076] Step 2.1.2.1: Transformer encoder ( Figure 2 (to the left of .b)

[0077] The encoder is mainly divided into two modules: Multi-head Self Attention layers (MSA) and Feed Forward Network (FFN).

[0078] MSA: Learns from the interactions between input vectors and calculates different weights for different objects to identify more relevant information. The calculation formula is shown below:

[0079]

[0080] Attention(Q,K,V)=Cos(Q,K)×V

[0081] Q, K, V are all sequences of input vectors X∈R N×d Q = XW, obtained through linear mapping. Q K = XW K V = XW V Here, Q represents the query vector, K represents the vector used to match Q, and V represents the content vector. First, the cosine similarity between Q and K is calculated using the Softmax function to obtain the weight information for each object. Then, this cosine similarity is multiplied by V to obtain the final weighted vector. This operation can be used to filter the vectors.

[0082] FFN: After obtaining the weighted vector, the vector needs to be input into FFN for further processing, as shown in the following formula:

[0083] FFN(attn x ) = MLP(LN(atten x ))+atten x

[0084] atten x The weighted vector obtained through MSA represents the input weights. LN (Layer Normalization) represents layer normalization, which prevents gradient vanishing and accelerates convergence. LN is used here because layer normalization is performed on each data point, thus not changing the data distribution. MLP stands for Multilayer Perceptron, which consists of two linear layers. The first linear layer expands the data dimension from D to 3D, and the second linear layer reduces the dimension back to D. The Gelu activation function is used between the layers to add non-linear learning capabilities to the network. Finally, residual operations are achieved by adding the input weighted vector to prevent gradient vanishing or gradient explosion caused by an overly deep model.

[0085] Step 2.1.2.2: Local encoder ( Figure 2 (the right side of .b)

[0086] The local encoder combines global attention from the self-attention mechanism with CNN to achieve a fusion of local and global information, enabling the extraction of effective white blood cell feature information. Since the self-attention mechanism learns the correlation between different patches and focuses more on global feature information, while locality is crucial for white blood cell images, convolution operations are added to the encoder to extract local information. The calculation formula is shown below:

[0087] z seq+1 =I2S(DW(S2I(z) seq )))

[0088] S2I reconstructs sequential data into two-dimensional image data, DW is a Depth-Wise convolution operation, and I2S converts two-dimensional image data into sequential data. Therefore, the local encoder reconstructs sequential data into two-dimensional data and obtains its local features through convolution operations.

[0089] Step 2.1.3: Identify and filter regions

[0090] Because the differences between similar images are extremely small, a region filtering module is added to enable the network to focus on the differences between different images. This module filters out tokens with relevance, and its calculation formula is as follows:

[0091] atten final =Πatten i

[0092] atten i This represents the attention weight of the i-th layer. By multiplying the attention weights of each layer, the tokens that the model pays the most attention to are selected for the final classification.

[0093] Step 2.2: Training the white blood cell dataset

[0094] Because WBCLformer can overfit on small datasets, this step is divided into two processes: pre-training on the ImageNet-2012 dataset and training on the white blood cell dataset.

[0095] Step 2.2.1: Pre-training ( Figure 1 (Model pre-training)

[0096] The model training uses the AdamW optimization algorithm to adjust parameters, with a learning rate of 5e-4, a weight decay of 1e-3, a first-order exponential decay rate of 0.9, and a second-order exponential decay rate of 0.999. The model is trained for a total of 200 rounds.

[0097] Step 2.2.2: Training the white blood cell dataset ( Figure 1 (Model training in)

[0098] The last layer of the model is replaced, and the pre-trained model parameters are used as the initial values ​​for training, after which the model is trained.

[0099] The model training uses the AdamW optimization algorithm to adjust parameters, and its training parameters are the same as those used in pre-training. However, its learning rate adjustment strategy employs warm-up and cosine annealing to ensure the model's training stability.

[0100] Step 3: Verification of WBCLformer model performance ( Figure 1 (Performance of the test model in the test)

[0101] The trained model is used to make predictions on the test set, and the test results are compared with the true values ​​according to the evaluation metrics. The model performance is evaluated using four metrics: Accuracy, Precision, Recall, and F1-Score. After obtaining these four metrics, the model is compared with current mainstream models. Figure 4 As shown, the models proposed in this invention are all in a leading position, and to verify the generalization ability of the models, the publicly available datasets PBC and ALL-IDB were also used for validation. Figure 5As shown, their accuracy rates are all at a leading low level, and the experimental results verify the effectiveness and generalizability of the present invention.

Claims

1. A fine-grained classification method for leukocytes based on attention mechanisms, characterized in that, The method specifically includes the following steps: Data collection, construction and training of the WBCLformer model, and evaluation of the WBCLformer model's performance; Step 1: Obtain white blood cell images as the basic dataset, and divide them into training and test sets; Step 2: Building and training the WBCLformer model The training of the WBCLformer model consists of three steps: building the neural network model, pre-training, and training on the white blood cell dataset. Step 2.1: Building the Neural Network Model The neural network consists of three parts: a feature extractor, an encoder, and a region filtering mechanism. Step 2.1.1: Feature Extractor First, the image data is converted into local feature data, and then a 3*3 convolution operation is used to obtain the image feature data. Step 2.1.2: Encoder There are two types of encoders in WBCLformer: one is the encoder in Transformer, and the other is a modified local encoder. Step 2.1.2.1: Transformer Encoder The encoder is divided into two modules: one is the Multi-head Self Attention layers, i.e., MSA, and the other is the Feed Forward Network, i.e., FFN. MSA: Learns from the interactions between input vectors and calculates different weights for different objects to find information about the regions of interest. FFN: After obtaining the weighted vector, the vector needs to be input into FFN for further processing; Step 2.1.2.2: Local Encoder The local encoder extracts local information by combining global attention from the self-attention mechanism with CNNs and incorporating convolution operations into the encoder; its calculation formula is shown below: With seq+1 =I2S(DW(S2I(z seq ))) Among them, S2I reconstructs sequence data into two-dimensional image data, DW is a Depth-Wise convolution operation, and I2S converts two-dimensional image data into sequence data. Step 2.1.3: Identify and filter regions The formula for filtering tokens based on relevance is as follows: eighteen final =Patten i atten i This represents the attention weight of the i-th layer. By multiplying the attention weights of each layer, the tokens that the model pays the most attention to are selected for the final classification. Step 2.2: Training the white blood cell dataset Pre-training on the ImageNet-2012 dataset and training on the white blood cell dataset; Step 2.2.1: Pre-training The model training uses the AdamW optimization algorithm to adjust parameters, with a learning rate of 5e-4, a weight decay of 1e-3, a first-order exponential decay rate of 0.9, a second-order exponential decay rate of 0.999, and a total of 150-250 training rounds. Step 2.2.2: Training the white blood cell dataset The last layer of the model is replaced, and the pre-trained model parameters are used as the initial values ​​for training, and then the model is trained. The model training uses the AdamW optimization algorithm to adjust parameters, and its training parameters are the same as those used in pre-training; its learning rate adjustment strategy uses warm-up and cosine annealing. Step 3: Verify the performance of the WBCLformer model The trained model is tested on a test set, the predicted results are compared with the actual values, and evaluation metrics are used for analysis.

2. The leukocyte fine-grained classification method based on attention mechanism according to claim 1, characterized in that: The acquisition of white blood cell images as the basic dataset is divided into training and test sets. The specific operation is as follows: sampling of the white blood cell image dataset is performed, and the dataset is divided into training and test sets according to the category in a ratio of 8:

2.

3. The leukocyte fine-grained classification method based on attention mechanism according to claim 1, characterized in that: The evaluation metrics used are accuracy, precision, recall, and F1 score. Before introducing the evaluation metrics, let's first introduce the confusion matrix: ●TP: Positive samples predicted as positive by the model ●TN: Negative samples predicted as negative by the model ●FP: Negative samples predicted as positive by the model ●FN: Positive samples predicted as negative by the model Accuracy: The percentage of correctly predicted results out of the total sample, calculated using the following formula. Precision: It represents the probability that a sample is actually positive among all samples predicted as positive; the formula is as follows: Recall: It represents the probability that a sample that is actually positive will be predicted as positive; the formula is as follows: F1 score: To balance precision and recall, the F1 score was introduced; the formula is as follows: Since this is a multi-class classification problem, the calculation method for precision and recall is changed. First, the precision and recall of each class are calculated, and then they are added together according to their weights to calculate the average precision and recall.

Images