Bone marrow cell classification method based on improved ResNet-50 network

Abstract

The invention discloses a bone marrow cell classification method based on an improved ResNet-50 network, and belongs to the field of medical auxiliary diagnosis of artificial intelligence. The method comprises the steps of designing a new classification model fused with multi-scale features, introducing a PSA module to improve a ResNet-50 network, introducing a CSAB module to enhance the edge feature extraction capability of an image, and helping the model to capture local details and global structure information of the bone marrow cell image; an ACAM module is introduced to improve the attention of the model to key features of different types of bone marrow cells; the improved ResNet-50 network is trained, and an optimal model obtained through training is classified. The high accuracy is ensured while the classification efficiency is improved, and compared with other models applied to bone marrow cell classification, the method has better performance.

Description

Technical Field

[0001] The present invention relates to the field of medical auxiliary diagnosis based on artificial intelligence, and in particular to a bone marrow cell classification method based on an improved ResNet-50 network. Background Art

[0002] Bone marrow cells are an important type of cell in the human body, which mainly include hematopoietic stem cells, precursor cells, mature blood cells, bone marrow stromal cells, endothelial cells and adipocytes, etc. Each cell has its own function, and these cells work together to maintain the health of the blood system and the normal functioning of the immune function. The examination and differentiation of bone marrow cell morphology is an indispensable part of the diagnosis and prognosis of blood system diseases, and has important clinical application value.

[0003] The traditional method of manual microscopic classification of bone marrow cells mainly relies on the intuitive observation of cells under a microscope and the doctor's experience and judgment. This method has high requirements for professional and technical personnel, low classification efficiency, and the classification results are easily affected by the subjective judgment of the inspectors. It is difficult to normalize and standardize, and it is difficult to deal with complex cell morphology; it cannot meet the clinical needs for patient diagnosis and evaluation. In order to solve the above problems, new technologies such as whole-slide imaging, deep learning algorithms, and artificial intelligence have been applied to the research and development of bone marrow cell morphology analyzers. However, since bone marrow specimens involve a variety of morphologically abnormal cells including metastatic tumor cells, their complexity and specificity make the research and application of bone marrow cell morphology analyzers face huge challenges. At present, manual microscopy is still the main means for bone marrow smear examination internationally.

[0004] Models based on deep learning can better capture subtle changes in cell morphology and learn the complex characteristics of bone marrow cells through large-scale data training, thereby achieving highly accurate identification and classification, which is of great significance for assisting doctors in diagnosis and providing a basis for clinicians to formulate treatment plans. However, most of the current research on bone marrow classification models based on deep learning focuses on data sets with limited sample numbers or insufficient categories, which limits the versatility of the model. In addition, most models for bone marrow cell classification are based on traditional CNN architectures, lack design for bone marrow cell smear characteristics, and have the problem of low classification accuracy. Summary of the invention

[0005] The present invention aims to replace the traditional bone marrow cell recognition method with artificial intelligence-based image processing technology, and proposes a bone marrow cell classification method based on an improved ResNet-50 with multi-scale feature fusion to improve the classification accuracy and effectiveness of artificial intelligence-assisted diagnosis.

[0006] In order to achieve the purpose of the invention, the present invention provides a bone marrow cell classification method based on an improved ResNet-50 with multi-scale feature fusion, comprising the following steps:

[0007] Step 1: Prepare a bone marrow cell image dataset, divide it into a training set and a test set, and introduce data augmentation technology to balance the number of samples in the training set.

[0008] Step 2: Introduce the PSA (Pyramid Squeeze Attention, PSA) module, build an improved ResNet-50 network with multi-scale feature extraction characteristics, and integrate the attention mechanism for classification.

[0009] Step 3: Design a multi-scale feature aggregation module in the network feature extraction part, and introduce the CSAB (Cell Spatial Attention Blocks, CSAB) module to enhance the ability to extract edge morphological features of cells.

[0010] Step 4: Before the classifier, the ACAM (Adaptive Channel Attention Module, ACAM) module is introduced to enable the model to pay more attention to the key features of distinguishing different categories of bone marrow cells.

[0011] Step 5: Train the improved ResNet-50 network.

[0012] Step 6: Input the bone marrow cell images in the test set into the trained optimal model for testing to obtain the classification results.

[0013] Furthermore, the step 1 comprises:

[0014] 1a) Prepare a bone marrow cell image dataset and divide it into a training set and a test set in a ratio of 8:2.

[0015] 2a) The training set is enhanced with the SMOTE technique for minority class samples, and multi-class samples are randomly undersampled to balance the number of samples.

[0016] Furthermore, the 3×3 convolutional layer in the residual block of the ResNet-50 network in step 2 is replaced by a PSA module. The ResNet network is divided into four modules, and the three-level features F1, F2, and F3 from blocks 2, 3, and 4 are used to obtain information at different levels of the image.

[0017] Furthermore, the PSA module first extracts the spatial information of different channels of the input feature map in a multi-scale manner through the SPC (Squeeze and Concat, SPC) module for compression and splicing; secondly, the SEWeight module is used to extract the attention of the feature map to obtain new attention weights. Third, the weights are calibrated using Softmax to obtain the recalibrated weights of the multi-scale channels. Fourth, the element-by-element product operation is applied to the recalibrated weights and the corresponding feature maps. Finally, a richer feature map containing multi-scale feature information is obtained as output, which helps the network to better process bone marrow cell images.

[0018] Furthermore, the CSAB module in step 3 uses edge detection and convolution operations to extract more detailed morphological features of cells from the three levels of feature maps F1, F2, and F3 using the Sobel operator, and further processes these features through the convolution layer to achieve the purpose of enhancing the model's representation ability for the input feature map.

[0019] Furthermore, the Sobel operator is used to calculate the horizontal and vertical gradients respectively, extract the edge information of the feature map in the horizontal and vertical directions, and then merge them into a total gradient map. The total gradient map is normalized and multiplied element by element with the original feature map to obtain a weighted feature map, which is then connected with the residual of the original input feature map to obtain the final feature map. After processing, the three feature maps F1', F2', and F3' are spliced ​​to obtain a new feature map.

[0020]

[0021] in Represents a concatenation operation on tensors.

[0022] Furthermore, the ACAM module in step 4 is composed of a feature mapping layer and a learnable adaptive graph convolution structure ACWM. It uses the idea of ​​graph convolutional networks to capture the relationship between pixels and strengthens the interaction between feature channels in an adaptive manner, allowing the network to dynamically adjust the weight of each channel and highlight the more important features that the model focuses on in classifying bone marrow cells.

[0023] Furthermore, for the feature map F3 output by the last layer of the backbone network, the dimension of the input feature map is first converted to 1×1×C through convolution and pooling operations, and then the new feature weights are obtained through ACWM and mapped back to the original feature map as channel weights. These weights are multiplied with the original feature map F3 to achieve feature reweighting and obtain the final output feature map F out , whose expression is:

[0024] F out =WF3(X1×X2+X3)

[0025] In the formula, X1 is a fixed matrix used to construct the benchmark of the attention matrix; X2 is the attention weight calculated by forward propagation, which is dynamically calculated during each forward propagation; X3 is a learnable parameter matrix, which is assigned a value of 1e-6 when the model is initialized and updated by gradient descent during training. During the training process, the gradient of the model parameters is updated according to the cross entropy loss function. The loss function expression is:

[0026]

[0027] Among them, p(x) represents the true distribution of the sample, and q(x) represents the predicted distribution of the model.

[0028] Furthermore, F 123 With F out Splice them together to train the classifier and get the final classification result.

[0029]

[0030] in Represents a concatenation operation on tensors.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] (1) The present invention designs a new classification model that integrates multi-scale features to classify bone marrow cells in bone marrow smears. Compared with manual microscopy methods, the model is more efficient and accurate. Compared with other models used for bone marrow cell classification, the model combines multi-scale feature extraction with an adaptive learning attention mechanism, so it can better capture the important features of the image and improve the ability to distinguish different types of cells, achieving better performance.

[0033] (2) The present invention designs a CSAB module to enhance the edge feature extraction of cell images, wherein the Sobel operator is used to extract the edge morphological features of cells from the three levels of feature maps F1, F2, and F3, and these features are further processed through a convolutional layer. Based on these features, the feature information of the image is effectively utilized, and the fusion of shallow information and deep information is realized, which helps the model capture the local details and global structural information of bone marrow cell images.

[0034] (3) The present invention designs the ACAM module, which captures the relationship between pixels by utilizing the idea of ​​graph convolutional networks, strengthens the interaction between feature channels in an adaptive manner, and allows the network to dynamically adjust the weight of each channel, highlighting the features that are more important for bone marrow cell classification, thereby improving the classification accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0036] Figure 1 The present invention is a general flow chart of bone marrow cell classification;

[0037] Figure 2 This is the overall structure diagram of the network involved in the present invention;

[0038] Figure 3 This is a schematic diagram of a PSA module according to the present invention;

[0039] Figure 4 This is a schematic diagram of a CSAB module according to the present invention;

[0040] Figure 5 This is a schematic diagram of the ACAM involved in the present invention. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0042] Embodiment 1:

[0043] This embodiment provides a bone marrow cell classification method based on an improved ResNet-50 network. The overall flow chart is as follows: Figure 1 As shown in the figure, first, a ResNet-50 classification network is built, and the PSA module, CSAB module and ACAM module are used to improve it; the bone marrow cell dataset is enhanced, the dataset is divided into a training set and a test set, and the training set is trained using the improved ResNet-50 classification network. The model with the best performance is used to test the test set, and finally the classification results are output.

[0044] Figure 2 The figure is the overall structure diagram of the improved ResNet-50 network in the present invention, and its overall structure includes a backbone network, feature fusion and an output module. Its specific structure is as follows: the backbone network uses the ResNet-50 network for feature extraction, and by introducing the PSA module, features are extracted and spliced ​​at different scales, thereby enhancing the model's ability to learn features; the feature fusion part introduces the CSAB module and the ACAM module, and with the help of edge detection and convolution operations, enhances the extraction of edge morphological features of cells and the focus on key features of different categories of bone marrow cells, thereby obtaining a new feature map; and after classification by the classifier of the output module, the output is output to achieve cell classification.

[0045] The bone marrow cell classification method based on the improved ResNet-50 network provided in this embodiment includes the following specific steps:

[0046] Step 1: Prepare a bone marrow cell image dataset, divide it into a training set and a test set, and introduce data augmentation technology to balance the number of samples in the training set.

[0047] 1a) Based on the public bone marrow cell dataset in "Highly accurate differentiation of bone marrow cellmorphologies using deep neural networks on a large image data set" by Matek et al., the data comes from 961 bone marrow smears diagnosed with various blood diseases, with a total of 171,374 bone marrow single cell images, which are divided into training set and test set in a ratio of 8:2.

[0048] 1b) SMOTE technology was introduced to generate synthetic samples from cell images with a number of less than 1,000 in the training set, and to amplify minority samples. At the same time, cell images with a number of more than 10,000 were randomly undersampled, and other types of cell images were not processed to alleviate the imbalance of the data set.

[0049] 1c) The dataset contains 21 types of bone marrow cell images, and the size of each image is 250×250.

[0050] Step 2: Introduce the PSA module, build an improved ResNet-50 network with multi-scale feature fusion characteristics, and combine it with the attention mechanism for classification.

[0051] The ResNet network is divided into four modules, and the three low-, medium-, and high-level features F1, F2, and F3 of block 2, block 3, and block 4 are used respectively to obtain information at different levels of the image.

[0052] Step 3: Design a multi-scale feature aggregation module in the backbone network and introduce the CSAB module to enhance the ability to extract edge morphological features of cells.

[0053] By introducing the CSAB module, with the help of edge detection and convolution operations, the Sobel operator is used to extract the edge morphological features of the cells from the three levels of feature maps F1, F2, and F3 obtained in step 2; the Sobel operator calculates the gradients in the horizontal and vertical directions respectively, extracts the edge information of the feature map in the horizontal and vertical directions, and then merges them into a total gradient map. The normalized total gradient map is element-by-element multiplied with the original feature map to obtain a weighted feature map, and a residual connection is performed with the original input feature map to obtain the final feature map, thereby improving the model's ability to express features. After processing, the three feature maps are spliced ​​to obtain a new feature map.

[0054]

[0055] in Represents the concatenation operation of the tensor, where F1′, F2′, and F3′ represent the feature maps enhanced by the CASB module.

[0056] Step 4: Before the classifier, the ACAM module is introduced to enable the model to focus more on the key features that distinguish different categories of bone marrow cells.

[0057] Specifically, ACAM consists of a feature mapping layer and a learnable adaptive graph convolution structure ACWM. First, the enhanced features obtained in step 3 are concatenated, and the dimension of the input feature map is converted to 1×1×C through convolution and pooling operations. The new feature weights are obtained by ACWM and then mapped back to the original feature map as the new channel weights. These weights are multiplied with the original feature map to achieve feature reweighting and obtain the final output feature map F out , whose expression is:

[0058] F out =WF3(X1×X2+X3)

[0059] In the formula, X1 is a unit matrix used to initialize the weights of the feature matrix; X2 is the attention weight calculated by forward propagation, which is dynamically calculated during each forward propagation; X3 is a learnable parameter matrix, which is assigned a value of 1e-6 when the model is initialized and updated by gradient descent during training; during training, the gradient of the model parameters is updated according to the cross entropy loss function, and the loss function expression is:

[0060]

[0061] Among them, p(x) represents the true distribution of the sample, and q(x) represents the predicted distribution of the model.

[0062] Furthermore, F 123 With F outSplice them together to train the classifier and get the final classification result.

[0063]

[0064] in Represents a concatenation operation on tensors.

[0065] Step 5: Train the improved ResNet-50 network.

[0066] All experiments in this embodiment are carried out on the same computer, and the experimental environment configuration is shown in Table 1:

[0067] Table 1 Experimental environment configuration

[0068] environment Configuration CPU i5-13400@2.50GHz GPU NVIDIARTX309024GB Memory 512G Deep Learning Frameworks Pytorch 1.11 Programming Environment Python 3.8.13

[0069] When the image is sent to the model for training, it will be cropped to a size of 224×224. During the training process, the SMOTE data enhancement technology is used to amplify the training samples, the initial learning rate is set to 0.001, the Adam optimizer training parameters are used, and the BatchSize is set to 32. A total of 20 epochs are trained, and the learning rate is decayed to 0.0001 after the 10th epoch.

[0070] In the present invention, the performance of the algorithm is evaluated by using indicators such as accuracy, precision, recall and specificity. Precision indicates the proportion of the number of positive samples correctly classified by the model in the total number of positive samples predicted by the model, reflecting the false positive rate of bone marrow cell classification; Recall indicates the ability of the model to classify positive samples, which is very important for ensuring the comprehensiveness and accuracy of diagnosis; Specificity represents whether the model can effectively distinguish normal bone marrow cells, that is, how many samples can be correctly excluded from the samples that actually belong to the negative class to avoid misclassifying them as positive.

[0071] The calculation methods of the above indicators are as follows:

[0072]

[0073]

[0074] Among them, TP represents correctly classified positive cells, TN represents correctly classified negative cells, FP represents incorrectly classified negative cells, and FN represents incorrectly classified positive cells.

[0075] Step 6: Input the bone marrow cell images in the test set into the best model obtained through training for testing and obtain the classification results.

[0076] In order to verify whether the PSA, CSAB, and ACAM modules can improve the model performance and enhance the model's ability to classify bone marrow cells, this application simultaneously conducted ablation experiments on the same dataset and objectively analyzed each module based on the accuracy indicator.

[0077] The experiments are divided into three different groups, including PSA, PSA+ACAM, PSA+ACAM, and PSA+CSAB+ACAM. The Accuracy results obtained from the ablation experiments are shown in Table 2. The mark √ represents the used module.

[0078] Table 2 Ablation experiment results

[0079] PSA CSAB ACAM Accuracy 0.844 √ 0.865 √ √ 0.869 √ √ 0.872 √ √ √ 0.875

[0080] The introduction of the PSA module significantly improved the performance of the model. This shows that the pyramid structure and attention mechanism of the PSA module enable the network to dynamically learn and emphasize the importance of features at different spatial locations, which means that the model can more accurately capture the key features of different cells, thereby improving the accuracy of classification. The addition of the CSAB module enables the model to more accurately locate and extract cell edge morphological features, helping the network to more accurately distinguish the true features of bone marrow cells from irrelevant information, thereby reducing recognition errors. The ACAM module effectively improves the performance of the model, which not only considers the relationship at the pixel level, but also the feature correlation at the channel level. In cell images, different channels emphasize different features of cells. ACAM can adaptively weight the features of each channel, allowing the model to more accurately extract and utilize these key information, thereby improving classification accuracy. After integrating these three modules, the performance of the model is significantly improved compared with the baseline network, and the classification accuracy of bone marrow cells is greatly improved.

[0081] In order to verify the performance of the proposed model MSAL-ResNet, this example also compares MSAL-ResNet with classic models such as ResNet-50, Densenet-121, Vit, ResNext-50, and DAGDNet and ASAE-Net models designed for medical images in the literature. All experiments were conducted under the same environment configuration and parameter settings.

[0082] For the ResNet-50 model, please refer to the introduction in “Deep residual learning for image recognition”; for the Densenet-121 model, please refer to the introduction in “Densely connected convolutional networks”; for the Vit model, please refer to the introduction in “An image is worth 16x16 words: Transformers for image recognition at scale”; for the ResNext-50 model, please refer to the introduction in “Aggregated residual transformations for deep neural networks”; for the DAGDNet model, please refer to the introduction in “Automated bone marrow cell classification through dual attention gates dense neural networks”; for the ASAE-Net model, please refer to the introduction in “Artificial intelligence-assisted diagnosis of hematologic diseases based on bone marrow smears using deep neural networks”.

[0083] Table 3 Comparative experimental results

[0084] Model Accuracy Recall Specificity ResNet-50 0.844 0.850 0.984 Densenet-121 0.827 0.829 0.981 ResNext-50 0.840 0.840 0.981 Vit 0.867 0.868 0.987 DAGDNet 0.835 0.835 0.981 ASAE-Net 0.852 0.853 0.985 MSAL-ResNet(Ours) 0.875 0.874 0.987

[0085] The results are shown in Table 3. It can be found that the MSAL-ResNet model performs better than other models, which shows that the bone marrow cell classification method proposed in this application is effective. Compared with the classic ResNet-50 model, the method of this application combines multi-scale feature extraction with adaptive learning attention mechanism, so that the model can better capture the important features of the image and improve the ability to distinguish different types of cells; compared with Densenet-121, Vit, ResNext-50, and DAGDNet and ASAE-Net models designed for medical images, the method of this application specifically designs a module to strengthen the feature learning of cells based on the characteristics of diverse bone marrow cell types and small inter-species differences. The model performs better than other models in the three key indicators of accuracy, recall rate and specificity, has a high comprehensive performance in classification tasks, and can effectively distinguish different categories of bone marrow cell images.

[0086] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Anyone familiar with this technology can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be based on the definition of the claims.

Claims

1. A bone marrow cell classification method based on an improved ResNet-50 network, characterized in that: The method comprises: Step 1: Prepare a bone marrow cell image dataset, divide it into a training set and a test set, and introduce data enhancement technology to balance the number of samples in the training set; Step 2: Introduce the PSA module, build an improved ResNet-50 network with multi-scale convolution kernels, and integrate the attention mechanism for classification; Step 3: Design a multi-scale feature aggregation module in the network feature extraction part, and introduce the CSAB module to enhance the ability to extract edge morphological features of cells; Step 4: Before the classifier, the ACAM module is introduced to enable the model to focus more on the key features that distinguish different types of bone marrow cells; Step 5: Train the improved ResNet-50 network; Step 6: Input the bone marrow cell image to be classified into the trained optimal model for testing to obtain the classification result.

2. The method according to claim 1, characterized in that The step 2 comprises: The 3×3 convolution layer in the residual block of the ResNet-50 network is replaced with the PSA module to extract the spatial information of the input feature map in a multi-branch manner, and multi-scale convolution kernels are used to extract feature maps of different scales, effectively extracting different spatial information on each channel of the feature map.

3. The method according to claim 2, characterized in that The ResNet network is divided into four modules. When acquiring features, the three-level features F1, F2, and F3 from blocks 2, 3, and 4 are used to acquire information at different levels of the image. In step 3, the CSAB module uses edge detection and convolution operations to extract the edge morphological features of cells from the three-level feature maps F1, F2, and F3 using the Sobel operator: The Sobel operator is used to calculate the gradients in the horizontal and vertical directions respectively, and the edge information of the feature map in the horizontal and vertical directions is extracted, and then merged into the total gradient map; After normalizing the total gradient map, perform element-by-element multiplication with the original feature map to obtain a weighted feature map, and perform a residual connection with the original input feature map to obtain the final feature map; Concatenate these three feature maps to get a new feature map F 123 .

4. The method according to claim 3, characterized in that: The ACAM module in step 4 is composed of a feature mapping layer and a learnable adaptive graph convolution structure ACWM, and step 4 includes: The dimension of the input feature map is converted to 1×1×C through convolution and pooling operations, and the new feature weights are obtained by ACWM and then mapped back to the original feature map as the new channel weights; Multiply the new channel weights by the original feature map to achieve feature reweighting and obtain the final output feature map F out To highlight the key features that distinguish different classes of myeloid cells.

5. The method according to claim 4, characterized in that In the step 2, the PSA module first extracts the spatial information of the input feature map in a multi-branch manner through the SPC module with multi-scale convolution kernels; secondly, the SEWeight module is used to extract the attention of feature maps of different scales to obtain the attention vector in the channel direction; thirdly, the channel attention vector is calibrated using Softmax to obtain the recalibrated weights of the multi-scale channels; fourthly, the element-by-element product operation is applied to the recalibrated weights and the corresponding feature maps.

6. The method according to claim 5, characterized in that The F 123 The expression is: Among them, F1', F2' and F3' represent the feature maps enhanced by the CASB module; The F out The expression is: F out =WF3(X1×X2+X3) Among them, X1 is a unit matrix; X2 is the attention weight calculated by forward propagation; X3 is a learnable parameter matrix, which is assigned a value of 1e-6 when the model is initialized. During the training process, the gradient of the model parameters is updated according to the cross entropy loss function. The expression of the cross entropy loss function is: Among them, p(x) represents the true distribution of the sample, and q(x) represents the predicted distribution of the model; The final classification results are: in Represents a concatenation operation on tensors.

7. The method according to claim 6, characterized in that The step 1 comprises: Step 1a, divide the training set and the test set into a ratio of 8:2; In step 2a, the SMOTE data enhancement technique is used and combined with the random undersampling technique to balance the number of training set samples.

8. The method according to claim 7, characterized in that In the step 1, for cell images with a number less than one thousand, synthetic samples are generated by using the SMOTE data enhancement technology to amplify minority samples; for cell images with a number greater than ten thousand, random undersampling is performed to reduce the number of samples.

Images