A method for classifying bone marrow cells based on an improved ResNet-50 network

Abstract

The application discloses a kind of based on improved ResNet-50 network's bone marrow cell classification method, belong to the medical auxiliary diagnosis field of artificial intelligence.The method includes: by designing a new classification model of fusion multiscale feature, introduce PSA module to improve ResNet-50 network, introduce CSAB module to enhance the edge feature extraction capability to image, help model capture the local details and global structure information of bone marrow cell image;Introduce ACAM module to improve the attention of model to different categories of bone marrow cell key features;By training the improved ResNet-50 network, the optimal model obtained by training is classified.In improving classification efficiency while ensuring high accuracy, compared with other bone marrow cell classification models applied has better performance.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-based medical auxiliary diagnosis, and in particular to a bone marrow cell classification method based on an improved ResNet-50 network. Background Technology

[0002] Bone marrow cells are an important type of cell in the human body, mainly including hematopoietic stem cells, precursor cells, mature blood cells, bone marrow stromal cells, endothelial cells, and adipocytes. Each type of cell has its own function, and these cells work together to maintain the health of the blood system and the normal functioning of the immune system. The examination of bone marrow cell morphology and differentiation is an indispensable and important step in the diagnosis and prognostic assessment of hematological diseases, and has significant clinical application value.

[0003] Traditional methods of manually classifying bone marrow cells under a microscope rely primarily on direct observation and the physician's experience. This method demands highly skilled technicians, has low classification efficiency, and is susceptible to subjective judgment by the laboratory personnel, making standardization and normalization difficult. Furthermore, it struggles to handle complex cell morphologies and fails to meet the clinical needs for patient diagnosis and evaluation. To address these issues, new technologies such as whole-slide imaging, deep learning algorithms, and artificial intelligence have been applied to the development of bone marrow cell morphology analyzers. However, the complexity and unique characteristics of bone marrow specimens, including metastatic tumor cells, present significant challenges to the research and application of bone marrow cell morphology analyzers. Currently, manual microscopy remains the primary method for bone marrow smear examination internationally.

[0004] Deep learning-based models can better capture subtle changes in cell morphology and learn the complex characteristics of bone marrow cells through large-scale data training, thus achieving highly accurate identification and classification. This is of great significance for assisting doctors in diagnosis and providing a basis for clinicians to formulate treatment plans. However, current research on deep learning-based bone marrow classification models mostly focuses on datasets with limited sample sizes or insufficient category detail, which limits the models' general applicability. In addition, most models for bone marrow cell classification are based on traditional CNN architectures and lack design for the characteristics of bone marrow cell smears, resulting in low classification accuracy. Summary of the Invention

[0005] This invention aims to replace traditional bone marrow cell identification methods with artificial intelligence-based image processing technology, proposing an improved ResNet-50 bone marrow cell classification method based on multi-scale feature fusion, thereby improving classification accuracy and the effectiveness of artificial intelligence-assisted diagnosis.

[0006] To achieve the objective of this invention, the present invention provides an improved ResNet-50 bone marrow cell classification method based on multi-scale feature fusion, comprising the following steps:

[0007] Step 1: Prepare a bone marrow cell image dataset, divide it into training and test sets, and introduce data augmentation techniques to balance the number of samples in the training set.

[0008] Step 2: Introduce the PSA (Pyramid Squeeze Attention) module to construct an improved ResNet-50 network with multi-scale feature extraction capabilities, and integrate the attention mechanism for classification.

[0009] Step 3: Design a multi-scale feature aggregation module in the network feature extraction part, and introduce the CSAB (CellSpatial AttentionBlocks, CSAB) module to enhance the ability to extract edge morphological features of cells.

[0010] Step 4: Before the classifier, introduce the ACAM (Adaptive Channel Attention Module) module to enable the model to pay more attention to the key features that distinguish different types of bone marrow cells.

[0011] Step 5: Train the improved ResNet-50 network.

[0012] Step 6: Input the bone marrow cell images from the test set into the trained optimal model for testing and obtain the classification results.

[0013] Further, step 1 includes:

[0014] 1a) Prepare a bone marrow cell image dataset and divide it into training and test sets in an 8:2 ratio.

[0015] 2a) Use SMOTE technology to augment the training set for minority class samples and randomly undersample multi-class samples to balance the number of samples.

[0016] Furthermore, in step 2, the 3×3 convolutional layer in the ResNet-50 network residual block is replaced with a PSA module. The ResNet network is then divided into four modules, using features F1, F2, and F3 from blocks 2, 3, and 4 respectively to acquire information at different levels of the image.

[0017] Furthermore, the PSA module first extracts spatial information from different channels of the input feature map in a multi-scale manner using the SPC (Squeeze and Concat) module for compression and concatenation. Second, it uses the SEWeight module to extract the attention levels of the feature map, obtaining new attention weights. Third, it uses Softmax to calibrate the weights, obtaining recalibrated weights for the multi-scale channels. Fourth, it applies element-wise multiplication to the recalibrated weights and the corresponding feature maps. Finally, it outputs a richer feature map containing multi-scale feature information, helping the network to better process bone marrow cell images.

[0018] Furthermore, in step 3, the CSAB module uses edge detection and convolution operations to extract more detailed morphological features of cells from the three layers of feature maps F1, F2, and F3 using the Sobel operator, and further processes these features through convolutional layers to enhance the model's ability to represent the input feature maps.

[0019] Furthermore, the Sobel operator is first used to calculate the gradients in the horizontal and vertical directions to extract the edge information of the feature maps in these directions, and then these edges are merged into a total gradient map. The total gradient map is then normalized and multiplied element-wise with the original feature map to obtain a weighted feature map. This weighted feature map is then concatenated with the residuals of the original input feature map to obtain the final feature map. After processing, the three feature maps F1', F2', and F3' are concatenated to obtain a new feature map.

[0020]

[0021] in This represents the tensor concatenation operation.

[0022] Furthermore, the ACAM module in step 4 consists of a feature mapping layer and a learnable adaptive graph convolutional structure (ACWM). It utilizes the concept of graph convolutional networks to capture the relationships between pixels and adaptively enhances the interactions between feature channels, allowing the network to dynamically adjust the weights of each channel and highlight the more important features that the model focuses on when classifying bone marrow cells.

[0023] Furthermore, for the feature map F3 output from the last layer of the backbone network, the dimension of the input feature map is first transformed to 1×1×C through convolution and pooling operations. Then, after obtaining new feature weights through ACWM, they are mapped back to the original feature map as channel weights. These weights are multiplied with the original feature map F3 to achieve feature reweighting, resulting in the final output feature map F. out Its expression is:

[0024] F out =WF3(X1×X2+X3)

[0025] In the formula, X1 is a fixed matrix used as the baseline for constructing the attention matrix; X2 is the attention weights calculated through forward propagation, which are dynamically calculated during each forward propagation; X3 is a learnable parameter matrix, assigned a value of 1e-6 during model initialization and updated through gradient descent during training. During training, the gradient of the model parameters is updated according to the cross-entropy loss function, the expression of which is:

[0026]

[0027] Where p(x) represents the true distribution of the sample and q(x) represents the predicted distribution of the model.

[0028] Furthermore, F 123 With F out The data is then combined to train the classifier, yielding the final classification result.

[0029]

[0030] in This represents the tensor concatenation operation.

[0031] The advantages of this invention compared to the prior art are:

[0032] (1) This invention designs a new classification model that integrates multi-scale features for classifying bone marrow cells in bone marrow smears. Compared with manual microscopic examination, it is more efficient and accurate. Compared with other models applied to bone marrow cell classification, due to the combination of multi-scale feature extraction and adaptive learning attention mechanism, the model can better capture the important features of the image and improve the ability to distinguish between different types of cells, thus achieving better performance.

[0033] (2) The present invention designs the CSAB module to enhance the extraction of edge features of cell images. The Sobel operator is used to extract the edge morphological features of cells from the feature maps F1, F2 and F3 at three levels, and these features are further processed by the convolutional layer. Based on these features, the feature information of the image is effectively utilized, and the fusion of shallow and deep information is realized, which helps the model capture the local details and global structural information of bone marrow cell images.

[0034] (3) The present invention designs an ACAM module, which uses the idea of ​​graph convolutional network to capture the relationship between pixels and strengthens the interaction between feature channels in an adaptive way. This allows the network to dynamically adjust the weight of each channel, highlighting the features that are more important for bone marrow cell classification, thereby improving the classification accuracy of the model. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating the overall process of bone marrow cell classification as described in this invention.

[0037] Figure 2 This is a diagram illustrating the overall network structure involved in the present invention;

[0038] Figure 3 This is a schematic diagram of the PSA module involved in the present invention;

[0039] Figure 4 This is a schematic diagram of the CSAB module involved in the present invention;

[0040] Figure 5 This is a schematic diagram of the ACAM involved in the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0042] Example 1:

[0043] This embodiment provides a bone marrow cell classification method based on an improved ResNet-50 network. The overall flowchart is as follows: Figure 1 As shown, a ResNet-50 classification network is first built and improved using the PSA, CSAB, and ACAM modules. The bone marrow cell dataset is then enhanced by dividing it into training and test sets. The improved ResNet-50 classification network is used to train the training set, and the best-performing model is then tested on the test set to finally output the classification results.

[0044] Figure 2 The diagram shows the overall structure of the improved ResNet-50 network in this invention. The overall structure includes a backbone network, feature fusion, and an output module. Specifically, the backbone network uses a ResNet-50 network for feature extraction. By introducing a PSA module, features are extracted and concatenated at different scales, enhancing the model's ability to learn features. The feature fusion part introduces CSAB and ACAM modules, using edge detection and convolution operations to enhance the extraction of cell edge morphology features and focus on key features of different types of bone marrow cells, resulting in new feature maps. After classification by the classifier in the output module, the cells are output, achieving cell classification.

[0045] The bone marrow cell classification method based on the improved ResNet-50 network provided in this embodiment includes the following specific steps:

[0046] Step 1: Prepare a bone marrow cell image dataset, divide it into training and test sets, and introduce data augmentation techniques to balance the number of samples in the training set.

[0047] 1a) Based on the publicly available bone marrow cell dataset in "Highly accurate differentiation of bone marrow cellmorphologies using deep neural networks on a large image dataset" by Matek et al., the data comes from bone marrow smears of 961 patients diagnosed with various blood diseases, totaling 171,374 single-cell images of bone marrow, which are divided into training and testing sets in an 8:2 ratio.

[0048] 1b) Introducing the SMOTE technique, synthetic samples are generated from cell images with fewer than 1,000 images in the training set, amplifying the minority class samples. At the same time, cell images with more than 10,000 images are randomly undersampled, while other cell types are left unprocessed, alleviating the imbalance of the dataset.

[0049] 1c) The dataset contains 21 types of bone marrow cell images, each image being 250×250 pixels in size.

[0050] Step 2: Introduce the PSA module to construct an improved ResNet-50 network with multi-scale feature fusion characteristics, and combine it with an attention mechanism for classification.

[0051] The ResNet network is divided into four modules, and the low, medium and high-level features F1, F2 and F3 of blocks 2, 3 and 4 are used to obtain information at different levels of the image.

[0052] Step 3: Design a multi-scale feature aggregation module in the backbone network and introduce the CSAB module to enhance the ability to extract edge morphological features of cells.

[0053] By introducing the CSAB module and utilizing edge detection and convolution operations, the Sobel operator is used to extract cell edge morphology features from the three-level feature maps F1, F2, and F3 obtained in step 2. The Sobel operator calculates the gradients in the horizontal and vertical directions respectively, extracting edge information in the horizontal and vertical directions of the feature maps, and then merges them into a total gradient map. The normalized total gradient map is multiplied element-wise with the original feature map to obtain a weighted feature map, which is then residually connected with the original input feature map to obtain the final feature map, improving the model's ability to express features. After processing, these three feature maps are concatenated to obtain a new feature map.

[0054]

[0055] in This represents the tensor concatenation operation, where F1′, F2′, and F3′ represent the feature maps enhanced by the CASB module.

[0056] Step 4: Before the classifier, introduce the ACAM module to enable the model to focus more on the key features that distinguish different types of bone marrow cells.

[0057] Specifically, ACAM consists of a feature mapping layer and a learnable adaptive graph convolutional structure (ACWM). First, the enhanced features obtained in step 3 are concatenated. Then, the dimensions of the input feature map are transformed to 1×1×C through convolution and pooling operations. New feature weights are obtained using ACWM and mapped back to the original feature map as new channel weights. These weights are then multiplied by the original feature map to achieve feature reweighting, resulting in the final output feature map F. out Its expression is:

[0058] F out =WF3(X1×X2+X3)

[0059] In the formula, X1 is an identity matrix used to initialize the weights of the feature matrix; X2 is the attention weights calculated through forward propagation, which are dynamically calculated during each forward propagation; X3 is a learnable parameter matrix, assigned a value of 1e-6 during model initialization and updated through gradient descent during training; during training, the gradients of the model parameters are updated according to the cross-entropy loss function, the expression of which is:

[0060]

[0061] Where p(x) represents the true distribution of the sample and q(x) represents the predicted distribution of the model.

[0062] Furthermore, F 123 With F outThe data is then combined to train the classifier, yielding the final classification result.

[0063]

[0064] in This represents the tensor concatenation operation.

[0065] Step 5: Train the improved ResNet-50 network.

[0066] All experiments in this embodiment were conducted on the same computer, and the experimental environment configuration is shown in Table 1:

[0067] Table 1 Experimental Environment Configuration

[0068] environment Configuration CPU i5-13400@2.50GHz GPU NVIDIA RTX 3090 24GB Memory 512G Deep learning framework PyTorch 1.11 Programming environment Python 3.8.13

[0069] When images are fed into the model for training, they are cropped to a size of 224×224. During training, SMOTE data augmentation technology is used to expand the training samples. The initial learning rate is set to 0.001, and the Adam optimizer is used to train the parameters. The batch size is set to 32. The training lasts for 20 epochs, and the learning rate is decayed to 0.0001 after the 10th epoch.

[0070] This invention uses metrics such as accuracy, precision, recall, and specificity to evaluate the algorithm's performance. Precision represents the percentage of correctly classified positive samples out of the total number of predicted positive samples, reflecting the false positive rate of bone marrow cell classification. Recall assesses the model's ability to classify positive samples, which is crucial for ensuring the comprehensiveness and accuracy of the diagnosis. Specificity represents whether the model can effectively distinguish normal bone marrow cells; that is, how many samples the model can correctly exclude from those that actually belong to the negative class, avoiding misclassification as positive.

[0071] The calculation methods for the above indicators are as follows:

[0072]

[0073]

[0074] In this context, TP represents correctly classified positive cells, TN represents correctly classified negative cells, FP represents incorrectly classified negative cells, and FN represents incorrectly classified positive cells.

[0075] Step 6: Input the bone marrow cell images from the test set into the best-trained model for testing and obtain the classification results.

[0076] To verify whether the PSA, CSAB, and ACAM modules can improve model performance and enhance the model's ability to classify bone marrow cells, this application conducted ablation experiments on the same dataset and objectively analyzed each module based on the accuracy metric.

[0077] The experiment was divided into three different groups: PSA, PSA+ACAM, PSA+ACAM, and PSA+CSAB+ACAM. The accuracy results of the ablation experiments are shown in Table 2, with √ indicating the module used.

[0078] Table 2 Ablation Experiment Results

[0079] PSA CSAB ACAM Accuracy 0.844 √ 0.865 √ √ 0.869 √ √ 0.872 √ √ √ 0.875

[0080] The introduction of the PSA module significantly improved the model's performance. This indicates that the pyramid structure and attention mechanism of the PSA module enable the network to dynamically learn and emphasize the importance of features at different spatial locations, meaning the model can more accurately capture the key features of different cells, thereby improving classification accuracy. The addition of the CSAB module allows the model to more accurately locate and extract cell edge morphological features, helping the network to more accurately distinguish between the true features and irrelevant information of bone marrow cells, thus reducing recognition errors. The ACAM module effectively improves model performance, considering not only pixel-level relationships but also channel-level feature correlations. In cell images, different channels emphasize different cell features; ACAM can adaptively weight the features of each channel, enabling the model to more accurately extract and utilize this key information, thereby improving classification accuracy. After integrating these three modules, the model's performance is significantly improved compared to the baseline network, and the classification accuracy of bone marrow cells is greatly enhanced.

[0081] To verify the performance of the proposed model MSAL-ResNet, this embodiment also compares MSAL-ResNet with classic models such as ResNet-50, Densenet-121, Vit, and ResNext-50, as well as DAGDNet and ASAE-Net models designed for medical images in the literature. All experiments were conducted under the same environment configuration and parameter settings.

[0082] For the ResNet-50 model, please refer to the introduction in "Deep residual learning for image recognition"; for the Densenet-121 model, please refer to the introduction in "Densely connected convolutional networks"; for the Vit model, please refer to the introduction in "An image is worth 16x16 words: Transformers for image recognition at scale"; for the ResNext-50 model, please refer to the introduction in "Aggregated residual transformations for deep neural networks"; for the DAGDNet model, please refer to the introduction in "Automated bone marrow cell classification through dual attention gates dense neural networks"; and for the ASAE-Net model, please refer to the introduction in "Artificial intelligence-assisted diagnosis of hematologic diseases based on bone marrow smears using deep neural networks".

[0083] Table 3 Comparison of experimental results

[0084] Model Accuracy Recall Specificity ResNet-50 0.844 0.850 0.984 Densenet-121 0.827 0.829 0.981 ResNext-50 0.840 0.840 0.981 Vit 0.867 0.868 0.987 DAGDNet 0.835 0.835 0.981 ASAE-Net 0.852 0.853 0.985 MSAL-ResNet(Ours) 0.875 0.874 0.987

[0085] The results are shown in Table 3. It can be seen that the MSAL-ResNet model outperforms other models, indicating that the bone marrow cell classification method proposed in this application is effective. Compared with the classic ResNet-50 model, the method in this application combines multi-scale feature extraction and adaptive learning attention mechanism, enabling the model to better capture important features of the image and improve the ability to distinguish between different types of cells. Compared with Densenet-121, Vit, ResNext-50, and DAGDNet and ASAE-Net models designed for medical images, the method in this application is specifically designed for the characteristics of diverse bone marrow cell types and small interspecies differences. It designs a module to enhance the feature learning of cells. The model outperforms other models in the three key indicators of accuracy, recall, and specificity, and has high comprehensive performance in classification tasks, effectively distinguishing different types of bone marrow cell images.

[0086] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the claims.

Claims

1. A bone marrow cell classification method based on an improved ResNet-50 network, characterized in that, The method includes: Step 1: Prepare a bone marrow cell image dataset, divide it into a training set and a test set, and introduce data augmentation techniques to balance the number of samples in the training set; Step 2: Replace the 3×3 convolutional layers in the residual blocks of the ResNet-50 network with PSA modules, and divide the ResNet network into four modules. When acquiring features, use features from three levels: block 2, block 3, and block 4. To obtain information at different levels of the image; Step 3: Design a multi-scale feature aggregation module in the network feature extraction part, and introduce the CSAB module to enhance the ability to extract cell edge morphology features, with three levels of feature maps. The CSAB module was used to extract more detailed morphological features of cells. After processing and average pooling, feature maps were obtained. , , three feature maps , , A new feature map is obtained by splicing. ; Step 4: For the features in Step 2 The data is processed through convolution and pooling operations, and then ACWM is used to obtain new weights. These weights are then compared with the original feature map. Multiplication achieves reweighting of features, yielding the output feature map. ; Step 5: Concatenate the feature maps obtained in Step 3 and Step 4 to obtain the final fused features; Step 6: Input the fused features to train the improved ResNet-50 network; Step 7: Input the bone marrow cell image to be classified into the best trained model for testing and obtain the classification result.

2. The method according to claim 1, characterized in that, Step 2 includes: The 3×3 convolutional layer in the residual block of the ResNet-50 network is replaced with a PSA module to extract spatial information of the input feature map in a multi-branch manner. Multi-scale convolutional kernels are used to extract feature maps at different scales, effectively extracting different spatial information on each channel of the feature map.

3. The method according to claim 2, characterized in that, In step 3, the CSAB module uses edge detection and convolution operations to process the feature maps at three levels. Extracting cell edge morphological features using the Sobel operator: The Sobel operator is used to calculate the gradients in the horizontal and vertical directions respectively, and the edge information of the feature map in the horizontal and vertical directions is extracted and then merged into a total gradient map. After normalizing, processing with ReLU and Conv, the total gradient map is multiplied element-wise with the original feature map to obtain a weighted feature map. This weighted feature map is then connected with the original input feature map via residual concatenation and processed with Conv to obtain the final feature map. The three feature maps are then concatenated after average pooling to obtain a new feature map. .

4. The method according to claim 3, characterized in that, The ACAM module in step 4 consists of a feature mapping layer and a learnable adaptive graph convolutional structure ACWM. Step 4 includes: The dimensions of the input feature map are transformed through convolution and pooling operations. After obtaining new feature weights using ACWM, they are mapped back to the original feature map and used as new channel weights. The new channel weights are multiplied by the original feature map to achieve feature reweighting, resulting in the final output feature map. To highlight the key characteristics that distinguish different types of bone marrow cells.

5. The method according to claim 4, characterized in that, In step 2, the PSA module first extracts the spatial information of the input feature map in a multi-branch manner using the SPC module with multi-scale convolutional kernels; secondly, it uses the SEWeight module to extract the attention of feature maps at different scales to obtain the attention vector in the channel direction; thirdly, it uses Softmax to calibrate the channel attention vector to obtain the recalibrated weights of the multi-scale channels; and fourthly, it applies the element-wise multiplication operation to the recalibrated weights and the corresponding feature maps.

6. The method according to claim 5, characterized in that, The The expression is: ， in, , and This represents the feature map after enhancement by the CSAB module and average pooling. The The expression is: ， in, It is an identity matrix; The attention weights are calculated through forward propagation. It is a learnable parameter matrix, assigned the value 1e-6 during model initialization. During training, the model parameters are updated according to the gradient of the cross-entropy loss function, which is expressed as: in, Represents the true distribution of the sample. Represents the model's predicted distribution; The final classification features are: ， in This represents the tensor concatenation operation.

7. The method according to claim 6, characterized in that, Step 1 includes: Step 1a: Divide the training set and the test set into an 8:2 ratio; Step 2a: SMOTE data augmentation technique is used, combined with random undersampling technique to balance the number of training set samples.

8. The method according to claim 7, characterized in that, In step 1, for cell images with fewer than 1,000 images, synthetic samples are generated using SMOTE data augmentation technology to amplify minority class samples; for cell images with more than 10,000 images, random undersampling is performed to reduce the number of samples.

Images