Bone marrow cell classification method, system, computer device and storage medium

Through the multi-level feature learning network and target detection network with feature fusion, the problem of low efficiency of automatic segmentation and classification of bone marrow cells is solved, high-precision bone marrow cell detection and classification is achieved, and problems such as cell density are overcome to meet clinical needs.

CN114580501BActive Publication Date: 2024-11-29SOUTH CHINA UNIV OF TECH
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210129374.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-11
Publication Date
2024-11-29
Estimated Expiration
2042-02-11

AI Technical Summary

Technical Problem

The existing automatic segmentation and classification methods of bone marrow cells are inefficient and have problems with cell density, agglomeration, extrusion or overlap. The classification accuracy of traditional methods is limited and depends on doctors' identification experience and subjective differences.

Method used

Using a multi-level feature learning network and target detection network based on feature fusion, bone marrow cell detection is performed through the YOLOX network, combining multi-level feature learning branches, fusion knowledge distillation branches and channel complementary information mining module to realize automatic detection and fine-grainedness of bone marrow cells. Classification.

Benefits of technology

It improves the detection accuracy and classification accuracy of bone marrow cells, overcomes the problems of cell density, agglomeration, extrusion or overlap, and achieves a high fine-grained classification accuracy of bone marrow cells to meet clinical application needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114580501B_ABST
    Figure CN114580501B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, system, computer device and storage medium for classifying bone marrow cells, the method comprising: obtaining a first data set and a second data set; constructing a bone marrow cell fine-grained classification network; obtaining multiple original bone marrow cell microscopic images; inputting each original bone marrow cell microscopic image into a trained bone marrow cell detection network for prediction, and outputting the predicted annotation of the cell boundary box in each original bone marrow cell microscopic image; segmenting the original bone marrow cell microscopic image according to the prediction result of the cell boundary box, so that each image contains only one complete bone marrow cell; inputting the segmented image into the bone marrow cell fine-grained classification network for classification, and outputting the classification result of the bone marrow cells. The present invention improves the classification accuracy of bone marrow cells by learning multi-layer features in a deep network and introducing a knowledge distillation mechanism to introduce fusion information into a single-level feature classifier.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a bone marrow cell classification method, system, computer equipment and storage medium, and belongs to the field of computer vision. Background Art

[0002] The diagnosis of leukemia mainly relies on clinical medical tests. According to the test results and the general disease classification standards, the specific type of leukemia is diagnosed. At present, the commonly used clinical test techniques in leukemia diagnosis mainly include bone marrow cytology, immunology, cytogenetics, molecular biology and bone marrow pathology. Among them, bone marrow cytology detection technology is the main project for leukemia diagnosis and treatment efficacy evaluation. Because of its simplicity, practicality and high detection accuracy, it is also the most widely used diagnostic technology in clinical practice. In bone marrow cytology detection, the doctor performs bone marrow puncture on the examinee, stains the bone marrow sample and makes a bone marrow smear for microscopic observation. By observing the bone marrow cell smear under a high-power microscope, the cell morphology, quantity and abnormality are analyzed, so as to determine the series and growth stage to which each cell belongs. However, this type of bone marrow cell morphology examination method is not only time-consuming and labor-intensive, but also relies on expert knowledge, has certain subjective differences, and lacks objective quantitative standards. Therefore, if a bone marrow cell automatic analysis system can be developed with the help of medical image analysis and processing and deep learning technology to realize automatic detection and classification of bone marrow cells, it will be beneficial to improve the inspection efficiency of bone marrow cells and improve the classification accuracy of bone marrow cells.

[0003] At present, the automatic segmentation and classification of bone marrow cells based on traditional methods is inefficient and has certain problems. First, in terms of cell segmentation, some work uses traditional methods (such as the watershed algorithm) to segment cells and identify individual cells. However, due to the characteristics of bone marrow smears, there are serious problems of dense, agglomerated, squeezed or overlapping cells, and the boundaries between some cells are unclear, making it difficult to accurately detect and segment individual cells. The traditional cell classification algorithm uses feature extraction and then feeds it into a classifier (such as a support vector machine) for classification. Feature extraction is mainly based on the doctor's recognition experience and is converted into a descriptor for extraction. However, the descriptor sometimes cannot express the doctor's recognition experience well. At the same time, due to the characteristics of bone marrow cells, some differences between classes are small, and the similarity between features is high, so the classification accuracy of traditional methods is limited. Summary of the invention

[0004] In view of this, the present invention provides a bone marrow cell classification method, system, computer device and storage medium, which achieves higher bone marrow cell fine-grained classification accuracy by constructing a multi-level feature learning network with feature fusion, introducing fusion information into the features of a single level; in addition, the present invention overcomes the problems of dense, agglomerated, squeezed or overlapping bone marrow cells by constructing a bone marrow cell detection network based on target detection, thereby achieving better bone marrow cell detection effects.

[0005] The first object of the present invention is to provide a method for classifying bone marrow cells.

[0006] The second object of the present invention is to provide a bone marrow cell classification system.

[0007] A third object of the present invention is to provide a computer device.

[0008] A fourth object of the present invention is to provide a storage medium.

[0009] The first object of the present invention can be achieved by adopting the following technical solutions:

[0010] A method for classifying bone marrow cells, comprising:

[0011] Acquire a first data set and a second data set;

[0012] Construct a multi-level feature learning network based on feature fusion;

[0013] Select the YOLOX network as the bone marrow cell detection network, use the first data set to train the bone marrow cell detection network, and obtain a trained bone marrow cell detection network;

[0014] The multi-level feature learning network is trained using the second data set, and the trained multi-level feature learning network is used as a bone marrow cell fine-grained classification network;

[0015] Obtain multiple microscopic images of primitive bone marrow cells;

[0016] Input each original bone marrow cell microscopic image into the trained bone marrow cell detection network for prediction, and output the predicted annotation of the cell bounding box in each original bone marrow cell microscopic image;

[0017] The original bone marrow cell microscopic images are segmented according to the prediction results of the cell bounding box, so that each image contains only one complete bone marrow cell;

[0018] The segmented image is input into the bone marrow cell fine-grained classification network for classification, and the classification result of the bone marrow cells is output.

[0019] Furthermore, the multi-level feature learning network includes a backbone network structure, a multi-level feature learning branch, a fusion knowledge distillation branch, and a channel complementary information mining module;

[0020] The backbone network structure adopts the ResNet50 structure without the global pooling layer and the fully connected layer, with a total of five convolutional layer groups;

[0021] The multi-level feature learning branch includes three feature learning modules, three feature classifiers and a maximum k pooling layer;

[0022] The fusion knowledge distillation branch is used to guide the feature classifiers at a single level to obtain fusion information;

[0023] The channel complementary information mining module is used to model the relationship between feature channels, so as to mine the channel complementary information in the features.

[0024] Furthermore, the fusion knowledge distillation branch is used to guide the feature classifier of a single level to obtain fusion information, as follows:

[0025] Aggregate the three first feature vectors obtained in the multi-level feature learning branch to obtain a second feature vector;

[0026] Using the second feature vector to train a fusion feature classifier, a predicted probability distribution is obtained;

[0027] The predicted probability distribution is trained using KL divergence loss, so that the feature classifier of a single level obtains fused feature knowledge;

[0028] The specific formula of the KL divergence loss is as follows:

[0029]

[0030] Furthermore, the channel complementary information in the mining feature is specifically formulated as follows:

[0031] M rel =softmax(-X Flat X Flat T )

[0032] X enh =X ori +f(X Flat ·M rel )

[0033] Where X∈R c×h×w represents the input feature map size, c, h, w represent the number of feature channels, the height of the feature map, and the width of the feature map; X Flat ∈R c×lRepresents the matrix form of the input feature map after spatial dimension expansion, l = h × w; M rel Represents the channel relationship matrix.

[0034] Furthermore, the method of using the second data set to train the multi-level feature learning network and using the trained multi-level feature learning network as a bone marrow cell fine-grained classification network specifically includes:

[0035] Inputting the second data set into the bone marrow cell detection network to obtain a second training set, and preprocessing the second training set;

[0036] Import the pre-trained parameters into the backbone network, initialize the multi-level feature learning network with feature fusion, and set the corresponding network parameters and training parameters;

[0037] Input the preprocessed second training set into the multi-level feature learning network, use the cross entropy loss function for loss calculation, and use the KL divergence loss function for distillation;

[0038] The stochastic gradient descent method is used to update the network parameters so that the loss is gradually reduced to convergence, thereby obtaining a trained multi-level feature learning network, which is used as a fine-grained classification network for bone marrow cells.

[0039] Further, the selecting of the YOLOX network as the bone marrow cell detection network and the use of the first data set to train the bone marrow cell detection network to obtain a trained bone marrow cell detection network specifically includes:

[0040] Performing data enhancement processing on the first data set;

[0041] The first data set after data enhancement is divided into a first training set and a first validation set;

[0042] Select the YOLOX network as the bone marrow cell detection network, use the first training set to train the bone marrow cell detection network, and use the detection box position loss function and category loss function during the training process;

[0043] The cosine annealing learning rate decay strategy was used to iteratively optimize the bone marrow cell detection network. The first validation set data was used for validation during the iterative process to obtain a trained bone marrow cell detection network.

[0044] Furthermore, the specific formula of the detection box position loss function is as follows:

[0045]

[0046] Among them, GIoU represents the detection box position loss function;

[0047] The specific formula of the category loss function is as follows:

[0048] BCE=-w i [y i logx i +(1-y i )log(1-x i )]

[0049] Among them, BCE represents the category loss function.

[0050] The second object of the present invention can be achieved by adopting the following technical solutions:

[0051] A bone marrow cell classification system, comprising:

[0052] A first acquisition unit, used to acquire a first data set and a second data set;

[0053] A construction unit for constructing a bone marrow cell detection network and a multi-level feature learning network based on feature fusion;

[0054] A first training unit is used to select a YOLOX network as a bone marrow cell detection network, and train the bone marrow cell detection network using a first data set to obtain a trained bone marrow cell detection network;

[0055] A second training unit is used to train the multi-level feature learning network using the second data set, and use the trained multi-level feature learning network as a bone marrow cell fine-grained classification network;

[0056] a second acquisition unit, used for acquiring a plurality of microscopic images of original bone marrow cells;

[0057] A prediction unit, used for inputting each original bone marrow cell microscopic image into a trained bone marrow cell detection network for prediction, and outputting a predicted annotation of a cell boundary box in each original bone marrow cell microscopic image;

[0058] A segmentation unit, used for segmenting the original bone marrow cell microscopic image according to the prediction result of the cell boundary box, so that each image contains only one complete bone marrow cell;

[0059] The classification unit is used to input the segmented image into the bone marrow cell fine-grained classification network for classification, and output the classification result of the bone marrow cells.

[0060] The third object of the present invention can be achieved by adopting the following technical solutions:

[0061] A computer device comprises a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned bone marrow cell classification method is implemented.

[0062] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0063] A storage medium stores a program, and when the program is executed by a processor, the above-mentioned bone marrow cell classification method is implemented.

[0064] The present invention has the following beneficial effects compared with the prior art:

[0065] (1) The present invention creatively proposes a bone marrow cell classification method based on deep learning, which realizes bone marrow cell detection, cutting and automatic classification, and can better meet the needs of clinical applications.

[0066] (2) The present invention transfers the target detection method in deep learning to the bone marrow cell detection task, which can overcome the problems of dense, agglomerated, squeezed or overlapped bone marrow cells to a certain extent and achieve better bone marrow cell detection effects.

[0067] (3) The present invention creatively designs a fine-grained classification network for bone marrow cells based on multi-level feature learning. By learning the multi-layer features in the deep network and introducing the knowledge distillation mechanism, the fusion information is introduced into the feature classifier of a single level, thereby achieving a higher classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0069] Figure 1 This is a flow chart of the bone marrow cell classification method according to Example 1 of the present invention.

[0070] Figure 2 This is a structural diagram of a multi-level feature learning network based on feature fusion in Example 1 of the present invention.

[0071] Figure 3 This is a structural diagram of a channel complementary information mining module according to Embodiment 1 of the present invention.

[0072] Figure 4 This is a flow chart of the bone marrow cell classification system according to Example 2 of the present invention.

[0073] Figure 5 This is a structural block diagram of a computer device according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0075] Embodiment 1:

[0076] Bone marrow cell classification is often regarded as a coarse-grained classification task, and a commonly used deep network (such as a residual network ResNet) is usually used to classify bone marrow cells. However, for practical application requirements, cells of the same genus level need to be divided into different types according to the degree of maturity of bone marrow cells. The classification accuracy is limited if only a deep network designed for coarse-grained classification is used. To this end, this embodiment provides a method for classifying bone marrow cells.

[0077] like Figure 1 As shown, the bone marrow cell classification method of this embodiment includes the following steps:

[0078] S101: Acquire a first data set and a second data set.

[0079] A set of bone marrow cell microscopic images of patients with different disease types is obtained as a data set, wherein the data set includes a plurality of bone marrow cell microscopic images of patients with different disease types, and the total number of cells contained in the bone marrow cell microscopic image of each patient ranges from 200 to 500.

[0080] The data set was screened and processed, specifically by removing blurred images, unclear cell labels, or microscopic images with a lot of impurities, and estimating the number of patient cells based on the microscopic images to ensure that each patient contains 200-300 valid cells; among which, valid cells do not include cells with unclear labels, dyes, and degenerate cells.

[0081] The screened data set is divided into a training set and a test set. Specifically, 80% of the patients' bone marrow cell microscopic image sets are randomly selected as the training set, and the remaining 20% ​​of the patients' bone marrow cell microscopic image sets are used as the test set.

[0082] 40 patients with different disease types were selected from the training set to form the first data set; the first data set contained more than 1,000 microscopic images of bone marrow cells under a 100x microscope; further, the first data set was annotated and converted into VOC format, so that all cells in each bone marrow cell microscopic image had a bounding box label.

[0083] The second data set in this embodiment is also obtained from the training set.

[0084] S102. Construct a multi-level feature learning network based on feature fusion.

[0085] like Figure 2 As shown, the multi-level feature learning network based on feature fusion constructed in this embodiment includes a backbone network structure, a multi-level feature learning branch, a fusion knowledge distillation branch and a channel complementary information mining module.

[0086] The specific structure of the multi-level feature learning network based on feature fusion constructed in this embodiment is as follows:

[0087] (1) The backbone network structure adopts the ResNet50 structure, which includes five convolutional layer groups (the first convolutional group, the second convolutional group, the third convolutional group, the fourth convolutional group, and the fifth convolutional group), and removes the global pooling layer and the fully connected layer in the original structure; the image input size of ResNet50 is 3*224*224.

[0088] (2) The multi-level feature learning branch is used to carefully mine features at different levels. This branch includes three feature learning modules, a maximum K value pooling layer, and three corresponding feature classifiers.

[0089] Furthermore, the feature learning module includes two layers of convolution and corresponding batch normalization layers and ReLU layers, wherein the two layers of convolution are used to adjust the number of feature channels and to mine information in detail, respectively; the feature learning module does not change the size of the feature map, but only changes the number of channels; the maximum K value pooling layer is used to obtain the first feature vector representing the feature, and this pooling strategy is a compromise between maximum pooling and global average pooling, using the average of the first K maximum activation values ​​on each feature map as the feature representation on the channel; the feature classifier includes two layers of fully connected layers and corresponding batch normalization layers and ReLU layers, which are used to obtain the prediction distribution of the corresponding classifier.

[0090] (3) The fusion knowledge distillation branch is used to enhance the comprehensive utilization of features at different levels. Specifically, the three first feature vectors obtained in the multi-level feature learning branch are aggregated to obtain the second feature vector; the second feature vector is used to train a fusion feature classifier to obtain a prediction distribution; this classifier is used as a teacher to guide the learning of the single-level classifier. Specifically, the KL divergence loss is used to narrow the distance between the single-level feature classifier and the fusion feature classifier, thereby achieving the effect of fusion feature guidance. The calculation formula of KL divergence is as follows:

[0091]

[0092] (4) The channel complementary information mining module is used to enhance the channel complementary information of features. Its specific structure is as follows: Figure 3 As shown, when the input feature map is X∈R c×h×w When (c, h, w represent the number of feature channels, the height of the feature map and the width of the feature map respectively), in this module, the input feature map is first expanded in spatial dimensions to obtain X Flat ∈R c×l , where l = h × w; then X Flat The matrix is ​​multiplied by its own transpose and inverted, and a channel relationship matrix M is obtained after calculation by the Softmax function. rel , M rel Represents the correlation between channels; after obtaining the matrix, use it with X Flat Perform matrix multiplication to obtain features containing channel correlation and fuse them to the original features in a certain way to achieve feature enhancement. The calculation formula for feature enhancement is as follows:

[0093] M rel =softmax(-X Flat X Flat T )

[0094] X enh =X ori +f(X Flat ·M rel ).

[0095] S103, selecting the YOLOX network as the bone marrow cell detection network, and using the first data set to train the bone marrow cell detection network to obtain a trained bone marrow cell detection network.

[0096] This embodiment is based on bone marrow cell detection of the YOLOX algorithm. The YOLOX algorithm is the latest version of the YOLO series network proposed in 2021. This network is used for transfer learning in this embodiment. The specific steps are as follows:

[0097] S1031. Perform data enhancement processing on the first data set.

[0098] Data enhancement processing was performed on the first data set. Specifically, each bone marrow cell microscopic image in the first data set was randomly flipped and the contrast and brightness were randomly enhanced. Then, staintools was used to enhance the staining data according to the characteristics of bone marrow cells. Finally, the mosaic data enhancement method and MixUp data enhancement method were used to enhance the first data set.

[0099] S1032: Divide the first data set that has completed data enhancement to obtain a first training set and a first validation set.

[0100] The first data set that has completed data enhancement is randomly divided to obtain a first training set and a first validation set; the first training set contains 808 microscopic images of bone marrow cells under a 100-fold microscope, and the first validation set contains 202 microscopic images of bone marrow cells under a 100-fold microscope.

[0101] S1033, select the YOLOX network for transfer learning, use the first training set to train the YOLOX network, and use the first validation set data for validation, as follows:

[0102] a. Set network parameters: In the overall network, select the Yolox-s network for training, use the SiLU activation function as the activation function in the backbone network, use the detection box position loss GIOU and the category loss BCE.

[0103] The specific formula of the loss function is as follows:

[0104]

[0105] Among them, GIOU represents the detection box position loss function;

[0106] BCE=-w i [y i logx i +(1-y i )log(1-x i )]

[0107] Among them, BCE represents the category loss function.

[0108] b. Network initialization: Import the pre-trained model on the COCO dataset.

[0109] c. Set training parameters: According to the characteristics of bone marrow cells, the threshold of non-maximum suppression nms is set to 0.4, the cosine annealing learning rate decay strategy is adopted, the initial learning rate is 0.00015625, and the batch size is set to 8.

[0110] d. In the above iterative process, the first validation set data is used for verification, and finally a YOLOX network with bone marrow cell detection capability is obtained, that is, a trained bone marrow cell detection network.

[0111] S104. Using the second data set to train the multi-level feature learning network, and using the trained multi-level feature learning network as a bone marrow cell fine-grained classification network.

[0112] The second data set is input into the bone marrow cell detection network to perform bone marrow cell prediction and obtain the predicted annotation of the cell bounding box in each bone marrow cell microscopic image. Each bone marrow cell microscopic image is segmented according to the predicted result of the cell bounding box so that each image contains only one complete bone marrow cell. Finally, multiple single complete bone marrow cell images are obtained by segmentation and correspond to the cell categories calibrated by the doctor, thereby constructing the second training set.

[0113] The training steps of the multi-level feature learning network based on feature fusion in this embodiment are as follows:

[0114] a. Network initialization: Import the pre-trained parameters on ImageNet into the structure of the backbone network ResNet50 and randomly initialize the parameters of the rest.

[0115] b. Set network parameters: Set the maximum K value pooling layer K to 5 and the distillation temperature T to 3.

[0116] c. Set training parameters: Use stochastic gradient descent optimization, set the weight decay coefficient to 1e-4, the momentum to 0.9, and the batch size to 64. The initial learning rate is set to 5e-3, which decays to the original 0.2 at the 25th and 40th epochs.

[0117] d. Load the second training set and preprocess it: Before loading the second training set, preprocess the training images in the second training set by randomly cropping the input image size to 224*224, and then perform online data enhancement through three methods: random vertical flipping, random rotation, and color jittering.

[0118] e. Train the network: Figure 2 The multi-level feature learning network based on feature fusion shown in the figure is iteratively trained, specifically: the feature classifiers of three different levels and the classifiers of the fused feature branches are jointly trained, and the cross entropy loss function is used to calculate the loss; at the same time, the KL divergence loss function is used for distillation between the feature classifiers of a single level and the fused feature classifiers; then the gradient backpropagation is performed to update the network parameters so that the loss is gradually reduced to convergence, thereby obtaining a trained multi-level feature learning network based on feature fusion; the trained multi-level feature learning network based on feature fusion is used as a fine-grained classification network for bone marrow cells.

[0119] S105. Acquire multiple microscopic images of primitive bone marrow cells.

[0120] The multiple original bone marrow cell microscopic images acquired in this embodiment come from the test set divided in step S101.

[0121] S106, input each original bone marrow cell microscopic image into the trained bone marrow cell detection network for prediction, output the predicted annotation of the cell boundary box in each original bone marrow cell microscopic image, and save it as a csv file.

[0122] S107. Segment the original bone marrow cell microscopic image according to the prediction result of the cell boundary box, so that each image contains only one complete bone marrow cell.

[0123] S108, inputting the segmented image into a bone marrow cell fine-grained classification network for classification, and outputting a classification result of the bone marrow cells.

[0124] The segmented image is input into the bone marrow cell fine-grained classification network to obtain the prediction results of the three-level classifiers and the prediction results of the fusion knowledge distillation branch. The results of multiple classifiers are integrated to finally obtain the classification results of bone marrow cells.

[0125] This embodiment is based on the total classification results of the target patient's bone marrow cells and the ratio calculation to assist in diagnosing the type of acute leukemia in the target patient. The specific steps are as follows:

[0126] The total classification results of the target patient's bone marrow cells are counted, and the number of each type of cells of the target patient is counted, and the proportion is calculated to obtain the statistical results of the target patient's bone marrow cells;

[0127] The statistical results of the target patient's bone marrow cells are compared with the reference value of normal nucleated cells in the bone marrow, and then the type of acute leukemia is classified and typed according to the internationally accepted classification standards to complete the qualitative and evaluative work, thereby assisting in the diagnosis of the target patient's acute leukemia type.

[0128] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment method may be implemented by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0129] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0130] Embodiment 2:

[0131] like Figure 4As shown, this embodiment provides a bone marrow cell classification system, which includes a first acquisition unit 401, a construction unit 402, a first training unit 403, a second training unit 404, a second acquisition unit 405, a prediction unit 406, a segmentation unit 407 and a classification unit 408. The specific functions of each unit are as follows:

[0132] A first acquisition unit 401 is used to acquire a first data set and a second data set;

[0133] A construction unit 402 is used to construct a bone marrow cell detection network and a multi-level feature learning network based on feature fusion;

[0134] A first training unit 403 is used to select a YOLOX network as a bone marrow cell detection network, and train the bone marrow cell detection network using a first data set to obtain a trained bone marrow cell detection network;

[0135] A second training unit 404 is used to train the multi-level feature learning network using the second data set, and use the trained multi-level feature learning network as a bone marrow cell fine-grained classification network;

[0136] The second acquisition unit 405 is used to acquire multiple microscopic images of original bone marrow cells;

[0137] A prediction unit 406 is used to input each original bone marrow cell microscopic image into the trained bone marrow cell detection network for prediction, and output a predicted annotation of a cell boundary box in each original bone marrow cell microscopic image;

[0138] The segmentation unit 407 is used to segment the original bone marrow cell microscopic image according to the prediction result of the cell boundary box so that each image contains only one complete bone marrow cell;

[0139] The classification unit 408 is used to input the segmented image into the bone marrow cell fine-grained classification network for classification, and output the classification result of the bone marrow cells.

[0140] The specific implementation of each unit in this embodiment can refer to the above-mentioned embodiment 1, and will not be described one by one here; it should be noted that the system provided in this embodiment is only illustrated by the division of the above-mentioned functional units. In actual applications, the above-mentioned functions can be assigned to different functional units as needed, that is, the internal structure can be divided into different functional units to complete all or part of the functions described above.

[0141] Embodiment 3:

[0142] like Figure 5As shown, this embodiment provides a computer device, which includes a processor 502, a memory, an input device 503, a display device 504 and a network interface 505 connected through a system bus 501. The processor 502 is used to provide computing and control capabilities, and the memory includes a non-volatile storage medium 506 and an internal memory 507. The non-volatile storage medium 506 stores an operating system, a computer program and a database. The internal memory 507 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium 506. When the computer program is executed by the processor 502, the bone marrow cell classification method of the above-mentioned embodiment 1 is implemented as follows:

[0143] Acquire a first data set and a second data set;

[0144] Construct a bone marrow cell detection network and a multi-level feature learning network based on feature fusion;

[0145] Select the YOLOX network as the bone marrow cell detection network, use the first data set to train the bone marrow cell detection network, and obtain a trained bone marrow cell detection network;

[0146] The multi-level feature learning network is trained using the second data set, and the trained multi-level feature learning network is used as a bone marrow cell fine-grained classification network;

[0147] Obtain multiple microscopic images of primitive bone marrow cells;

[0148] Input each original bone marrow cell microscopic image into the trained bone marrow cell detection network for prediction, and output the predicted annotation of the cell bounding box in each original bone marrow cell microscopic image;

[0149] The original bone marrow cell microscopic images are segmented according to the prediction results of the cell bounding box, so that each image contains only one complete bone marrow cell;

[0150] The segmented image is input into the bone marrow cell fine-grained classification network for classification, and the classification result of the bone marrow cells is output.

[0151] Embodiment 4:

[0152] This embodiment provides a storage medium, which is a computer-readable storage medium, storing a computer program. When the computer program is executed by a processor, the bone marrow cell classification method of the above embodiment 1 is implemented as follows:

[0153] Acquire a first data set and a second data set;

[0154] Construct a bone marrow cell detection network and a multi-level feature learning network based on feature fusion;

[0155] Select the YOLOX network as the bone marrow cell detection network, use the first data set to train the bone marrow cell detection network, and obtain a trained bone marrow cell detection network;

[0156] The multi-level feature learning network is trained using the second data set, and the trained multi-level feature learning network is used as a bone marrow cell fine-grained classification network;

[0157] Obtain multiple microscopic images of primitive bone marrow cells;

[0158] Input each original bone marrow cell microscopic image into the trained bone marrow cell detection network for prediction, and output the predicted annotation of the cell bounding box in each original bone marrow cell microscopic image;

[0159] The original bone marrow cell microscopic images are segmented according to the prediction results of the cell bounding box, so that each image contains only one complete bone marrow cell;

[0160] The segmented image is input into the bone marrow cell fine-grained classification network for classification, and the classification result of the bone marrow cells is output.

[0161] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0162] In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device, or device. In this embodiment, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program used by or in combination with an instruction execution system, device, or device. The computer program contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0163] The computer readable storage medium can be written in one or more programming languages ​​or a combination thereof to execute the computer program of the present embodiment, and the programming language includes an object-oriented programming language, such as Java, Python, C++, and also includes a conventional procedural programming language, such as C language or a similar programming language. The program can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0164] In summary, the present invention achieves a higher fine-grained classification accuracy of bone marrow cells by constructing a multi-level feature learning network with feature fusion and introducing the fusion information into a single-level feature classifier; in addition, the present invention overcomes the problems of dense, agglomerated, squeezed or overlapping bone marrow cells by constructing a bone marrow cell detection network based on target detection, thereby achieving a better bone marrow cell detection effect.

[0165] The above is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and inventive concept of the present invention within the scope disclosed by the present invention, which shall fall within the protection scope of the present invention.

Claims

1. A method for classifying bone marrow cells, It is characterized in that The method comprises: Acquire a first data set and a second data set; Construct a multi-level feature learning network based on feature fusion; Select the YOLOX network as the bone marrow cell detection network, use the first data set to train the bone marrow cell detection network, and obtain a trained bone marrow cell detection network; The multi-level feature learning network is trained using the second data set, and the trained multi-level feature learning network is used as a bone marrow cell fine-grained classification network; Obtain multiple microscopic images of primitive bone marrow cells; Input each original bone marrow cell microscopic image into the trained bone marrow cell detection network for prediction, and output the predicted annotation of the cell bounding box in each original bone marrow cell microscopic image; The original bone marrow cell microscopic images are segmented according to the prediction results of the cell bounding box, so that each image contains only one complete bone marrow cell; The segmented image is input into the bone marrow cell fine-grained classification network for classification, and the classification result of the bone marrow cells is output; The multi-level feature learning network includes a backbone network structure, a multi-level feature learning branch, a fusion knowledge distillation branch, and a channel complementary information mining module; The backbone network structure adopts the ResNet50 structure without the global pooling layer and the fully connected layer, with a total of five convolutional layer groups; The multi-level feature learning branch includes three feature learning modules, three feature classifiers and a maximum k pooling layer; The fusion knowledge distillation branch is used to guide the feature classifiers at a single level to obtain fusion information; The channel complementary information mining module is used to model the relationship between feature channels, so as to mine the channel complementary information in the features; The fusion knowledge distillation branch is used to guide the feature classifiers at a single level to obtain fusion information, as follows: Aggregate the three first feature vectors obtained in the multi-level feature learning branch to obtain a second feature vector; Using the second feature vector to train a fusion feature classifier, a predicted probability distribution is obtained; The predicted probability distribution is trained using KL divergence loss, so that the feature classifier of a single level obtains fused feature knowledge; The specific formula of the KL divergence loss is as follows: The channel complementary information in the mining feature is specifically formulated as follows: M rel =softmax(-X Flat X Flat T ) X enh =X ori +f(X Flat ·M rel ) Where X∈R c×h×w represents the input feature map size, c, h, w represent the number of feature channels, the height of the feature map, and the width of the feature map; X Flat ∈R c×l Represents the matrix form of the input feature map after spatial dimension expansion, l = h × w; M rel Represents the channel relationship matrix.

2. The method for classifying bone marrow cells according to claim 1, It is characterized in that The method of training the multi-level feature learning network using the second data set and using the trained multi-level feature learning network as a fine-grained classification network for bone marrow cells specifically includes: Inputting the second data set into the bone marrow cell detection network to obtain a second training set, and preprocessing the second training set; Import the pre-trained parameters into the backbone network, initialize the multi-level feature learning network with feature fusion, and set the corresponding network parameters and training parameters; Input the preprocessed second training set into the multi-level feature learning network, use the cross entropy loss function for loss calculation, and use the KL divergence loss function for distillation; The stochastic gradient descent method is used to update the network parameters so that the loss is gradually reduced to convergence, thereby obtaining a trained multi-level feature learning network, which is used as a fine-grained classification network for bone marrow cells.

3. The method for classifying bone marrow cells according to claim 1, It is characterized in that The step of selecting the YOLOX network as the bone marrow cell detection network and training the bone marrow cell detection network using the first data set to obtain a trained bone marrow cell detection network specifically includes: Performing data enhancement processing on the first data set; The first data set after data enhancement is divided into a first training set and a first validation set; Select the YOLOX network as the bone marrow cell detection network, use the first training set to train the bone marrow cell detection network, and use the detection box position loss function and category loss function during the training process; The cosine annealing learning rate decay strategy was used to iteratively optimize the bone marrow cell detection network. The first validation set data was used for validation during the iterative process to obtain a trained bone marrow cell detection network.

4. The method for classifying bone marrow cells according to claim 3, It is characterized in that The specific formula of the detection box position loss function is as follows: Among them, GIoU represents the detection box position loss function; The specific formula of the category loss function is as follows: BCE=-w i [y i logx i +(1-y i )log(1-x i )] Among them, BCE represents the category loss function.

5. A bone marrow cell classification system, It is characterized in that The system comprises: A first acquisition unit, used to acquire a first data set and a second data set; A construction unit for constructing a multi-level feature learning network based on feature fusion; A first training unit is used to select a YOLOX network as a bone marrow cell detection network, and train the bone marrow cell detection network using a first data set to obtain a trained bone marrow cell detection network; A second training unit is used to train the multi-level feature learning network using the second data set, and use the trained multi-level feature learning network as a bone marrow cell fine-grained classification network; a second acquisition unit, used for acquiring a plurality of microscopic images of original bone marrow cells; A prediction unit, used for inputting each original bone marrow cell microscopic image into a trained bone marrow cell detection network for prediction, and outputting a predicted annotation of a cell boundary box in each original bone marrow cell microscopic image; A segmentation unit, used for segmenting the original bone marrow cell microscopic image according to the prediction result of the cell boundary box, so that each image contains only one complete bone marrow cell; A classification unit is used to input the segmented image into a bone marrow cell fine-grained classification network for classification, and output a classification result of the bone marrow cells; The multi-level feature learning network includes a backbone network structure, a multi-level feature learning branch, a fusion knowledge distillation branch, and a channel complementary information mining module; The backbone network structure adopts the ResNet50 structure without the global pooling layer and the fully connected layer, with a total of five convolutional layer groups; The multi-level feature learning branch includes three feature learning modules, three feature classifiers and a maximum k pooling layer; The fusion knowledge distillation branch is used to guide the feature classifiers at a single level to obtain fusion information; The channel complementary information mining module is used to model the relationship between feature channels, so as to mine the channel complementary information in the features; The fusion knowledge distillation branch is used to guide the feature classifiers at a single level to obtain fusion information, as follows: Aggregate the three first feature vectors obtained in the multi-level feature learning branch to obtain a second feature vector; Using the second feature vector to train a fusion feature classifier, a predicted probability distribution is obtained; The predicted probability distribution is trained using KL divergence loss, so that the feature classifier of a single level obtains fused feature knowledge; The specific formula of the KL divergence loss is as follows: The channel complementary information in the mining feature is specifically formulated as follows: M rel =softmax(-X Flat X Flat T ) X enh =X ori +f(X Flat ·M rel ) Where X∈R c×h×w represents the input feature map size, c, h, w represent the number of feature channels, the height of the feature map, and the width of the feature map; X Flat ∈R c×l Represents the matrix form of the input feature map after spatial dimension expansion, l = h × w; M rel Represents the channel relationship matrix.

6. A computer device comprising a processor and a memory for storing a program executable by the processor, It is characterized in that When the processor executes the program stored in the memory, the bone marrow cell classification method according to any one of claims 1 to 4 is implemented.

7. A storage medium storing a program, wherein when the program is executed by a processor, the bone marrow cell classification method according to any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • Bone marrow cell classification method and classification device based on depth learning

    CN109360198A

  • Morphological feature fused microscopic image bone marrow cell counting method and system

    CN113096096A

  • Bone marrow cell classification and identification method and system based on multiple features and multiple classifiers

    CN113177927A