A Microscopic Image-Based Bone Marrow Cell Identification Method Based on Feature Mixing

Abstract

This invention discloses a method for identifying bone marrow cells from microscopic images based on feature fusion, comprising: acquiring labeled microscopic images of bone marrow cells to construct a dataset, dividing it into a training set and a test set; performing data augmentation on the images in the training set to obtain an augmented training set; feeding the augmented training set into an improved Mask R-CNN network for training to obtain an optimal model; inputting the test set into the optimal model to obtain prediction information, including: the location bounding box, category, and segmentation mask information of each bone marrow cell; drawing this information onto the original image; and marking the predicted bone marrow cell category above the location bounding box, thereby completing the identification of bone marrow cells. This invention improves the generalization of bone marrow cell image recognition across various color styles.

Description

Technical Field

[0001] This invention relates to the technical field of bone marrow blood cell target identification, and in particular to a method for bone marrow cell identification based on feature mixing microscopic images. Background Technology

[0002] The main challenge in automatically identifying bone marrow cells stems from the diversity and density of cell types in microscopic images. While modern deep learning technology has made significant progress in image recognition, variations in equipment model, type of staining agent, staining quality, and color style can greatly affect recognition accuracy when processing microscopic cell images.

[0003] Existing deep learning models typically perform well on specific training sets, but often lack generalization ability; that is, their performance drops sharply when faced with new color styles different from the training dataset. This is particularly evident in bone marrow cell identification, because differences in image color styles caused by different experimental conditions and equipment settings can make it difficult for models to accurately identify cell types in new samples. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for identifying bone marrow cells in microscopic images based on feature fusion. This method involves interactive fusion of feature layer statistical information of batch images in the shallow network of the feature extraction module of the improved Mask R-CNN network, thereby improving the network's generalization ability to recognize color style differences caused by different microscope models, staining conditions, etc.

[0005] To achieve the above objectives, the technical solution provided by this invention is as follows: a method for identifying bone marrow cells in microscopic images based on feature fusion. This method is based on an improved Mask R-CNN network to achieve target identification of bone marrow cells. The improved Mask R-CNN network is an improvement on the feature extraction module of the traditional Mask R-CNN network, that is, a shallow feature fusion module is added to the shallow network of the feature extraction module. The shallow feature fusion module is an improvement on the style feature fusion module, which is called MixStyle. The specific improvements are as follows: ① A current farthest point sampling module is added before MixStyle. The current farthest point sampling module is called CFPS. The CFPS calculates the Euclidean distance between the shallow features of each pair of image samples and selects the sample with the largest Euclidean distance to the current sample and provides it to MixStyle as input; ② An internal channel fusion module is added after MixStyle. The internal channel fusion module is called InterMix. The InterMix performs random style feature fusion between the feature channels of the sample using MixStyle.

[0006] The specific implementation of the microscopic image bone marrow cell identification method includes the following steps:

[0007] 1) Obtain labeled bone marrow cell microscopic images to construct a dataset, where the annotations for each image include the location bounding box, category, and segmentation mask of each cell. Then, divide the dataset into a training set and a test set.

[0008] 2) Perform data augmentation on the images in the training set to obtain an augmented training set after shape and color transformation;

[0009] 3) The enhanced training set is fed into the improved Mask R-CNN network. More out-of-distribution color style features are generated through the shallow feature mixing module of the feature extraction module, and feature information of different types of bone marrow cells is obtained through the feature extraction module. The extracted feature information is fused with the feature pyramid module FPN for multi-scale feature fusion, and then the region recommendation module RPN is used to obtain the region of interest. Finally, it is input into the classifier, bounding box regression module and mask segmentation module of the improved Mask R-CNN network to obtain the prediction results of bone marrow cells. Among them, binary cross-entropy and Smooth-L1 loss function are used to calculate the loss value of classification and regression in backpropagation. After multiple iterations, the loss value is minimized to obtain the optimal model.

[0010] 4) Input the test set into the optimal model to obtain prediction information, including: the location bounding box, category and segmentation mask information of each bone marrow cell. Draw this information onto the original image and mark the predicted bone marrow cell category above the location bounding box to complete the identification of bone marrow cells.

[0011] Furthermore, the data enhancement includes: randomly adjusting the image contrast, histogram equalization, inverting pixel values, randomly rotating the image, randomly adjusting the image brightness, randomly adjusting the image sharpness, and flipping the image in the horizontal or vertical direction.

[0012] Furthermore, the shallow feature blending module includes:

[0013] The current farthest point sampling module, called CFPS, calculates the Euclidean distance between shallow features of each pair of images in the same batch. Then, for the current base sample X, it selects the sample point C that is farthest away from it as the subsequent feature fusion object. CFPS is expressed by the following formula:

[0014]

[0015] In the formula, X represents the currently selected base sample, and i is the index of the current sample X in the batch. j Let D(X,X) represent the j-th sample. j ) represents X and X jThe Euclidean distance between them, argmax represents the distance between the functions D(X,X) and argmax. j The sample at which the maximum value is obtained;

[0016] The style feature mixing module, called MixStyle, normalizes Mask R-CNN training by perturbing the style information of source domain instances. It achieves style feature mixing by calculating the feature statistics of two instances and mixing them through random convex weights. The specific process is represented by the following formula:

[0017] γ mix =λσ(f)+(1-λ)σ(f′)

[0018] β mix =λμ(f)+(1-λ)μ(f′)

[0019]

[0020] In the formula, γ mix β represents the standard deviation obtained after shallow feature mixing. mix denoted as the average value obtained after shallow feature mixing, f and f′ represent the two sets of sample feature maps used for mixing, μ and σ represent the functions used to calculate the channel average and standard deviation, respectively, and λ is the random weight sampled from the beta distribution;

[0021] The internal channel mixing module, called InterMix, randomly selects two channels from each feature channel of the sample itself for feature mixing. The mixing method uses MixStyle to obtain more color style features outside the distribution.

[0022] Furthermore, the shallow feature blending module specifically performs the following operations:

[0023] The image feature C' with the greatest style difference is selected from the batch of images using CFPS. Then, C' is style-blended with the current image feature X' to obtain feature M. InterMix is ​​then used to blend the internal channel features of the newly generated feature M to generate more implicit color style features. In the training of the improved Mask R-CNN network, the shallow feature blending module is inserted between the shallow networks of the feature extraction module of the improved Mask R-CNN network. During the training process, the style blending transformation of the shallow feature blending module generates some color style features that the improved Mask R-CNN network has not seen before, thereby improving the generalization and robustness of the improved Mask R-CNN network.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] 1. Based on traditional style transformation methods, this invention proposes the Current Farthest Point Sampling Module (CFPS) to select mixed samples, thereby improving the feature mixing differences and diversity.

[0026] 2. This invention proposes an internal channel mixing module (InterMix), which further enhances the generated implicit style features.

[0027] 3. This invention incorporates an improved style feature mixing module (MixStyle) into the traditional target detection model method, and proposes an improved Mask R-CNN network based on shallow feature mixing for target detection of unlabeled bone marrow blood cells, which has high recognition accuracy and out-of-distribution generalization. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the logical flow of the method of the present invention.

[0029] Figure 2 The diagram shows the structure of the Mask R-CNN network to improve it. In the diagram, ResBlock is the residual module in the ResNet50 feature extraction network, RPN is the region recommendation module, and FPN is the feature pyramid module. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0031] In this embodiment, 20 cell categories were identified in bone marrow cells, including early erythroblasts, intermediate erythroblasts, late erythroblasts, primitive myelocytes, promyelocytes, abnormal promyelocytes, neutrophils, neutrophils, band neutrophils, neutrophils, eosinophils, eosinophils, primitive monocytes, monocytes, pre-existing lymphocytes, lymphocytes, plasma cells, myeloma cells, smear cells, and other cells.

[0032] like Figure 1As shown, this embodiment discloses a method for identifying bone marrow cells in microscopic images based on feature fusion. This method is based on an improved Mask R-CNN network to achieve target identification of bone marrow cells. The improved Mask R-CNN network is an improvement on the feature extraction module of the traditional Mask R-CNN network. Specifically, a shallow feature fusion module is added to the shallow network of the feature extraction module. The shallow feature fusion module is an improvement on the style feature fusion module (which can be called MixStyle). The specific improvements are as follows: ① A current farthest point sampling module (which can be called CFPS) is added before MixStyle. The CFPS calculates the Euclidean distance between each pair of image samples and selects the sample with the largest Euclidean distance to the current sample as input to MixStyle; ② An internal channel fusion module (which can be called InterMix) is added after MixStyle. The InterMix performs random style feature fusion between the various feature channels of the sample using MixStyle. The specific steps of this method are as follows:

[0033] 1) Obtain labeled bone marrow cell microscopic images to construct a dataset, where the annotations for each image include the location bounding box, category, and segmentation mask of each cell. Then, divide the dataset into a training set and a test set.

[0034] The categories are 20 types of cells, including early erythroblasts, intermediate erythroblasts, late erythroblasts, primitive granulocytes, promyelocytes, abnormal promyelocytes, neutrophils, neutrophils, band neutrophils, neutrophils, eosinophils, eosinophils, primitive monocytes, monocytes, pre-existing lymphocytes, lymphocytes, plasma cells, myeloma cells, smear cells, and other cells. The annotation includes the location border, category, and segmentation mask for each cell.

[0035] 2) Perform data augmentation on the images in the training set to obtain an augmented training set after shape and color transformations. Data augmentation techniques aim to simulate variations that may be encountered under various real-world imaging conditions, thereby enhancing the model's adaptability to different imaging settings. This step is crucial for improving the model's stability and accuracy in practical applications.

[0036] The data augmentation includes: a) randomly adjusting the image contrast; b) histogram equalization; c) inverting pixel values, i.e., changing pixel value x to 255-x; d) randomly rotating the image by an angle from 0 to 180 degrees to fill blank areas with 0; e) randomly selecting a value from 0 to 4, setting all bits below x in the RGB channels of each pixel to 0; f) randomly selecting a value from 0 to 255, inverting all bits above x; g) randomly adjusting the image brightness; h) randomly adjusting the image sharpness; and i) flipping the image horizontally or vertically with a probability of 0.5.

[0037] 3) The enhanced training set is fed into the improved Mask R-CNN network. More out-of-distribution color style features are generated through the shallow feature mixing module of the feature extraction module, and feature information of different types of bone marrow cells is obtained through the feature extraction module. The extracted feature information is fused with the feature pyramid module FPN for multi-scale feature fusion, and then the region recommendation module RPN is used to obtain the region of interest. Finally, the data is input into the classifier, bounding box regression module and mask segmentation module of the improved Mask R-CNN network to obtain the prediction results of bone marrow cells. Among them, binary cross-entropy and Smooth-L1 loss function are used to calculate the loss values ​​of classification and regression in backpropagation. After multiple iterations, the loss value is minimized to obtain the optimal model.

[0038] like Figure 2 As shown, the shallow feature blending module includes the following modules:

[0039] Current Farthest Point Sampling Module (CFPS): This module calculates the Euclidean distance between shallow features of each pair of images in the same batch. Then, for the current base sample X, it selects the sample point C that is furthest away from it as the subsequent feature fusion object. CFPS is expressed by the following formula:

[0040]

[0041] In the formula, X represents the currently selected base sample, and i is the index of the current sample X in the batch. j Let D(X,X) represent the j-th sample. j ) represents X and X j The Euclidean distance between them, argmax represents the distance between the functions D(X,X) and argmax. j The sample at which the maximum value is obtained;

[0042] Style Feature Mixing Module (also known as MixStyle): This module normalizes Mask R-CNN training by perturbing the style information of source domain instances. It achieves style feature mixing by calculating the feature statistics of two instances and mixing them through random convex weights. The specific process is represented by the following formula:

[0043] γ mix =λσ(f)+(1-λ)σ(f′)

[0044] β mix =λμ(f)+(1-λ)μ(f′)

[0045]

[0046] In the formula, γ mix β represents the standard deviation obtained after shallow feature mixing. mix denoted as the average value obtained after shallow feature mixing, f and f′ represent the two sets of sample feature maps used for mixing, μ and σ represent the functions used to calculate the channel average and standard deviation, respectively, and λ is the random weight sampled from the beta distribution.

[0047] Internal Channel Mixing Module (InterMix): Existing style fusion methods do not consider the feature style information of each channel within the sample features. However, there is a lot of potential pixel difference information between different channels. Utilizing this information can help us generate more potential new color styles while limiting the color style changes to exceed the realistic range. Therefore, we propose an internal channel mixing module (InterMix). This module randomly selects two channels from each feature channel of the sample itself for feature mixing using the MixStyle method, thereby obtaining more implicit style features.

[0048] The overall process of the shallow feature mixing module is as follows: From the batch of images, the current farthest point sampling module (CFPS) selects the image feature C' with the largest style difference. Then, C' is style-mixed with the current image feature X' to obtain feature M. Next, the internal channel mixing module (InterMix) is used to perform internal feature mixing on the newly generated feature M, thereby generating more implicit color style features. Specifically, during the training of the improved Mask R-CNN network, the shallow feature mixing module is inserted between the shallow layers of the improved Mask R-CNN network. During training, as the style mixing transformation of the shallow feature mixing module occurs, some color style features that the improved Mask R-CNN network has not yet seen are generated, thereby improving the generalization and robustness of the improved Mask R-CNN network.

[0049] 4) Input the test set into the optimal model to obtain prediction information, including: the location bounding box, category and segmentation mask information of each bone marrow cell. Draw this information onto the original image and mark the predicted bone marrow cell category above the location bounding box to complete the identification of bone marrow cells.

[0050] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for microscopic image bone marrow cell recognition based on shallow feature mixing, characterized in that, The method is based on an improved Mask R-CNN network to realize target recognition of bone marrow cells, the improved Mask R-CNN network is to improve the feature extraction module of the traditional Mask R-CNN network, that is, a shallow feature mixing module is added in the shallow network of the feature extraction module, the shallow feature mixing module is improved on the basis of a style feature mixing module, the style feature mixing module is called MixStyle, and specific improvements are as follows: A current farthest point sampling module is added before MixStyle, the current farthest point sampling module is called CFPS, the CFPS is obtained by calculating the Euclidean distance between the shallow features of two image samples, and the sample with the maximum Euclidean distance from the current sample is selected as the input of MixStyle; An internal channel mixing module is added after MixStyle, the internal channel mixing module is called InterMix, and the InterMix is random style feature mixing between each feature channel of the sample using MixStyle. The specific implementation of the microscopic image bone marrow cell recognition method comprises the following steps: 1) Obtain labeled bone marrow cell microscopic images to construct a data set, wherein the label of each image comprises the position frame, class and segmentation mask of each cell, and then divide the data set into a training set and a test set; 2) Data augmentation is performed on the images in the training set to obtain an enhanced training set after shape and color transformation; 3) The enhanced training set is input into the improved Mask R-CNN network, more distributed color style features are generated through the shallow feature mixing module of the feature extraction module, and the feature information of different types of bone marrow cells is obtained through the feature extraction module, the extracted feature information is subjected to feature multi-scale fusion with the feature pyramid module FPN, and then the region of interest is obtained through the region proposal network RPN, and finally the improved Mask R-CNN network is input into the classifier, the frame regression module and the mask segmentation module to obtain the prediction result of the bone marrow cell; wherein the binary cross entropy is used to calculate the loss value of the classification in the back propagation, and the Smooth-L1 loss function is used to calculate the loss value of the regression, and the optimal model is obtained after multiple iterations until the loss value is minimum; 4) The test set is input into the optimal model to obtain the prediction information, including the position frame, class and segmentation mask information of each bone marrow cell, the information is drawn on the original image, and the predicted bone marrow cell class is labeled above the position frame, thereby completing the recognition of the bone marrow cell.

2. The microscopic image bone marrow cell recognition method based on shallow feature mixing according to claim 1, characterized in that, The data augmentation comprises: randomly adjusting the contrast of the image, histogram equalization, reversing the pixel value, randomly rotating the image, randomly adjusting the brightness of the image, randomly adjusting the sharpening degree of the image, and flipping the image in the horizontal or vertical direction.

3. The microscopic image bone marrow cell recognition method based on shallow feature mixing according to claim 1, characterized in that, The shallow feature mixing module comprises: A current farthest point sampling module: this module calculates the Euclidean distance of the shallow features between two images in the same batch, and then selects the sample point C farthest from the current base sample X as the subsequent feature fusion object, and CFPS is represented by the following formula: ； where X represents the currently selected base sample, i is the index of the current sample X in the batch, represents the jth sample, represents the Euclidean distance between X and argmax represents the sample that makes the function take the maximum value. A style feature mixing module: this module standardizes Mask R-CNN training by disturbing the style information of the source domain instance, calculates the feature statistical information of two instances, and mixes them through random convex weights to achieve the purpose of style feature mixing, and the specific process is represented by the following formula: ； ； ； wherein, denotes the standard deviation after mixing the shallow features, denotes the mean value after mixing the shallow features, , denote two sets of sample feature maps for mixing, and denote functions for computing the channel mean and standard deviation, respectively, is a random weight sampled from a beta distribution; An internal channel mixing module: this module randomly selects two channels in each feature channel of the sample itself for feature mixing, and uses MixStyle for mixing, thereby obtaining more distributed color style features.

4. The microscopic image bone marrow cell recognition method based on shallow feature mixing according to claim 1, characterized in that, The shallow feature mixing module specifically performs the following operations: From the batch images, the image features C' with the greatest style difference are selected using the CFPS, and then the feature M is obtained by mixing the style of C' and the current image feature X'; the internal channel features of the newly generated feature M are mixed using the InterMix, so as to generate more implicit color style features; wherein, when the improved Mask R-CNN network is trained, the shallow feature mixing module is inserted between the shallow networks of the feature extraction module of the improved Mask R-CNN network, and some color style features that the improved Mask R-CNN network has not seen before are generated in the training process through the style mixing transformation of the shallow feature mixing module, so as to improve the generalization and robustness of the improved Mask R-CNN network.

Images