Leukocyte classification method, device, electronic device and storage medium
By combining the leukocyte classification model and the t-SNE algorithm for high-dimensional feature dimensionality reduction and clustering analysis, the regional attention layer is used to identify important areas, which solves the problem of failure to distinguish unknown cells in the existing technology, and improves the accuracy of leukocyte classification meet clinical testing needs.
Patent Information
- Application Number
- CN202411032506.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-30
AI Technical Summary
The existing deep learning methods are difficult to distinguish unknown categories of cells in leukocyte classification, resulting in low classification accuracy in clinical examinations and unable to meet actual needs.
The pre-trained leukocyte classification model combined with the t-SNE algorithm for high-dimensional feature dimensionality reduction and clustering analysis. The important areas are automatically identified through the regional attention layer, and unknown cells are screened out, and the high-dimensional features of known categories are input into all Connection layers are categorized more accurately.
Effectively distinguishing unknown cells into cells improves the accuracy of leukocyte classification, allowing the model to more accurately classify leukocytes and meet the needs of clinical testing.
Smart Images

Figure CN118552954B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of cell classification, and in particular to a method, device, electronic device and storage medium for classifying white blood cells. Background Art
[0002] White blood cell classification plays an important role in disease diagnosis and treatment. Manual microscopic examination is often used to identify and classify white blood cells, that is, the morphology, type and number of white blood cells in the sample are manually analyzed and counted. This processing method has low analysis efficiency and is easily affected by subjective factors, resulting in low classification accuracy. In view of this, the relevant technology introduces deep learning methods to establish classification models to automatically classify white blood cells based on images.
[0003] However, deep learning methods rely on complete data sets to train model networks. Generally, the data sets used for training only contain images of normal white blood cells of known categories. Due to the scarcity of cells of unknown categories such as abnormal white blood cells, it is impossible to collect enough data images to establish labels, so that the trained model can only classify normal white blood cells of known categories. Abnormal white blood cells and other unknown categories of cells will also be classified into known categories of normal white blood cells. As a result, the model itself cannot distinguish cells of unknown categories and cannot meet the actual needs of clinical testing.
[0004] There is currently no effective technical solution to the above problems. Summary of the invention
[0005] The purpose of the present application is to provide a leukocyte classification method, device, electronic device and storage medium to distinguish cells of unknown categories and then perform leukocyte classification processing.
[0006] In a first aspect, the present application provides a leukocyte classification method for performing leukocyte classification, the method comprising the following steps:
[0007] Inputting the cell image to be classified into a pre-trained leukocyte classification model, wherein the leukocyte classification model includes a stem portion, a trunk portion, and a fully connected layer connected in sequence;
[0008] Obtaining high-dimensional features of different cell data output by the backbone, and reducing the high-dimensional features into three-dimensional features to be analyzed based on the t-SNE algorithm;
[0009] Identify unknown category cells according to the distance relationship between the three-dimensional features to be analyzed and the cluster center, and select the remaining high-dimensional features corresponding to white blood cells, wherein the cluster center is the cluster center of the three-dimensional features of the known category white blood cells;
[0010] The high-dimensional features corresponding to the known categories of white blood cells are input into the fully connected layer for classification.
[0011] The white blood cell classification method of the present application combines the normal white blood cell classification model obtained through training and the t-SNE algorithm to perform white blood cell classification, and performs dimensionality reduction and cluster analysis on the high-dimensional features generated during the classification process of the white blood cell classification model, so that the white blood cell classification method of the present application has the function of distinguishing cells of unknown categories, and the remaining high-dimensional features corresponding to known categories of white blood cells are screened out and input into the fully connected layer for more accurate white blood cell classification.
[0012] The white blood cell classification method, wherein the main body is composed of a basic module, and the basic module includes a first convolution module, a regional attention layer and a second convolution module connected in sequence.
[0013] In this example, the regional attention layer can automatically identify and focus on important areas or information in the input data, thereby improving the performance and accuracy of the corresponding model decision. It can assign different attention weights to different areas of the input data, so that the model can pay more attention to important areas when processing data and ignore unimportant areas. This can help the model better capture the key information in the input data and improve the model's decision-making ability.
[0014] The white blood cell classification method, wherein the regional attention layer includes three parallel branches, two of which include an average pooling layer, two 1×1 convolutional layers, a BN layer and a HardSwish activation function layer connected in sequence, and the remaining parallel branches are directly connected branches.
[0015] The method for classifying white blood cells further comprises the following steps:
[0016] A heat map of the image area that the backbone focuses on when making predictions is obtained based on the Grad-CAM model.
[0017] In the white blood cell classification method, the step of identifying unknown category cells according to the distance relationship between the three-dimensional features to be analyzed and the cluster center, and screening the remaining high-dimensional features corresponding to the known category white blood cells is performed based on the following formula:
[0018] ;
[0019] Among them, P = 0 means that the three-dimensional feature to be analyzed corresponds to a known category of white blood cells, P = 1 means that the three-dimensional feature to be analyzed corresponds to an unknown category of cells, e is a natural constant, K is the total number of categories of known categories of white blood cells, corresponding to the total number of cluster centers, a i is the average distance between the three-dimensional feature of the leukocyte of the i-th known category and the corresponding cluster center; di is the distance between the three-dimensional feature to be analyzed and the cluster center of the i-th known category of white blood cells.
[0020] The white blood cell classification method, wherein the stem includes three 3×3 convolutional layers.
[0021] The leukocyte classification method, wherein the training process of the leukocyte classification model comprises:
[0022] Constructing an initial model, and training the initial model based on a pre-constructed cell image dataset to obtain a basic classification model;
[0023] Simplifying the basic classification model based on structural reparameterization to obtain a simplified model;
[0024] The simplified model is subjected to Int8 quantization processing, and the simplified model after Int8 quantization processing is rebuilt in the TensorRT framework to obtain the white blood cell classification model.
[0025] In a second aspect, the present application further provides a leukocyte classification device for classifying leukocytes, the device comprising:
[0026] A first input module is used to input the cell image to be classified into a pre-trained leukocyte classification model, wherein the leukocyte classification model includes a stem, a trunk and a fully connected layer connected in sequence;
[0027] A dimensionality reduction module, used for obtaining high-dimensional features of different cell data output by the backbone, and reducing the high-dimensional features into three-dimensional features to be analyzed based on a t-SNE algorithm;
[0028] A screening module, used to identify unknown category cells according to the distance relationship between the three-dimensional features to be analyzed and the cluster center, and screen out the remaining high-dimensional features corresponding to the known category of white blood cells, wherein the cluster center is the cluster center of the three-dimensional features of the known category of white blood cells;
[0029] The second input module is used to input the high-dimensional features corresponding to the known categories of white blood cells into the fully connected layer for classification.
[0030] The white blood cell classification device of the present application combines the normal white blood cell classification model obtained through training and the t-SNE algorithm to perform white blood cell classification, and performs dimensionality reduction and cluster analysis on the high-dimensional features generated during the classification process of the white blood cell classification model, so that the white blood cell classification device of the present application has the function of distinguishing cells of unknown categories, and screens out the remaining high-dimensional features corresponding to known categories of white blood cells and inputs them into the fully connected layer for more accurate white blood cell classification.
[0031] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are executed.
[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps in the method provided in the first aspect are executed.
[0033] As can be seen from the above, the present application provides a white blood cell classification method, device, electronic device and storage medium, wherein the white blood cell classification method combines the normal white blood cell classification model obtained through training and the t-SNE algorithm to perform white blood cell classification, and performs dimensionality reduction and cluster analysis on the high-dimensional features generated during the classification process of the white blood cell classification model, so that the white blood cell classification method of the present application has the function of distinguishing cells of unknown categories, and the remaining high-dimensional features corresponding to known categories of white blood cells are screened out and input into the fully connected layer for more accurate white blood cell classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 A flowchart of a method for classifying white blood cells provided in an embodiment of the present application.
[0035] Figure 2 A schematic diagram of the network structure of the leukocyte classification model used in the leukocyte classification method provided in an embodiment of the present application.
[0036] Figure 3 Schematic diagram of the network structure of the regional attention layer.
[0037] Figure 4 Schematic diagram of the structural reparameterization process of the basic module.
[0038] Figure 5 Heatmap of image regions that different classification models focus on.
[0039] Figure 6 The spatial distribution and three-directional projection diagrams of the three-dimensional characteristics of known types of white blood cells.
[0040] Figure 7 The spatial distribution of the three-dimensional features to be analyzed corresponding to the unknown category cells and the three-directional projection diagrams.
[0041] Figure 8 A schematic diagram of the structure of a white blood cell classification device provided in an embodiment of the present application.
[0042] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0043] Figure numerals: 201, first input module; 202, dimensionality reduction module; 203, screening module; 204, second input module; 301, processor; 302, memory; 303, communication bus. DETAILED DESCRIPTION
[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0045] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0046] First, please refer to Figure 1 Some embodiments of the present application provide a method for classifying leukocytes, which is used for classifying leukocytes. The method comprises the following steps:
[0047] S1. Inputting the cell image to be classified into a pre-trained leukocyte classification model, the leukocyte classification model includes a stem, a trunk, and a fully connected layer connected in sequence;
[0048] S2, obtaining high-dimensional features of different cell data output by the main body, and reducing the high-dimensional features to three-dimensional features to be analyzed based on the t-SNE algorithm;
[0049] S3, identifying the unknown category cells according to the distance relationship between the three-dimensional features to be analyzed and the cluster center, and screening out the remaining high-dimensional features corresponding to the known category white blood cells, the cluster center being the cluster center of the three-dimensional features of the known category white blood cells;
[0050] S4. Input the high-dimensional features corresponding to the known categories of white blood cells into the fully connected layer for classification.
[0051] Specifically, the white blood cell classification model is a pre-trained image classification model, which can be used to classify white blood cells in images containing white blood cells, such as for distinguishing lymphocytes, basophils, neutrophils, eosinophils and monocytes, etc. In the embodiment of the present application, it is preferably used to distinguish 9 types of cells including segmented neutrophils, monocytes, lymphocytes, primitive lymphocytes, primitive red blood cells, primitive granulocytes, eosinophils, basophils and band-shaped neutrophils.
[0052] More specifically, the white blood cell classification model can be trained using an existing model training method, such as the gradient descent method. After the training is completed, the above steps S3 and S4 can be introduced for secondary training for fine-tuning to make the white blood cell classification model more suitable for classifying the above categories of white blood cells.
[0053] More specifically, the cell image to be classified is an image that needs to be classified for white blood cells, and should contain an appropriate amount of cell data.
[0054] More specifically, the stem and trunk of the leukocyte classification model are functional modules that divide the network structure based on feature processing, wherein the stem is a module used to extract low-level features of the input cell image to be classified, that is, in the initial stage, local feature extraction processing is performed on the cell image to be classified to obtain the relationship between adjacent pixels in the image, and represented by low-level features, such as edges, textures, etc. These low-level features are usually local and specific, and can provide a basis for subsequent trunk operations, so that the trunk can further learn and extract higher-level features; the trunk is used to obtain higher-level features based on the extraction and / or combination of low-level features transmitted from the stem. These higher-level features are generally manifested as high-dimensional features, which are usually more abstract and complex than the features extracted from the stem, and can better represent the content of the input image to provide support for the final classification task; the fully connected layer is located at the last part of the leukocyte classification model, and is used to receive the feature map composed of high-dimensional features output by the trunk, and convert it into a one-dimensional feature vector for classification. It is generally used in combination with the softmax function to convert the output one-dimensional feature vector into a probability distribution to achieve leukocyte classification.
[0055] More specifically, based on the module composition and corresponding functional roles of the above-mentioned white blood cell classification model, it can be known that the white blood cell classification model is based on a one-dimensional feature vector converted from high-dimensional features of different cell data for classification. In essence, it is used to analyze the probability that the cell data in the cell image to be classified belongs to different known categories of white blood cells to classify white blood cells. For cells other than known categories of white blood cells (such as abnormal white blood cells), corresponding probabilities are also generated and white blood cell division is performed, thereby affecting the accuracy of the classification results; therefore, the white blood cell classification method of the embodiment of the present application introduces steps S3-S4 in the white blood cell classification process to remove these unknown category cells. The unknown category cells refer to cells other than the known category of white blood cells that the white blood cell classification model is suitable for classification, that is, cells other than the aforementioned 9 categories of white blood cells.
[0056] More specifically, the t-SNE (t-distributed Stochastic Neighbor Embedding) algorithm is a dimensionality reduction algorithm that can map data in a high-dimensional space to a low-dimensional space and retain the local characteristics of the data set. The white blood cell classification method in the embodiment of the present application mainly uses the t-SNE algorithm to reduce the high-dimensional features into three-dimensional features to be analyzed, and can also be further visualized; wherein, the high-dimensional features corresponding to different cell data output by the main body can effectively represent the characteristics of the corresponding cell data, and the three-dimensional features to be analyzed obtained by dimensionality reduction based on the t-SNE algorithm retain the local characteristics of the corresponding high-dimensional features, and the cluster center is the cluster center of the three-dimensional features of the known category of white blood cells, that is, the cluster center is determined by clustering a series of three-dimensional features obtained by dimensionality reduction based on the high-dimensional features output by the main body according to the image of the known category of white blood cells, and the distance relationship between the three-dimensional features to be analyzed and the cluster center reflects the distance between the three-dimensional features to be analyzed and the cluster center. Analyze the similarity between the three-dimensional features and the three-dimensional features in the cluster corresponding to the cluster center; in the embodiment of the present application, there are multiple cluster centers, corresponding to the total number of types of known categories of white blood cells that can be distinguished by the white blood cell classification model, that is, different clusters correspond to different categories of known categories of white blood cells; if the cell data to be classified belongs to a known category of white blood cells, then the three-dimensional features to be analyzed corresponding to the cell data should be relatively close to at least one cluster center, so according to the distance relationship between the three-dimensional features to be analyzed and the cluster center, it can be quickly determined whether the high-dimensional features corresponding to these three-dimensional features to be analyzed are high-dimensional features of known categories of white blood cells, so that the white blood cell classification method of the embodiment of the present application can quickly remove the high-dimensional features corresponding to the unknown category cells based on step S3, and filter out the remaining high-dimensional features corresponding to the known category of white blood cells and input them into the fully connected layer for more accurate white blood cell classification.
[0057] More specifically, step S3 is equivalent to performing dimensionality reduction and clustering on the white blood cell image features extracted by the white blood cell classification model and the normal white blood cell features (the features corresponding to the aforementioned known category of white blood cells) to compare the differences between the two in order to distinguish whether the corresponding cell data belongs to a known category of white blood cells or an unknown category of cells, so that the white blood cell classification method of the embodiment of the present application has the function of distinguishing unknown category cells.
[0058] The leukocyte classification method of the embodiment of the present application combines the normal leukocyte classification model obtained through training and the t-SNE algorithm to perform leukocyte classification, and performs dimensionality reduction and cluster analysis on the high-dimensional features generated during the classification process of the leukocyte classification model, so that the leukocyte classification method of the embodiment of the present application has the function of distinguishing cells of unknown categories, and the remaining high-dimensional features corresponding to known categories of leukocytes are screened out and input into the fully connected layer for more accurate leukocyte classification.
[0059] It should be noted that, in actual use, cells of unknown category are generally abnormal white blood cells. In the embodiment of the present application, there is no need to consider providing data labels of abnormal white blood cells to assist in the training of the white blood cell classification model, so as to achieve the distinction of abnormal white blood cells, which effectively solves the problem of insufficient recognition ability of the white blood cell classification model due to insufficient abnormal cell data.
[0060] In some preferred embodiments, the backbone is composed of basic modules, which include a first convolution module, a regional attention layer, and a second convolution module connected in sequence.
[0061] Specifically, the regional attention layer has a regional attention mechanism, that is, when processing input data, it can automatically identify and focus on important areas or information in the input data, thereby improving the performance and accuracy of the corresponding model decision. It can assign different attention weights to different areas of the input data, so that the model can pay more attention to important areas when processing data and ignore unimportant areas, which can help the model better capture the key information in the input data and improve the decision-making ability of the model.
[0062] More specifically, if Figure 4 As shown, the regional attention layer connects the first convolution module and the second convolution module, and can identify the important regional information in the corresponding features according to the features output by the first convolution module, and then adjust the weight distribution so that the second convolution module pays more attention to the important regional information for feature processing, so that the high-dimensional features output by the main body can better characterize the characteristics of the cell data to improve the accuracy of white blood cell classification.
[0063] In some preferred embodiments, the first convolution module includes multiple basic parallel branches and one jump connection branch containing a batch normalization (Batch Normalization, BN) layer, wherein the number of basic parallel branches can be given according to the complexity of the visual task; each basic parallel branch includes a convolution layer and a batch normalization layer connected to the convolution layer.
[0064] In some preferred embodiments, the second convolution module includes a plurality of basic parallel branches and a jump connection branch including a batch normalization (Batch Normalization, BN) layer, wherein the number of basic parallel branches can be given according to the complexity of the visual task.
[0065] In some preferred embodiments, the convolutions of the convolution layers of the basic parallel branches in the first convolution module are all 3×3Depthwise convolutions, and the convolutions of the convolution layers of the basic parallel branches in the first convolution module are all 1×1Depthwise convolutions (depthwise convolutions).
[0066] Specifically, in the embodiments of the present application, Depthwise convolution helps to reduce the complexity of the model, reduce memory consumption, and reduce the risk of overfitting, and also facilitates subsequent structural reparameterization.
[0067] In some preferred embodiments, the number of basic parallel branches in the first convolution module and the number of basic parallel branches in the second convolution module are both 4.
[0068] Specifically, in the model training stage, the feature vector input into the first convolution module is input into the four basic parallel branches and batch normalization layer therein for parallel processing and batch normalization to obtain four feature vectors. Then, these four feature vectors are added to obtain a combined feature vector, and the ReLU activation function is applied to it. The calculation method of the ReLU activation function is as follows:
[0069] f(x)=max(0,x) (1)
[0070] Among them, x represents the input; after activation, the first convolution module passes the combined feature vector processed by the ReLU activation function to the regional attention layer.
[0071] In some preferred embodiments, Figure 2 As shown, the main body includes a four-layer structure, and the four-layer structure includes 2, 2, 4, and 1 basic modules along the feature transfer direction, corresponding to Block1, Block2, Block3, and Block4 in the figure respectively.
[0072] Specifically, in this embodiment, the number of convolution cores of basic modules in different layer structures is different, and the number of convolution cores (convolution width) of basic modules in the same layer structure is the same, such as the number of convolution cores of Block1_1 and Block1_2 in the first layer structure is 64 both.
[0073] More specifically, the number of structural layers of the main body and the number of basic modules in each layer can be designed according to usage requirements. The above design method can ensure that the white blood cell classification model obtained through training can accurately and efficiently complete the white blood cell classification task.
[0074] In some preferred embodiments, Figure 3 As shown in Figure 1, the regional attention layer includes three parallel branches, two of which include an average pooling layer, two 1×1 convolutional layers, a BN layer, and a HardSwish activation function layer connected in sequence, and the remaining parallel branches are directly connected branches.
[0075] Specifically, the above two 1×1 convolutional layers are regular convolutions.
[0076] More specifically, based on the foregoing, the regional attention layer receives a combined feature vector processed by the ReLU activation function, which is processed in turn by two non-directly connected parallel branches, namely, an average pooling layer, two 1×1 convolutional layers, a BN layer, and a HardSwish activation function layer, to obtain two feature vectors. Finally, the two feature vectors are multiplied with the combined feature vector transmitted by the direct branch to obtain a new combined feature vector, so as to realize the feature learning process to read the leukocyte cell morphology and subcellular structure localization in a targeted manner.
[0077] More specifically, the HardSwish activation function layer is calculated as follows:
[0078] (2)
[0079] Here, x represents the input.
[0080] More specifically, the average pooling layers in the two parallel branches are the X-direction average pooling (XAvgPool) and the Y-direction average pooling (Y AvgPool) shown in the figure.
[0081] In some preferred embodiments, the method further comprises the following steps:
[0082] S4. Based on the Grad-CAM model, obtain the heat map of the image area that the main cadre pays attention to when making predictions.
[0083] Specifically, Grad-CAM (Gradient-weighted Class Activation Mapping) is a technology used in deep learning to visualize and understand model decisions; the white blood cell classification method of the embodiment of the present application uses the Grad-CAM model to visualize the image area that the white blood cell classification model pays attention to when making predictions, so as to observe the decision-making mechanism inside the white blood cell classification model and make it explainable, so that the traditional attention mechanism is no longer in a "black box" state, that is, in an unobservable state.
[0084] More specifically, in order to more clearly reflect the areas that the white blood cell classification model focuses on when making decisions and their corresponding importance, the white blood cell classification method of the embodiment of the present application uses a heat map to represent the image areas that the white blood cell classification model focuses on when making predictions.
[0085] In some preferred embodiments, step S4 comprises:
[0086] S41, extracting a feature map of a leukocyte classification model for processing a cell image to be classified based on forward propagation;
[0087] S42, calculating the gradient weight of the category score with respect to the feature map based on back propagation;
[0088] S43, multiplying multiple feature maps and corresponding gradient weights, and summing them up to obtain a heat map.
[0089] Specifically, the purpose of the above processing is to obtain the area that the white blood cell classification model focuses on when making decisions. It can also be used for comparative analysis in different classification types to distinguish the advantages and disadvantages of different models, such as Figure 5 As shown, the basic network in the figure refers to the existing white blood cell classification model, which is directly composed of multiple conventional convolutions. The basic network + attention mechanism in the figure refers to the existing white blood cell classification model combined with the classification model determined by the main body in this application. The white blood cell classification model of this application in the figure refers to the classification model determined by the basic network combined with the stem and the main body in this application; according to the diagram, it can be seen that the white blood cell classification model of the embodiment of the present application can more accurately focus on the image features of the cell data that need to be distinguished; among them, adding the stem can make the focus area more fit the entire cell area, indicating that the stem can effectively expand the receptive field of the model and improve the range of the focus area; adding a main body with a regional attention layer can enable the model to pay attention to more cell nucleus area features.
[0090] In some preferred embodiments, step S3 comprises:
[0091] S31, extracting the cluster center and the average distance from the three-dimensional data within each cluster to the cluster center, the calculation process is as follows:
[0092] Based on the high-dimensional features of known types of white blood cells output by the main body of the white blood cell classification model, the 3D features of known types of white blood cells are obtained by dimensionality reduction based on the t-SNE algorithm;
[0093] The three-dimensional features of known categories of white blood cells are clustered based on the K-means algorithm, where the number of cluster centers of the K-means algorithm is set in the range of [1,30], and the performance of each candidate cluster number is evaluated based on the silhouette coefficient to automatically select the optimal cluster number;
[0094] Calculate the cluster center of each cluster, and calculate the average distance from the three-dimensional data within each cluster to the corresponding cluster center;
[0095] S32, calculating the distance between the three-dimensional feature to be analyzed and different cluster centers, and screening out high-dimensional features corresponding to known categories of white blood cells according to the relationship between the distance and the average distance.
[0096] Specifically, this implementation can quickly determine whether the corresponding cell data belongs to a known category of white blood cells by analyzing whether there is a cluster center with an expected distance from the three-dimensional feature to be analyzed.
[0097] More specifically, after obtaining the three-dimensional features of known categories of white blood cells based on t-SNE algorithm and setting them in three-dimensional space, the distribution of these three-dimensional features is as follows: Figure 6 As shown, after placing the three-dimensional feature to be analyzed into the distribution map, the position difference between the three-dimensional feature to be analyzed and the three-dimensional feature of the known category of white blood cells can be intuitively observed. Figure 7 The black dots in the middle represent the distribution of the three-dimensional features to be analyzed corresponding to the cells belonging to the unknown category calculated in step S32.
[0098] In general, the distances obtained by the above calculations can be compared and analyzed one by one with the average distances. In order to further improve the analysis efficiency, in some preferred embodiments, the unknown category cells are identified according to the distance relationship between the three-dimensional features to be analyzed and the cluster center, and the remaining high-dimensional features corresponding to the known categories of white blood cells are screened based on the following formula:
[0099] (3)
[0100] Among them, P = 0 means that the three-dimensional feature to be analyzed corresponds to a known category of white blood cells, P = 1 means that the three-dimensional feature to be analyzed corresponds to an unknown category of cells, e is a natural constant, K is the total number of categories of known categories of white blood cells, corresponding to the total number of cluster centers, a i is the average distance between the three-dimensional feature of the leukocyte of the i-th known category and the corresponding cluster center; d iis the distance between the three-dimensional feature to be analyzed and the cluster center of the i-th known category of white blood cells.
[0101] It should be noted that step S32 can also be calculated using the above formula.
[0102] Specifically, based on the above content, it can be seen that the above clustering is determined based on the silhouette coefficient, that is, the clustering ranges determined based on the corresponding average distances between different clusters do not intersect. In this case, if the cell data belongs to the known category of white blood cells, its corresponding three-dimensional feature to be analyzed has only one The value of is less than 1, which makes P output 0. On the contrary, if the cell data belongs to an unknown category of cells, the corresponding three-dimensional feature to be analyzed corresponds to The values of are all greater than 1, so that P output is 1. The white blood cell classification method of the embodiment of the present application can quickly screen out high-dimensional features corresponding to known categories of white blood cells based on the above formula.
[0103] In some preferred embodiments, the stem includes three 3×3 convolutional layers.
[0104] Specifically, the 3×3 convolution layer is a regular convolution layer. Setting three 3×3 convolution layers at the stem can increase the receptive field of the white blood cell classification model (i.e., the entire network). The equivalent receptive field size is the same as that of a single-layer 7×7 regular convolution, but the increase in the number of parameters is less than that of a single-layer 7×7 regular convolution, which is beneficial to improving the training efficiency and processing efficiency of the model.
[0105] In some preferred embodiments, the training process of the leukocyte classification model includes:
[0106] A1. Build an initial model and train the initial model based on a pre-built cell image dataset to obtain a basic classification model;
[0107] A2, simplifying the basic classification model based on structural reparameterization to obtain a simplified model;
[0108] A3. Perform Int8 quantization on the streamlined model and rebuild the streamlined model after Int8 quantization in the TensorRT framework to obtain the white blood cell classification model.
[0109] Specifically, the initial model is a neural network model; the basic classification model obtained by training in step A1 can be directly used as a white blood cell classification model, that is, in some other embodiments, the training process of the white blood cell classification model only includes step A1; the purpose of adding steps A2 and A3 to the white blood cell classification method in the embodiment of the present application is to improve the classification efficiency of the white blood cell classification model finally obtained by simplifying and lightweight processing.
[0110] More specifically, the pre-constructed cell image dataset is a cell image dataset that contains white blood cells and has annotated white blood cell categories, and is divided into a training set, a validation set, and a test set to facilitate training, validation, and testing.
[0111] More specifically, step A1 preferably uses the gradient descent method to train and optimize the initial model, that is, when the initial model converges during the training process and the loss value no longer changes, it is saved as the basic classification model.
[0112] More specifically, in this embodiment, the structure of the basic classification model corresponds to the structure of the aforementioned white blood cell classification model, that is, the first convolution module includes a plurality of basic parallel branches and a skip connection branch containing a batch normalization layer, and the second convolution module includes a plurality of basic parallel branches and a skip connection branch containing a batch normalization layer; Figure 4 As shown, the structural reparameterization process of step A2 is actually a processing process of structural reparameterization of the first convolution module and the second convolution module, wherein the first convolution module is preferably reparameterized as a 3×3Depthwise convolution layer, and the second convolution module is preferably reparameterized as a 1×1Depthwise convolution layer; the structural reparameterization is based on the properties that convolution calculations satisfy the distributive law, associative law, and symmetry, and the parameters of multiple convolution layers combined with the BN layer are added and assigned to a single convolution layer to generate an equivalent single-branch structure used in the reasoning stage, which brings faster reasoning speed while maintaining the accuracy unchanged.
[0113] More specifically, the structural reparameterization process can be divided into three stages:
[0114] B1, merge the basic parallel branches into a convolutional layer;
[0115] B2, merge multiple convolutional layers reconstructed based on basic parallel branches into a merged convolutional layer;
[0116] B3, merge the merged convolution layer and the skip connection branch into a Depthwise convolution layer;
[0117] Specifically, the main purpose of the above processing is to remove the BN layer to reduce the inference delay in the inference stage.
[0118] More specifically, the streamlined model obtained in step A2 can efficiently complete the classification reasoning of flow cells and can be used directly in the reasoning stage, that is, in some embodiments, it can be directly used as the white blood cell classification model in step S1; in order to further improve the classification reasoning efficiency of the model, the white blood cell classification method of the embodiment of the present application performs Int8 quantization on the streamlined model, and constructs the final white blood cell classification model in the TensorRT framework based on the streamlined model after Int8 quantization as an inference engine, so as to make full use of GPU hardware acceleration and realize efficient real-time classification reasoning to complete white blood cell classification. The inference engine can be extended to embedded devices and mobile terminals for deployment, so it can be deployed in cell classification instruments, such as edge computing devices Jetson AGX Orin.
[0119] More specifically, Int8 quantization processing is used to map the neural network weight file of the streamlined model into int8 data. In an embodiment of the present application, the network weight file is preferably converted into an ONNX format file, which has good compatibility and can be used with a variety of deep learning frameworks so that it can be flexibly deployed in different objects; parsing the ONNX format file can build a white blood cell classification model corresponding to the streamlined model in the TensorRT (NVIDIA TensorRT) framework. The white blood cell classification model used as an inference engine can make full use of GPU hardware acceleration and has efficient real-time inference performance.
[0120] In some preferred embodiments, the loss function used in the initial model training process is a cross entropy loss function, which is expressed as:
[0121] (4)
[0122] Among them, H is the loss value, n is the sample number in the training set, and p n is the category value of sample n, q n is the predicted category of sample n using the initial model, where N is the total number of samples in the training set.
[0123] Second, please refer to Figure 8 Some embodiments of the present application also provide a leukocyte classification device for classifying leukocytes, the device comprising:
[0124] A first input module 201 is used to input the cell image to be classified into a pre-trained leukocyte classification model, where the leukocyte classification model includes a stem, a trunk and a fully connected layer connected in sequence;
[0125] A dimension reduction module 202 is used to obtain high-dimensional features of different cell data output by the main body, and reduce the high-dimensional features to three-dimensional features to be analyzed based on the t-SNE algorithm;
[0126] The screening module 203 is used to identify the unknown category of cells according to the distance relationship between the three-dimensional features to be analyzed and the cluster center, and screen out the remaining high-dimensional features corresponding to the known category of white blood cells, and the cluster center is the cluster center of the three-dimensional features of the known category of white blood cells;
[0127] The second input module 204 is used to input the high-dimensional features corresponding to the known categories of white blood cells into the fully connected layer for classification.
[0128] The white blood cell classification device of the embodiment of the present application combines the normal white blood cell classification model obtained through training and the t-SNE algorithm to perform white blood cell classification, and performs dimensionality reduction and cluster analysis on the high-dimensional features generated during the classification process of the white blood cell classification model, so that the white blood cell classification device of the embodiment of the present application has the function of distinguishing cells of unknown categories, and screens out the remaining high-dimensional features corresponding to known categories of white blood cells and inputs them into the fully connected layer for more accurate white blood cell classification.
[0129] In some preferred embodiments, the device further comprises:
[0130] The visualization explanation module (not shown) is used to obtain the heat map of the image area that the main cadre pays attention to when making predictions based on the Grad-CAM model.
[0131] In some preferred embodiments, the leukocyte classification device of the embodiments of the present application is used to execute the leukocyte classification method provided in the first aspect above.
[0132] Third, please refer to Fig. 9 Some embodiments of the present application also provide a structural diagram of an electronic device. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores computer-readable instructions executable by the processor 301. When the electronic device is running, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation of the above embodiments.
[0133] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method in any optional implementation of the above embodiment is executed. The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.
[0134] In summary, the embodiments of the present application provide a white blood cell classification method, device, electronic device and storage medium, wherein the white blood cell classification method combines the normal white blood cell classification model obtained through training and the t-SNE algorithm to perform white blood cell classification, and performs dimensionality reduction and cluster analysis on the high-dimensional features generated during the classification process of the white blood cell classification model, so that the white blood cell classification method of the embodiments of the present application has the function of distinguishing cells of unknown categories, and screens out the remaining high-dimensional features corresponding to known categories of white blood cells and inputs them into the fully connected layer for more accurate white blood cell classification.
[0135] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0136] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0137] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0138] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.
[0139] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A method for classifying leukocytes, for classifying leukocytes, characterized in that: The method comprises the following steps: S1. Inputting the cell image to be classified into a pre-trained leukocyte classification model, wherein the leukocyte classification model comprises a stem, a trunk and a fully connected layer connected in sequence; S2, obtaining high-dimensional features of different cell data output by the main body, and reducing the high-dimensional features into three-dimensional features to be analyzed based on the t-SNE algorithm; S3, identifying unknown category cells according to the distance relationship between the three-dimensional feature to be analyzed and the cluster center, and screening out the remaining high-dimensional features corresponding to the known category white blood cells, wherein the cluster center is the cluster center of the three-dimensional features of the known category white blood cells; S4, inputting the high-dimensional features corresponding to the known categories of white blood cells into the fully connected layer for classification; Step S3 includes: S31, extracting the cluster center and the average distance from the three-dimensional data within each cluster to the cluster center; S32, calculating the distance between the three-dimensional feature to be analyzed and different cluster centers, and screening out high-dimensional features corresponding to known categories of white blood cells according to the relationship between the distance and the average distance, and the step of screening high-dimensional features is based on the following formula: ; Among them, P = 0 means that the three-dimensional feature to be analyzed corresponds to a known category of white blood cells, P = 1 means that the three-dimensional feature to be analyzed corresponds to an unknown category of cells, e is a natural constant, K is the total number of categories of known categories of white blood cells, corresponding to the total number of cluster centers, a i is the average distance between the three-dimensional feature of the leukocyte of the i-th known category and the corresponding cluster center; d i is the distance between the three-dimensional feature to be analyzed and the cluster center of the i-th known category of white blood cells.
2. The method for classifying leukocytes according to claim 1, characterized in that: The main body is composed of basic modules, which include a first convolution module, a regional attention layer and a second convolution module connected in sequence.
3. The method for classifying leukocytes according to claim 2, characterized in that: The regional attention layer includes three parallel branches, two of which include an average pooling layer, two 1×1 convolutional layers, a BN layer and a HardSwish activation function layer connected in sequence, and the remaining parallel branches are directly connected branches.
4. The method for classifying leukocytes according to claim 2, characterized in that: The method further comprises the following steps: A heat map of the image area that the backbone focuses on when making predictions is obtained based on the Grad-CAM model.
5. The method for classifying leukocytes according to claim 1, characterized in that: The stem includes three 3×3 convolutional layers.
6. The method for classifying leukocytes according to claim 1, characterized in that: The training process of the leukocyte classification model includes: Constructing an initial model, and training the initial model based on a pre-constructed cell image dataset to obtain a basic classification model; Simplifying the basic classification model based on structural reparameterization to obtain a simplified model; The simplified model is subjected to Int8 quantization processing, and the simplified model after Int8 quantization processing is rebuilt in the TensorRT framework to obtain the white blood cell classification model.
7. A leukocyte classification device for classifying leukocytes, characterized in that: The device comprises: A first input module is used to input the cell image to be classified into a pre-trained leukocyte classification model, wherein the leukocyte classification model includes a stem, a trunk and a fully connected layer connected in sequence; A dimensionality reduction module, used for obtaining high-dimensional features of different cell data output by the backbone, and reducing the high-dimensional features into three-dimensional features to be analyzed based on a t-SNE algorithm; A screening module, used to identify unknown category cells according to the distance relationship between the three-dimensional features to be analyzed and the cluster center, and screen out the remaining high-dimensional features corresponding to the known category of white blood cells, wherein the cluster center is the cluster center of the three-dimensional features of the known category of white blood cells; A second input module, used for inputting the high-dimensional features corresponding to the known categories of white blood cells into the fully connected layer for classification; The step of identifying unknown category cells according to the distance relationship between the three-dimensional feature to be analyzed and the cluster center, and screening out the remaining high-dimensional features corresponding to known category white blood cells includes: Extract the cluster center and the average distance from the three-dimensional data within each cluster to the cluster center; The distance between the three-dimensional feature to be analyzed and the different cluster centers is calculated, and the high-dimensional features corresponding to the known categories of white blood cells are screened out according to the relationship between the distance and the average distance. The high-dimensional features are screened based on the following formula: ; Among them, P = 0 means that the three-dimensional feature to be analyzed corresponds to a known category of white blood cells, P = 1 means that the three-dimensional feature to be analyzed corresponds to an unknown category of cells, e is a natural constant, K is the total number of categories of known categories of white blood cells, corresponding to the total number of cluster centers, a i is the average distance between the three-dimensional feature of the leukocyte of the i-th known category and the corresponding cluster center; d i is the distance between the three-dimensional feature to be analyzed and the cluster center of the i-th known category of white blood cells.
8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 6 are executed.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are executed.
