Multi-domain unbalanced blood cell classification method and device and storage medium

Abstract

The invention discloses a multi-domain unbalanced blood cell classification method and device and a storage medium. The method comprises the following steps: acquiring a to-be-classified cell image; inputting the to-be-classified cell image into a blood cell classification model to obtain a blood cell classification result; wherein the blood cell classification model is obtained through the following steps: obtaining a multi-domain unbalanced blood cell image set; performing dyeing enhancement processing on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each blood cell image; performing local alignment processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image, and calculating local alignment loss; domain invariant contrast learning processing is carried out, and domain invariant supervised contrast loss is calculated; and according to the local alignment loss and the domain invariant supervised contrast loss, performing multi-stage training processing on a preset deep learning model to obtain a blood cell classification model. The blood cell classification is realized, and the classification accuracy is improved. The method can be widely applied to the technical field of image recognition.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a multi-domain unbalanced blood cell classification method, device and storage medium. Background Art

[0002] Blood cells, also known as "blood corpuscles", are cells that exist in the blood and can flow throughout the body with the blood. Traditional blood cell classification uses manual inspection under a microscope to identify blood cells, which is a repetitive, labor-intensive and time-consuming process. In traditional classification models based on deep learning, since blood cell images collected from different laboratories usually show significant domain shift and data imbalance, the domain shift will cause the generalization performance of traditional classification models to decline and perform poorly on unseen data sets. Data imbalance also causes traditional classification models to perform poorly on minority classes and have low classification accuracy.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiments of the present invention provide a multi-domain unbalanced blood cell classification method, device and storage medium, which effectively improve the classification accuracy and model precision.

[0005] On the one hand, an embodiment of the present invention provides a multi-domain unbalanced blood cell classification method, comprising the following steps:

[0006] Acquire the cell image to be classified;

[0007] Inputting the image of cells to be classified into the blood cell classification model to obtain the blood cell classification result;

[0008] Wherein, the blood cell classification model is obtained by the following steps:

[0009] Acquire a multi-domain unbalanced blood cell image set, wherein the multi-domain unbalanced blood cell image set includes a plurality of blood cell images;

[0010] Performing dye enhancement processing on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each of the blood cell images, wherein the domain transformation sample set includes a plurality of domain transformation samples;

[0011] performing local alignment processing according to the blood cell image and a domain transformation sample set corresponding to the blood cell image, and calculating a local alignment loss;

[0012] performing domain-invariant contrast learning processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image, and calculating the domain-invariant supervised contrast loss;

[0013] According to the local alignment loss and the domain-invariant supervised contrast loss, a preset deep learning model is subjected to multi-stage training processing to obtain the blood cell classification model.

[0014] In some embodiments, performing dye enhancement processing on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each of the blood cell images includes:

[0015] Performing image decomposition on each of the blood cell images in the multi-domain unbalanced blood cell image set to obtain a corresponding density map and a staining matrix;

[0016] According to the number of clusters, clustering the plurality of staining matrices using a preset clustering method to obtain a plurality of clusters;

[0017] Setting a domain label for the blood cell image corresponding to the staining matrix in each of the clusters;

[0018] Selecting a transformed image set from the multi-domain unbalanced blood cell image set, the transformed image set includes a preset number of transformed images, and the domain label of each transformed image in the transformed image set is different;

[0019] Selecting a transformed image from the transformed image set as the image to be normalized;

[0020] Normalizing the density map corresponding to the blood cell image and the density map corresponding to the image to be normalized to obtain a normalized map;

[0021] Calculating a domain transformation sample according to the normalized image, the illumination light intensity and the staining matrix corresponding to the image to be normalized;

[0022] A plurality of the domain transformation samples are combined to obtain the domain transformation sample set.

[0023] In some embodiments, performing local alignment processing and calculating local alignment loss according to the blood cell image and the domain transformation sample set corresponding to the blood cell image includes:

[0024] Extracting features from the blood cell image to obtain a blood cell feature map;

[0025] Extracting features from each domain transformation sample in the domain transformation sample set to obtain a corresponding domain transformation feature map;

[0026] Performing feature mapping on the blood cell feature map using a nonlinear multilayer perceptron to obtain a blood cell feature embedding;

[0027] Performing feature mapping on the domain transformation feature map using a nonlinear multilayer perceptron to obtain a domain transformation feature embedding;

[0028] Calculating a pixel-level alignment loss based on the blood cell feature embedding and the domain transformed feature embedding;

[0029] The local alignment loss is calculated according to the number of image samples, the spatial position index of the feature map and the pixel-level alignment loss.

[0030] In some embodiments, performing domain-invariant contrast learning processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image and calculating the domain-invariant supervised contrast loss includes:

[0031] Acquire a blood cell feature projection of the blood cell image and a domain transformation feature projection corresponding to each domain transformation sample in the domain transformation sample set;

[0032] Performing L2 normalization on the blood cell feature projection;

[0033] Performing L2 normalization on the domain transformed feature projection;

[0034] Calculating an average feature based on the blood cell feature projection after L2 normalization and the domain transformation feature projection after L2 normalization;

[0035] Calculate positive samples based on class labels;

[0036] The domain-invariant supervised contrast loss is calculated based on the temperature coefficient, the average feature and the positive sample.

[0037] In some embodiments, the multi-stage training process is performed on the preset deep learning model according to the local alignment loss and the domain-invariant supervised contrast loss to obtain the blood cell classification model, including:

[0038] Calculating the representation learning phase loss according to the loss ratio hyperparameter, the local alignment loss, and the domain-invariant supervised contrast loss;

[0039] Performing a first-stage training on the preset deep learning model using the representation learning stage loss;

[0040] Calculate the classification training phase loss based on the number of categories, hot-encoded labels, and model prediction probability distribution;

[0041] The preset deep learning model after the first stage training is trained in the second stage using the classification training stage loss to obtain the blood cell classification model.

[0042] In some embodiments, the method further comprises:

[0043] Conducting a performance evaluation experiment on the blood cell classification model;

[0044] The performance evaluation experiment of the blood cell classification model comprises the following steps:

[0045] Set the number of clusters, loss scale hyperparameters, learning rate, and number of image samples;

[0046] Performing model training on a preset test model according to the number of iterations, the number of clusters, the loss ratio hyperparameter, the learning rate, and the number of image samples to obtain a target test model;

[0047] The target test model is used to classify the test data set to obtain performance evaluation test results.

[0048] In some embodiments, the calculating pixel-level alignment loss according to the blood cell feature embedding and the domain transformation feature embedding comprises:

[0049] According to the blood cell feature embedding and the domain transformation feature embedding, the pixel level alignment loss is calculated by a pixel level alignment loss calculation formula, and the pixel level alignment loss calculation formula is:

[0050]

[0051] In the formula, is the pixel-level alignment loss, E i,p For the blood cell feature embedding, is the domain transformation feature embedding, k is the number of clusters, ‖·‖ 2 is the L2 norm.

[0052] On the other hand, an embodiment of the present invention provides a multi-domain unbalanced blood cell classification device, comprising:

[0053] The first module is used to obtain the image of cells to be classified;

[0054] The second module is used to input the image of cells to be classified into the blood cell classification model to obtain the blood cell classification result;

[0055] Wherein, the blood cell classification model is obtained by the following steps:

[0056] Acquire a multi-domain unbalanced blood cell image set, wherein the multi-domain unbalanced blood cell image set includes a plurality of blood cell images;

[0057] Performing dye enhancement processing on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each of the blood cell images, wherein the domain transformation sample set includes a plurality of domain transformation samples;

[0058] performing local alignment processing according to the blood cell image and a domain transformation sample set corresponding to the blood cell image, and calculating a local alignment loss;

[0059] performing domain-invariant contrast learning processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image, and calculating the domain-invariant supervised contrast loss;

[0060] According to the local alignment loss and the domain-invariant supervised contrast loss, a preset deep learning model is subjected to multi-stage training processing to obtain the blood cell classification model.

[0061] In another aspect, an embodiment of the present invention provides a computer device, comprising:

[0062] at least one processor;

[0063] at least one memory for storing at least one program;

[0064] When the at least one program is executed by the at least one processor, the at least one processor implements the described method.

[0065] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0066] The beneficial effects of the present invention are as follows:

[0067] The embodiment of the present invention first obtains a cell image to be classified, and then inputs the cell image to be classified into a blood cell classification model to obtain a blood cell classification result, thereby realizing blood cell classification and improving classification accuracy. The blood cell classification model is obtained by the following steps: first, a multi-domain unbalanced blood cell image set is obtained, and then a staining enhancement process is performed on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each blood cell image, and then a local alignment process is performed based on the blood cell image and the domain transformation sample set corresponding to the blood cell image, and the local alignment loss is calculated, and a domain-invariant contrast learning process is performed to calculate the domain-invariant supervised contrast loss, and finally, a multi-stage training process is performed on the preset deep learning model based on the local alignment loss and the domain-invariant supervised contrast loss to obtain a blood cell classification model, thereby improving the model accuracy.

[0068] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0070] Figure 1 This is a flow chart of a multi-domain unbalanced blood cell classification method according to an embodiment of the present invention;

[0071] Figure 2 This is a flow chart of obtaining a blood cell classification model according to an embodiment of the present invention;

[0072] Figure 3 A schematic diagram of a dye enhancement process according to an embodiment of the present invention;

[0073] Figure 4 A schematic diagram of a local alignment process according to an embodiment of the present invention;

[0074] Figure 5 A schematic diagram of a domain-invariant contrastive learning processing process according to an embodiment of the present invention;

[0075] Figure 6 This is a diagram of a blood cell classification network model architecture according to an embodiment of the present invention;

[0076] Figure 7 A schematic diagram of average results of comparative experiments of different methods on a public data set according to an embodiment of the present invention;

[0077] Figure 8 A schematic diagram of average results of comparative experiments of different methods on a non-public data set according to an embodiment of the present invention;

[0078] Fig. 9 A schematic diagram of a visual comparison result predicted by a blood cell classification model according to an embodiment of the present invention;

[0079] Fig.10 This is a schematic diagram of the structure of a multi-domain unbalanced blood cell classification device according to an embodiment of the present invention;

[0080] Fig.11 The figure is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0082] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0083] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0084] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0085] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0086] Blood cells: Also known as "blood corpuscles", are cells that exist in the blood and can flow throughout the body with the blood. For mammals, blood cells mainly include the following three types: Red blood cells: The main function is to transport oxygen. White blood cells: They mainly play an immune role. When pathogens invade the human body, white blood cells can pass through the capillary walls, concentrate at the site of pathogen invasion, surround the pathogens and then engulf them. Platelets: They play an important role in the hemostasis process. Blood cells account for about 45% of the blood volume, including red blood cells, white blood cells and platelets. Under normal physiological conditions, blood cells and platelets have certain morphological structures and relatively stable numbers.

[0087] In the related art, blood cell classification, as a universal cell analysis method, can include leukemia, anemia, infection, autoimmune disorders, and other blood-related disease cells. Therefore, accurate and timely classification of blood cells is crucial in clinical practice and plays a key role in timely detection of blood cells. In clinical practice, blood cells are traditionally identified by manual inspection using a microscope by skilled experts, which is a repetitive, labor-intensive and time-consuming process. In recent years, deep learning models that solve real-world classification tasks in various domains have made significant progress. Some researchers have tried to apply deep learning models to blood cell classification and have shown promising results. However, blood cell images collected from various laboratories and hospitals often exhibit significant domain shifts and data imbalances. Domain shifts may lead to decreased generalization performance of the model and poor performance on unseen datasets, while data imbalance may cause the model to perform poorly on minority classes and have low classification accuracy.

[0088] In view of this, this embodiment performs unbalanced multi-domain blood cell classification through staining-aware domain alignment. Specifically, a staining-based enhancement method is first introduced to generate domain-transformed samples containing domain-specific information of the source domain while retaining domain-invariant features. The original blood cell image and the generated domain-transformed samples will enter the same feature extraction network (such as Resnet50). In order to encourage the model to capture domain-invariant features, a local alignment constraint is designed to keep the original blood cell image and the domain-transformed samples consistent at the feature map level. In addition, this embodiment proposes a domain-invariant supervised contrastive learning strategy, which can reduce the influence of domain-specific features and promote discriminative feature learning by taking the average features of the original blood cell image and the domain-transformed samples as anchors. Finally, the training process is decoupled into a representation learning phase and a classification training phase, which alleviates the problem of data imbalance. Experimental results on four public blood cell datasets and one real dataset show that this embodiment achieves significantly superior state-of-the-art results and improves classification accuracy.

[0089] A multi-domain unbalanced blood cell classification method provided in an embodiment of the present application relates to the field of image recognition technology. A multi-domain unbalanced blood cell classification method provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a multi-domain unbalanced blood cell classification method, etc., but is not limited to the above forms.

[0090] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0091] The following is a detailed explanation of the embodiments of the present application in conjunction with the accompanying drawings:

[0092] Figure 1 is an optional flow chart of a multi-domain unbalanced blood cell classification method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to step S101 to step S102.

[0093] Step S101, obtaining an image of cells to be classified;

[0094] Step S102: input the image of cells to be classified into a blood cell classification model to obtain a blood cell classification result.

[0095] Steps S101 to S102 shown in the embodiment of the present application realize blood cell classification and improve classification accuracy.

[0096] In some embodiments, the image of the cell to be classified may be first obtained, and then the image of the cell to be classified may be input into a blood cell classification model to obtain a blood cell classification result, so as to achieve blood cell classification. The accuracy of the blood cell classification result mainly depends on the model accuracy of the blood cell classification model.

[0097] Figure 2 It is an optional flowchart of a process of obtaining a blood cell classification model provided in an embodiment of the present application, which may include but is not limited to steps S201 to S205.

[0098] Step S201, obtaining a multi-domain unbalanced blood cell image set, where the multi-domain unbalanced blood cell image set includes a plurality of blood cell images;

[0099] Step S202, performing dye enhancement processing on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each blood cell image, where the domain transformation sample set includes a plurality of domain transformation samples;

[0100] Step S203, performing local alignment processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image, and calculating the local alignment loss;

[0101] Step S204, performing domain-invariant contrast learning processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image, and calculating the domain-invariant supervised contrast loss;

[0102] Step S205: According to the local alignment loss and the domain-invariant supervised contrast loss, a multi-stage training process is performed on the preset deep learning model to obtain a blood cell classification model.

[0103] Steps S201 to S205 shown in the embodiment of the present application obtain a blood cell classification model, thereby improving the accuracy of the model.

[0104] In step S201 of some embodiments, a multi-domain unbalanced blood cell image set may be obtained through a cell database. The multi-domain unbalanced blood cell image set may also be obtained in other ways, not limited thereto. The multi-domain unbalanced blood cell image set includes a plurality of blood cell images, and the number of image samples may be set as the number of blood cell images.

[0105] In some embodiments, in step S202, staining enhancement processing is performed on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each blood cell image, which may include but is not limited to the following steps:

[0106] Perform image decomposition on each blood cell image in the multi-domain unbalanced blood cell image set to obtain a corresponding density map and staining matrix;

[0107] According to the number of clusters, multiple staining matrices are clustered using a preset clustering method to obtain several clusters;

[0108] Set domain labels for the blood cell images corresponding to the staining matrix in each cluster;

[0109] Selecting a transformed image set from a multi-domain unbalanced blood cell image set, the transformed image set includes a preset number of transformed images, and the domain label of each transformed image in the transformed image set is different;

[0110] Selecting a transformed image from the transformed image set as the image to be normalized;

[0111] Normalizing the density map corresponding to the blood cell image and the density map corresponding to the image to be normalized to obtain a normalized map;

[0112] Calculating a domain transformation sample according to the normalization map, the illumination light intensity and the staining matrix corresponding to the image to be normalized;

[0113] Multiple domain transformation samples are combined to obtain a domain transformation sample set.

[0114] In some embodiments, the dye enhancement process is as follows Figure 3 As shown in the figure, SNMF stands for sparse non-negative matrix factorization method. The role of the dyeing solution is to color the blood cell image to enhance the visibility of the cell structure. The final color effect depends on the specific type and amount of the dyeing solution absorbed by the blood cells. Each blood cell image in the multi-domain unbalanced blood cell image set can be first decomposed by the Beer-Lambert law to obtain the corresponding density map H and the corresponding staining matrix W, and the sparse non-negative matrix factorization (SNMF) method is used to optimize the density map H and the staining matrix W. Among them, the mathematical expression for image decomposition of the blood cell image is: Where V is the optical density generated by Beer-Lambert transformation, V∈R 3×n , X is the blood cell image in RGB space, X∈R 3×n , n is the number of pixels, X 0 is the illumination intensity (e.g., the illumination intensity for an 8-bit image is 255), W is the coloring matrix, W∈R 3×r , H is the density map, H∈R r×n , r is the normalized color basis. It can be understood that, assuming that there are N samples in the multi-domain unbalanced blood cell image set mini-batch, each sample x i Decomposed into the coloring matrix W i and density map Hi , where i = 1, 2…N. Then, according to the number of clusters, the preset clustering method is used to cluster multiple staining matrices to obtain several clusters. Exemplarily, in order to select samples with significant domain differences, the preset clustering method can be set to the k-means algorithm, the number of clusters is set to k, and the k-means algorithm is used to cluster the staining matrices corresponding to each blood cell image in the multi-domain unbalanced blood cell image set to obtain k clusters. Then, a domain label is set for the blood cell image corresponding to the staining matrix in each cluster. Exemplarily, each cluster generated by k-means has a unique domain label, which is a sample x in each cluster. i Assign the domain label d corresponding to the cluster i ∈{1,2,…,k}.

[0115] Then, a transformation image set is selected from the multi-domain unbalanced blood cell image set, wherein the transformation image set includes a preset number of transformation images, and the domain labels of each transformation image in the transformation image set are different. Exemplarily, when the number of clusters is set to k, excluding the current cluster, a blood cell image can be randomly selected from each of the other clusters with different domain labels (a total of k-1 clusters) as a transformation image to obtain k-1 transformation images. Each selected transformation image has a corresponding density map and staining matrix. Since there may be scale differences between density maps, additional normalization processing is required. A transformation image can be selected from the transformation image set as the image to be normalized, and the density map H corresponding to the blood cell image can be normalized. i The density map H corresponding to the image to be normalized t Normalization is performed to obtain a normalized graph, wherein the calculation formula of the normalized graph is: GY = Norm (H i ,H t ), where GY is the normalized graph, Norm is the normalization process, and H i is the density map corresponding to the blood cell image, H t is the density map corresponding to the image to be normalized. Then, according to the normalized map, the illumination light intensity and the staining matrix corresponding to the image to be normalized, the domain transformation sample is calculated, where the calculation formula of the domain transformation sample is: In the formula, is the domain transformation sample, X 0 is the illumination light intensity, W t is the staining matrix corresponding to the image to be normalized. Finally, multiple domain transformation samples are combined to obtain a domain transformation sample set. It can be understood that for each blood cell image x i , k-1 staining matrices W from different domain labels are selected i , using k-1 different W i and H iTo synthesize k-1 domain transformation samples Specifically, for x i There are k-1 clusters with different domain labels, and a staining matrix W can be randomly selected from each cluster. t , to the blood cell image x i The density map G i Perform a re-staining operation. For each blood cell image x in the multi-domain unbalanced blood cell image set mini-batch i , both generate corresponding domain transformation samples Each blood cell image x i Will generate k-1 domain transformation samples A total of N(k-1) domain transformed samples are generated.

[0116] In some embodiments, in step S203, local alignment processing is performed according to the blood cell image and the domain transformation sample set corresponding to the blood cell image to calculate the local alignment loss, which may include but is not limited to the following steps:

[0117] Extract features from the blood cell image to obtain a blood cell feature map;

[0118] Perform feature extraction on each domain transformation sample in the domain transformation sample set to obtain a corresponding domain transformation feature map;

[0119] The blood cell feature map is feature mapped using a nonlinear multilayer perceptron to obtain blood cell feature embedding;

[0120] The domain transformation feature map is feature mapped using a nonlinear multilayer perceptron to obtain a domain transformation feature embedding;

[0121] Calculate pixel-level alignment loss based on blood cell feature embedding and domain transformed feature embedding;

[0122] The local alignment loss is calculated based on the number of image samples, the spatial position index of the feature map, and the pixel-level alignment loss.

[0123] In some embodiments, the local alignment process is as follows Figure 4 As shown, the domain transformation samples generated by dye enhancement processing The structure of the original blood cell image x iThe structure of is consistent with that of . In order to utilize this feature, a local alignment constraint can be used in combination with a projection head Projector to minimize the difference between the original blood cell image and the domain transformed sample at the feature map level. The blood cell image can be firstly feature extracted to obtain a blood cell feature map, and feature extraction can be performed on each domain transformed sample in the domain transformed sample set to obtain the corresponding domain transformed feature map. For example, the blood cell feature map F can be extracted from the final layer (before the global average pooling layer) of the backbone network (ResNet50) i,p Domain transformation feature map in, D b is the number of output channels of the backbone network, and p is the spatial position index of the feature map. In order to prevent the performance degradation due to the collapse of the representation, a nonlinear multilayer perceptron can be used to represent the blood cell feature map F. i,p Perform feature mapping to obtain blood cell feature embedding E i,p , and use nonlinear multilayer perceptron to transform the feature map Perform feature mapping to obtain domain transformation feature embedding Then, according to the blood cell feature embedding and domain transformation feature embedding, the pixel-level alignment loss is calculated by the pixel-level alignment loss calculation formula. The pixel-level alignment loss calculation formula is: In the formula, is the pixel-level alignment loss, E i,p Embedding for blood cell features, is the domain transformation feature embedding, k is the number of clusters, ‖·‖ 2 is the L2 norm. Finally, the local alignment loss is calculated based on the number of image samples, the spatial position index of the feature map, and the pixel-level alignment loss. The calculation formula of the local alignment loss is: In the formula, is the local alignment loss, N is the number of image samples (i.e., the number of blood cell images in the multi-domain unbalanced blood cell image set), and p is the spatial position index of the feature map.

[0124] In some embodiments, in step S204, domain-invariant contrastive learning processing is performed according to the blood cell image and the domain transformation sample set corresponding to the blood cell image to calculate the domain-invariant supervised contrast loss, which may include but is not limited to the following steps:

[0125] Acquire a blood cell feature projection of the blood cell image and a domain transformation feature projection corresponding to each domain transformation sample in the domain transformation sample set;

[0126] Perform L2 normalization on the blood cell feature projection;

[0127] Perform L2 normalization on the domain transformed feature projection;

[0128] Calculate the average feature based on the L2-normalized blood cell feature projection and the L2-normalized domain transformation feature projection;

[0129] Calculate positive samples based on class labels;

[0130] Based on the temperature coefficient, average features, and positive samples, the domain-invariant supervised contrastive loss is calculated.

[0131] In some embodiments, the domain-invariant contrastive learning process is as follows: Figure 5 As shown in the figure, in the presence of domain shift, the selected anchor points may contain domain-specific features, which may affect the performance of contrastive learning to a certain extent. The blood cell feature projection of the blood cell image and the domain transformation feature projection corresponding to each domain transformation sample in the domain transformation sample set can be obtained first. For example, in each small batch, after the backbone network (ResNet50) f b (·) and projection head (Projector)f p (·) After that, the original blood cell image x i and domain transformation samples Their feature projections are obtained respectively. Then the blood cell feature projection is L2 normalized, and the domain transformation feature projection is L2 normalized to use the dot product as the distance metric. Then, the average feature a is calculated based on the L2 normalized blood cell feature projection and the L2 normalized domain transformation feature projection. i , and use the average feature as the anchor point to obtain an anchor point that is less affected by domain-specific features. It can be understood that the anchor point can be obtained by calculating the average of the feature projection between the original blood cell image and the domain-transformed sample. Then, according to the class label, the positive sample is calculated, where the calculation formula of the positive sample is: In the formula, is a positive sample, y i is x i The class label, z j is the blood cell feature projection, is the domain transformation feature projection. Finally, the domain invariant supervision contrast loss is calculated based on the temperature coefficient, average feature and positive sample, where the calculation formula of the domain invariant supervision contrast loss is: In the formula, is the domain invariant supervised contrast loss, N is the number of image samples, and a i is the average feature, z k is the domain transformed feature projection of the kth domain transformed sample, τ is the temperature coefficient, and |·| is the number of elements in the set.

[0132] In some embodiments, in step S205, a multi-stage training process is performed on a preset deep learning model according to the local alignment loss and the domain-invariant supervised contrast loss to obtain a blood cell classification model, which may include but is not limited to the following steps:

[0133] Calculate the representation learning phase loss based on the loss ratio hyperparameter, local alignment loss, and domain-invariant supervised contrastive loss;

[0134] Perform the first phase of training on the preset deep learning model using the loss in the representation learning phase;

[0135] Calculate the classification training phase loss based on the number of categories, hot-encoded labels, and model prediction probability distribution;

[0136] The preset deep learning model after the first stage training is trained in the second stage using the classification training stage loss to obtain a blood cell classification model.

[0137] In some embodiments, in a multi-stage training process, the training process is divided into two stages: a representation learning stage and a classification training stage. This two-stage training method enables the model to more intensively learn features related to the minority class, thereby improving the performance of the minority class. The representation learning stage loss can be first calculated based on the loss ratio hyperparameter, the local alignment loss, and the domain-invariant supervised contrast loss, where the calculation formula for the representation learning stage loss is: In the formula, To represent the loss in the learning phase, is the domain-invariant supervised contrast loss, β is the loss ratio hyperparameter, which is used to control the ratio of the two loss terms. is the local alignment loss. Then, the representation learning phase loss is used to train the preset deep learning model in the first phase, that is, the training in the representation learning phase. Then, the classification training phase loss is calculated based on the number of categories, hot encoding labels, and model prediction probability distribution. The calculation formula for the classification training phase loss is: In the formula, is the loss in the classification training phase, is the cross entropy loss function, C is the number of categories, y i is the hot encoding label, The model predicts the probability distribution. Finally, the classification training phase loss is used to perform the second phase training on the preset deep learning model after the first phase training, that is, the training in the classification training phase, to obtain the blood cell classification model.

[0138] In some embodiments, the blood cell classification network model architecture diagram is as follows Figure 6As shown in the figure, the original blood cell image can be dye-enhanced to obtain multiple domain transformation samples, which are then simultaneously input into the ResNet50 backbone network and weight-shared. The features output by the network are then locally aligned. After global average pooling, feature flattening, and projection head, domain-invariant contrastive learning is combined to perform multi-stage training to obtain a blood cell classification model. In the figure, SBA (Stain-baesdaugmentation) represents dye enhancement, LA (Local alignment) represents local alignment, GAP (Global average pooling) represents global average pooling, and DISC (Domain-invariant supervised contrastive learning) represents domain-invariant contrastive learning.

[0139] In some embodiments, the method further comprises:

[0140] Conduct performance evaluation experiments on blood cell classification models;

[0141] The performance evaluation experiment of the blood cell classification model includes the following steps:

[0142] Set the number of clusters, loss scale hyperparameters, learning rate, and number of image samples;

[0143] According to the number of iterations, the number of clusters, the loss ratio hyperparameter, the learning rate, and the number of image samples, the preset test model is trained to obtain the target test model;

[0144] The target test model is used to classify the test data set and obtain the performance evaluation experimental results.

[0145] In some embodiments, a performance evaluation experiment can be performed on the blood cell classification model. This embodiment has conducted a large number of experiments on four unbalanced blood cell image datasets, namely Acevedo-20, LDWBC, Raabin-WBC, and Zheng-18, and also used a private real dataset for testing. The number of clusters, loss ratio hyperparameter, learning rate, and number of image samples can be set first, and then the preset test model can be trained according to the number of iterations, the number of clusters, the loss ratio hyperparameter, the learning rate, and the number of image samples to obtain the target test model. Finally, the target test model is used to classify the test dataset to obtain the performance evaluation experimental results. Exemplarily, it can be performed on a server equipped with two NVIDIA 4090 graphics cards and an Intel i9-10900X processor. The number of clusters k and the loss ratio hyperparameter β are set to 3 and 0.1 respectively, the learning rate is set to 0.00005, and the size of the mini-batch of each data set is set to 32. In the representation learning stage, the model training iterated 4000 steps, and in the classification training stage, the model also iterated 4000 steps. In order to improve model training and evaluation, this embodiment performs a five-category classification task including neutrophils, eosinophils, basophils, monocytes and lymphocytes. Due to the imbalance of the data set, the F1-macro score is used as an evaluation indicator to reflect the classification performance of the model when processing unbalanced data. In addition, F1-micro is also reported, which provides a comprehensive indicator through the overall statistics of all categories. The method of this embodiment (hereinafter referred to as SADA) is compared with several state-of-the-art methods, namely ERM, CORAL, Mixup, SelfReg and BoDA. To ensure fair comparison, all algorithms use the same backbone network (i.e., ResNet50 pre-trained on ImageNet).

[0146] For all datasets, a leave-one-out strategy is used for evaluation, where one dataset is used as the target domain and the remaining datasets are used as the source domain. The average results of three experiments of different comparison methods and the method of this embodiment are shown in Figure 7As shown. It can be observed that the method of this embodiment achieves an average F1-micro score of 84.4% and an F1-macro score of 68.6%, exceeding the second highest performing method by 4.4% and 3.9%, respectively. For each dataset, the method of this embodiment almost achieved the best results, except for the Raabin-WBC dataset, where the method of this embodiment performed comparable to the Mixup method with an F1-micro score of 87.4% and an F1-macro score of 61.9%. On the contrary, the method of this embodiment significantly outperforms the second highest performing method on the Acevedo-20 dataset, with F1-micro and F1-macro scores of 79.9% and 81.6%, respectively, exceeding the second highest performing method by 9.2% and 8.0%. After training on four public datasets, the method of this embodiment was evaluated on a private real dataset. As shown Figure 8 As shown, the method of this embodiment outperforms other comparison methods with an F1-micro score of 85.4% and an F1-macro score of 85.1%, exceeding the second highest performing method by 2.1% and 1.8% respectively. The SADA method is specifically designed for the classification of unbalanced blood cell datasets, surpassing other methods and performing well. In order to evaluate the effectiveness of SADA, ablation experiments were performed on four public datasets. The method of this embodiment is compared with the following two variants: (1) SADAw / o la represents SADA without local alignment constraints; (2) SADAw / sc represents SADA with the domain-invariant supervised contrast loss replaced by the supervised contrast loss. The other configurations of these two variants remain the same as the original design to ensure fair comparison. The evaluation results are listed in Figure 7 It can be observed that the lack of local alignment constraints will lead to a certain degree of degradation in the overall performance of the model. At the same time, the domain-invariant supervised contrastive learning loss proposed in this embodiment is significantly better than the supervised contrastive learning loss. In order to further explore the classification performance of the model, the prediction results of the blood cell classification model are visualized by GradCam. The visualization comparison results of the blood cell classification model prediction are shown in Figure 2. Fig. 9 As shown, it can be seen that compared with other methods, the method of this embodiment can focus on the blood cells in the image more comprehensively and accurately.

[0147] In some embodiments, the method of this embodiment includes dye enhancement, local alignment, and domain-invariant contrastive learning. Dye enhancement effectively enriches the diversity of the sample domain and further increases the number of samples. Through local alignment constraints, the model can capture domain-invariant features at the feature map level. Domain-invariant contrastive learning uses the original blood cell image and the centroid of the domain-converted sample as anchors to enhance the model's ability to learn discriminative features. Extensive experiments on four public blood cell datasets and one private real dataset show that the method of this embodiment effectively improves classification accuracy.

[0148] The beneficial effects of implementing the embodiments of the present invention include: the embodiments of the present invention first obtain the cell image to be classified, and then input the cell image to be classified into the blood cell classification model to obtain the blood cell classification result, thereby realizing the blood cell classification and improving the classification accuracy. Among them, the blood cell classification model is obtained by the following steps: first, a multi-domain unbalanced blood cell image set is obtained, and then the multi-domain unbalanced blood cell image set is subjected to dye enhancement processing to obtain a domain transformation sample set corresponding to each blood cell image, and then local alignment processing is performed according to the blood cell image and the domain transformation sample set corresponding to the blood cell image, and the local alignment loss is calculated, and domain-invariant contrast learning processing is performed to calculate the domain-invariant supervised contrast loss, and finally, according to the local alignment loss and the domain-invariant supervised contrast loss, a preset deep learning model is subjected to multi-stage training processing to obtain a blood cell classification model, thereby improving the model accuracy.

[0149] like Fig.10 As shown, an embodiment of the present invention further provides a multi-domain unbalanced blood cell classification device, comprising:

[0150] The first module 801 is used to obtain the image of cells to be classified;

[0151] The second module 802 is used to input the image of cells to be classified into the blood cell classification model to obtain the blood cell classification result;

[0152] Among them, the blood cell classification model is obtained through the following steps:

[0153] Acquire a multi-domain unbalanced blood cell image set, the multi-domain unbalanced blood cell image set including a plurality of blood cell images;

[0154] Performing dye enhancement processing on a multi-domain unbalanced blood cell image set, obtaining a domain transformation sample set corresponding to each blood cell image, wherein the domain transformation sample set includes a plurality of domain transformation samples;

[0155] Performing local alignment processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image, and calculating the local alignment loss;

[0156] According to the blood cell image and the domain transformation sample set corresponding to the blood cell image, domain invariant contrastive learning processing is performed to calculate the domain invariant supervised contrastive loss;

[0157] According to the local alignment loss and domain-invariant supervised contrast loss, the preset deep learning model is trained in multiple stages to obtain a blood cell classification model.

[0158] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] like Fig.11 As shown, an embodiment of the present invention further provides a computer device, including:

[0160] at least one processor 901;

[0161] At least one memory 902, used to store at least one program;

[0162] When at least one program is executed by at least one processor, the at least one processor implements Figure 1 The method shown.

[0163] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0164] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, Figure 1 The method shown.

[0165] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0166] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multi-domain unbalanced blood cell classification method, characterized in that: The following steps are involved: Acquire the cell image to be classified; Inputting the image of cells to be classified into the blood cell classification model to obtain the blood cell classification result; Wherein, the blood cell classification model is obtained by the following steps: Acquire a multi-domain unbalanced blood cell image set, wherein the multi-domain unbalanced blood cell image set includes a plurality of blood cell images; Performing dye enhancement processing on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each of the blood cell images, wherein the domain transformation sample set includes a plurality of domain transformation samples; performing local alignment processing according to the blood cell image and a domain transformation sample set corresponding to the blood cell image, and calculating a local alignment loss; performing domain-invariant contrast learning processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image, and calculating the domain-invariant supervised contrast loss; According to the local alignment loss and the domain-invariant supervised contrast loss, a preset deep learning model is subjected to multi-stage training processing to obtain the blood cell classification model.

2. The method according to claim 1, characterized in that The step of performing dye enhancement processing on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each of the blood cell images includes: Performing image decomposition on each of the blood cell images in the multi-domain unbalanced blood cell image set to obtain a corresponding density map and a staining matrix; According to the number of clusters, clustering the plurality of staining matrices using a preset clustering method to obtain a plurality of clusters; Setting a domain label for the blood cell image corresponding to the staining matrix in each of the clusters; Selecting a transformed image set from the multi-domain unbalanced blood cell image set, the transformed image set includes a preset number of transformed images, and the domain label of each transformed image in the transformed image set is different; Selecting a transformed image from the transformed image set as the image to be normalized; Normalizing the density map corresponding to the blood cell image and the density map corresponding to the image to be normalized to obtain a normalized map; Calculating a domain transformation sample according to the normalization map, the illumination light intensity and the staining matrix corresponding to the image to be normalized; A plurality of the domain transformation samples are combined to obtain the domain transformation sample set.

3. The method according to claim 1, characterized in that The performing local alignment processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image and calculating the local alignment loss includes: Extracting features from the blood cell image to obtain a blood cell feature map; Extracting features from each domain transformation sample in the domain transformation sample set to obtain a corresponding domain transformation feature map; Performing feature mapping on the blood cell feature map using a nonlinear multilayer perceptron to obtain a blood cell feature embedding; Performing feature mapping on the domain transformation feature map using a nonlinear multilayer perceptron to obtain a domain transformation feature embedding; Calculating a pixel-level alignment loss based on the blood cell feature embedding and the domain transformed feature embedding; The local alignment loss is calculated according to the number of image samples, the spatial position index of the feature map and the pixel-level alignment loss.

4. The method according to claim 1, characterized in that: The performing domain invariant contrast learning processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image and calculating the domain invariant supervised contrast loss includes: Acquire a blood cell feature projection of the blood cell image and a domain transformation feature projection corresponding to each domain transformation sample in the domain transformation sample set; Performing L2 normalization on the blood cell feature projection; Performing L2 normalization on the domain transformed feature projection; Calculating an average feature based on the blood cell feature projection after L2 normalization and the domain transformation feature projection after L2 normalization; Calculate positive samples based on class labels; The domain-invariant supervised contrast loss is calculated based on the temperature coefficient, the average feature and the positive sample.

5. The method according to claim 1, characterized in that: The method of performing multi-stage training processing on a preset deep learning model according to the local alignment loss and the domain-invariant supervised contrast loss to obtain the blood cell classification model includes: Calculating the representation learning phase loss according to the loss ratio hyperparameter, the local alignment loss, and the domain-invariant supervised contrast loss; Performing a first-stage training on the preset deep learning model using the representation learning stage loss; Calculate the classification training phase loss based on the number of categories, hot-encoded labels, and model prediction probability distribution; The preset deep learning model after the first stage training is trained in the second stage using the classification training stage loss to obtain the blood cell classification model.

6. The method according to claim 1, characterized in that The method further comprises: Conducting a performance evaluation experiment on the blood cell classification model; The performance evaluation experiment of the blood cell classification model comprises the following steps: Set the number of clusters, loss scale hyperparameters, learning rate, and number of image samples; Performing model training on a preset test model according to the number of iterations, the number of clusters, the loss ratio hyperparameter, the learning rate, and the number of image samples to obtain a target test model; The target test model is used to classify the test data set to obtain performance evaluation test results.

7. The method according to claim 3, characterized in that The calculating pixel-level alignment loss according to the blood cell feature embedding and the domain transformation feature embedding comprises: According to the blood cell feature embedding and the domain transformation feature embedding, the pixel level alignment loss is calculated by a pixel level alignment loss calculation formula, and the pixel level alignment loss calculation formula is: In the formula, is the pixel-level alignment loss, E i,p For the blood cell feature embedding, is the domain transformation feature embedding, k is the number of clusters, and ‖·‖2 is the L2 norm.

8. A multi-domain unbalanced blood cell classification device, characterized in that: include: The first module is used to obtain the image of cells to be classified; The second module is used to input the image of cells to be classified into the blood cell classification model to obtain the blood cell classification result; Wherein, the blood cell classification model is obtained by the following steps: Acquire a multi-domain unbalanced blood cell image set, wherein the multi-domain unbalanced blood cell image set includes a plurality of blood cell images; Performing dye enhancement processing on the multi-domain unbalanced blood cell image set to obtain a domain transformation sample set corresponding to each of the blood cell images, wherein the domain transformation sample set includes a plurality of domain transformation samples; performing local alignment processing according to the blood cell image and a domain transformation sample set corresponding to the blood cell image, and calculating a local alignment loss; performing domain-invariant contrast learning processing according to the blood cell image and the domain transformation sample set corresponding to the blood cell image, and calculating the domain-invariant supervised contrast loss; According to the local alignment loss and the domain-invariant supervised contrast loss, a preset deep learning model is subjected to multi-stage training processing to obtain the blood cell classification model.

9. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images