Bone marrow cell classification and identification method based on SCKannform neural network
The bone marrow cell classification method using the SCKansformer neural network solves the complexity problem in bone marrow blood cell image recognition, achieving accurate classification of forty types of bone marrow blood cells and improving the accuracy of disease diagnosis and treatment.
Patent Information
- Application Number
- CN202410826656.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-11-29
AI Technical Summary
Existing technologies face challenges in bone marrow and blood cell image recognition, including issues such as image color differences, low resolution, insufficient contrast, uneven category distribution, and small cell differences. These problems make accurate classification difficult, especially for bone marrow and blood cell classification and recognition in rare disease categories.
A bone marrow cell classification method based on SCKansformer neural network is adopted, which combines SCConv encoder, global-local attention encoder and Kansformer encoder. The method reduces redundant information through separation-reconstruction and segmentation-transformation-fusion strategy, uses multi-head self-attention module to learn long-distance dependencies and extracts local features through deep separable convolution, so as to achieve accurate classification of forty types of bone marrow blood cells.
It enables accurate classification of bone marrow blood cells in microscopic images, improves clinicians' ability to diagnose major hematological diseases, provides important diagnostic and treatment reference standards, and enhances the ability to diagnose diseases such as acute leukemia.
Smart Images

Figure CN119049042A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision deep learning and medical image processing, and in particular to a bone marrow cell classification and recognition method based on an SCKansformer neural network. Background Art
[0002] In recent years, the incidence and mortality of malignant tumors, such as acute leukemia, have shown a clear upward trend worldwide. In the actual diagnosis process, the hospital initially mainly uses cytological examinations based on peripheral blood smears and bone marrow smears to determine the specific type of malignant tumor diseases. For example, based on the results of cytological examinations, doctors determine the specific type of leukemia in patients and provide targeted treatment plans. In cytological examinations, medical personnel mainly focus on the number of various white blood cells in samples from leukemia patients and the difference between the proportion of samples and healthy samples, and rely on counting the proportion of primitive cells to determine the category of the disease and the development of the disease. Therefore, the key to the accuracy of bone marrow or peripheral blood smear examinations lies in whether the different types of blood cells in the sample can be accurately counted, especially the statistical counting results of bone marrow smear cytological examinations. These data are important reference standards for assisting clinicians in diagnosing and treating diseases.
[0003] Clinicians can make a comprehensive diagnosis based on abnormal blood cell morphology, such as observing whether there are primitive immature blood cells, whether there are toxic granules and vacuolar degeneration in neutrophils, whether the red blood cell morphology is normal, and whether there are abnormal morphological changes such as spherical red blood cells, elliptical red blood cells or acanthocytes, so as to better help clinical patients diagnose and monitor diseases. When making accurate diagnosis and treatment, especially for the identification of major dangerous diseases, it is not enough to use only peripheral blood smear examination, and bone marrow smear cell morphology examination must be relied on. This method is mainly suitable for the diagnosis of hematopoietic system diseases, such as leukemia, aplastic anemia, multiple myeloma and other major malignant diseases.
[0004] In addition, compared with image recognition in natural scenes, blood cell classification for major blood diseases such as leukemia auxiliary diagnosis often faces the following challenges due to the need for bone marrow image cytology examination: 1) The bone marrow blood cell images produced by different hospitals, mechanical equipment, and imaging environments have different colors, making them difficult to accurately identify. 2) Due to the low incidence of major dangerous diseases, bone marrow blood cell datasets often have long-tail problems, with very small sample distributions for rare disease categories and particularly large sample distributions for common disease categories. Therefore, the category distribution is extremely uneven. 3) Unlike natural images obtained in sunlight, microscopic images of bone marrow blood samples are obtained through specialized imaging equipment and processing techniques. Therefore, microscopic images of bone marrow blood cells usually exhibit lower resolution and contrast, which may result in fewer features than natural images. Images under the microscope are difficult to be recognized by the model. 4) In bone marrow smear cytology examinations, the differences between different types of bone marrow blood cells are small, such as the differences between early erythroblasts, mid-stage erythroblasts, and late erythroblasts, which are extremely small and difficult to classify. Summary of the invention
[0005] In view of the deficiencies of the prior art, the present invention proposes a bone marrow cell classification and recognition method based on the SCKansformer neural network, which can effectively cope with the complexity of cell classification in microscopic images and achieve accurate classification of forty types of bone marrow blood cells.
[0006] In order to solve the above technical problems, the technical solution of the present invention is:
[0007] A bone marrow cell classification and recognition method based on SCKansformer neural network comprises the following steps:
[0008] Step 1, constructing a SCKansformer neural network model, wherein the SCKansformer neural network model includes an SCConv encoder, a global-local attention encoder and a Kansformer encoder; Step 2: Establish and preprocess a bone marrow blood cell dataset, and divide the dataset into a training set and a test set; Step 3: Use the training set to train the SCKansformer neural network model, and optimize the parameters of the SCKansformer neural network model based on the characteristics of bone marrow blood cell images; Step 4: Use the test set to test the trained SCKansformer neural network model, and then obtain the network model, and finally realize the bone marrow blood cell image classification and recognition function. Preferably, the SCConv encoder comprises a spatial reconstruction unit and a channel reconstruction unit, which respectively reduce spatial and channel redundancy through separation-reconstruction and separation transform fusion strategies. Preferably, the global-local attention encoder includes a multi-head self-attention module and a local module, wherein the multi-head self-attention module learns long-distance dependencies through a self-attention mechanism, and the local module extracts local features between adjacent pixels through deep separable convolution. Preferably, the Kansformer encoder consists of alternating multi-head self-attention layers and KAN networks, with layer normalization applied before each block and residual connections applied after each block. Preferably, in step 2, the blood cells in the images in the data set are labeled with types. Preferably, in step 2, the preprocessing method is: first randomly cropping to a size of 224*224, and performing data enhancement by random horizontal inversion, random rotation and mixup. Preferably, in step 3, the training method of the SCKansformer neural network model is: The training set is used as input, implemented by Python, the IDE is PyCharm, the PyTorch deep learning framework is used, it is carried out on a single GPU, the operating system is Ubuntu 18.04, the training lasts for 100 cycles, the batch size is 128, and it takes 12.6 hours. Preferably, in step 4, the evaluation method of the trained SCKansformer neural network model is: Among them, TP, FP, FN, and TN are true positive, false positive, false negative, and true negative, respectively. TP and TN are samples predicted to be true, and FP and FN are samples predicted to be false. Preferably, the implementation method of the Kansformer encoder is as follows: after the output of the global-local attention encoder integrates the category vector, image block embedding and position encoding into an integrated input embedding vector, the relationship between adjacent pixels is further obtained through the Global-LocalAttention Encoder, and the processed feature vector can be fed into the Kansformer Encoder, through the Kansformer Encoder composed of serially stacked Kansformer Encoder Blocks, and finally the features corresponding to the learnable category embedding vector are extracted for image classification. Preferably, after the output vector of the global-local attention encoder is input into the Kansformer encoder, the output of the SCConv encoder is used as the effective input sequence length of the Kansformer encoder. The Kansformer encoder uses a constant latent vector size D in all its layers, flattens the image block, and uses a trainable linear projection map to D dimensions. The output operation is called Patch Embedding, and its state at the output of the Kansformer encoder is used as an image classification flag. Preferably, the method in which the spatial reconstruction unit performs spatial reconstruction on the feature map is: First, the input feature map is processed by Group Normalization to separate the feature map with rich information from the feature map with less information. The expression is as follows: Among them, the input feature map X∈R N×C×H×W , N is the batch axis, C is the channel axis, H and W are the spatial height and width axes; μ and σ are the mean and standard deviation of X, ε is a small positive constant added for segmentation stability, and γ and β are trainable affine transformations; The amount of information carried by these different feature maps is then evaluated by the scaling factor γ, expressed as follows: The normalized correlation weight Wγ is obtained by taking the scaling factor γ as the metric The obtained relevant weight Wγ is mapped to the range of 0 to 1 through the sigmoid function, and gated by the threshold. The weight exceeding the threshold is assigned to 1 to generate the information weight W1, and the weight below the threshold is assigned to 0 to obtain the non-information weight W2; Then, the input feature X is multiplied by the two weighted features to obtain the key features with large information content. and redundant features with less information Finally, a cross reconstruction operation is performed to fully fuse the two weighted information features, enhance the information flow between the two, and finally obtain the spatially refined feature map Xw, as shown in the following formula: X ω = Gate(Sigmoid(W γ (GN(X)))). Preferably, the channel reconstruction unit adopts a separation-transformation-fusion strategy to reduce feature redundancy, and replaces the standard convolution with three operators: Split, Transformi and Fus. Preferably, the Split operation refines the input spatial feature X ωThe number of channels in one part is αC, and the number of channels in the other part is (1-α)C, where α is a hyperparameter used to segment the features. Then, the number of channels of the two sets of features is compressed using a 1*1 convolution kernel, and the results are X up and X low . Preferably, the Transform operation converts the input X up As the input of rich feature extraction, GWC and PWC are used for efficient convolution operations, and then add operations are performed to obtain the output Y1; the input X low As a supplement to the rich feature extraction, a PWC efficient convolution operation is performed, and the result is concat-operated with the original input to obtain the output Y2. Preferably, the Fus operation first uses global average pooling to combine global spatial information and channel statistical information to obtain pooled results S1 and S2, and then performs a Softmax operation on S1 and S2 to obtain a feature weight vector β 1 and β 1 Finally, the feature weight vector is used to obtain the output Y = β 1 Y 1 +β 2 Y 2 , where Y is the feature after channel extraction. Preferably, the method for the multi-head self-attention module to learn long-distance dependencies is: MHA(Q,K,V)=Concat(Attention(Q 1 , K 1 , V 1 ), ..., Attention(Q i , K i V i )) Attention(Q,K,V)=Cos(Q,K)×V Among them, Q represents the query information, K represents the vector used to match Q, also known as the index, and V represents the content vector. Q, K, V are composed of the input vector sequence X∈R N×d The mapping formula obtained by linear mapping is as follows, where W Q , W K , W V ∈R d×d : Q=XW Q K=XW K V=XWV . Preferably, the method for extracting local features by the local module is: First, the image is treated as a serialized input, the image is divided into a series of tiles, and each tile is converted into a vector representation as a serialized vector The image serialization vector is reconstructed into a two-dimensional feature map through the Sequence to Image (S2I) operation, the purpose of which is to restore the physical position and adjacency between pixels; A 1*1 convolution kernel with a step size of 1 is used to expand the dimension, and batch normalization and h-swish activation function are used to solve the gradient disappearance problem and add nonlinearity to the model; After the dimension is expanded, the feature map z is obtained by a depth-separable convolution with a step size of 1 and 3*3 out ; Use a 1*1 convolution with a stride of 1 to restore the dimension; After local feature extraction, the feature map is re-expanded into a one-dimensional serialized vector Conversion of one-dimensional serialized vector to trainable serialized vector concatenated as the input vector for the next layer. The present invention has the following characteristics and beneficial effects:
[0009] By adopting the above technical scheme, the present invention aims at automatic classification and recognition of blood cells in microscopic images. By integrating Kansformer Encoder (KAN), SCConv encoder, and global-local attention encoder, it can effectively cope with the complexity of cell classification in microscopic images and achieve accurate classification of forty types of bone marrow blood cells. This method can not only provide an important reference standard for clinicians to diagnose and treat major dangerous diseases, but also further enhance the diagnosis ability of clinicians for acute leukemia and other major blood diseases, which is of great significance in clinical auxiliary diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0011] Figure 1 The overall structure diagram of the SCKansformer neural network model of the present invention;
[0012] Figure 2Schematic diagram of the SCConv module of the present invention;
[0013] Figure 3 This is a schematic diagram of the SRU module of the present invention;
[0014] Figure 4 This is a schematic diagram of a CRU module of the present invention;
[0015] Figure 5 Schematic diagram of the global-local attention encoder module of the present invention. DETAILED DESCRIPTION
[0016] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0017] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.
[0018] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.
[0019] The present invention provides a bone marrow cell classification and recognition method based on an SCKansformer neural network. The SCKansformer neural network model consists of an SCConv encoder (SCConv Encoder), a global-local attention encoder (Global-Local Attention Encoder) and a Kansformer encoder (Kansformer Encoder).
[0020] Specifically, Figure 1 As shown, the method comprises the following steps:
[0021] Step 1: Construct an SCKansformer neural network model for bone marrow cell classification and recognition, wherein the SCKansformer neural network model consists of an SCConv encoder, a global-local attention encoder, and a Kansformer encoder.
[0022] Specifically, the SCKansformer neural network model reshapes the microscopic image X∈R^(H×W×C) into a series of two-dimensional image patch sequences X_p∈R^(N×(p^2·C)) after the PatchEmbedding operation, where (H,W) is the resolution of the original image, C is the number of channels, (P,P) is the resolution of each image patch, and N=HW / P^2 is the number of patches obtained. The sequence is then encoded by the SCConv Encoder to process the spatial and channel redundant features, and the processed result is used as the effective input sequence length of Kansformer. Kansformer uses a constant latent vector size D in all its layers, so we flatten the image patches and use a trainable linear projection to map them to D dimensions. The output operation is called PatchEmbedding. Similar to BERT's [class] token, an embedding is added before the Embedded Patch sequence, and its state at the output of the Kansformer encoder is used as an image classification flag. After the input embedding vector that combines the category vector, image block embedding and position encoding is integrated, it is necessary to first pass the Global-Local AttentionEncoder to further obtain the relationship between adjacent pixels. The processed feature vector can be fed into the KansformerEncoder. The Kansformer Encoder, which is composed of a series of Kansformer Encoder Blocks stacked together, finally extracts the features corresponding to the learnable category embedding vector (class token) for image classification.
[0023] The SCConv encoder module of the SCKansformer neural network model mainly consists of two units, namely the Spatial Reconstruction Unit (SRU) and the Channel Reconstruction Unit (CRU). The SRU uses a separation-reconstruction method to reduce spatial redundancy, while the CRU uses a segmentation-conversion-fusion method to reduce channel redundancy. These two units work together to reduce the redundant information of features in CNN. The patch expansion layer uses a linear layer to increase the feature dimension to twice the original. Then, the resolution of the input feature is expanded to twice the resolution of the input feature through the rearrange operation, and the feature dimension is reduced to one-fourth of the input feature dimension.
[0024] The global-local attention encoder of the SCKansformer neural network model is mainly composed of a multi-head self-attention module and a local module. The encoder can not only pay attention to the long-distance dependencies of microscopic cell images, but also effectively extract local features through deep separable convolution, and further obtain the multivariate relationship between adjacent pixels. The multi-head self-attention module calculates different weights of different objects through mutual learning between input vectors (self-attention mechanism), so as to find more concerned regional information; the local module first divides the serialized vector into trainable serialized vectors and the image serialization vector The image sequence vector is reconstructed into a two-dimensional feature map through the Sequence to Image (S2I) operation, the purpose of which is to restore the physical position and adjacency between pixels. A 1*1 convolution kernel with a step size of 1 is used to expand the dimension, and batch normalization and h-swish activation function are used to solve the gradient disappearance problem and add nonlinearity to the model. After expanding the dimension, a depth-wise convolution with a step size of 1 and a 3*3 depth-wise convolution is used to obtain the feature map z out . Use a 1*1 convolution with a step size of 1 to restore the dimension. After completing the local feature extraction, the feature map is re-expanded into a one-dimensional serialized vector Concatenate the one-dimensional serialized vector with the trainable serialized vector as the input vector of the next layer.
[0025] The Kansformer encoder of the SCKansformer neural network model consists of alternating multi-head self-attention layers (MSA) and KAN networks. Layer normalization (Layer Norm) is applied before each block, and residual connection (Residual Connection) is applied after each block. After the microscopic image X∈R^(H×W×C) is subjected to the Patch Embedding operation, it is reshaped into a series of two-dimensional image block sequences X_p∈R^(N×(p^2·C)), where (H,W) is the resolution of the original image, C is the number of channels, (P,P) is the resolution of each image block, and N=HW / P^2 is the number of blocks obtained. The sequence is then encoded by SCConvEncoder to process the spatial and channel redundant features, and the processed result is used as the effective input sequence length of Kansformer. Kansformer uses a constant latent vector size D in all its layers, so we flatten the image blocks and map them to D dimensions using a trainable linear projection. The output operation is called Patch Embedding. Similar to BERT's [class] token, an embedding is added before the Embedded Patch sequence, and its state at the output of the Kansformer encoder is used as the image classification flag. After the input embedding vector that combines the category vector, image patch embedding, and position encoding into one, it is necessary to first pass the Global-Local Attention Encoder to further obtain the relationship between adjacent pixels. The processed feature vector can be fed into the Kansformer Encoder. Through the Kansformer Encoder composed of serially stacked Kansformer Encoder Blocks, the features corresponding to the learnable category embedding vector (class token) are finally extracted for image classification.
[0026] Step 2, establish a bone marrow blood cell data set. Due to the difficulty of collecting and annotating bone marrow blood sample images and the high fine-grained classification of bone marrow blood cells themselves, the academic community lacks a large-scale bone marrow blood cell data set with accurate annotation and high fine-grainedness, which provides a data basis for related development. For this reason, in the present embodiment, a bone marrow blood cell data set (Bone Marrow Blood Cell Fine-Grained Classification Dataset, BMCD-FGCD) with 10k+ data quantity, 40 classified fine-grained and accurately annotated by senior professional doctors is established. BMCD-FGCD contains 92335 bone marrow blood cell images in total, a total of 40 blood cell categories. This paper divides the data set according to categories, and the training set contains 73877 samples, and the test set contains 18458 samples. At the same time, in order to facilitate the development of the field and make positive contributions, it is decided to make the BMCD-FGCD private data set public and provide it to other researchers for use.
[0027] Specifically, in this embodiment, the original images of the BMCD-FGCD dataset are derived from the bone marrow blood sample images in the hospital data system. These images are collected from patients who visited the Hematology Department of Zhejiang Hospital and were diagnosed by professional doctors as needing to undergo puncture to collect bone marrow samples for auxiliary diagnosis. The establishment of the BMCD-FGCD dataset includes the following three steps:
[0028] (1) Collection phase:
[0029] Under the guidance of a professional doctor, the patient lies in a prone or lateral position. The doctor disinfects the skin of the patient to be punctured and injects local anesthetics to locally infiltrate and anesthetize the area. The clinician uses a disposable puncture device, a bone marrow puncture needle, to pass the needle through the bone cortex and into the bone marrow cavity. After the needle enters the bone marrow cavity, the doctor pulls out the needle core, connects an empty 10ml syringe to remove the bone marrow fluid, and makes a bone marrow smear sample. After the smear sample is dried and stained, a professional physician performs a cytological examination under a microscope, and the cell image and content are entered into the electronic medical record system of Zhejiang Hospital.
[0030] Under the supervision and guidance of professional doctors, the team selected high-quality bone marrow blood images of patients of different ages, genders, and diseases from the electronic medical record system (image archiving system) of Zhejiang Hospital, and ensured the privacy of the data source in the process.
[0031] (2) Image processing stage:
[0032] The acquired bone marrow blood images will be imported into professional image processing software or systems to ensure that they meet the requirements for further analysis and annotation. Images that do not meet the standards will be pre-processed, such as image clarity enhancement or standardization, to ensure that the generated image quality is suitable for subsequent algorithm processing.
[0033] (3) Marking stage:
[0034] This step involves identifying and labeling the type of each blood cell, such as lymphocytes, neutrophils, monocytes, etc. The labeling process requires medical expertise to ensure the accuracy and quality of the image data set. The team processed the collected data with high granularity and high accuracy. In this step, a total of 40 types of bone marrow blood cells were identified and labeled under the guidance and help of several senior physicians.
[0035] Step 3. Use the training set to train the SCKansformer neural network model. Combined with the characteristics of bone marrow cell images, this method has been implemented in Python. The IDE is PyCharm, and the PyTorch deep learning framework is used. The experiment was performed on a single GPU (NVIDA GeForce RTX 4090Ti). The operating system is Ubuntu 18.04. The training lasted 100 cycles with a batch size of 128 and took 12.6 hours. The values of all hyperparameters in the ablation experiment are shown. The input image is first randomly cropped to a size of 224*224, and the data is enhanced by data augmentation methods such as random horizontal inversion, random rotation, and mixup.
[0036] Specifically, in this embodiment, random horizontal inversion, random rotation and mixup are used for data enhancement.
[0037] Furthermore, in this embodiment, the AdamW optimizer and the cosine annealing algorithm are used to prevent the model from falling into a local optimum. In order to verify the effectiveness of the model, the image size is first adjusted to 256*256 during the verification process, and then the center is cropped to 224*224.
[0038] Step 4: In order to comprehensively evaluate the performance of different models in the class-imbalanced bone marrow cell dataset (BMCD-FGCD), the following common evaluation indicators are used, including: Accuracy, Precision, Recall and F1 score. Their calculation formulas are as follows:
[0039]
[0040]
[0041] In the formula, TP (True Positive), FP (False Positive), FN (False Negative), and TN (True Negative) are true positive, false positive, false negative, and true negative, respectively. TP and TN are samples predicted to be true, and FP and FN are samples predicted to be false.
[0042] Step 5: Apply the completed evaluated SCKansformer neural network model to realize the bone marrow blood cell image classification and recognition function.
[0043] Specific:
[0044] First, the SCConv encoder module of the SCKansformer neural network model is mainly composed of two units, namely the Spatial Reconstruction Unit (SRU) and the Channel Reconstruction Unit (CRU). The SRU uses a separation-reconstruction method to reduce spatial redundancy, while the CRU uses a segmentation-conversion-fusion method to reduce channel redundancy. These two units work together to reduce the redundant information of features in CNN. The patch expansion layer uses a linear layer to increase the feature dimension to twice the original value for the input feature. Then, the resolution of the input feature is expanded to twice the resolution of the input feature through the rearrange operation, and the feature dimension is reduced to one-fourth of the input feature dimension.
[0045] Specifically, the SRU module: The sequence information of the input microscopic image is mixed after the Patch operation. In order to reasonably use the spatial redundancy of the features, the SRU module is used to perform spatial refinement operations. This unit reduces spatial redundancy through the separation-reconstruction strategy. The structure is as follows Figure 2 and Figure 3 shown.
[0046] First, the input feature map is processed by Group Normalization to separate the feature map with rich information from the feature map with less information. N×C×H×W , N is the batch axis, C is the channel axis, H and W are the spatial height and width axes. μ and σ are the mean and standard deviation of X, ε is a small positive constant added for segmentation stability, and γ and β are trainable affine transformations. As shown in the following formula
[0047]
[0048] The amount of information carried by these different feature maps is then evaluated by the scaling factor γ. As shown in the following formula, the normalized correlation weight Wγ is obtained by taking the scaling factor γ as a metric, which measures the importance of different feature maps.
[0049]
[0050] The obtained correlation weight Wγ is mapped to the range of 0 to 1 through the sigmoid function and gated by the threshold. In this embodiment, the threshold is set to 0.5, and the weight exceeding the threshold is assigned to 1 to generate the information weight W1, and the weight below the threshold is assigned to 0 to obtain the non-information weight W2. Subsequently, the input feature X is multiplied by the two weighted features to obtain the key features with large information content. and redundant features with less information Finally, a cross-reconstruction operation is performed to fully integrate the two weighted information features, enhance the information flow between the two, and finally obtain the spatially refined feature map Xw. As shown in the following formula:
[0051] X ω = Gate(Sigmoid(W γ (GN(X))))
[0052] Furthermore, the CRU module: Figure 4 As shown in the figure, since there is still spatial redundancy in spatial refinement, the CRU module is introduced to improve its performance. The specific results are shown in the figure. This unit adopts the strategy of separation transformation fusion to reduce feature redundancy and replaces the standard convolution with three operators: Split, Transformi and Fus.
[0053] The Split operation refines the input space into features X ω The number of channels in one part is αC, and the number of channels in the other part is (1-α)C, where α is a hyperparameter. In this embodiment, α is set to 0.5 for segmentation features. Then, the number of channels of the two sets of features is compressed using a 1*1 convolution kernel, and the results are X up and X low .
[0054] The Transform operation converts the input X up As the input of rich feature extraction, GWC and PWC are performed respectively, that is, efficient convolution operations. GWC reduces the number of parameters and calculations, and PWC makes up for information loss and helps information flow across feature channels. Then the add operation is performed to obtain the output Y1, and the input X low As a supplement to the rich feature extraction, PWC is performed, and the result is concat-operated with the original input to obtain the output Y2.
[0055] The Fus operation uses an adaptive method to merge Y1 and Y2. Specifically, it first uses global average pooling to combine global spatial information and channel statistics to obtain pooled results S1 and S2. Then, a Softmax operation is performed on S1 and S2 to obtain the feature weight vector β 1 and β 1 Finally, the feature weight vector is used to obtain the output Y = β 1 Y 1 +β 2 Y 2 , where Y is the feature after channel extraction.
[0056] Then, if Figure 5 As shown in the figure, the global-local attention encoder is mainly composed of a multi-head self-attention module and a local module. The encoder can not only pay attention to the long-distance dependencies of microscopic cell images, but also effectively extract local features through deep separable convolution, and further obtain the multivariate relationship between adjacent pixels.
[0057] A further configuration of this embodiment is a multi-head self-attention module: This module calculates different weights of different objects through interactive learning between input vectors (self-attention mechanism), so as to find more concerned area information. The self-attention calculation formula is shown as follows:
[0058]
[0059] Attemtion(Q,K,V)=Cos(Q,K)×V
[0060] It is called self-attention because Q, K, V are all derived from the input vector sequence X∈R N×d The mapping formula obtained by linear mapping is as follows, where W Q , W K , W V ∈R d×d :
[0061] Q=XW Q
[0062] K=XW K
[0063] V=XW V
[0064] As shown in the above formula, Q represents the query information, K represents the vector used to match Q, also known as the index, and V represents the content vector. First, calculate the transpose multiplication of Q and K, that is, the inner product of the point features, to obtain the correlation between the feature points, and then divide by d represents the dimension of the hidden layer. This operation has two purposes: 1) to prevent the value of the Softmax function from being too large in the subsequent use, causing the partial derivative to approach 0. 2) to make QK T The distribution with an expectation of 0 and a variance of 1 is equivalent to a normalization operation. Then the Softmax function is used to calculate the weight information of each object, as shown in formula (3.6), where e represents a natural constant and z i represents the i-th element of vector z, The purpose of the Softmax function is to obtain the probability distribution of different elements in the vector z, that is, to obtain the vector The weight information of each object in . Finally, the weight information is used to match V to obtain the weighted output vector.
[0065]
[0066] In order to focus on the input image sequence from multiple angles in parallel and improve the model's expressiveness and generalization capabilities, a multi-head self-attention mechanism is used. The multi-head self-attention mechanism performs multiple attention calculations on each input image sequence, focusing on its relationship with other image sequences from multiple angles, so that the model can jointly focus on information from different representation subspaces at different positions, thereby capturing more comprehensive and detailed global semantic information. The calculation formula of the multi-head self-attention mechanism is to splice different attention heads. This parallel operation can improve the efficiency of the model and speed up the training and reasoning process of the model. As shown in the following formula:
[0067] MHA(Q,K,V)=Concat(Attention(Q 1 , K1, V 1 ),…,Attention(Q 1 , K 1 , V i ))
[0068] Local module: Since the self-attention mechanism is only used on the flattened feature map, only the global dependencies between all input image sequences are captured, which will result in a lack of information interaction between adjacent pixels, that is, the local dependencies between nearby pixels cannot be modeled. To solve this problem, this paper proposes a local module, which introduces deep separable convolution to model the local dependencies between adjacent pixels. The local module uses convolution to learn local features. Their similarities are that they first expand the dimension and then restore it to the original dimension. The advantage of this is that it can learn more nonlinear relationships between features and make full use of the global and local information features of microscopic cell images.
[0069] The main process of the local module is shown in the figure: 1) First, the serialized vector is divided into trainable serialized vectors and the image serialization vector 2) The image serialization vector is reconstructed into a two-dimensional feature map through the Sequence to Image (S2I) operation, the purpose of which is to restore the physical position and adjacency between pixels. 3) A 1*1 convolution kernel with a step size of 1 is used to expand the dimension, and batch normalization and h-swish (as shown in formula (3.8)) activation function are used to solve the gradient disappearance problem and add nonlinearity to the model. 4) After the dimension is expanded, a depth-wise convolution (Depth-wise Convolution) with a step size of 1 and 3*3 is used to obtain the feature map z out . 5) Use a 1*1 convolution with a step size of 1 to restore the dimension. 6) After completing the local feature extraction, the feature map is re-expanded into a one-dimensional serialized vector 7) Concatenate the one-dimensional serialized vector with the trainable serialized vector as the input vector for the next layer. This process can be represented by the following series of formulas:
[0070] h-swish(x)=x*sigmoid(x)
[0071]
[0072]
[0073] z expand =h-swish(BN(Conv(z img )))
[0074] z out =DW(z expand )
[0075] z squeeze =BN(Conv(z out )))
[0076]
[0077] Finally, the Kansformer encoder of the SCKansformer neural network model consists of alternating multi-head self-attention layers (MSA) and KAN networks. Layer normalization (Layer Norm) is applied before each block, and residual connection (Residual Connection) is applied after each block. After the microscopic image X∈R^(H×W×C) is subjected to the Patch Embedding operation, it is reshaped into a series of two-dimensional image block sequences X_p∈R^(N×(p^2·C)), where (H,W) is the resolution of the original image, C is the number of channels, (P,P) is the resolution of each image block, and N=HW / P^2 is the number of blocks obtained. The sequence is then encoded by SCConvEncoder to process the spatial and channel redundant features, and the processed result is used as the effective input sequence length of Kansformer. Kansformer uses a constant latent vector size D in all its layers, so the image blocks are flattened and mapped to D dimensions using a trainable linear projection. The output operation is called Patch Embedding. Similar to BERT's [class] token, an embedding is added before the Embedded Patch sequence, and its state at the output of the Kansformer encoder is used as the image classification flag. After the input embedding vector that combines the category vector, image patch embedding, and position encoding into one, it is necessary to first pass the Global-Local Attention Encoder to further obtain the relationship between adjacent pixels. The processed feature vector can be fed into the Kansformer Encoder. Through the Kansformer Encoder composed of serially stacked Kansformer Encoder Blocks, the features corresponding to the learnable category embedding vector (class token) are finally extracted for image classification.
[0078] The complexity of the SCKansformer model was experimentally compared with other typical classification models and advanced microscopic image cell classification models to further verify the advancement of the SCKansformer model. Compared with the VGG series models, although these classic models perform stably in traditional image classification tasks, their network depth leads to significant computational overhead and memory consumption. These problems are particularly prominent when processing high-resolution microscopic images. In contrast, SCKansformer uses a more efficient KAN network to replace the traditional MLP layer, thereby reducing memory consumption. In addition, through the global-local attention mechanism, SCKansformer is good at capturing subtle features in microscopic images. This has led to an improvement in the F1 score and accuracy, which are 9.79 and 8.73 percentage points higher than the highest value of the VGG model, respectively. Compared with the ResNet series models, SCKansformer also shows excellent performance in processing microscopic images. The highest F1 score of the ResNet series model is 75.55, while SCKansformer reaches an astonishing 84.34, an increase of 8.79 percentage points. In terms of accuracy, the best performance of the ResNet series model is 75.48, while the SCKansformer model reaches 83.23, an improvement of 7.75 percentage points. By combining MSA and local modules, SCKansformer effectively captures the global and local features of microscopic images, thereby significantly improving the classification accuracy. Compared with advanced classification models such as ViT and lightweight EfficientNetV2, the advantages of SCKansformer are more obvious. Compared with ViT and EfficientNetV2, the accuracy of the SCKansformer model has increased by 5.81 and 6.03 percentage points, respectively. This significant improvement is mainly attributed to the SCConv encoder and global-local attention mechanism of SCKansformer, which can effectively capture the fine-grained features of microscopic images. This feature makes up for the shortcomings of ViT and EfficientNetV2 in processing high-resolution microscopic images, especially in capturing local features. Compared with traditional leukemia cell classification models such as WBCsNet and BloodCaps, the SCKansformer model has shown excellent performance in various indicators. In terms of accuracy, the SCKansformer model is 10.80 and 4.92 percentage points higher than the above models, respectively. By introducing the KAN network and SCConv encoder, SCKansformer effectively solves the limited feature extraction capability and parameter redundancy inherent in the traditional CNN model WBCsNet. In addition, by adding the GLAE encoder, SCKansformer also makes up for the high computational complexity and insufficient global information capture of the BloodCaps capsule network.
[0079] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.
Claims
1. A bone marrow cell classification and recognition method based on SCKansformer neural network, characterized in that: The steps include: Step 1, constructing a SCKansformer neural network model, wherein the SCKansformer neural network model includes an SCConv encoder, a global-local attention encoder and a Kansformer encoder; Step 2: Establish and preprocess a bone marrow blood cell dataset, and divide the dataset into a training set and a test set; Step 3: Use the training set to train the SCKansformer neural network model, and optimize the parameters of the SCKansformer neural network model based on the characteristics of bone marrow blood cell images; Step 4: Use the test set to test the trained SCKansformer neural network model, and then obtain the network model, and finally realize the bone marrow blood cell image classification and recognition function.
2. A bone marrow cell classification and identification method based on SCKansformer neural network according to claim 1, characterized in that: The SCConv encoder includes a spatial reconstruction unit and a channel reconstruction unit, which respectively reduce spatial and channel redundancy through separation-reconstruction and separation transform fusion strategies.
3. The method for classifying and identifying bone marrow cells based on the SCKansformer neural network according to claim 2, characterized in that: The global-local attention encoder includes a multi-head self-attention module and a local module. The multi-head self-attention module learns long-distance dependencies through a self-attention mechanism, and the local module extracts local features between adjacent pixels through deep separable convolution.
4. The method for classifying and identifying bone marrow cells based on the SCKansformer neural network according to claim 3, characterized in that: The Kansformer encoder consists of alternating multi-head self-attention layers and KAN networks, with layer normalization applied before each block and residual connections applied after each block.
5. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 1, characterized in that: In step 2, blood cells on the images in the data set are labeled with types.
6. The method for classifying and identifying bone marrow cells based on the SCKansformer neural network according to claim 1, characterized in that: In step 2, the preprocessing method is: first randomly crop the image to a size of 224*224, and perform data enhancement by random horizontal inversion, random rotation and mixup.
7. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 1, characterized in that: In step 3, the training method of the SCKansformer neural network model is: The training set is used as input, implemented by Python, the IDE is PyCharm, the PyTorch deep learning framework is used, it is carried out on a single GPU, the operating system is Ubuntu 18.04, the training lasts for 100 cycles, the batch size is 128, and it takes 12.6 hours.
8. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 1, characterized in that: In step 4, the evaluation method of the trained SCKansformer neural network model is: Among them, TP, FP, FN, and TN are true positive, false positive, false negative, and true negative, respectively. TP and TN are samples predicted to be true, and FP and FN are samples predicted to be false.
9. The method for classifying and identifying bone marrow cells based on the SCKansformer neural network according to claim 4, characterized in that: The implementation method of the Kansformer encoder is as follows: after the output of the global-local attention encoder integrates the category vector, image block embedding and position encoding into an integrated input embedding vector, the relationship between adjacent pixels is further obtained through the Global-LocalAttention Encoder. The processed feature vector can be fed into the Kansformer Encoder, and finally the features corresponding to the learnable category embedding vector are extracted for image classification through the Kansformer Encoder composed of serially stacked Kansformer Encoder Blocks.
10. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 4, characterized in that: After the output vector of the global-local attention encoder is input into the Kansformer encoder, the output of the SCConv encoder is used as the effective input sequence length of the Kansformer encoder. The Kansformer encoder uses a constant latent vector size D in all its layers, flattens the image block, and uses a trainable linear projection map to D dimensions. The output operation is called Patch Embedding, and its state at the output of the Kansformer encoder is used as an image classification flag.
11. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 4, characterized in that: The method for the spatial reconstruction unit to perform spatial reconstruction on the feature map is: First, the input feature map is processed by Group Normalization to separate the feature map with rich information from the feature map with less information. The expression is as follows: Among them, the input feature map X∈R N×C×H×W , N is the batch axis, C is the channel axis, H and W are the spatial height and width axes; μ and σ are the mean and standard deviation of X, ε is a small positive constant added for segmentation stability, and γ and β are trainable affine transformations; The amount of information carried by these different feature maps is then evaluated by the scaling factor γ, expressed as follows: The normalized correlation weight Wγ is obtained by taking the scaling factor γ as the metric The obtained relevant weight Wγ is mapped to the range of 0 to 1 through the sigmoid function, and gated by the threshold. The weight exceeding the threshold is assigned to 1 to generate the information weight W1, and the weight below the threshold is assigned to 0 to obtain the non-information weight W2; Then, the input feature X is multiplied by the two weighted features to obtain the key features with large information content. and redundant features with less information Finally, a cross reconstruction operation is performed to fully fuse the two weighted information features, enhance the information flow between the two, and finally obtain the spatially refined feature map Xw, as shown in the following formula: X ω =Gate(Sigmoid(W γ (GN(X))))。 12. The method for classifying and identifying bone marrow cells based on the SCKansformer neural network according to claim 11, characterized in that: The channel reconstruction unit adopts a separation-transformation-fusion strategy to reduce feature redundancy, and replaces the standard convolution with three operators: Split, Transformi and Fus.
13. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 12, characterized in that: The Split operation refines the input space feature X ω The number of channels in one part is αC, and the number of channels in the other part is (1-α)C, where α is a hyperparameter used to segment the features. Then, the number of channels of the two sets of features is compressed using a 1*1 convolution kernel, and the results are X up and X low .
14. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 13, characterized in that: The Transform operation converts the input X up As the input of rich feature extraction, GWC and PWC are used for efficient convolution operations, and then add operations are performed to obtain the output Y1; the input X low As a supplement to the rich feature extraction, a PWC efficient convolution operation is performed, and the result is concat-operated with the original input to obtain the output Y2.
15. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 14, characterized in that: The Fus operation first uses global average pooling to combine the global spatial information and channel statistical information to obtain the pooled results S1 and S2, then performs a Softmax operation on S1 and S2 to obtain feature weight vectors β1 and β1, and finally uses the feature weight vector to obtain the output Y=β1Y1+β2Y2, where Y is the feature after channel extraction.
16. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 4, characterized in that: The method for the multi-head self-attention module to learn long-distance dependencies is: MHA(Q,K,V)=Concat(Attention(σ1,K1,V1),...,Attention(Q i ,K i ,V i )) Attention(Q,K,V)=Cos(Q,K)×V Among them, Q represents the query information, K represents the vector used to match Q, also known as the index, and V represents the content vector. Q, K, V are composed of the input vector sequence X∈R N×d The mapping formula obtained by linear mapping is as follows, where W Q , W K , W V ∈R d×d : Q=XW Q K=XW K V=XW V 。 17. The method for classifying and identifying bone marrow cells based on SCKansformer neural network according to claim 16, characterized in that: The method for extracting local features by the local module is: First, the image is treated as a serialized input, the image is divided into a series of tiles, and each tile is converted into a vector representation as a serialized vector The image serialization vector is reconstructed into a two-dimensional feature map through the Sequence to Image (S2I) operation, the purpose of which is to restore the physical position and adjacency between pixels; A 1*1 convolution kernel with a step size of 1 is used to expand the dimension, and batch normalization and h-swish activation function are used to solve the gradient disappearance problem and add nonlinearity to the model; After the dimension is expanded, the feature map Z is obtained by a depth-separable convolution with a step size of 1 and 3*3 out ; Use a 1*1 convolution with a stride of 1 to restore the dimension; After local feature extraction, the feature map is re-expanded into a one-dimensional serialized vector Conversion of one-dimensional serialized vector to trainable serialized vector concatenated as the input vector for the next layer.
Citation Information
Cited By
Lithium battery health state prediction method and system
CN119438958A
Hyperspectral image classification method and system, electronic equipment and storage medium
CN120014356A
Construction method of medical examination recommendation model based on progressive spatial-temporal feature fusion
CN120256716A
