A method for leukocyte detection based on multi-scale fusion and deformable self-attention

Abstract

This invention discloses a white blood cell detection method based on multi-scale fusion and deformable self-attention, comprising the following steps: S1, acquiring and preprocessing a white blood cell target detection dataset; S2, constructing and training an MFDS-DETR model; S3, extracting features through a backbone network to obtain a feature map; S4, obtaining a multi-scale fused feature vector through a high-level feature fusion pyramid; S5, obtaining the final output vector through a deformable self-attention module; S6, using the final output vector from step S5 as input, extracting global features through an encoder; S7, using the extracted global features as input, obtaining the final target detection result through a decoder. This invention uses high-level semantic features as weights to filter low-level features and adds the filtered features to the high-level semantic features point by point, improving the model's feature extraction and representation capabilities. The use of deformable self-attention to extract global white blood cell features makes model training more convenient.

Description

Technical Field

[0001] This invention relates to the fields of computer vision deep learning and medical image processing technology, specifically to a white blood cell detection method based on multi-scale fusion and deformable self-attention. Background Technology

[0002] In routine blood tests performed in hospitals, a microscope is typically used to observe the patient's white blood cells. Automated white blood cell classification requires segmenting the white blood cells from the patient's blood microscopic image, followed by automatic classification and counting using a white blood cell classifier. This method usually requires high-precision medical image segmentation technology or significant human resources, and can be affected by image quality or environmental factors. Although automated white blood cell classification is more efficient and accurate than manual counting by doctors under a microscope, it still has limitations. Therefore, to further simplify the process and eliminate the image segmentation step, this chapter presents a study on white blood cell target detection, achieving a one-step process—direct white blood cell classification and counting from blood microscopic images—which helps doctors quickly diagnose conditions.

[0003] The following problems exist in the detection of white blood cells in blood microscopy: 1) Different hospitals use different equipment to collect blood images, resulting in color differences in the generated images. 2) White blood cell images have limited features, which makes white blood cell detection difficult. 3) Different hospitals use different magnification levels, causing the same type of white blood cells to appear at different sizes in different blood images. Furthermore, different white blood cells themselves have different sizes, which further complicates white blood cell detection. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a white blood cell detection method based on multi-scale fusion and deformable self-attention. By fusing feature maps of different scales and using a deformable self-attention module, the feature extraction and expression capabilities of the model are improved.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] A method for detecting white blood cells based on multi-scale fusion and deformable self-attention includes the following steps:

[0007] S1. Obtain and preprocess the white blood cell target detection dataset;

[0008] S2. Construct and train the MFDS-DETR model.

[0009] The MFDS-DETR model includes a backbone network, a high-level feature fusion pyramid, a deformable self-attention module, an encoder, and a decoder.

[0010] The backbone network adopts ResNet-50 with residual connections, and a convolutional block is added to the ResNet-50. The convolutional block includes two 1*1 convolutions and one 3*3 convolution. The high-level filtering feature fusion pyramid includes a feature selection module and a feature fusion module. The deformable self-attention module includes an offset module and an attention module.

[0011] S3. Using the trained MFDS-DETR model, the preprocessed dataset is taken as input and feature maps of different scales are extracted through the backbone network.

[0012] S4. In the high-level screening-feature fusion pyramid, the feature selection module selects features from feature maps at different scales. Then, the feature fusion module fuses high-level and low-level information at multiple scales, so that each layer of features has rich semantic information.

[0013] S5. In the deformable self-attention module, offset output vectors of different scales are obtained through the offset module. Finally, the offset output vectors of different scales are weighted through the attention module to obtain the final output vector.

[0014] S6. Using the final output vector from step S5 as input, extract global features through the encoder;

[0015] S7. The extracted global features are used as input and the final target detection result is obtained through the decoder.

[0016] Preferably, the data preprocessing method in step S1 is as follows:

[0017] The white blood cells in the image data of the white blood cell target detection dataset are labeled as (x1, y1) and (x2, y2) using LabelMe, where x1 and y1 represent the coordinates of the upper left corner and x2 and y2 represent the coordinates of the lower right corner. The label format is then converted into (x1, y1) and the width and height of the label box. The label files of all images are then integrated into COCO format label data.

[0018] After obtaining training data in COCO format, the input images need to be augmented. This is done by augmenting the training dataset using operations such as random cropping, center cropping, adding salt-and-pepper noise, and Gaussian noise.

[0019] Preferably, the feature selection module includes a channel attention module and a dimension matching module. The channel attention module includes global average pooling and global max pooling, and the dimension matching module includes 1*1 convolution.

[0020] The feature selection method using the feature selection module is as follows: First, global average pooling and global max pooling are performed. Then, the calculated features are summed. Subsequently, the weight value of each channel is calculated using the sigmoid activation function. Finally, the weight f of each channel is output. CA ∈R C×1×1 The obtained weight information is then multiplied with the feature map at its own scale to obtain the filtered feature map; in the dimension matching module, a 1*1 convolution is used to reduce the number of channels of the feature map at each scale to 256.

[0021] Preferably, the feature fusion module includes transposed convolution, bilinear interpolation combination, and CA module;

[0022] The method for obtaining multi-scale fused feature vectors through the feature fusion module is as follows: Given an input high-level feature f high ∈R C×H×W Input low-level features First, the high-level features are expanded by using a transposed convolution with a stride of 2 and a kernel size of 3*3. Subsequently, to unify the sizes of high-level and low-level features, bilinear interpolation was used to downsample or upsample the high-level features, resulting in... Where C represents the number of channels in the feature map, H represents the feature map height, and W represents the feature map width; after obtaining features of consistent size, the CA module is used to convert high-level features into corresponding attention weights to filter low-level features. Finally, the filtered low-level features are fused with the high-level features to obtain a multi-scale fused feature vector, as shown in the following expression:

[0023] f att =BL(T-Conv(f) high ))

[0024] f out =f low *CA(f att )+f att

[0025] Preferably, the method for obtaining the offset output vectors at different scales in step S5 is as follows:

[0026] Before the multi-scale fused feature vector is input into the offset module, it first needs to be converted into a feature map. Then, the input query vector is obtained based on the coordinates of the reference point. A linear transformation is applied to the query vector to obtain the offset Δp of the query vector. q Then, based on the offset of each reference point, the point of interest (i.e., the sampling point) is found, and the output offset of that point is obtained using bilinear interpolation. valueEach query vector has H attention heads, and for each attention head there are K offset points.

[0027] Preferably, the method for obtaining the final output vector in step S5 is as follows:

[0028] First, a linear transformation is performed on the input query vector. Then, the Softmax function is used to obtain the weight vector for each offset. The output of each offset is multiplied by the corresponding weight vector and summed to obtain the sample value. value Then, each focus point in the reference point is concatenated to obtain the final sampled output vector. output Finally, the sampled output vector is linearly transformed to obtain the final output, Output, as shown in the following expression:

[0029] Weight = Softmax(WQ)

[0030]

[0031]

[0032] Output = W * Sample output .

[0033] The encoder has six layers, and each layer consists of a deformable self-attention and FFN network structure.

[0034] The encoder extracts global features by adding spatial location encoding and scale encoding to the multi-scale feature fusion vector as input to the encoder. This input vector extracts global features of white blood cells through variable self-attention, and then normalizes the output vector through a residual structure. The output vector is then used to learn white blood cell features through an FFN network structure.

[0035] Preferably, the decoder includes a self-attention feature extraction module and a cross-attention feature extraction module;

[0036] The self-attention feature extraction module includes a self-attention module and an FFN (Flexible Input Network). The self-attention module learns from the interaction between input vectors (self-attention mechanism) and calculates the weights of different objects to find more relevant region information. The self-attention calculation formula is as follows:

[0037]

[0038] Attention(Q,K,V)=Cos(Q,K)×V

[0039] The cross-attention feature extraction module has the same structure as the encoder, but the inputs to the encoder and the cross-attention mechanism come from the output position encoding and the global features extracted from the last layer of the encoder, respectively.

[0040] Preferably, the loss function of the MFDS-DETR model is a joint loss function composed of classification loss, regression loss, and auxiliary loss, expressed as follows:

[0041]

[0042] This invention has the following characteristics and beneficial effects:

[0043] This invention proposes a leukocyte detection model based on multi-scale feature fusion and deformable self-attention (MFDS-DETR). This model improves the feature representation ability of the model by fusing feature maps at different scales, thus solving the multi-scale problem in leukocyte datasets.

[0044] Furthermore, to address the limitation of traditional CNN detection in failing to capture global features of white blood cell images, a deformable self-attention approach is employed to extract these global features. This eliminates the need to examine all possible image spaces, reduces the number of parameters in traditional models, and accelerates training. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a structural diagram of the MFDS-DETR model of the present invention.

[0047] Figure 2 This is a structural diagram of the HS-FPN of the present invention.

[0048] Figure 3 Selecting a feature fusion module for this invention

[0049] Figure 4 Visualization comparing the complexity of self-attention and deformable self-attention in this invention.

[0050] Figure 5 This invention relates to a deformable self-attention module.

[0051] Figure 6This invention compares the recognition performance of different detection models on a white blood cell target detection dataset.

[0052] Figure 7 This is a graph showing the AP value curves for different multi-scale fusion methods of the present invention.

[0053] Figure 8 This is a graph showing the AP value curves corresponding to different encoder layers in this invention.

[0054] Figure 9 This is a graph showing the AP value curves corresponding to different decoder layers in this invention.

[0055] Figure 10 AP value curves corresponding to different positional encodings in this invention

[0056] Figure 11 The graph shows the AP value curves corresponding to different loss functions of this invention.

[0057] Figure 12 Visualization of sampling points and corresponding attention weights for different attention heads in this invention.

[0058] Figure 13 Visualization of the sampling points and attention weights of the last layer encoder in this invention.

[0059] Figure 14 Visualization of the sampling points and attention weights of the last layer decoder in this invention.

[0060] Figure 15 The image shows the prediction results of MFDS-DETR on the WBCDD dataset (the black box represents Ground Truth, and the boxes of different colors represent their prediction boxes). Detailed Implementation

[0061] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0062] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0063] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0064] This invention provides a method for detecting white blood cells based on multi-scale fusion and deformable self-attention, comprising the following steps:

[0065] S1. Obtain and preprocess the white blood cell target detection dataset;

[0066] S2. Construct and train the MFDS-DETR model.

[0067] Specifically, the MFDS-DETR model structure is as follows: Figure 1 As shown, the MFDS-DETR model mainly consists of the following parts: backbone network, high-level select-feature pyramid network (HS-FPN), deformable self-attention module, encoder, and decoder.

[0068] S3. Using the trained MFDS-DETR model, the preprocessed dataset is used as input in the backbone network. Features are extracted through the backbone network to obtain feature maps. First, 1*1 convolution is used to reduce the number of channels, then 3*3 convolution is used to reduce the size of the feature map, and finally 1*1 convolution is used to expand the number of channels, thereby outputting feature maps of different scales.

[0069] For the backbone network, MFDS-DETR uses an improved version of ResNet-50 to extract features. ResNet-50 uses residual connections to solve the gradient vanishing problem and makes the model converge faster, avoiding the degradation problem of deep neural networks. Table 1 shows the network structure of the improved ResNet-50 in MFDS-DETR. Since white blood cell images have fewer features and require deeper semantic information, this backbone network adds a convolutional block to the original ResNet-50 model to extract deeper information. This convolutional block uses a similar structure to that in ResNet-50, first using a 1*1 convolution to reduce the number of channels, then using a 3*3 convolution to reduce the feature map size, and finally using a 1*1 convolution to expand the number of channels.

[0070] The specific parameters of the backbone network are shown in Table 1:

[0071] Table 1. Model parameters of the backbone network

[0072]

[0073] S4. In the high-level filtering feature fusion pyramid, the feature selection module selects features from feature maps of different scales to obtain high-level features and low-level features. Then, the feature fusion module is used to fuse the high-level information and low-level information at multiple scales to obtain a multi-scale fused feature vector.

[0074] The High-Level Filtering-Feature Fusion Pyramid (HS-FPN) can be divided into two parts: 1) Feature Selection and 2) Feature Fusion. First, feature selection is performed on feature maps at different scales. Then, a feature fusion module is used to fuse high-level and low-level information. The specific implementation of feature selection is as follows:

[0075] The feature selection module mainly consists of two parts: a channel attention (CA) module and a dimension matching module. For example... Figure 2 As shown in the CA module, for the input feature map f in ∈R C×H×W(C represents the number of channels in the feature map, H represents the feature map height, and W represents the feature map width). First, the feature maps pass through two pooling layers: global average pooling and global max pooling. Then, the calculated features are summed. Next, the sigmoid activation function is used to calculate the weight value for each channel. Finally, the weight f for each channel is output. CA ∈R C×1×1 In the channel attention module, global average pooling and global max pooling are used to calculate the average and maximum values ​​for each channel. Average pooling aims to capture all information from the feature map evenly, without losing too much information, while max pooling aims to extract the most representative information from each channel. Therefore, using both pooling methods in the CA module can extract the most representative information from each channel without losing too much information. The obtained weights are then multiplied by the feature map at its own scale to obtain the filtered feature map. Since feature maps at different scales have different numbers of channels, dimension matching is required before feature fusion. In the dimension matching module, 1*1 convolutions are used to reduce the number of channels in the feature map at each scale to 256.

[0076] For feature fusion, a common method is to directly add upsampled high-level features to low-level features pixel by pixel, so that each layer of features has rich semantic information. However, the above method simply adds pixels from different feature layers without selecting features. To address this issue, this invention designs a Select Feature Fusion (SFF) module, which uses high-level features as weights to filter important semantic information in low-level features, thus achieving targeted feature fusion. Figure 3 As shown, given an input high-level feature f high ∈R C×H×W Input low-level features First, the high-level features are expanded by using transposed convolutions (T-Conv) with a stride of 2 and a kernel size of 3*3 to increase the feature size. Subsequently, to unify the sizes of high-level and low-level features, bilinear interpolation is used to downsample or upsample the high-level features, resulting in... After obtaining features of consistent size, the CA module is used to convert high-level features into corresponding attention weights to filter low-level features. Finally, the filtered low-level features are fused with high-level features to improve the feature representation capability of the model. Formulas (1) and (2) represent the process of selecting and fusing features. In the process of image upsampling, this invention uses a combination of transposed convolution and bilinear interpolation to restore the scale of high-level features. The advantage of bilinear interpolation is that it is a simple and fast method that achieves image magnification and reduction through pixel operations. The advantages of transposed convolution are: 1) Adapting to data through learnable parameters: The output of transposed convolution is not only an amplification of the feature map, but also a certain reconstruction of the input through convolution. It first fills with 0 to expand the feature map, and then uses a convolution kernel to perform convolution operations. This process is adjusted through learnable parameters (convolution kernel). 2) Handling non-uniform sampling: Transposed convolution can sample different regions of the input image at different positions of the output image, unlike bilinear interpolation which can only select a few surrounding points. Therefore, it can handle non-uniform sampling.

[0077] f att =BL(T-Conv(f) high )) (1)

[0078] f out =f low *CA(f att )+f att (2)

[0079] S5. In the deformable self-attention module, offset output vectors of different scales are obtained through the offset module. Finally, the offset output vectors of different scales are weighted through the attention module to obtain the final output vector.

[0080] Regarding the deformable self-attention module, unlike traditional self-attention mechanisms, it only focuses on a set of key sampling points around the image reference point and does not need to consider the size of the feature map. For example... Figure 4 As shown, in the square matrix, circles represent attention between keys and queries, orange indicates retained connections, white indicates deleted connections, and the gray lines on the diagonal represent the position of the reference point. Compared to the traditional self-attention mechanism, which has a quadratic exponential complexity with respect to the input, the deformable self-attention module reduces the complexity of attention with respect to the input by sparsifying the query, achieving linear complexity with respect to the input. The deformable self-attention module mainly consists of two important parts: the offset module and the attention module.

[0081] Offset module: such as Figure 5As shown, before the vector is input to the offset module, it first needs to be converted into a feature map, and then the input query vector is obtained based on the coordinates of the reference point. A linear transformation is then applied to the query vector to obtain its offset Δp. q Furthermore, a linear transformation is applied to the input feature map to obtain the content feature map. Then, based on the offset of each reference point, the point of interest for that reference point, i.e., the sampling point, is found, and the output offset of that point is obtained using bilinear interpolation. value As shown in the figure, for each query vector, there are H attention heads, and for each attention head, there are K offset points. In the experiments in this chapter, H is 8 and K is 4.

[0082] Attention module: such as Figure 5 As shown, in the attention module, the input query vector is first linearly transformed, then the Softmax function is used to obtain the weight vector for each offset. Finally, the output of each offset learned in the offset module is multiplied by the corresponding weight vector and summed to obtain the sample value. value Then, each focus point in the reference point is concatenated to obtain the final sampled output vector Sample. output Finally, the sampled output vector is linearly transformed to obtain the final output, Output. Its specific formulas are shown in (3), (4), (5), and (6).

[0083] Weight = Softmax(WQ) (3)

[0084]

[0085]

[0086] Output = W * Sample output (6)

[0087] Because MFDS-DETR uses the HS-FPN module to perform multi-scale fusion of the backbone network's feature input, the encoder input is a multi-scale feature map. To more fully extract leukocyte feature information from different scales, a multi-scale deformable attention module is employed. This module not only learns offsets based on the scale of the input reference point but also learns offsets at different scales based on the relative positions of the normalized reference point at different scales. For each scale's offset, a deformable self-attention offset module calculates the offset output vector for that reference point. Finally, the weighted sum of the offset output vectors from different scales yields the final output vector.

[0088] S6. Using the final output vector from step S5 as input, extract global features through the encoder;

[0089] Regarding the encoder and decoder sections, the encoder's role is to extract global features from the white blood cell image. For example... Figure 1 As shown, the encoder input consists of multi-scale features with added spatial location and scale encoding. A single encoder layer comprises a deformable self-attention module and an FFN module. The most crucial element in the deformable self-attention is the reference point location. Therefore, the reference point initialization in the deformable self-attention module follows these steps: 1) Find the center coordinates of each pixel at each scale. 2) Normalize by dividing by the width and height of the feature maps at different scales. After obtaining the reference point locations, the output vector is obtained using the multi-scale deformable self-attention module. It is worth noting that the deformable self-attention module typically uses eight attention heads, which can be obtained from... Figure 12 As observed, each head focuses on offsets in different directions. After obtaining the output vector, it is normalized using a residual structure (Add & Norm method) to prevent gradient vanishing and accelerate model convergence. The output vector is then passed through an FFN network structure to further learn white blood cell features. FFN is a multilayer perceptron that expands and reduces dimensionality to enable the model to learn more nonlinear relationships between features. In the MFDS-DETR model, a 6-layer encoder is used, and this invention demonstrates through ablation experiments that the 6-layer encoder performs optimally.

[0090] S7. The extracted global features are used as input and the final target detection result is obtained through the decoder.

[0091] The decoder's role is to learn the relationships between different detection feature representations, that is, to learn the target's location and category information. For example... Figure 1 As shown, a single-layer decoder consists of two parts: a self-attention feature extraction module and a cross-attention feature extraction module. 1) The self-attention feature extraction module consists of two parts: a self-attention module and an FFN. 2) The cross-attention feature extraction module is structurally similar to the encoder, but unlike the encoder, the keys and queries of the cross-attention mechanism both come from the output positional encoding (Object Queries), and its values ​​come from the global features extracted by the last layer of the encoder.

[0092] Unlike traditional object detection tasks, this model uses a Transformer architecture, treating object detection as a set prediction task. The difficulty in training the model lies in how to score the prediction set based on the original target set. To address this, the Hungarian algorithm is used to match the original target set and the prediction set. As shown in formula (8), assuming the target set is y and the prediction set is... The Hungarian algorithm is used to find the matching method that minimizes the loss function. The calculation formula for the loss function is shown in Equation (9), which mainly consists of classification loss and regression loss function.

[0093]

[0094]

[0095] (1) Classification loss

[0096] Since 100 object lookup boxes are set in the decoder, while the actual target of a white blood cell image is only 2 or 3 white blood cells, this will cause a serious imbalance between positive and negative samples. Therefore, the Focal loss function is introduced.

[0097] The Focal loss function was first proposed by Kaiming He in RetinaNet, primarily used to address the problems of imbalanced positive and negative samples and learning from difficult-to-classify samples in object detection. Due to the long-tail phenomenon in current datasets, Focal loss can be used as the model's classification loss. It is a derivative of the cross-entropy loss function, and its calculation formula is as follows:

[0098]

[0099] (2) Regression loss

[0100] Bounding box detection is a crucial step in object detection. Therefore, to enable the model to learn the position of the bounding box, a regression loss function is set. Existing regression loss functions include L1, Smoothing L1, IoU, and GIoU. Since the most commonly used L1 loss function is affected by scale, L1 and GIoU loss functions are used as regression loss functions in the joint loss function, as shown in equation (10). y and This represents the predicted bounding box and the target bounding box paired using the Hungarian algorithm, λ GIoU and λ L1 These are two hyperparameters, representing the weights of GIoU and L1 loss. The loss function is represented by GIoU, while the L1 loss function is the absolute value of the difference between the position of the target box and the predicted box.

[0101]

[0102] The formulas for calculating the GIoU loss function are shown in (11)(12)(13), assuming the original target box is a box. y The predicted target box is First, the IoU value between the two bounding boxes is calculated, which is the intersection of the two boxes divided by their union. Furthermore, to address the issue of non-overlapping bounding boxes failing to reflect distance, the minimum bounding rectangle (Area) of the target and predicted bounding boxes is first calculated. c and use IoU values ​​with Area c The GIoU is obtained by subtracting the percentages of areas not covered by either bounding box. The advantages of the GIoU loss function are: 1) It is scale-insensitive and unaffected by object size. 2) GIoU considers not only the overlapping area between the target and predicted bounding boxes but also the non-overlapping area, better reflecting the overlap between the predicted and target bounding boxes. 3) It solves the problem that IoU cannot reflect distance when the predicted and target bounding boxes do not intersect.

[0103]

[0104]

[0105]

[0106] In addition, an auxiliary function is also incorporated into the loss function. Unlike the two loss functions mentioned above, this auxiliary loss is mainly used to accelerate model training. In the original model, only the output of the last layer of the encoder is used to predict the target, while the auxiliary loss is used for prediction of the output of each layer of the encoder, and the classification and loss function of each layer are calculated for model training. Therefore, the final loss function is calculated as shown in Equation (10), that is, each decoder layer is used as the last layer of the model to predict the result, where N is the number of decoder layers.

[0107] Classification loss and regression loss are used to optimize the model, i.e., to find the most suitable matching value, while auxiliary loss is used to accelerate the convergence of the model. It mainly calculates the classification loss and regression loss of each decoder layer.

[0108]

[0109] To verify the effectiveness of the above-mentioned white blood cell detection method based on multi-scale fusion and deformable self-attention, the following specific embodiments are provided:

[0110] In this embodiment, the acquired dataset

[0111] Three datasets were used for validation: the White Blood Cell Detection Dataset (WBCDD), the LISC dataset, and the BCCD dataset. The WBCDD dataset was collected internally, while the LISC and BCCD datasets were publicly available datasets used to validate the model's generalization ability. Tables 2 and 3 show the number of each cell type in the different datasets. Figure 6 Images of white blood cells from various datasets are displayed.

[0112] WBCDD was collected from a local hospital. The images in this dataset were obtained by doctors in the laboratory department through microscopic observation of patients' blood images. Under the guidance of professional doctors, the LabelMe tool was used to annotate the white blood cell bounding boxes. This white blood cell dataset contains 684 white blood cell image samples, representing five types of white blood cells: neutrophils (NEU), eosinophils (EOS), monocytes (MON), basophils (BAS), and lymphocytes (LYM), totaling 1257 white blood cells. Before the experiment, the dataset was divided into categories, with the training set containing 540 image samples and the test set containing 144 image samples.

[0113] LISC is an early white blood cell dataset. It consists of blood images obtained from peripheral blood of healthy volunteers. Smears were stained using the Gismo-right technique and observed using a Sony SSCD50AP camera under a 100x Axioskope 40 microscope. A hematologist then classified the 250 blood images into five fine-grained categories. However, since this is a classification dataset, the LabelMe tool was used for white blood cell annotation to enable its use in object detection models. Before the experiment, the dataset was divided into training and testing sets, with 200 images in the training set and 50 images in the testing set.

[0114] The BCCD dataset is a publicly available, labeled dataset of blood cell images called the Blood CellCount Dataset. This dataset is also stained using the Gismo-right technique and observed using a CCD color camera under a conventional optical microscope at 100x magnification, and annotated by a hematologist. The dataset contains 364 images, divided into three classes: white blood cells (WBCs), red blood cells (RBCs), and platelets. Before experiments, the dataset was split into training and testing sets, with the training set containing 292 images and the testing set containing 72 images.

[0115] Table 2. Number of different white blood cell categories in LISC and WBCDD

[0116]

[0117] Table 3. Number of cells of different categories in the BCCD dataset

[0118]

[0119] In this embodiment, step S2 primarily utilizes Python to construct the MFDS-DETR model, employing the PyTorch deep learning framework. Since the MFDS-DETR model cannot converge quickly on small datasets, it is first trained on the publicly available MS COCO dataset and then fine-tuned on a white blood cell target detection dataset using transfer learning. The hardware for this experiment consisted of a 24GB GPU (NVIDIA GeForce RTX3090) and the operating system was Ubuntu 20.04.

[0120] The MFDS-DETR model was trained with a batch size of 100. The learning rate for the backbone network was set to 0.00002, the encoder and decoder to 0.0002, and the HS-FPN to 0.0003. The learning rate decayed using StepLR, decreasing to 0.1 every 40 batches. The AdamW optimizer was used to fine-tune the model, with β1 = 0.9, β2 = 0.999, and weight decay set to 0.0001.

[0121] In this embodiment, since the obtained patient blood images do not contain labeled white blood cells, the LabelMe tool is first used to label and classify the white blood cells in the blood images in order to construct our own white blood cell target detection dataset. LabelMe is a web-based open-source tool developed by the Computer Science and Artificial Intelligence Laboratory (CSAIL) at MIT for manually annotating image datasets.

[0122] Before training, the dataset must be preprocessed. LabelMe uses the format (x1, y1), (x2, y2), where x1 and y1 represent the coordinates of the top-left corner, and x2 and y2 represent the coordinates of the bottom-right corner. Each image has a JSON-formatted annotation file. Therefore, before training, the annotation information is preprocessed. First, the annotation format is converted to (x1, y1) along with the width and height of the bounding box. Then, the annotation files from all images in the training and test sets are combined into a COCO-formatted annotation dataset.

[0123] After obtaining the training data in COCO format, data augmentation of the input images is necessary. The calculated mean and variance of each channel in this dataset are [0.8063, 0.7413, 0.6825] and [0.1429, 0.1497, 0.1121], respectively. The mean is relatively high, while the variance is relatively low. This can cause the model to prioritize samples with high means during training, leading to overfitting and poor performance on the validation set. Therefore, to address the overfitting problem, data augmentation of the training dataset is performed. In the experiments, random cropping, center cropping, and the addition of salt-and-pepper noise and Gaussian noise are used for data augmentation.

[0124] In this embodiment, predictive analysis is performed using the constructed MFDS-DETR model:

[0125] Unlike classification tasks, in object detection tasks, TP represents the number of predicted boxes with an IoU greater than a threshold with the target box, FP represents the number of predicted boxes with an IoU less than or equal to the threshold, and FN represents the number of predicted boxes with an IoU of 0, i.e., the number of no true target boxes detected. Therefore, in object detection tasks, precision represents the percentage of correctly predicted boxes among all predicted boxes, and recall represents the percentage of all correctly predicted boxes among the target boxes. Simply using precision or recall to evaluate the entire detector is unfair because different detection tasks have different metrics. Therefore, to more fairly evaluate detectors, AP is usually used to evaluate the model. The AP calculation formula follows these steps: 1) Select an IoU threshold. 2) For each class, sort all predicted boxes from highest to lowest confidence. 3) Starting with the predicted boxes with high confidence, calculate precision and recall sequentially. 4) Calculate the AP for that class. class That is, the area under the PR curve, and its formula is shown in (15). 5) For all categories of AP class The average value is used to obtain the AP at this IoU threshold. i 6) For different thresholds, repeat steps 2-5 to calculate AP at different thresholds. i Subsequently, for different thresholds of AP i The final AP value is obtained by averaging, as shown in Equation (16), where the IoU threshold is 10 thresholds with an interval of 0.05 from 0.5 to 0.95.

[0126]

[0127]

[0128] The effectiveness and generalization ability of the MFDS-DETR model were validated through experiments on different white blood cell target detection datasets. These experiments were conducted on three datasets: the White Blood Cell Target Detection Dataset (WBCDD), LISC, and BCCD. MFDS-DETR was compared with commonly used models in the target detection field, such as Faster R-CNN, SSD, RetinaNet, and DETR. Table 4 shows the results of the MFDS-DETR model on the white blood cell target detection dataset. The AP and AP of the proposed MFDS-DETR model on the white blood cell target detection dataset are shown in Table 4. 50 The accuracy rates reached 79.7% and 97.2% respectively, indicating that the model's use of multi-scale and global feature extraction methods contributes to the accuracy of white blood cell target detection. Compared to the traditional two-stage target detection model (Faster R-CNN), this model achieves higher accuracy in AP and AP. 50 These improvements were 21.5% and 23.5% respectively. Compared to the traditional single-stage multi-scale feature extraction object detection model (SSD), AP and AP... 50 These improvements were 15.5% and 16.7%, respectively. Compared to the global feature extraction object detection model (DETR), AP and AP... 50 The improvements were 12.9% and 10.8%, respectively. To further illustrate in which aspect the model improved the detection results, this invention calculated the AP value for each type of white blood cell for validation. As shown in Table 4, in the white blood cell target detection dataset, compared with the baseline model (Deformable DETR), the AP values ​​for eosinophils and lymphocytes were significantly improved, by 10.5% and 6.6%, respectively.

[0129] Table 4 shows the results of the MFDS-DETR model on the white blood cell target detection dataset.

[0130]

[0131] To further verify the model's generalization ability, experiments were conducted on the LISC and BCCD datasets. As shown in Tables 5 and 6, on the LISC dataset, AP and AP... 75 It is optimal, despite its AP 50While it doesn't outperform Faster R-CNN and the baseline model (Deformable DETR), the difference is only 0.1%, and it achieves optimal detection results for all cell types except neutrophils. This is because neutrophils in the LISC dataset are segmented neutrophils, which have significantly different nuclei compared to other types of white blood cells, allowing the two-stage target detection model to more accurately locate and classify them. On the BCCD dataset, it achieves optimal results across all metrics compared to other models.

[0132] Table 5 Comparison of recognition performance of different detection models on the LISC dataset.

[0133]

[0134] Table 6 Comparison of recognition performance of different detection models on BCCD

[0135]

[0136]

[0137] Because of the scale differences among leukocytes, this invention designs HS-FPN to selectively fuse high-level semantic information and low-level features in order to more accurately locate and classify leukocytes. To demonstrate that the designed HS-FPN can more effectively fuse multi-scale features, it is compared with other multi-scale feature fusion methods (FPN, BiFPN, PaFPN, FaPN). As shown in Table 7, the AP of HS-FPN is improved by 3.6% compared to FPN, and the AP... 50 and AP 75 The results were improved by 4.1% and 4.6% respectively, and the HS-FPN model performed best in white blood cell detection compared to other advanced FPN models. This proves that the HS-FPN model can locate high-level semantic information in low-level features by using high-level semantics as weights to filter low-level features, and can more effectively fuse high-level semantic information of white blood cell images with low-level features. Figure 7 The AP curves for different FPNs are shown, among which Figure 7 'a' is the AP indicator curve, while Figure 7 b is the AP curve when the IoU threshold is 0.5.

[0138] Table 7 Comparison of different multi-scale feature fusion methods

[0139]

[0140] Furthermore, as shown in Table 7, the upsampling combination of transposed convolution and bilinear interpolation in HS-FPN performs better than the upsampling method using only bilinear interpolation.

[0141] In MFDS-DETR, the encoder is crucial for the model to learn global features. To verify the importance of the model learning global features, this invention validates this by changing the number of encoder layers. As shown in Table 8, without the encoder, AP decreased by 2.8%. 50 and AP 75 The performance decreased by 3.2% and 3.0% respectively. Using a single encoder layer not only failed to improve performance but actually decreased slightly, indicating that a single encoder layer is insufficient to accurately extract global features from an image. The AP (Average Per Second) increased with the number of encoder layers. Therefore, experiments demonstrate that learning global features is crucial, and the encoder plays a key role in understanding white blood cells and their locations. Figure 8 The AP curves for different encoder layers are shown, where Figure 8 'a' is the AP index curve. Figure 8 b is the AP curve when the IoU threshold is 0.5.

[0142] Table 8. Importance of the encoder to the model

[0143]

[0144] In MFDS-DETR, the decoder is the component that simulates the relationship between feature representations of different detections. To verify the importance of the decoder, this invention verifies it by changing the number of layers in the decoder. (See Table 9 and...) Figure 9 As shown, the detection performance decreases continuously with the decrease in the number of encoder layers. Compared to using only one encoder layer as the model's prediction output, its AP and AP are significantly lower. 50 and AP 75 They decreased by 1.6%, 2.7%, and 2.6%, respectively.

[0145] Table 9. Importance of the Decoder for the Model

[0146]

[0147] In MFDS-DETR, there are three types of positional encoding: spatial positional encoding, scale encoding, and output positional encoding (object queries).

[0148] Spatial location coding: In order to learn the global features of an image, the image needs to be serialized. Therefore, in order to indicate the position of the serialized image in the original image, location coding needs to be added to each sequence block. Unlike DETR, this spatial location coding is only added to the encoder. Because deformable convolution is used, the reference point in the decoder is determined by the output position, so the key does not need to be added to the spatial location coding.

[0149] Scale coding: MFDS-DETR is designed for multi-scale inputs. The location coding at different scales will be the same, which will make it impossible to distinguish the location. Therefore, in order to distinguish inputs at different scales, scale coding is added on the basis of spatial location coding.

[0150] Output location encoding: This is an essential location encoding, which is used to output the location of the predicted bounding box.

[0151] There are generally two types of spatial location coding: Learned PE and SinPE, as shown in Table 10. Experiments have shown that spatial location coding and scale coding are necessary. Without spatial location coding and scale coding, AP decreased by 3.1% and 3.4% respectively, and the encoder performed better with SinPE than with Learned PE. Figure 10 The AP curves using different positional encodings are shown.

[0152] Table 4. Importance of positional encoding for the model.

[0153]

[0154] To verify the importance of the different components of the joint loss function in this model, ablation experiments were conducted by permuting and combining different losses within the joint loss function. The MFDS-DETR joint loss function employs three types of losses: classification loss, regression loss, and auxiliary loss. The regression loss consists of the L1 bounding box loss function and the GIoU loss function. The classification loss function is crucial for model training and cannot be removed. Therefore, this invention trained a model without bounding box distance loss and a model without GIoU loss. Furthermore, to further verify the importance of the auxiliary loss, a model without auxiliary loss was also trained. Figure 11 As shown in Table 11, the GIoU loss is more important than the L1 loss. Without the L1 loss, the AP decreased by 1.4%, while without the GIoU loss function, the AP decreased by 1.9%. In the experiments, compared with the model without auxiliary loss, the AP decreased by 4.5%, thus demonstrating that the auxiliary loss function plays an important role in model training.

[0155] Table 11. Importance of different loss functions for the model

[0156]

[0157] To more intuitively understand what subspace information multi-head attention learns in MFDS-DETR, we visualize the deformable self-attention of each attention head based on the coordinates of the center point of the bounding box. This involves visualizing the offset coordinates learned by the center point at different attention heads and scales. Each sampling point is marked with a solid circle, its color representing the corresponding attention weight, while the reference point is displayed as a black cross. For example... Figure 12 As shown, for different attention heads, it learns the offset in different directions to finally obtain all sampling points of the reference point at different scales and in different directions.

[0158] This invention visualizes the sampling points and attention weights of the last layer of the encoder and decoder in the model, further clarifying what information the attention module of MFDS-DETR learns. Figure 13 and Figure 14 As shown, the representation of sampling points and attention weights is the same as... Figure 12 Similarly, the predicted bounding box boundaries are represented by different colored bounding boxes for different cells, with their category and confidence scores displayed above them. This attention module learns the features of different cells by adjusting the weights assigned to different sampling points. As can be seen from the attention of the encoder and decoder, sampling points closer to the target detection point receive higher attention.

[0159] In order to more clearly demonstrate the effectiveness of the model's predictions, Figure 15 The image shows the model's predicted categories and locations on the WBCDD dataset, along with the bounding boxes and categories in the original images. Black boxes represent Ground Truth, while other colored boxes represent the MFDS-DETR prediction results and confidence levels. Green represents lymphocytes, orange represents neutrophils, purple represents eosinophils, blue represents basophils, and orange represents monocytes. The image demonstrates that the model achieves high prediction confidence and accurate predicted locations for all five white blood cell types.

[0160] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A method for detecting white blood cells based on multi-scale fusion and deformable self-attention, characterized in that, Includes the following steps: S1. Obtain and preprocess the white blood cell target detection dataset; S2. Construct and train the MFDS-DETR model. The MFDS-DETR model includes a backbone network, a high-level feature fusion pyramid, a deformable self-attention module, an encoder, and a decoder. The backbone network adopts ResNet-50 with residual connections, and a convolutional block is added to the ResNet-50. The convolutional block includes two 1*1 convolutions and one 3*3 convolution. The high-level filtering feature fusion pyramid includes a feature selection module and a feature fusion module. The deformable self-attention module includes an offset module and an attention module. S3. Using the trained MFDS-DETR model, the preprocessed dataset is used as input in the backbone network. Features are extracted through the backbone network to obtain feature maps. First, 1*1 convolution is used to reduce the number of channels, then 3*3 convolution is used to reduce the size of the feature map, and finally 1*1 convolution is used to expand the number of channels, thereby outputting feature maps of different scales. S4. In the high-level filtering feature fusion pyramid, the feature selection module selects features from feature maps of different scales to obtain high-level features and low-level features. Then, the feature fusion module is used to fuse the high-level information and low-level information at multiple scales to obtain a multi-scale fused feature vector. S5. In the deformable self-attention module, offset output vectors of different scales are obtained through the offset module. Finally, the offset output vectors of different scales are weighted through the attention module to obtain the final output vector. The method for obtaining the offset output vectors at different scales in step S5 is as follows: Before the multi-scale fused feature vector is input into the offset module, it first needs to be converted into a feature map. Then, the input query vector is obtained based on the coordinates of the reference point. A linear transformation is applied to the query vector to obtain the offset Δp of the query vector. q Then, based on the offset of each reference point, the point of interest (i.e., the sampling point) is found, and the output offset of that point is obtained using bilinear interpolation. value Each query vector has H attention heads, and for each attention head there are K offset points; The method for obtaining the final output vector in step S5 is as follows: First, perform a linear transformation on the input query vector; then, use the Softmax function to obtain the weight vector for each offset; multiply each offset output by the corresponding weight vector and add them together to obtain the sample value Sample. value Then, each focus point in the reference point is concatenated to obtain the final sampled output vector Sample. output Finally, the sampled output vector is linearly transformed to obtain the final output, Output, as shown in the following expression: Weight = Softmax(WQ) Output=W*Sample output ; S6. Using the final output vector from step S5 as input, extract global features through the encoder; S7. The extracted global features are used as input and the final target detection result is obtained through the decoder.

2. The leukocyte detection method based on multi-scale fusion and deformable self-attention according to claim 1, characterized in that, The data preprocessing method in step S1 is as follows: The white blood cells in the image data of the white blood cell target detection dataset are labeled with LabelMe as (x1,y1) and (x2,y2), where x1,y1 represent the coordinates of the upper left corner and x2,y2 represent the coordinates of the lower right corner. The label format is then converted into (x1,y1) and the width and height of the label box. The label files of all images are then integrated into COCO format label data. After obtaining the training data in COCO format, the input images need to be augmented. This is done by augmenting the training dataset using random cropping, center cropping, and adding salt-and-pepper noise and Gaussian noise.

3. The leukocyte detection method based on multi-scale fusion and deformable self-attention according to claim 1, characterized in that, The feature selection module includes a channel attention module and a dimension matching module. The channel attention module includes global average pooling and global max pooling. The dimension matching module includes 1*1 convolution. The feature selection method using the feature selection module is as follows: First, global average pooling and global max pooling are performed. Then, the calculated features are summed. Subsequently, the weight value of each channel is calculated using the sigmoid activation function. Finally, the weight f of each channel is output. CA ∈R C×1×1 The obtained weight information is then multiplied with the feature map at its own scale to obtain the filtered feature map; in the dimension matching module, a 1*1 convolution is used to reduce the number of channels of the feature map at each scale to 256.

4. The leukocyte detection method based on multi-scale fusion and deformable self-attention according to claim 2, characterized in that, The feature fusion module employs a combination of transposed convolution, bilinear interpolation, and a CA module. The method for obtaining multi-scale fused feature vectors through the feature fusion module is as follows: Given an input high-level feature f high ∈R C×H×W Input low-level features First, the high-level features are expanded by using transposed convolutions with a stride of 2 and a kernel size of 3*3. Subsequently, to unify the sizes of high-level and low-level features, bilinear interpolation was used to downsample or upsample the high-level features, resulting in... Where C represents the number of channels in the feature map, H represents the feature map height, and W represents the feature map width; after obtaining features of consistent size, the CA module is used to convert high-level features into corresponding attention weights to filter low-level features. Finally, the filtered low-level features are fused with the high-level features to obtain a multi-scale fused feature vector, as shown in the following expression: f att =BL(T-Conv(f high )) f out =f low *CA(f att )+f att 。 5. The leukocyte detection method based on multi-scale fusion and deformable self-attention according to claim 4, characterized in that, The encoder has six layers, and each layer consists of a deformable self-attention and FFN network structure. The encoder extracts global features by adding spatial location encoding and scale encoding to the final output vector as input to the encoder; initializing reference points through variable self-attention to obtain the output vector; normalizing the output vector through a residual structure; and then learning white blood cell features through an FFN network structure.

6. The leukocyte detection method based on multi-scale fusion and deformable self-attention according to claim 5, characterized in that, The decoder includes a self-attention feature extraction module and a cross-attention feature extraction module; The self-attention feature extraction module includes a self-attention module and an FFN. The self-attention module learns by interacting with the input vectors and calculates the weights of different objects to find more relevant information in the region. The self-attention calculation formula is as follows: Attention(Q,K,V)=Cos(Q,K)×V The cross-attention feature extraction module has the same structure as the encoder, but the inputs to the encoder and the cross-attention mechanism come from the output position encoding and the global features extracted from the last layer of the encoder, respectively.

7. The leukocyte detection method based on multi-scale fusion and deformable self-attention according to claim 6, characterized in that, The loss function of the MFDS-DETR model is a joint loss function composed of classification loss, regression loss, and auxiliary loss, and its expression is as follows:

Images