Whole film bone marrow cell automatic counting method and system based on visual large model and self-supervised learning

Abstract

The application provides a whole bone marrow smear automatic counting method and system based on a visual large model and self-supervised learning. An ROI segmentation model is used to segment an interested region from the bone marrow smear. A TOI classification model is used to cut a slice from the bone marrow smear corresponding to the interested region, and perform binary classification to select an interested slice. A cell segmentation model based on the visual large model is used to segment all cells on the interested slice. A cell classification model based on the self-supervised learning is used to classify the cells, and finally counting is performed. The application has strong generalization and high accuracy, and can provide valuable reference for pathologists of various medical centers.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence learning technology, specifically to an automatic whole-slice bone marrow cell counting method and system based on a large visual model and self-supervised learning. Background Technology

[0002] Bone marrow aspiration smear cytology count (BMADCC) plays a crucial role in the diagnosis and treatment of hematologic malignancies. Specifically, it is the process by which pathologists determine the proportions of various cell types in the bone marrow.

[0003] Bone marrow aspiration smear cytological differential count is helpful in diagnosing acute lymphoblastic leukemia (ALL), acute myeloid leukemia (AML), angioimmunoblastic T-cell lymphoma (AITL), Burkitt lymphoma, chronic lymphocytic leukemia (CLL), chronic myeloid leukemia (CML), chronic myeloid monocytic leukemia (CMML), classical Hodgkin lymphoma (cHL), diffuse large B-cell lymphoma (DLBCL), essential thrombocythemia (ET), follicular lymphoma (FL), mantle cell lymphoma (MCL), mucosa-associated lymphoid tissue lymphoma (MALT), multiple myeloma (MM), NK / T-cell lymphoma (NKTL), prolymphocytic leukemia (PLL), immune thrombocytopenic purpura (ITP), and other hematologic malignancies.

[0004] Traditional bone marrow aspiration smear cell classification and counting is performed manually by pathologists using a microscope. It is considered the gold standard, widely used for diagnosing and monitoring various blood and bone marrow diseases. However, it has two main drawbacks: 1) Labor-intensive: Manual counting is a time-consuming and labor-intensive task, requiring pathologists to dedicate significant time. Prolonged continuous work can affect the accuracy of cell classification and counting. 2) Subjectivity: Inconsistencies in cell classification and counting experience levels among different pathologists lead to subjective bias.

[0005] With the rapid development of deep learning, deep learning-based visual models have demonstrated outstanding performance in image processing tasks such as object detection, semantic segmentation, and image classification. These models enable quantitative routine tasks in computer-aided diagnosis, thereby accelerating the process, reducing bias, and improving the consistency of results. To address the problems associated with manual counting, many researchers have applied deep learning algorithms to automate cell classification and counting in bone marrow aspiration smears. This significantly improves the accuracy and efficiency of cell classification and counting, possessing significant theoretical and practical value. These studies can be broadly categorized into two main approaches based on their processes: slice-based counting methods and whole-smear-based counting methods.

[0006] Slice-based counting methods require manually slicing the entire image into small pieces (i.e., square bone marrow images), selecting appropriate slices as input for cell classification and counting, and then performing cell detection and classification. Wang et al. used the Faster-RCNN object detection algorithm and feature pyramid network to detect six cell types in the slices. However, the classification accuracy of the cell detection model still needs improvement. In contrast, Chandradevan achieved higher overall accuracy by using a separate cell classification model. They developed a two-stage system for standard clinical cell classification, manually selecting BMA slices containing a large number of cells, using Faster-RCNN to detect all cells as single-class objects, and then using VGG for cell classification. Yu et al. applied deep convolutional neural networks to automatically detect and classify nucleated cells in bone marrow slices. These methods require manual selection of the optimal slice, making the process slow and unsuitable for full-process cell classification and counting, limiting their scalability in clinical diagnostic work.

[0007] Compared to slice-based counting methods, whole-slice-based counting methods do not require manual slice selection. Instead, they automatically identify slices of interest (TOIs) from the whole slice, detect cells in these slices, and finally classify and count these cells, achieving intelligent end-to-end cell classification and counting. These methods can be further classified based on the slice extraction method: grid-based counting methods and ROI-based counting methods.

[0008] Grid-based counting methods directly slice the entire slide into uniformly sized slices and then select TOIs from these slices. Tayebi et al. developed an end-to-end automated bone marrow cytology system that slices the entire slide into uniform slices, uses DenseNet121 for binary classification to distinguish between TOIs and non-TOIs, and then uses YOLOv4 for cell detection and classification within the TOIs. This method achieved high accuracy and showed a strong correlation with manual counting results. Lewis et al. proposed a more precise automated workflow that uses EfficientNetV2S to classify uniformly sized slices into four classes (optimal, granular, hemodiluted, and external), uses Faster-RCNN to detect cells of a single class in the optimal slice (i.e., TOI), and uses EfficientNetV2L for cell classification. Multiple experiments demonstrated the feasibility of automatically generating cell classification counts from WSIs. Both methods slice the entire slide into slices, but these slices may include cell aggregation, overstained, or blank areas, resulting in many invalid slices that need to be filtered, making the process particularly time-consuming. In practice, the suitable region for cell classification counting is the body-tail junction, known as the region of interest (ROI).

[0009] Region of Interest (ROI)-based counting methods identify ROIs from the entire slice and then extract slices from these regions for subsequent cell detection and classification. Wang et al. proposed a hierarchical framework that utilizes a multi-resolution pyramid and Cascade R-CNN to identify suitable bounding boxes as ROIs from the entire slice. Then, another Cascade R-CNN is used to perform BMA cell detection and classification within these regions, achieving efficient nucleated cell classification and counting analysis from WSI. Su et al. further explored methods for automatically extracting high-quality slice images and accurately locating and identifying nucleated cells, proposing the ROI-BMC-DNNet analysis framework. This framework uses a pyramid network and encoder-decoder to segment ROIs and then extracts high-quality patches from these regions. Subsequently, a slice quality assessment network and a cell detection network are used to automatically identify and count bone marrow nucleated cells. Compared to grid-based counting methods that slice the entire slice into uniformly sized slices, ROI-based counting methods produce higher-quality slices with less data, thus reducing processing time during slice classification.

[0010] However, both slice-based and whole-slice-based counting methods have significant drawbacks. They are only applicable to bone marrow aspiration smears from specific diseases and medical centers, require substantial labeled data, time, and computational resources, and exhibit poor generalization. Specifically: 1) Supervised learning issues: 1. Object detection models can only detect known cell types in the dataset, making them suitable only for specific detection tasks. When cell types change, additional cell bounding boxes need to be labeled; 2. Image classification models can only learn limited representations in the current dataset. When applied to different medical centers, a large amount of labeled data and retraining are required to maintain high accuracy. 2) Efficiency and staining issues: 1. Grid-based counting methods generate a large amount of irrelevant data when slicing the whole slice into slices, increasing the time and computational resources required for slice classification models; 2. BMA staining differences may exist between different medical centers, resulting in poor performance for ROI segmentation and TOI classification based on traditional deep learning models, making them unsuitable for other centers.

[0011] In recent years, artificial intelligence, especially in the field of computer vision, has made rapid progress, with significant advancements in large-scale models and self-supervised learning. These models can effectively solve problems.1) Regarding large-scale models, Segment AnythingModel (SAM), trained on the SA-1B dataset containing 11 million images and over 1 billion masks, demonstrates strong zero-shot generalization capabilities. Even when domain-specific images differ significantly from SAM's training data, high-precision segmentation results can be achieved through fine-tuning with a small amount of labeled data using cues. SAM has recently attracted widespread attention in the field of medical imaging. For example, methods such as MedSAM and Med-SA have been optimized for general medical images. In the field of cell segmentation, methods such as CellSAM, Guided Prompting SAM, and UNSAM have also shown significant results. Furthermore, self-supervised learning initially constructed supervisory signals from within images, such as the jigsaw puzzle prediction task proposed by Doersch et al. and the image rotation prediction task proposed by Noroozi et al. Subsequently, contrastive learning has become one of the mainstream methods of self-supervised learning, learning useful feature representations by comparing the similarities and differences between samples. Notable examples include MoCo proposed by He et al. and SimCLR introduced by Chen et al. Building on these advancements, the MAE model proposed by He et al. uses image masking and reconstruction methods to train a feature extractor with strong generalization capabilities from a large amount of unlabeled data, achieving excellent classification performance through fine-tuning with a small amount of labeled data. For problem 2), Su et al. addressed the time and computational efficiency issues caused by a large number of invalid slices by segmenting the Region of Interest (ROI), which constitutes only a small portion of the WSI, and then extracting high-quality slices from these regions. However, previous studies have not addressed the color differences in BMA smears from different medical centers, nor have they addressed the significant texture differences between ROI and non-ROI image pairs, or between TOI and non-TOI image pairs. Summary of the Invention

[0012] In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide an automatic counting method and system for whole-slice bone marrow cells based on visual large model and self-supervised learning.

[0013] To achieve the above-mentioned objectives of this invention, this invention provides an automated whole-slice bone marrow cell counting method based on a large visual model and self-supervised learning, comprising the following steps:

[0014] Obtain the full bone marrow smear dataset;

[0015] A deep learning framework for intelligent cell classification and counting is constructed, including an ROI segmentation model, a TOI classification model, a cell segmentation model based on a large visual model, and a cell classification model based on self-supervised learning.

[0016] The ROI segmentation model segments the region of interest from the bone marrow smear; the TOI classification model cuts out slices from the bone marrow smear corresponding to the region of interest and performs binary classification to select slices of interest; the cell segmentation model segments all cells on the slices of interest; the cell classification model classifies the cells; and finally, the cells are counted.

[0017] Optionally, the ROI segmentation model employs an encoder to extract features at different levels from the input original image; a texture extraction module extracts texture features at different levels from the encoder by calculating the covariance of the features; a decoder connects the texture features with the features at different levels extracted by the encoder and inputs them into the decoder for decoding; and a first texture depth supervision module supervises the texture features at different levels of the decoder using the texture image extracted from the original image.

[0018] Optionally, the texture extraction module performs convolution calculation on the input original image and calculates the covariance matrix between the channels of each pixel in the convolutional feature map;

[0019] Construct an upper triangular matrix of size (C1×(C1+1)) / 2 containing the same information as the covariance matrix, and reshape the upper triangular matrix into a vector;

[0020] Perform the same operation as described above on each pixel of the input original image to obtain a feature map;

[0021] The feature map is concatenated with the original input image to obtain the original feature information and the texture information extracted from the original feature map.

[0022] Optionally, the first texture depth supervision module includes:

[0023] The first texture prediction block receives the output of the decoder and performs convolution to obtain the texture prediction. Figure 1 Texture prediction Figure 1 Upsampling to the same resolution as the original input image yields texture prediction. Figure 2 ;

[0024] The first texture-aided loss calculation block calculates texture prediction. Figure 2 Texture-assisted loss between the texture image and the texture image extracted from the original image.

[0025] Optionally, the TOI classification model uses a CNN backbone network to extract multi-scale features from the input bone marrow smear slices;

[0026] During training, texture pseudo-labels and contour pseudo-labels are generated for depth supervision. A second texture depth supervision module processes each scale feature of the bone marrow smear slices extracted by the CNN backbone network, predicts the texture, and enlarges the texture to match the size of the original image. An auxiliary loss is calculated using the predicted texture results and texture pseudo-labels, followed by gradient backpropagation. Similarly, a contour depth supervision module processes each scale feature of the bone marrow smear slices extracted by the CNN backbone network, predicts the contour, and enlarges the contour to the same size as the original image. An auxiliary loss is calculated using the predicted contour and contour pseudo-labels, followed by gradient backpropagation.

[0027] Optional,

[0028] The second texture depth supervision module includes:

[0029] The second texture prediction block extracts texture and performs texture prediction by calculating feature covariance.

[0030] The second texture auxiliary loss calculation block samples the texture features predicted by the second texture prediction block at each level to the original image size, calculates the auxiliary loss using texture pseudo-labels, and performs gradient backpropagation to achieve optimized texture extraction.

[0031] Optionally, the contour depth supervision module uses contour prediction blocks to perform average pooling and max pooling on the input features, and then concatenates and performs sigmoid operations on the resulting features to obtain attention weights; each element of the attention weights is subtracted from 1 and multiplied element-wise with the input features to enhance the contour features, and convolution is used for contour prediction.

[0032] The contour-assisted loss calculation block samples the contour map predicted by the contour prediction block to the original image size, calculates the auxiliary loss using contour pseudo-labels, and performs gradient backpropagation to achieve contour extraction.

[0033] Optionally, the cell segmentation model includes an image encoder, a cue encoder, and a mask decoder.

[0034] The image encoder acquires the image output by the TOI classification model, rescales the image and fills the shorter side to obtain the target input resolution, and uses convolution operations to obtain n channels;

[0035] The cue encoder generates point, box, text, and mask cues. Sparse cues are mapped to n-dimensional vector embeddings. Specifically, point and box are represented using positional encoding. A point is represented as the sum of the positional encoding and one of two learned embeddings, which indicate whether the point is in the foreground or background. A box is represented by an embedding pair. The learned embeddings are summed for each cue type and the free-form text using the text encoder. Mask cues use convolutional embeddings and are summed element-wise in the image embeddings.

[0036] The mask decoder maps image embedding, cue embedding, and output token to a mask;

[0037] The cell segmentation model freezes the official weights of the image encoder and cue encoder, and only updates the mask decoder weights;

[0038] The SegAny method is used to predict cell masks based on points, and the geometric center point of the cell mask is used as the foreground point to fine-tune the mask decoder weights; during cell counting inference, SegEvery is used for segmentation.

[0039] In the cell segmentation fine-tuning stage, each TOI image is labeled with a single-cell mask; in the cell segmentation verification stage, each TOI image is labeled with all single-cell masks, all of which are used for verification.

[0040] Optionally, the cell classification model includes:

[0041] The masking module divides the image output by the cell segmentation model into regular, non-overlapping small blocks, randomly samples the non-overlapping small blocks and makes them visible, and masks the remaining small blocks.

[0042] The encoder embeds the location of the visible small blocks;

[0043] The decoder takes input consisting of encoded visible patches and masks, with position embeddings added to all encoded visible patches and masks. The decoder includes Transformer blocks, and the decoder predicts the pixel values ​​of each masked patch to reconstruct the input. Each element in the decoder output represents a pixel value vector of a patch. The last layer of the decoder is a linear projection, and the number of its output channels is equal to the number of pixel values ​​in a patch.

[0044] The reconstruction module reconstructs the decoder's output to form a reconstructed image.

[0045] This invention also proposes an automated whole-smear bone marrow cell counting system, including a data acquisition module, a processing module, and a storage module for acquiring bone marrow smear datasets;

[0046] The data acquisition module is communicatively connected to the processing module, sending the bone marrow smear dataset to it; the processing module is communicatively connected to the storage module, which stores at least one executable instruction. The executable instruction causes the processing module to perform the operations corresponding to the above-described automatic whole-smear bone marrow cell counting method based on the bone marrow smear dataset, automatically classifying and counting the whole-smear bone marrow cells.

[0047] The beneficial effects of this invention are:

[0048] In this invention, the ROI segmentation model demonstrates significant effectiveness in texture information extraction, mitigating the impact of staining differences (bone marrow aspiration smears typically exhibit staining variations, particularly noticeable texture differences in the body, tail, and body-tail junction of thumbnails); the TOI classification model possesses powerful capabilities in texture and contour feature extraction, achieving high accuracy even on relatively small datasets, resolving significant texture differences between image pairs of slices of interest and those of non-interest; the cell segmentation model is more accurate than traditional object detection and image segmentation methods, improving segmentation performance, specifically exhibiting strong generalization ability and achieving accurate segmentation without additional fine-tuning; the cell classification model demonstrates even stronger feature extraction and powerful classification performance, requiring minimal additional annotation and fine-tuning to achieve effective classification based on specific cell types. Therefore, this invention possesses strong generalization and high accuracy, providing valuable reference for pathologists in various medical centers.

[0049] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0050] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0051] Figure 1 This is a schematic diagram of the principle of the present invention;

[0052] Figure 2 This is the network architecture diagram of the ROI segmentation model;

[0053] Figure 3 This is a schematic diagram of the network structure of the texture extraction module;

[0054] Figure 4 This is a schematic diagram of the network structure of the first texture prediction block;

[0055] Figure 5 This is a schematic diagram of the network structure of the TOI classification model;

[0056] Figure 6This is a schematic diagram of the network structure of the second texture prediction block;

[0057] Figure 7 This is a schematic diagram of the network structure of the contour prediction block;

[0058] Figure 8 This is a schematic diagram of the network structure of the cell segmentation model;

[0059] Figure 9 This is a schematic diagram of the network structure of a mask decoder;

[0060] Figure 10 This is a block diagram illustrating the principle of a cell classification model. Detailed Implementation

[0061] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0062] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0063] like Figure 1 As shown, this invention provides an automatic whole-slice bone marrow cell counting method based on a large visual model (SAM) and self-supervised learning, comprising the following steps:

[0064] Obtain the bone marrow smear dataset. In this embodiment, bone marrow aspiration smears are scanned using a 100x oil immersion objective, and corresponding thumbnails (reduced by a factor of 100) are generated using Python to obtain the bone marrow smear dataset. A deep learning framework for cell classification and counting is constructed, including the ROI segmentation model TextureUnet, the TOI classification model TCNet, a cell segmentation model, and a cell classification model. The ROI segmentation model is used to segment the region of interest from the bone marrow smear; the TOI classification model is used to cut slices from the bone marrow smear corresponding to the region of interest, and binary classification is performed to select the slice of interest; the cell segmentation model is used to segment all cells on the slice of interest; and the cell classification model is used to classify and count the cells. A 16-category cell classification and counting histogram is generated by calculating the cell proportion by dividing the number of each cell type by the total number of cells.

[0065] Among them, such as Figure 2As shown, the ROI segmentation model TextureUnet utilizes an enhanced encoder-decoder structure developed based on the Unet framework, employing a ResNet-based encoder for feature extraction. For a single input image, it is first fed into the encoder to extract its multi-level feature maps. Then, the feature maps at different levels are input into the texture extraction module TEM to extract texture feature maps. Next, the texture feature maps are concatenated with the feature maps from the encoder and input into the decoder for decoding. The decoded result is the intermediate features of each layer and the final predicted feature ROI mask. This embodiment designs a first texture depth supervision module TDSM1, which uses the texture feature maps extracted from the original input image to supervise the feature maps at different levels of the decoder (each layer uses MSE loss summation). Specifically,

[0066] The texture extraction module (TEM) extracts texture features from different levels of features extracted by the encoder by calculating the covariance of the features. For example... Figure 3 As shown, the texture extraction module TEM takes a feature map f with a resolution of C×H×W as the original image input, where C is the number of channels, and H and W are the height and width of the feature map f, respectively. First, a 1×1 convolution is performed on the feature map f to obtain a feature map fk with a resolution of C1×H×W. Then, the covariance matrix between the channels of each pixel in the feature map fk is calculated, resulting in a covariance matrix Ck with a resolution of C1×C1. The covariance matrix between features measures the co-occurrence of features and can be used to extract texture information. Since the covariance matrix Ck is a symmetric matrix, an upper triangular matrix can be used, which contains the same information as the covariance matrix Ck but has a size of (C1×(C1+1)) / 2. Then, the upper triangular matrix is ​​reshaped into a vector, resulting in a vector vk of size (C1×(C1+1)) / 2. The same operation is performed on each pixel of the input feature map f to obtain a feature map fv with a resolution of (C1×(C1+1)) / 2×H×W. Finally, the feature map fv is concatenated with the input feature map f to obtain a feature map ft with a resolution of (C_1+(C_1×(C_1+1)) / 2)×H×W. The feature map ft contains the original feature information and the texture information extracted from the original feature map f, and then it is sent to the decoder.

[0067] During the training of the ROI segmentation model, the first texture depth supervision module (TDSM1) uses texture images extracted from the original image to supervise the texture features at different levels of the decoder. Specifically, the first texture depth supervision module (TDSM1) includes a first texture prediction block (TPB1) and a first texture auxiliary loss calculation block (TALCB1). The first texture depth supervision module (TDSM1) receives the output from the decoder and uses 1×1 convolution processing to obtain texture predictions with a resolution of 1×H×W. Figure 1 t. Then, as Figure 4 As shown, texture prediction Figure 1 The image is upsampled to the same resolution as the original input image to obtain a texture prediction with a resolution of 1×H1×W1. Figure 2 tu. The first texture-aided loss calculation block calculates texture prediction. Figure 2 The texture-aided loss Lt is calculated between tu and the texture image extracted from the original input image. Through the implementation of the first texture depth supervision module TDSM1, the network can be trained to extract texture information from the input image and use it to assist in the segmentation of regions of interest.

[0068] To fully consider the characteristics of bone marrow smears and enhance texture and contour feature extraction, the TOI classification model TCNet is used for texture and contour perception. The input to TCNet is an RGB color image of a bone marrow smear, and the output is the image classification result, i.e., TOI or non-TOI. During training, this embodiment uses the OpenCV LBP (Local Binary Pattern) algorithm and the Canny edge detection algorithm to generate texture pseudo-labels and contour pseudo-labels for depth supervision. Specifically, the TOI classification model TCNet consists of three basic components: a CNN backbone network (ResNet-50) for extracting image feature representations, a second texture depth supervision module TDSM2, and a contour depth supervision module CDSM.

[0069] like Figure 5 As shown, given input I∈R (3×H×W) A CNN backbone network (ResNet-50) is used to extract multi-scale features. This refers to the features at each scale of the bone marrow smear slice. During the training phase, the extracted features (F2, F3, F4, F5) serve as input to the second texture depth supervision module TDSM2 and the contour depth supervision module CDSM. For the features... The input features f2 are simultaneously sent to the second texture depth supervision module TDSM2 and the contour depth supervision module CDSM. The second texture depth supervision module TDSM2 processes the input features f2 to predict the texture. The predicted texture T2 is then enlarged to match the size of the original image (H×W). An auxiliary loss is calculated using the prediction results and texture pseudo-labels, followed by gradient backpropagation. Similarly, the contour depth supervision module CDSM processes the input features to predict contours. The image is then upscaled to the same size as the original image (H×W). The auxiliary loss is calculated using the predicted results and contour pseudo-labels, followed by gradient backpropagation. The same operation is performed on the remaining features f3, f4, and f5 to achieve depth supervision of each texture and contour layer. The CNN backbone network of the TOI classification model ultimately outputs the classification result.

[0070] The second texture depth supervision module TDSM 2 includes: a second texture prediction block TPB2 and a second texture auxiliary loss calculation block TALCB2. For example... Figure 6 As shown, the second texture prediction block, TPB2, extracts texture and performs texture prediction by calculating feature covariance. It takes a feature map with resolution C×H×W as input and performs multiple calculations using a 1×1 convolution kernel to obtain a feature map with resolution C1×H×W. To improve computational efficiency, C1 is set to a small value (24 in this paper), and the obtained feature map is used to learn multiple aspects of the texture in subsequent operations. Next, the feature map is input into a texture extraction module to calculate the covariance matrix between feature channels at each location to capture the correlation between different responses in the convolutional features. This texture extraction module can use the same network structure as the texture extraction module TEM mentioned above. The covariance matrix between features measures the co-occurrence of features, describes the combination of features, and is used to represent texture information. For feature map f... k Each pixel in C1×H×W Calculate its covariance matrix as and The inner product between them. Since the covariance matrix (C1×C1) is diagonally symmetric, the upper triangular part of the matrix is ​​used to represent texture features, and the result is reshaped into a feature vector. The same operation is performed on each pixel in the feature map, and the results are concatenated to obtain a feature map containing texture information. The obtained feature map is subjected to a 1×1 convolution operation to obtain the predicted texture feature map. The second texture auxiliary loss calculation block TALCB2 samples the predicted texture features of the second texture prediction block TPB2 to the original image size, calculates the auxiliary loss using texture pseudo-labels, and performs gradient backpropagation to achieve optimized texture extraction.

[0071] The Contour Depth Supervision Module (CDSM) comprises a contour prediction block (CPB) and a contour-aided loss calculation block (CALCB). In the contour prediction block (CPB), a reverse attention mechanism is introduced to enhance sensitivity to contour features. For example... Figure 7 As shown, the input features are first subjected to average pooling and max pooling, and then concatenated. Next, a 7×7 convolution and a sigmoid operation are performed on the resulting features to obtain attention weights. Then, each element of the attention weights is subtracted from the first convolution, and the result is multiplied element-wise with the input features to enhance the features. Through these processes, the contour features are enhanced. Finally, a 3×3 convolution is used for contour prediction. The contour-aided loss calculation block (CALCB) samples the contour prediction block (CPB) at each layer to the original image size, calculates the auxiliary loss using contour pseudo-labels, and performs gradient backpropagation to achieve contour extraction.

[0072] The cell segmentation model is built on the Transformer vision model and has three components, such as... Figure 8 As shown, a powerful image encoder, a flexible cue encoder, and an efficient mask decoder are presented. Supporting point, box, and text cues, and trained on the SA-1B dataset containing 11 million images and over 1 billion masks, it demonstrates strong segmentation and generalization capabilities. It supports two segmentation methods: SegAny, which predicts the mask of a single object based on points or boxes, and SegEvery, which predicts the mask of all objects in an image. To segment all cells, this embodiment chooses ViT-H (632M parameters) as the backbone network. When the cell segmentation model SAM performs poorly in a specific domain, accuracy can be improved through fine-tuning. Since the training images of the original large visual model SAM are natural images, which differ from bone marrow cell images, direct cell segmentation requires improvement; therefore, this paper uses a cell mask dataset for fine-tuning. By freezing the official weights of the image encoder and cue encoder, SAM is directly fine-tuned; these encoders effectively handle feature extraction and point cues, and only the decoder weights are updated. The SegAny method is used to predict cell masks based on points, and the weights of the mask decoder are fine-tuned using the geometric center point of the cell mask as the foreground point. The fine-tuning method involves calculating the DICE loss and BCE loss of the cell mask and the model's predicted mask, followed by backpropagation. During the cell segmentation fine-tuning phase, a single-cell mask is labeled for each TOI image. During validation and testing, i.e., during cell counting inference, SegEvery is used for segmentation. No bounding boxes or text hints are used here because SegEvery, being point-based, improves segmentation performance. During the cell segmentation validation phase, all single-cell masks are labeled for each TOI image and used entirely for validation. When applied to other medical centers, due to the relatively fixed cell morphology, the fine-tuned SAM exhibits strong generalization ability, achieving accurate segmentation without further fine-tuning.

[0073] Specifically, the image encoder acquires the image output by the TOI classification model, rescales the image and pads the shorter side to obtain the target input resolution, and uses convolution operations to obtain n channels. In this embodiment, the input resolution is 1024*1024, and the output channels n are always 256. The image encoder can be any network with an image embedding of output size C×H×W. Motivated by scalability and strong pre-training, ViT pre-trained with MAE is used with minimal adjustments to handle high-resolution inputs, specifically ViT-H / 16 with 14×14 window attention and four equally spaced global attention blocks. The output of the image encoder is a 16-fold downsized embedding of the input image. Since the runtime goal is to process each cue in real time, a large number of image encoder flops can be afforded because they are computed only once per image, not once per cue. As is standard practice, the input resolution of 1024×1024 is obtained by rescaling the image and padding the shorter side. Therefore, the image embedding is 64×64. To reduce the channel dimension, 1×1 convolutions are used to obtain 256 channels, followed by 3×3 convolutions to obtain 256 channels. Each convolution is followed by layer normalization.

[0074] The cue encoder considers two sets of cues: sparse (points, boxes, text) and dense (masks). Sparse cues are represented by positional encodings for points and boxes, with learned embeddings summed for each cue type and the free-form text using an off-the-shelf text encoder employing a CLIP multimodal neural network model. Dense cues use convolutional embeddings, summed element-wise in the image embeddings. Specifically, sparse cues are mapped to 256-dimensional vector embeddings. A point is represented as the sum of a positional encoding and one of two learned embeddings indicating whether the point is in the foreground or background. A box is represented using a pair of embeddings: (1) the positional encoding of its top-left corner is summed with the learned embedding representing the "top-left" corner; (2) the same structure, but using the learned embedding representing the "bottom-right" corner. Finally, for the free-form text, a text encoder from CLIP (generally any text encoder is possible) is used. Dense cues have a spatial correspondence with the image. The input mask is set to a resolution 4 times lower than the input image, and then scaled down by a factor of 4 using two 2×2 convolutions with a stride of 2, one for each of the output channels 4 and 16. A final 1×1 convolution maps the channel dimensions to 256. Each layer is separated by GELU activation and layer normalization. Masking and image embedding elements are then added. If no masking cue is available, a learned embedding representing "no mask" is added at each image embedding location.

[0075] The mask decoder efficiently maps image embeddings, cue embeddings, and output tokens to a mask. The design employs a modification of the Transformer decoder block, followed by a dynamic mask prediction head. The improved decoder block updates all embeddings in both directions using cue self-attention and cross-attention (cues to image embeddings and vice versa). After running both blocks, the image embeddings are upsampled, and a multilayer perceptron (MLP) maps the output tokens to a dynamic linear classifier, which then computes the mask foreground probability for each image location.

[0076] Specifically, the mask decoder effectively maps image embeddings and a set of cue embeddings to an output mask. To combine these inputs, the standard Transformer decoder is modified, drawing inspiration from the Transformer segmentation model. Before applying this mask decoder, a learned output token embedding is first inserted into the set of cue embeddings; this token will be used in the decoder's output. For simplicity, these embeddings (excluding image embeddings) are collectively referred to as "tokens".

[0077] Mask decoder design such as Figure 9 As shown, each decoder layer performs four steps: (1) self-attention to tokens, (2) cross-attention from tokens (as queries) to the image embedding, (3) pointwise MLP update for each token, and (4) cross-attention from the image embedding (as queries) to tokens. The final step updates the image embedding with cue information. During the cross-attention process, the image embedding is treated as a set of 64×64 256-dimensional vectors. Each self / cross-attention and MLP has residual connections, layer normalization, and a dropout of 0.1 during training. The next decoder layer obtains the updated tokens and the updated image embedding from the previous layer. A two-layer decoder is used in this embodiment.

[0078] To ensure the decoder has access to critical geometric information, positional encodings are added to the image embedding whenever they participate in the attention layer. Furthermore, whenever they participate in the attention layer, the entire original cue token set (including their positional encodings) is re-added to the updated token set. This allows for strong dependencies on the geometric position and type of the cue tokens.

[0079] After running the decoder, the updated image embedding is upsampled by a factor of 4 using two transposed convolutional layers (now it's 4 times smaller relative to the input image). Then, the tokens are again used for image embedding, with the updated output tokens embedding passed to a small 3-layer MLP that outputs a vector matching the channel dimensions of the upsampled image embedding. Finally, the mask is predicted using the spatial dot product between the upsampled image embedding and the MLP output.

[0080] The Transformer uses an embedding dimension of 256. The MLP block in the Transformer has a larger internal dimension of 2048, but the MLP is only applied to a relatively small number of cue tokens (rarely greater than 20). However, there is a 64×64 image embedding in the cross-attention layer, and to improve computational efficiency, the channel dimensions of the query, key, and value are reduced by a factor of 2 to 128. All attention layers use 8 heads.

[0081] The transposed convolutions used for embedding the upsampled output image are 2×2 with a stride of 2, output channel sizes of 64 and 32, and GELU activation. They are separated by layer normalization.

[0082] The cell classification model employs a self-supervised learning model (MAE) with the ViT series as the backbone network, such as ViT-B. It accurately classifies all cells segmented by the TOI cell segmentation model and trains in two phases: a cell image reconstruction phase and a cell classification fine-tuning phase. In the cell image reconstruction phase, the self-supervised learning model (MAE) learns general feature representations through image reconstruction from a large amount of unlabeled cell data. In the cell classification fine-tuning phase, the MAE encoder acts as a general feature extractor, fine-tuning with a small amount of labeled cell data to obtain a high-accuracy cell classification model. When applied to other medical centers, the powerful feature extraction capabilities of the self-supervised learning model (MAE) demonstrate strong classification performance, achieving effective classification of specific cell types with minimal additional labeling and fine-tuning.

[0083] Self-supervised learning models (MAEs) are a simple autoencoder method that can reconstruct the original signal given partial observations. For example... Figure 10 As shown, like all autoencoders, this embodiment has an encoder that maps the observed signal to a latent representation, and a decoder that reconstructs the original signal from the latent representation. Unlike classic autoencoders, this embodiment employs an asymmetric design, allowing the encoder to operate only on a portion of the observed signal (without mask tokens), and a lightweight decoder to reconstruct the complete signal from the latent representation and mask tokens.

[0084] Masking. According to ViT, the image is divided into regular, non-overlapping patches. The input image size is typically 224*224, and the patch size is typically 16*16. A subset of these patches is then sampled, and the remaining patches are masked (i.e., removed). Sampling strategy: Random patches are sampled without replacement, following a uniform distribution, known as "random sampling." Random sampling with a high masking ratio (i.e., the proportion of patches removed) largely eliminates redundancy, thus creating a task that cannot be easily solved by extrapolating from visible neighboring patches. Uniform distribution prevents potential center bias (i.e., more masked patches near the image center). Finally, the highly sparse input creates opportunities to design efficient encoders.

[0085] The MAE encoder is a ViT, but it only applies to visible, unoccluded patches. Like in a standard ViT, the encoder embeds patches by adding linear projections of positional embeddings, then processes the results through a series of Transformer blocks. However, the encoder here in this paper only runs on a small fraction (e.g., 25%) of the complete set. Masked patches are removed; masked tokens are not used. This allows a very large encoder to be trained with only a small fraction of the computation and memory. The complete set is processed by a lightweight decoder.

[0086] The input to the MAE decoder is a complete set of tokens, consisting of encoded visible patches and mask tokens. See also Figure 10 Each mask token is a shared learned vector representing a missing patch to be predicted. Positional embeddings are added to all tokens in this complete set; without this, the mask tokens would have no information about their location in the image. The decoder has another set of Transformer blocks. The MAE decoder is only used during pre-training to perform the image reconstruction task (only the encoder is used to generate image representations for recognition). Therefore, the decoder architecture can be flexibly designed in a way that is independent of the encoder design. Experiments are conducted with a very small decoder, narrower and shallower than the encoder. For example, the default decoder has <10% more computation per token compared to the encoder. Through this asymmetric design, the entire token set is processed only by a lightweight decoder, which significantly reduces pre-training time.

[0087] Reconstruction Objective: In this embodiment, MAE reconstructs the input by predicting the pixel values ​​of each masked patch. Each element in the decoder output is a vector of pixel values ​​representing a patch. The final layer of the decoder is a linear projection whose output channels are equal to the number of pixel values ​​in a patch. The decoder output is reconstructed to form the reconstructed image. The loss function calculates the mean squared error (MSE) between the reconstructed image and the original image in pixel space, only calculating the loss on the masked patch, similar to BERT.

[0088] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0089] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A whole film bone marrow cell automatic counting method based on a visual large model and self-supervised learning, characterized in that, The method comprises the following steps: Obtaining a bone marrow smear data set; Building a cell classification and counting deep learning framework, including an ROI segmentation model, a TOI classification model, a cell segmentation model based on a visual large model, and a cell classification model based on self-supervised learning; The ROI segmentation model segments the region of interest from the bone marrow smear; the TOI classification model cuts out a slice from the bone marrow smear corresponding to the region of interest and performs binary classification to select the slice of interest; the cell segmentation model segments all cells on the slice of interest; the cell segmentation model comprises an image encoder, a prompt encoder, and a mask decoder; The image encoder obtains the image output by the TOI classification model, rescales the image and fills in the shorter side to obtain a target input resolution, and obtains n channels using a convolution operation; The prompt encoder generates point, box, text, and mask prompts, and sparse prompts are mapped to n-dimensional vector embeddings; specifically, a point and a box are represented by position encoding, and a point is represented as the sum of position encoding and one of two learned embeddings, which indicate whether the point is in the foreground or the background; a box is represented by an embedding pair, and the sum of learned embeddings is summed for each prompt type and free-form text using a text encoder; the mask prompt uses a convolution embedding and is summed element-wise in the image embedding; The mask decoder maps the image embedding, the prompt embedding, and the output token to the mask; The official weights of the image encoder and the prompt encoder are frozen, and only the mask decoder weights are updated; The cell mask is predicted based on the point using the SegAny method, and the geometric center point of the cell mask is used as the foreground point to fine-tune the mask decoder weights; During cell counting inference, segmentation is performed using SegEvery; During the cell segmentation fine-tuning stage, each TOI image is labeled with a single cell mask; during the cell segmentation verification stage, each TOI image is labeled with all single cell masks, all of which are used for verification; The cell classification model classifies the cells; finally, the cells are counted; the cell classification model comprises: A masking module divides the image output by the cell segmentation model into regular non-overlapping small blocks, randomly samples the non-overlapping small blocks and makes them visible, and masks the remaining small blocks; An encoder embeds the positions of the visible small blocks; A decoder inputs the encoded visible small blocks and the mask, all of which are added with position embeddings; the decoder comprises a Transformer block; the decoder predicts the pixel values of each masked small block to reconstruct the input; each element in the decoder output represents a pixel value vector of a small block; the last layer of the decoder is a linear projection, and the number of output channels is equal to the number of pixel values in a small block; A reconstruction module reconstructs the output of the decoder to form a reconstructed image.

2. The method of claim 1, wherein the method is based on a visual large model and self-supervised learning. The ROI segmentation model adopts an encoder to extract different levels of features of an input original image; adopts a texture extraction module to extract texture features of the different levels of features extracted by the encoder by calculating the covariance of the features; adopts a decoder to connect and input the texture features and the different levels of features extracted by the encoder into the decoder for decoding; and adopts a first texture depth supervision module to supervise the texture features at different levels of the decoder by using a texture image extracted from the original image.

3. The method of claim 2, wherein the method is based on a visual large model and self-supervised learning. The texture extraction module performs convolution calculation on the input original image, and calculates a covariance matrix between channels of each pixel in a feature map after convolution; An upper triangular matrix with the same information as the covariance matrix and a size of (C1x(C1+1)) / 2 is constructed, and the upper triangular matrix is reshaped into a vector; The same operation is performed on each pixel of the input original image to obtain a feature map; The feature map and the input original image are connected to obtain original feature information and texture information extracted from the original feature map.

4. The method of claim 2, wherein the method is based on a visual large model and self-supervised learning. The first texture depth supervision module comprises: A first texture prediction block receives an output of the decoder and performs convolution to obtain a texture prediction map one, upsamples the texture prediction map one to the same resolution as the input original image to obtain a texture prediction map two, and a first texture auxiliary loss calculation block calculates a texture auxiliary loss between the texture prediction map two and a texture image extracted from the original image. The TOI classification model adopts a CNN backbone network to extract multi-scale features of an input bone marrow smear slice; 5. The method of claim 1, wherein the method is based on a visual large model and self-supervised learning. In the training process, texture pseudo-labels and contour pseudo-labels for depth supervision are generated; A second texture depth supervision module is used to process each scale feature of the bone marrow smear slice extracted by the CNN backbone network, predict a texture, enlarge the texture to match the size of the original image, calculate an auxiliary loss by using the predicted texture and the texture pseudo-labels, and then perform gradient back propagation; and a contour depth supervision module is used to process each scale feature of the bone marrow smear slice extracted by the CNN backbone network, predict a contour, and enhance the contour to the same size as the original image; an auxiliary loss is calculated by using the predicted contour and the contour pseudo-labels, and then gradient back propagation is performed. The second texture depth supervision module comprises:

6. The method of claim 5, wherein the method is based on a visual large model and self-supervised learning. A second texture prediction block extracts a texture by calculating feature covariance and performs texture prediction; A second texture auxiliary loss calculation block upsamples the texture features predicted by the second texture prediction block at each level to the size of the original image, calculates an auxiliary loss by using the texture pseudo-labels, and performs gradient back propagation to realize optimized texture extraction. The contour depth supervision module adopts a contour prediction block to perform average pooling and maximum pooling on the input features, connects them, then performs convolution operation and Sigmoid operation on the obtained features to obtain attention weights, subtracts each element of the attention weights from 1, and element-wise multiplies the input features to enhance the contour features, and uses convolution to perform contour prediction.

7. The method of claim 5, wherein the method is based on a visual large model and self-supervised learning. ​ The contour auxiliary loss calculation block samples the contour prediction block on the contour map predicted by each layer to the original image size, calculates the auxiliary loss with the contour pseudo label, and performs gradient back propagation to realize contour extraction.

8. A whole bone marrow cell automatic counting system, characterized by, The data acquisition module, the processing module and the storage module are included for acquiring a bone marrow smear data set; The data acquisition module is in communication connection with the processing module and sends the bone marrow smear data set thereto; the processing module and the storage module are in communication connection, and the storage module is used for storing at least one executable instruction, which makes the processing module execute the operation corresponding to the automatic counting method of whole film bone marrow cells based on visual large model and self-supervised learning as claimed in any one of claims 1-7 based on the bone marrow smear data set, and automatically classifies and counts the whole film bone marrow cells.

Images