Full-piece bone marrow cell automatic counting method and system based on visual large model and self-supervised learning

Abstract

The invention provides a full bone marrow cell automatic counting method and system based on a visual large model and self-supervised learning. The full bone marrow cell automatic counting method comprises the following steps: segmenting a region of interest from a bone marrow smear by adopting an RO I segmentation model; cutting out slices from the bone marrow smear corresponding to the region of interest by adopting a TO I classification model, and carrying out binary classification to select interested slices; segmenting all cells on the interested slice by adopting a cell segmentation model based on a visual large model; and classifying the cells by adopting a cell classification model based on self-supervised learning, and finally counting. The method has strong generalization and high accuracy, and can provide valuable reference for pathologists of various medical centers.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence learning technology, and in particular to a whole-slice bone marrow cell automatic counting method and system based on a large visual model and self-supervised learning. Background Art

[0002] Bone marrow aspirate smear differential cell count (BMADCC) plays a vital role in the diagnosis and treatment of hematological malignancies. Specifically, it is a process by which pathologists determine the proportions of various cell types in the bone marrow.

[0003] Bone marrow aspirate smear cell differential count helps diagnose acute lymphocytic leukemia (ALL), acute myeloid leukemia (AML), angioimmunoblastic T-cell lymphoma (AITL), Burkitt lymphoma (Burkitt), chronic lymphocytic leukemia (CLL), chronic myeloid leukemia (CML), chronic myelomonocytic leukemia (CMML), classical Hodgkin lymphoma (cHL), diffuse large B-cell lymphoma (DLBCL), essential thrombocythemia (ET), follicular lymphoma (FL), mantle cell lymphoma (MCL), mucosa-associated lymphoid tissue lymphoma (MALT), multiple myeloma (MM), NK / T-cell lymphoma (NKTL), prolymphocytic leukemia (PLL), immune thrombocytopenic purpura (ITP) and other hematological malignancies.

[0004] Traditional bone marrow aspirate smear cell differential counts are performed manually by pathologists using a microscope. Considered the gold standard, this method is widely used to diagnose and monitor a variety of blood and bone marrow diseases. However, it has two major disadvantages: 1) Labor-intensive: Manual counting is a time-consuming and labor-intensive task that requires a lot of time from pathologists. Long hours of continuous work may affect the accuracy of cell classification and counting. 2) Subjectivity: The inconsistency in the level of experience in cell differential counting between different pathologists leads to subjective bias.

[0005] With the rapid development of deep learning, deep learning-based visual models have performed well in image processing tasks such as target detection, semantic segmentation, and image classification. These models make quantitative routine tasks in computer-aided diagnosis possible, thereby speeding up the process, reducing bias, and improving the consistency of results. In order to solve the problems associated with manual counting, many researchers have applied deep learning algorithms to automate bone marrow aspirate smear cell classification counting. This significantly improves the accuracy and efficiency of cell classification counting and has important theoretical and practical value. These studies can be divided into two main methods according to their process: slice-based counting methods and whole-slide-based counting methods.

[0006] Slice-based counting methods require manual slicing of the whole slice image into small pieces (i.e., square bone marrow images), selecting appropriate slices as input for cell classification counting, and then performing cell detection and classification. Wang et al. used the Faster-RCNN target detection algorithm and feature pyramid network to detect 6 cell types in the slices. However, the classification accuracy of the cell detection model still needs to be improved. In contrast, Chandradevan achieved higher overall accuracy by using a separate cell classification model. They developed a two-stage system for standard clinical cell classification, manually selecting BMA slices containing a large number of cells, using Faster-RCNN to detect all cells as a single category object, and then using VGG for cell classification. Yu et al. applied deep convolutional neural networks to automatically detect and classify bone marrow nucleated cells in slices. These methods require manual selection of the best slices, making the process slow and unsuitable for full-process cell classification counting, limiting their scalability in clinical diagnostic work.

[0007] Compared with slice-based counting methods, whole-slice counting methods do not require manual slice selection. Instead, they automatically identify slices of interest (TOI) from the whole slice, detect cells in these slices, and finally classify and count these cells to achieve intelligent full-process cell classification and counting. These methods can be further classified according to the method of slice extraction: grid-based counting methods and ROI-based counting methods.

[0008] Grid-based counting methods directly cut whole slides into uniform-sized slices and then select TOIs from these slices. Tayebi et al. developed an end-to-end automated bone marrow cytology system that cuts whole slides into uniform slices, uses DenseNet121 for binary classification to distinguish TOIs from non-TOIs, and then uses YOLOv4 for cell detection and classification in TOIs. This method achieves high accuracy and shows strong correlation with manual counting results. Lewis et al. proposed a more accurate automated workflow that uses EfficientNetV2S to classify uniform-sized slices into four categories (optimal, granular, hemodilution, and external), uses Faster-RCNN to detect cells of single-category objects in the best slices (i.e., TOIs), and uses EfficientNetV2L for cell classification. Multiple experiments demonstrated the feasibility of automatically generating cell classification counts from WSIs. Both methods cut whole slides into slices, but these slices may include cell aggregation, over-staining, or blank areas, resulting in many invalid slices that need to be filtered, making the process particularly time-consuming. In practice, the area suitable for cell classification counting is the body-tail junction, called the region of interest (ROI).

[0009] ROI-based counting methods identify ROIs from the whole slide and then extract slices from these regions for subsequent cell detection and classification. Wang et al. proposed a hierarchical framework that uses multi-resolution pyramids and Cascade R-CNN to identify bounding boxes suitable as ROIs from the whole slide. Another Cascade R-CNN was then used to detect and classify BMA cells within these regions, achieving effective nucleated cell classification and counting analysis from WSI. Su et al. further explored methods for automatically extracting high-quality slice images and accurately locating and identifying nucleated cells, and proposed a ROI-BMC-DNNet analysis framework. The framework uses a pyramid network and an encoder-decoder to segment ROIs and then extract high-quality patches from these regions. A slice quality assessment network and a cell detection network are then used to automatically identify and count bone marrow nucleated cells. Compared with grid-based counting methods that cut the whole slide into slices of uniform size, ROI-based counting methods produce higher quality slices with less data, thereby reducing processing time during slice classification.

[0010] However, both slice-based and whole-slice counting methods have obvious problems. They are only applicable to bone marrow aspirate smears of specific diseases and medical centers, require a large amount of labeled data, time and computing resources, and have poor generalization. Specifically: 1) Supervised learning problems: 1. The target detection model can only detect known cell types in the dataset, making it only applicable to specific detection tasks. When the cell type changes, additional annotation of cell bounding boxes is required; 2. The image classification model can only learn limited representations in the current dataset. When applied to different medical centers, a large amount of labeled data and retraining are required to maintain high accuracy. 2) Efficiency and staining issues: 1. The grid-based counting method generates a large amount of irrelevant data when cutting the whole slice into slices, which increases the time and computing resources required for the slice classification model; 2. There may be staining differences in BMAs from different medical centers, making the ROI segmentation and TOI classification based on traditional deep learning models less effective and not applicable to other centers.

[0011] In recent years, artificial intelligence, especially in the field of computer vision, has made rapid progress, with significant progress in large models and self-supervised learning. These models can effectively solve the problem 1). In terms of large models, Segment Anything Model (SAM) is trained on the SA-1B dataset, which contains 11 million images and more than 1 billion masks, and shows strong zero-shot generalization capabilities. Even if the domain-specific images are significantly different from the training data of SAM, high-precision segmentation results can be achieved by fine-tuning with a small amount of annotated data using prompts. SAM has recently attracted widespread attention in the field of medical imaging. For example, methods such as MedSAM and Med-SA have been optimized for general medical images. In the field of cell segmentation, methods such as CellSAM, Guided Prompting SAM, and UNSAM have also shown significant results. In addition, self-supervised learning initially constructed supervisory signals from within the image, such as the jigsaw prediction task proposed by Doersch et al. and the image rotation prediction task proposed by Noroozi et al. Subsequently, contrastive learning became one of the mainstream methods of self-supervised learning, learning useful feature representations by comparing the similarities and differences between samples. Notable examples include MoCo proposed by He et al. and SimCLR introduced by Chen et al. Building on these advances, the MAE model proposed by He et al. uses image masks and reconstruction methods to train a feature extractor with strong generalization ability from a large amount of unlabeled data, and achieves excellent classification performance through fine-tuning on a small amount of labeled data. For problem 2), Su et al. solved the time and computational efficiency issues caused by a large number of invalid slices by segmenting ROIs, which only account for a small part of the WSI, and then extracting high-quality slices from these regions. However, previous studies did not address the color differences of BMA smears from different medical centers, nor did they address the significant texture differences between ROI and non-ROI image pairs, and TOI and non-TOI image pairs. Summary of the invention

[0012] In order to overcome the defects in the prior art, the present invention aims to provide a method and system for automatic counting of whole-slice bone marrow cells based on a large visual model and self-supervised learning.

[0013] In order to achieve the above-mentioned object of the present invention, the present invention provides a method for automatic counting of whole-slice bone marrow cells based on a large visual model and self-supervised learning, comprising the following steps:

[0014] Obtain full-slice bone marrow smear dataset;

[0015] Construct a deep learning framework for intelligent cell classification and counting, including ROI segmentation model, TOI classification model, cell segmentation model based on visual big model, and cell classification model based on self-supervised learning;

[0016] The ROI segmentation model segments the region of interest from the bone marrow smear; the TOI classification model cuts slices from the bone marrow smear corresponding to the region of interest and performs binary classification to select the slice of interest; the cell segmentation model segments all cells on the slice of interest; the cell classification model classifies the cells; and finally counts them.

[0017] Optionally, the ROI segmentation model uses an encoder to extract features of different levels of the input original image; uses a texture extraction module to extract texture features of different levels of features extracted by the encoder by calculating the covariance of the features; uses a decoder to connect the texture features with the features of different levels extracted by the encoder and input them into the decoder for decoding; uses a first texture depth supervision module to supervise the texture features of different levels of the decoder using the texture image extracted from the original image.

[0018] Optionally, the texture extraction module performs convolution calculation on the input original image, and calculates the covariance matrix between channels of each pixel of the feature map after the convolution;

[0019] Construct an upper triangular matrix of size (C1×(C1+1)) / 2 containing the same information as the covariance matrix, and reshape the upper triangular matrix into a vector;

[0020] Perform the same operation as above on each pixel of the input original image to obtain a feature map;

[0021] The feature map is connected to the input original image to obtain the original feature information and the texture information extracted from the original feature map.

[0022] Optionally, the first texture depth supervision module includes:

[0023] The first texture prediction block receives the output of the decoder and performs convolution to obtain the texture prediction Figure 1 , the texture prediction Figure 1 Upsample to the same resolution as the original input image to get the texture prediction Figure 2 ;

[0024] The first texture-assisted loss calculation block calculates texture predictions Figure 2 The texture-assisted loss between the texture image extracted from the original image.

[0025] Optionally, the TOI classification model uses a CNN backbone network to extract multi-scale features of the input bone marrow smear slices;

[0026] During the training process, texture pseudo labels and contour pseudo labels are generated for deep supervision; each scale feature of the bone marrow smear slice extracted by the CNN backbone network is processed using the second texture deep supervision module, the texture is predicted and the texture is enlarged to match the size of the original image, and the auxiliary loss is calculated using the predicted texture result and the texture pseudo label, and then gradient back propagation is performed; each scale feature of the bone marrow smear slice extracted by the CNN backbone network is processed using the contour deep supervision module, the contour is predicted and the contour is lifted to the same size as the original image; the auxiliary loss is calculated using the predicted contour and the contour pseudo label, and then gradient back propagation is performed.

[0027] Optional,

[0028] The second texture depth supervision module comprises:

[0029] The second texture prediction block extracts texture and performs texture prediction by calculating feature covariance;

[0030] The second texture auxiliary loss calculation block samples the texture features predicted at each level in the second texture prediction block to the original image size, calculates the auxiliary loss using the texture pseudo-labels, and performs gradient backpropagation to achieve optimized texture extraction.

[0031] Optionally, the contour deep supervision module uses the contour prediction block to perform average pooling and maximum pooling on the input features, and concatenates them, then performs convolution and sigmoid operations on the obtained features to obtain attention weights; subtract each element of the attention weight from 1, and perform element-by-element multiplication with the input features to enhance the contour features, and use convolution for contour prediction;

[0032] The contour auxiliary loss calculation block samples the contour map predicted by the contour prediction block at each layer to the original image size, calculates the auxiliary loss with the contour pseudo-label, and performs gradient backpropagation to achieve contour extraction.

[0033] Optionally, the cell segmentation model includes an image encoder, a hint encoder and a mask decoder,

[0034] The image encoder takes the image output by the TOI classification model, rescales the image and pads the shorter side to obtain the target input resolution, and uses a convolution operation to obtain n channels;

[0035] The prompt encoder generates point, box, text, and mask prompts. Sparse prompts are mapped to n-dimensional vector embeddings. Specifically, position encoding is used to represent points and boxes. A point is represented as the sum of the position encoding and one of the two learned embeddings. The two learned embeddings indicate whether the point is in the foreground or background. A box is represented by an embedding pair, and the learned embeddings are summed for each prompt type and free-form text using the text encoder. Mask prompts use convolutional embeddings and are summed element-wise in the image embedding.

[0036] The mask decoder maps the image embedding, hint embedding, and output token to a mask;

[0037] The cell segmentation model freezes the official weights of the image encoder and the hint encoder and only updates the mask decoder weights;

[0038] Use the SegAny method to predict the cell mask based on the point, and use the geometric center point of the cell mask as the foreground point to fine-tune the mask decoder weights; when inferring cell counts, use SegEvery for segmentation;

[0039] In the cell segmentation fine-tuning stage, each TOI image is annotated with a single cell mask; in the cell segmentation verification stage, each TOI image is annotated with all single cell masks, all of which are used for verification.

[0040] Optionally, the cell classification model includes:

[0041] The masking module divides the image output by the cell segmentation model into regular non-overlapping patches, randomly samples the non-overlapping patches and makes them visible, and masks the remaining patches;

[0042] Encoder,embeds the position of visible patches;

[0043] Decoder, the input consists of the encoded visible patches and masks. All the encoded visible patches and masks are embedded with positions. The decoder includes a Transformer block. The decoder predicts the pixel values ​​of each masked patch to reconstruct the input. Each element in the decoder output represents a vector of pixel values ​​in a patch. The last layer of the decoder is a linear projection whose output channels are equal to the number of pixel values ​​in a patch.

[0044] The reconstruction module reconstructs the output of the decoder to form a reconstructed image.

[0045] The present invention also proposes a whole-slice bone marrow cell automatic counting system, comprising a data acquisition module, a processing module and a storage module for acquiring a bone marrow smear data set;

[0046] The data acquisition module is communicatively connected with the processing module and sends the bone marrow smear data set to the processing module; the processing module is communicatively connected with the storage module, and the storage module is used to store at least one executable instruction, and the executable instruction enables the processing module to perform operations corresponding to the above-mentioned method for automatic counting of whole-slice bone marrow cells based on a large visual model and self-supervised learning based on the bone marrow smear data set, so as to automatically classify and count the whole-slice bone marrow cells.

[0047] The beneficial effects of the present invention are:

[0048] In the present invention, the ROI segmentation model has a significant effect in extracting texture information, reducing the influence of staining differences (because bone marrow aspirate smears usually show staining variations, especially the body, tail and body-tail junction of thumbnails have obvious texture differences); the TOI classification model has a strong ability in extracting texture and contour features, and can also achieve high accuracy on relatively small data sets, solving the significant texture differences between the slices of interest and the slices of non-interest image pairs; the cell segmentation model is more accurate than traditional target detection and image segmentation methods, improves segmentation performance, and specifically has a strong generalization ability, and can achieve accurate segmentation without additional fine-tuning; the cell classification model has a stronger feature extraction effect and a strong classification performance, and can achieve effective classification according to specific cell types with minimal additional annotation and fine-tuning. Therefore, the present invention has strong generalization and high accuracy, and can provide valuable reference for pathologists in various medical centers.

[0049] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0051] Figure 1 It is a principle block diagram of the present invention;

[0052] Figure 2 This is the network architecture diagram of the ROI segmentation model;

[0053] Figure 3 It is a schematic diagram of the network structure of the texture extraction module;

[0054] Figure 4 is a schematic diagram of the network structure of the first texture prediction block;

[0055] Figure 5 It is a schematic diagram of the network structure of the TOI classification model;

[0056] Figure 6is a schematic diagram of the network structure of the second texture prediction block;

[0057] Figure 7 It is a schematic diagram of the network structure of the contour prediction block;

[0058] Figure 8 It is a schematic diagram of the network structure of the cell segmentation model;

[0059] Fig. 9 is a schematic diagram of the network structure of the mask decoder;

[0060] Fig.10 This is the principle block diagram of the cell classification model. DETAILED DESCRIPTION

[0061] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0062] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0063] like Figure 1 As shown, the present invention provides a method for automatic counting of whole-slice bone marrow cells based on a visual large model SAM and self-supervised learning, comprising the following steps:

[0064] Obtain a bone marrow smear data set. In this embodiment, a 100x oil-immersion objective is used to scan a bone marrow aspiration smear, and its corresponding thumbnail (reduction factor is 100) is generated by python to obtain a bone marrow smear data set. A deep learning framework for cell classification and counting is constructed, including a ROI segmentation model TextureUnet, a TOI classification model TCNet, a cell segmentation model, and a cell classification model; a region of interest is segmented from a bone marrow smear using a ROI segmentation model; a slice is cut out from a bone marrow smear corresponding to a region of interest using a TOI classification model, and a binary classification is performed to select a slice of interest; all cells on a slice of interest are segmented using a cell segmentation model; and cells are classified and counted using a cell classification model. The cell ratio is calculated by dividing the number of each cell type by the total number of cells, and a 16-class cell classification count histogram is generated.

[0065] Among them, Figure 2As shown, the ROI segmentation model TextureUnet utilizes an enhanced encoder-decoder structure, which is developed based on the Unet framework and uses a ResNet-based encoder for feature extraction. For a single input original image, it is first sent to the encoder to extract its multi-level feature map. The feature maps at different levels are then input into the texture extraction module TEM to extract the texture feature map. The texture feature map is then connected to the feature map from the encoder and input into the decoder for decoding. The decoding obtains the intermediate features of each layer and the final predicted feature ROI mask. This embodiment designs a first texture deep supervision module TDSM1, which uses the texture feature map extracted from the original input image to supervise the feature maps at different levels of the decoder (each layer uses MSE loss and then sums them). Specifically,

[0066] The texture extraction module TEM extracts texture features of different levels of features extracted by the encoder by calculating the covariance of the features. Figure 3 As shown in Figure 1, the texture extraction module TEM takes the feature map f with a resolution of C×H×W as the original image input, where C is the number of channels, and H and W are the height and width of the feature map f, respectively. First, a 1×1 convolution calculation is performed on the feature map f to obtain a feature map fk with a resolution of C1×H×W, and then the covariance matrix between the channels of each pixel of the feature map fk is calculated to obtain a covariance matrix Ck with a resolution of C1×C1. The covariance matrix between features measures the co-occurrence of features and can be used to extract texture information. Since the covariance matrix Ck is a symmetric matrix, an upper triangular matrix can be used, which contains the same information as the covariance matrix Ck, but with a size of (C1×(C1+1)) / 2. Then, the upper triangular matrix is ​​reshaped into a vector to obtain a vector vk with a size of (C1×(C1+1)) / 2. The same operation is performed on each pixel of the input feature map f to obtain a feature map fv with a resolution of (C1×(C1+1)) / 2×H×W. Finally, the feature map fv is concatenated with the input feature map f to obtain a feature map ft with a resolution of (C_1+(C_1×(C_1+1)) / 2)×H×W. The feature map ft contains the original feature information and the texture information extracted from the original feature map f, which is then fed into the decoder.

[0067] During the training process of the ROI segmentation model, the first texture depth supervision module TDSM1 uses the texture image extracted from the original image to supervise the texture features of different levels of the decoder. Specifically, the first texture depth supervision module TDSM1 includes a first texture prediction block TPB1 and a first texture auxiliary loss calculation block TALCB1. The first texture depth supervision module TDSM1 receives the output from the decoder and uses a 1×1 convolution process to obtain a texture prediction with a resolution of 1×H×W. Figure 1 t. Then, if Figure 4 As shown, the texture prediction Figure 1 t is upsampled to the same resolution as the original input image, resulting in a texture prediction with a resolution of 1×H1×W1 Figure 2 tu. The first texture-assisted loss calculation block calculates texture predictions Figure 2 The texture-assisted loss Lt between tu and the texture image extracted from the original input image. Through the implementation of the first texture deep supervision module TDSM1, the network can be trained to extract texture information from the input image and use it to assist the segmentation of the sensory interest region.

[0068] In order to fully consider the characteristics of bone marrow smears and enhance texture and contour feature extraction, the TOI classification model TCNet is used for texture and contour perception. The input of the TOI classification model TCNet is the RGB color image of the bone marrow smear, and the output is the image classification result, i.e., TOI or non-TOI. During the training process, this embodiment uses the LBP (Local Binary Pattern) algorithm and the Canny edge detection algorithm in OpenCV to generate texture pseudo-labels and contour pseudo-labels for deep supervision. Specifically, the TOI classification model TCNet includes three basic components: a CNN backbone network (ResNet-50) for extracting image feature representation, a second texture depth supervision module TDSM2, and a contour depth supervision module CDSM. Specifically,

[0069] like Figure 5 As shown, given input I∈R (3×H×W) , using CNN backbone network (ResNet-50) to extract multi-scale features That is, each scale feature of the bone marrow smear slice. In the training phase, the extracted features (F2, F3, F4, F5) are used as the input of the second texture depth supervision module TDSM2 and the contour depth supervision module CDSM. It is sent to the second texture depth supervision module TDSM2 and the contour depth supervision module CDSM at the same time. The second texture depth supervision module TDSM2 processes the input feature f2 and predicts the texture The predicted texture T2 is then scaled up to match the size of the original image (H×W). The auxiliary loss is calculated using the predicted results and the texture pseudo-labels, and then the gradient is back-propagated. Similarly, the contour deep supervision module CDSM processes the input features and predicts the contour And it is lifted to the same size (H×W) as the original image. The auxiliary loss is calculated using the prediction results and the contour pseudo-labels, and then the gradient is back-propagated. The same operation is performed on the remaining features f3, f4, and f5 to achieve deep supervision of textures and contours at each layer. The CNN backbone network of the TOI classification model finally outputs the classification results.

[0070] The second texture depth supervision module TDSM 2 includes: a second texture prediction block TPB2 and a second texture auxiliary loss calculation block TALCB2. Figure 6 As shown, the second texture prediction block TPB2 extracts texture and performs texture prediction by calculating feature covariance. It takes a feature map with a resolution of C×H×W as input, uses a 1×1 convolution kernel for multiple calculations, and obtains a feature map with a resolution of C1×H×W. In order to improve computational efficiency, C1 is set to a smaller value (this paper sets it to 24), and the obtained feature map is used to learn multiple aspects of texture in subsequent operations. Next, the feature map is input into a texture extraction module to calculate the covariance matrix between feature channels at each position to capture the correlation between different responses in the convolutional features. The texture extraction module here can use the same network structure as the above-mentioned texture extraction module TEM. The covariance matrix between features measures the co-occurrence of features, describes the combination of features, and is used to represent texture information. For the feature map f k Each pixel in ∈C1×H×W Compute its covariance matrix as and Since the covariance matrix (C1×C1) has a diagonal symmetric property, the upper triangular part of the matrix is ​​used to represent the texture features, and the result is reshaped into a feature vector. The same operation is performed on each pixel in the feature map, and the results are connected to obtain a feature map containing texture information. The obtained feature map is subjected to a 1×1 convolution operation to obtain the predicted texture feature map. The second texture auxiliary loss calculation block TALCB2 samples the texture features predicted by the second texture prediction block TPB2 at each level to the original image size, calculates the auxiliary loss using the texture pseudo-label, and performs gradient backpropagation to achieve optimized texture extraction.

[0071] The contour deep supervision module CDSM includes: contour prediction block CPB and contour auxiliary loss calculation block CALCB. In the contour prediction block CPB, a reverse attention mechanism is introduced to enhance the sensitivity to contour features. Figure 7 As shown, the input features are first average pooled and max pooled and concatenated, and then the obtained features are subjected to 7×7 convolution and Sigmoid operations to obtain attention weights. Next, each element of the attention weight is subtracted from 1 and multiplied element-by-element with the input features to achieve feature enhancement. Through the above processing, the contour features are enhanced. Finally, 3×3 convolution is used for contour prediction. The contour auxiliary loss calculation block CALCB samples the contour map predicted by the contour prediction block CPB at each layer to the original image size, calculates the auxiliary loss with the contour pseudo label, and performs gradient backpropagation to achieve contour extraction.

[0072] The cell segmentation model is built on the basis of the Transformer visual model and has three components, such as Figure 8 As shown, a powerful image encoder, a flexible hint encoder and an efficient mask decoder. It supports point, box and text hints, and is trained on the SA-1B dataset containing 11 million images and more than 1 billion masks, demonstrating strong segmentation and generalization capabilities. It supports two segmentation methods: SegAny, which predicts the mask of a single object based on a point or box, and SegEvery, which predicts the mask of all objects in the image. In order to segment all cells, this embodiment selects ViT-H (with 632M parameters) as the backbone network. When the cell segmentation model SAM performs poorly in a specific field, the accuracy can be improved by fine-tuning. Since the training images of the original visual large model SAM are natural images, which are different from bone marrow cell images, the direct segmentation of cells needs to be improved. In this paper, the cell mask dataset is used for fine-tuning. SAM is directly fine-tuned by freezing the official weights of the image encoder and hint encoder, which effectively handle feature extraction and point hints, and only the decoder weights are updated. The SegAny method is used to predict the cell mask based on points, and the mask decoder weights are fine-tuned with the geometric center point of the cell mask as the foreground point. The fine-tuning method is: calculate the dice loss and bce loss of the cell mask and the predicted mask of the model, and then perform backpropagation. During the cell segmentation fine-tuning stage, a single cell mask is annotated for each TOI image. During validation and testing, that is, when cell counting inference is performed, SegEvery is used for segmentation. No box or text prompts are used here, because SegEvery is based on point prompts, which can improve segmentation performance. In the cell segmentation verification stage, all single cell masks are annotated for each TOI image, and all are used for verification. When applied to other medical centers, due to the relatively fixed cell morphology, the fine-tuned SAM shows strong generalization ability and can achieve accurate segmentation without additional fine-tuning.

[0073] Specifically, the image encoder takes the image output by the TOI classification model, rescales the image and pads the shorter side to the target input resolution, and uses convolution operations to obtain n channels, which in this example is 1024*1024, and the output channels n is always 256. The image encoder can be any network that outputs an image embedding of size C×H×W. Motivated by scalability and obtaining strong pre-training, ViT pre-trained with MAE is minimally adjusted to handle high-resolution inputs, specifically ViT-H / 16 with 14×14 window attention and four equally spaced global attention blocks. The output of the image encoder is a 16-fold downscaled embedding of the input image. Since the runtime goal is to process each prompt in real time, a large number of image encoder flops can be afforded because they are only calculated once per image, not once per prompt. Following standard practice, the input resolution of 1024×1024 is obtained by rescaling the image and padding the shorter side. Therefore, the image embedding is 64×64. To reduce the channel dimension, 1×1 convolutions are used to obtain 256 channels, and then 3×3 convolutions are used to obtain 256 channels. Each convolution is followed by layer normalization.

[0074] The hint encoder considers two sets of hints: sparse (points, boxes, text) and dense (masks). Sparse hints represent points and boxes via positional encodings and sum over learned embeddings for each hint type and free-form text using an off-the-shelf text encoder with a neural network model from CLIP Multimodal. Dense hints use convolutional embeddings and sum element-wise over the image embedding. Specifically, sparse hints are mapped to 256-dimensional vector embeddings. A point is represented as the sum of a positional encoding and one of two learned embeddings that indicate whether the point is in the foreground or background. A box is represented by an embedding pair: (1) the positional encoding of its top-left corner is summed with a learned embedding representing “top-left corner”; (2) the same structure but using a learned embedding representing “bottom-right corner”. Finally, to represent free-form text, the text encoder from CLIP is used (in general any text encoder is possible). Dense hints have a spatial correspondence with the image. The input mask is 4 times lower resolution than the input image, and then it is downscaled by a factor of 4 using two 2×2 convolutions with stride 2 for output channels 4 and 16 respectively. The final 1×1 convolution maps the channel dimension to 256. Each layer is separated by a GELU activation and layer normalization. Mask and image embedding elements are then added. If there is no mask hint, a learned embedding representing "no mask" is added to each image embedding position.

[0075] The mask decoder efficiently maps the image embedding, hint embedding, and output tokens to the mask. The design employs a modification of the Transformer decoder block followed by a dynamic mask prediction head. The modified decoder block uses hint self-attention and cross-attention in both directions (hint to image embedding and vice versa) to update all embeddings. After running both blocks, the image embedding is upsampled and a multi-layer perceptron (MLP) maps the output tokens to a dynamic linear classifier which then computes the mask foreground probability for each image location.

[0076] Specifically, the mask decoder effectively maps an image embedding and a set of prompt embeddings to an output mask. To combine these inputs, the standard Transformer decoder is modified with inspiration from the Transformer segmentation model. Before applying the mask decoder, a learned output token embedding is first inserted into the set of prompt embeddings, which will be used for the output of the decoder. For simplicity, these embeddings (excluding the image embedding) are collectively referred to as "tokens".

[0077] The mask decoder is designed as Fig. 9 As shown, each decoder layer performs 4 steps: (1) self-attention on tokens, (2) criss-cross attention from tokens (as queries) to image embeddings, (3) point-wise MLP updates each token, and (4) criss-cross attention from image embeddings (as queries) to tokens. The last step updates the image embedding with the hint information. During the criss-cross attention process, the image embedding is treated as a set of 64×64 256-dimensional vectors. Each self / criss-cross attention and MLP has residual connections, layer normalization, and dropout of 0.1 during training. The next decoder layer obtains the updated tokens and updated image embeddings from the previous layer. A two-layer decoder is used in this embodiment.

[0078] To ensure that the decoder has access to critical geometric information, positional encodings are added to the image embedding whenever they participate in an attention layer. Additionally, the entire original hint tokens (including their positional encodings) are re-appended to the updated tokens whenever they participate in an attention layer. This allows for strong dependencies on the geometric position and type of hint tokens.

[0079] After running the decoder, the updated image embedding is upsampled by a factor of 4 using two transposed convolutional layers (it is now 4 times smaller relative to the input image). The tokens are then again involved in the image embedding and the updated output token embedding is passed to a small 3-layer MLP that outputs a vector that matches the channel dimension of the upscaled image embedding. Finally, the spatial dot product between the upsampled image embedding and the MLP output is used to predict the mask.

[0080] The Transformer uses an embedding dimension of 256. The internal dimension of the MLP block in the Transformer is larger, 2048, but the MLP is only applied to a relatively small number of prompt tokens (rarely greater than 20). However, in the criss-cross attention layer there is a 64×64 image embedding, and for computational efficiency the channel dimension of the query, key, and value is reduced by a factor of 2 to 128. All attention layers use 8 heads.

[0081] The transposed convolutions used to upsample the output image embeddings are 2×2 with stride 2, output channel sizes 64 and 32, and have GELU activations. They are separated by layer normalization.

[0082] The cell classification model adopts the self-supervised learning model MAE, using the ViT series as the backbone network, such as ViT-B, to accurately classify all cells segmented by the cell segmentation model in TOI, and is trained in two stages: the cell image reconstruction stage and the cell classification fine-tuning stage. In the cell image reconstruction stage, the self-supervised learning model MAE learns universal feature representations by reconstructing images from a large amount of unlabeled cell data. In the cell classification fine-tuning stage, the self-supervised learning model MAE encoder is used as a universal feature extractor and fine-tuned using a small amount of labeled cell data to obtain a highly accurate cell classification model. When applied to other medical centers, due to the powerful feature extraction capabilities of the self-supervised learning model MAE, it demonstrated strong classification performance, achieving effective classification according to specific cell types with minimal additional annotation and fine-tuning.

[0083] Self-supervised learning model MAE is a simple self-encoding method that can reconstruct the original signal given some observations. Fig.10 As shown, like all autoencoders, this embodiment has an encoder that maps the observed signal to a latent representation, and a decoder that reconstructs the original signal from the latent representation. Unlike classic autoencoders, this embodiment uses an asymmetric design that allows the encoder to operate on only part of the observed signal (without mask tokens) and a lightweight decoder to reconstruct the full signal from the latent representation and mask tokens.

[0084] Masking. According to ViT, the image is divided into regular non-overlapping patches. The input image size is generally 224*224, and the patch size is generally 16*16. Then a subset of the patches is sampled, and the remaining patches are masked (i.e. deleted). Sampling strategy: Sampling random patches without replacement, following a uniform distribution, is called "random sampling". Random sampling with a high masking ratio (i.e. the proportion of removed patches) largely eliminates redundancy, creating a task that cannot be easily solved by extrapolating from visible neighboring patches. Uniform distribution prevents potential center bias (i.e. there are more masked patches near the center of the image). Finally, highly sparse input creates opportunities for designing efficient encoders.

[0085] The MAE encoder is a ViT, but only applied to visible, unoccluded patches. Just like in the standard ViT, the encoder embeds patches by adding a linear projection of the positional embedding, and then processes the result through a series of Transformer blocks. However, the encoder here in this paper only operates on a small fraction (e.g., 25%) of the full set. Masked patches are removed; mask tokens are not used. This allows very large encoders to be trained with only a fraction of the compute and memory. The full set is processed by a lightweight decoder.

[0086] The input to the MAE decoder is a set of tokens consisting of the encoded visible patch and mask tokens. Fig.10 , each mask token is a shared learned vector indicating the presence of a missing patch to be predicted. A position embedding is added for all tokens in this complete set; without this, the mask tokens would have no information about their position in the image. The decoder has another series of Transformer blocks. The MAE decoder is only used to perform the image reconstruction task during pre-training (only the encoder is used to generate image representations for recognition). Therefore, the decoder architecture can be flexibly designed in a way that is independent of the encoder design. Experiment with very small decoders, narrower and shallower than the encoder. For example, the default decoder has <10% of the computation per token compared to the encoder. With this asymmetric design, the entire set of tokens is processed only by the lightweight decoder, which greatly reduces pre-training time.

[0087] Reconstruction Objective: The MAE in this example reconstructs the input by predicting the pixel values ​​of each masked patch. Each element in the decoder output is a vector of pixel values ​​representing a patch. The last layer of the decoder is a linear projection whose number of output channels is equal to the number of pixel values ​​in a patch. The output of the decoder is reconstructed to form a reconstructed image. The loss function calculates the mean squared error (MSE) between the reconstructed image and the original image in pixel space, and only calculates the loss on the masked patch, similar to BERT.

[0088] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0089] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for automatic counting of whole-slice bone marrow cells based on a large visual model and self-supervised learning, characterized in that: The following steps are involved: Obtain bone marrow smear dataset; Construct a deep learning framework for cell classification and counting, including ROI segmentation model, TOI classification model, cell segmentation model based on visual big model, and cell classification model based on self-supervised learning; The ROI segmentation model segments the region of interest from the bone marrow smear; the TOI classification model cuts slices from the bone marrow smear corresponding to the region of interest and performs binary classification to select the slice of interest; the cell segmentation model segments all cells on the slice of interest; the cell classification model classifies the cells; and finally counts them.

2. The method for automatic whole-slice bone marrow cell counting based on a visual large model and self-supervised learning according to claim 1, characterized in that: The ROI segmentation model uses an encoder to extract features of different levels of the input original image; uses a texture extraction module to extract texture features of the features of different levels extracted by the encoder by calculating the covariance of the features; uses a decoder to connect the texture features with the features of different levels extracted by the encoder and input them into the decoder for decoding; uses a first texture depth supervision module to supervise the texture features of different levels of the decoder using a texture image extracted from the original image.

3. The method for automatic counting of whole-slice bone marrow cells based on a large visual model and self-supervised learning according to claim 2, characterized in that: The texture extraction module performs convolution calculation on the input original image, and calculates the covariance matrix between the channels of each pixel in the convolved feature map; Construct an upper triangular matrix of size (C1×(C1+1)) / 2 containing the same information as the covariance matrix, and reshape the upper triangular matrix into a vector; Perform the same operation as above on each pixel of the input original image to obtain a feature map; The feature map is connected to the input original image to obtain the original feature information and the texture information extracted from the original feature map.

4. The method for automatic counting of whole-slice bone marrow cells based on a visual large model and self-supervised learning according to claim 2, characterized in that: The first texture depth supervision module comprises: The first texture prediction block receives the output of the decoder and performs convolution to obtain a texture prediction image 1, and upsamples the texture prediction image 1 to the same resolution as the input original image to obtain a texture prediction image 2; The first texture-assisted loss calculation block calculates the texture-assisted loss between the texture prediction image 2 and the texture image extracted from the original image.

5. The method for automatic whole-slice bone marrow cell counting based on a visual large model and self-supervised learning according to claim 1, characterized in that: The TOI classification model uses a CNN backbone network to extract multi-scale features of the input bone marrow smear slices; Generate texture pseudo labels and contour pseudo labels for deep supervision during training; The second texture deep supervision module is used to process each scale feature of the bone marrow smear slice extracted by the CNN backbone network, predict the texture and enlarge the texture to match the size of the original image, and the auxiliary loss is calculated using the predicted texture result and the texture pseudo-label, and then the gradient back propagation is performed; the contour deep supervision module is used to process each scale feature of the bone marrow smear slice extracted by the CNN backbone network, predict the contour and lift the contour to the same size as the original image; the auxiliary loss is calculated using the predicted contour and the contour pseudo-label, and then the gradient back propagation is performed.

6. The method for automatic whole-slice bone marrow cell counting based on a visual large model and self-supervised learning according to claim 5, characterized in that: The second texture depth supervision module comprises: The second texture prediction block extracts texture and performs texture prediction by calculating feature covariance; The second texture auxiliary loss calculation block samples the texture features predicted at each level in the second texture prediction block to the original image size, calculates the auxiliary loss using the texture pseudo-labels, and performs gradient backpropagation to achieve optimized texture extraction.

7. The method for automatic whole-slice bone marrow cell counting based on a visual large model and self-supervised learning according to claim 5, characterized in that: The contour deep supervision module uses the contour prediction block to perform average pooling and maximum pooling on the input features and connect them. Then, it performs convolution and sigmoid operations on the obtained features to obtain attention weights. Each element of the attention weight is subtracted from 1 and multiplied element-by-element with the input features to enhance the contour features. Convolution is used for contour prediction. The contour auxiliary loss calculation block samples the contour map predicted by the contour prediction block at each layer to the original image size, calculates the auxiliary loss with the contour pseudo-label, and performs gradient backpropagation to achieve contour extraction.

8. The method for automatic whole-slice bone marrow cell counting based on a visual large model and self-supervised learning according to claim 1, characterized in that: The cell segmentation model includes an image encoder, a cue encoder, and a mask decoder; The image encoder takes the image output by the TOI classification model, rescales the image and pads the shorter side to obtain the target input resolution, and uses a convolution operation to obtain n channels; The prompt encoder generates point, box, text, and mask prompts. Sparse prompts are mapped to n-dimensional vector embeddings. Specifically, position encoding is used to represent points and boxes. A point is represented as the sum of the position encoding and one of the two learned embeddings. The two learned embeddings indicate whether the point is in the foreground or background. A box is represented by an embedding pair, and the learned embeddings are summed for each prompt type and free-form text using the text encoder. Mask prompts use convolutional embeddings and are summed element-wise in the image embedding. The mask decoder maps the image embedding, hint embedding, and output token to a mask; Freeze the official weights of the image encoder and hint encoder, and only update the mask decoder weights; Use the SegAny method to predict the cell mask based on the point, and use the geometric center point of the cell mask as the foreground point to fine-tune the mask decoder weights; In cell counting inference, SegEvery is used for segmentation; In the cell segmentation fine-tuning stage, each TOI image is annotated with a single cell mask; in the cell segmentation verification stage, each TOI image is annotated with all single cell masks, all of which are used for verification.

9. The method for automatic whole-slice bone marrow cell counting based on a large visual model and self-supervised learning according to claim 1, characterized in that: The cell classification model includes: The masking module divides the image output by the cell segmentation model into regular non-overlapping patches, randomly samples the non-overlapping patches and makes them visible, and masks the remaining patches; Encoder,embeds the position of visible patches; Decoder, the input consists of the encoded visible patches and masks. All the encoded visible patches and masks are embedded with positions. The decoder includes a Transformer block. The decoder predicts the pixel values ​​of each masked patch to reconstruct the input. Each element in the decoder output represents a vector of pixel values ​​in a patch. The last layer of the decoder is a linear projection whose output channels are equal to the number of pixel values ​​in a patch. The reconstruction module reconstructs the output of the decoder to form a reconstructed image.

10. A whole-slice bone marrow cell automatic counting system, characterized in that: It includes a data acquisition module, a processing module and a storage module for acquiring a bone marrow smear data set; The data acquisition module is communicatively connected with the processing module and sends the bone marrow smear data set to the processing module; the processing module is communicatively connected with the storage module, and the storage module is used to store at least one executable instruction, and the executable instruction enables the processing module to perform operations corresponding to the method for automatic counting of whole-slice bone marrow cells based on a large visual model and self-supervised learning as described in any one of claims 1 to 9 based on the bone marrow smear data set, and automatically classify and count the whole-slice bone marrow cells.

Images