A bone marrow cell few sample classification method, system and electronic equipment

Abstract

The application discloses a kind of bone marrow cell few sample classification method, system and electronic equipment, method includes: obtaining bone marrow cell dataset, constructs few sample training set and test set establishes the multi-scale multi-modal few sample classification network model based on text multi-source generation selection;According to the few sample training set, the few sample classification network model is trained, and the trained bone marrow cell classification network model is obtained;The bone marrow cell image of test is input into the bone marrow cell classification network model, and the prediction class of bone marrow cell image is obtained, realizes the bone marrow cell image classification under the condition of few sample training.The present application selects histopathology pre-training model for bone marrow cell classification, and designs text multi-source generation selection module, standardizes the semantic description of each category of bone marrow cell, and further improves the few sample classification accuracy of bone marrow cell, which can be widely applied in the field of computer vision technology.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system and electronic device for classifying bone marrow cells in small samples. Background Technology

[0002] Previous studies on bone marrow cell classification have relied on building large datasets for supervised training. However, labeling bone marrow cells with classifiers is extremely resource-intensive; a single bone marrow smear typically requires labeling 200-300 single-cell classes, necessitating experienced physicians working continuously for over four hours to complete. In practical applications, collecting and labeling images from each hospital and building large datasets for individual training is not feasible.

[0003] Many deep learning models possess a certain degree of generalization ability, and by using a small amount of labeled data and applying various methods to adjust the classification model or data—that is, performing few-shot classification—they can achieve good classification results on multiple datasets. Therefore, in the application of bone marrow cell classification, we consider an application scenario: for newly acquired images, we annotate a subset of cell labels to construct a labeled few-shot dataset for classification. This study investigates the few-shot classification task of bone marrow cells in this scenario, exploring its application effectiveness.

[0004] Following the release of CLIP by OpenAI, large-scale foundational models have become a new research hotspot. CLIP utilizes billions of internet image-text pairs to build large-scale multimodal models. During training, it aligns images and text, acquiring rich visual language knowledge through contrastive learning, thus exhibiting better generalization performance and demonstrating superior few-shot classification capabilities on various datasets. However, when applied to few-shot classification of bone marrow cells, its classification performance is poor. The main problem lies in:

[0005] (1) Lack of precise text descriptions for categories. In CLIP model training, the category text of natural images is matched with the images after training. In downstream tasks, input text such as "a picture of a dog" can successfully match the image of a dog, thanks to the text-image matching process in the pre-training stage. However, for categories such as bone marrow cells that have not been pre-trained, directly using the category text cannot match the image. Therefore, it is necessary to generate more general and discriminative text descriptions, such as "the color is blue-purple" and "the cell nucleus is located in the center of the image," to accurately match image features and improve model performance;

[0006] (2) The design did not take into account the fine-grained characteristics of bone marrow cells. Bone marrow cell classification is a fine-grained classification task. When classifying, the fine-grained requirements need to be considered. When classifying a small number of samples, only the overall image features are considered, without paying attention to the detailed features of the image.

[0007] (3) The pre-training dataset differs greatly from the bone marrow cell dataset. CLIP pre-training data consists of common natural images, which do not include bone marrow cell images. Furthermore, natural images and bone marrow cell images differ greatly, so the model's good classification ability cannot be inherited in practical applications. Summary of the Invention

[0008] In view of this, embodiments of the present invention provide a method, system, and electronic device for classifying small samples of bone marrow cells with high accuracy.

[0009] One aspect of this invention provides a method for classifying bone marrow cells in small samples, comprising the following steps:

[0010] Obtain a bone marrow cell dataset and construct a small-sample training and test set;

[0011] A multi-scale, multi-modal, few-shot classification network model based on multi-source text generation selection is established. The few-shot classification network model utilizes a histopathology pre-trained model and incorporates a multi-source text generation selection module. This module generates category text descriptions for input to the network.

[0012] The few-sample classification network model is trained using the few-sample training set to obtain a trained bone marrow cell classification network model.

[0013] The tested bone marrow cell images are input into the bone marrow cell classification network model to obtain the predicted category of the bone marrow cell images, thus achieving bone marrow cell image classification under the condition of few-sample training.

[0014] Optionally, the few-shot classification network model includes a text feature extractor, an image feature extractor, and an image-text similarity score calculation module;

[0015] The text feature extractor is used to extract text features using a text-transformer model;

[0016] The image feature extractor is used to extract image features using the ViT-B16 model;

[0017] The image-text similarity score calculation module is used to obtain text and image similarity scores using the feature dot product method;

[0018] The multi-source text generation selection module includes a classification branch at the original image scale and a classification branch at the subject image scale. The subject image is obtained by localization and segmentation of the original image. The classification branch at the original image scale is used to extract the overall features of the image, while the classification branch at the subject image scale is used to extract the detailed features of the image.

[0019] Optionally, the category text description generated by the multi-source text generation selection module includes single text content generated by the GPT-3 large language model, text and image content generated by the GPT-4 large language model, and supplementary content of professional medical knowledge;

[0020] The multi-source text generation and selection module adaptively selects text that matches the image as the category text description input to the network through a selection function;

[0021] The expression for the selection function is:

[0022]

[0023] Where F represents the selection function; α and β represent proportionality coefficients; C i Normalized image features representing each category; T j Represents each text feature; cos(T) i ,T j The expression represents calculating the cosine similarity between two different texts.

[0024] Optionally, the establishment of a multi-scale, multi-modal, few-shot classification network model based on text multi-source generation selection includes the following steps:

[0025] The original image and corresponding text are fed into a multimodal network for classification, and prediction results are obtained from the global perspective of the cell image.

[0026] The original image is cropped and enlarged to obtain the main image;

[0027] The main image and corresponding text are fed into the multimodal network for classification, and prediction results are obtained from the details of the cell image.

[0028] By combining the classification branches at the original image scale and the classification branches at the subject image scale, the final classification result is obtained through ensemble learning in the case of few-sample classification.

[0029] Optionally, the overall image extracted from the classification branch at the original image scale is the original cell image, which includes the complete cell nucleus, cytoplasmic structure, and surrounding background environment;

[0030] The main image extracted by the classification branch at the main image scale is the cell nucleus and part of its surrounding environment.

[0031] Optionally, the process of extracting the subject image through the classification branch of the subject image scale includes the following steps:

[0032] The original image is fed into the image segmentation model SAM to segment out the cell nucleus mask. The cell nucleus region in the cell nucleus mask is set to white and assigned a value of 1, while the rest of the image except for the cell nucleus region is set to black and assigned a value of 0.

[0033] Calculate the maximum connected region of the white area based on the pixels;

[0034] The rectangle covered by the calculated maximum connected region is calculated to obtain the main body positioning frame;

[0035] The original image is cropped based on the coordinates of the main subject positioning box and enlarged to the same size as the original image to obtain the final main subject image.

[0036] Optionally, obtaining the bone marrow cell dataset and constructing a small-sample training and test set includes the following steps:

[0037] Images of some bone marrow cells were collected, and each bone marrow cell was labeled with a fine-grained category.

[0038] The dataset is randomly divided into a training set and a test set in an 8:2 ratio.

[0039] In the training set, 1, 2, 4, 8, and 16 images of each category are randomly selected as the training dataset for few-shot classification.

[0040] All image formats are standardized to obtain the final few-sample training and testing sets. The image formats include image file types and image sizes.

[0041] Another aspect of this invention provides a bone marrow cell small sample classification system, comprising:

[0042] The first module is used to obtain a bone marrow cell dataset and construct a small-sample training and test set.

[0043] The second module is used to establish a multi-scale, multi-modal, few-shot classification network model based on text multi-source generation selection; wherein, the few-shot classification network model applies a histopathology pre-trained model, and the few-shot classification network model adds a multi-source text generation selection module; the multi-source text generation selection module is used to generate category text descriptions for input to the network;

[0044] The third module is used to train the few-sample classification network model based on the few-sample training set to obtain a trained bone marrow cell classification network model.

[0045] The fourth module is used to input the tested bone marrow cell images into the bone marrow cell classification network model to obtain the predicted category of the bone marrow cell images, thereby achieving bone marrow cell image classification under the condition of few-sample training.

[0046] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0047] The memory is used to store programs;

[0048] The processor executes the program to implement the method described above.

[0049] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0050] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0051] This invention obtains a bone marrow cell dataset, constructs a few-shot training set and a test set, and establishes a multi-scale, multi-modal few-shot classification network model based on multi-source text generation selection. The few-shot classification network model utilizes a histopathology pre-trained model and incorporates a multi-source text generation selection module. This module generates category text descriptions for input to the network. The few-shot classification network model is trained using the few-shot training set to obtain a trained bone marrow cell classification network model. Test bone marrow cell images are input into the model to obtain predicted categories, achieving bone marrow cell image classification under few-shot training conditions. This invention selects a histopathology pre-trained model for bone marrow cell classification and designs a multi-source text generation selection module to standardize the semantic descriptions of various bone marrow cell categories, thereby improving the accuracy of few-shot bone marrow cell classification. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1A flowchart illustrating the overall steps of an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of an implementation environment provided for an embodiment of the present invention;

[0055] Figure 3 This is an implementation flowchart of the present invention in a specific scenario.

[0056] Figure 4 This is a structural diagram of a multi-scale, multi-modal, few-shot classification network model based on text multi-source generation selection provided in an embodiment of the present invention;

[0057] Figure 5 This is a structural diagram of the multi-source text generation and selection module in the few-shot classification network model provided in this embodiment of the invention.

[0058] Figure 6 A schematic diagram of the structure of a system provided in an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0061] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0062] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0064] The bone marrow cell small sample classification method provided in this invention relates to the field of computer vision technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the bone marrow cell small sample classification method, but is not limited to the above forms.

[0065] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0066] To address the problems existing in related technologies, this invention provides a method for classifying bone marrow cells with few samples. It selects a histopathological pre-trained model for bone marrow cell classification to improve overall classification accuracy; designs a multi-source text generation and selection module to standardize the semantic descriptions of various bone marrow cell categories; and applies a multi-scale approach, combining overall and subject images for classification, focusing on both overall and fine-grained features, thereby improving the accuracy of bone marrow cell classification with few samples.

[0067] Specifically, such as Figure 1 As shown, a method for classifying bone marrow cells in a small sample according to the present invention includes the following steps:

[0068] S100. Obtain the bone marrow cell dataset and construct a small-sample training and test set.

[0069] S200. Establish a multi-scale, multi-modal, few-shot classification network model based on text multi-source generation selection; wherein, the few-shot classification network model applies a histopathology pre-trained model, and the few-shot classification network model adds a multi-source text generation selection module; the multi-source text generation selection module is used to generate category text descriptions for input networks;

[0070] S300. Train the few-sample classification network model based on the few-sample training set to obtain a trained bone marrow cell classification network model.

[0071] S400. Input the tested bone marrow cell image into the bone marrow cell classification network model to obtain the predicted category of the bone marrow cell image, thereby achieving bone marrow cell image classification under the condition of few-sample training.

[0072] Optionally, the few-shot classification network model includes a text feature extractor, an image feature extractor, and an image-text similarity score calculation module;

[0073] The text feature extractor is used to extract text features using a text-transformer model;

[0074] The image feature extractor is used to extract image features using the ViT-B16 model;

[0075] The image-text similarity score calculation module is used to obtain text and image similarity scores using the feature dot product method;

[0076] The multi-source text generation selection module includes a classification branch at the original image scale and a classification branch at the subject image scale. The subject image is obtained by localization and segmentation of the original image. The classification branch at the original image scale is used to extract the overall features of the image, while the classification branch at the subject image scale is used to extract the detailed features of the image.

[0077] Optionally, the category text description generated by the multi-source text generation selection module includes single text content generated by the GPT-3 large language model, text and image content generated by the GPT-4 large language model, and supplementary content of professional medical knowledge;

[0078] The multi-source text generation and selection module adaptively selects text that matches the image as the category text description input to the network through a selection function;

[0079] The expression for the selection function is:

[0080]

[0081] Where F represents the selection function; α and β represent proportionality coefficients; C i Normalized image features representing each category; T j Represents each text feature; cos(T) i ,T j The expression represents calculating the cosine similarity between two different texts.

[0082] Optionally, the establishment of a multi-scale, multi-modal, few-shot classification network model based on text multi-source generation selection includes the following steps:

[0083] The original image and corresponding text are fed into a multimodal network for classification, and prediction results are obtained from the global perspective of the cell image.

[0084] The original image is cropped and enlarged to obtain the main image;

[0085] The main image and corresponding text are fed into the multimodal network for classification, and prediction results are obtained from the details of the cell image.

[0086] By combining the classification branches at the original image scale and the classification branches at the subject image scale, the final classification result is obtained through ensemble learning in the case of few-sample classification.

[0087] Optionally, the overall image extracted from the classification branch at the original image scale is the original cell image, which includes the complete cell nucleus, cytoplasmic structure, and surrounding background environment;

[0088] The main image extracted by the classification branch at the main image scale is the cell nucleus and part of its surrounding environment.

[0089] Optionally, the process of extracting the subject image through the classification branch of the subject image scale includes the following steps:

[0090] The original image is fed into the image segmentation model SAM to segment out the cell nucleus mask. The cell nucleus region in the cell nucleus mask is set to white and assigned a value of 1, while the rest of the image except for the cell nucleus region is set to black and assigned a value of 0.

[0091] Calculate the maximum connected region of the white area based on the pixels;

[0092] The rectangle covered by the calculated maximum connected region is calculated to obtain the main body positioning frame;

[0093] The original image is cropped based on the coordinates of the main subject positioning box and enlarged to the same size as the original image to obtain the final main subject image.

[0094] Optionally, obtaining the bone marrow cell dataset and constructing a small-sample training and test set includes the following steps:

[0095] Images of some bone marrow cells were collected, and each bone marrow cell was labeled with a fine-grained category.

[0096] The dataset is randomly divided into a training set and a test set in an 8:2 ratio.

[0097] In the training set, 1, 2, 4, 8, and 16 images of each category are randomly selected as the training dataset for few-shot classification.

[0098] All image formats are standardized to obtain the final few-sample training and testing sets. The image formats include image file types and image sizes.

[0099] The specific implementation process of this invention will be described in detail below using a specific application scenario as an example:

[0100] like Figure 2 As shown, the training environment for the bone marrow cell small sample classification method provided in this embodiment of the invention can be applied to, for example... Figure 2 In the application environment shown in 202, labeled bone marrow cell few-sample images are input into computer device 201. Computer device 201 first processes the dataset to obtain a processed few-sample dataset. Then, computer device 201 trains a multi-scale, multi-modal few-sample classification network model based on text multi-source generation selection using the training set, obtaining a trained multi-scale, multi-modal few-sample classification network based on text multi-source generation selection. Random bone marrow cell images from the test set are input into the trained multi-scale, multi-modal few-sample classification network based on text multi-source generation selection in computer device 201 to obtain the cell category prediction results corresponding to the bone marrow cell images, thus achieving cell classification. The computer device in 202 can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers.

[0101] In one embodiment, such as Figure 3 As shown, a method for classifying bone marrow cells in small samples is provided, including the following steps:

[0102] S201. Obtain the dataset and process it.

[0103] Optionally, step S201 specifically includes:

[0104] S2011. Collect and label the dataset.

[0105] In accordance with clinical laboratory standards, this embodiment collected bone marrow smears stained with Wright's stain from some newly diagnosed leukemia patients and bone marrow transplant donors at the hospital. All smears were collected from the subjects during their individual disease examinations at the hospital and were already included in the hospital's database. Subjects were not specifically recruited for this study. All data has been anonymized, does not contain the subjects' identity information, and no follow-up was conducted on the patients.

[0106] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0107] To construct a fine-grained classification dataset for bone marrow cells, this embodiment cropped individual cells from the original images and labeled each cropped bone marrow cell with its category. Based on the diagnostic criteria for acute leukemia and the developmental evolution of bone marrow cells, and taking into account the actual number of cells collected, this embodiment ultimately constructed a bone marrow cell dataset with 19 categories, totaling 22,201 cell images.

[0108] S2012. Preprocess the dataset to obtain a preprocessed small sample dataset.

[0109] Preprocessing of the dataset: The image format of all bone marrow cells in the dataset was standardized. This standardization included uniform image file type and image size. The image type was JPG, and the uniform image size was 224×224.

[0110] The dataset was randomly divided into a total training set and a test set in an 8:2 ratio. In the total training set, 1, 2, 4, 8, and 16 images of each category were randomly selected as the training dataset for few-shot classification.

[0111] S202. Establish a multi-scale, multi-modal, few-shot classification network model based on text multi-source generation selection.

[0112] like Figure 4As shown, the multi-scale, multimodal, few-shot classification network model based on text multi-source generation selection is built on the multimodal large model CLIP.

[0113] The CLIP model consists of three parts: a text feature extractor, an image feature extractor, and an image-text similarity score calculation. The text feature extractor uses a text-transformer to extract text features; the image feature extractor uses ViT-B16 to extract image features; and the text and image similarity scores are obtained using feature dot product.

[0114] Using the pre-trained Quilt-Net model trained on histopathological images and text, freeze the text feature extractor and image feature extractor, and train only the network's classifier.

[0115] This embodiment of the few-shot classification network model first adds a text multi-source generation and selection module to generate broader and more accurate text descriptions, such as... Figure 5 As shown, the structure includes two parts: multi-source text generation and text sorting selection.

[0116] When generating text, three approaches are selected: GPT-3 single text generation, professional medical knowledge supplementation, and GPT-4 text and image generation to generate a wide range of text descriptions.

[0117] The input questions to GPT-3 are mainly divided into general description and specific description. In the general description, GPT-3 is expected to provide a comprehensive description of a certain type of cell. In the specific description, GPT-3 is expected to provide detailed descriptions, such as the specific color, size, and texture characteristics of a certain type of cell. In professional medicine, descriptions of various cell types are collected from medical laboratory atlases. In the input question of GPT-4, cell images from the dataset are input, and it is assumed that GPT-4 is a professional micropathologist who expects it to provide a visual description.

[0118] The text sorting selection section sorts text based on image-text similarity and adaptively selects the most matching text.

[0119] First, the image features and text features are multiplied pairwise to obtain the corresponding image-text features. Then, the features of images of the same category are added together according to the category of the image features to obtain the corresponding total category image-text features. Then, the elements in the matrix are normalized column by column, changing the original elements to their proportion in the 19 categories. At this point, the normalized image-text features of all texts for each category of images are obtained for subsequent sorting and selection.

[0120] Submodule optimization is used to select the final set, and the score function during selection is shown below, which also promotes the selection of a more discriminative and differentiated text set.

[0121]

[0122] Furthermore, the few-shot classification network model in this embodiment also applies the concept of multi-scale, with branches at both the original image and the subject image scales simultaneously focusing on global cell features and fine-grained features. The specific process is as follows:

[0123] The original image and corresponding text are fed into a multimodal network for classification, and prediction results are obtained from the global perspective of the cell image.

[0124] The original image is cropped and enlarged to obtain the main image, which mainly consists of the cell nucleus and its surrounding area;

[0125] The main image and corresponding text are fed into a multimodal network for classification, and prediction results are obtained from cell image details.

[0126] The overall image classification branch and the subject image classification branch are combined to obtain the final classification result through ensemble learning when classifying with few samples.

[0127] The overall image is the original cell image, containing the complete cell nucleus and cytoplasm structure, as well as a significant amount of surrounding background. The main image, consisting of the cell nucleus and a small portion of its surroundings, is derived from the original image after cropping and magnification. This image contains only the cell nucleus, allowing observation of chromatin aggregation within the nucleus and providing more detailed information, free from the influence of the surrounding environment. The specific acquisition process is as follows:

[0128] The original image is fed into the image segmentation model SAM. The segmentation model segments out the nucleus mask. The nucleus region in the nucleus mask is white and assigned a value of 1, while the rest is black and assigned a value of 0.

[0129] The central region, specifically the cell nucleus, is the largest region; therefore, the largest connected region of the white area is calculated based on the pixels.

[0130] The rectangle covered by the calculated maximum connected region is then calculated to obtain the main body positioning box.

[0131] The original image is cropped based on the coordinates of the subject's positioning box and then enlarged to the same size as the original image to obtain the final subject image.

[0132] S203. Using the bone marrow cell few-sample dataset, train the few-sample classification network model to obtain a trained few-sample classification network model.

[0133] The specific steps for training a few-shot classification network model in this embodiment are as follows:

[0134] (1) Network initialization: Import the Quit-Net pre-trained parameters and randomly initialize the parameters of the remaining network structures;

[0135] (2) Network parameters were set: the learning rate was set to 0.01, the number of samples per batch during training was 32, and the total training time was 50 epochs. The Adam optimizer was used, the momentum was set to 0.9, and the weight decay coefficient was 1e-3. In the linear prediction of the CLIP model, the linear classifier of sklearn was used, and multiple experiments were conducted to select the value of the C parameter in the function, and the best performance on the test set was recorded.

[0136] (3) Load the few-sample training dataset into the network.

[0137] (4) Preprocessing the training images: The images in the bone marrow cell small sample dataset are preprocessed, specifically by performing online data augmentation on the images through three methods: random vertical flipping, random rotation, and color perturbation.

[0138] (5) Train the model: Figure 3 The few-shot classification network shown is trained iteratively, specifically as follows:

[0139] Input image is I total Selected text T total At this time, I total T total The input data to be fed into the network is processed by a text feature extractor to obtain text features. Image features are obtained after image feature extraction. Calculate the feature similarity score G between text features and image features. total The calculation formula is shown below.

[0140]

[0141] The feature similarity score G total The data is fed into a classifier to classify 19 categories. The classifier learns the weights for each category. After training, the classifier provides the predicted probability P for each category. total The loss function is calculated by combining the predicted category probability with the true image label y. The formula for calculating the loss function is shown below.

[0142] L total =CE(P total ,y)

[0143] The main body of the input image is the original image I. total Image I of the main body obtained after segmentation and processing local Selected text T local Text features are obtained through a text feature extractor. Image features are obtained after image feature extraction. Calculate the feature similarity score G between text features and image features. local The calculation formula is shown below.

[0144]

[0145] The feature similarity score G local The data is fed into a classifier to classify 19 categories. The classifier is configured identically to the overall part, and the predicted category probabilities P are obtained. local The loss function is calculated by combining the predicted category probability with the true image label y. The formula for calculating the loss function is shown below, where CE is the cross-entropy loss.

[0146] L local =CE(P local ,y)

[0147] The overall loss function is shown below, where α and β are hyperparameters for different branches.

[0148] L=αL total +βL local =αCE(P total ,y)+βCE(P local ,y) (4-5)

[0149] S204. Input the bone marrow cell images from the test set into the trained multi-scale, multimodal, few-shot classification network model based on text multi-source generation selection to obtain the predicted category of the bone marrow cell images and achieve bone marrow cell image classification.

[0150] The bone marrow cell images from the input test set are used to extract features and calculate the predicted probability distributions P for the two branches. total P local The total predicted probability P is P total P local The sum of these probabilities is then used to select the position with the highest category prediction probability P as the final predicted category for the test image.

[0151] This embodiment conducts experiments on a test bone marrow cell dataset. The trained few-shot classification network model is used to predict the class of random bone marrow cell images, and the classification performance of the few-shot classification network is quantitatively measured by the accuracy of the class prediction. The accuracy is calculated as the proportion of correctly predicted images out of all test bone marrow cell images, explicitly representing the network's prediction performance for the target bone marrow cell images.

[0152] In the experiment, a consistent pre-trained model and feature extractor were used, and 10 random selections of the few-sample dataset were performed. The average results on the test set were recorded. The experimental results are shown in Table 1, which describes the classification accuracy of bone marrow cells in few samples.

[0153] Table 1

[0154] Model 1shot 2shot 4shot 8shot 16shot CLIP linear mapping 18.570 24.663 30.126 35.049 42.963 This embodiment uses a few-shot classification network model. 25.458 29.586 35.049 38.214 43.085

[0155] As can be seen from Table 1, this embodiment performs better than CLIP linear mapping. Comparing the experimental results, it was found that the network proposed in this invention can improve the accuracy of classification with few samples. The improvement effect is the best when the sample size is smaller, such as 1 shot. When the amount of data increases, the improvement in model accuracy is limited.

[0156] like Figure 6 As shown, this embodiment provides a bone marrow cell small sample classification system. The system includes a dataset acquisition unit 601, a model building unit 602, a model training unit 603, and a classification prediction unit 604. The specific functions of each unit are as follows:

[0157] The dataset acquisition unit 601 is used to acquire and process the dataset, which includes a few-sample training dataset and a test dataset.

[0158] The model building unit 602 is used to build a multi-scale, multi-modal, few-shot classification network model based on text multi-source generation and selection. The few-shot classification network model is based on the CLIP network, applies a histopathology pre-trained model, and adds a text multi-source generation and selection module, including classification branches at two scales: the original image and the subject image.

[0159] The model training unit 603 is used to train the few-sample classification network model using a few-sample dataset of bone marrow cells to obtain a trained few-sample classification network model.

[0160] The classification prediction unit 604 is used to input bone marrow cell images from the test set into a trained bone marrow cell few-sample classification network model to obtain the predicted category of the bone marrow cell images, thereby achieving bone marrow cell image classification.

[0161] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0162] This embodiment provides a computer device, which can be a computer, such as... Figure 7As shown, the processor 702, memory, input device 703, display device 704, and network interface 705 are connected via system bus 701. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 706 and internal memory 707. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory 707 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the processor 702 executes the computer programs stored in the memory, it implements the aforementioned bone marrow cell classification method.

[0163] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0164] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for classifying small samples of bone marrow cells.

[0165] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0166] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0167] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0168] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0169] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0171] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0172] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0173] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0174] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0175] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for classifying bone marrow cells in small samples, characterized in that, Includes the following steps: Obtain a bone marrow cell dataset and construct a small-sample training and test set; A multi-scale, multi-modal, few-shot classification network model based on multi-source text generation selection is established. The few-shot classification network model utilizes a histopathology pre-trained model and incorporates a multi-source text generation selection module. This module generates category text descriptions for input to the network. The few-sample classification network model is trained using the few-sample training set to obtain a trained bone marrow cell classification network model. The tested bone marrow cell images are input into the bone marrow cell classification network model to obtain the predicted category of the bone marrow cell images, thus achieving bone marrow cell image classification under the condition of few sample training. The few-shot classification network model includes a text feature extractor, an image feature extractor, and an image-text similarity score calculation module; The text feature extractor is used to extract text features using a text-transformer model; The image feature extractor is used to extract image features using the ViT-B16 model; The image-text similarity score calculation module is used to obtain text and image similarity scores using the feature dot product method; The multi-scale, multimodal, few-shot classification network model includes a classification branch at the original image scale and a classification branch at the subject image scale, wherein the subject image is obtained by localization and segmentation of the original image; the classification branch at the original image scale is used to extract the overall features of the image; and the classification branch at the subject image scale is used to extract the detailed features of the image. The category text descriptions generated by the multi-source text generation and selection module include single text content generated by the GPT-3 large language model, text and image content generated by the GPT-4 large language model, and supplementary content of professional medical knowledge; The multi-source text generation and selection module adaptively selects text that matches the image as the category text description input to the network through a selection function; The expression for the selection function is: in, Represents the selection function; , Represents the proportionality coefficient; Normalized image features representing each category; Represents each text feature; This represents calculating the cosine similarity between two different texts; The establishment of a multi-scale, multi-modal, few-shot classification network model based on text multi-source generation selection includes the following steps: The original image and corresponding text are fed into a multimodal network for classification, and prediction results are obtained from the global perspective of the cell image. The original image is cropped and enlarged to obtain the main image; The main image and corresponding text are fed into the multimodal network for classification, and prediction results are obtained from the details of the cell image. By combining the classification branches at the original image scale and the classification branches at the subject image scale, the final classification result is obtained through ensemble learning in the case of few-sample classification.

2. The method for classifying bone marrow cells in small samples according to claim 1, characterized in that, The overall image extracted from the classification branch at the original image scale is the original cell image, which contains the complete cell nucleus, cytoplasmic structure, and surrounding background environment. The main image extracted by the classification branch at the main image scale is the cell nucleus and part of its surrounding environment.

3. The method for classifying bone marrow cells in small samples according to claim 2, characterized in that, The process of extracting the subject image through the classification branch at the subject image scale includes the following steps: The original image is fed into the image segmentation model SAM to segment out the cell nucleus mask. The cell nucleus region in the cell nucleus mask is set to white and assigned a value of 1, while the rest of the image except for the cell nucleus region is set to black and assigned a value of 0. Calculate the maximum connected region of the white area based on the pixels; The rectangle covered by the calculated maximum connected region is calculated to obtain the main body positioning frame; The original image is cropped based on the coordinates of the main subject positioning box and enlarged to the same size as the original image to obtain the final main subject image.

4. A method for classifying bone marrow cells in small samples according to any one of claims 1-3, characterized in that, The process of obtaining the bone marrow cell dataset and constructing a small-sample training and test set includes the following steps: Images of some bone marrow cells were collected, and each bone marrow cell was labeled with a fine-grained category. The dataset is randomly divided into a training set and a test set in an 8:2 ratio. In the training set, 1, 2, 4, 8, and 16 images of each category are randomly selected as the training dataset for few-shot classification. All image formats are standardized to obtain the final few-sample training and testing sets. The image formats include image file types and image sizes.

5. A system for implementing the bone marrow cell small sample classification method as described in any one of claims 1-4, characterized in that, include: The first module is used to obtain a bone marrow cell dataset and construct a small-sample training and test set. The second module is used to establish a multi-scale, multi-modal, few-shot classification network model based on text multi-source generation selection; wherein, the few-shot classification network model applies a histopathology pre-trained model, and the few-shot classification network model adds a multi-source text generation selection module; the multi-source text generation selection module is used to generate category text descriptions for input to the network; The third module is used to train the few-sample classification network model based on the few-sample training set to obtain a trained bone marrow cell classification network model. The fourth module is used to input the tested bone marrow cell images into the bone marrow cell classification network model to obtain the predicted category of the bone marrow cell images, thereby achieving bone marrow cell image classification under the condition of few-sample training.

6. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 4.

Images