Pathological image cell classification method based on visual language large model and prompt learning

Abstract

The invention discloses a pathological image cell classification method based on a visual language large model and prompt learning, and the method comprises the steps: constructing a cell classification frame through employing an instance perception prompt learning module, a prompt conditionalization module, a local visual text matching module, and an image coding and decoding network, and specifically comprises the steps: in the instance perception prompt learning module, carrying out the prompt conditionalization of a local visual text; respectively extracting features of an input image and a text by utilizing a visual language large model so as to fully mine instance specific information and obtain instance perception text features; inputting the text features into a prompt conditional module to modulate image features, and inputting the text features into a local visual text matching module to enhance semantic association with the image features; constructing a network overall loss function; and iteratively training the classification network by using an optimizer. According to the method, the technical problems of incomplete prompt and semantic ambiguity of cell characteristics caused by large instance difference in the prior art are solved, so that the classification performance of the model in multiple data sets can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image cell classification, and specifically is a pathological image cell classification method based on a visual language large model and prompt learning. Background Art

[0002] As an important part of pathological image analysis, automated cell classification can assist pathologists in disease diagnosis, thereby improving the efficiency and accuracy of pathological diagnosis. It has important research significance and broad application prospects. In recent years, with the continuous accumulation of data resources in the field of cell classification, pathological image cell classification methods have achieved certain success. However, the inherent morphological and staining heterogeneity of pathological image cells poses a severe challenge to traditional methods trained with a single dataset, resulting in limited performance of such cell classification models, especially in terms of their generalization ability. In view of this, the current research community is keen to explore the construction of a general model that can be generalized across datasets, aiming to enhance the universality and robustness of the model by integrating multiple data sources. Unfortunately, this strategy has always been plagued by the problem of differences in annotation systems between datasets caused by different sources of pathological tissues, which directly leads to semantic inconsistencies in the training and generalization stages of the model, thus becoming a key problem hindering the development of general models.

[0003] To address the challenge of differences between datasets, a series of solutions have been proposed in the research community, such as multi-task learning strategies (Graham, S.; Vu, QD; Jahanifar, M.; Raza, SEA; Minhas, F.; Snead, D.; and Rajpoot, N. One model is all you need: Multi-task learningenables simultaneous histology image segmentation and classification. Medical Image Analysis, 2023, 83: 102685.) and label integration strategies (Zhang, W.; Zhang, J.; Wang,X.; Yang, S.; Huang, J.; Yang, W.; Wang, W.; and Han, X. Merging nucleusdatasets by correlation-based cross-training. Medical Image Analysis, 2023,84: 102705.), which aim to extract more generalized image features and make full use of the category semantic information of different annotation systems. However, these existing strategies often face the limitation of surging computing resources or fail to realize an end-to-end cell classification framework for pathological images. Recently, a prompt-based integrated universal model, UniCell (Huang, J.; Li,H.; Wan, X.; and Li, G. UniCell: Universal Cell Nucleus Classification viaPrompt Learning. In Proceedings of the AAAI Conference on ArtificialIntelligence, 2024, 38(3): 2348-2356.) was proposed. The core design concept of this framework is to discover and utilize the cell knowledge shared between datasets, while avoiding the common redundant training problem in multi-source data integration processing. Specifically, this universal model first adopts advanced learnable cell category text prompt technology to unify the annotation systems of different datasets, and then selectively combines category text prompts to generate dataset prompts according to the characteristics of different dataset annotation systems, thereby improving the generalization performance of the cell classification model.

[0004] However, although the above methods have made remarkable progress by representing cell semantic information, cell classification using textual prompts still faces two major challenges, namely instance differences and semantic ambiguity. Specifically, pathological image instances often show significant differences in cell category composition within and between datasets. Therefore, dataset prompt methods that rely on fixed patterns cannot flexibly utilize contextual information from rich image instances, resulting in incomplete or even inappropriate prompts. At the same time, the semantic relationship between many cells in a single image instance and their true categories is also extremely complex. Therefore, the above-mentioned prompt-based general cell classification model is prone to produce semantically ambiguous cell features, and it is difficult to learn generalizable cell representations to accurately identify different cell categories from multiple datasets. Summary of the invention

[0005] The present invention aims to address the deficiencies of the above-mentioned prior art and proposes a pathological image cell classification method based on a large visual language model and prompt learning, in order to solve the technical problems of incomplete prompts and semantic ambiguity of cell features caused by image instance differences in the prior art, thereby improving the cell classification performance of the model on multiple data sets.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0007] The method for classifying pathological image cells based on a large visual language model and prompt learning is characterized in that it comprises the following steps:

[0008] Step 1: Get the annotated After standardization and normalization of the pathological image dataset, the preprocessed Pathology image datasets; Representative The total number of preprocessed image instances corresponding to the data set; The pathological image dataset Pathological image , The corresponding real annotation is denoted as ;and ,in, Indicates The real labeling of the cell center coordinates, Indicates The true annotations of cell categories, and ∈{1,2,…,K}; K represents the total number of cell categories; P represents the total number of cells of all categories, for The true label of the cell category is The total number of cells;

[0009] Get Corresponding instance prompt ;

[0010] Get The corresponding category hints for all cell categories in the pathological image dataset ;

[0011] Step 2: Construct a network based on the visual language model and prompt learning cell classification, including: instance-aware prompt learning module, prompt conditioning module, local visual text matching module and the first Image encoding and decoding network ;

[0012] Step 2.1: The instance-aware prompt learning module is based on , as well as ,get Instance-aware categorical text features ;in, represents the instance-aware category text feature corresponding to the k-th cell category;

[0013] Step 2.2, Image encoding and decoding network The encoder pair in Process and obtain Image features ;

[0014] Step 2.3, the prompt conditioning module is based on as well as ,get Cue-conditioned image features ;

[0015] Step 2.4, Image encoding and decoding network The decoder pair in Process and obtain local cell image features ;

[0016] Step 2.5, Image encoding and decoding network The classification head pair in Process, generate The prediction results , including: The predicted center coordinates of the cells and Predicted class of cells ; , represents the total number of predicted cells;

[0017] Step 2.6: According to Image encoding and decoding network right The prediction results , using formula (1) to obtain the predicted center coordinates and categories of the predicted cells matching the real cells:

[0018] (1)

[0019] In formula (1), represents the linear sum distribution function; and Respectively represent Image encoding and decoding network The obtained The image contains The predicted center coordinates and categories of the predicted cells that match the real cells;

[0020] Step 3: Construct loss function:

[0021] Step 3.1: Use formula (2) to construct The classification loss :

[0022] (2)

[0023] In formula (2), represents the balanced cross entropy loss function; represents the mean absolute error loss function;

[0024] Step 3.2: Use formula (3) to construct Hungarian matching loss :

[0025] (3)

[0026] In formula (3), Indicates Image encoding and decoding network The obtained The prediction results of the image The true cell category corresponding to the unmatched predicted cells; , and Represents 3 loss weight coefficients;

[0027] Step 3.3: The local visual text matching module uses formula (4) to construct the first The local visual text matching loss function corresponding to the dataset :

[0028] (4)

[0029] In formula (4), represents the feature similarity measurement function, is the temperature parameter;

[0030] Step 3.4: Use formula (5) to construct The overall loss corresponding to the dataset :

[0031] (5)

[0032] In formula (5), , and They are , and The weight coefficient of

[0033] Step 4: Use the Adam optimizer to train the pathological image cell classification network based on the visual language model and prompt learning, and minimize With this goal in mind, the pathological image cell classification network based on the visual language large model and prompt learning is iterated, and after several rounds of training, the optimal classification network is selected to identify cells in the image.

[0034] The pathological image cell classification network based on the visual language large model and prompt learning described in the present invention is also characterized in that the step 2.1 includes:

[0035] Step 2.1.1. Definition The learnable tokens are and The learnable tokens are ;

[0036] Step 2.1.2: Generate using formula (6) A refinement of the category text tokens :

[0037] (6)

[0038] In formula (6), represents a set of parameters to be learned that control the scale of the mapping learnable tokens; represents a mapping function; Represents element-by-element multiplication operation;

[0039] Step 2.1.3: Use the text encoder to and Encode and get Example text embedding and the text embedding matrix of K cell categories ;in, Represents the category text embedding of the cell category with the actual label k;

[0040] Step 2.1.4: Use the visual encoder to Encode and get Image embedding , thus As a query to the Transformer decoder, As the key and value of the Transformer decoder, and the Transformer decoder Calibrate and generate Visual domain cue embedding ;

[0041] Step 2.1.5: Generate using formula (7) Enhanced instance text embedding :

[0042] (7)

[0043] In formula (7), represents a set of control visual domain cue embeddings The parameters to be learned of the scale;

[0044] Step 2.1.6: Use formula (8) to get Specific instance text features :

[0045] (8)

[0046] In formula (8), represents the specificity mask; represents the masked cross attention operation; Representation layer normalization function; represents a feed-forward neural network;

[0047] Step 2.1.7: and Splice to get Instance-aware categorical text features .

[0048] Further, the step 2.3 includes:

[0049] Step 2.3.1: Use formula (9) to get The two conditional vectors and :

[0050] (9)

[0051] In formula (9), and Represents 2 different multi-layer perceptrons;

[0052] Step 2.3.2: Use formula (10) to Perform conditional adjustments to obtain prompt conditional image features :

[0053] (10) .

[0054] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the pathological image cell classification method, and the processor is configured to execute the program stored in the memory.

[0055] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the pathological image cell classification method when the computer program is executed by a processor.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. The present invention proposes an instance-aware prompt learning module; this module uses a large visual language model and prompt learning technology to not only fully utilize the shared knowledge from multiple data sets, but also flexibly captures rich contextual information from different pathological image instances, avoiding incomplete and inappropriate prompts caused by differences in image instances.

[0058] 2. The present invention proposes a prompt conditioning module; this module uses the linear modulation technology at the feature level to constrain the semantic information of the corresponding cell category to constrain the cell image features, so that the features from the visual and language modalities can be fully interactively integrated, making the subsequent language supervision optimization process more effective.

[0059] 3. The present invention proposes a local visual-text matching loss; this module explicitly utilizes language-guided supervision information to enhance the semantic association between local cell images and corresponding category text prompts, avoiding the problem of non-generalization of cell features due to semantic ambiguity, thereby improving the cell classification performance of the model on multiple data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] In this embodiment, a pathological image cell classification method based on a large visual language model and prompt learning is provided. Figure 1 As shown, it mainly includes the following steps:

[0063] Step 1: Get the annotated After standardization and normalization of the pathological image dataset, the preprocessed Pathology image datasets;

[0064] Step 1.1. In this example, the cell classification datasets CoNSeP, MoNuSAC, Lizard and OCELOT are used. For any of the above datasets, this example randomly divides them into two subsets, uses one of the subsets as the test set each time, and randomly selects 20% of the data from the other subset as the validation set, and the rest as the training set, and only uses the image data in the training set for training.

[0065] Step 1.2: Augment the training set of the cell classification dataset by rotating it by 0°, 90°, 180°, and 270°, as well as vertical and horizontal flipping. The rotation and flipping operations are combined, so that the amount of training data is increased by 8 times. Then, each pathological image is standardized and normalized using equations (1) and (2). The specific calculation method is as follows:

[0066] (1)

[0067] (2)

[0068] In formula (1) and formula (2), For pathological images The standardized results, and Represent the mean and variance of all pathological image pixels in the dataset, is the result after image normalization.

[0069] Step 1.3: The standardized and normalized data set is used as the preprocessed data set; The pathological image dataset Pathological image , The corresponding real annotation is denoted as ;and , among which The actual annotation of the cell center coordinates is denoted by , No. The true label of each cell category is denoted by ∈{1,2,…,K}; K represents the total number of cell categories; P represents the total number of cells of all categories, for The actual label of the cell category is The total number of cells;

[0070] Get Corresponding instance prompt ; Get The corresponding category hints of the overall cell categories in the pathological image dataset ;

[0071] Step 2: Construct a network based on the visual language model and prompt learning cell classification, including: instance-aware prompt learning module, prompt conditioning module, local visual text matching module and the first Image encoding and decoding network ;

[0072] Step 2.1: Instance-aware cue learning module based on , as well as ,get Instance-aware categorical text features ,in, represents the instance-aware category text feature corresponding to the k-th cell category;

[0073] Step 2.1.1, obtain and the instance text description "A H&E image of [Cancer_type]", where Cancer_type represents a disease type, and the kth cell category text description [Category_k], where Category_k represents the cell category name; learnable tokens are introduced in the prefix part of the instance text description and the category text description respectively and , thus forming an instance text prompt and category text hints ;

[0074] Step 2.1.2: Use formula (3) to promote arrive Information sharing, updating the category text token to generate a refined category text token :

[0075] (3)

[0076] In formula (3), represents a set of parameters to be learned that control the scale of the mapping learnable tokens; represents a mapping function; Represents an element-by-element multiplication operation; Combined with the category text description ;Will Combined with the instance text description ;

[0077] Step 2.1.3: Use the text encoder to and Encode and get Example text embedding and the text embedding matrix of K cell categories ;in, Represents the category text embedding of the cell category with the actual label k;

[0078] Step 2.1.4: Use the visual encoder to Encode and get Image embedding , thus As a query to the Transformer decoder, As the key and value of the Transformer decoder, and use formula (4) to Calibrate and generate Visual domain cue embedding :

[0079] (4)

[0080] Step 2.1.5: Use formula (5) to embed the visual domain cue into and instance text embedding Perform feature fusion to generate Enhanced instance text embedding :

[0081] (5)

[0082] In formula (5), represents a set of control visual domain cue embeddings The parameters to be learned of the scale;

[0083] Step 2.1.6: Use formula (6) to embed the enhanced instance text And K category text embedding matrices Perform dataset-specific feature fusion to obtain dataset-specific instance text features :

[0084] (6)

[0085] In formula (6), represents a specificity mask; represents the masked cross attention operation; Representation layer normalization function; represents a feed-forward neural network;

[0086] Step 2.1.7: and Splice to get Instance-aware categorical text features .

[0087] Step 2.2, Image encoding and decoding network The encoder pair in Process and obtain Image features ;

[0088] In this embodiment, the encoder consists of a Swin-Transformer feature extractor and three Transformer encoders;

[0089] Step 2.3: Prompt the conditional module based on as well as ,get Cue-conditioned image features ;

[0090] Step 2.3.1: Use formula (7) to Perform mapping processing and obtain 2 conditional vectors and :

[0091] (7)

[0092] In formula (7), and Represents 2 different multi-layer perceptrons;

[0093] Step 2.3.2: Use formula (8) to analyze the image features Perform conditional adjustments to obtain prompt conditional image features :

[0094] (8)

[0095] Step 2.4, Image encoding and decoding network The decoder pair in Process and obtain local cell image features ; In this embodiment, the decoder is composed of multiple Transformer decoders;

[0096] Step 2.5, Image encoding and decoding network The classification head pair in Process, generate The prediction results ,Include: prediction cells, The predicted center coordinates of the cells and Predicted class of cells ;

[0097] The classification head consists of two feedforward neural networks, and its specific network structure information is shown in Table 1; FFN1 is the regression branch, and FFN2 is the classification branch; Indicates The number of cell types contained in the dataset;

[0098] Table 1: Classification head network structure information

[0099]

[0100] Step 2.6: According to Image encoding and decoding network right The prediction results , using formula (1) to obtain the predicted center coordinates and categories of the predicted cells matching the real cells:

[0101] (1)

[0102] In formula (1), represents the linear sum distribution function; and Respectively represent Image encoding and decoding network The obtained The image contains The predicted center coordinates and categories of the predicted cells that match the real cells;

[0103] Step 3: Construct loss function:

[0104] Step 3.1: Use formula (10) to construct The classification loss :

[0105] (10)

[0106] In formula (10), express The real labeling of cells in the Represents the image encoding and decoding network right The prediction results; represents the balanced cross entropy loss function; represents the mean absolute error loss function;

[0107] Step 3.2: Use formula (11) to construct Hungarian matching loss :

[0108] (3)

[0109] In formula (11), Indicates Image encoding and decoding network The obtained The prediction results of the image The true cell category corresponding to the unmatched predicted cells is the background category; , and Represents three loss weight coefficients. In this embodiment, , and Set to 5, 2 and 2 respectively;

[0110] Step 3.3: Use formula (12) to construct The local visual text matching loss function corresponding to the dataset :

[0111] (12)

[0112] In formula (12), represents the feature similarity measurement function, It is a temperature parameter;

[0113] Step 3.4: Use formula (13) to construct The overall loss corresponding to the dataset :

[0114] (13)

[0115] In formula (13), , and They are , and The weight coefficient of , and Set to 1, 0.1 and 1 respectively.

[0116] Step 4: Use the Adam optimizer to train the pathological image cell classification network based on the visual language model and prompt learning, and minimize With this goal in mind, the pathological image cell classification network based on the visual language large model and prompt learning is iterated, and after several rounds of training, the optimal classification network is selected to identify cells in the image.

[0117] Step 4.1, set the learning rate of the Adam optimizer to 10-4, and iterate the pathological image cell classification network based on the visual language large model and prompt learning;

[0118] Step 4.2: After each iteration, the cell classification network is verified on the validation set;

[0119] Step 4.2.1: In this embodiment, the threshold method is first used to convert the prediction results middle The cell category probability of the predicted position is compared with the preset threshold value 0.5. If the predicted category probability is greater than 0.5, the predicted result of the position is determined to be a cell; otherwise, the predicted result of the position is determined to be background;

[0120] Step 4.2.2: For the prediction results determined to be cells, in this embodiment, the Hungarian matching algorithm is used to assign a prediction result to each real cell based on the coordinate distance, so that the total distance between all real cells and the corresponding matched predicted cells is minimized;

[0121] Step 4.2.3, use Score( ) is used as the evaluation index, and the cell classification results are evaluated using formulas (14) to (16):

[0122] (14)

[0123] (15)

[0124] (16)

[0125] In formula (14) to formula (16), Yes Detection Score, It is Classification of categories Score, All the data in the dataset Average classification of cell classes Score. represents the number of true positive detections, Represents the number of false positives detected, Represents the number of false negative examples detected. Cell types, Further divided into: the number of true positive examples , the number of false positive examples in classification And the number of false negative examples .

[0126] In this embodiment, the corresponding true label region is established according to the center position coordinates of the cell, that is, a circular region with a radius of 6 pixels and the true position coordinates of each cell as the center. If there are more than one cell position prediction results in the true label region, the cell prediction result closest to the true position coordinates is regarded as a true positive example. If the cell position prediction result is not in any true label region, the prediction result is regarded as a false positive example; the remaining prediction results are false negative examples.

[0127] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0128] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

[0129] In addition, in order to quantitatively evaluate the performance of the proposed method, this embodiment demonstrates the comparison of the proposed method with HoverNet (Graham, S.; Vu, QD; Raza, SEA; Azam, A.; Tsang, YW; Kwak,JT; and Rajpoot, N. Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical ImageAnalysis, 2019, 58: 101563.), MCSpatNet (Abousamra, S.; Belinsky, D.; VanArnam, J.; Allard, F.; Yee, E.; Gupta, R.; Kurc, T.; Samaras, D.; Saltz, J. ;and Chen, C. Multi-class cell detection using spatial context representation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, 4005-4014.), Cellpose (Stringer, C.; Wang, T.; Michaelos, M.; andPachitariu, M. Cellpose: a generalist algorithm for cellular segmentation.Nature Methods, 2021, 18(1): 100–106.), Omnipose (Cutler, KJ; Stringer, C.;Lo, TW; Rappez, L.; Stroustrup, N.; Brook Peterson, S.; Wiggins, PA; and Mougous, JD Omnipose: a high-precision morphology-independent solution for bacterial cell segmentation. Nature Methods, 2022, 19(11): 1438–1448.), UperNet ConvNeXt (Liu, Z.; Mao, H.; Wu, C.-Y.; Feichtenhofer, C.; Darrell, T.; and Xie, S. A convnet for the 2020s. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 2022, 1197611986.), PGT (Huang, J.; Li, H.; Sun, W.; Wan, Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(3): 2348-2356.) and other single dataset and multi-dataset cell classification methods on the cell classification datasets CoNSeP, MoNuSAC, Lizard and OCELOT. The results are shown in Table 2. .

[0130] Table 2: Performance comparison of the proposed method with other single-dataset and multi-dataset cell classification methods on cell classification datasets CoNSeP, MoNuSAC, Lizard and OCELOT:

[0131]

[0132] From the above results, it can be seen that the method proposed by the present invention can perform cell classification more accurately, efficiently and automatically.

Claims

1. A pathological image cell classification method based on a large visual language model and prompt learning, characterized in that: The steps include: Step 1: Get the annotated After standardization and normalization of the pathological image dataset, the preprocessed Pathology image datasets; Representative The total number of preprocessed image instances corresponding to the data set; The pathological image dataset Pathological image , The corresponding real annotation is denoted as ;and ,in, Indicates The real labeling of the cell center coordinates, Indicates The true annotations of cell categories, and ∈{1,2,…,K}; K represents the total number of cell categories; P represents the total number of cells of all categories, for The true label of the cell category is The total number of cells; Get Corresponding instance prompt ; Get The corresponding category hints for all cell categories in the pathological image dataset ; Step 2: Construct a network based on the visual language model and prompt learning cell classification, including: instance-aware prompt learning module, prompt conditioning module, local visual text matching module and the first Image encoding and decoding network ; Step 2.1: The instance-aware prompt learning module is based on , as well as ,get Instance-aware categorical text features ;in, represents the instance-aware category text feature corresponding to the k-th cell category; Step 2.2, Image encoding and decoding network The encoder pair in Process and obtain Image features ; Step 2.3, the prompt conditioning module is based on as well as ,get Cue-conditioned image features ; Step 2.4, Image encoding and decoding network The decoder pair in Process and obtain local cell image features ; Step 2.5, Image encoding and decoding network The classification head pair in Process, generate The prediction results , including: The predicted center coordinates of the cells and Predicted class of cells ; , represents the total number of predicted cells; Step 2.6: According to Image encoding and decoding network right The prediction results , using formula (1) to obtain the predicted center coordinates and categories of the predicted cells matching the real cells: (1) In formula (1), represents the linear sum distribution function; and Respectively represent Image encoding and decoding network The obtained The image contains The predicted center coordinates and categories of the predicted cells that match the real cells; Step 3: Construct loss function: Step 3.1: Use formula (2) to construct The classification loss : (2) In formula (2), represents the balanced cross entropy loss function; represents the mean absolute error loss function; Step 3.2: Use formula (3) to construct Hungarian matching loss : (3) In formula (3), Indicates Image encoding and decoding network The obtained The prediction results of the image The true cell category corresponding to the unmatched predicted cells; , and Represents 3 loss weight coefficients; Step 3.3: The local visual text matching module uses formula (4) to construct the first The local visual text matching loss function corresponding to the dataset : (4) In formula (4), represents the feature similarity measurement function, is the temperature parameter; Step 3.4: Use formula (5) to construct The overall loss corresponding to the dataset : (5) In formula (5), , and They are , and The weight coefficient of Step 4: Use the Adam optimizer to train the pathological image cell classification network based on the visual language model and prompt learning, and minimize With this goal in mind, the pathological image cell classification network based on the visual language large model and prompt learning is iterated, and after several rounds of training, the optimal classification network is selected to identify cells in the image.

2. The pathological image cell classification network based on visual language large model and prompt learning according to claim 1 is characterized in that: The step 2.1 comprises: Step 2.1.

1. Definition The learnable tokens are and The learnable tokens are ; Step 2.1.2: Generate using formula (6) A refinement of the category text tokens : (6) In formula (6), represents a set of parameters to be learned that control the scale of the mapping learnable tokens; represents a mapping function; Represents element-by-element multiplication operation; Step 2.1.3: Use the text encoder to and Encode and get Example text embedding and the text embedding matrix of K cell categories ;in, Represents the category text embedding of the cell category with the actual label k; Step 2.1.4: Use the visual encoder to Encode and get Image embedding , thus As a query to the Transformer decoder, As the key and value of the Transformer decoder, and the Transformer decoder Calibrate and generate Visual domain cue embedding ; Step 2.1.5: Generate using formula (7) Enhanced instance text embedding : (7) In formula (7), represents a set of control visual domain cue embeddings The parameters to be learned of the scale; Step 2.1.6: Use formula (8) to get Specific instance text features : (8) In formula (8), represents the specificity mask; represents the masked cross attention operation; Representation layer normalization function; represents a feed-forward neural network; Step 2.1.7: and Splice to get Instance-aware categorical text features .

3. The pathological image cell classification network based on visual language large model and prompt learning according to claim 2 is characterized in that: The step 2.3 includes: Step 2.3.1: Use formula (9) to get The two conditional vectors and : (9) In formula (9), and Represents 2 different multi-layer perceptrons; Step 2.3.2: Use formula (10) to Perform conditional adjustments to obtain prompt conditional image features : (10)。 4. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the pathological image cell classification method according to any one of claims 1 to 3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the pathological image cell classification method according to any one of claims 1 to 3 are executed.

Images