Small-sample full-section pathological image classification method and system based on multi-modal large model, and computer readable storage medium

Abstract

The invention belongs to the technical field of image analysis, and more specifically relates to a small-sample full-section pathological image classification method and system based on a multi-modal large model, and a computer readable storage medium. The method comprises the steps of obtaining a multi-instance data set, and dividing the multi-instance data set into a training set and a test set; a complete pathology multi-instance data set composed of M full-slice pathology images is given, each full-slice pathology image comprises U image blocks, each full-slice pathology image has a package level label, and each image block has an instance level label; k full-slice pathological images are selected from the data set, L image blocks are selected from each selected full-slice pathological image, and a training data set is constructed; constructing a double small sample learning new normal form network model, and performing learning; and finally carrying out two-level classification. The problems that in the prior art, a large amount of annotation data is needed, the annotation cost is high, popularization in clinical application is difficult, and data scarcity is difficult to solve are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image analysis, and more specifically, to a small sample whole slice pathology image classification method, system and computer-readable storage medium based on a multimodal large model. Background Art

[0002] Whole-slice pathology image (WSI) classification is an important problem in computational pathology and is widely used in cancer diagnosis, pathological analysis and other fields. Unlike natural images, whole-slice pathology images are huge in size. Using deep learning methods, they need to be divided into a series of image blocks and a classification model is applied to each image block. The Patch Level fully supervised learning method is the most direct way to train a classification model. It annotates each image block and then performs end-to-end training.

[0003] The Slide Level weakly supervised learning method models the whole-slide pathology image classification task as a multiple instance learning (MIL) problem, where each image block is defined as an Instance, each WSI is considered as a Bag, and only the Slide-Level weakly supervised labels are used to train the whole-slide pathology image classification network. However, this study is limited to the large amount of available whole-slide pathology image data.

[0004] Chinese patent document CN118097315A discloses a small sample image classification method based on contrastive learning, including S1, constructing a base class data set and a new class data set; S2, extracting multiple N-way K-shot meta-tasks on the base class data set for network training, dividing each meta-task into a labeled support set and a labeled query set, and using the labeled support set and the labeled query set as network input, outputting a mixed loss, and back-propagating the loss to adjust network parameters; S3, extracting multiple N-way K-shot meta-tasks on the new class data set for network testing, dividing each meta-task into a labeled support set and an unlabeled query set to input into the network, and outputting the network's classification result for the query set.

[0005] In summary, the existing full-slice pathology image classification methods can be divided into two categories. The fully supervised learning method based on Patch Label, this method labels each patch and then uses a deep learning model for training. However, this method requires a large amount of labeled data, and the labeling cost is high, making it difficult to promote in clinical applications. The weakly supervised learning method based on Slide Label, this method only uses Slide-level labels for training to avoid expensive labeling. However, this method cannot utilize patch-level supervision information and relies on a large amount of training data, making it difficult to solve the problem of data scarcity. Summary of the invention

[0006] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide a small sample whole-slice pathology image classification method based on a multimodal large model to solve the recognition problem that cannot utilize patch-level supervision information, relies on a large amount of training data, and is difficult to solve the problem of data scarcity.

[0007] The detailed technical scheme of the present invention is as follows: A small sample full-slice pathological image classification method based on a multimodal large model, the method comprising: S1. Obtain a complete pathology multi-instance dataset and divide it into a training set and a test set; Given a complete pathology multi-instance dataset consisting of M full-slice pathology images, the M full-slice pathology images contain image blocks, M and U are positive integers; for each full-slice pathology image, it has a package-level label, and for each image block, it has an instance-level label; S2. Select K full-slice pathology images from the training set, and select L image blocks from each selected full-slice pathology image to construct a training data set, where K and L are both positive integers and K <M、L<U; S3. Construct a new paradigm network model for dual small sample learning and use the training data set and test set for learning; The dual small sample learning new paradigm network model includes a learnable image cache branch and a priori knowledge branch based on a multimodal pre-trained neural network, specifically: Construct a learnable image cache branch, including: image encoder, preprocessing module, cache model, knowledge retrieval module; The image encoder extracts features of image blocks of the pathological image and stores them in a cache model; The preprocessing module converts the labeled labels into one-hot encoding, generates pseudo labels for unlabeled image blocks, converts the pseudo labels into learnable pseudo labels using learnable, and stores them in the cache model; The cache model includes a feature cache and a label cache, which are used to store features and labels respectively; The knowledge retrieval module adopts an attention mechanism, takes the feature as the key, takes the label as the value and combines the key to be retrieved to obtain the prediction result of the knowledge retrieval module, and the key to be retrieved is the feature of the image block to be tested extracted by the image encoder; Construct prior knowledge branches based on multimodal pre-trained neural networks, including GPT-4V, text encoder, and text-image fusion classifier; The GPT-4V generates a description of the image in combination with pathological knowledge to form prompt words for detecting each category of images; The text encoder extracts features of prompt words of different categories to obtain prompt word features; The text-image fusion classifier obtains a classification result of a priori knowledge branch based on the prompt word, the prompt word feature and the feature of the image block to be tested; Finally, the prediction results of the knowledge retrieval module and the classification results of the prior knowledge branch are weighted and fused to obtain the final prediction results and loss function.

[0008] Furthermore, the prediction results are classified in two stages: image block level classification and image level classification: In the image block level classification stage, the image blocks are classified to obtain the classification results of each image block; In the image-level classification stage, the classification results of all image blocks of each image are pooled to obtain the classification result of the entire image, and finally the classification result of each full-slice pathology image is output, that is, positive and non-positive.

[0009] S4, image block level classification, i.e., the final prediction result of the new paradigm network model of dual small sample learning; Image-level classification: pooling operation is performed on the classification results of all image blocks of each image to obtain the classification result of the entire image, and finally the classification result of each full-slice pathology image is output, that is, positive and non-positive.

[0010] Furthermore, the complete pathology multi-instance dataset is divided into a training set and a test set, and S1 specifically includes: For the training set, given by A training set of complete pathology multi-instances consisting of full-slice pathology images , any full-slice pathology image Depend on image blocks, i.e., the i-th image , Any image block in is considered as an instance, All image blocks in a package together form a bag. represents the jth image block of the i-th image, ; Images of whole slide pathology images in the training set , with corresponding package-level tags , and the training set Corresponding to the package training label set ; Training set The image blocks in Corresponding to instance-level labels , forming the instance training label set ; Among them, is the total number of image patches in the M-th image.

[0011] For the test set, given a complete pathological multi-instance test set composed of whole-slide pathological images ; The images of the whole-slide pathological images in the test set correspond to bag-level labels , and the test set corresponds to a bag test label set , is the corresponding bag-level label; The image patches in the test set have corresponding instance-level labels , that is, an instance test label set .

[0012] Furthermore, the specific steps of the learnable image cache branch are as follows: First, select K whole-slide pathological images from the training set to construct a training data set , where is a positive integer and K < M; Then, for each whole-slide pathological image in the training data set , only label image patches to obtain a labeled instance-level label set , L represents the number of image patches labeled in each whole-slide pathological image, where L is a positive integer and L < U; except for the labeled image patches, the remaining U - L unlabeled image patches form an unlabeled instance-level pseudo-label set.

[0013] Furthermore, the processing steps of the learnable image cache branch are as follows: S31. Input the training data set into the image encoder to obtain the features of each image patch and store them in the feature cache: (1); In formula (1), represents the image encoder; S32. The preprocessing module converts the labels of the labeled image patches into one-hot encoding and stores them in the label cache; Generate pseudo-labels for the unlabeled image patches and use learnable to convert the pseudo-labels into learnable pseudo-labels, which are learnable labels , and stored in the tag buffer: (2); ,represent All unlabeled instance pseudo labels in, where h represents the training set The number of unlabeled image patches in , h=UL; S33. After constructing the feature buffer and the label buffer, prediction is performed through knowledge retrieval: The knowledge retrieval module is implemented using the attention mechanism, which treats features and labels as one-to-one key-value pairs. As key, use Indicates that represents the number of keys in the knowledge base, represents the feature dimension of each key, where R is a real matrix; label For the value, use the matrix express, The dimension of the representative value; extract the features of the image blocks in the test set to obtain the image features to be retrieved , the image features to be retrieved are used as the key to be retrieved express, Represents the number of keys to be retrieved; Finally, by calculating the dot product of the query and all keys To get the similarity matrix, then pass Converted to weights, and finally with the matrix Multiply them together to get the prediction result: Image cache branch prediction results obtained through the knowledge retrieval module for: (3); S34. Calculate the cross entropy between the image cache branch prediction value and the gold standard as the loss function: (4); In formula (4), Represents the cross entropy function, which is used to calculate the loss between the predicted result and the true label. Refers to the image cache branch prediction value, refers to the true label, i.e., the gold standard; Refers to the training set samples Features obtained through encoding by the image encoder; the image cache branch prediction value is a calculated value before the image cache branch prediction result is obtained.

[0014] Finally, the learnable labels in the cached model can be optimized by minimizing the above loss function: (5); At the same time, in order to use the supervision information to optimize the feature space obtained by the image encoder, The value of is set to be learnable: (6).

[0015] Furthermore, a threshold of unlabeled image blocks is set. When the number of unlabeled image blocks in the image is greater than the threshold, the representative image blocks are first selected through the K-means clustering algorithm to construct a core set: Randomly select K data points, i.e. unlabeled image blocks, as the initial center points of each cluster; calculate the distance between each unlabeled image block and the center point, and assign the unlabeled image block to the nearest cluster, where a cluster refers to a collection of unlabeled image blocks; calculate the average value of the unlabeled image blocks in each cluster as the new center point; repeat the steps of assigning and updating the center point until the stopping condition is met; when the stopping condition is met, the algorithm ends, each data point belongs to a cluster, and the center points of all clusters form a core set; the core set is regarded as a collection of unlabeled image blocks.

[0016] Furthermore, the processing steps of the prior knowledge branch based on the multimodal pre-trained neural network are as follows: S35, inputting the labeled image blocks into GPT-4V, allowing GPT-4V to generate descriptions of the image blocks in combination with relevant pathological knowledge to form prompt words for detecting each category of images; S36, then extract the features of the prompt words of different categories through the text encoder to obtain the prompt word features , (7); (8); In formula (7) ~ formula (8), For Category The prompt word encoding is composed of multiple word vectors Composition, l represents the total number of word vectors.

[0017] S37, in the text image fusion classifier, the image block features of the test set encoded by the image encoder are given, that is, the test image block features , The set of full-slice pathology image block features representing the entire test set is a set of text features containing all categories; The classification results of the prior knowledge branch under N categories are: (9); In formula (9), is the temperature coefficient, learned by CLIP in the pre-training stage, refers to the possibility that the test image patch belongs to category c, For Category The text features of It represents the sum of all categories from category 0 to category N, where N represents the total number of categories.

[0018] Furthermore, different from S36, in the prior knowledge branch, the prompt words for each category of images are set to be learnable, specifically: Add the prompt words obtained in each category Learnable tokens , c is the index of the category, indicating different categories; D represents the number of learning tokens, where learning tokens refer to the introduced trainable vectors; the final learnable prompt word is obtained by combination, where each token is a 512-dimensional learnable feature: (10); In formula (10), Indicates that in a given category The final learnable prompt vector after combining all the learning tokens is obtained by combining D learning tokens in sequence. The new prompt word feature for, (11); In formula (11), refers to the text encoder, Refers to the category The cue word encoding combined with the learning token; Based on the new prompt word features And the test image patch features , is a given image feature , the probability that the image block is classified as c, that is, the classification result of the prior knowledge branch : (12).

[0019] Furthermore, the loss function of the training data of the prior knowledge branch module is calculated: (13); Continuously optimize learnable tokens: (14); Similarly, for CLIP The parameters are no longer optimized.

[0020] Furthermore, the prediction result based on the knowledge retrieval module and the classification result of the text image fusion classifier are fused by weight to obtain the final prediction result and the total loss function as follows: (15); (16); In formula (15) ~ formula (16), is the fusion weight of the two branches, and the optimal value is quickly obtained through grid search.

[0021] In another aspect of the present invention, a small sample whole-slice pathology image classification system based on a multimodal large model is provided, the system comprising: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to execute the small sample whole slice pathology image classification method based on the multimodal large model as described above.

[0022] In another aspect of the present invention, a machine-readable storage medium is provided, which stores executable instructions, and when the instructions are executed, the machine executes the small sample whole-slice pathology image classification method based on the multimodal large model as described above.

[0023] Compared with the prior art, the present invention has the following beneficial effects: The small sample whole-slice pathology image classification method, system and computer-readable storage medium based on a multimodal large model proposed in the present invention can obtain PatchLevel supervision information by only annotating a small number of image blocks in a small number of whole-slice pathology images, greatly reducing the annotation cost and time, and achieving SOTA performance on the CAMELYON16 and TCGA-RENAL datasets. At the same time, based on the pre-trained visual language models CLIP and GPT-4V, it uses fewer parameters, has a fast training speed, is easy to deploy, and can be quickly migrated to a variety of whole-slice pathology image classification tasks without retraining. The caching mechanism and prompt word technology improve the model interpretability, and can be used for cancer detection, pathology research and medical education, with broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the small sample whole slice pathology image classification method based on the multimodal large model described in the present invention.

[0025] Figure 2 It is a schematic diagram of the double-level data labeling strategy described in Example 1 of the present invention.

[0026] Figure 3It is a schematic diagram of the structure of the new paradigm network model of dual small sample learning described in Example 1 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0029] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0030] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0031] Example 1 Ginseng Figure 1 This embodiment provides a small sample full-slice pathological image classification method based on a multimodal large model, the method comprising: S1, divide the complete pathology multi-instance dataset into training set and test set; Given a complete pathology multi-instance dataset consisting of M full-slice pathology images, the M full-slice pathology images contain image blocks, M and U are positive integers; for each full-slice pathology image, there is a package-level label, and for each image block, there is an instance-level label; Specifically, dividing the complete pathology multi-instance dataset into a training set and a test set includes: For the training set, given by A training set of complete pathology multi-instances consisting of full-slice pathology images , M represents the total number of full-slice pathology images, where the image of any full-slice pathology image Depend on image blocks, i.e., the i-th image , Any image block in is considered as an instance, All image blocks in a package together form a bag. represents the jth image block of the i-th image, is the total number of image blocks in the i-th image; Images of whole slide pathology images in the training set , with corresponding package-level tags , and the training set Corresponding to the package training label set ; Training set The image blocks in , with corresponding instance-level labels , i.e., the instance training label set ; in, is the total number of image blocks in the Mth image.

[0032] For the test set, given by A complete pathology multi-instance test set consisting of full-slice pathology images ; Images of whole slide pathology images in the test set , corresponding to the package-level label , and the test set Corresponding package test tag set , for The corresponding package-level label; Test Set The image blocks in , with corresponding instance-level labels , that is, the instance test label set .

[0033] S2, perform two-level data annotation and build a training data set; From the training set First, K full-slice pathology images are selected, and then each full-slice pathology image is labeled image patches, and construct them as training datasets , the corresponding label set is the annotated instance-level label set , K and L are both positive integers and K <M、L<U; ; Specifically, Figure 2 As shown, a two-level data annotation strategy is adopted: First, from the complete pathology multi-instance dataset First, select K full-slice pathology images to construct a training dataset ,in Much smaller than ; Then, for the training dataset In each full-slice pathology image, only the annotation image patches, and obtain an instance-level label set , L represents the number of annotated image blocks in each full-slice pathology image, where Much smaller than the number of all image blocks in a full-slice pathology image ; Except for the marked image blocks, the remaining UL unlabeled image blocks constitute the unlabeled instance-level pseudo label set.

[0034] In this setting, the training dataset The number of At the same time, the number of image patches in the full-slice pathology image is much smaller than the number of annotated image patches in the complete fully supervised dataset.

[0035] Preferably, in conventional small sample learning, the training data is usually modeled in an “N-way K-shot” manner, where N-way represents N categories. -shot represents labeled samples; Under the dual-level data annotation strategy, the shot in N-way K-shot is composed of two scales: bag and instance. Therefore, the dual small sample full-slice pathology image classification task is constructed in the form of "N-way K-bshot&L-ishot", where K-bshot represents K bags and L-ishot represents L annotated instance-level labels.

[0036] Specifically, take the simulation of small sample learning scenarios as an example: In order to simulate the annotation strategy of FAST in real scenarios, a dual small sample scenario with both bag and instance small samples is constructed on the training data set with complete instance-level annotations. First, simulate the small sample WSI data collection process in clinical practice, sample all full-slice pathology images under each category in the training set, that is, select K full-slice pathology images, and set the sampling number to 1, 2, 4, 8, and 16 respectively. The sampled full-slice pathology images can be used for training, but the unsampled full-slice pathology images are completely invisible; Then we continue to simulate the expert's small sample instance annotation process. In each full-slice pathology image obtained by sampling, we first use the K-means clustering method to select The image blocks are used as the sampling core set, and then 16 image blocks are randomly sampled from each category in the sampling core set as the labeled image blocks, namely the labeled image blocks, and the other image blocks are used as unlabeled samples. Through this simulation process, for the CAMELYON16 and TCGA-RENAL datasets, small sample training datasets with bshot of 1, 2, 4, 8, 16 and ishot of 16 were obtained respectively. Considering the randomness of small sample learning, 5 random small sample sampling and model training were performed for all scenes, and the mean and variance of model prediction were reported.

[0037] During training, you can use and To train the model, the annotation cost can be significantly reduced, while the full test set is still used for testing. and Testing the model can more accurately evaluate the performance of the model, improve the reliability of the classification results, and adapt to clinical application scenarios.

[0038] S3, build a dual small sample learning new paradigm (FAST) network model and train it with the training data set; The dual few-shot learning new paradigm (FAST) network model includes a learnable image cache branch and a prior knowledge branch based on a multimodal pre-trained neural network (CLIP), such as Figure 3 As shown; Construct a learnable image cache branch, including: image encoder, preprocessing module (learnable), cache model, knowledge retrieval module; The image encoder extracts features of image blocks of the full-slice pathology image and stores them in a cache model; preferably, the image encoder is an image encoder in CLIP, which is a deep learning model developed by OpenAI to enable computers to understand and process the relationship between images and text. The CLIP model has two main parts: an image encoder and a text encoder; The preprocessing module converts the labeled labels into one-hot encoding, generates pseudo labels for unlabeled image blocks and converts the pseudo labels into learnable pseudo labels using learnable; The cache model includes a feature cache and a label cache, which are used to store features and labels respectively; The knowledge retrieval module adopts an attention mechanism, taking features as keys and labels as values ​​in combination with the keys to be retrieved to obtain the prediction results of the knowledge retrieval module.

[0039] Construct prior knowledge branches based on multimodal pre-trained neural networks, including GPT-4V, text encoder, and text-image fusion classifier; The GPT-4V generates a description of the image in combination with pathological knowledge to form prompt words for detecting each category of images; The text encoder extracts features of prompt words of different categories to obtain prompt word features; preferably, the text encoder is a CLIP text encoder; The text-image fusion classifier obtains a classification result based on the prompt word and the prompt word feature; Finally, the prediction results of the knowledge retrieval module and the classification results of the text-image fusion classifier are weighted and fused to obtain the final prediction results and loss function.

[0040] Furthermore, the processing steps of the learnable image cache branch are as follows: S31. Input the training data set into the image encoder to obtain the features of each image block , and store it in the feature buffer: (1); In formula (1), Stands for Image Encoder.

[0041] S32, the preprocessing module converts the label of the labeled image block into a one-hot code and stores it in a label buffer; Generate pseudo labels for unlabeled image blocks, and use learnable to convert pseudo labels into learnable pseudo labels, that is, learnable labels , and stored in the tag buffer: (2); ,represent All unlabeled instance pseudo labels in, where h represents the training dataset The number of unlabeled image patches in .

[0042] S33, after constructing the feature buffer and the label buffer, prediction is performed through knowledge retrieval, such as Figure 3 As shown in the knowledge retrieval module; Specifically, the attention mechanism is used to implement the knowledge retrieval module, which regards features and labels as one-to-one key-value pairs, and the transposition of features As key, use Indicates that represents the number of keys in the knowledge base, represents the feature dimension of each key, where R is a real matrix; label For the value, use the matrix express, The dimension of the representative value; extract the features of the image blocks in the test set to obtain the image features to be retrieved , the image features to be retrieved are used as the key to be retrieved express, Represents the number of keys to be retrieved; Finally, by calculating the dot product of the query and all keys To get the similarity matrix, then pass Converted to weights, and finally with the matrix Multiply them together to get the prediction result: Image cache branch prediction results obtained through the knowledge retrieval module for: (3).

[0043] S34. Calculate the cross entropy between the image cache branch prediction value and the gold standard as the loss function: (4); In formula (4), Represents the cross entropy function, which is used to calculate the loss between the predicted result and the true label. Refers to the image cache branch prediction value, refers to the true label, i.e., the gold standard; Refers to the training set data block, that is, the training set image block Features obtained through encoding by the image encoder; the image cache branch prediction value is a calculated value before the image cache branch prediction result is obtained.

[0044] Finally, the learnable labels in the cached model can be optimized by minimizing the above loss function: (5); At the same time, in order to use the supervision information to optimize the feature space obtained by the image encoder, The value of is set to be learnable: (6); For the network Parameters are not optimized.

[0045] Furthermore, the processing steps of the prior knowledge branch based on the multimodal pre-trained neural network are as follows: In order to further utilize the prior knowledge of the multimodal large model, a learnable instance classifier for prompt words was constructed based on the text encoder of CLIP. Since the feature spaces of the text encoder and image encoder of CLIP are aligned, image classification can be achieved by calculating the similarity between text and image. However, unlike natural image classification, pathological image classification requires more professional and targeted prompt words.

[0046] S35, such as Figure 3As shown in the prior branch of , the labeled image blocks are input into GPT-4V, and GPT-4V combines relevant pathological knowledge to generate descriptions of the image blocks. These descriptions include information such as cell types, tissue structures, and pathological characteristics that appear in the images, forming prompt words for detecting each category of images. These prompt words are natural language phrases used to describe the characteristics of image categories, such as "cancer cells", "normal tissue", etc.

[0047] S36, then extract the features of the prompt words of different categories through the text encoder to obtain the prompt word features , (7); (8); In formula (7) ~ formula (8), For Category The prompt word encoding is composed of multiple word vectors Composition, l represents the total number of word vectors.

[0048] S37, in the text image fusion classifier, given the image block features of the test set encoded by the image encoder, that is, the test image block , It represents the feature vector of a single test image block; The set of features representing the entire test full-slice pathology image is a set of text features containing all categories; The classification results of the prior knowledge branch under N categories are: (9); In formula (9), is the temperature coefficient, learned by CLIP in the pre-training stage, refers to the possibility that the test image patch belongs to category c, For Category The text features of It represents the sum of all categories from category 0 to category N, where N represents the total number of categories. Represents any category from category 0 to category N.

[0049] S4, image blocks are classified into two levels of images; Image patch-level classification, i.e. the final prediction result of the FAST network model; Image-level classification: pooling operation is performed on the classification results of all image blocks of each image to obtain the classification result of the entire image, and finally the classification result of each full-slice pathology image is output, that is, positive and non-positive.

[0050] Example 2 This embodiment is based on Embodiment 1, and differs from Embodiment 1 in that: Set the cue words for each category of images to be learnable; Specifically, add the prompt words obtained in each category Learnable tokens , c is the index of the category, indicating different categories; D represents the number of learning tokens; learning tokens refer to the introduced trainable vectors, which can capture specific information or context in the text, and their learning process is part of the model training, and the final learnable prompt word is obtained by combination, where each token is a 512-dimensional learnable feature: (10); In formula (10), Indicates that in a given category The final learnable prompt word after combining all the learning tokens can be used to learn the prompt vector, which is the result obtained by combining D learning tokens in sequence. The new prompt word feature for, (11); In formula (11), refers to the text encoder, Refers to the category The cue word encoding combined with the learning token; Specifically, based on the new prompt word features And the test image patch features , is a given image feature When , the probability that the image block is classified as c, that is, the classification result of the prior knowledge branch: (12); Calculate the loss function of the training data of the prior knowledge branch module: (13); Continuously optimize learnable tokens: (14).

[0051] Furthermore, the prediction result based on the knowledge retrieval module and the classification result of the text-image fusion classifier are weighted and fused to obtain the final prediction result. And the total loss function Specifically: (15); (16); In formula (15) ~ formula (16), is the fusion weight of the two branches, which can be regarded as a hyperparameter. Since the entire network has few parameters and fast inference speed, the optimal value of this parameter can be quickly obtained through grid search.

[0052] This example first uses the CAMELYON16 dataset as the processing object, and the results are: Whether in terms of instance-level AUC or package-level AUC, the present invention demonstrates significant performance advantages in different small sample learning scenarios. Under different sample sizes, the present method surpasses the comparison methods Tip-Adapter-F and Tip-Adapter. Specifically, the performance of different methods under different sample sizes is analyzed: First, under zero-sample conditions, the instance-level and package-level AUCs of zero-sample CLIP are very low and can hardly complete the classification task. However, in the present invention, in the extremely small sample case where there is only one package in each category and only 16 instances are annotated in each package, the instance-level AUC mean can reach 0.8400, which is significantly improved compared with zero-sample CLIP.

[0053] Secondly, as the number of training samples increases, the performance of the method continues to show a significant improvement trend. When the number of package samples is only 16 and only 16 instances are annotated in each package, the instance-level AUC and package-level AUC of the present invention reach 0.9151 and 0.9161, respectively. 0.0200 and 0.8197 0.0474, which is close to the fully supervised method that uses all instance annotations. However, the annotation amount of this method is only 0.22% of that of the fully supervised method.

[0054] Secondly, this example uses the TCGA-RENAL dataset as the processing object, and the results are: When the number of packages reaches 4 or more, the results of the small sample method begin to improve significantly. The present invention also surpasses other methods in all indicators in the scenarios with 4, 8 and 16 packages, especially in the scenario with 16 packages, where the present method significantly surpasses the comparison method. The mean AUC of package-level classification for chromophobe cell renal carcinoma (chRCC) reaches 0.9234, which is only 0.0033 different from the fully supervised method. The mean AUC of package classification for clear cell renal carcinoma (ccRCC) and papillary renal carcinoma (pRCC) also reaches 0.9254 and 0.9216, respectively, which is only 0.0218 and 0.036 different from the fully supervised method.

[0055] This experimental result further demonstrates that in multi-category complex cancer recognition tasks, the present invention can also approach the level of fully supervised methods at a very low annotation cost, reaching the current best level of small sample learning.

[0056] Example 3 This embodiment provides a small sample whole-slice pathological image classification system based on a multimodal large model, the system comprising: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to execute the small sample whole slice pathology image classification method based on the multimodal large model as described above.

[0057] In this embodiment, the device includes but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0058] Example 4 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the small sample whole-slice pathology image classification method based on a multimodal large model as described above.

[0059] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0060] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0061] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code may be downloaded from a server computer or a cloud via a communication network.

[0062] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A small sample full-slice pathological image classification method based on a multimodal large model, characterized in that: The method comprises: S1. Obtain a complete pathology multi-instance dataset and divide it into a training set and a test set; Given a complete pathology multi-instance dataset consisting of M full-slice pathology images, the M full-slice pathology images contain image blocks, M and U are positive integers; for each full-slice pathology image, it has a package-level label, and for each image block, it has an instance-level label; S2. Select K full-slice pathology images from the training set, and select L image blocks from each selected full-slice pathology image to construct a training data set, where K and L are both positive integers and K <M、L<U; S3. Construct a new paradigm network model for dual small sample learning and use the training data set and test set for learning; The dual small sample learning new paradigm network model includes a learnable image cache branch and a priori knowledge branch based on a multimodal pre-trained neural network, specifically: Construct a learnable image cache branch, including: image encoder, preprocessing module, cache model, knowledge retrieval module; The image encoder extracts features of image blocks of the pathological image and stores them in a cache model; The preprocessing module converts the labeled labels into one-hot encoding, generates pseudo labels for unlabeled image blocks, converts the pseudo labels into learnable pseudo labels using learnable, and stores them in the cache model; The cache model includes a feature cache and a label cache, which are used to store features and labels respectively; The knowledge retrieval module adopts an attention mechanism, takes the feature as the key, takes the label as the value and combines the key to be retrieved to obtain the prediction result of the knowledge retrieval module, and the key to be retrieved is the feature of the image block to be tested extracted by the image encoder; Construct a priori knowledge branch based on multimodal pre-trained neural networks, including GPT-4V, text encoder, and text-image fusion classifier; The GPT-4V generates a description of the image in combination with pathological knowledge to form prompt words for detecting each category of images; The text encoder extracts features of prompt words of different categories to obtain prompt word features; The text-image fusion classifier obtains a classification result based on the prompt word, the prompt word feature and the feature of the image block to be tested; Finally, the prediction results based on the knowledge retrieval module and the classification results of the text-image fusion classifier are weighted and fused to obtain the final prediction results and loss function. S4, image block level classification, i.e., the final prediction result of the new paradigm network model of dual small sample learning; Image-level classification: pooling operation is performed on the classification results of all image blocks of each image to obtain the classification result of the entire image, and finally the classification result of each full-slice pathology image is output, that is, positive and non-positive.

2. The small sample whole slice pathology image classification method based on multimodal large model according to claim 1 is characterized in that: The S1 specifically includes: For the training set, given by A training set of complete pathology multi-instances consisting of full-slice pathology images , any full-slice pathology image Depend on image blocks, i.e., the i-th image , Any image block in is considered as an instance, All image blocks in a package together form a bag. represents the jth image block of the i-th image; Images of whole slide pathology images in the training set , with corresponding package-level tags , and the training set Corresponding to the package training label set ; Training set The image blocks in Corresponding to instance-level labels , forming the instance training label set ； in, is the total number of image blocks in the Mth image; For the test set, given by A complete pathology multi-instance test set consisting of full-slice pathology images ; Images of whole slide pathology images in the test set , corresponding to the package-level label , and the test set Corresponding package test tag set , for The corresponding package-level label; Test Set The image blocks in , with corresponding instance-level labels , that is, the instance test label set .

3. The small sample whole slice pathology image classification method based on multimodal large model according to claim 2 is characterized in that: The S2 specifically includes: First, from the training set First, select K full-slice pathology images to construct a training dataset ,in is a positive integer and K <M; Then, for the training dataset In each full-slice pathology image, only the annotation image patches, and obtain an instance-level label set , L represents the number of annotated image blocks in each full-slice pathology image, where L is a positive integer and L <U。 4. The small sample whole slice pathology image classification method based on multimodal large model according to claim 3 is characterized in that: The processing steps of the learnable image cache branch are as follows: S31, input the training data set into the image encoder, obtain the features of each image block, and store them in the feature buffer: （1）； In formula (1), represents image encoder; S32, the preprocessing module converts the label of the labeled image block into a one-hot code and stores it in a label buffer; Generate pseudo labels for unlabeled image blocks, and use learnable to convert pseudo labels into learnable pseudo labels, that is, learnable labels , and stored in the tag buffer: （2）； ,represent All unlabeled instance pseudo labels in, where h represents the training set The number of unlabeled image patches in ; S33. After constructing the feature buffer and the label buffer, prediction is performed through knowledge retrieval: The knowledge retrieval module is implemented using the attention mechanism, which treats features and labels as one-to-one key-value pairs. As key, use Indicates that represents the number of keys in the knowledge base, represents the feature dimension of each key, where R is a real matrix; label For the value, use the matrix express, The dimension of the representative value; extract the features of the image blocks in the test set to obtain the image features to be retrieved , the image features to be retrieved are used as the key to be retrieved express, represents the number of keys to be retrieved; then, by calculating the dot product of the query and all keys To obtain the similarity matrix, Converted to weights, and finally with the matrix Multiply them together to get the prediction result: Image cache branch prediction results obtained through the knowledge retrieval module for: （3）； S34. Calculate the cross entropy between the image cache branch prediction value and the gold standard as the loss function: （4）； In formula (4), Represents the cross entropy function, which is used to calculate the loss between the predicted result and the true label. Refers to the image cache branch prediction value, refers to the true label, i.e., the gold standard; Refers to the training set samples Features obtained by encoding with an image encoder; the image cache branch prediction value is a calculated value before the image cache branch prediction result is obtained; Finally, the learnable labels in the cached model can be optimized by minimizing the above loss function: （5）； At the same time, in order to use the supervision information to optimize the feature space obtained by the image encoder, The value of is set to be learnable: （6）。 5. The small sample whole slice pathology image classification method based on multimodal large model according to claim 3 or 4, characterized in that: The processing steps of the prior knowledge branch based on the multimodal pre-trained neural network are as follows: S35, inputting the labeled image blocks into GPT-4V, allowing GPT-4V to generate descriptions of the image blocks in combination with relevant pathological knowledge, wherein the descriptions include cell types, tissue structures, and pathological feature information appearing in the images, forming prompt words for detecting each category of images; S36, then extract the features of the prompt words of different categories through the text encoder to obtain the prompt word features , （7）； （8）； In formula (7) ~ formula (8), For Category The prompt word encoding is composed of multiple word vectors Composition, l represents the total number of word vectors; S37, in the text image fusion classifier, the image block features of the test set encoded by the image encoder are given, that is, the test image block features , Represents the set of full-slice pathology image patch features of the entire test set; The classification results of the prior knowledge branch under N categories are: （9）； In formula (9), is the temperature coefficient, learned by CLIP in the pre-training phase, refers to the possibility that the test image patch belongs to category c, For Category The text features of It represents the sum of all categories from category 0 to category N, where N represents the total number of categories.

6. The small sample whole slice pathology image classification method based on multimodal large model according to claim 5 is characterized in that: The prompt words for each category of images are set to be learnable, specifically: Add the prompt words obtained in each category Learnable tokens , c is the index of the category, indicating different categories; D represents the number of learning tokens, and learning tokens refer to the introduced trainable vectors; the final learnable prompt word is obtained by combination, where each token is a 512-dimensional learnable feature: （10）； In formula (10), Indicates that in a given category The final learnable prompt word after combining all learning tokens is the result obtained by combining D learning tokens in sequence. The new prompt word feature for, （11）； In formula (11), refers to the text encoder, Refers to the category The cue word encoding combined with the learning token; Based on the new prompt word features And the test image patch features , is the feature of a given image block , the probability that the image block is classified as c, that is, the classification result of the prior knowledge branch : （12）； Furthermore, the loss function of the training data of the prior knowledge branch module is calculated: （13）； Continue to optimize learnable tokens: （14）。 7. The small sample whole slice pathology image classification method based on multimodal large model according to claim 6 is characterized in that: The prediction result based on the knowledge retrieval module and the classification result of the text-image fusion classifier are fused by weight to obtain the final prediction result and loss function: （15）； （16）； In formula (15) ~ formula (16), is the fusion weight of the two branches, which can be regarded as a hyperparameter. Since the entire network has few parameters and fast inference speed, the optimal value of this parameter can be quickly obtained through grid search.

8. The small sample whole slice pathology image classification method based on multimodal large model according to claim 1 or 7, characterized in that: Set the unlabeled image block threshold. When the number of unlabeled image blocks in the image is greater than the unlabeled image block threshold, first select representative image blocks through the K-means clustering algorithm to construct the core set: Randomly select K data points, i.e. unlabeled image blocks, as the initial center points of each cluster; calculate the distance between each unlabeled image block and the center point, and assign the unlabeled image block to the nearest cluster, where a cluster refers to a collection of unlabeled image blocks; calculate the average value of the unlabeled image blocks in each cluster as the new center point; repeat the steps of assigning and updating the center point until the stopping condition is met; when the stopping condition is met, the algorithm ends, each data point belongs to a cluster, and the center points of all clusters form a core set; the core set is regarded as a collection of unlabeled image blocks.

9. A small sample whole slice pathology image classification system based on a multimodal large model, characterized in that: The system comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of the small sample whole slice pathology image classification method based on a multimodal large model are implemented as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images