A multi-cue learning method for HER2 prediction in breast cancer

Abstract

This invention discloses a breast cancer HER2 prediction method using multi-cue learning. The method first acquires image features from breast cancer WSI pathology images. A large language model is then used to generate descriptive text with multiple cues. A multimodal pre-trained model is then used to embed learnable cues into the descriptive text to generate multiple cues. A text encoder is then used to generate multiple cues. A text-guided feature aggregation module is then used to aggregate global visual features of category and staining level, which are then fed into a global feature fusion module for integration to obtain HER2 features. Finally, the cosine similarity between the slice-level cues and the HER2 features is calculated to obtain HER2 prediction results for breast cancer WSI pathology images. The method is then optimized using a cross-entropy loss. This method effectively captures semantic information related to text descriptions, mimics the process of HER2 assessment by pathologists, and improves HER2 prediction accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image processing and HER2 prediction, and in particular relates to a breast cancer HER2 prediction method based on multiple prompt learning. Background Art

[0002] Breast cancer is the second most common cancer worldwide and also has the highest incidence and mortality rates among women, ranking fourth in global mortality. In clinical diagnosis, tumor tissue is typically stained with hematoxylin andeosin (H&E), and morphological evaluation is performed under a light microscope. Immunohistochemistry (IHC) is also used to assess the expression of biomarkers, such as human epidermal growth factor receptor 2 (HER2). HER2 gene amplification leads to significantly increased expression of HER2 protein on the cell surface, which is closely associated with accelerated tumor cell proliferation and enhanced invasiveness. Accurate assessment of HER2 status is not only an important basis for the diagnosis and classification of breast cancer but also plays a key role in clinical treatment decisions. Therefore, HER2 scoring is a core component of breast cancer diagnosis, classification, and treatment. In clinical practice, pathologists determine a HER2 score (0, 1+, 2+, or 3+) based on the proportion of invasive cancer cells with varying degrees of staining. Cases with a score of 0 or 1+ are classified as negative; cases with a score of 3+ are classified as positive; and cases with a score of 2+ are considered equivocal and require further testing for HER2 gene amplification via fluorescence in situ hybridization (FISH). HER2 scoring not only guides clinical treatment but also provides an important basis for breast cancer research and drug development. Large-scale analysis of HER2 status can assess the clinical efficacy of new targeted drugs and promote the continuous optimization of personalized treatment plans. However, traditional HER2 assessment methods rely primarily on subjective judgment by pathologists and have significant limitations. Interobserver and intraobserver variability can affect the consistency and reliability of diagnostic results. Furthermore, with high-resolution whole-slide images (WSIs) and complex tumor tissue architecture, traditional methods struggle to achieve standardization and quantitative analysis.

[0003] With the rapid development of digital pathology and artificial intelligence technologies, the use of deep learning algorithms for HER2 scoring can, to a certain extent, overcome the problems of traditional methods. Currently, there are three mainstream technologies: the first is based on small slice (patch) classification. This method divides the WSI into smaller patches and then uses machine learning or deep learning algorithms to extract features and classify each patch. This method can handle complex cellular structures and staining patterns, but requires a large amount of small patch image annotation data to train the model. The second is based on multi-instance learning (MIL), which aims to address the lack of fine-grained annotation for HER2 scoring. This method uses the annotation of the entire WSI image and combines it with the multi-instance learning paradigm to extract a holistic representation of the pathology image. Talha Qaiser et al. (Talha Qaiser and Nasir M Rajpoot. “Learning where to see: a novel attention model for automated immunohistochemical scoring”. In: IEEE transactions on medical imaging 38.11 (2019), pp. 2620–2631.) introduced reinforcement learning to the HER2 scoring task. By mimicking the way pathologists analyze tissue sections, they were able to intelligently select the most diagnostically valuable regions from large, whole-slide images for analysis, rather than processing all sub-image patches to predict HER2 scores. Wenqi Lu et al. (Wenqi Lu et al. “SlideGraph+: Whole slide image level graphs to predict HER2 status in breast cancer”. In: Medical Image Analysis 80 (2022), p. 102486.) employed a graph neural network to construct node features using cell nucleus features, DAB density features, and deep features to predict HER2 scores directly from H&E-stained WSIs. In addition, SahanYorucSelcuk et al. (SahanYorucSelcuk et al. “Automated HER2 Scoring inBreast Cancer Images UsingDeep Learning and Pyramid Sampling”. In: BMEF (BMEFrontiers) (2024).) proposed a pyramid sampling method to introduce low-resolution images, thereby focusing on large-scale features such as tissue hierarchical structure and tumor area to help more accurately predict HER2 scores.The last type is segmentation-based methods, which focus on improving the interpretability of scores by accurately identifying and segmenting regions of interest, such as cell nuclei and tumor regions; this method can analyze the characteristics of specific structures in more detail, thereby providing more accurate biomarker expression assessments.

[0004] However, each of the three existing methods has its own shortcomings. The first, based on a patch classifier, requires annotations for each patch before classifier training, and this process relies on the participation of professional pathologists. However, this detailed annotation is extremely costly, especially given the scarcity of qualified pathologists. The second, based on the MIL, can overcome the high cost of pathology image annotation to some extent. However, because MIL is trained based on the annotations of the entire WSI as supervision information, it requires a large amount of labeled data to achieve good results, and the results are often difficult to interpret. Therefore, for the HER2 score prediction task, the number of samples in the existing publicly available, rigorously annotated HER2 score datasets is very small, and MIL models trained on such datasets often suffer from poor generalization performance. Finally, segmentation-based methods also require a significant amount of resources to provide detailed tissue region and cell nucleus annotations, even more than the first method, because a patch image often contains multiple cell nuclei, and providing annotations for each nucleus is more time-consuming. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a breast cancer HER2 prediction method based on multiple prompt learning. The method aims to use the visual description of the pathology prior to guide the aggregation of image features, simulate the actual situation of pathologists when performing the HER2 scoring task, and help improve the accuracy of HER2 scoring.

[0006] To achieve the above object, the present invention adopts a breast cancer HER2 prediction method based on multiple prompt learning, comprising the following steps:

[0007] Obtain breast cancer WSI pathology images and divide them into multiple patches. Use an image encoder to extract image features of breast cancer WSI pathology images.

[0008] A large language model is used to generate instance-level category descriptive text and instance-level staining degree descriptive text; slice-level descriptive text is generated by referring to and adopting descriptions in standard clinical assessment guidelines;

[0009] The instance-level category descriptive text, instance-level coloring degree descriptive text, slice-level descriptive text and their corresponding category tokens and learnable prompts are input into the multimodal pre-trained model for assembly. The instance-level category prompt, instance-level coloring degree prompt and slice-level prompt are obtained through word segmentation and word embedding processing.

[0010] Use a text encoder to perform integration operations on instance-level category hints, instance-level coloring hints, and slice-level hints to obtain instance-level category hint features, instance-level coloring hint features, and slice-level hint features;

[0011] The instance-level category hint features and image features, as well as the instance-level coloration hint features and image features are respectively input into the text-guided feature aggregation module for aggregation to obtain the category global visual features and the coloration global visual features;

[0012] The category global visual features and the staining degree global visual features are input into the global feature fusion module for integration to obtain the HER2 features of the breast cancer WSI pathology image;

[0013] The cosine similarity between the slice-level cue features and the HER2 features of breast cancer WSI pathology images was calculated to obtain the HER2 prediction results of breast cancer WSI pathology images and optimized using cross-entropy loss.

[0014] As a preferred technical solution, the obtaining of instance-level category hints, instance-level staining degree hints, and slice-level hints is specifically as follows:

[0015] Obtain the category tokens corresponding to the instance-level category descriptive text, instance-level coloring degree descriptive text, and slice-level descriptive text respectively and assemble them to obtain the instance-level category descriptive text, instance-level coloring degree descriptive text, and slice-level descriptive text with category information;

[0016] The learnable prompts corresponding to the instance-level category descriptive text, instance-level staining degree descriptive text, and slice-level descriptive text are initialized by fixed templates respectively;

[0017] The instance-level category descriptive text with category information, the instance-level coloring degree descriptive text and the slice-level descriptive text are segmented, and the corresponding learnable prompts are word embedded to obtain instance-level category prompts, instance-level coloring degree prompts and slice-level prompts.

[0018] As a preferred technical solution, the execution integration operation is specifically as follows:

[0019] Use a text encoder to encode instance-level category hints, instance-level coloring hints, and slice-level hints respectively to obtain instance-level category hint encoding, instance-level coloring hint encoding, and slice-level hint encoding;

[0020] The instance-level category hint coding, instance-level coloring hint coding and slice-level hint coding are integrated respectively. k The average operation is performed on the dimensions to obtain instance-level category hint features, instance-level staining degree hint features, and slice-level hint features.

[0021] As a preferred technical solution, the text-guided feature aggregation module includes a cross-attention layer and an output layer;

[0022] The obtained category global visual features and staining degree global visual features are specifically:

[0023] The instance-level category hint feature and image feature, the instance-level coloring hint feature and image feature are input into the cross attention layer to obtain the first attention feature and the second attention feature;

[0024] The instance-level category hint feature and the first attention feature, the instance-level coloring degree hint feature and the second attention feature are respectively sent to the output layer to obtain the category global visual feature and the coloring degree global visual feature.

[0025] As a preferred technical solution, the global feature fusion module includes an input layer, a cross attention layer, a mean layer and an output layer;

[0026] The HER2 features of the WSI pathological image of breast cancer are obtained as follows:

[0027] The category global visual feature and the staining intensity global visual feature are sent to the input layer and spliced ​​in the quantity dimension to obtain the global visual feature;

[0028] A new learnable category query token is defined for the slice-level hint feature, and is input into the cross-attention layer with the global visual feature to obtain the third attention feature;

[0029] Use the mean layer to average the global visual features in the first dimension to obtain the global average visual features;

[0030] The third attention feature and the global average visual feature are input into the output layer to generate the HER2 feature of the breast cancer WSI pathological image.

[0031] As a preferred technical solution, the HER2 prediction result of the breast cancer WSI pathological image is obtained as follows:

[0032] Calculate the cosine similarity between the slice-level hint features and the HER2 features of breast cancer WSI pathology images;

[0033] The HER2 classification probability of breast cancer WSI pathology images was calculated based on cosine similarity, and the HER2 prediction results of breast cancer WSI pathology images were obtained;

[0034] The parameters of the learnable hints, text-guided feature aggregation module, and global feature fusion module are optimized using cross-entropy loss.

[0035] Another aspect of the present invention provides a breast cancer HER2 prediction system based on multiple prompt learning, which is applied to the above-mentioned breast cancer HER2 prediction method, and includes an image feature extraction module, a description text generation module, a multiple prompt generation module, a multiple prompt representation module, a multiple prompt aggregation module, a multiple prompt integration module and a HER2 prediction module;

[0036] The image feature extraction module is used to obtain a breast cancer WSI pathology image and divide it into multiple patches, and use an image encoder to extract image features of the breast cancer WSI pathology image;

[0037] The descriptive text generation module is used to generate instance-level category descriptive text and instance-level staining degree descriptive text using a large language model; and generate slice-level descriptive text by referring to and using descriptions in standard clinical assessment guidelines;

[0038] The multiple prompt generation module is used to assemble instance-level category descriptive text, instance-level coloring degree descriptive text, slice-level descriptive text and their corresponding category tokens, and learnable prompts into a multimodal pre-training model, and obtain instance-level category prompts, instance-level coloring degree prompts and slice-level prompts through word segmentation and word embedding processing;

[0039] The multiple prompt representation module is used to use a text encoder to perform integration operations on instance-level category prompts, instance-level color degree prompts, and slice-level prompts to obtain instance-level category prompt features, instance-level color degree prompt features, and slice-level prompt features;

[0040] The multiple prompt aggregation module is used to input the instance-level category prompt feature and image feature, the instance-level color degree prompt feature and image feature into the text-guided feature aggregation module for aggregation, so as to obtain the category global visual feature and the color degree global visual feature;

[0041] The multiple prompt integration module is used to input the category global visual features and the staining degree global visual features into the global feature fusion module for integration to obtain the HER2 features of the breast cancer WSI pathology image;

[0042] The HER2 prediction module is used to calculate the cosine similarity between the slice-level prompt features and the HER2 features of the breast cancer WSI pathology image, obtain the HER2 prediction results of the breast cancer WSI pathology image and optimize them using the cross entropy loss.

[0043] In another aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the above-mentioned breast cancer HER2 prediction method.

[0044] Another aspect of the present invention provides an electronic device, comprising:

[0045] at least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores computer program instructions that can be executed by the at least one processor. The computer program instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned breast cancer HER2 prediction method.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] 1. The method of the present invention can be trained in a parameter-efficient manner, reducing dependence on large-scale datasets and improving the generalization performance of the model. The present invention uses a pre-trained large language model to generate multiple prompt descriptive text, then uses a multimodal pre-trained model to embed learnable prompts into the text to generate multiple prompts. At the same time, a text-guided feature aggregation module is designed to aggregate WSI-level visual features related to the text. Finally, a global feature fusion module is used to integrate the WSI-level visual features related to category and staining degree to obtain a WSI-level HER2 feature representation.

[0049] 2. The method of the present invention can effectively capture semantic information related to text descriptions, imitate the process of pathologists evaluating HER2, guide the model to focus on tissue areas of different types and staining degrees through instance-level category hints and instance-level staining degree hints, and use the intermediate calculated attention matrix of text hint features and image features to generate heat maps, thereby improving the interpretability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1The figure is a flow chart of a breast cancer HER2 prediction method based on multiple prompt learning according to an embodiment of the present invention.

[0052] Figure 2 This is a flowchart of a breast cancer HER2 prediction method based on multiple prompt learning according to an embodiment of the present invention.

[0053] Figure 3 This is a schematic diagram of the structure of a breast cancer HER2 prediction system based on multiple prompt learning according to an embodiment of the present invention.

[0054] Figure 4 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0057] The method proposed in this paper aims to use pathology prior visual descriptions to guide the aggregation of image features, simulating the actual situation of pathologists when performing HER2 scoring tasks; these visual descriptions are generated by a large language model (LLM) and contain morphological information related to tissue region classification and staining degree. Through these descriptions, the model can better focus on visual features relevant to the task. Figure 1 、 Figure 2 In one embodiment of the present application, a method for predicting HER2 in breast cancer using multiple prompt learning is provided, comprising the following steps:

[0058] S1. Obtain breast cancer WSI pathology images and divide them into multiple patches (Patches) using an image encoder E I Extracting image features from breast cancer WSI pathology images ,in In For the n A patch, M is the number of patches segmented from the breast cancer WSI pathology image, d is the feature dimension. In this embodiment, the image encoder E I The ViT-B / 32 (Vision Transformer) structure in the pre-trained CLIP model is used for implementation. The parameters of the image encoder need to be frozen during the operation ( Figure 2 snowflake logo).

[0059] S2. Use a large language model to generate instance-level category descriptive text and instance-level staining degree descriptive text; refer to and use the descriptions in standard clinical assessment guidelines to generate slice-level descriptive text.

[0060] In this example, the large language model uses GPT-4 to generate instance-level category descriptive text and instance-level staining level descriptive text. Specifically, through question-and-answer interaction with GPT-4, for example, "Please provide visual descriptions of five HER2 immunohistochemistry pathology images, one for each tissue category and staining level," multiple descriptive texts are generated to increase diversity and richness. For slide-level descriptive text, reference is made to and adapted from the descriptions in the standard clinical assessment guideline (Antonio C Wolff et al. "Human epidermal growth factor receptor 2 testing in breast cancer: American Society of Clinical Oncology / College of American Pathologist sclinical practice guideline focused update." In: Archives of pathology & laboratory medicine 142.11 (2018), pp. 1364–1382.).

[0061] S3. The instance-level category descriptive text, instance-level coloring degree descriptive text, slice-level descriptive text and their corresponding category tokens and learnable prompts are input into the multimodal pre-training model for assembly, and instance-level category prompts, instance-level coloring degree prompts and slice-level prompts are obtained through word segmentation and word embedding processing.

[0062] Furthermore, the present invention uses a multimodal pre-training model to generate multiple prompts, specifically:

[0063] S3.1. Obtain the category tokens corresponding to the instance-level category descriptive text, the instance-level staining degree descriptive text, and the slice-level descriptive text respectively and assemble them to obtain the instance-level category descriptive text, the instance-level staining degree descriptive text, and the slice-level descriptive text with category information. For example, assemble according to the following template: "An immunohistochemical pathological image of <cls>, which is <description>", where cls is the category token and description is the descriptive text. The category tokens for instance-level category descriptive text include Normal, Tumor, Lymphocyte, and Stroma; the category tokens for instance-level staining degree descriptive text include None, Weak, Moderate, and Strong; and the category tokens for slice-level descriptive text include HER2 scores, which include 0, 1+, 2+, and 3+.

[0064] S3.2. Initialize the learnable hints corresponding to the instance-level category descriptive text, instance-level staining degree descriptive text, and slice-level descriptive text through fixed templates p In this embodiment, the fixed template adopts the above-mentioned assembly template.

[0065] S3.3. Perform word segmentation on the instance-level category descriptive text, instance-level coloring degree descriptive text, and slice-level descriptive text with category information, and perform word embedding on the corresponding learnable hints to obtain instance-level category hints. , instance-level coloring hint and slice-level hints ,in N c is the number of category tokens corresponding to the instance-level category descriptive text, N s is the number of category tokens corresponding to the instance-level coloring descriptive text, k is the number of corresponding descriptive texts, L is the length of the corresponding prompt text, d is the embedding dimension. The prompt corresponding to each descriptive text can be expressed as:

[0066] T =[ p 1, p 2, ..., p m ; cls ; x 1, x 2, ..., x n ],

[0067] in, p 1, p 2, ..., p m represents a learnable hint corresponding to the descriptive text, m is the number of learnable cues, cls is the category token corresponding to the descriptive text, x 1, x 2, ..., x n is the word embedding obtained by converting the corresponding descriptive text, n is the number of word embeddings, while satisfying m + n +1= L .

[0068] In this embodiment, the multimodal pre-trained model uses the CLIP model pre-trained on 400 million image-text pairs to generate multiple prompts, where k =5, L =77.

[0069] S4. Use the text encoder to perform integration operations on instance-level category hints, instance-level coloring degree hints, and slice-level hints to obtain instance-level category hint features, instance-level coloring degree hint features, and slice-level hint features.

[0070] In order to obtain a more comprehensive and complete semantic representation, an integration operation is performed on the generated multiple prompts, specifically:

[0071] S4.1. Using a Text Encoder E T Encode the instance-level category hint, instance-level staining hint, and slice-level hint respectively to obtain the instance-level category hint encoding , instance-level coloring hint encoding and slice-level hint encoding In this embodiment, the text encoder E T Use the Transformer structure in the pre-trained CLIP model.

[0072] S4.2. Ensemble the instance-level category hint coding, instance-level coloring hint coding, and slice-level hint coding. k Perform average operation on the dimension to obtain instance-level category hint features , instance-level staining level hint features and slice-level hint features , l c Represents the category in the slice-level hint c Among them, , mean () represents the average operation, which is an integration method.

[0073] S5. Input the instance-level category hint feature and image feature, the instance-level coloring degree hint feature and image feature into the text-guided feature aggregation module respectively for aggregation to obtain the category global visual feature and the coloring degree global visual feature.

[0074] However, there are still significant differences between the initialized IHC pathology image features and the prompt features; the method of calculating the aggregation weights by the dot product of image and text features may introduce additional interference, thereby weakening the performance of the prompt-based pooling strategy. Therefore, the present invention designs a text-guided feature aggregation module (Context-guided Feature Aggregator, CFA), which is a variant structure based on Transformer, which uses prompt features to guide image features to aggregate into prompt-related global visual features, including a cross-attention layer and an output layer; wherein the cross-attention layer is used to calculate the cross-attention between the prompt features and the image features to generate attention features; the output layer is used to add the attention features to the prompt features, and obtain the global visual features through the MLP layer and the residual connection. Thus, the steps of obtaining the category global visual features and the staining degree global visual features are specifically as follows:

[0075] S5.1. Use the cross attention layer to calculate the cross attention between the instance-level category hint feature and the image feature, and between the instance-level coloring hint feature and the image feature, to obtain the first attention feature and the second attention feature. The process is described as follows:

[0076] ,

[0077] ,

[0078] in, is the first attention feature, is the second attention feature, is the learnable projection layer parameter matrix, initialized to the identity matrix , to prevent the image features and hint features extracted by the image encoder from being destroyed in the initial stage.

[0079] S5.2. Input the instance-level category hint feature and the first attention feature, the instance-level coloring hint feature and the second attention feature into the output layer respectively to obtain the category global visual feature and the coloring global visual feature. The process is expressed as:

[0080] ,

[0081] ,

[0082] in, X cls is the category global visual feature, X sin is the global visual feature of the degree of staining, MLP It is a multi-layer perceptron.

[0083] S6. Input the category global visual features and the staining degree global visual features into the global feature fusion module for integration to obtain the HER2 features of the breast cancer WSI pathology image.

[0084] Since the HER2 score focuses on the staining expression within the invasive cancer area, the present invention designs an effective global feature fusion module (Global Feature Fusion, GFF) to integrate the global visual features of each category and each staining degree into the HER2 feature of WSI. The global feature fusion module (GFF) includes an input layer, a cross-attention layer, a mean layer, and an output layer. The input layer is used to splice the global visual features of each category and each staining degree in the quantitative dimension to obtain the global visual feature. The cross-attention layer is used to perform cross-attention calculations on the learnable category query token and the global visual feature to obtain the attention feature. The mean layer is used to average the global visual feature in the quantitative dimension to obtain the average feature. The output layer is used to add the attention feature and the average feature, and obtain the HER2 feature of WSI through the MLP layer and residual connection. Therefore, the process of obtaining the HER2 feature of breast cancer WSI pathology image is as follows:

[0085] S6.1. First, the category global visual feature and the coloring degree global visual feature are sent to the input layer and spliced ​​in the quantity dimension to obtain the global visual feature .

[0086] S6.2. Then, a new learnable category query token is defined for the slice-level hint feature. , and the global visual feature input cross attention layer into the interaction of global visual features to obtain the third attention matrix; the process is described as:

[0087] ,

[0088] in, is the third attention matrix.

[0089] S6.3. Using the mean layer to measure global visual features X G In the first dimension (i.e. N c + N s dimension) and take the average value to get the global average visual feature mean ( X G ).

[0090] S6.4. Finally, the third attention feature The global average visual feature is input and output layer, and after addition, it passes through the MLP layer to generate the HER2 feature of the breast cancer WSI pathology image. The process is described as follows:

[0091] ,

[0092] in, HER2 features of breast cancer WSI pathological images, It is the category token in the category global visual feature.

[0093] S7. Calculate the cosine similarity between the slice-level cue features and the HER2 features of the breast cancer WSI pathology image, obtain the HER2 prediction results of the breast cancer WSI pathology image, and optimize them using the cross entropy loss.

[0094] Specifically, the process of step S7 is:

[0095] S7.1. First, calculate the slice-level hint features HER2 features in WSI pathological images of breast cancer x Cosine similarity of cos ( l c , x ), cos () is the cosine similarity function.

[0096] S7.2. Next, the HER2 classification probability of the breast cancer WSI pathology image is calculated based on the cosine similarity, and the HER2 prediction result of the breast cancer WSI pathology image is obtained, which is expressed as:

[0097] ,

[0098] in, p ( y = c | x ) indicates HER2 signature x Medium category c The probability of τ is the temperature coefficient, used to control the slice-level hint feature HER2 signature x Scale of similarity.

[0099] S7.3. Use cross-entropy loss to optimize the parameters of the learnable prompt, text-guided feature aggregation module, and global feature fusion module. The cross-entropy loss function is: L = - logp ( y = c | x ).

[0100] The method designed in this paper was tested on the private ZJH-HER2 dataset and the public HER2C dataset (Talha Qaiser et al. "Her 2 challenge contest: a detailed assessment of automated her 2 scoring algorithms in whole slide images of breast cancer tissues". In: Histopathology 72.2 (2018), pp. 227–238). Five MIL-based pathology image classification methods were selected for comparison:

[0101] ABMIL (Maximilian Ilse, JakubTomczak, and Max Welling. "Attention-based deep multiple instancelearning". In: International conference onmachine learning. PMLR. 2018, pp. 2127–2136) is an early and well-known attention-based MIL method that aggregates image features by utilizing the attention mechanism.

[0102] CLAM (Ming Y Lu et al. "Data-efficient and weakly supervised computational pathology on whole-slide images". In: Nature biomedicalengineering 5.6 (2021), pp. 555–570.) extends ABMIL to multi-category problems and introduces instance-level clustering constraints to optimize the feature space.

[0103] TransMIL (Zhuchen Shao et al. "Transmil: Transformer based correlatedmultiple instance learning for whole slide imageclassification". In: Advancesin neural information processing sys tems 34 (2021), pp. 2136–2147.) applies the self-attention mechanism to the pathological image classification task, further improving the performance of the MIL-based method.

[0104] TOP (Linhao Quetal. "The rise of ai language pathologists: Exploring two-level prompt learning ing for few-shot weakly-supervised whole slide image classification". In: Advances in Neural Information Processing Systems 36(2024).) uses a pre-trained vision-language model to incorporate prior textual information into the image feature aggregation process, providing richer semantic information, especially improving performance in small sample scenarios.

[0105] ViLa-MIL (Jiangbo Shi et al. "ViLa-MIL: Dual-scale Vision-LanguageMultiple Instance Learning for Whole Slide Image Classification". In:Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 11248–11258.) further enhances the performance of vision-language models by using dual-scale visual description text cues based on a frozen large language model (LLM).

[0106] To adapt TOP and ViLa-MIL to the HER2 scoring task, their original prompt templates were modified in this experiment. However, both methods still follow the methods described in their respective papers, using LLM to generate prior text descriptions. To maintain consistency with the ViLa-MIL method, image features were extracted at 5× and 10× magnification. The final results are shown in Table 1:

[0107] Table 1 Experimental results

[0108]

[0109] In this experiment, the performance of each method was evaluated using the ACC and F1-Score metrics. As shown in Table 1, compared to five existing methods, our method (Ours) achieved an ACC of 79.2% and an F1-Score of 77.6% on the ZJH-HER2 dataset, and 85.7% and 87.1% on the HER2C dataset, significantly outperforming the comparison methods. Notably, the cue-learning-based method (using CLIP-I and CLIP-I&T as the backbone network) outperformed the traditional attention-based MIL method (using a ResNet50 backbone network). This is primarily due to the text-guided image feature aggregation module, which injects rich semantic context by incorporating prior textual information and further aligns information between different modalities. Although TOP and ViLa-MIL also utilize text cues for feature aggregation, our method performs better. This improvement is attributed to our proposed multi-cue learning strategy, which decouples category cues from staining intensity cues, guiding the model to more precisely focus on regions of varying tissue types and staining intensities. In addition, this method also combines a global feature fusion module to combine staining degree information with category information, effectively simulating the decision-making process of pathologists when scoring.

[0110] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0111] Based on the same concept as the HER2 prediction method for breast cancer using multiple prompts learning in the above-mentioned embodiment, the present invention also provides a HER2 prediction system for breast cancer using multiple prompts learning, which can be used to implement the above-mentioned HER2 prediction method for breast cancer using multiple prompts learning. For ease of explanation, the schematic diagram of the structure of the embodiment of the HER2 prediction system for breast cancer using multiple prompts learning only shows the parts relevant to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and the device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0112] See also Figure 3 In another embodiment of the present application, a breast cancer HER2 prediction system based on multiple prompt learning is provided, the system comprising an image feature extraction module, a description text generation module, a multiple prompt generation module, a multiple prompt representation module, a multiple prompt aggregation module, a multiple prompt integration module and a HER2 prediction module;

[0113] The image feature extraction module is used to obtain breast cancer WSI pathology images and divide them into multiple patches, and use an image encoder to extract image features of breast cancer WSI pathology images;

[0114] The descriptive text generation module is used to generate instance-level category descriptive text and instance-level staining degree descriptive text using a large language model; and to generate slice-level descriptive text by referring to and adopting descriptions in standard clinical assessment guidelines;

[0115] The multi-cue generation module is used to assemble instance-level category descriptive text, instance-level coloring degree descriptive text, slice-level descriptive text and their corresponding category tokens, as well as learnable cues, into a multimodal pre-trained model. The module then obtains instance-level category cues, instance-level coloring degree cues, and slice-level cues through word segmentation and word embedding.

[0116] The multiple prompt representation module is used to use the text encoder to perform integration operations on instance-level category prompts, instance-level coloring degree prompts, and slice-level prompts to obtain instance-level category prompt features, instance-level coloring degree prompt features, and slice-level prompt features;

[0117] The multi-cue aggregation module is used to input instance-level category cue features and image features, instance-level coloring degree cue features and image features into the text-guided feature aggregation module for aggregation, thereby obtaining category global visual features and coloring degree global visual features.

[0118] The multiple prompt integration module is used to input the category global visual features and the staining degree global visual features into the global feature fusion module for integration to obtain the HER2 features of breast cancer WSI pathology images;

[0119] The HER2 prediction module is used to calculate the cosine similarity between the slice-level cue features and the HER2 features of breast cancer WSI pathology images, obtain the HER2 prediction results of breast cancer WSI pathology images and optimize them using cross-entropy loss.

[0120] It should be noted that the breast cancer HER2 prediction system with multiple prompt learning of the present invention corresponds one-to-one to the breast cancer HER2 prediction method with multiple prompt learning of the present invention. The technical features and beneficial effects described in the embodiment of the above-mentioned breast cancer HER2 prediction method with multiple prompt learning are applicable to the embodiment of breast cancer HER2 prediction with multiple prompt learning. For specific details, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0121] Furthermore, in the implementation of the HER2 prediction system for breast cancer using multiple prompts learning described in the above embodiment, the logical division of the various program modules is merely illustrative. In actual applications, the above functions may be assigned to different program modules as needed, for example, based on the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the HER2 prediction system for breast cancer using multiple prompts learning may be divided into different program modules to complete all or part of the functions described above.

[0122] See also Figure 4 In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a program is stored. When the program is executed by a processor, the above-mentioned breast cancer HER2 prediction method based on multiple prompt learning can be implemented, specifically:

[0123] Obtain breast cancer WSI pathology images and divide them into multiple patches. Use an image encoder to extract image features of breast cancer WSI pathology images.

[0124] A large language model is used to generate instance-level category descriptive text and instance-level staining degree descriptive text; slice-level descriptive text is generated by referring to and adopting descriptions in standard clinical assessment guidelines;

[0125] The instance-level category descriptive text, instance-level coloring degree descriptive text, slice-level descriptive text and their corresponding category tokens and learnable prompts are input into the multimodal pre-trained model for assembly. The instance-level category prompt, instance-level coloring degree prompt and slice-level prompt are obtained through word segmentation and word embedding processing.

[0126] Use a text encoder to perform integration operations on instance-level category hints, instance-level coloring hints, and slice-level hints to obtain instance-level category hint features, instance-level coloring hint features, and slice-level hint features;

[0127] The instance-level category hint features and image features, as well as the instance-level coloration hint features and image features are respectively input into the text-guided feature aggregation module for aggregation to obtain the category global visual features and the coloration global visual features;

[0128] The category global visual features and the staining degree global visual features are input into the global feature fusion module for integration to obtain the HER2 features of the breast cancer WSI pathology image;

[0129] The cosine similarity between the slice-level cue features and the HER2 features of breast cancer WSI pathology images was calculated to obtain the HER2 prediction results of breast cancer WSI pathology images and optimized using cross-entropy loss.

[0130] In another embodiment, the present application provides an electronic device for implementing the above-mentioned method for predicting HER2 levels in breast cancer using multiple prompt learning, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores computer program instructions executable by the at least one processor, and when the computer program instructions are executed by the at least one processor, the at least one processor is enabled to perform the above-mentioned method for predicting HER2 levels in breast cancer using multiple prompt learning, specifically:

[0131] Obtain breast cancer WSI pathology images and divide them into multiple patches. Use an image encoder to extract image features of breast cancer WSI pathology images.

[0132] A large language model is used to generate instance-level category descriptive text and instance-level staining degree descriptive text; slice-level descriptive text is generated by referring to and adopting descriptions in standard clinical assessment guidelines;

[0133] The instance-level category descriptive text, instance-level coloring degree descriptive text, slice-level descriptive text and their corresponding category tokens and learnable prompts are input into the multimodal pre-trained model for assembly. The instance-level category prompt, instance-level coloring degree prompt and slice-level prompt are obtained through word segmentation and word embedding processing.

[0134] Use a text encoder to perform integration operations on instance-level category hints, instance-level coloring hints, and slice-level hints to obtain instance-level category hint features, instance-level coloring hint features, and slice-level hint features;

[0135] The instance-level category hint features and image features, as well as the instance-level coloration hint features and image features are respectively input into the text-guided feature aggregation module for aggregation to obtain the category global visual features and the coloration global visual features;

[0136] The category global visual features and the staining degree global visual features are input into the global feature fusion module for integration to obtain the HER2 features of the breast cancer WSI pathology image;

[0137] The cosine similarity between the slice-level cue features and the HER2 features of breast cancer WSI pathology images was calculated to obtain the HER2 prediction results of breast cancer WSI pathology images and optimized using cross-entropy loss.

[0138] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0139] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.< / description> < / cls>

Claims

1. A breast cancer HER2 prediction method based on multiple prompt learning, characterized in that: The steps include: Obtain breast cancer WSI pathology images and divide them into multiple patches. Use an image encoder to extract image features of breast cancer WSI pathology images. A large language model is used to generate instance-level category descriptive text and instance-level staining degree descriptive text; slice-level descriptive text is generated by referring to and adopting descriptions in standard clinical assessment guidelines; The instance-level category descriptive text, instance-level coloring degree descriptive text, slice-level descriptive text and their corresponding category tokens and learnable prompts are input into the multimodal pre-trained model for assembly. The instance-level category prompt, instance-level coloring degree prompt and slice-level prompt are obtained through word segmentation and word embedding processing. The obtaining of instance-level category hints, instance-level staining degree hints, and slice-level hints is specifically as follows: Obtain the category tokens corresponding to the instance-level category descriptive text, instance-level coloring degree descriptive text, and slice-level descriptive text respectively and assemble them to obtain the instance-level category descriptive text, instance-level coloring degree descriptive text, and slice-level descriptive text with category information; The learnable prompts corresponding to the instance-level category descriptive text, instance-level staining degree descriptive text, and slice-level descriptive text are initialized by fixed templates respectively; The instance-level category descriptive text, instance-level coloring degree descriptive text, and slice-level descriptive text with category information are segmented, and the corresponding learnable prompts are word-embedded to obtain instance-level category prompts, instance-level coloring degree prompts, and slice-level prompts; Use a text encoder to perform integration operations on instance-level category hints, instance-level coloring hints, and slice-level hints to obtain instance-level category hint features, instance-level coloring hint features, and slice-level hint features; The instance-level category hint features and image features, as well as the instance-level coloration hint features and image features are respectively input into the text-guided feature aggregation module for aggregation to obtain the category global visual features and the coloration global visual features; The category global visual features and the staining degree global visual features are input into the global feature fusion module for integration to obtain the HER2 features of the breast cancer WSI pathology image; The cosine similarity between the slice-level cue features and the HER2 features of breast cancer WSI pathology images was calculated to obtain the HER2 prediction results of breast cancer WSI pathology images and optimized using cross-entropy loss.

2. The method for predicting HER2 in breast cancer according to claim 1, wherein The execution integration operation is specifically as follows: Use a text encoder to encode instance-level category hints, instance-level coloring hints, and slice-level hints respectively to obtain instance-level category hint encoding, instance-level coloring hint encoding, and slice-level hint encoding; The instance-level category hint coding, instance-level coloring hint coding and slice-level hint coding are integrated respectively. k The average operation is performed on the dimensions to obtain instance-level category hint features, instance-level staining degree hint features, and slice-level hint features.

3. The method for predicting HER2 in breast cancer according to claim 1, wherein The text-guided feature aggregation module includes a cross-attention layer and an output layer; The obtained category global visual features and staining degree global visual features are specifically: The instance-level category hint feature and image feature, the instance-level coloring hint feature and image feature are input into the cross attention layer to obtain the first attention feature and the second attention feature; The instance-level category hint feature and the first attention feature, the instance-level coloring degree hint feature and the second attention feature are respectively sent to the output layer to obtain the category global visual feature and the coloring degree global visual feature.

4. The method for predicting HER2 in breast cancer according to claim 1, wherein The global feature fusion module includes an input layer, a cross attention layer, a mean layer and an output layer; The HER2 features of the WSI pathological image of breast cancer are obtained as follows: The category global visual feature and the staining intensity global visual feature are sent to the input layer and spliced ​​in the quantity dimension to obtain the global visual feature; A new learnable category query token is defined for the slice-level hint feature, and is input into the cross-attention layer with the global visual feature to obtain the third attention feature; Use the mean layer to average the global visual features in the first dimension to obtain the global average visual features; The third attention feature and the global average visual feature are input into the output layer to generate the HER2 feature of the breast cancer WSI pathological image.

5. The method for predicting HER2 in breast cancer according to claim 1, wherein The HER2 prediction result of the breast cancer WSI pathological image is specifically obtained as follows: Calculate the cosine similarity between the slice-level hint features and the HER2 features of breast cancer WSI pathology images; The HER2 classification probability of breast cancer WSI pathology images was calculated based on cosine similarity, and the HER2 prediction results of breast cancer WSI pathology images were obtained; The parameters of the learnable hints, text-guided feature aggregation module, and global feature fusion module are optimized using cross-entropy loss.

6. A breast cancer HER2 prediction system based on multiple prompt learning, characterized in that: The breast cancer HER2 prediction method applied to any one of claims 1-5 comprises an image feature extraction module, a description text generation module, a multiple prompt generation module, a multiple prompt representation module, a multiple prompt aggregation module, a multiple prompt integration module and a HER2 prediction module; The image feature extraction module is used to obtain a breast cancer WSI pathology image and divide it into multiple patches, and use an image encoder to extract image features of the breast cancer WSI pathology image; The descriptive text generation module is used to generate instance-level category descriptive text and instance-level staining degree descriptive text using a large language model; and generate slice-level descriptive text by referring to and using descriptions in standard clinical assessment guidelines; The multiple prompt generation module is used to assemble instance-level category descriptive text, instance-level coloring degree descriptive text, slice-level descriptive text and their corresponding category tokens, and learnable prompts into a multimodal pre-training model, and obtain instance-level category prompts, instance-level coloring degree prompts and slice-level prompts through word segmentation and word embedding processing; The obtaining of instance-level category hints, instance-level staining degree hints, and slice-level hints is specifically as follows: Obtain the category tokens corresponding to the instance-level category descriptive text, instance-level coloring degree descriptive text, and slice-level descriptive text respectively and assemble them to obtain the instance-level category descriptive text, instance-level coloring degree descriptive text, and slice-level descriptive text with category information; The learnable prompts corresponding to the instance-level category descriptive text, instance-level staining degree descriptive text, and slice-level descriptive text are initialized by fixed templates respectively; The instance-level category descriptive text, instance-level coloring degree descriptive text, and slice-level descriptive text with category information are segmented, and the corresponding learnable prompts are word-embedded to obtain instance-level category prompts, instance-level coloring degree prompts, and slice-level prompts; The multiple prompt representation module is used to use a text encoder to perform integration operations on instance-level category prompts, instance-level color degree prompts, and slice-level prompts to obtain instance-level category prompt features, instance-level color degree prompt features, and slice-level prompt features; The multiple prompt aggregation module is used to input the instance-level category prompt feature and image feature, the instance-level color degree prompt feature and image feature into the text-guided feature aggregation module for aggregation, so as to obtain the category global visual feature and the color degree global visual feature; The multiple prompt integration module is used to input the category global visual features and the staining degree global visual features into the global feature fusion module for integration to obtain the HER2 features of the breast cancer WSI pathology image; The HER2 prediction module is used to calculate the cosine similarity between the slice-level prompt features and the HER2 features of the breast cancer WSI pathology image, obtain the HER2 prediction results of the breast cancer WSI pathology image and optimize them using the cross entropy loss.

7. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the breast cancer HER2 prediction method according to any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the breast cancer HER2 prediction method according to any one of claims 1 to 5.

Images