A tumor pathology analysis method and system based on multimodal artificial intelligence

Abstract

The present invention relates to the field of pathology analysis technology, and more specifically to a tumor pathology analysis method based on multimodal artificial intelligence, comprising the following steps: acquiring multi-center, multi-type whole-slice tissue images, cutting the whole-slice images into pathology images, and dividing the standardized pre-processed pathology images into training and test sets; constructing a contrastive language and image model for capturing key features in pathology images through a two-stage pre-training process; and fine-tuning the model using the pre-trained weights of the contrastive language and image model as initial weights and adding task-specific fully connected layers to obtain a pathology analysis model for implementing different classification and regression tasks in tumor pathology analysis. By integrating multimodal information from images and text, the present invention can more comprehensively assess tumor type, stage, and prognosis, thereby fully leveraging the advantages of multi-source data and effectively addressing the current bottlenecks in tumor pathology analysis technology.

Description

Technical Field

[0001] The present invention relates to the technical field of pathological diagnosis, and in particular to a tumor pathology analysis method and system based on multimodal artificial intelligence. Background Art

[0002] Tumor pathology diagnosis is a crucial step in cancer treatment, directly influencing the development of subsequent treatment plans. Currently, traditional pathology diagnosis relies on expert pathologists analyzing images of tumor tissue sections using microscopes and combining them with the patient's clinical data to make judgments. This method is not only time-consuming and labor-intensive, but also highly subjective and easily influenced by factors such as expert experience and fatigue, leading to uncertainty in diagnostic results. Therefore, improving the automation, accuracy, and efficiency of tumor pathology diagnosis has always been a major issue in medical research and clinical applications.

[0003] In recent years, deep learning technology has made significant progress in the field of medical image analysis. However, existing pathological diagnostic methods mostly focus on analyzing image data. However, tumor assessment often requires the integration of information from multiple modalities, such as genomic data, molecular markers, and clinical data. Traditional methods fail to effectively integrate this diverse information source, resulting in a lack of comprehensiveness and accuracy in tumor assessment. In particular, single image analysis methods often fail to provide sufficient support for tumor type, staging, and prognosis assessment.

[0004] Therefore, existing tumor pathology diagnosis technologies are unable to integrate information data from all aspects, and thus cannot obtain accurate and comprehensive tumor pathology assessment results. Summary of the Invention

[0005] The purpose of the present invention is to provide a tumor pathology analysis method and system based on multimodal artificial intelligence to solve the technical problem in the existing technology that tumor pathology diagnosis is difficult to integrate information data from all aspects, thereby failing to obtain accurate and comprehensive tumor pathology assessment results.

[0006] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0007] A tumor pathology analysis method based on multimodal artificial intelligence comprises the following steps:

[0008] By acquiring multi-center and multi-type whole-slice images of tissues, the whole-slice images are cut into pathological images, and the pathological images after standardized preprocessing are divided into a training set and a test set;

[0009] A comparative language and image model is constructed to capture key features in pathological images through a two-stage pre-training process. The comparative language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding. The image encoder uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning. Unlabeled data refers to whole-slice images of tissues that have not been manually annotated with pathological categories or pathological feature labels. The text encoder uses a text encoder from the comparative language and image pre-training model. This text encoder is a model based on the Transformers structure. Through weakly supervised learning and an attention mechanism, it integrates slice-level features into global pathological features of the whole-slice image, thereby constructing a holistic understanding of pathological images. Finally, multimodal data of microscopic imaging and anatomical site information is integrated to enhance the representation ability of anatomical information by combining image and text information.

[0010] Pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, use early stopping strategy to avoid overfitting, and maximize validation set performance by adjusting hyperparameters;

[0011] By using the pre-trained weights of the contrastive language and image models as the initial weights and adding task-specific fully connected layers, the model is fine-tuned to obtain a pathology analysis model for implementing different classification and regression tasks in tumor pathology analysis.

[0012] As a preferred solution of the present invention, the pathological image preprocessing method includes:

[0013] The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image;

[0014] The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images;

[0015] The resolution of each pathological image was set to 1.0 pixel / μm, and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

[0016] As a preferred embodiment of the present invention, the method for constructing the comparative language and image model includes:

[0017] Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the specimen;

[0018] Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , the anatomical part information The text embedding formula is:

[0019] ;

[0020] Where, is the fixed-dimensional text feature representation of T, where T is the anatomical part information, For text encoder;

[0021] Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , the text feature representation The optimization formula is:

[0022] ;

[0023] Where, is the optimized text feature representation, MLP is a multi-layer perceptron;

[0024] Through the image encoder From the whole slide image I Extract the image features of the whole slice image , the image features The extraction formula is:

[0025] ;

[0026] Where, is the image feature of the full slice image, I is the full slice image, is an image encoder;

[0027] The processed text features and image features Splicing and fusion to obtain comprehensive feature representation ;

[0028] Use the pooling strategy based on the attention mechanism to aggregate the contextual information of each pathological image and obtain the weighted feature representation of the whole slice image , the weighted feature representation of the whole slice image The weighted aggregation formula is: ;

[0029] in, ;

[0030] Where, is the weighted feature representation of the full slice image, is the jth pathological image feature obtained by cutting the full slice image, To pass and comprehensive feature representation The corresponding correlation measurement between The attention score, is the correlation measure operator.

[0031] As a preferred solution of the present invention, the method for adding a fully connected layer for a specific task includes:

[0032] The weighted feature representation of the full slice image Input to the fully connected layer of a specific task, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is:

[0033] ;

[0034] Where, is the prediction result of a specific task, is a fully connected layer.

[0035] As a preferred solution of the present invention, the specific tasks of the fully connected layer include: tumor origin prediction, gene mutation prediction and prognosis prediction, wherein the fully connected layer for tumor origin prediction is set as an 18-category weakly supervised classification task, the fully connected layer for gene mutation prediction is set as a binary classification task, and the fully connected layer for prognosis prediction is set as a regression model that outputs a death risk score.

[0036] As a preferred embodiment of the present invention, the weakly supervised learning and contrastive learning pre-training method of the contrastive language and image models includes:

[0037] Use a weakly supervised learning strategy, adopt the Adam optimizer, set the initial learning rate to 0.0003, and adjust the learning rate by the cosine annealing method;

[0038] Apply the early stopping strategy and terminate the training when the AUROC indicator of the validation set does not improve for 10 consecutive epochs;

[0039] AUROC, sensitivity, and specificity validation set performance indicators were calculated separately for each anatomical site, and hyperparameters were optimized to maximize the average AUROC indicator across anatomical sites.

[0040] As a preferred embodiment of the present invention, the present invention provides a tumor pathology analysis system based on multimodal artificial intelligence, which is applied to a tumor pathology analysis method based on multimodal artificial intelligence. The system includes:

[0041] A data acquisition and processing unit is used to acquire multi-center and multi-type whole-slice images of tissues, cut the whole-slice images into pathological images, and divide the standardized pre-processed pathological images into a training set and a test set;

[0042] A model building unit is used to build a comparative language and image model for capturing key features in pathological images through a two-stage pre-training process, wherein the comparative language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding. The image encoder uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning, wherein unlabeled data refers to whole-slice images of tissues that have not been manually labeled with pathological categories or pathological feature labels. The text encoder uses a text encoder from the comparative language and image pre-training model. The text encoder is a model based on the Transformers structure. Through weakly supervised learning and an attention mechanism, slice-level features are integrated into global pathological features of the whole-slice image to build a holistic understanding of the pathological image. Finally, multimodal data of microscopic imaging and anatomical site information are integrated to enhance the representation capability of anatomical information by combining image and text information.

[0043] Model training and optimization unit, used to pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, using early stopping strategy to avoid overfitting, and maximizing validation set performance by adjusting hyperparameters;

[0044] The model fine-tuning unit is used to fine-tune the model by using the pre-trained weights of the comparative language and image models as the initial weights and adding a fully connected layer for a specific task to obtain an analysis model for implementing different classification and regression tasks in tumor pathology analysis;

[0045] Among them, the pre-trained weights of the contrast language and image models refer to the weights obtained through one-stage self-supervised learning on large-scale unlabeled data. These weights serve as initial parameters to help the model quickly adapt to specific tasks, reduce dependence on labeled data, and improve training efficiency; the fine-tuning process includes adjustments to the model weights and structure, such as adding 18 categories of weakly supervised classifiers when predicting tumor origin, or replacing them with regression models when predicting tumor prognosis and adjusting the model weight values ​​through automated training; the model shares knowledge between different tasks by sharing one-stage pre-trained weights and feature layers, thereby improving the model's expressiveness and generalization capabilities.

[0046] As a preferred solution of the present invention, the preprocessing method of the pathological image by the data acquisition and processing unit includes:

[0047] The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image;

[0048] The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images;

[0049] The resolution of each pathological image was set to 1.0 pixel / μm, and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

[0050] As a preferred embodiment of the present invention, the method for constructing the comparative language and image models by the model construction unit includes:

[0051] Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the specimen;

[0052] Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , the anatomical part information The text embedding formula is:

[0053] ;

[0054] Where, for Fixed-dimensional text feature representation, For anatomical location information, For text encoder;

[0055] Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , the text feature representation The optimization formula is:

[0056] ;

[0057] Where, is the optimized text feature representation, MLP is a multi-layer perceptron;

[0058] Through the image encoder From the whole slide image I Extract the image features of the whole slice image , the image features The extraction formula is:

[0059] ;

[0060] Where, is the image feature of the full slice image, I is the full slice image, is an image encoder;

[0061] The processed text features and image features Splicing and fusion to obtain comprehensive feature representation :

[0062] Use the pooling strategy based on the attention mechanism to aggregate the contextual information of each pathological image and obtain the weighted feature representation of the whole slice image , the weighted feature representation of the whole slice image The weighted aggregation formula is:

[0063] ;

[0064] in, ;

[0065] Where, is the weighted feature representation of the full slice image, is the jth pathological image feature obtained by cutting the full slice image, To pass and comprehensive feature representation The corresponding correlation measurement between The attention score, is the correlation measure operator.

[0066] As a preferred solution of the present invention, the method for adding a fully connected layer to a specific task by the model fine-tuning unit includes:

[0067] The weighted feature representation of the full slice image Input to the fully connected layer of a specific task, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is:

[0068] ;

[0069] Where, is the prediction result of a specific task, is a fully connected layer.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] The present invention constructs a comparative language and image model to extract slice-level feature representations from unlabeled data through self-supervised learning, and integrates slice-level features into global pathological features of the whole slice image through weakly supervised learning and attention mechanism, thereby building a holistic understanding of pathological images and obtaining multimodal data that integrates microscopic imaging and anatomical site information. By combining image and text information, the representation ability of anatomical information is enhanced. By fusing the multimodal information of image and text, the type, stage and prognosis of the tumor can be more comprehensively evaluated, thereby fully utilizing the advantages of multi-source data. This is an effective way to solve the current bottleneck of tumor pathology analysis technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0073] Figure 1 This is a flow chart of a tumor pathology analysis method based on multimodal artificial intelligence provided by an embodiment of the present invention;

[0074] Figure 2 is a flow chart of a pathological image preprocessing method provided by an embodiment of the present invention;

[0075] Figure 3 Schematic diagram of the model structure of the tumor pathology analysis method based on multimodal artificial intelligence provided by an embodiment of the present invention;

[0076] Figure 4 This is a model training flow chart of a tumor pathology analysis method based on multimodal artificial intelligence provided by an embodiment of the present invention;

[0077] Figure 5 This is a block diagram of a tumor pathology analysis system based on multimodal artificial intelligence provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0079] like Figure 1 As shown, the present invention provides a tumor pathology analysis method based on multimodal artificial intelligence, comprising the following steps:

[0080] By acquiring multi-center and multi-type whole-slice images of tissues, the whole-slice images are cut into pathological images, and the standardized pre-processed pathological images are divided into training sets and test sets;

[0081] A comparative language and image model is constructed to capture key features in pathological images through a two-stage pre-training process. The comparative language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding. The image encoder uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning. Unlabeled data refers to whole-slice images of tissues that have not been manually annotated with pathological categories or pathological feature labels. The text encoder uses a text encoder from the comparative language and image pre-training model. This text encoder is a model based on the Transformers structure. Through weakly supervised learning and an attention mechanism, it integrates slice-level features into global pathological features of the whole-slice image, thereby constructing a holistic understanding of pathological images. Finally, multimodal data of microscopic imaging and anatomical site information is integrated to enhance the representation ability of anatomical information by combining image and text information.

[0082] Pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, use early stopping strategy to avoid overfitting, and maximize validation set performance by adjusting hyperparameters;

[0083] By using the pre-trained weights of the contrastive language and image models as the initial weights and adding task-specific fully connected layers, the model is fine-tuned to obtain an analysis model for implementing different classification and regression tasks in tumor pathology analysis.

[0084] like Figure 2 As shown in Figure 2, the preprocessing methods for pathological images include:

[0085] The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image;

[0086] The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images;

[0087] The resolution of each pathological image was set to 1.0 pixel / μm, and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

[0088] Methods for building contrastive language and image models include:

[0089] Obtain anatomical location information for each full-slice image , anatomical information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the specimen;

[0090] Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , anatomical information The text embedding formula is:

[0091] ;

[0092] Where, is the fixed-dimensional text feature representation of T, where T is the anatomical part information, For text encoder;

[0093] Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , text feature representation The optimization formula is:

[0094] ;

[0095] Where, is the optimized text feature representation, MLP is a multi-layer perceptron;

[0096] Through the image encoder From the whole slide image I Extract the image features of the whole slice image , image features The extraction formula is:

[0097] ;

[0098] Where, is the image feature of the full slice image, I is the full slice image, is an image encoder;

[0099] The processed text features and image features Splicing and fusion to obtain comprehensive feature representation ;

[0100] Use the pooling strategy based on the attention mechanism to aggregate the contextual information of each pathological image and obtain the weighted feature representation of the whole slice image , weighted feature representation of full-slice images The weighted aggregation formula is:

[0101] ;

[0102] in, ;

[0103] Where, is the weighted feature representation of the full slice image, is the jth pathological image feature obtained by cutting the full slice image, To pass and comprehensive feature representation The corresponding correlation measurement between The attention score, is the correlation measure operator.

[0104] Methods for adding fully connected layers for specific tasks include:

[0105] The weighted feature representation of the full slice image Input to the fully connected layer of a specific task, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is:

[0106] ;

[0107] Where, is the prediction result of a specific task, is a fully connected layer.

[0108] The specific tasks of the fully connected layer include: tumor origin prediction, gene mutation prediction and prognosis prediction. Among them, the fully connected layer for tumor origin prediction is set as an 18-category weakly supervised classification task, the fully connected layer for gene mutation prediction is set as a binary classification task, and the fully connected layer for prognosis prediction is set as a regression model that outputs a death risk score.

[0109] like Figure 4 As shown, the weakly supervised learning and contrastive learning pre-training methods for contrastive language and image models include:

[0110] Use a weakly supervised learning strategy, adopt the Adam optimizer, set the initial learning rate to 0.0003, and adjust the learning rate by the cosine annealing method;

[0111] Apply the early stopping strategy and terminate the training when the AUROC indicator of the validation set does not improve for 10 consecutive epochs;

[0112] AUROC, sensitivity, and specificity validation set performance indicators were calculated separately for each anatomical site, and hyperparameters were optimized to maximize the average AUROC indicator across anatomical sites.

[0113] like Figure 5 As shown, the present invention provides a tumor pathology analysis system based on multimodal artificial intelligence, which is applied to a tumor pathology analysis method based on multimodal artificial intelligence. The system includes:

[0114] A data acquisition and processing unit is used to obtain multi-center and multi-type whole-slice images of tissues, cut the whole-slice images into pathological images, and divide the standardized pre-processed pathological images into a training set and a test set;

[0115] A model building unit is used to build a comparative language and image model for capturing key features in pathological images through a two-stage pre-training process, wherein the comparative language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding. The image encoder uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning, wherein unlabeled data refers to whole-slice images of tissues that have not been manually labeled with pathological categories or pathological feature labels. The text encoder uses a text encoder from the comparative language and image pre-training model. The text encoder is a model based on the Transformers structure. Through weakly supervised learning and an attention mechanism, slice-level features are integrated into global pathological features of the whole-slice image to build a holistic understanding of the pathological image. Finally, multimodal data of microscopic imaging and anatomical site information are integrated to enhance the representation capability of anatomical information by combining image and text information.

[0116] Model training and optimization unit, used to pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, using early stopping strategy to avoid overfitting, and maximizing validation set performance by adjusting hyperparameters;

[0117] The model fine-tuning unit is used to fine-tune the model by using the pre-trained weights of the comparative language and image models as the initial weights and adding task-specific fully connected layers to obtain an analysis model for different classification and regression tasks in tumor pathology analysis;

[0118] like Figure 3 As shown, adding 18 categories of weakly supervised classifiers to the language and image models can realize the model structure for predicting tumor origin, adding a regression model to the language and image models can realize the model structure for predicting tumor prognosis, and adding a binary classifier to the language and image models can realize the model structure for predicting gene mutation.

[0119] In contrast, the pre-trained weights of language and image models refer to the weights obtained through a one-stage self-supervised learning on large-scale unlabeled data. These weights serve as initial parameters to help the model quickly adapt to specific tasks, reduce dependence on labeled data, and improve training efficiency.

[0120] The fine-tuning process includes adjusting the model weights and structure, such as adding an 18-category weakly supervised classifier when predicting tumor origin, or replacing it with a regression model when predicting tumor prognosis and adjusting the model weight values ​​through automated training; the model shares knowledge between different tasks by sharing the first-stage pre-training weights and feature layers, thereby improving the model's expressiveness and generalization capabilities.

[0121] The preprocessing method of the data acquisition and processing unit for pathological images includes:

[0122] The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image;

[0123] The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images;

[0124] The resolution of each pathological image was set to 1.0 pixel / μm, and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

[0125] The model building unit constructs a method for comparing language and image models including:

[0126] Obtain anatomical location information for each full-slice image , anatomical information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the specimen;

[0127] Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , anatomical information The text embedding formula is:

[0128] ;

[0129] Where, is the fixed-dimensional text feature representation of T, where T is the anatomical part information, For text encoder;

[0130] Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , text feature representation The optimization formula is:

[0131] ;

[0132] Where, is the optimized text feature representation, MLP is a multi-layer perceptron;

[0133] Through the image encoder From the whole slide image I Extract the image features of the whole slice image , image features The extraction formula is:

[0134] ;

[0135] Where, is the image feature of the whole slice image, I is the whole slice image, is an image encoder;

[0136] The processed text features and image features Splicing and fusion to obtain comprehensive feature representation ;

[0137] Use the pooling strategy based on the attention mechanism to aggregate the contextual information of each pathological image and obtain the weighted feature representation of the whole slice image , weighted feature representation of full-slice images The weighted aggregation formula is:

[0138] ;

[0139] in, ;

[0140] Where, is the weighted feature representation of the full slice image, is the jth pathological image feature obtained by cutting the full slice image, To pass and comprehensive feature representation The corresponding correlation measurement between The attention score, is the correlation measure operator.

[0141] The model fine-tuning unit adds fully connected layers to specific tasks in the following ways:

[0142] The weighted feature representation of the full slice image Input to the task-specific fully connected layer, and output the prediction result corresponding to the specific task. The fully connected layer calculation formula is:

[0143] ;

[0144] Where, is the prediction result of a specific task, is a fully connected layer.

[0145] The present invention constructs a comparative language and image model to extract slice-level feature representations from unlabeled data through self-supervised learning, and integrates slice-level features into global pathological features of the whole slice image through weakly supervised learning and attention mechanism, thereby building a holistic understanding of pathological images and obtaining multimodal data that integrates microscopic imaging and anatomical site information. By combining image and text information, the representation ability of anatomical information is enhanced. By fusing the multimodal information of image and text, the type, stage and prognosis of the tumor can be more comprehensively evaluated, thereby fully utilizing the advantages of multi-source data. This is an effective way to solve the current bottleneck of tumor pathology analysis technology.

[0146] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A tumor pathology analysis method based on multimodal artificial intelligence, characterized in that: The following steps are involved: By acquiring multi-center and multi-type whole-slice images of tissues, the whole-slice images are cut into pathological images, and the pathological images after standardized preprocessing are divided into a training set and a test set; A comparative language and image model is constructed for capturing key features in pathological images through a two-stage pre-training process. The comparative language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding. The image encoder constituting the histopathology image branch uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning. The unlabeled data refers to whole-slice images of tissues that have not been manually annotated with pathology categories or pathology feature labels. The text encoder constituting the text branch uses a text encoder from the comparative language and image pre-training model. The text encoder is a model based on the Transformers structure and integrates slice-level features into global pathology features of the whole-slice image through weakly supervised learning and an attention mechanism. Pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, use early stopping strategy to avoid overfitting, and maximize validation set performance by adjusting hyperparameters; By using the pre-trained weights of the contrastive language and image models as the initial weights and adding task-specific fully connected layers, the model is fine-tuned to obtain a pathology analysis model for different classification and regression tasks in tumor pathology analysis. The method for constructing the comparative language and image model includes: Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the sample; Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , the anatomical part information The text embedding formula is: ； Where, for Fixed-dimensional text feature representation, For anatomical location information, For text encoder; Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate an optimized text feature table , the text feature representation The optimization formula is: ； Where, is the optimized text feature representation, MLP is a multi-layer perceptron; Through the image encoder From the whole slide image I Extract the image features of the whole slice image , the image features The extraction formula is: ； Where, is the image feature of the full slice image, I is the full slice image, is an image encoder; The processed text features and image features Splicing and fusion to obtain comprehensive feature representation ; Use the pooling strategy based on the attention mechanism to aggregate the contextual information of each pathological image and obtain the weighted feature representation of the whole slice image , the weighted feature representation of the whole slice image The weighted aggregation formula is: ; in, ; Where, is the weighted feature representation of the full slice image, is the jth pathological image feature obtained by cutting the full slice image, To pass and comprehensive feature representation The corresponding correlation measurement between The attention score, is the correlation measurement operator; The specific tasks of the fully connected layer include: tumor origin prediction, gene mutation prediction and prognosis prediction. Among them, the fully connected layer for tumor origin prediction is set as an 18-category weakly supervised classification task, the fully connected layer for gene mutation prediction is set as a binary classification task, and the fully connected layer for prognosis prediction is set as a regression model that outputs a death risk score.

2. The method for tumor pathology analysis based on multimodal artificial intelligence according to claim 1, characterized in that: The pathological image preprocessing method includes: The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image; The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images; The resolution of each pathological image was set to 1.0 pixel / μm, and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

3. The tumor pathology analysis method based on multimodal artificial intelligence according to claim 1, characterized in that: The method for adding a fully connected layer for the specific task includes: The weighted feature representation of the full slice image Input to the fully connected layer of a specific task, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is: ； Where, is the prediction result of a specific task, is a fully connected layer.

4. The tumor pathology analysis method based on multimodal artificial intelligence according to claim 1, characterized in that: The weakly supervised learning and contrastive learning pre-training methods of the contrastive language and image models include: Use a weakly supervised learning strategy, adopt the Adam optimizer, set the initial learning rate to 0.0003, and adjust the learning rate by the cosine annealing method; Apply the early stopping strategy and terminate the training when the AUROC indicator of the validation set does not improve for 10 consecutive epochs; AUROC, sensitivity, and specificity validation set performance indicators were calculated separately for each anatomical site, and hyperparameters were optimized to maximize the average AUROC indicator across anatomical sites.

5. A tumor pathology analysis system based on multimodal artificial intelligence, characterized in that: A method for tumor pathology analysis based on multimodal artificial intelligence as described in any one of claims 1 to 4, the system comprising: A data acquisition and processing unit is used to acquire multi-center and multi-type whole-slice images of tissues, cut the whole-slice images into pathological images, and divide the standardized pre-processed pathological images into a training set and a test set; A model building unit is configured to construct a comparative language and image model for capturing key features in pathological images through a two-stage pre-training process, wherein the comparative language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding. The image encoder constituting the histopathology image branch uses a self-supervised visual transformer backbone network and employs self-supervised learning to extract slice-level feature representations from unlabeled data, wherein the unlabeled data refers to whole-slice images of tissues that have not been manually labeled with pathology categories or pathology feature labels. The text encoder constituting the text branch uses a text encoder from the comparative language and image pre-training model. The text encoder is a model based on a Transformers structure and integrates slice-level features into global pathology features of the whole-slice image through weakly supervised learning and an attention mechanism. Model training and optimization unit, used to pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, using early stopping strategy to avoid overfitting, and maximizing validation set performance by adjusting hyperparameters; The model fine-tuning unit is used to fine-tune the model by using the pre-trained weights of the comparative language and image models as the initial weights and adding a fully connected layer for a specific task to obtain an analysis model for implementing different classification and regression tasks in tumor pathology analysis.

6. The tumor pathology analysis system based on multimodal artificial intelligence according to claim 5, characterized in that: The data acquisition and processing unit preprocesses the pathological image including: The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image; The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images; The resolution of each pathological image was set to 1.0 pixel / μm, and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

7. The tumor pathology analysis system based on multimodal artificial intelligence according to claim 5, characterized in that: The method for constructing the contrast language and image model by the model construction unit includes: Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the sample; Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , the anatomical part information The text embedding formula is: ； Where, for Fixed-dimensional text feature representation, For anatomical location information, For text encoder; Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , the text feature representation The optimization formula is: ； Where, is the optimized text feature representation, MLP is a multi-layer perceptron; Through the image encoder From the whole slide image I Extract the image features of the whole slice image , the image features The extraction formula is: ； Where, is the image feature of the full slice image, I is the full slice image, is an image encoder; The processed text features and image features Splicing and fusion to obtain comprehensive feature representation ; Use the pooling strategy based on the attention mechanism to aggregate the contextual information of each pathological image and obtain the weighted feature representation of the full slice image , the weighted feature representation of the whole slice image The weighted aggregation formula is: ； in, ; Where, is the weighted feature representation of the full slice image, is the jth pathological image feature obtained by cutting the full slice image, To pass and comprehensive feature representation The corresponding correlation measurement between The attention score, is the correlation measure operator.

8. The multimodal artificial intelligence-based tumor pathology analysis system according to claim 7, characterized in that: The method of adding a fully connected layer to a specific task by the model fine-tuning unit includes: The weighted feature representation of the full slice image Input to the fully connected layer of a specific task, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is: ； Where, is the prediction result of a specific task, is a fully connected layer.

Images