Tumor pathology analysis method and system based on multi-modal artificial intelligence

Abstract

The invention relates to the technical field of pathology analysis, in particular to a tumor pathology analysis method based on multi-modal artificial intelligence, which comprises the following steps: obtaining a multi-center and multi-type tissue full-slice image, cutting the full-slice image into a pathology image, and dividing the pathology image after standardization preprocessing into a training set and a test set; constructing a comparison language and an image model for capturing key features in the pathological image through a two-stage pre-training process; a comparison language and an image model are used for pre-training a weight as an initial weight, a task-specific full-connection layer is added, and the model is finely adjusted to obtain a pathological analysis model for realizing different classification regression tasks in tumor pathological analysis. According to the method, the type, staging and prognosis of the tumor can be evaluated more comprehensively by fusing the multi-modal information of the image text, so that the advantages of multi-source data can be fully utilized, and the method is an effective way for solving the bottleneck of the current tumor pathology analysis technology.

Description

Technical Field

[0001] The present invention relates to the technical field of pathological diagnosis, and in particular to a tumor pathology analysis method and system based on multimodal artificial intelligence. Background Art

[0002] Tumor pathology diagnosis is a crucial step in the cancer treatment process and directly affects the formulation of subsequent treatment plans. At present, traditional pathology diagnosis methods rely on pathology experts to analyze tumor tissue slice images through a microscope and make judgments based on the patient's clinical data. This method is not only time-consuming and labor-intensive, but also highly subjective and easily affected by factors such as expert experience and fatigue, resulting in uncertainty in diagnostic results. Therefore, improving the automation, accuracy and efficiency of tumor pathology diagnosis has always been an important issue in medical research and clinical applications.

[0003] In recent years, deep learning technology has made significant progress in the field of medical image analysis, but existing pathological diagnosis methods mostly focus on the analysis of image data. However, tumor assessment usually requires the combination of information from multiple modalities, such as genomic data, molecular markers, and clinical data. Traditional methods fail to effectively integrate information from these different sources, making tumor assessment lack of comprehensiveness and accuracy. In particular, a single image analysis method often cannot provide sufficient support in terms of tumor type, staging, and prognosis assessment.

[0004] Therefore, the existing tumor pathology diagnosis technology is difficult to integrate various information data, and thus cannot obtain accurate and comprehensive tumor pathology evaluation results. Summary of the invention

[0005] The purpose of the present invention is to provide a tumor pathology analysis method and system based on multimodal artificial intelligence to solve the technical problem that tumor pathology diagnosis in the prior art is difficult to integrate information data from all aspects, thereby failing to obtain accurate and comprehensive tumor pathology evaluation results.

[0006] In order to solve the above technical problems, the present invention specifically provides the following technical solutions: A tumor pathology analysis method based on multimodal artificial intelligence comprises the following steps: By acquiring multi-center and multi-type whole-slice images of tissues, the whole-slice images are cut into pathological images, and the pathological images after standardized preprocessing are divided into a training set and a test set; A contrastive language and image model is constructed to capture key features in pathological images through a two-stage pre-training process, wherein the contrastive language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding. The image encoder uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning, wherein unlabeled data refers to whole-slice images of tissues that have not been manually labeled with pathological categories or pathological feature labels; while the text encoder uses a text encoder from the contrastive language and image pre-training model, which is a model based on the Transformers structure. Through weakly supervised learning and attention mechanisms, slice-level features are integrated into global pathological features of whole-slice images to construct a holistic understanding of pathological images; finally, multimodal data of microscopic imaging and anatomical site information are integrated to enhance the representation capability of anatomical information through the combination of image and text information; Pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, use early stopping strategy to avoid overfitting, and maximize validation set performance by adjusting hyperparameters; By using the pre-trained weights of the contrastive language and image models as the initial weights and adding task-specific fully connected layers, the model is fine-tuned to obtain a pathology analysis model for implementing different classification and regression tasks in tumor pathology analysis.

[0007] As a preferred solution of the present invention, the pathological image preprocessing method includes: The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image; The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images; The resolution of each pathological image was set to 1.0 μm / pixel and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

[0008] As a preferred embodiment of the present invention, the method for constructing the comparative language and image model comprises: Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the specimen; Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , the anatomical part information The text embedding formula is: ; In the formula, is the fixed-dimensional text feature representation of T, T is the anatomical part information, is a text encoder; Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , the text feature representation The optimization formula is: ; In the formula, Optimized text feature representation, MLP is a multi-layer perceptron; Through the image encoder From the whole slice image Extract the image features of the whole slice image , the image features The extraction formula is: ; In the formula, is the image feature of the full slice image, is a full-slice image, is an image encoder; The processed text features With image features Splicing and fusion to obtain comprehensive feature representation ; A pooling strategy based on the attention mechanism is used to aggregate the contextual information of each pathological image to obtain a weighted feature representation of the full slice image. , the weighted feature representation of the whole slice image The weighted aggregation formula is: ; in, ; In the formula, is the weighted feature representation of the full slice image, is the jth pathological image obtained by cutting the full slice image, for The attention score, yes With comprehensive feature representation The correlation measure between .

[0009] As a preferred solution of the present invention, the method for adding a fully connected layer for a specific task includes: The weighted feature representation of the whole slice image Input to the fully connected layer of a specific task, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is: ; In the formula, is the prediction result of a specific task, is a fully connected layer.

[0010] As a preferred solution of the present invention, the specific tasks of the fully connected layer include: tumor origin prediction, gene mutation prediction and prognosis prediction, wherein the fully connected layer for tumor origin prediction is set as an 18-category weakly supervised classification task, the fully connected layer for gene mutation prediction is set as a binary classification task, and the fully connected layer for prognosis prediction is set as a regression model that outputs a death risk score.

[0011] As a preferred embodiment of the present invention, the weakly supervised learning and contrastive learning pre-training method of contrastive language and image models includes: Weakly supervised learning strategy was used, Adam optimizer was used, the initial learning rate was 0.0003, and the learning rate was adjusted by cosine annealing method; Apply the early stopping strategy and terminate the training when the AUROC indicator of the validation set does not improve for 10 consecutive epochs; The AUROC index, sensitivity, and specificity validation set performance indicators were calculated separately for each anatomical site, and the hyperparameters were optimized to maximize the average AUROC index across anatomical sites.

[0012] As a preferred embodiment of the present invention, the present invention provides a tumor pathology analysis system based on multimodal artificial intelligence, which is applied to a tumor pathology analysis method based on multimodal artificial intelligence. The system includes: A data acquisition and processing unit, used for acquiring multi-center and multi-type tissue whole-slice images, cutting the whole-slice images into pathological images, and dividing the pathological images after standardized preprocessing into a training set and a test set; A model building unit is used to build a contrastive language and image model for capturing key features in pathological images through a two-stage pre-training process, wherein the contrastive language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding, and the image encoder uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning, wherein unlabeled data refers to whole-slice images of tissues that have not been manually labeled with pathological categories or pathological feature labels; and the text encoder uses a text encoder from the contrastive language and image pre-training model, which is a model based on the Transformers structure, and integrates slice-level features into global pathological features of whole-slice images through weakly supervised learning and attention mechanisms, thereby building a holistic understanding of pathological images; finally, multimodal data of microscopic imaging and anatomical site information are integrated, and the representation capability of anatomical information is enhanced through the combination of image and text information; Model training and optimization unit, used to pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, use early stopping strategy to avoid overfitting, and maximize validation set performance by adjusting hyperparameters; A model fine-tuning unit is used to fine-tune the model by using the pre-trained weights of the contrast language and image models as the initial weights and adding a fully connected layer for a specific task to obtain an analysis model for implementing different classification and regression tasks in tumor pathology analysis; Among them, the pre-trained weights of the contrast language and image models refer to the weights obtained through a one-stage self-supervised learning on large-scale unlabeled data. These weights are used as initial parameters to help the model quickly adapt to specific tasks, reduce dependence on labeled data, and improve training efficiency; the fine-tuning process includes adjustments to the model weights and structure, such as adding 18 categories of weakly supervised classifiers when predicting tumor origin, or replacing it with a regression model when predicting tumor prognosis and adjusting the model weight value through automated training; the model shares knowledge between different tasks by sharing the first-stage pre-trained weights and feature layers, thereby improving the model's expressiveness and generalization capabilities.

[0013] As a preferred solution of the present invention, the preprocessing method of the pathological image by the data acquisition and processing unit includes: The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image; The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images; The resolution of each pathological image was set to 1.0 μm / pixel, and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

[0014] As a preferred solution of the present invention, the method for constructing the model building unit for comparing language and image models includes: Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the specimen; Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , the text embedding formula of the anatomical part information is: ; In the formula, is the fixed-dimensional text feature representation of T, T is the anatomical part information, is a text encoder; Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , the text feature representation The optimization formula is: ; In the formula, is the optimized text feature representation, MLP is a multi-layer perceptron; Through the image encoder From the whole slice image Extract the image features of the whole slice image , the image features The extraction formula is: ; In the formula, is the image feature of the full slice image, is a full-slice image, is an image encoder; The processed text features With image features Splicing and fusion to obtain comprehensive feature representation ; A pooling strategy based on the attention mechanism is used to aggregate the contextual information of each pathological image to obtain a weighted feature representation of the full slice image. , the weighted feature representation of the whole slice image The weighted aggregation formula is: ; in, ; In the formula, is the weighted feature representation of the full slice image, is the jth pathological image obtained by cutting the full slice image, for The attention score, yes With comprehensive feature representation The correlation measure between .

[0015] As a preferred solution of the present invention, the method for adding a fully connected layer to a specific task by the model fine-tuning unit includes: The weighted feature representation of the whole slice image Input to the fully connected layer of a specific task, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is: ; In the formula, is the prediction result of a specific task, is a fully connected layer.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention constructs a comparative language and image model to extract slice-level feature representation from unlabeled data through self-supervised learning, and integrates slice-level features into global pathological features of the whole slice image through weakly supervised learning and attention mechanism, constructs a holistic understanding of the pathological image, and obtains multimodal data integrating microscopic imaging and anatomical site information. By combining image and text information, the representation ability of anatomical information is enhanced. By fusing the multimodal information of image and text, the type, stage and prognosis of the tumor can be more comprehensively evaluated, thereby making full use of the advantages of multi-source data. It is an effective way to solve the current bottleneck of tumor pathology analysis technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0018] Figure 1 is a flow chart of a tumor pathology analysis method based on multimodal artificial intelligence provided by an embodiment of the present invention; Figure 2 is a flow chart of a pathological image preprocessing method provided by an embodiment of the present invention; Figure 3 is a schematic diagram of a model structure of a tumor pathology analysis method based on multimodal artificial intelligence provided by an embodiment of the present invention; Figure 4 is a model training flow chart of a tumor pathology analysis method based on multimodal artificial intelligence provided by an embodiment of the present invention; Figure 5 It is a block diagram of a tumor pathology analysis system based on multimodal artificial intelligence provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, the present invention provides a tumor pathology analysis method based on multimodal artificial intelligence, comprising the following steps: By acquiring multi-center and multi-type whole-slice images of tissues, the whole-slice images are cut into pathological images, and the pathological images after standardized preprocessing are divided into a training set and a test set; A contrastive language and image model is constructed to capture key features in pathological images through a two-stage pre-training process, wherein the contrastive language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding. The image encoder uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning, wherein unlabeled data refers to whole-slice images of tissues that have not been manually labeled with pathological categories or pathological feature labels; while the text encoder uses a text encoder from the contrastive language and image pre-training model, which is a model based on the Transformers structure. Through weakly supervised learning and attention mechanisms, slice-level features are integrated into global pathological features of whole-slice images to construct a holistic understanding of pathological images; finally, multimodal data of microscopic imaging and anatomical site information are integrated to enhance the representation capability of anatomical information through the combination of image and text information; Pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, use early stopping strategy to avoid overfitting, and maximize validation set performance by adjusting hyperparameters; By using the pre-trained weights of the contrastive language and image models as the initial weights and adding task-specific fully connected layers, the model is fine-tuned to obtain an analysis model for implementing different classification and regression tasks in tumor pathology analysis.

[0021] like Figure 2 As shown, the pathological image preprocessing method includes: The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image; The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images; The resolution of each pathological image was set to 1.0 μm / pixel and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

[0022] The method for constructing the contrastive language and image model includes: Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the specimen; Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , the anatomical part information The text embedding formula is: ; In the formula, is the fixed-dimensional text feature representation of T, T is the anatomical part information, is a text encoder; Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , the text feature representation The optimization formula is: ; In the formula, is the optimized text feature representation, MLP is a multi-layer perceptron; Through the image encoder From the whole slice image Extract the image features of the whole slice image , the image features The extraction formula is: ; In the formula, is the image feature of the full slice image, is a full-slice image, is an image encoder; The processed text features With image features Splicing and fusion to obtain comprehensive feature representation ; A pooling strategy based on the attention mechanism is used to aggregate the contextual information of each pathological image to obtain a weighted feature representation of the full slice image. , the weighted feature representation of the whole slice image The weighted aggregation formula is: ; in, ; In the formula, is the weighted feature representation of the full slice image, is the jth pathological image obtained by cutting the full slice image, for The attention score, yes With comprehensive feature representation The correlation measure between .

[0023] The method for adding a fully connected layer for the specific task includes: The weighted feature representation of the whole slice image Input to the fully connected layer of a specific task, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is: ; In the formula, is the prediction result of a specific task, is a fully connected layer.

[0024] The specific tasks of the fully connected layer include: tumor origin prediction, gene mutation prediction and prognosis prediction, wherein the fully connected layer for tumor origin prediction is set as an 18-category weakly supervised classification task, the fully connected layer for gene mutation prediction is set as a binary classification task, and the fully connected layer for prognosis prediction is set as a regression model that outputs a death risk score.

[0025] like Figure 4 As shown, the weakly supervised learning and contrastive learning pre-training method of contrastive language and image models includes: Weakly supervised learning strategy was used, Adam optimizer was used, the initial learning rate was 0.0003, and the learning rate was adjusted by cosine annealing method; Apply the early stopping strategy and terminate the training when the AUROC indicator of the validation set does not improve for 10 consecutive epochs; The AUROC index, sensitivity, and specificity validation set performance indicators were calculated separately for each anatomical site, and the hyperparameters were optimized to maximize the average AUROC index across anatomical sites.

[0026] like Figure 5 As shown, the present invention provides a tumor pathology analysis system based on multimodal artificial intelligence, which is applied to a tumor pathology analysis method based on multimodal artificial intelligence. The system includes: A data acquisition and processing unit, used for acquiring multi-center and multi-type tissue whole-slice images, cutting the whole-slice images into pathological images, and dividing the pathological images after standardized preprocessing into a training set and a test set; A model building unit is used to build a contrastive language and image model for capturing key features in pathological images through a two-stage pre-training process, wherein the contrastive language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding, and the image encoder uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning, wherein unlabeled data refers to whole-slice images of tissues that have not been manually labeled with pathological categories or pathological feature labels; and the text encoder uses a text encoder from the contrastive language and image pre-training model, which is a model based on the Transformers structure, and integrates slice-level features into global pathological features of whole-slice images through weakly supervised learning and attention mechanisms, thereby building a holistic understanding of pathological images; finally, multimodal data of microscopic imaging and anatomical site information are integrated, and the representation capability of anatomical information is enhanced through the combination of image and text information; Model training and optimization unit, used to pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, use early stopping strategy to avoid overfitting, and maximize validation set performance by adjusting hyperparameters; A model fine-tuning unit, which is used to fine-tune the model by using the pre-trained weights of the contrast language and image models as the initial weights and adding task-specific fully connected layers to obtain an analysis model for implementing different classification and regression tasks in tumor pathology analysis; like Figure 3 As shown, adding 18 categories of weakly supervised classifiers to the contrast language and image model can realize the model structure for predicting tumor origin, adding a regression model to the contrast language and image model can realize the model structure for predicting tumor prognosis, and adding a binary classifier to the contrast language and image model can realize the model structure for predicting gene mutation.

[0027] The pre-trained weights of contrast language and image models refer to the weights obtained through a one-stage self-supervised learning on large-scale unlabeled data. These weights serve as initial parameters to help the model quickly adapt to specific tasks, reduce dependence on labeled data, and improve training efficiency.

[0028] The fine-tuning process includes adjusting the model weights and structure, such as adding 18 categories of weakly supervised classifiers when predicting tumor origin, or replacing it with a regression model when predicting tumor prognosis and adjusting the model weight values ​​through automated training; the model shares knowledge between different tasks by sharing the first-stage pre-trained weights and feature layers, thereby improving the model's expressiveness and generalization capabilities.

[0029] The data acquisition and processing unit preprocesses the pathological image including: The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image; The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images; The resolution of each pathological image was set to 1.0 μm / pixel, and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

[0030] The method for constructing the model building unit for comparing language and image models includes: Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the specimen; Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation , the text embedding formula of the anatomical part information is: ; In the formula, is the fixed-dimensional text feature representation of T, T is the anatomical part information, is a text encoder; Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , the text feature representation The optimization formula is: ; In the formula, is the optimized text feature representation, MLP is a multi-layer perceptron; Through the image encoder From the whole slice image Extract the image features of the whole slice image , the image feature extraction formula is: ; In the formula, is the image feature of the full slice image, is a full-slice image, is an image encoder; The processed text features With image features Splicing and fusion to obtain comprehensive feature representation ; A pooling strategy based on the attention mechanism is used to aggregate the contextual information of each pathological image to obtain a weighted feature representation of the full slice image. , the weighted feature representation of the whole slice image The weighted aggregation formula is: ; in, ; In the formula, is the weighted feature representation of the full slice image, is the jth pathological image obtained by cutting the full slice image, for The attention score, yes With comprehensive feature representation The correlation measure between .

[0031] The method of adding a fully connected layer to a specific task by the model fine-tuning unit includes: The weighted feature representation of the whole slice image Input to the task-specific fully connected layer, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is: ; In the formula, is the prediction result of a specific task, is a fully connected layer.

[0032] The present invention constructs a comparative language and image model to extract slice-level feature representation from unlabeled data through self-supervised learning, and integrates slice-level features into global pathological features of the whole slice image through weakly supervised learning and attention mechanism, constructs a holistic understanding of the pathological image, and obtains multimodal data integrating microscopic imaging and anatomical site information. By combining image and text information, the representation ability of anatomical information is enhanced. By fusing the multimodal information of image and text, the type, stage and prognosis of the tumor can be more comprehensively evaluated, thereby making full use of the advantages of multi-source data. It is an effective way to solve the current bottleneck of tumor pathology analysis technology.

[0033] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A tumor pathology analysis method based on multimodal artificial intelligence, characterized in that: The following steps are involved: By acquiring multi-center and multi-type whole-slice images of tissues, the whole-slice images are cut into pathological images, and the pathological images after standardized preprocessing are divided into a training set and a test set; A contrastive language and image model is constructed for capturing key features in pathological images through a two-stage pre-training process, wherein the contrastive language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding, the image encoder constituting the histopathology image branch uses a self-supervised visual transformer backbone network to extract slice-level feature representations from unlabeled data using self-supervised learning, wherein the unlabeled data refers to whole-slice images of tissues that have not been manually labeled with pathological categories or pathological feature labels; the text encoder constituting the text branch uses a text encoder from the contrastive language and image pre-training model, wherein the text encoder is a model based on the Transformers structure, and integrates slice-level features into global pathological features of the whole-slice image through weakly supervised learning and an attention mechanism; Pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, use early stopping strategy to avoid overfitting, and maximize validation set performance by adjusting hyperparameters; By using the pre-trained weights of the contrastive language and image models as the initial weights and adding task-specific fully connected layers, the model is fine-tuned to obtain a pathology analysis model for implementing different classification and regression tasks in tumor pathology analysis.

2. The tumor pathology analysis method based on multimodal artificial intelligence according to claim 1, characterized in that: The pathological image preprocessing method comprises: The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image; The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images; The resolution of each pathological image was set to 1.0 μm / pixel and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

3. The tumor pathology analysis method based on multimodal artificial intelligence according to claim 1, characterized in that: The method for constructing the contrastive language and image model includes: Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the sample; Pre-trained text encoder using contrastive language and image models The anatomical part information is embedded to Converted into a fixed-dimensional text feature representation , The anatomical site information The text embedding formula is: ； In the formula, is the fixed-dimensional text feature representation of T, T is the anatomical part information, is a text encoder; Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , the text feature representation The optimization formula is: ； In the formula, is the optimized text feature representation, MLP is a multi-layer perceptron; Through the image encoder From the whole slice image Extract the image features of the whole slice image , the image features The extraction formula is: ； In the formula, is the image feature of the full slice image, is a full-slice image, is an image encoder; The processed text features With image features Splicing and fusion to obtain comprehensive feature representation ; A pooling strategy based on the attention mechanism is used to aggregate the contextual information of each pathological image to obtain a weighted feature representation of the full slice image. , the weighted feature representation of the whole slice image The weighted aggregation formula is: ； in, ; In the formula, is the weighted feature representation of the full slice image, is the jth pathological image obtained by cutting the full slice image, for The attention score, yes With comprehensive feature representation The correlation measure between .

4. The tumor pathology analysis method based on multimodal artificial intelligence according to claim 1, characterized in that: The method for adding a fully connected layer for the specific task includes: The weighted feature representation of the whole slice image Input to the fully connected layer of a specific task, and output the prediction result corresponding to the specific task. The calculation formula of the fully connected layer is: ； In the formula, is the prediction result of a specific task, is a fully connected layer.

5. The tumor pathology analysis method based on multimodal artificial intelligence according to claim 1, characterized in that: The specific tasks of the fully connected layer include: tumor origin prediction, gene mutation prediction and prognosis prediction, wherein the fully connected layer for tumor origin prediction is set as an 18-category weakly supervised classification task, the fully connected layer for gene mutation prediction is set as a binary classification task, and the fully connected layer for prognosis prediction is set as a regression model that outputs a death risk score.

6. The tumor pathology analysis method based on multimodal artificial intelligence according to claim 1, characterized in that: The weakly supervised learning and contrastive learning pre-training method of contrastive language and image models includes: Weakly supervised learning strategy was used, Adam optimizer was used, the initial learning rate was 0.0003, and the learning rate was adjusted by cosine annealing method; Apply the early stopping strategy and terminate the training when the AUROC indicator of the validation set does not improve for 10 consecutive epochs; The AUROC index, sensitivity, and specificity validation set performance indicators were calculated separately for each anatomical site, and the hyperparameters were optimized to maximize the average AUROC index across anatomical sites.

7. A tumor pathology analysis system based on multimodal artificial intelligence, characterized in that: A tumor pathology analysis method based on multimodal artificial intelligence as described in any one of claims 1 to 6, the system comprising: A data acquisition and processing unit, used for acquiring multi-center and multi-type tissue whole-slice images, cutting the whole-slice images into pathological images, and dividing the pathological images after standardized preprocessing into a training set and a test set; A model building unit, used to build a contrastive language and image model for capturing key features in pathological images through a two-stage pre-training process, wherein the contrastive language and image model includes a histopathology image branch for image encoding and a text branch for anatomical site encoding, the image encoder uses a self-supervised visual transformer backbone network, and uses self-supervised learning to extract slice-level feature representations from unlabeled data, wherein unlabeled data refers to tissue whole-slice images that have not been manually labeled with pathological categories or pathological feature labels; and the text encoder uses a text encoder from the contrastive language and image pre-training model, which is a model based on the Transformers structure, and integrates slice-level features into global pathological features of the whole-slice image through weakly supervised learning and attention mechanisms; Model training and optimization unit, used to pre-train and optimize contrastive language and image models through weakly supervised learning and contrastive learning, use early stopping strategy to avoid overfitting, and maximize validation set performance by adjusting hyperparameters; The model fine-tuning unit is used to fine-tune the model by using the pre-trained weights of the contrast language and image models as the initial weights and adding a fully connected layer for a specific task to obtain an analysis model for implementing different classification and regression tasks in tumor pathology analysis.

8. The tumor pathology analysis system based on multimodal artificial intelligence according to claim 7, characterized in that: The data acquisition and processing unit preprocesses the pathological image including: The Otsu threshold method was used to remove the background in the whole-slice image and retain the tissue area in the whole-slice image; The whole-slice images with background removed were cropped into non-overlapping patches of 256 × 256 pixels as pathological images; The resolution of each pathological image was set to 1.0 μm / pixel, and the magnification was set to ×10 to unify the brightness and contrast of the pathological images.

9. The tumor pathology analysis system based on multimodal artificial intelligence according to claim 7, characterized in that: The method for constructing the model building unit for comparing language and image models includes: Obtain anatomical location information for each full-slice image , the anatomical part information The format is "This is a pathology image from [anatomical site]", where [anatomical site] is the anatomical location of the sample; Pre-trained text encoder using contrastive language and image models Anatomical information Embed to embed anatomical information Converted into a fixed-dimensional text feature representation, the text embedding formula of the anatomical part information is: ； In the formula, is the fixed-dimensional text feature representation of T, T is the anatomical part information, is a text encoder; Text feature representation obtained by text embedding through multi-layer perceptron MLP Perform optimization processing to generate optimized text feature representation , the text feature representation The optimization formula is: ； In the formula, is the optimized text feature representation, MLP is a multi-layer perceptron; Through the image encoder From the whole slice image Extract the image features of the whole slice image , the image features The extraction formula is: ； In the formula, is the image feature of the full slice image, is a full-slice image, is an image encoder; The processed text features With image features Splicing and fusion to obtain comprehensive feature representation ; A pooling strategy based on the attention mechanism is used to aggregate the contextual information of each pathological image to obtain a weighted feature representation of the full slice image. , the weighted feature representation of the whole slice image The weighted aggregation formula is: ； in, ; In the formula, Weighted feature representation of full-slice images, is the jth pathological image obtained by cutting the full slice image, for The attention score, yes With comprehensive feature representation The correlation measure between .

10. The tumor pathology analysis system based on multimodal artificial intelligence according to claim 9, characterized in that: The method of adding a fully connected layer to a specific task by the model fine-tuning unit includes: The weighted feature representation of the full slice image is input into the fully connected layer of the specific task, and the prediction result corresponding to the specific task is output. The calculation formula of the fully connected layer is: ； In the formula, is the prediction result of a specific task, is a fully connected layer.

Images