Weakly supervised gastric cancer tissue pathology image classification method and system based on contrastive learning multimodality

Abstract

The present invention discloses a weakly supervised gastric cancer tissue pathology image classification method and system based on contrastive learning multimodality. The present invention solves at least two difficult problems in traditional tissue pathology image classification methods: lack of fine-grained labels and insufficient use of clinical information. In response to these problems, the present invention first uses the self-supervised network SimCLR model to extract features from tissue pathology images. This method can learn meaningful features from unlabeled data by itself, thereby effectively extracting feature representations of tissue pathology images, reducing the need for fine-grained labels and alleviating the burden of data annotation; secondly, by using a multimodal fusion method, the features of tissue pathology images of different magnifications are fused with clinical structured information to provide the classification model with richer features for learning, thereby achieving more accurate survival predictions to meet the needs of clinical auxiliary diagnosis.

Description

Technical Field

[0001] The present invention relates to a weakly supervised gastric cancer tissue pathology image classification method based on contrastive learning multimodality, and belongs to the fields of computer vision and medical data analysis and mining. Background Art

[0002] Gastric cancer is one of the major cancers threatening people's public health. In 2020, the global incidence of gastric cancer ranked fifth and the mortality rate ranked fourth. In clinical settings, pathologists usually make diagnoses based on the morphological features of tissue pathology images observed at different scales and referring to medical reports. However, the manual diagnosis process is time-consuming, subjective, and non-repeatable. With the rapid development of digital pathology and deep learning technology, many deep learning-based cancer prognosis prediction models have been proposed to assist doctors in clinical diagnosis. At present, there are many studies on cancer prognosis modeling through medical images such as CT and MRI, but for the field of cancer diagnosis, tissue pathology images with high resolution and multi-level magnification are one of the gold standards for cancer diagnosis and prognosis.

[0003] The multi-resolution and large amount of information characteristics of tissue pathology images lead to information redundancy in deep learning networks, which cannot effectively extract key features of images. In addition, only using a single image feature ignores data information in clinical scenarios, which is not suitable for actual application scenarios and has low accuracy. Summary of the invention

[0004] The present invention provides a weakly supervised gastric cancer tissue pathology image classification method based on contrastive learning multimodality, which only requires coarse-grained annotation information at the image level. At the same time, it can be used for gastric cancer tissue pathology image classification by combining clinical information with a multimodal method for fusion, thereby improving the generalization ability and accuracy of the model; based on the classification results, survival status prediction is achieved to meet the needs of clinical auxiliary diagnosis.

[0005] The technical solution of the present invention is:

[0006] According to one aspect of the present invention, a weakly supervised gastric cancer tissue pathology image classification method based on contrastive learning multimodality is provided, comprising: S1, collecting gastric cancer patient tissue pathology images and corresponding clinical structured data; S2, preprocessing the tissue pathology images and clinical structured data to obtain tissue pathology image sub-blocks at different magnifications and corresponding preprocessed clinical structured data; S3, inputting the tissue pathology image sub-blocks at different magnifications into a feature extraction model for contrastive learning training, and obtaining feature extraction model weights of tissue pathology images at different preset magnifications; S4, importing the weights obtained in S3 into the feature extraction model to obtain a tissue pathology image feature extractor at different preset magnifications; Input into the feature extractor, and output the corresponding tissue pathology image sub-block feature representation; at the same time, input the preprocessed clinical structured data into the customized multi-layer perceptron to obtain the clinical structured data feature representation of the same dimension as the tissue pathology image sub-block feature representation; S5, divide the tissue pathology image sub-block feature representation output by S4 and the one-to-one corresponding clinical structured data feature representation into training set, validation set and test set; S6, input the training set data of S5 into the multimodal fusion model for training, and then use the validation set of S5 to verify the model performance obtained from each batch of training, and save the parameters of the multimodal fusion model that best obtains the best indicators on the validation set as the weight of the final classification model; finally, verify the saved model performance on the test set of S5.

[0007] Preprocessing of each tissue pathology section includes: removing unmatched data, dividing it into sub-blocks of preset size using a sliding window, discarding blank sub-blocks, and performing staining standardization.

[0008] The preprocessing of each tissue pathology image is specifically as follows: based on the survival label information of the clinical structured data, the tissue pathology images with no survival status information, no total survival information, survival status of 0 and total survival less than y years, and survival status of 1 and total survival greater than y years are eliminated; the images of the tissue pathology images at two different preset magnifications are cropped into sub-blocks of a preset size using a sliding window, and the sub-blocks with a blank background exceeding a first preset threshold are filtered out; all the sub-blocks are color standardized. After the above operations are completed, the cropped sub-blocks are saved separately according to the corresponding tissue pathology images.

[0009] Preprocessing of clinical structured data includes: selecting attributes related to patient survival, removing unmatched data, discretizing continuous data in clinical structured data and performing one-hot encoding; among them, clinical structured data includes attribute features and survival labels.

[0010] The multilayer perceptron customization is specifically as follows: the multilayer perceptron consists of three fully connected layers, respectively having 128, 256, and 512 nodes, and uses a ReLU activation function.

[0011] The multimodal fusion model consists of three modules: a multi-instance aggregation module, a tensor fusion module, and a classifier; the feature representations of the sub-blocks of the tissue pathology image at different preset magnifications are spliced ​​to generate instance-level embeddings; and the package-level feature embeddings are generated from the instance-level features through the multi-instance aggregation module; the instance-level embedding, the package-level feature embedding and the clinical structured data feature representation are multimodally fused through the tensor fusion module, and a multimodal fusion vector is output; finally, the survival status prediction result is output through a classifier composed of a fully connected layer.

[0012] The multi-instance aggregation module is used to obtain the key instance feature p with the highest score by performing maximum pooling on all instance-level features in the sub-tile instance-level embedding. key , each instance feature p f Decomposed into a query vector q f Sum value vector v f ; respectively expressed as: q f =W a *p f , v f =W v *p f , f=1,…,n, where W a and W v is a weight matrix; then a method similar to the self-attention mechanism is used to calculate the attention score between the key instance feature and each instance feature; the obtained attention score Score is used as the weight for each value vector v f The weighted information vectors of all instance features are weighted and added together to form a package-level feature embedding.

[0013] The method expression similar to the self-attention mechanism is:

[0014]

[0015] Where <*,*> represents the inner product of two eigenvectors.

[0016] The tensor fusion module is used to expand the single-modal feature dimension to the same dimension using 1 for clinical structured data feature representation, instance-level embedding and package-level feature embedding, and then calculate the Kronecker product of all modal features to obtain a multimodal fusion vector.

[0017] According to another aspect of the present invention, a weakly supervised gastric cancer tissue pathology image classification system based on contrastive learning multimodality is provided, comprising a module of any one of the methods described above.

[0018] The beneficial effect of the present invention is that the present invention solves at least two difficult problems in the traditional histopathology image classification method: lack of fine-grained labels and insufficient use of clinical information. In response to these problems, the present invention first uses the self-supervised network SimCLR model to extract features from histopathology images. This method can learn meaningful features from unlabeled data by itself, thereby effectively extracting feature representations of histopathology images, reducing the need for fine-grained labels and alleviating the burden of data annotation; secondly, by using a multimodal fusion method, the features of histopathology images at different magnifications are fused with clinical structured information, providing the classification model with richer features for learning, achieving more accurate survival predictions, and meeting the needs of clinical auxiliary diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the process of the present invention;

[0020] Figure 2 This is a survival prediction result diagram using 5-fold tissue pathology images provided according to an embodiment of the present invention;

[0021] Figure 3 This is a survival prediction result diagram using 20x histopathological images provided according to an embodiment of the present invention;

[0022] Figure 4 is a result diagram of survival prediction using clinical information provided by an embodiment of the present invention;

[0023] Figure 5 This is a survival prediction result diagram using 5x and 20x histopathological image fusion according to an embodiment of the present invention;

[0024] Figure 6 This is a survival prediction result diagram using a multimodal method provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The invention will be further described below in conjunction with the accompanying drawings and embodiments, but the content of the invention is not limited to the scope of the embodiments.

[0026] Example 1: Figure 1-6As shown in the figure, a weakly supervised gastric cancer tissue pathology image classification method based on contrastive learning multimodality includes: S1, collecting HE-stained tissue pathology images and corresponding clinical structured data of gastric cancer patients diagnosed by clinical doctors in a tumor hospital; S2, preprocessing the tissue pathology images and clinical structured data to obtain tissue pathology image sub-blocks at different magnifications and corresponding preprocessed clinical structured data; S3, inputting the tissue pathology image sub-blocks at different magnifications into a feature extraction model for contrastive learning training, and obtaining feature extraction model weights of tissue pathology images at different preset magnifications; wherein the feature extraction model is a SimCLR model with ResNet50 as the backbone network; S4, importing the weights obtained in S3 into the feature extraction model to obtain feature extraction weights of tissue pathology images at different preset magnifications. extractor; input the histopathological image sub-blocks at different magnifications into the feature extractor, and output the corresponding histopathological image sub-block feature representation; at the same time, input the preprocessed clinical structured data into the custom multi-layer perceptron to obtain the clinical structured data feature representation of the same dimension as the histopathological image sub-block feature representation; S5, divide the histopathological image sub-block feature representation output by S4 and the one-to-one corresponding clinical structured data feature representation into a training set, a validation set and a test set; S6, input the training set data of S5 into the multimodal fusion model for training, and then use the validation set of S5 to verify the model performance obtained from each batch of training, and save the parameters of the multimodal fusion model that best obtains the best indicators on the validation set as the weight of the final classification model; finally, verify the saved model performance on the test set of S5.

[0027] Furthermore, preprocessing each tissue pathology section includes: removing unmatched data, dividing into sub-blocks of preset sizes using a sliding window, discarding blank sub-blocks, and performing staining standardization.

[0028] Furthermore, preprocessing of each tissue pathology image includes:

[0029] Based on the survival label information of clinical structured data, histopathological images with no survival status information, no overall survival information, survival status of 0 and overall survival less than y years, and survival status of 1 and overall survival greater than y years were eliminated;

[0030] Use the openslide toolkit in the Python library to use a sliding window to crop the tissue pathology images at two different preset magnifications into sub-blocks of preset sizes, and filter out sub-blocks with blank backgrounds exceeding the first preset threshold; use the Vahadane method in the staintools toolkit to color standardize all sub-blocks. After the above operations are completed, save the cropped sub-blocks separately according to the corresponding tissue pathology images. In the above technical solution, the use of two different preset magnifications can effectively improve the prediction accuracy while avoiding the waste of computing resources caused by too many magnifications. The values ​​of the first preset magnification and the second preset magnification are in a multiple relationship;

[0031] Furthermore, preprocessing of clinical structured data includes: selecting attributes related to patient survival, removing unmatched data, discretizing continuous data in clinical structured data and performing one-hot encoding; wherein, clinical structured data includes attribute characteristics and survival labels, attribute characteristics include gender, age, TN stage, etc., and survival labels include survival status and overall survival.

[0032] Furthermore, the clinical structured data is preprocessed specifically as follows: the attribute features whose missing value ratio in the clinical structured data exceeds the second preset threshold are deleted; the remaining missing values ​​are interpolated using the mean method; the data that cannot be matched with the tissue pathology image is removed; the continuous data in the clinical structured data is discretized, and the characteristic value of each attribute is uniquely encoded to obtain the preprocessed clinical structured data. It should be noted that the data that cannot be matched with the tissue pathology image is removed specifically as follows: the clinical structured data that does not have a corresponding tissue pathology image ID is matched with the attribute ID in the clinical structured data according to the tissue pathology image ID; the continuous data in the clinical structured data is discretized, and the discretization is to divide the levels according to the intervals such as the patient's age (for example, the age attribute feature of the patient's age 0-30 is assigned 0; the age attribute feature of the patient's age 30-50 is assigned 1, etc.).

[0033] Furthermore, in S3, the feature extraction model SimCLR framework used is as follows:

[0034] SimCLR uses a random data enhancement method to generate a corresponding sample pair x for each sub-block. i 、x j , including random grayscale with a probability of 0.2, random color jitter with a probability of 0.8, random cropping, random Gaussian blur, and random horizontal flip. i represents the original sample, x j Represents the sample after data enhancement. i 、xj Input the convolutional encoder with resnet-50 as the backbone network to generate feature representation h i 、h j , and then the image is mapped to z through a nonlinear fully connected layer i 、z j , calculate the mapping representation z of two sample pairs i 、z j The cosine similarity between . Its definition is as follows:

[0035]

[0036] SimCLR uses NT-Xent loss as the contrast loss, which is defined as follows:

[0037]

[0038] Among them, τ is an adjustable temperature coefficient used to adjust the scale of cosine similarity, and k represents the number of sample pairs involved in the comparison.

[0039] Furthermore, the multi-layer perceptron is customized specifically as follows: the multi-layer perceptron is composed of three fully connected layers, respectively having 128, 256, and 512 nodes, and using a ReLU activation function. Since the pre-processed clinical structured data obtained after pre-processing the clinical structured data still only contains feature vectors of 52 dimensions, the multi-layer perceptron customized by the present invention is used to map the low-dimensional pre-processed clinical structured data features to a high-dimensional space, thereby making it easier to fuse with image features.

[0040] Furthermore, the multimodal fusion model consists of three modules: a multi-instance aggregation module, a tensor fusion module, and a classifier; the feature representations of the sub-blocks of the tissue pathology image at different preset magnifications are spliced ​​to generate instance-level embeddings; and the package-level feature embeddings are generated from the instance-level features through the multi-instance aggregation module; the instance-level embeddings, package-level feature embeddings and the clinical structured data feature representations obtained by S4 are multimodally fused through the tensor fusion module to output a multimodal fusion vector; finally, the survival status prediction result is output through a classifier composed of a fully connected layer.

[0041] Furthermore, the feature representations of the histopathological image sub-blocks at different preset magnifications are concatenated to generate the histopathological image sub-block instance-level embedding E a ={p1, p2, ..., p n}, where n represents the number of histopathological image sub-blocks corresponding to the first preset magnification;

[0042] Furthermore, the multi-instance aggregation module is used to obtain the key instance feature p with the highest score by performing maximum pooling on all instance-level features in the sub-tile instance-level embedding. key , each instance feature p f Decomposed into a query vector q f Sum value vector v f ; respectively expressed as: q f =W a *p f , V F =W v *p F , f=1,…,n, where W a and W v is a weight matrix; then a method similar to the self-attention mechanism is used to calculate the attention score between the key instance feature and each instance feature;

[0043] The obtained attention score Score is used as a weight for each value vector v f Weighted, and the weighted information vectors of all instance features are added together to form the package-level feature embedding E b , expressed as:

[0044]

[0045] Furthermore, the method expression similar to the self-attention mechanism is:

[0046]

[0047] Where <*,*> represents the inner product of two eigenvectors.

[0048] Furthermore, the tensor fusion module represents the clinical structured data feature E t , instance-level embedding E a and the bag-level feature embedding E b Use 1 to expand the single-modal feature dimension to the same dimension, and then calculate the Kronecker product of all modal features to obtain the multimodal fusion vector h fusion , the expression is as follows:

[0049]

[0050] Furthermore, the classifier is used to input the multimodal fusion vector obtained by the tensor fusion module into the classifier to obtain a binary classification result. The classifier consists of two fully connected layers and an adaptive average pooling layer, with 512, 128 and 1 nodes respectively, and the last layer outputs the patient's survival status prediction result.

[0051] The classifier is optimized using the COX risk function model. The COX risk function is as follows:

[0052]

[0053] Among them, i Represents the output of the current last layer, t i represents the survival time of the current patient, δ i is whether the i-th patient dies, (δ i =1, indicating that the current patient is dead). The Adam optimizer is used to minimize the loss function, with an initial learning rate of 0.00001, a decay rate of 0.00001, and a training cycle of 100 times.

[0054] The model is trained and validated using the training set and validation set, and C-index and accuracy (ACC) are used as performance indicators. C-index is used to evaluate the consistency between the predicted risk of the dataset and the actual survival status. Accuracy is used to evaluate the classification accuracy of the model. The definitions of the two indicators are as follows:

[0055]

[0056] Among them, n is the number of pairs that can be compared, Y represents the model's predicted value for the current patient, s is the actual observed value, that is, the patient's survival time, and I[~] is an indicator function. If the condition in the brackets is true, its value is 1, otherwise it is 0. The C-index value range is 0 to 1. The larger the C-index value, the better the prediction performance of the model, and vice versa.

[0057]

[0058] Among them, TP means that the true value is the number of samples predicted correctly in the positive class; TN means that the true value is the number of samples predicted correctly in the negative class; FP means that the true value is the number of samples predicted as positive in the negative class; FN means that the true value is the number of samples predicted as negative in the positive class.

[0059] Further, combined with experimental data, the following is explained:

[0060] Table 1 Experimental results under different modes

[0061] Experimental methods ACC C-Index 5× 60.3％ 0.614 20× 60.4％ 0.603 Clinical structured information 58.5％ 0.587 5×+20× 64.1％ 0.603 Method of the present invention 67.9％ 0.68

[0062] According to Table 1, compared with the single-modal survival prediction method, the weakly supervised gastric cancer patient survival prediction method based on multimodal contrast learning of histopathological images proposed in the present invention has better performance. Compared with the conventional histopathological image classification task, it combines the global features at 5 times the resolution, the local fine-grained features at 20 times the resolution and the structured clinical information, and the sample features are richer, realizing the complementarity between the image and table features, which can produce more reliable predictions. Figure 2-Figure 6 As shown, it is a confusion matrix diagram obtained by other single-mode methods and the method of the present invention (the present invention is as shown in Figure 6 As shown in the result diagram, it can be seen that the classification performance of the present invention is higher than that of other models. Figure 2 The single-modal method shown in FIG. 5 is to use the SimCLR model to obtain the sub-block feature representation of the 5× tissue pathology image sub-block, and then directly process the sub-block feature representation through the multi-instance aggregation module and the classifier to obtain the output result; Figure 3 Similarly; Figure 4 The corresponding method is to use a multi-layer perceptron to predict clinical structured information after preprocessing; Figure 5 The method under different magnifications shown is to input 5× and 20× tissue pathology image sub-blocks into the SimCLR model to obtain sub-block feature representation, and then directly perform feature splicing on the sub-block feature representation and perform multi-instance aggregation module and classifier processing to obtain the output result; the method of the present invention adopts the first preset magnification of 5 times and the second preset magnification of 20 times; the value of y is set to 3.

[0063] The above cases are used to illustrate the effectiveness and feasibility of the present invention. The focus of the present invention is to use multiple different modal data to predict the survival of gastric cancer patients, but the number of modal features is not limited to the fixed number in the implementation case. At the same time, the use of contrastive learning weak supervision method is to solve the problem that effective feature extraction cannot be performed due to the lack of fine-grained labels in tissue pathology images, rather than targeting a specific case, and it is not intended to limit the present invention. Within the spirit of the present invention and the scope of protection of the claims, any modification or improvement of the present invention will be limited by the scope of protection of the present invention.

[0064] According to another aspect of an embodiment of the present invention, a weakly supervised gastric cancer tissue pathology image classification system based on contrastive learning multimodality is provided, comprising:

[0065] An acquisition module, used to acquire pathological images of gastric cancer patients and corresponding clinical structured data;

[0066] The first acquisition module is used to preprocess the histopathological image and clinical structured data to obtain histopathological image sub-blocks at different magnifications and corresponding preprocessed clinical structured data;

[0067] The second acquisition module is used to input the histopathological image sub-blocks at different magnifications into the feature extraction model for comparative learning training to obtain the feature extraction model weights of the histopathological images at different preset magnifications; wherein the feature extraction model is a SimCLR model with ResNet50 as the backbone network;

[0068] The third acquisition module is used to import the weights obtained by the second acquisition module into the feature extraction model to obtain the histopathological image feature extractors at different preset magnifications; the histopathological image sub-blocks at different magnifications are input into the feature extractor, and the corresponding histopathological image sub-block feature representations are output; at the same time, the pre-processed clinical structured data is input into a custom multi-layer perceptron to obtain the clinical structured data feature representation of the same dimension as the histopathological image sub-block feature representation;

[0069] A partitioning module, used for partitioning the histopathological image sub-block feature representation output by the third acquisition module and the one-to-one corresponding clinical structured data feature representation into a training set, a validation set and a test set;

[0070] The classification module is used to input the training set data into the multimodal fusion model for training, and then use the validation set to verify the model performance obtained from each batch of training, and save the parameters of the multimodal fusion model that best achieves the best indicators on the validation set as the weight of the final classification model; finally, verify the saved model performance on the test set; and classify the tissue pathology images to be classified.

[0071] According to another aspect of an embodiment of the present invention, a processor is provided, wherein the processor is used to run a program, wherein when the program is run, any one of the above-mentioned weakly supervised gastric cancer tissue pathology image classification methods based on contrastive learning multimodality is executed.

[0072] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A weakly supervised gastric cancer tissue pathology image classification method based on contrastive learning multimodality, characterized in that: include: S1. Collect pathological images and corresponding clinical structured data of gastric cancer patients; S2, preprocessing the histopathological image and clinical structured data to obtain histopathological image sub-blocks at different magnifications and corresponding preprocessed clinical structured data; S3, inputting the histopathological image sub-blocks at different magnifications into the feature extraction model for comparative learning training, and obtaining the feature extraction model weights of the histopathological images at different preset magnifications; S4, importing the weights obtained in S3 into the feature extraction model to obtain a histopathological image feature extractor at different preset magnifications; inputting the histopathological image sub-blocks at different magnifications into the feature extractor, and outputting the corresponding histopathological image sub-block feature representation; at the same time, inputting the preprocessed clinical structured data into a customized multi-layer perceptron to obtain a clinical structured data feature representation of the same dimension as the histopathological image sub-block feature representation; S5, dividing the histopathological image sub-block feature representation output by S4 and the one-to-one corresponding clinical structured data feature representation into a training set, a validation set and a test set; S6, input the training set data of S5 into the multimodal fusion model for training, then use the validation set of S5 to verify the performance of the model obtained by each batch of training, and save the parameters of the multimodal fusion model that best obtains the best index on the validation set as the weight of the final classification model; Finally, the performance of the saved model is verified on the test set of S5; Preprocessing of each tissue pathology section includes: removing unmatched data, using a sliding window to cut into sub-blocks of preset size, discarding blank sub-blocks, and performing staining standardization; The multimodal fusion model consists of three modules: a multi-instance aggregation module, a tensor fusion module, and a classifier; the feature representations of the sub-blocks of the pathological images at different preset magnifications are spliced ​​to generate instance-level embeddings; and the package-level feature embeddings are generated from the instance-level features through the multi-instance aggregation module; the instance-level embeddings, package-level feature embeddings and clinical structured data feature representations are multimodally fused through the tensor fusion module to output a multimodal fusion vector; and finally, the survival status prediction result is output through a classifier composed of a fully connected layer; The tensor fusion module is used to expand the single-modality dimension to the same dimension using 1 for clinical structured data feature representation, instance-level embedding and package-level feature embedding, and then calculate the Kronecker product of each modality to obtain a multi-modal fusion vector.

2. The method for weakly supervised gastric cancer tissue pathology image classification based on contrastive learning multimodality according to claim 1 is characterized in that: The specific preprocessing of each tissue pathology image is as follows: Based on the survival label information of clinical structured data, histopathological images with no survival status information, no overall survival information, survival status of 0 and overall survival less than y years, and survival status of 1 and overall survival greater than y years were eliminated; The histopathological images at two different preset magnifications are cropped into sub-blocks of preset sizes using a sliding window, and the sub-blocks whose blank background exceeds the first preset threshold are filtered out; all the sub-blocks are color standardized. After the above operations are completed, the cropped sub-blocks are saved separately according to the corresponding histopathological images.

3. The method for weakly supervised gastric cancer tissue pathology image classification based on contrastive learning multimodality according to claim 1, characterized in that: Preprocessing of clinical structured data includes: selecting attributes related to patient survival, removing unmatched data, discretizing continuous data in clinical structured data and performing one-hot encoding; among them, clinical structured data includes attribute features and survival labels.

4. The method for weakly supervised gastric cancer tissue pathology image classification based on contrastive learning multimodality according to claim 1, characterized in that: The multilayer perceptron customization is specifically as follows: the multilayer perceptron consists of three fully connected layers, respectively having 128, 256, and 512 nodes, and uses a ReLU activation function.

5. The method for weakly supervised gastric cancer tissue pathology image classification based on contrastive learning multimodality according to claim 1, characterized in that: The multi-instance aggregation module is used to obtain the key instance feature p with the highest score by performing maximum pooling on all instance-level features in the sub-tile instance-level embedding. key , each instance feature p f Decomposed into a query vector q f Sum value vector v f ; respectively expressed as: q f =W a *p f , v f =W v *p f , f=1,…,n, where W a and W v is a weight matrix; then a method similar to the self-attention mechanism is used to calculate the attention score between the key instance feature and each instance feature; the obtained attention score Score is used as the weight for each value vector v f The weighted information vectors of all instance features are weighted and added together to form a package-level feature embedding.

6. The method for weakly supervised gastric cancer tissue pathology image classification based on contrastive learning multimodality according to claim 5, characterized in that: The method expression similar to the self-attention mechanism is: Where <*,*> represents the inner product of two eigenvectors.

7. A weakly supervised gastric cancer tissue pathology image classification system based on contrastive learning multimodality, characterized by: A module comprising the method according to any one of claims 1 to 6.

Images