Full-section classification method and device for tissue pathology of generic cancer species

Abstract

The invention relates to a medical image processing technology, and discloses a pan cancer species histopathologic full-slice classification method and device, which is a trained multi-modal pan cancer species classification model based on histopathologic full-slice image data and full-slice diagnosis report pairing data of various tissue parts and various cancer species, and has the advantages of wide application range and high efficiency. And full slices of various pathological tissues can be classified. For classification tasks of different tissue parts, only text prompts (prompts) are used, i.e., a candidate cancer species list is set, and zero-sample full-slice classification can be realized without training. And for rare cancer species, full-slice classification can be carried out. Besides, the model has high expansibility, in order to better adapt to downstream tasks, better classification performance can be achieved by combining slice embedding with a linear classifier for use, freezing a slice embedding part and only finely adjusting the linear classifier, and the requirement for the training data size is low.

Description

Technical Field

[0001] The invention relates to medical image processing technology, and in particular to a method and device for classifying pan-cancer tissue pathology full slices. Background Art

[0002] Currently, there are mainly the following methods for the task of classifying whole tissue pathology sections:

[0003] Image classification methods or semantic segmentation methods based on classic convolutional neural networks (CNNs) both require cutting the entire slice into a series of image blocks, then classifying or segmenting each image block, and finally fusing the results of all image blocks into slice-level results based on the classification decision strategy or segmentation region fusion. The entire process is not end-to-end, and the fusion decision from image blocks to slice levels varies for different classification tasks, and generalization of different tasks cannot be achieved. In addition, methods based on semantic segmentation require labeling of regions of interest (ROI), which is time-consuming and labor-intensive, and labeling consistency is difficult to ensure.

[0004] Compared with method 1, the classification method based on multiple instance learning (MIL) can achieve end-to-end slice-level classification, but it cannot efficiently extract global features for the whole slice, and it is difficult to capture the effective area for slices with a small proportion of positive areas, resulting in reduced classification accuracy. Both method 1 and method 2 cannot achieve zero-shot classification, that is, they cannot effectively predict unseen cancer types; and when a single model continuously learns multiple classification tasks, catastrophic forgetting will occur, and it is easy to overfit the current learning task. In addition, the current method does not fully utilize the rich and diverse semantic information in clinical diagnosis reports to assist in the classification of whole tissue pathology slices. Summary of the invention

[0005] The present invention aims to solve the problems in the prior art of lack of high-quality annotated data; semantic segmentation data involves the annotation of regions of interest (ROI), which is time-consuming and difficult to ensure the consistency of annotation; most current algorithms are based on separate modeling of single tissue sites and do not have zero-shot capabilities, i.e., they cannot effectively predict unseen cancer types; the identification of new cancer types in untrained new tissue sites requires retraining and modeling, and the features of other trained tissue sections cannot be efficiently shared, and training requires more data. A method and device for pan-cancer tissue pathology full-slice classification is provided.

[0006] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0007] A pan-cancer tissue pathology whole-slice classification method, the method comprising:

[0008] Construction of training datasets, including pre-training dataset PD and whole-slice diagnostic report paired dataset W2R;

[0009] The encoder module training includes the tile encoder TiE, the slice encoder SE and the text encoder TeE. The dataset PD is used to train the tile encoder TiE, and the diagnostic report paired dataset W2R is used to train the slice encoder SE and the text encoder TeE.

[0010] Classification prediction of whole slice of tissue pathology, based on the above trained encoder model, input the whole slice image WSI to be classified i And the candidate cancer list D, synonym expansion is performed on each cancer type in the original candidate cancer list D, model reasoning is performed in a test enhancement manner, multiple prediction results are obtained, and the full slice classification output is finally obtained using the fusion method.

[0011] Preferably, the training of the encoder module includes:

[0012] The dataset PD is used to train the tile encoder TiE;

[0013] Based on the full-slice diagnosis report paired dataset W2R, the preprocessing module and the tile encoder TiE, the image triplet loss and contrast loss are used to train the slice encoder SE and text encoder TeE with the Tile Dropout strategy.

[0014] Preferably, the Tile Dropout strategy is used to train the slice encoder SE and the text encoder TeE, including:

[0015] For each foreground tile t i Generate a random number r i ∈[0,1];

[0016] If r i <p drop , then discard tile t i , that is, the block will not be input into the model in this round of training; i ≥p drop , then keep the block to participate in training; where p drop is the probability of discarding;

[0017] The input set after random Tile Dropout is T train ={t i |r i ≥p drop}, T train The size is ntrain ≤n, where n is the number of original foreground patches.

[0018] As a preference, the image triplet loss and contrast loss are used to train the slice encoder SE and the text encoder TeE with the Tile Dropout strategy, and the total loss function L total The image triplet loss L image and contrast loss L con Composition; among them, L total =λL image +(1-λ)L con , λ is a hyperparameter;

[0019]

[0020] Where N is the batch size, that is, the number of image triplets in each training batch;

[0021]

[0022] Where I is the normalized embedding of the WSI image after being encoded by the slice encoder SE, is the embedding of the reference sample image, is with Image embedding of samples belonging to the same category, is with Image embeddings of samples belonging to different categories, d(…) is the distance between two embeddings, and α is a margin hyperparameter used to ensure and The distance and and There is a sufficient interval between the distances. The meaning of this loss function is: when and The distance exceeds and When the distance is greater than 0 and the margin condition is not met, the loss value is positive, otherwise it is zero; when the loss is zero, it means and The distance in the feature space is close enough, and and The distance between is far enough; d(…) uses the cosine distance to measure the difference between two normalized embedding vectors. The calculation formula is as follows:

[0023]

[0024] Among them, A and B are normalized embeddings respectively;

[0025] Contrastive loss L con The calculation is as follows:

[0026]

[0027] Among them, x i and j are the normalized embeddings of the i-th image pair and the j-th text pair, respectively. Since a triple contains 3 pairs of full slices and the corresponding clinical diagnosis reports, N′=N*3, and σ is the temperature parameter learned during training, which is used to scale the logarithmic probability. The contrast loss function is obtained by the sum of the cross entropy losses of image-to-text and text-to-image.

[0028] Preferably, the classification prediction of the whole tissue pathology slide is tested by synonym expansion to enhance the model reasoning, including:

[0029] Construct a synonym table, construct a synonym table of a mapping table between cancer types and their synonyms, and the synonym table is obtained by manual collection or by using medical knowledge bases, dictionaries, etc.; suppose that the cancer type name list D = [d1, d2, ... d n ], for each cancer name d i , there is a corresponding synonym set S(d i )={S i1 ,S i2 ,…S im}, where d i and S i1 ,S i2 ,…S im are synonyms, each S(d i ) may contain different numbers of synonyms; then the synonym expansion set is: T(d i )={d i ,S i1 ,S i2 ,…S im}, that is, T(d i )={t i1 ,t i2 ,…t ik}, where k = m + 1, including the original cancer name and all synonyms. i If there is no synonym, then T(d i ) contains only the name of the original cancer species i an element;

[0030] Generate a synonym-based test combination DL by replacing the cancer type d in the original cancer type list D i For its synonym expansion set T(di ) to generate the synonym set D'1 of the cancer type list D. Similarly, multiple synonym sets of D can be obtained by traversing, that is, the test combination Where M is T(d i ) maximum number of elements.

[0031] Multiple inferences, given a full slice image WSI i and the cancer list D' in the above test combination DL k , then the whole slice WSI i Belong to D' k The predicted probability of each cancer type in is y k =f(WSI i ;D' k ), where f(…) is the model reasoning process;

[0032] Traverse the cancer type list in DL and perform model inference in sequence to obtain multiple prediction results. Among them, based on the trained encoder module, for the input full slice WSI i and cancer list D' k , the model reasoning process of f(…) is:

[0033] Get the normalized slice embedding vector v: full slice WSI i The preprocessing module PM processes the foreground tiles into a series of foreground tiles, which are then encoded into tile embedding sequences by the tile encoder TiE. Finally, the tile embedding sequences are aggregated and mapped into slice embedding vectors v by the slice encoder SE.

[0034] Get the normalized text embedding vector set T corresponding to the cancer type list: cancer type list D' k Each cancer name in is encoded into a normalized text embedding vector t by the text encoder TeE, then the set T is D' k The vector set consisting of the text embedding t of all cancer types in ;

[0035] Obtaining whole-slice WSI i Belong to D' k The predicted probability y of each cancer type in k : Calculate each text embedding vector t in T i The cosine similarity between the slice embedding vector v and the full slice WSI is obtained by applying the softmax activation function i Belongs to cancer type c∈D' k The probability y k , the specific calculation formula is:

[0036]

[0037] Where exp is the natural exponential function and σ is the temperature parameter learned during the training of the above encoder module.

[0038] Results fusion: The multiple prediction results obtained above are summarized by fusion method, and finally the full slice classification result after test enhancement is output; the fusion method of probability average is used to obtain the fused prediction probability vector Final whole slice WSI i The classification category is y fused The position corresponding to the maximum value in corresponds to the cancer type in the cancer type list D.

[0039] In order to solve the above technical problems, the present invention provides a pan-cancer tissue pathology whole-slice classification device, which comprises:

[0040] The building module of the training dataset includes the pre-training dataset PD and the whole-slice diagnostic report paired dataset W2R;

[0041] The training module of the encoder module includes a tile encoder TiE, a slice encoder SE and a text encoder TeE; the dataset PD is used to train the tile encoder TiE, and the diagnosis report paired dataset W2R is used to train the slice encoder SE and the text encoder TeE;

[0042] The classification prediction module of the whole slice of tissue pathology is based on the above-mentioned trained encoder model and inputs the whole slice image WSI to be classified i And the candidate cancer list D, synonym expansion is performed on each cancer type in the original candidate cancer list D, model reasoning is performed in a test enhancement manner, multiple prediction results are obtained, and the full slice classification output is finally obtained using the fusion method.

[0043] The present invention has significant technical effects due to the adoption of the above technical solution:

[0044] The present invention is based on the multimodal pan-cancer classification model trained based on the histopathological full-slice image data of various tissue sites and various cancer types and the paired data of the full-slice diagnostic report. It has a wide range of applications and can classify full slices of various pathological tissues. For classification tasks at different tissue sites, only text prompts are used, that is, a list of candidate cancer types is set, and zero-sample full-slice classification can be achieved without training. For rare cancer types, full-slice classification can also be performed. In addition, the model has strong scalability. In order to better adapt to downstream tasks, by combining slice embedding with a linear classifier, freezing the slice embedding part, and only fine-tuning the linear classifier, better classification performance can be achieved, and the requirement for the amount of training data is low.

[0045] The present invention introduces image triplet loss and cooperates with contrast loss to train a pan-cancer multimodal classification model, which can fully utilize and share the slice image feature information of different tissues and different cancer types, as well as the semantic feature information of the diagnosis report, and realize feature alignment of the two modalities. For classification tasks of different tissue parts and different cancer types, only text prompts are used, that is, a list of candidate cancer types is set, and zero-sample pan-cancer full-slice classification can be achieved without training, rather than separate modeling for different classification tasks.

[0046] The Tile Dropout strategy is introduced during model training: by randomly discarding some foreground tiles, the model will not be overly dependent on certain specific tile combinations, which plays a role in data enhancement and makes the model more robust; in addition, it can also reduce the computational overhead during training and accelerate the training process;

[0047] The present invention introduces a test enhancement of synonym expansion during model reasoning to increase the input diversity of the model, thereby allowing the model to more robustly handle disease names with different expressions and improve the accuracy of the model in pan-cancer classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is the overall flow chart of the present invention;

[0049] Figure 2 This is a training flowchart of the encoder module of the present invention.

[0050] Figure 3 The figure is a flow chart of classification prediction of whole tissue pathology sections of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0052] Example 1

[0053] A pan-cancer tissue pathology whole-slice classification method. Figure 1 The method includes the construction of training dataset, the training of encoder module and the classification prediction of whole tissue pathology sections.

[0054] The construction of the training dataset includes the construction of the pre-training dataset PD and the paired dataset W2R of whole-slice images and diagnostic reports. PD contains whole-slice images (WSI) of various resolutions from various data sources and various tissue sites. The slice resolution includes various mpp (microns-per-pixel), specifically 2.0, 1.0, 0.5 and 0.25mpp, corresponding to sampling at 5x, 10x, 20x and 40x magnification, respectively. W2R consists of whole-slice images and corresponding clinical diagnostic reports, and does not include data that lacks WSI or one of the clinical diagnostic reports. Sensitive patient information is desensitized.

[0055] Figure 2 In the training of the encoder module, the training involves the training of three submodules: the tile encoder module TiE, the slice encoder module SE and the text encoder module TeE. The training process is divided into two stages: stage one, the training of the tile encoder module TiE; stage two, the training of the slice encoder module SE and the text encoder module TeE.

[0056] Stage 1: Training of the patch encoder module TiE;

[0057] (1) Construction of foreground image block dataset PD': The pre-training dataset PD is processed by the pre-processing module PM to obtain the foreground image block dataset PD'. Specifically, let WSI PD is a full-slice image from the pre-training dataset PD, then WSI PD After being processed by the preprocessing module PM, the corresponding foreground tiles set is obtained. Right now where f PM is the preprocessing module PM, each t i represents a foreground tile, and n is the full slice WSI PD The number of foreground blocks included, n may be different for different WSIs. In the embodiment, the preprocessing module PM adopts the same preprocessing process as the CLAM (clustering-constrained-attention multiple-instance learning) tool. i The size is 224x 224.

[0058] (2) Obtaining the tile encoder TiE: Based on the PD' dataset, the self-supervised learning framework DINOv2 is used for training to obtain the tile encoder TiE. The pre-trained tile encoder TiE is used to encode the tile image into a tile embedding, which serves as the basis for subsequent slice-level feature extraction and the entire full-slice classification task. In the embodiment, the model architecture of TiE is the ViT-H / 14 architecture with a patch size of 14; the tile image size of the input TiE is 224x 224, and the mpp of the image is 0.5; other training configurations remain the same as the original DINOv2.

[0059] Phase 2: training of the slice encoder module SE and the text encoder module TeE;

[0060] Based on the full slice image and diagnosis report paired dataset W2R, the preprocessing module PM and the tile encoder TiE trained in the above stage 1, the slice encoder module SE and the text encoder module TeE are trained with the Tile Dropout strategy using the image triplet loss and contrast loss. Specifically:

[0061] (1) Normalized slice embedding latent feature F s ' lide Acquisition: The WSI image is processed by the preprocessing module PM to obtain the foreground tile sequence T, and the Tile Dropout strategy is used to randomly discard some tiles; the remaining foreground tile sequence T train After the pre-trained tile encoder module TiE in stage 1, the normalized tile embedding sequence corresponding to the foreground tile sequence is obtained Where n is the number of foreground tiles, and different WSIs generally correspond to different n, and m is the length of the tile embedding feature; finally, the tile embedding sequence F is aggregated and mapped into the normalized slice embedding latent feature by the slice encoder module SE

[0062] Among them, the Tile Dropout strategy is:

[0063] For each foreground tile t i Generate a random number r i ∈[0,1];

[0064] If r i <p drop , then discard tile t i , that is, the block will not be input into the model in this round of training; i ≥p drop , then keep the block to participate in training; where p drop is the probability of discarding. In practical applications, the discarding probability p dropIt needs to be carefully selected. If it is too high, too much key information may be lost, resulting in the inability of the model to learn effectively; if it is too low, the data enhancement effect of dropout cannot be fully utilized. drop Take 0.01.

[0065] The input set after random Tile Dropout is T train ={t i |r i ≥p drop}, T train The size is n train ≤n, where n is the number of original foreground patches.

[0066] Alternative solutions to Tile Dropout: first calculate the tile embeddings for all foreground tiles, then randomly mask out some of the tile embeddings and first randomly discard some foreground tiles, then calculate the tile embeddings for the remaining foreground tiles. The latter is the current description scheme, and the two are alternative solutions to each other.

[0067] In the model training phase, the Tile Dropout strategy is used to randomly discard some foreground tiles, so that the model will not be overly dependent on certain specific tile combinations, which plays a role in data enhancement and makes the model more robust; in addition, it can also reduce the computational overhead during training and accelerate the training process. In the model inference phase, there is no Tile Dropout operation, that is, all foreground tiles participate in the final prediction to obtain stable and accurate prediction output.

[0068] The slice encoder module SE consists of a visual encoding layer VL and a linear projection layer VP. VL is responsible for embedding the tile into the sequence Further feature extraction and global information aggregation to obtain slice embedding features The linear projection layer VP consists of a fully connected layer with v input nodes and u output nodes and a normalization layer with Euclidean L2 norm, which is responsible for embedding the slice into the feature F slide Further converted into normalized slice embedding latent features

[0069] Preferably, the visual coding layer VL adopts a Perceiver network, which has high computational efficiency, strong cross-modal fusion capability, and the ability to capture global features, enhancing the diagnostic ability and robustness of the model. In the embodiment, u=5120, that is, a WSI image is ultimately encoded into a one-dimensional feature vector with a length of 5120; otherwise, m=2560, v=1280.

[0070] (2) Normalized text embedding latent features F t ' extObtaining: Assume a clinical diagnosis report is X = {x1, x2, ..., x n}, where x i Represents the words or characters in the report, then the report X is encoded by the text encoder module TeE to obtain the normalized text embedding latent features That is Ft' ext =f TeE (X;θ), where f TeE is the text encoder module TeE, θ is f TeE The model parameters in .

[0071] Furthermore, the text encoder module TeE consists of a text encoding layer TL and a linear projection layer TP. TL is responsible for encoding the text sequence X of the diagnosis report into a text embedding The linear projection layer TP consists of a fully connected layer with t input nodes and u output nodes and a normalization layer with Euclidean L2 norm, which is responsible for embedding the text Further converted into normalized text embedding latent features Its feature vector length is the same as the above slice embedding hidden feature F s ' lide In the embodiment, t=1024, u=5120, that is, a text sequence is first encoded into a one-dimensional feature vector with a length of 1024 by the text encoding layer TL in the text encoder module TeE, and then further mapped into a one-dimensional feature vector with a length of 5120 by the linear projection layer TP.

[0072] Preferably, the text encoding layer TL adopts the BioGPT structure. Since BioGPT is pre-trained based on large-scale biomedical corpus, it is better at understanding and encoding complex language expressions, terminology and logic in the biomedical field than general language models, thereby more accurately encoding complex medical terms and phrases into semantic vectors. This can not only improve the accuracy and robustness of the model, but also improve the model's ability to interpret and diagnose diseases.

[0073] (3) Calculation and back propagation of loss function: The total loss function L total The image triplet loss L image and contrast loss L con Composition, based on the full-slice image and diagnosis report paired dataset W2R, freeze the parameters of the pre-trained tile encoder module TiE, and use the loss function L total , perform back propagation to train the parameters of the slice encoder module SE and the text encoder module TeE until convergence.

[0074] Among them, L total =λL image +(1-λ)L con, λ is a hyperparameter;

[0075]

[0076] Where N is the batch size, i.e., the number of image triplets in each training batch; further,

[0077]

[0078] Where I is the normalized embedding of the WSI image, that is, the normalized slice embedding latent feature Specifically, is the embedding of the reference sample image, is with Image embedding of samples belonging to the same category, is with Image embeddings of samples belonging to different categories, d(…) is the distance between two embeddings, and α is a margin hyperparameter used to ensure and The distance and and There is a sufficient interval between the distances. The meaning of this loss function is: when and The distance exceeds and When the distance is greater than 0 and the margin condition is not met, the loss value is positive, otherwise it is zero; when the loss is zero, it means and The distance in the feature space is close enough, and and The distance between is far enough; d(…) uses the cosine distance to measure the difference between two normalized embedding vectors. The calculation formula is as follows:

[0079]

[0080] Among them, A and B are normalized embeddings respectively;

[0081] Image triplet loss L image The trained model is forced to make the embeddings of images with the same label closer, while the embeddings of images with different labels are farther away in the feature space, thereby producing embedding features with good discrimination, which helps to improve the image classification accuracy.

[0082] Contrastive loss L con The calculation is as follows:

[0083]

[0084] Among them, x i is the normalized slice embedding latent feature of the i-th pair of images, y j is the normalized text embedding latent feature of the jth pair of diagnostic reports; since a triple contains 3 pairs of full slices and the corresponding clinical diagnostic reports, N' = N * 3, where N is the batch size; σ is the temperature parameter learned during training, which is used to scale the logarithmic probability. Contrastive loss L con It is obtained by the sum of the cross entropy losses of image-to-text and text-to-image.

[0085] Contrastive loss function L con The goal is to maximize the cosine similarity between paired image and text projections, while minimizing the similarity between unpaired image and text projections. Based on contrastive loss, during model training, the parameters of the slice encoder module SE and the aligned text encoder module TeE are simultaneously updated through contrastive learning, which helps to realize cross-modal alignment applications, such as zero-shot image classification; multi-modal feature alignment also helps to improve the generalization of the model.

[0086] Figure 3 In the classification prediction of tissue pathology whole slice, based on the above trained encoder module, the whole slice image WSI to be classified is input i And the candidate cancer list D, synonym expansion is performed on each cancer type in the original candidate cancer list D, model reasoning is performed in a test enhancement manner, multiple prediction results are obtained, and the full slice classification output is finally obtained using the fusion method.

[0087] Specifically, a synonym table is constructed, which is a mapping table between cancer types and their synonyms. The synonym table is obtained by manual collection or by using a medical knowledge base, dictionary, etc.; suppose that the cancer type name list D = [d1, d2, ... d n ], for each cancer name d i , there is a corresponding synonym set S(d i )={S i1 ,S i2 ,…S im}, where d i and S i1 ,S i2 ,…S im are synonyms, each S(d i ) may contain different numbers of synonyms; then the synonym expansion set is: T(d i )={d i ,Si1 ,S i2 ,…S im}, that is, T(d i )={t i1 ,t i2 ,…t ik}, where k = m + 1, including the original cancer name and all synonyms. i If there is no synonym, then T(d i ) contains only the name of the original cancer species i an element;

[0088] Generate a synonym-based test combination DL by replacing the cancer type d in the original cancer type list D i For its synonym expansion set T(d i ) to generate the synonym set D'1 of the cancer type list D. Similarly, multiple synonym sets of D can be obtained by traversing, that is, the test combination Where M is T(d i ) maximum number of elements.

[0089] Multiple inferences, given a full slice image WSI i and the cancer list D' in the above test combination DL k , then the whole slice WSI i Belong to D' k The predicted probability of each cancer type in is y k =f(WSI i ;D' k ), where f(…) is the model inference process. By traversing the cancer type list in DL and performing model inference in sequence, multiple prediction results can be obtained.

[0090] Among them, based on the trained encoder module, for the input full slice WSI i and cancer list D' k , the model reasoning process of f(…) is:

[0091] (1) Obtain the normalized slice embedding latent feature vector v: Based on the trained tile encoder module TiE and slice encoder module SE, according to the above F′ slide The calculation process can obtain the normalized slice embedding latent feature vector v;

[0092] (2) Obtain the normalized text embedding latent feature set T corresponding to the cancer type list: Based on the trained text encoder TeE, according to the above F′ text Calculation process, cancer type list D' k Each cancer name in is encoded into a normalized text embedding latent feature vector t by TeE, then the set T is D' kThe vector set consisting of the text embedding latent feature vectors t of all cancer types in ;

[0093] (3) Obtaining whole-slice WSI i Belong to D' k The predicted probability y of each cancer type in k :Calculate the hidden feature vector t of each text embedding in T i The cosine similarity between the slice embedding latent feature vector v and the full slice WSI is obtained by applying the softmax activation function i Belongs to cancer type c∈D' k The probability y k , the specific calculation formula is:

[0094]

[0095] Where exp is the natural exponential function and σ is the temperature parameter learned during the training of the above encoder module.

[0096] The results are fused, and the multiple prediction results obtained above are summarized by the fusion method, and finally the full slice classification result after test enhancement is output. Preferably, the probability average fusion method is used to obtain the fused prediction probability vector Final whole slice WSI i The classification category is y fused The position corresponding to the maximum value in corresponds to the cancer type in the cancer type list D.

[0097] Example 2

[0098] Based on Example 1, this example is a pan-cancer tissue pathology whole-slice classification device, which includes:

[0099] The building module of the training dataset includes the pre-training dataset PD and the whole-slice diagnostic report paired dataset W2R;

[0100] The training module of the encoder module includes a tile encoder TiE, a slice encoder SE and a text encoder TeE; the dataset PD is used to train the tile encoder TiE, and the diagnosis report paired dataset W2R is used to train the slice encoder SE and the text encoder TeE;

[0101] The classification prediction module of the whole slice of tissue pathology is based on the above-mentioned trained encoder model and inputs the whole slice image WSI to be classified i And the candidate cancer list D, synonym expansion is performed on each cancer type in the original candidate cancer list D, model reasoning is performed in a test enhancement manner, multiple prediction results are obtained, and the full slice classification output is finally obtained using the fusion method.

Claims

1. A method for pan-cancer tissue pathology whole section classification, the method comprising: Construction of training datasets, including pre-training dataset PD and whole-slice diagnostic report paired dataset W2R; The encoder module training includes the tile encoder TiE, the slice encoder SE and the text encoder TeE. The dataset PD is used to train the tile encoder TiE, and the diagnostic report paired dataset W2R is used to train the slice encoder SE and the text encoder TeE. Classification prediction of whole slice of tissue pathology, based on the above trained encoder model, input the whole slice image WSI to be classified i And the candidate cancer list D, synonym expansion is performed on each cancer type in the original candidate cancer list D, model reasoning is performed in a test enhancement manner, multiple prediction results are obtained, and the full slice classification output is finally obtained using the fusion method.

2. A pan-cancer tissue pathology whole section classification method according to claim 1, characterized in that: The training of the encoder module includes: The dataset PD is used to train the tile encoder TiE; Based on the full-slice diagnosis report paired dataset W2R, the preprocessing module and the tile encoder TiE, the image triplet loss and contrast loss are used to train the slice encoder SE and the text encoder TeE with the Tile Dropout strategy.

3. A pan-cancer tissue pathology whole section classification method according to claim 2, characterized in that: The TileDropout strategy is used to train the slice encoder SE and text encoder TeE, including, For each foreground tile t i Generate a random number r i ∈[0,1]; If r i <p drop , then discard tile t i , that is, the block will not be input into the model in this round of training; i ≥p drop , then keep the block to participate in training; where p drop is the probability of discarding; The input set after random Tile Dropout is T train ={t i |r i ≥p drop }, T train The size is n train ≤n, where n is the number of original foreground patches.

4. A pan-cancer tissue pathology whole section classification method according to claim 2, characterized in that: Using the image triplet loss and contrast loss, the tile dropout strategy is used to train the slice encoder SE and the text encoder TeE. The total loss function L total The image triplet loss L image and contrast loss L con Composition; among them, L total =λL image +(1-λ)L con , λ is a hyperparameter; Where N is the batch size, that is, the number of image triplets in each training batch; Where I is the normalized embedding of the WSI image after being encoded by the slice encoder SE, is the embedding of the reference sample image, is with Image embedding of samples belonging to the same category, is with Image embeddings of samples belonging to different categories, d(…) is the distance between two embeddings, and α is a margin hyperparameter used to ensure and The distance and and There is a sufficient interval between the distances. The meaning of this loss function is: when and The distance exceeds and When the distance is greater than 0 and the margin condition is not met, the loss value is positive, otherwise it is zero; when the loss is zero, it means and The distance in the feature space is close enough, and and The distance between is far enough; d(…) uses the cosine distance to measure the difference between two normalized embedding vectors. The calculation formula is as follows: Among them, A and B are normalized embeddings respectively; Contrastive loss L con The calculation is as follows: Among them, x i and j are the normalized embeddings of the i-th image pair and the j-th text pair, respectively. Since a triple contains 3 pairs of full slices and the corresponding clinical diagnosis reports, N′=N*3, and σ is the temperature parameter learned during training, which is used to scale the logarithmic probability. The contrast loss function is obtained by the sum of the cross entropy losses of image-to-text and text-to-image.

5. The method for pan-cancer tissue pathology whole section classification according to claim 1, characterized in that: Classification prediction of whole tissue pathology slides is tested through synonym expansion to enhance model reasoning, including: Construct a synonym table, and construct a synonym table of a mapping table between cancer types and their synonyms. The synonym table is obtained by manual collection or by using medical knowledge bases, dictionaries, etc.; suppose that the cancer type name list D = [d1, d2, ... d n ], for each cancer name d i , there is a corresponding synonym set S(d i )={S i1 ,S i2 ,…S im }, where d i and S i1 ,S i2 ,…S im are synonyms, each S(d i ) may contain different numbers of synonyms; then the synonym expansion set is: T(d i )={d i ,S i1 ,S i2 ,…S im }, that is, T(d i )={t i1 ,t i2 ,…t ik }, where k = m + 1, including the original cancer name and all synonyms. i If there is no synonym, then T(d i ) contains only the name of the original cancer species i an element; Generate a synonym-based test combination DL by replacing the cancer type d in the original cancer type list D i For its synonym expansion set T(d i ) to generate the synonym set D'1 of the cancer type list D. Similarly, multiple synonym sets of D can be obtained by traversing, that is, the test combination Where M is T(d i ) maximum number of elements. Multiple inferences, given a full slice image WSI i And the list of cancer types in the above test combination DL Whole slice WSI i belong The predicted probability of each cancer type in Where f(…) is the model reasoning process; Traverse the cancer type list in DL and perform model inference in sequence to obtain multiple prediction results. Among them, based on the trained encoder module, for the input full slice WSI i and cancer types list The model reasoning process is: Get the normalized slice embedding vector v: full slice WSI i The preprocessing module PM processes the foreground tiles into a series of foreground tiles, which are then encoded into tile embedding sequences by the tile encoder TiE. Finally, the tile embedding sequences are aggregated and mapped into slice embedding vectors v by the slice encoder SE. Get the normalized text embedding vector set T corresponding to the cancer type list: Cancer type list Each cancer name in is encoded into a normalized text embedding vector t by the text encoder TeE, then the set T is The vector set consisting of the text embedding t of all cancer types in ; Obtaining whole-slice WSI i belong The predicted probability y of each cancer type in k : Calculate each text embedding vector t in T i The cosine similarity between the slice embedding vector v and the full slice WSI is obtained by applying the softmax activation function i Type of cancer The probability y k , the specific calculation formula is: Where exp is the natural exponential function, and σ is the temperature parameter learned during the training of the above encoder module; Results fusion: The multiple prediction results obtained above are summarized by fusion method, and finally the full slice classification result after test enhancement is output; the fusion method of probability average is used to obtain the fused prediction probability vector Final whole slice WSI i The classification category is y fused The position corresponding to the maximum value in corresponds to the cancer type in the cancer type list D.

6. A pan-cancer tissue pathology whole-slice classification device, comprising: The building module of the training dataset includes the pre-training dataset PD and the whole-slice diagnostic report paired dataset W2R; The training module of the encoder module includes a tile encoder TiE, a slice encoder SE and a text encoder TeE; the dataset PD is used to train the tile encoder TiE, and the diagnosis report paired dataset W2R is used to train the slice encoder SE and the text encoder TeE; The classification prediction module of the whole slice of tissue pathology is based on the above-mentioned trained encoder model and inputs the whole slice image WSI to be classified i And the candidate cancer list D, synonym expansion is performed on each cancer type in the original candidate cancer list D, model reasoning is performed in a test enhancement manner, multiple prediction results are obtained, and the full slice classification output is finally obtained using the fusion method.

Images