Weak supervision full-view digital slice classification method and device based on causal intervention

Abstract

The invention discloses a weak supervision full-view digital slice classification method and device based on causal intervention, and the method comprises the steps: segmenting a full-view digital slice (WSI) into a plurality of image blocks, forming a package containing a plurality of instances, and obtaining a package level label of the package; performing feature extraction on each image block by using a feature extractor to obtain instance features; processing the instance features by adopting a multi-instance learning (MIL) model to obtain packet level features; constructing a confusion set, and representing confusion features frequently appearing in the core mode in the training process through the confusion set; performing causal intervention on the packet level features based on the confusion set to obtain intervened packet level features; and performing classification based on the intervened packet level features to obtain a classification result of the WSI. According to the method, the problems of inaccurate classification and poor robustness caused by attention deviation and false correlation interference in traditional weak supervision full-view digital slice classification are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital slice processing, in particular to a weakly supervised whole-slide digital slice classification method and device based on causal intervention. BACKGROUND

[0002] Classification of whole-slide digital slice (WSI) is crucial in the fields of pathological diagnosis and medical research. WSI contains a large amount of tissue cell information, and accurate classification is helpful for disease diagnosis (such as cancer typing), prognosis, and the like. However, the cost of obtaining instance-level (such as single cell, tissue block) labels is extremely high, and actual multiple depends on package-level labels (only knowing the class of the entire WSI), forming a weakly supervised learning scenario.

[0003] When the traditional MIL model relies on the attention mechanism to aggregate instance features, it is easy to over-focus on high-frequency and typical patterns, and ignore key but low-frequency diagnostic information. For example, in lung cancer pathological slice classification, the model may continuously focus on the features of alveolar type structures due to their high frequency, but it may not be able to recognize solid type structures that are similar in shape to other cancer types, resulting in classification bias. There are non-diagnostic high-frequency co-occurring features in WSI, such as staining artifacts, regular noise introduced by tissue processing, and the like. When the model is trained, it may learn false associations between these features and class labels. For example, unevenly stained areas may be misjudged by the model as diagnostic evidence due to their frequent coexistence with certain cancer samples, reducing the robustness and accuracy of classification.

[0004] To solve the above problems, researchers have tried to optimize the MIL model, such as improving the pooling strategy and introducing more complex feature extractors, but the effect is limited. On the one hand, new pooling methods (such as simple graph pooling and attention variants) do not fundamentally distinguish between causal and false associations; on the other hand, simply improving the feature extraction capability cannot eliminate the interference of confusing factors on the classification decision, and the model is still prone to being misled by non-diagnostic features, making it difficult to meet the demand for accurate classification in the clinic.

[0005] In summary, the existing weakly supervised WSI classification method has defects in accuracy, robustness, and explainability due to the inability to effectively solve the attention bias and false correlation problem, and there is an urgent need for innovative technical solutions to introduce new perspectives such as causal inference, reshape feature learning and classification logic, and improve the reliability of WSI classification to assist pathological diagnosis and medical research. SUMMARY

[0006] Therefore, the present application provides a weakly supervised whole-slide digital slice classification method and device based on causal intervention, which solves the problem of attention bias and false correlation interference of the multiple instance learning (MIL) model in the traditional weakly supervised whole-slide digital slice (WSI) classification, resulting in inaccurate classification and poor robustness.

[0007] In order to achieve the above object, the present application provides the following technical scheme: a weakly supervised whole slide image classification method based on causal intervention, comprising the following steps:

[0008] The whole slide image (WSI) is divided into a plurality of image blocks, a bag containing a plurality of instances is formed, and a bag-level label of the bag is obtained;

[0009] A feature extractor is used to extract features of each image block to obtain instance features;

[0010] A multi-instance learning model is used to process the instance features to obtain bag-level features;

[0011] A confusion set is constructed to represent confusion features in core patterns that frequently occur during the training process;

[0012] The bag-level features are intervened based on the confusion set to obtain intervened bag-level features;

[0013] Classification is performed based on the intervened bag-level features to obtain a classification result of the WSI.

[0014] As a preferred scheme of the weakly supervised whole slide image classification method based on causal intervention, the construction of the confusion set specifically comprises:

[0015] A teacher model is used to process the WSI in the training set to obtain an attention score of the instance in each bag;

[0016] The top p% of high attention instances are selected as key instances according to the attention score;

[0017] The K-Means algorithm is used to cluster the key instances, and the clustering center is taken as the confusion set;

[0018] A momentum update strategy is used to update the confusion set, and the specific update formula is:

[0019] C = mu * C + (1-mu) * C (e)

[0020] In the formula, C is the updated global confusion set, C (e) is the confusion set of the current training period, mu is the momentum factor, 0 < mu < 1, and e represents the number of training periods.

[0021] As a preferred scheme of the weakly supervised whole slide image classification method based on causal intervention, before the WSI in the training set is processed by the teacher model, it further comprises:

[0022] The teacher model is trained, specifically: the teacher model is trained using the package level label, and cross entropy loss is minimized, so that the teacher model can identify high attention instances, and the classification rule of the teacher model is:

[0023]

[0024] In the formula, The teacher model outputs the package feature in the training period e, W1 is a classifier composed of a full connection layer, Y represents the classification result, and X represents the input full field digital slice.

[0025] As an optimal solution of the weakly supervised full field digital slice classification method based on causal intervention, the package level feature is intervened causally based on the confusion set, specifically including:

[0026] The package level feature output by the student model is taken as a query vector, the confusion set is taken as a key matrix and a value matrix, the interaction relationship between the two is modeled through a cross attention module of multi-head self-attention, and a projection representation is obtained.

[0027] In the cross attention module of multi-head self-attention, the calculation method of the hth head is:

[0028]

[0029] In the formula, is the mathematical expectation of the projection representation output by the hth attention head; P(c) is the confusion set class prior probability; Q h is the query vector of the hth attention head, which is composed of the hth segment dimension of the student model package feature ; K h is the key vector of the hth attention head, which is obtained by multiplying the confusion set C and the learnable matrix ; V h is the value vector of the hth attention head, which is obtained by multiplying the confusion set C and the learnable matrix ; d h is the feature dimension of a single attention head; U is the class cluster number of the confusion set;

[0030] Classification is performed based on the projection representation and the package level feature, and the specific classification formula is:

[0031] P(Y|do(X))≈P(Y|Z s )=ω(σ(Z s ))

[0032] In the formula, P(Y|do(X)) represents the probability that the sample belongs to the class Y after the causal intervention `do` operation is performed on the input X; Z sa bag-level feature vector output by the student model; ω(·) represents a Softmax activation function, which normalizes the classification score into a probability distribution; σ(·) represents a classifier of the student model, which maps the input feature to the original classification score.

[0033] As an optimal solution of the weakly supervised whole slide image classification method based on causal intervention, the bag-level feature is modified based on the projection representation, and the specific formula is:

[0034]

[0035] In the formula, is a bag-level feature vector output by the student model in a training cycle e; is a mathematical expectation of the projection representation of the confusion set C in the training cycle e, representing the comprehensive influence of the confusion factor on the bag feature.

[0036] As an optimal solution of the weakly supervised whole slide image classification method based on causal intervention, the feature extractor includes a ResNet-18 model based on ImageNet pre-training and / or a CTransPath model based on MoCoV3 self-supervised framework pre-training, wherein the ResNet-18 model extracts a 1024-dimensional feature vector, and the CTransPath model extracts a 768-dimensional feature vector.

[0037] As an optimal solution of the weakly supervised whole slide image classification method based on causal intervention, the method further includes: before classifying the WSI, performing feature fusion on the bag-level feature, and the feature fusion includes a topology-aware pooling based on a graph neural network, an aggregation strategy based on a self-attention mechanism of a Transformer, or a parallel feature fusion strategy based on a multi-branch.

[0038] The application also provides a weakly supervised whole slide image classification device based on causal intervention, comprising:

[0039] A segmentation module is configured to segment a whole slide image (WSI) into a plurality of image blocks, form a bag containing a plurality of instances, and obtain a bag-level label of the bag.

[0040] A feature extraction module is configured to extract features of each image block by using a feature extractor to obtain instance features.

[0041] A processing module is configured to process the instance features by using a multi-instance learning model to obtain bag-level features.

[0042] A construction module is configured to construct a confusion set, and the confusion set is used to represent confusion features in a core mode frequently appearing in a training process.

[0043] an intervention module, configured to perform causal intervention on the bag-level feature based on the confusion set, to obtain an intervened bag-level feature;

[0044] a classification module, configured to perform classification based on the intervened bag-level feature, to obtain a classification result of the WSI.

[0045] As an optimal solution of the weakly supervised whole slide digital slice classification device based on causal intervention, the construction module comprises:

[0046] a screening sub-module, configured to process the WSIs in the training set by using the teacher model, to obtain an attention score of an instance in each bag;

[0047] screening top p% high-attention instances as key instances according to the attention score;

[0048] a clustering sub-module, configured to cluster the key instances by using a K-Means algorithm, and take a clustering center as the confusion set;

[0049] an updating sub-module, configured to update the confusion set by using a momentum updating strategy, and a specific updating formula is:

[0050] C=μ*C+(1-μ)*C (e)

[0051] In the formula, C is an updated global confusion set, C (e) is a confusion set in a current training period, μ is a momentum factor, 0<μ<1, and e represents a number of training periods.

[0052] As an optimal solution of the weakly supervised whole slide digital slice classification device based on causal intervention, the intervention module comprises:

[0053] an interaction sub-module, configured to take a bag-level feature output by the student model as a query vector, take the confusion set as a key matrix and a value matrix, model an interaction relationship between the two by using a cross-attention module of a multi-head self-attention, and obtain a projection representation;

[0054] In the cross-attention module of the multi-head self-attention, a calculation manner of an hth head is:

[0055]

[0056] In the formula, is a mathematical expectation of the projection representation output by the hth attention head; P(c) is a confusion set class prior probability; Q h is a query vector of the hth attention head, which is composed of an hth segment dimension of the bag feature output by the student model; K h is a key vector of the hth attention head, which is composed of the confusion set C and a learnable matrix is multiplied to obtain h is the value vector of the h-th attention head, which is obtained by multiplying the confusion set C and the learnable matrix is multiplied to obtain d h is the feature dimension of a single attention head; U is the number of class clusters of the confusion set;

[0057] The correction sub-module is used for correction based on the projection representation and the package-level feature, and the specific formula is:

[0058]

[0059] In the formula, is the package-level feature vector output by the student model in the training period e; is the mathematical expectation of the projection representation of the confusion set C in the training period e, representing the comprehensive influence of the confusion factor on the package feature.

[0060] As an optimal solution of the weakly supervised full-view digital slice classification device based on causal intervention, in the classification module, classification is performed based on the projection representation and the package-level feature, and the specific classification formula is:

[0061] P(Y|do(X))≈P(Y|Z s )=ω(σ(Z s ))

[0062] In the formula, P(Y|do(X)) represents the probability that the sample belongs to the category Y after the causal intervention `do` operation is performed on the input X; Z s is the package-level feature vector output by the student model; ω(·) represents the Softmax activation function, which normalizes the classification score into a probability distribution; σ(·) represents the classifier of the student model, which maps the input feature into the original classification score.

[0063] The present application has the following advantages:

[0064] First, by using the MIL basic process and causal intervention, the attention deviation and false association problems of the traditional MIL are solved: the confusion set explicitly models the interference factors (such as staining artifacts and high-frequency non-diagnostic patterns), and the causal intervention cuts off the false path, so that the model focuses on the real diagnostic features (such as tumor cell morphology). In the Camelyon16 and TCGA-NSCLC dataset tests, compared with the traditional MIL method, the F1 score is improved by 1.13% to 6.68%, and the misjudgment rate (such as the confusion of adenocarcinoma and squamous carcinoma) is reduced.

[0065] Second, enhance model robustness: dynamically construct confusion set, adapt to the change of confusion mode in training (such as the difference of staining of different batches of WSI); the features after causal intervention are more stable, and the classification performance can be maintained under low-quality features (such as simple features extracted by ResNet-18), and the robustness is better than SOTA methods (such as TransMIL, CLAM).

[0066] Third, improve medical interpretability: confusion set visualization, trace the interference factors that the model focuses on; causal intervention disassembles the interaction of "bag feature-confusion feature", which helps pathologists understand model decision (such as confirming whether the model misjudges non-diagnostic areas), which meets the clinical demand for AI interpretability.

[0067] Fourth, adapt to weakly supervised clinical scenarios: only rely on bag-level labels (without instance annotation), reduce the cost of pathological annotation; "plug and play" causal intervention strategy can be seamlessly integrated into existing MIL pipeline (such as replacing traditional pooling module), with low engineering cost and accelerating clinical landing (such as remote pathological diagnosis and large-scale cancer screening). BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can be derived from the provided drawings without creative labor.

[0069] The structures, proportions, sizes, etc. shown in the specification are only used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and are not used to limit the conditions that the present application can be implemented, so they do not have technical significance. Any modification of structure, change of proportion relationship or adjustment of size, without affecting the effect and purpose that the present application can produce, should still fall within the scope of the technical content disclosed by the present application.

[0070] Figure 1 A flowchart of a weakly supervised whole slide digital slice classification method based on causal intervention provided in an embodiment of the present application is shown in the figure.

[0071] Figure 2 A causal graph model in a weakly supervised whole slide digital slice classification method based on causal intervention provided in an embodiment of the present application is shown in the figure.

[0072] Figure 3 A network framework of a weakly supervised whole slide digital slice classification method based on causal intervention provided in an embodiment of the present application is shown in the figure.

[0073] Figure 4 FIG. 1 shows an architecture of a weakly supervised whole slide image classification system according to an embodiment of the present application. DETAILED DESCRIPTION

[0074] The present application is described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0075] It is well known that the purpose of WSI classification is to achieve effective identification and classification of the entire tissue section, which is of great significance for pathological diagnosis, disease prediction and biomedical research. It is a unique image processing and analysis task, which is different from traditional image classification tasks, because the resolution of WSI is extremely high (for example, 10000x10000), usually containing millions or even billions of pixels, and containing complex tissue structure and cell information, so the analysis of WSI is a challenging task.

[0076] In actual scenarios, pixel-level or instance-level labels are difficult to obtain because they rely on professional pathologists for fine annotation, and the annotation process is time-consuming and labor-intensive. This results in that in the WSI classification task, traditional pixel or instance-based supervised learning methods cannot be directly applied. Therefore, the classification of WSI is usually carried out in a weakly supervised manner. The "weak" only reflects that the label of WSI is only a WSI-level label, that is, the entire section is given a general class label (such as cancer type, benign and malignant, etc.), without providing detailed annotation information at the pixel or instance level.

[0077] In the related art, a common WSI classification strategy is based on Multiple Instance Learning (MIL). In the MIL framework, each WSI is treated as a "bag" containing multiple image patches, and the label of the entire bag (i.e., the WSI-level label) is used for the training process. Specifically, the model attempts to learn how to select the most representative image patches (or instances) from each bag (i.e., each WSI) and predict the label of the entire bag based on these instances. A mainstream training pipeline for MIL-based WSI classification, given a WSI containing only WSI-level labels, first segments it into multiple image patches (instances). Then, a deep learning network pre-trained on ImageNet or self-supervised learning is used to extract the feature representation of each instance. Finally, the instance features in the bag are input into the MIL pooling layer to aggregate the bag-level representation and used for classification. The current innovations of this process mainly focus on the design of the pooling operator: for example, topology-aware pooling based on graph neural networks, self-attention mechanism aggregation strategy based on Transformers, and multi-branch parallel feature fusion strategy, etc.

[0078] Existing weakly supervised WSI classification pipelines usually adopt a bag-level label supervision paradigm, only using WSI-level coarse-grained class labels to measure the global consistency between the entire WSI and the given label by defining a bag-level loss function. This process usually contains the following two core steps: feature extraction and feature fusion. Due to the high resolution, high dimensionality, and strong heterogeneity of WSI data, existing methods mainly focus on designing better feature fusion networks. For example, Ilse et al. proposed ABMIL, which uses a neural network-based permutation-invariant aggregation operator to calculate the contribution of each instance; Zhang et al. (2022) developed a double-layer feature distillation MIL (DTFDMIL) for WSI classification, which uses a double-layer ABMIL and introduces pseudo-bags to assist model training. Considering the context information between instances; Xiang et al. proposed ILRA-MIL, which introduced an iterative low-rank attention mechanism in the multi-head self-attention network to model the global interaction between instances; Li et al. proposed a dual-stream multiple instance learning framework DSMIL, which integrates instance-level discriminators and bag-level aggregators through a hierarchical co-attention mechanism to explicitly simulate the local-to-global causal dependency in WSI.

[0079] While MIL-based methods have proven effectiveness in detecting cancer-related cases, their reliance on weak supervision introduces representational biases that can hinder robust generalization: models trained with WSI-level labels only (e.g., “lung adenocarcinoma (LUAD)” vs. “lung squamous cell carcinoma (LUSC)”) tend to anchor decision boundaries on highly distinguishable patterns far from class centroids (e.g., typical acinar structures in lung LUAD), while overlooking marginally valuable borderline patterns (e.g., solid-type structures similar in morphology to LUSC). During training, many MIL frameworks enhance the association between bag-level labels and key pathological patterns. However, this can cause models to learn non-causal shortcut patterns, such as stain-induced technical pathology confounders or tissue-related structural artifacts. The problem at hand stems from the over-optimization of patterns with class disparity during model training. These patterns, while important, contain non-causal spurious associations, causing models to learn confounded paths and introducing unexplainable non-causal dependencies.

[0080] The learning goal of causal inference is to eliminate spurious biases, ensuring that conclusions drawn from data truly reflect causal relationships between variables. In recent years, more and more computer vision tasks have benefited from causal relationships, such as image classification, object detection, image generation, and video understanding. On the WSI classification problem, some scholars have also proposed multi-instance learning methods combined with causal inference in recent years, such as Lin et al. proposed a causal inference method based on backdoor adjustment IBMIL, which constructed a confusion set on bag-level features, and alleviated the model from capturing spurious associations between bags and labels due to prior context misguidance. Chen et al. proposed a method based on front-door adjustment CaMIL, which intervened the influence of shortcut features (e.g., image color) in instances on prediction. In contrast, the present invention expands the confusion details that can exist in the MIL model learning process, and more specifically, the method proposed by the present invention is also based on backdoor adjustment like IBMIL, but the difference lies in that the present invention introduces a new variable to characterize the prior of the context. In terms of training strategy, IBMIL adopts a three-stage sequential training paradigm, using the model trained in the first stage to construct a static confusion set to intervene in the model training in the third stage. CaMIL adopts a one-stage intervention strategy, dynamically generating prototypes by randomly sampling instances from each training batch to intervene in training. In contrast, the present invention proposes an improved single-stage three-step causal intervention framework, the core innovation of which is to use distillation learning as the basis, dynamically constructing a confusion set using the knowledge learned by the teacher model to intervene in the training of the student model. The following is the specific content of the embodiments of the present invention.

[0081] Embodiment 1

[0082] Reference Figure 1Embodiment 1 of the present invention provides a weakly supervised full-field digital slice classification method based on causal intervention, comprising the following steps:

[0083] S1, dividing the full-view digital slide WSI into multiple image blocks, forming a package containing multiple instances, and obtaining a package-level label of the package;

[0084] S2. Using a feature extractor to extract features from each of the image blocks to obtain instance features;

[0085] S3. Processing the instance features using a multi-instance learning model to obtain packet-level features;

[0086] S4. Constructing a confusion set, using the confusion set to characterize the confusion features in the core patterns that frequently appear during the training process;

[0087] S5. Perform causal intervention on the packet-level features based on the confusion set to obtain packet-level features after intervention;

[0088] S6. Classify based on the packet-level features after the intervention to obtain a classification result of the WSI.

[0089] As a weakly supervised learning method, the core idea of ​​multiple instance learning is to treat each sample as a "bag" containing multiple instances and only label the entire bag, rather than each instance within the bag. In this framework, a bag is labeled positive if and only if it contains at least one positive instance; and a bag labeled negative means that all instances within it are negative. Formally, let X = {(x1, y1), ..., (x n ,y n )} is represented as a package containing n instances, where x i Represents an instance, y i Represented as instance x i Each package X has a unique corresponding label Y, which is defined as:

[0090]

[0091] In the package-level multi-instance learning method, model learning usually consists of three stages: 1) Feature extraction: extracting embeddings from each instance for subsequent learning based on the deep learning network f(·); 2) Feature fusion: assigning weights to each instance based on the feature fusion network g(·), and applying these weights to the corresponding features to generate a comprehensive package-level feature; 3) Classification decision: using the package-level features to train the classifier σ(·) to perform the final classification of the entire package. The above process is shown in formula (2):

[0092]

[0093] In the causal inference framework, a causal graph explicitly models the causal relationships between variables by directed acyclic graphs, where nodes represent random variables in the system and directed edges represent direct causal effects between variables. In the learning of weakly supervised MIL, the embodiment believes that the prediction results of the bag are not only affected by the features of the instances in the bag itself, but also significantly affected by the high-frequency information of the class difference, which contains the key clues for class distinction. From the perspective of information theory, the information expressed by the bag-level features can be understood as the projection of the bag on the high-frequency information, which means that the information in the bag does not exist in isolation, but is inherently related to the high-frequency information. Based on the above definition, the embodiment constructs a causal graph model to address the representation bias problem caused by excessive attention to discriminative instances in multi-instance learning, as shown in Figure 2 .

[0094] Figure 2 In the above SCM, part (a) captures the causal relationship of the classifier in WSI classification, and part (b) is the intervened SCM with blocked backdoor paths. Among them, C→X: C represents high-frequency information, and this path represents that C shows different patterns or distributions in different classes of X; C→M←X: variable M represents the transformed representation of WSI X on high-frequency information C; X→Y←M: for the prediction Y of WSI, it is not only affected by the direct features of X, but also contains the influence of non-causal high-frequency information M introduced by C. More precisely, the path X→Y represents that the features or attributes in X directly determine the prediction result Y. The path M→Y reveals the influence of the non-causal high-frequency information representation M introduced by the high-frequency information C on the prediction result Y.

[0095] As can be seen from the above SCM, C, as a confounder, establishes a spurious correlation between C and Y through the backdoor path X←C→M→Y. Therefore, the embodiment proposes a framework for eliminating confounding effects using causal intervention P(Y|do(X)). Among them, the operation do(·) means that a specific value is forcibly assigned to the variable X.

[0096] Since physical intervention is difficult to implement in the WSI classification scenario, the embodiment uses backdoor adjustment to approximately implement P(Y|do(X)). The core idea is to cut off the link X←C in part (b) of the above SCM by layering the confounder variable C as C={c1, c2,... c U} and forcing the value of C to be c1, c2,... c Figure 2 , thereby decoupling the conditional dependence of C on X. Formally, the backdoor adjustment of part (a) in the above SCM is: Figure 2

[0097]

[0098] ​where M denotes the projection of WSIX on the high-frequency information C. The specific implementation of the function h(·) is given below. P(c i ) denotes the prior distribution of the confusion variable C, which determines the influence of the context c i on the prediction.

[0099] Referring to Figure 3 , in the DVCausIF proposed in this embodiment, in each training cycle e, the teacher model first traverses the entire training data, then filters out all key instances in the training set to generate a confusion set, and finally uses the confusion set to intervene in the training of the student model. The entire process includes three key steps:

[0100] Step 1: Training of the teacher model. The purpose of this embodiment is to maximize P(Y|X) for learning a MIL-based classification model to identify high-attention instances. This model is called the teacher model, and its classification rule is shown in equation (4):

[0101]

[0102] where, denotes the training cycle e teacher model output bag feature, and W1 is a fully connected layer regarded as a classifier. During the training process, the teacher model minimizes the cross-entropy loss to encourage the model to give a high score to the correct class and a low score to the wrong class.

[0103] Step 2: Construction of the confusion set. The high-frequency information C is unobservable, not to mention stratifying it. In fact, not all high-frequency components have an impact on the model's decision, and the components that truly confuse the model's decision often exist in key instances (i.e., high-attention instances) and are encoded by the feature extraction network into high-level semantic features containing space, texture, color, etc. The commonality of key instance features reflects the core patterns that frequently appear during the training process, among which there will be potential confusion features that affect the model's decision. Therefore, this embodiment uses the K-Means algorithm to cluster all key instances in the training set, and regards the cluster centers as the confusion set.

[0104] Specifically, for the whole training set, the embodiment utilizes the attention network in the teacher model to output the instance attention scores in each bag, and filters the top p% high attention instances for the subsequent generation of the confusion set. Differently, the test set does not participate in the extraction of key instances and the generation of the confusion set. As the training progresses, the key degree of different instances for the improvement of model performance is not fixed, and the features associated with them that can cause the model to produce confusion are also in continuous evolution. In view of this dynamic characteristic, in order to more accurately capture the key confusion situation faced by the model at different training stages, the embodiment constructs a confusion set based on the "teacher model" currently being trained within each training period e. By adopting a momentum update strategy to smoothly integrate the periodic confusion sets. More specifically, by maintaining a global confusion feature set C, at the end of each training period t, according to the current confusion set C ( e) Momentum update C:

[0105] C = μ * C + (1 - μ) * C (e) (5)

[0106] where, U is the number of class clusters in the data set, and d represents the dimension of the instance feature. is the i-th cluster center. μ (0 < μ < 1) is the momentum factor. Initially, C = C (0) .

[0107] Step three: intervention training. Given the bag features and the projected representation m (e) output by the student model at training period e, use a linear model with softmax (denoted as ω) as the classifier: Then:

[0108]

[0109] Further apply the optimized P(Y|do(X)) of Normalized Weighted Geometric Mean. As formula (7):

[0110]

[0111] Where the linear model in the student model is designed as: where is the classification layer, is the fully connected layer. Then, formula (7) is derived as:

[0112]

[0113] In this embodiment, a cross attention module based on multi-head autonomous intention is implemented Specifically, by combining the bag features ( As the query vector, C is considered as the key matrix and the value matrix, which explicitly models the interaction between the two. In this process, the multi-head attention mechanism captures the feature correlation patterns in different semantic spaces by computing multiple attention heads in parallel. The h-th head is defined as follows:

[0114]

[0115] Where, and Represents the learnable weight matrix.d h = l / H, H represents the number of heads. Finally, m (e) The output of multi-head attention is spliced ​​and combined with the prior probability P(c i ) is obtained by multiplying them.

[0116]

[0117] The goal of this example is to maximize P(Y|do(X)) for causal learning without spurious correlations. Since the prediction of a packet comes from the intervention of the confusion set on the packet features, we have:

[0118] In order to verify the effect of the embodiment of the present invention, experiments were conducted on two public WSI datasets, Camelyon16 and TCGA-NSCLC. The Camelyon16 dataset contains 399 annotated WSIs (270 for model training and 129 for independent testing), and each WSI is marked as negative or positive for lymph node metastasis; the TCGA-NSCLC dataset is derived from the Cancer Genome Atlas (TCGA) project and contains a total of 1046 WSIs (512 of which are lung squamous cell carcinoma LUSC subtypes and 534 are lung adenocarcinoma LUAD subtypes). For the above datasets, the existing pre-trained features are directly reused as model input, where each WSI has a feature matrix F∈R n×d , where n represents the number of instances in a single WSI after block processing, and d represents the instance feature dimension. This feature matrix is ​​generated by the following two encoders: 1) a ResNet-18 model pre-trained on ImageNet, which extracts a 1024-dimensional feature vector for each instance; and 2) a CTransPath model pre-trained on the MoCo V3 self-supervised framework, which extracts a 768-dimensional feature vector for each patch.

[0119] Table 1 Experimental results of combining different baselines with DVCausIF on the Camelon16 dataset (%)

[0120]

[0121]

[0122] Table 2 Experimental results of different baselines combined with DV-CausIF on TCGA-NSCLC dataset (%)

[0123]

[0124] During training, the official split was adopted for the Camelyon16 dataset, and five-fold cross-validation was adopted for the TCGA-NSCLC dataset. In the experimental part, the precision, recall, accuracy, recall, and area under curve (AUC) scores of each model are reported. The training period of all models is 50, and the learning rate is 0.0001. Regarding the confusion set, U is 16, p is 30, and the feature dimension l of the bag is 512. When updating the momentum, μ is 0.9. The number of heads H in the attention mechanism is 4. The hardware resources required for the experiment are as follows: L2048G graphics memory, 100G content. The CPU is 20 cores, Xeon(R) Platinum8457C.

[0125] The performance of the method DV-CausIF proposed in the present application and the baseline on the WSI classification task is integrated on two datasets. The experimental results are shown in Tables 1 and 2, wherein the model without using any inference is denoted as vanilla model, Δovervanilla represents the difference between “w / DV-CausIF” and “vanilla”, and ΔMean represents the average of all “Δovervanilla” in the column. From Tables 1 and 2, two findings can be seen: 1) the causal inference generally improves the indicators of the baseline on each dataset. DV-CausIF is more common than IBMIL and CaMIL in improving the model. Most obviously, on the Camelyon16 dataset based on the CTransPath extracted, IBMIL and CaMIL assisted DSMIL improved the F1 score by 1.81% and 6.34%, respectively, and the method of the present application improved by 6.68%. Since spurious correlations are ubiquitous, the model bias caused by spurious correlations can be effectively reduced by introducing a causal inference method. 2) Under the same feature extractor condition, the performance of DV-CausIF on TCGA-NSCLC is more stable than that of Camelyon16. This is because more bags are more conducive to the training of the model, and at the same time, there is enough information to approximate the confusion set.

[0126] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server, etc. The method of the embodiments can also be applied to a distributed scenario, and be completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present disclosure, and the multiple devices can interact with each other to complete the method of the present disclosure.

[0127] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0128] Embodiment 2

[0129] Referring to Figure 4 Embodiment 2 of the present disclosure also provides a device for weakly supervised whole slide image (WSI) classification based on causal intervention, comprising:

[0130] A segmentation module 100 is configured to segment a WSI into a plurality of image blocks, form a bag containing a plurality of instances, and obtain a bag-level label of the bag.

[0131] A feature extraction module 200 is configured to extract features of each of the image blocks by using a feature extractor, to obtain instance features.

[0132] A processing module 300 is configured to process the instance features by using a multi-instance learning model, to obtain bag-level features.

[0133] A construction module 400 is configured to construct a confusion set, and use the confusion set to represent confusion features in a core mode that frequently occurs in a training process.

[0134] An intervention module 500 is configured to perform causal intervention on the bag-level features based on the confusion set, to obtain intervened bag-level features.

[0135] A classification module 600 is configured to perform classification based on the intervened bag-level features, to obtain a classification result of the WSI.

[0136] In the embodiments, the construction module 400 comprises:

[0137] The screening submodule 401 is configured to process the WSI in the training set by using the teacher model to obtain the attention score of each instance in the bag;

[0138] The top p% of high attention instances are screened out as key instances according to the attention score;

[0139] The clustering submodule 402 is configured to cluster the key instances by using the K-Means algorithm, and take the clustering center as the confusion set;

[0140] The updating submodule 403 is configured to update the confusion set by using a momentum updating strategy, and the specific updating formula is:

[0141] C = μ * C + (1-μ) * C (e)

[0142] In the formula, C is the updated global confusion set, C (e) is the confusion set of the current training period, μ is a momentum factor, 0 < μ < 1, and e represents the number of training periods.

[0143] In the embodiment, the intervention module 500 includes:

[0144] The interaction submodule 501 is configured to take the bag-level feature output by the student model as a query vector, take the confusion set as a key matrix and a value matrix, model the interaction relationship between the two by using a multi-head self-attention cross-attention module, and obtain a projection representation.

[0145] In the multi-head self-attention cross-attention module, the calculation manner of the hth head is:

[0146]

[0147] In the formula, is the mathematical expectation of the projection representation output by the hth attention head; P(c) is the confusion set class prior probability; Q h is the query vector of the hth attention head, which is composed of the hth segment of dimensions of the bag feature output by the student model; K h is the key vector of the hth attention head, which is obtained by multiplying the confusion set C and a learnable matrix ; V h is the value vector of the hth attention head, which is obtained by multiplying the confusion set C and a learnable matrix ; d h is the feature dimension of a single attention head; and U is the class cluster number of the confusion set.

[0148] The correction submodule 502 is configured to correct based on the projection representation and the bag-level feature, and the specific formula is:

[0149]

[0150] wherein, is the bag-level feature vector output by the student model at training epoch e; is the mathematical expectation of the projected representation of the confusion set C at training epoch e, characterizing the combined influence of the confounders on the bag features;

[0151] In the classification module 600, classification is performed based on the projected representation and the bag-level features, and the specific classification formula is:

[0152] P(Y|do(X))≈P(Y|Z s )=ω(σ(Z s ))

[0153] wherein, P(Y|do(X)) represents the probability that the sample belongs to the class Y after the causal intervention `do` operation is performed on the input X; Z s is the bag-level feature vector output by the student model; ω(·) represents a Softmax activation function, which normalizes the classification scores into a probability distribution; σ(·) represents a classifier of the student model, which maps the input features into original classification scores.

[0154] It should be noted that the information interaction and execution process between the modules of the above apparatus, since the same technical effects as the method embodiment in Embodiment 1 of the present application based on the same concept, the specific content can be referred to the description of the method embodiment in the foregoing method embodiment of the present application, which will not be repeated here.

[0155] Embodiment 3

[0156] Embodiment 3 of the present application provides a non-transitory computer readable storage medium, the computer readable storage medium has a program code of a weakly supervised whole slide digital slide classification method based on causal intervention stored therein, and the program code includes instructions for executing the weakly supervised whole slide digital slide classification method based on causal intervention of Embodiment 1 or any possible implementation manner thereof.

[0157] The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (Solid State Disk, SSD)) and the like.

[0158] Embodiment 4

[0159] Embodiment 4 of the present application provides an electronic device, comprising a memory and a processor.

[0160] The processor and the memory complete the communication between each other through the bus; the memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the weakly supervised full-view digital slice classification method based on causal intervention of embodiment 1 or any possible implementation manner thereof.

[0161] Specifically, the processor can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which realizes by reading software codes stored in the memory. The memory can be integrated in the processor or exist independently outside the processor.

[0162] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable systems. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode.

[0163] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by a general computing system, which can be concentrated on a single computing system or distributed on a network composed of multiple computing systems, and optionally, they can be realized by program codes executable by the computing system, so that they can be stored in the storage system and executed by the computing system, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0164] Although the present application has been described in detail with general description and specific embodiments above, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present application. Therefore, these modifications or improvements made on the basis of not deviating from the spirit of the present application, all belong to the scope of protection claimed by the present application.

Claims

1. A weakly supervised full-field digital slice classification method based on causal intervention, characterized by: The following steps are involved: Segmenting the full-field digital slice WSI into multiple image blocks to form a package containing multiple instances, and obtaining a package-level label of the package; Using a feature extractor to extract features from each of the image blocks to obtain instance features; Using a multi-instance learning model to process the instance features to obtain packet-level features; Constructing a confusion set, wherein the confusion set represents the confusion features in the core patterns that frequently appear during the training process; Performing causal intervention on the package-level features based on the confusion set to obtain package-level features after intervention; Classification is performed based on the packet-level features after the intervention to obtain a classification result of the WSI.

2. The weakly supervised full-field digital slide classification method based on causal intervention according to claim 1 is characterized in that: The construction of the obfuscation set specifically includes: Use the teacher model to process the WSI in the training set to obtain the attention score of each instance in the bag; Filter out the top p% of high-attention instances as key instances based on the attention scores; Clustering the key instances using a K-Means algorithm, and using cluster centers as the confusion set; The momentum update strategy is used to update the confusion set. The specific update formula is: C=μ*C+(1-μ)*C (e) Where C is the updated global confusion set, C (e) is the confusion set of the current training cycle, μ is the momentum factor, 0<μ<1, and e represents the number of training cycles.

3. The weakly supervised full-field digital slide classification method based on causal intervention according to claim 2 is characterized in that: Before using the teacher model to process the WSI in the training set, it also includes: The teacher model is trained by using the packet-level labels to minimize the cross-entropy loss so that the teacher model can identify high-attention instances. The classification rule of the teacher model is: Where, The output package features of the teacher model in the training cycle e are shown in Figure 1. W1 is the classifier composed of fully connected layers, Y represents the classification result, and X represents the input full-view digital slice.

4. The weakly supervised full-field digital slide classification method based on causal intervention according to claim 1 is characterized in that: The causal intervention on the packet-level feature based on the confusion set specifically includes: The packet-level features output by the student model are used as the query vector, and the confusion set is used as the key matrix and value matrix. The interaction between the two is modeled through the cross-attention module of the multi-head self-attention to obtain the projected representation; In the cross-attention module of the multi-head self-attention, the h-th head is calculated as: Where, is the mathematical expectation of the projection representation of the h-th attention head output; P(c) is the prior probability of the confusion set category; Q h is the query vector of the h-th attention head, which is represented by the student model package features The hth dimension of K h is the key vector of the h-th attention head, which is composed of the confusion set C and the learnable matrix W h K Multiply to get; V h is the value vector of the h-th attention head, which is composed of the confusion set C and the learnable matrix W h V Multiply to get; d h is the feature dimension of a single attention head; U is the number of clusters in the confusion set; Classification is performed based on the projection representation and the package-level features. The specific classification formula is: P(Y|do(X))≈P(Y|Z s )=ω(σ(Z s )) Where P(Y|do(X)) represents the probability that the sample belongs to category Y after the causal intervention `do` operation is performed on the input X; Z s is the packet-level feature vector output by the student model; ω(·) represents the Softmax activation function, which normalizes the classification score to a probability distribution; σ(·) represents the classifier of the student model, which maps the input features to the original classification score.

5. The weakly supervised full-field digital slide classification method based on causal intervention according to claim 4 is characterized in that: The correction is performed based on the projection representation and the package-level features. The specific formula is: Where, is the packet-level feature vector output by the student model at training cycle e; It is the mathematical expectation of the projection representation of the confusion set C in the training cycle e, which represents the comprehensive impact of the confusion factors on the package features.

6. The weakly supervised full-field digital slide classification method based on causal intervention according to claim 1 is characterized in that: The feature extractor includes a ResNet-18 model pre-trained based on ImageNet and / or a CTransPath model pre-trained based on the MoCoV3 self-supervised framework, wherein the ResNet-18 model extracts a 1024-dimensional feature vector and the CTransPath model extracts a 768-dimensional feature vector.

7. The weakly supervised full-field digital slide classification method based on causal intervention according to any one of claims 1 to 6, characterized in that: Also includes: Before classifying the WSI, the packet-level features are subjected to feature fusion, wherein the feature fusion includes topology-aware pooling driven by a graph neural network, a Transformer-based self-attention mechanism aggregation strategy, or a multi-branch-based parallel feature fusion strategy.

8. A weakly supervised full-field digital slice classification device based on causal intervention, characterized in that: include: a segmentation module, configured to segment the full-field digital slide WSI into multiple image blocks, form a package containing multiple instances, and obtain a package-level label of the package; A feature extraction module, configured to extract features from each of the image blocks using a feature extractor to obtain instance features; A processing module, configured to process the instance features using a multi-instance learning model to obtain packet-level features; A construction module, configured to construct a confusion set, wherein the confusion set is used to represent confusion features in core patterns that frequently appear during training; An intervention module, configured to perform causal intervention on the packet-level features based on the confusion set to obtain the packet-level features after intervention; A classification module is used to perform classification based on the packet-level features after the intervention to obtain a classification result of the WSI.

9. The weakly supervised full-field digital slide classification device based on causal intervention according to claim 8 is characterized in that: The building blocks include: The filtering submodule is used to process the WSI in the training set using the teacher model to obtain the attention score of the instance in each package; Filter out the top p% of high-attention instances as key instances based on the attention scores; A clustering submodule, configured to cluster the key instances using a K-Means algorithm and use the cluster centers as the confusion sets; The update submodule is used to update the confusion set using a momentum update strategy. The specific update formula is: C=μ*C+(1-μ)*C (e) Where C is the updated global confusion set, C (e) is the confusion set of the current training cycle, μ is the momentum factor, 0<μ<1, and e represents the number of training cycles.

10. The weakly supervised full-field digital slide classification device based on causal intervention according to claim 8, characterized in that: The intervention module includes: The interaction submodule is used to take the packet-level features output by the student model as the query vector and the confusion set as the key matrix and value matrix, and to model the interaction between the two through the cross-attention module of the multi-head self-attention to obtain the projected representation; In the cross-attention module of the multi-head self-attention, the h-th head is calculated as: Where, is the mathematical expectation of the projection representation of the h-th attention head output; P(c) is the prior probability of the confusion set category; Q h is the query vector of the h-th attention head, which is represented by the student model package features The hth dimension of K h is the key vector of the h-th attention head, which is composed of the confusion set C and the learnable matrix W h K Multiply to get; V h is the value vector of the h-th attention head, which is composed of the confusion set C and the learnable matrix W h V Multiply to get; d h is the feature dimension of a single attention head; U is the number of clusters in the confusion set; The correction submodule is used to perform correction based on the projection representation and the packet-level features. The specific formula is: Where, is the packet-level feature vector output by the student model at training cycle e; It is the mathematical expectation of the projection representation of the confusion set C in the training cycle e, representing the comprehensive impact of the confusion factors on the packet features; In the classification module, classification is performed based on the projection representation and the package-level features. The specific classification formula is: P(Y|do(X))≈P(Y|Z s )=ω(σ(Z s )) Where P(Y|do(X)) represents the probability that the sample belongs to category Y after the causal intervention `do` operation is performed on the input X; Z s is the packet-level feature vector output by the student model; ω(·) represents the Softmax activation function, which normalizes the classification score to a probability distribution; σ(·) represents the classifier of the student model, which maps the input features to the original classification score.

Images