A pathological image classification method based on self-motivated multiple-instance learning

Abstract

This invention discloses a pathological image classification method based on self-motivated multi-instance learning. This method promotes better and more reliable decision-making by exploring the potential relationships between instances and the reciprocal relationship between feature representation and label prediction. The invention captures and aggregates tumor feature representations by introducing label-related category priors provided by pseudo-packet prediction to achieve high-accuracy pathological image label prediction. Conversely, the network is optimized based on the prediction results to further improve feature representation and thus improve pseudo-packet label prediction, forming a self-motivated learning mechanism. This invention introduces a multi-level feature fusion strategy to explore knowledge of current instances and global historical instances, while constructing a time comparison module to improve the robustness of feature representation and alleviate representation bias and overfitting problems. Furthermore, the self-motivated feature fusion module utilizes the mutual refinement mechanism between pseudo-packet prediction and feature representation to enhance the accuracy and reliability of pathological image classification.

Description

Technical Field

[0001] This invention belongs to the field of pathological image technology and relates to a pathological image classification method, specifically a pathological image classification method based on self-motivated multi-instance learning (SMILE). Background Technology

[0002] Pathological tissue images contain rich information on cell morphology and are the gold standard for clinical cancer diagnosis. However, manually examining multi-scale, complex tissue structures using an optical microscope to diagnose diseases is a tedious task for pathologists. Recently, digital whole-slide images (WSI) combined with deep learning have become an effective tool for assisting diagnosis. However, due to the irregularity of tumor tissue and the gigapixel resolution of WSI, collecting finely annotated WSI images of tumor regions is both laborious and time-consuming.

[0003] To overcome these limitations, weakly supervised methods, such as Multi-Instance Learning (MIL), utilize WSI labels (e.g., positive or negative) from clinical reports to train models for WSI classification tasks. Specifically, MIL-based techniques treat each WSI as a "bag" and aggregate disease-related representations from thousands of unlabeled instances sampled from the WSI for bag-level label prediction. This process can be summarized in three key steps: instance sampling, feature representation, and decision-making. However, due to the gigapixel count of WSIs, MIL-based algorithms still face significant challenges in capturing key tumor features for accurate label prediction.

[0004] To find reasonable solutions, some studies have integrated reinforcement learning, clustering algorithms, attention scores, and pseudo-labels into the instance sampling process, making the sampled pseudo-packets closer to the actual package labels. Regarding feature representation, researchers have incorporated traditional max pooling, average pooling, and Top-K instances into popular attention mechanisms, Transformers, and graph neural networks to select instances with significant disease information or fuse instance features for WSI package label prediction. Furthermore, to optimize the package label decision-making process, the authors of DTFD were the first to attempt dividing packages into multiple pseudo-packets (or sub-packets) and introducing pseudo-labels for these pseudo-packets. By predicting pseudo-packets to refine and aggregate package-level feature representations, this method improves the accuracy of package label decisions.

[0005] While these efforts have improved the accuracy of pathology image label prediction to some extent, the performance gains rely too heavily on complex design frameworks and loss constraints of specific processes (sampling, feature representation, or decision-making), while the joint representation and mutual learning among these three processes have been largely unexplored. Incorrect associations between feature representations and labels accumulate in the network, ultimately leading to unsatisfactory bag label predictions and exacerbating the risk of model overfitting. Particularly for small and dispersed WSI tumor tissues, these methods struggle to generate reliable and robust feature representations aligned with bag labels. These shortcomings impair the model's robustness and accuracy capabilities, such as... Figure 1 As shown. Summary of the Invention

[0006] To address the significant overfitting and low confidence issues inherent in existing multi-instance learning methods, this invention provides a pathological image classification method based on self-motivated multi-instance learning. This method promotes better and more reliable decision-making by exploring the potential relationships between instances and the reciprocal relationship between feature representation and label prediction. Specifically, a multi-level feature fusion and temporal contrast module is introduced to improve the robustness of feature representation and alleviate the overfitting problem. Simultaneously, label prediction is utilized to capture and aggregate label-related discriminative features to improve decision accuracy.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A pathological image classification method based on self-motivated multi-instance learning includes the following steps:

[0009] Step 1: Given a whole slice dataset of pathological images It contains N WSI X i and its corresponding label Y i In the WSI classification task based on multi-instance learning, X i Considered as the i-th "packet", containing B i Sampling instances Where H and W represent the height and width of an instance, b∈[1,B] i ], except for the "bag" tag Y i All instances outside of {0,1,…,C} Corresponding tags Unknown, where C represents the number of disease types in dataset X;

[0010] Step 2: Using pathological WSI images X i As input to the instance sampling strategy, the non-background organized regions of WSI are cropped without overlap, resulting in a large number of image patches, referred to as instances. Furthermore, a pre-trained deep neural network is used as the encoder to embed instances into feature vectors. At time t, sampling The set of feature vectors corresponding to the instances M represents the number of instance feature vectors contained in the sampled pseudo-packet, and T represents the input image X. i The dynamic interactive sampling of pseudo-packets of size M is executed a certain number of times; the specific steps of the instance sampling strategy are as follows: at time t, decision-driven similarity sampling selects the current sampled pseudo-packet. In the example attention score P AS The pre-P value above 0.5 PBCS ×M instances, P PBCS A token label probability score is assigned to the pseudo-package, and then the selected instance is measured against package X. i The cosine similarity of features among the remaining unsampled instances is used to select the most similar P. PBCS ×M instances enter the next pseudo-packet Meanwhile, the remaining (1-P) PBCS )×M instances using a random sampling strategy S RAND The above process can be expressed by the following formula:

[0011]

[0012] Step 3: Introduce a multi-layer attention selection and fusion module, which includes a Transformer-based module and an attention-based feature fusion module, to... and a randomly initialized token As input, the pseudo-packet instance feature vector is fused using the Transformer module. Enter An attention-based feature fusion module enables cross-aggregation of the current token across pseudo-packets. and historical tokens Information between them is represented by a token. The specific steps are as follows:

[0013] Step 31: Design a Token Position Vector Come and learn about fake packages Spatial information in;

[0014] Step 3.2: Randomly initialize a token vector. Extracting with the Transformer module Local context information of instance features is entered into the Token In the above process, it is described as follows:

[0015]

[0016] Step 33, in order to improve Consistency with bag labels, the attention-based feature fusion module utilizes To selectively merge historical tokens Global semantic information to obtain rich token representations The process is described as follows: the pseudo-packet token at time t is mapped to a query (Q), and the tokens before time t are mapped to keys (K) and values ​​(V). Next, the matrix similarity between Q and K is calculated and transformed into attention weights through Softmax. Based on attention weight Effective information is extracted from historical knowledge V. Subsequently, Q is added to the extracted saliency historical information matrix to enhance the token feature representation at time t. The above process can be represented as follows:

[0017]

[0018]

[0019]

[0020] Step Four It is passed to a multilayer perceptron A pseudo-packet level decision-making module was built to infer the probability of pseudo-packet labels. The above process is expressed as follows:

[0021]

[0022] Step 5: Design a time comparison module using linear and normalized layers to explore all historical tokens. To obtain stable and effective information between the model and its features, and to enhance the generalization and robustness of the model's feature representation, the specific steps are as follows:

[0023] For the pseudo-packet token at time t From the positive window W p Selected positive samples It tends to be more similar, while from the positive window W p Externally selected negative samples They are not very similar. Information-rich representations are extracted by minimizing the InfoNCE loss, a process expressed as:

[0024]

[0025] Where S is a similarity measure expressed as a feature, and τ is a temperature coefficient. They represent and The class probability;

[0026] Step Six: Introduce an adaptive feature enhancement module. Utilize pseudo-packet label prediction as a priori guide for the class-related model to capture the most informative token representations, calculate adaptive weights, and then... By selecting and aggregating disease-dependent information, more discriminative enhanced features can be generated. The specific steps are as follows:

[0027] Adaptive enhancement feature module with pseudo-packet token and the corresponding pseudo-label probability As input, capture with the highest probability The most crucial token use and The ratio is used as the weight. use polymerization and Formation of enhanced features The process is described as follows:

[0028]

[0029]

[0030] Step 7: Based on the token output by the adaptive enhancement feature module, use a class label. Combines the multi-head attention mechanism in the Transformer module to fuse sampled pseudo-packet tokens. and enhanced features Entering a global-level package feature representation The adaptive enhancement feature module and the Transformer module constitute the self-stimulating fusion feature module. The specific steps are as follows:

[0031] The Transformer module is equipped with an initialization class marker. Further integration of fake package tokens and enhanced features To obtain a global representation at the packet level, specifically, first concatenate... and For a feature map H i After a linear transformation, the data is transformed into Q, K, and V, and then segmented into multi-head representations. Attention scores are calculated for Q and K, multiplied by V to extract features, and then fused into Q. This process is described as follows:

[0032]

[0033]

[0034] Among them, W q W k W v W o It is a linear transformation matrix. The query, key, and value are split into a multi-head representation, where Head represents the number of multi-heads, and d k The dimension representing each head;

[0035] Step 8: Global-level package feature representation output by the self-motivation fusion feature module. It is passed into a multilayer perceptron The built package decision module infers package X i Tag probability The process is described as follows:

[0036]

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] This invention promotes better and more reliable decision-making by exploring the potential relationships between instances and the reciprocal relationship between feature representation and label prediction. It captures and aggregates tumor feature representations by introducing label-related category priors provided by pseudo-packet prediction, achieving high-accuracy pathological image label prediction. Conversely, the network is optimized based on the prediction results to further enhance feature representation, thereby improving pseudo-packet label prediction, forming a self-motivated learning mechanism. This invention introduces a multi-level feature fusion strategy to explore knowledge of current instances and global historical instances, while constructing a temporal comparison module to improve the robustness of feature representation and alleviate representation bias and overfitting problems. Furthermore, the self-motivated feature fusion module utilizes the mutual refinement mechanism between pseudo-packet prediction and feature representation to enhance the accuracy and reliability of pathological image classification. The SMILE framework as a whole forms a multi-level dynamic reciprocal learning end-to-end structure, including convolutional neural network-based feature extraction, multi-layer attention-based feature fusion, a temporal comparison module, self-motivated feature fusion, and pseudo-packet and package label prediction modules constructed from multi-layer perceptrons. In tests on common pathological image datasets, SMILE outperforms current mainstream methods such as multi-instance learning based on package and pseudo-packet levels. The experimental results fully demonstrate the high performance of this method in pathological image classification tasks and its application potential in real-world scenarios. Attached Figure Description

[0039] Figure 1 Examples of overfitting issues in common MIL methods for WSI classification;

[0040] Figure 2A flowchart of a pathological image classification method based on self-motivated multi-instance learning;

[0041] Figure 3 A diagram illustrating the structure of the SEMIL network for self-motivation through multiple examples;

[0042] Figure 4 Multi-level attention selection fusion module (MASF);

[0043] Figure 5 This is an attention-based feature fusion module (AFF).

[0044] Figure 6 For Time Comparison Module (TCM);

[0045] Figure 7 This is for Adaptive Enhancement Feature Module (AAF). Detailed Implementation

[0046] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0047] This invention provides a pathological image classification method based on self-motivated multiple instance learning, wherein the definition of multiple instance learning is as follows:

[0048] Given a whole slice dataset of pathological images It contains N WSIX i and its corresponding label Y i In the WSI classification task based on multi-instance learning, X i Considered as the i-th "packet", containing B i Sampling instances Where H and W represent the height and width of an instance, b∈[1,B] i ]. In addition to the "bag" tag Y i All instances outside of {0,1,…,C} Corresponding label y i,b Unknown, where C represents the number of disease types in dataset X. Taking binary classification as an example, multi-instance learning is defined as:

[0049]

[0050] This process can be interpreted as follows: if an instance is labeled as 1 (positive), then the final prediction for WSI is "positive". Otherwise, it is labeled as 0 (negative). In this invention, a pre-trained deep neural network is used as the encoder to... i,b Embedded into a D-dimensional feature vector Then, feature representation and decision-making strategies are designed to improve feature v. i,b And the model's representation learning ability, to promote robust and generalized feature representation for bag label prediction. The accuracy and reliability of [the system / mechanism].

[0051] like Figure 2 As shown, the method is implemented based on a self-motivated multiple instance learning (SEMIL) network, which is a multi-level pathological image classification network that promotes mutual learning. Figure 3 As shown, the network framework includes an instance sampling strategy, a Multi-layer Attention Selection Fusion (MASF) module for enhancing feature representation, a Self-Motivated Fusion Feature (SMFF) module, a Pseudo-bag-level Decision-making (PBDM) module, and a Bag-level Decision-making (BDM) module for pathological image labeling. The core objective of this invention, SMILE, is to enhance better and more reliable decision-making for pathological image labels by exploring the coupling relationships between instances and leveraging the association learning between feature representation and pseudo-bag label prediction.

[0052] Specifically, the SMILE network accepts a pathological WSI image X. i As input, the non-background organized regions of WSI are first cropped without overlap into a large number of image patches, which are called instances. Furthermore, a deep neural network pre-trained on ImageNet or a large number of pathological images is used as the encoder to embed instances into feature vectors. B here i This represents the number of sampled instances. Taking the operation at time t as an example, the model samples... The set of feature vectors corresponding to the instances Here, M represents the number of instance feature vectors contained in the sampling pseudo-packet, and T represents the input image X. i The number of times the dynamic interactive sampling pseudo-packet of size M is executed. Subsequently, the multi-layer attention selection fusion module MASF is used... And a randomly initialized token. As input, and by introducing a Transformer Module (TRM) to fuse pseudo-package instance feature vectors. Enter Subsequently, Employ an attention-based feature fusion (AFF) module to achieve cross-packet aggregation of the current token. and historical tokens Information entering between thereby improving The feature representation containing the most information prompts the pseudo-packet label probability output by the Pseudo-Bag Decision-Making (PBDM) module. Approaching the label Y i In addition, we introduced a Time Contrastive Module (TCM) that compares all historical tokens. To enhance the generalization and robustness of the model's feature representation, Self-Incentivized Fusion Feature (SMFF) subsequently introduces an Adaptive Augmentation Feature (AAF) strategy. The goal is to leverage pseudo-packet label predictions as class-related priors to guide the model in capturing the most informative token representations, and to compute adaptive weights from... By selecting and aggregating disease-dependent information, more discriminative enhanced features can be generated. Finally, use a class token (CLS). Combine the multi-head attention mechanism in the Transformer module to fuse sampled pseudo-packet tokens. and enhanced features Entering a global-level package feature representation at last, It is passed into the Packet Decision-Making (BDM) module to infer packet X. i Tag probability The following sections will provide a detailed introduction to the key components of the SMILE framework.

[0053] I. Instance Sampling Strategy:

[0054] To explore the impact of the intrinsic relationship between instance sampling and bag labeling on pathological image classification performance, the maximum average value (Pij) of the multi-head attention score in the first Transformer module was extracted. AS The probability score of the pseudo-packet tag (P) output by the pseudo-packet decision module PBDM and the pseudo-packet tag token. PBCSFurthermore, a decision-driven similarity sampling (DMSS) strategy was designed. DMSS This is used to generate pseudo packets at the same time for instances with abundant sampling information and similar packet labels.

[0055] The specific steps are as follows: Taking the operation at time t as an example, DMSS selects the current sampled pseudo-packet. China P AS The pre-P value above 0.5 PBCS ×M instances, where M represents the number of pseudo packets generated by sampling. It contains M instance feature vectors. Then, the selected instances are measured against the package X. i The cosine similarity of features among the remaining unsampled instances is used to select the most similar P. PBCS ×M instances enter the next pseudo-packet Meanwhile, the remaining ((1-P) PBCS )×M) instances are sampled using a random sampling (RAND) strategy. RAND The DMSS strategy encourages SMILE to focus on instances associated with packet tags in order to generate pseudo-packets that are closer to the packet tags. At time t, the above process can be expressed by the formula:

[0056]

[0057] In this invention, because the positive WSIs in the Camelyon16 dataset contain a small percentage (on average less than 10%) of tumor tissue regions, it is difficult to generate pseudo-packets consistent with the positive packet labels using the DMSS strategy. In the TCGA Lung dataset, positive WSIs contain over 80% tumor tissue regions, and the DMSS strategy can sample more tumor instances into a single pseudo-packet. Therefore, for the TCGALung dataset, this invention utilizes the DMSS strategy to encourage the model to select more instances with consistent packet labels and rich information into the same pseudo-packet. For the Camelyon16 dataset, a random sampling strategy is used, combined with a multi-level fusion feature strategy to aggregate tumor information to improve the consistency between pseudo-packet feature expression and packet labels.

[0058] II. Multi-layer Attention Selection Fusion (MASF) module:

[0059] To improve the accuracy of prediction, discriminative feature representation, such as Figure 4As shown, the Multi-Layer Attention Selection Fusion (MASF) module proposed in this invention includes a Transformer-based module (TRM) and an Attention-based Feature Fusion (AFF) module. MASF has the following characteristics:

[0060] (1) Explore the local and global coupling relationships between historical tokens and current pseudo-packet instances in the current WSI to achieve information-rich pseudo-packet tokens. express;

[0061] (2) Study the feature semantic distance between multiple pseudo-packet tokens to enhance token functionality. This indicates access to a richer token representation. This helps to mitigate biases such as model overfitting, pseudo-packet representation, and inconsistent packet labels.

[0062] The specific steps are as follows: First, design a position vector. Come and learn about fake packages Spatial information within. Then, a token vector is randomly initialized. Extracting with TRM module Local context information of instance features is entered into the Token In the above process, it is described as follows:

[0063]

[0064] In order to improve The AFF module utilizes consistency with package tags. To selectively merge historical tokens Global semantic information to obtain rich token representations like Figure 5 As shown.

[0065] The process is described as follows: the pseudo-packet token at time t is mapped to a query (Q), and the tokens before time t are mapped to keys (K) and values ​​(V). Next, the matrix similarity between Q and K is calculated and transformed into attention weights through Softmax. Based on weight Valid information is extracted from historical knowledge V. Subsequently, Q is added to the extracted saliency historical information matrix to enhance the token feature representation at time t. This process can fully integrate discriminative information from historical data that is beneficial for false label prediction, thereby improving the accuracy of false label prediction. The above process is represented as follows:

[0066]

[0067] III. Pseudo-Bag Decision-Making (PBDM)

[0068] The data is passed to a Multi-Layer Perceptron (MLP) consisting of linear layers and a Softmax activation function as a Pseudo-Bag Decision-Making (PBDM) module to infer the probability of pseudo-bag labels. The above MASF process is expressed as follows:

[0069]

[0070] IV. Time Contrastive Module (TCM)

[0071] To achieve more stable feature representation, this invention utilizes a linear layer and a normalization layer to design a TCM for full exploration. Generalization and robustness information, such as Figure 6 As shown.

[0072] The specific steps are as follows: For example, from the positive window W p Selected positive samples Tend to be more similar, while from W p Externally selected negative samples They are not very similar. This encourages MASF to extract information-rich representations by minimizing the InfoNCE loss. The process is expressed as:

[0073]

[0074] Here, S is a similarity measure expressed as a feature. The temperature coefficient τ is set to 0.1 to encourage the model to pay attention to feature changes. They represent and The class probability.

[0075] V. Adaptive Augmentation Feature (AAF) Module

[0076] Accurate predictions and high-quality tokens can provide prior information related to the labels, thereby aggregating the features with the most information and achieving better performance in pathological image label prediction. Therefore, this invention proposes to fully utilize information from... Disease-related prior knowledge is used to aggregate and optimize package label prediction representations that are beneficial for decision-making. For example... Figure 7 As shown, this invention develops an Adaptive Augmentation Feature (AAF) module to obtain the most relevant features of the tag, while incorporating effective information from other tokens.

[0077] The specific steps are as follows: The AAF module uses a pseudo-packet token. and the corresponding pseudo-label probability As input, capture with the highest probability The most crucial token Because distinctive expressions may be scattered across tokens, using and The ratio is used as the weight. use Aggregation will and form This approach can extract more complementary and target-related knowledge while protecting... Free from random interference.

[0078]

[0079] VI. Self-Motivated Fusion Feature (SMFF) Module

[0080] Based on the enhanced token output by the AAF module, this invention proposes a Self-Motivated Fusion Feature (SMFR) module, which consists of a Transformer Module (TRM) and an AAF module.

[0081] The specific steps are as follows: Use the Transformer module to equip an initialization class marker. Further integration of fake package tokens and enhanced features To obtain a global representation at the packet level, specifically, the first step is to concatenate... and For a feature map H iThe data is linearly transformed into query Q, key K, and value V, and then segmented into a multi-head representation. Attention scores are calculated for Q and K, multiplied by V to extract features, and then fused into Q. This process... By fully selecting and aggregating discriminative and abundant information most relevant to the package label, the risk of overfitting due to excessive reliance on optimal feature representation is mitigated, thus promoting robust and generalizable decision-making. This process can be described as follows:

[0082]

[0083] Among them W q W k W v W o It is a linear transformation matrix. The query, key, and value are split into a multi-head representation, where Head represents the number of multi-heads, and d k This represents the dimension of each head.

[0084] VII. Bag Decision-Making (BDM) Module

[0085] Finally, class tags with package-level global representation. An MLP constructed from linear layers and a softmax activation function is fed into the packet decision-making (BDM) module to infer and predict packet label probabilities.

[0086]

[0087] VIII. Loss Function

[0088] To achieve mutual promotion and learning between feature representation and prediction, and to obtain high-quality feature representation and better model decision-making capabilities, this invention explores bag probability on SMILE. Fake packet probability The composite constraint loss function for time consistency with TCM includes Bag Cross-Entropy Loss (BCE). Pseudo-Bag Cross-Entropy Loss (PBCE) and TCM loss These losses can be expressed as:

[0089]

[0090] Among them, L SMILE λ represents the final loss function used to optimize the SMILE method proposed in this invention. PBCEand λ TCM These are balancing parameters that are adjusted from initial values ​​to final values ​​(0 to 0.1, 0.5 to 0) by integrating the period and cosine curves to guide the model from exploring coarse information to more refined feature representations. and Y represents the predicted packet label probability and the pseudo-packet label probability. i This indicates the actual label on the package. Represents the input image X i Includes B i The number of times the dynamic interactive sampling of a pseudo-packet of size M is executed per instance.

[0091] Experimental results

[0092] In this invention, experiments were conducted to perform extensive comparisons and analyses on the commonly used CAMELYON16 and TCGA lung cancer datasets. The proposed SMILE method was compared with mainstream multi-instance learning methods, including bag-level methods (MeanPooling, MaxPooling, ABMIL, RNNMIL, DSMIL, CLAM, TransMIL, IAT, and MHIM) and pseudo-bag or pseudo-label methods (DTFD, DGMIL, and MuRCL). The area under the curve (AUC), accuracy, and F1 score were used for evaluation, with a threshold of 0.5. SMILE was trained for 100 iterations using the AdaMax optimizer with an initial learning rate of 0.0004. As shown in Table 1, the proposed method improves accuracy by 7.1% on the CAMELYON16 dataset compared to the state-of-the-art MHIM method. On the TCGA Lung dataset, it improves accuracy by 3.2% compared to IAT, validating its high accuracy and robustness in pathological image classification.

[0093] Table 1 Comparison of Model Training Results

[0094]

[0095] Figure 1 Examples of overfitting issues exist in common MIL methods for WSI classification. Packet-level methods (ABMIL and TransMIL) and pseudo-packet-level methods (DTFD) suffer from overfitting and pseudo-packet representation and packet label bias on the training and test sets of CAMELYON16. Bias is measured by the difference between packet-level and packet-level losses in the training set. The SMILE method proposed in this invention alleviates bias while demonstrating significant generalization ability.

[0096] This invention innovatively explores the latent relationships between instances and the mutual refinement between prediction and feature representation to achieve better labeling decisions for pathological images. To obtain high-quality representations and accurate predictions, this invention develops a Multi-layer Attention Selection Fusion (MASF) scheme to aggregate knowledge from current and past instances, while introducing a Time Contrastive Module (TCM) to eliminate inconsistencies between sub-packet representations and labels, mitigating overfitting. For robust packet labeling decisions, this invention utilizes pseudo-packet prediction to design a Self-Motivated Fusion Feature (SMFF) strategy to capture and aggregate the most informative and saliency feature representations. Simultaneously, it introduces correlation optimization between feature representations and label predictions to mitigate error accumulation. Extensive experiments on multiple benchmark datasets demonstrate that the proposed SMILE method exhibits impressive classification performance and generalization ability.

Claims

1. A pathological image classification method based on self-motivated multi-instance learning, characterized in that... The method includes the following steps: Step 1: Given a whole slice dataset of pathological images It contains N WSI X i and its corresponding label Y i In the WSI classification task based on multi-instance learning, X i Considered as the i-th "packet", containing B i Sampling instances Where H and W represent the height and width of an instance, b∈[1,B] i ], except for the "bag" tag Y i All instances outside of {0,1,…,C} Corresponding tags Unknown, where C represents the number of disease types in dataset X; Step 2: Using pathological WSI images X-rays i As input to the instance sampling strategy, the non-background organized regions of WSI are cropped without overlap, resulting in a large number of image patches, referred to as instances. Furthermore, a pre-trained deep neural network is used as the encoder to embed instances into feature vectors. At time t, sampling The set of feature vectors corresponding to the instances M represents the number of instance feature vectors contained in the sampled pseudo-packet, and T represents the input image X. i The number of times a pseudo-packet of size M is executed in a dynamic interactive sampling method; Step 3: Introduce a multi-layer attention selection and fusion module, which includes a Transformer-based module and an attention-based feature fusion module, to... and a randomly initialized token As input, the pseudo-packet instance feature vector is fused using the Transformer module. Enter An attention-based feature fusion module enables cross-aggregation of the current token across pseudo-packets. and historical tokens Information between them is represented by a token. Step Four It is passed to a multilayer perceptron A pseudo-packet level decision-making module was built to infer the probability of pseudo-packet labels. Step 5: Design a time comparison module using linear and normalized layers to explore all historical data. Stable and effective information between them to enhance the generalization and robustness of the model's feature representation; Step Six: Introduce an adaptive feature enhancement module. Utilize pseudo-packet label prediction as a priori guide for the class-related model to capture the most informative token representations, calculate adaptive weights, and then... By selecting and aggregating disease-dependent information, more discriminative enhanced features can be generated. Step 7: Based on the token output by the adaptive enhancement feature module, use a class label. Combined with the multi-head attention mechanism in the Transformer module, the sampled pseudo-packet is fused. and enhanced features Entering a global-level package feature representation The adaptive enhancement feature module and the Transformer module constitute the self-motivated fusion feature module; Step 8: Global-level package feature representation output by the self-motivation fusion feature module It is passed into a multilayer perceptron The built package decision module infers package X i Tag probability 2. The pathological image classification method based on self-motivated multi-instance learning according to claim 1, characterized in that... The specific steps of the instance sampling strategy in step two are as follows: at time t, decision-driven similarity sampling selects the current sampled pseudo-packet. In the example attention score P AS The pre-P value above 0.5 PBCS ×M instances, P PBCS A token label probability score is assigned to the pseudo-package, and then the selected instance is measured against package X. i The cosine similarity of features among the remaining unsampled instances is used to select the most similar P. PBCS ×M instances enter the next pseudo-packet Meanwhile, the remaining (1-P) PBCS )×M instances using a random sampling strategy S RAND The above process can be expressed by the following formula:

3. The pathological image classification method based on self-motivated multi-instance learning according to claim 1, characterized in that... The specific steps of step three are as follows: Step 31: Design a Token Vector Come and learn about fake packages Spatial information in; Step 3.2: Randomly initialize a token vector. Extracting with the Transformer module Local context information of instance features is entered into the Token In the above process, it is described as follows: Step 33, in order to improve Consistency with bag labels, the attention-based feature fusion module utilizes To selectively integrate history Global semantic information to obtain rich token representations The process is described as follows: the pseudo-packet token at time t is mapped to query Q, and the tokens before time t are mapped to key K and value V. Next, the matrix similarity between Q and K is calculated and transformed into attention weights through Softmax. Based on attention weight Effective information is extracted from historical knowledge V. Subsequently, Q is added to the extracted saliency historical information matrix to enhance the token feature representation at time t. The above process can be represented as follows:

4. The pathological image classification method based on self-motivated multi-instance learning according to claim 1, characterized in that... The process of step four is expressed as follows:

5. The pathological image classification method based on self-motivated multi-instance learning according to claim 1, characterized in that... The specific steps of step five are as follows: For the pseudo-packet token at time t From the positive window W p Selected positive samples Tend to be more similar, while from the positive window W p Externally selected negative samples They are not very similar. Information-rich representations are extracted by minimizing the InfoNCE loss, a process expressed as: Where S is a similarity measure expressed as a feature, and τ is a temperature coefficient. They represent and The class probability.

6. The pathological image classification method based on self-motivated multi-instance learning according to claim 1, characterized in that... The specific steps of step six are as follows: Adaptive feature enhancement module with pseudo-packet and the corresponding pseudo-label probability As input, capture with the highest probability The most crucial token use and The ratio is used as the weight. use polymerization and Formation of enhanced features The process is described as follows:

7. The pathological image classification method based on self-motivated multi-instance learning according to claim 1, characterized in that... The specific steps of step seven are as follows: The Transformer module is equipped with an initialization class marker. Further integration of fake packages and enhanced features To obtain a global representation at the packet level, specifically, first concatenate... and For a feature map H i After a linear transformation, the data is transformed into a query Q, a key K, and a value V, and then segmented into a multi-head representation. Attention scores are calculated for Q and K, multiplied by V to extract features, and then fused into Q. This process is described as follows: Among them, W q W k W v W o It is a linear transformation matrix. The query, key, and value are split into a multi-head representation, where Head represents the number of multi-heads, and d k This represents the dimension of each head.

8. The pathological image classification method based on self-motivated multi-instance learning according to claim 1, characterized in that... The process of step eight is described as follows:

Images