Pathology classification prediction method based on diffusion heterogeneous difficult instance mining

Abstract

The invention relates to a pathology classification prediction method based on diffusion heterogeneous difficult instance mining. The method comprises the following steps: selecting a public pathology image WSI data set; traversing each packet in the D by using a pre-training feature learning model to obtain all instance feature sets to form a training set D '; constructing a difficult instance mining model M, wherein the M comprises a prototype-based difficult instance mining module MPHIM and a dual-view adversarial learning module MDVAL; inputting MPHIM to obtain a difficult instance set, taking the difficult instance set as the input of MDVAL, and outputting to obtain a pathological prediction result of the WSI image; and constructing a loss function of the model M, and training the model M to finally obtain a trained M '. By using the method of the invention, partial case content in the WSI image can be rapidly and accurately identified, and high-value reference content is provided for diagnosis personnel.

Description

Technical Field

[0001] This invention relates to the field of pathological image analysis, and in particular to a pathological classification prediction method based on the mining of difficult instances of diffusion heterogeneity. Background Technology

[0002] The rapid development of computational pathology has made digital pathological image (WSI) analysis a key technology for assisting diagnosis and improving the accuracy of disease detection, such as cancer. These pathological images have astonishing resolution, some even containing billions of pixels. Accurately labeling disease-related regions in such massive amounts of image data is both time-consuming and labor-intensive, and extremely challenging in practice. Multiple instance learning (MIL) technology has opened up a new solution to this problem. This technology transforms pathological image analysis into a weakly supervised learning task, treating each pathological image as a "bag," and the smaller patches within the image as "instances." Each "bag" has only one label indicating whether it contains a lesion instance.

[0003] In recent years, Multi-Instance Learning (MIL) has attracted much attention in the field of pathological image analysis. Existing MIL research can be summarized into two main categories: instance label-level algorithms and instance feature-level algorithms. The former predicts individual instance labels and aggregates them to obtain bag labels; the latter aggregates all instance features into high-level bag embeddings for bag classification. However, the imbalance in the ratio of positive to negative instances within a bag makes it difficult to obtain ideal robust results by directly aggregating bag features from all instance features. Hard sample mining strategies provide an effective solution to this challenge. Hard sample mining aims to solve the problems of sample imbalance and uneven distribution of easy and difficult samples in training data. Its core idea is to prioritize the most challenging samples for the current model during training. Some studies have pioneered the introduction of hard instance mining into the MIL framework, directly locating difficult negative instances in false positive bags through attention scores, significantly improving the classification performance of pathological images. Other studies have used a teacher-student model to select key instances that are beneficial to classification and indirectly mine difficult instances by masking these instances during training, effectively enhancing the model's classification ability and generalization performance. These methods rely on uniform criteria learned from the entire dataset, which may fail to capture subtle feature changes in a single pathological image, limiting their effectiveness in identifying image-specific hard instances.

[0004] In a single pathological image, the lesion area typically occupies only a small proportion. Therefore, even within positive "packets," the vast majority of instances are negative. This causes the MIL model to suffer from severe sample imbalance during the learning process. To address this challenge, researchers have proposed various solutions, such as using attention scores or image patch probabilities to filter salient instances, and then aggregating the features of these instances for pathological image classification. However, research shows that these salient instances are often easy to classify, making them less than ideal for training robust and discriminative pathological image classification models.

[0005] Recent studies have introduced hard case mining techniques, employing uniform difficulty assessment criteria to select hard cases. However, these methods ignore the inherent visual diversity of pathological images. In practical applications, pathological images exhibit rich visual variations due to differences in staining agent type and quality, scanning equipment, and acquisition parameters; the diversity of tissue structure and cell morphology further exacerbates this difference. This significant visual heterogeneity poses a challenge to existing uniform hard case mining strategies, making it difficult to comprehensively uncover potential hard cases. By ignoring visual heterogeneity, these uniform mining strategies often lead to inaccurate case selection, resulting in misclassification of the final pathology report. Summary of the Invention

[0006] In view of the above-mentioned problems in the existing technology, the technical problem to be solved by the present invention is: how to more accurately extract difficult instances in pathological images in order to improve the prediction accuracy of pathological classification.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A pathological classification prediction method based on mining difficult instances of diffusion heterogeneity includes the following steps:

[0009] S100: Select a publicly available pathological image dataset. D contains several pathological image WSIs, where each WSI is considered a bag, and each segmented image patch within a bag is considered an instance within that bag. The definition of D is as follows:

[0010] D = {(B1,Y1),…,(B i ,Y i ),…,(B n ,Y n )}

[0011] Among them, B i Let B represent the i-th packet. i It contains m instances of {d1, ...,d2}. j ,…,d m} and j∈[1,m], Y i Indicates assignment to B i The label, B i Each instance within the instance has a corresponding real label y;

[0012] S200: Using a pre-trained feature learning model, select the i-th bag for instance feature extraction to obtain the instance feature set. The calculation expression is as follows:

[0013]

[0014] in, b represents a collection of instances. j This represents the j-th instance d. j Corresponding instance features;

[0015] Traverse each bag in D to obtain the instance feature set corresponding to each bag. All instance feature sets constitute the training set D'.

[0016] S300: Construct a hard instance mining model M, M including a prototype-based hard instance mining module M PHIM and dual-view adversarial learning module M DVAL The M PHIM Includes a typical negative prototype generation module and a hard instance mining module HIM, the M PHIM Used for mining difficult instances in WSI; the M DVAL It includes two multi-instance learning models (MIL) for outputting classification results; the typical negative prototype generation module includes a diffusion model (DiT), a trainable feature adapter (Adapter), and a feature embedding layer.

[0017] S400: will Enter M PHIM Obtain the set of difficult instances Then As M DVAL The input is the pathological prediction result of the WSI image;

[0018] S500: The loss function for constructing model M Use D' as input to M and utilize M is trained, and the model parameters of M are updated using the gradient descent backpropagation principle. Training stops when the value no longer decreases or when the maximum number of iterations is reached. At this point, the trained M' is obtained.

[0019] S600: Select the WSI image X to be predicted, input X into M', and obtain the prediction result of the pathological classification of X.

[0020] Preferably, the set of difficult instances is obtained in S400. The steps are as follows:

[0021] S410: Will Perform pooling operation and incorporate initial noise. The generated noise after adding instance feature information The calculation formula is as follows:

[0022]

[0023] in, This indicates a pooling operation.

[0024] S411: Will As input to the DiT diffusion model, the canonical negative prototype q is calculated, and the calculation expression is as follows:

[0025]

[0026] Where t represents the t-th time step, This indicates the calculation process of DiT;

[0027] S412: Input q into the Adapter and fine-tune it to obtain q′. The calculation formula is as follows:

[0028]

[0029] in, This represents a trainable feature adapter (Adapter).

[0030] S413: Combining q and q′ yields the diffused and denoised canonical negative prototype q″, expressed as follows:

[0031] q″=q+q′;

[0032] S420: Calculate instance feature b j The similarity score s with q″ j The calculation formula is as follows:

[0033]

[0034] Where ||·|| represents L 2 Norm;

[0035] S421: Repeat S420, calculate... The similarity scores of all instance features and q″ are represented by a set.

[0036] S422: For S i Sort the instances in ascending order and set a hyperparameter θ. Select the instances with the highest similarity scores (top θ percentages) as the set of difficult instances.

[0037] Preferably, the loss function is constructed in S500. The steps are as follows:

[0038] S510: Input q″ into the feature embedding layer and output the feature embedding vector ε. - (q″), change ε - (q″) is used as the input to the binary classifier to calculate the B... iPrediction results for normal instances The calculation expression is as follows:

[0039]

[0040] Where, ε - (·) represents the feature embedding layer. Indicates a binary classification header;

[0041] S511: Utilizing And a fine-tuning loss function is constructed using cross-entropy loss. The calculation expression is as follows:

[0042]

[0043] Among them, y nor Labels indicating normal portions of a pathological image;

[0044] S520: M DVAL The two multi-instance learning models MIL in the model are defined as a normal perspective MIL network and a disease perspective MIL network, respectively. As M DVAL Input, Bag-level embeddings are obtained by embedding and aggregation through the MIL network from a normal perspective. nor , Bag-level embeddings f are obtained by embedding and aggregation of the MIL network from a disease perspective. dis The calculation formula is as follows:

[0045]

[0046] Where, ε nor (·) and ε dis (·) represent the feature embedding layers of the normal perspective network and the disease perspective network, respectively. and These represent the feature aggregators corresponding to the normal perspective network and the disease perspective network, respectively.

[0047] S521: via f dis and Calculate the loss function of a disease-perspective network in binary classification. The calculation formula is as follows:

[0048]

[0049] Where Y represents the image instance type;

[0050] S522: Constructing a binary classification loss function The calculation expression is as follows:

[0051]

[0052] in, This indicates a binary category header, and y represents the package label;

[0053] S530: Integration and Obtain the total loss function of the model The calculation formula is as follows:

[0054]

[0055] Where γ represents a non-negative coefficient.

[0056] Preferably, the loss function of the disease perspective network in S521 The categories are as follows:

[0057] For diagnostic tasks, the loss function adapts to the true label y, and the binary classification loss function is expressed as:

[0058]

[0059] For classification tasks other than diagnostic tasks, the loss function uses the disease type label, and the binary classification loss function is expressed as follows:

[0060]

[0061] Among them, y dis This indicates the disease type label in the case image.

[0062] Compared with the prior art, the present invention has at least the following advantages:

[0063] This invention proposes an innovative hard instance mining strategy based on image generation—Heterogeneous Diffusion Hard Instance Mining (DH). 2 IM), and applied it to MIL pathological image classification. In DH 2 The IM strategy extracts visual context information from each pathological image, guiding a fine-tuned diffusion model to generate a negative instance for each image. This negative instance serves as the standard negative prototype for the pathological image, and instances highly similar to the prototype are considered easily classifiable. This approach achieves image-level hard instance mining. However, despite filtering out many typical instances, the problem of imbalanced instance distribution persists. To prevent the dilution of key features during instance aggregation, this invention designs a dual-view classification network, embedding and aggregating instances of each category from different perspectives of interest. Furthermore, this invention employs dual-view adversarial learning based on the outputs of these classification networks, prompting each network to focus more on instances of the target category from its corresponding perspective.

[0064] 1. This invention proposes a novel Multi-Instance Learning (MIL) framework based on difficult instance mining—DH 2 IM-MIL, used for WSI classification, DH 2 The main advantage of IM-MIL lies in its image-level hard instance mining criteria, which can more accurately identify information-rich instances; to date, this should be the first work to perform adaptive instance mining for each WSI.

[0065] 2. This invention introduces a novel generative hard instance mining method—diffusion heterogeneous hard instance mining (DH). 2 IM). DH 2 IM employs an image-level adaptive mechanism to dynamically filter easily classifiable instances. It generates image-specific canonical prototypes through a visual contextual conditional diffusion model. These prototypes serve as a reference to adaptively distinguish irrelevant instances from information-rich difficult instances in each WSI.

[0066] 3. This invention designs a lightweight and efficient dual-view adversarial learning strategy, which prompts each classification network to competitively aggregate and classify its target category instances, thereby mitigating the bias caused by the imbalance in instance distribution. Attached Figure Description

[0067] Figure 1 For the present invention DH 2 A schematic diagram of the IM model.

[0068] Figure 2 For DH 2 Performance of IM on the CAMELYON dataset under different mining ratios θ and scaling factors γ.

[0069] Figure 3 Visualization of image patches generated by the positive perspective (disease perspective) network and the negative perspective (normal perspective) network in a dual-view network; the middle curve delineates the tumor region, and the brighter the image patch, the higher the attention score.

[0070] Figure 4 t-SNE visualization for mining homogeneous (MHIM-MIL) and heterogeneous hard instances; the figure shows all instances of a pathological image (WSI), where circles represent positive instances, boxes represent hard instances mined during training, and the rest represent negative instances.

[0071] Figure 5 Visualization of image patches generated for AB-MIL (baseline) and HDHIM-MIL; curves delineate tumor regions, and the highlighted areas within the curves represent high-probability areas where the corresponding location is a tumor. Detailed Implementation

[0072] The present invention will now be described in further detail.

[0073] This invention designs a method for mining diffusion heterogeneous hard instances (DH). 2 This method cleverly utilizes visual context information to modulate the diffusion model, generating a unique canonical prototype for each image. By filtering out simple instances that are highly similar to the prototype, it achieves accurate mining of difficult instances that are adaptive to the image. At the same time, to alleviate the problem of class imbalance in the mined instances, this method innovatively introduces a dual-view adversarial learning strategy—constructing two specialized classification networks, each focusing on different perspectives, to collaboratively generate more discriminative bag embedding representations. The method of this invention adopts a modular design and can be flexibly integrated into various MIL frameworks.

[0074] See Figures 1-5 A pathological classification prediction method based on the mining of difficult instances of diffusion heterogeneity includes the following steps:

[0075] S100: Select a publicly available pathological image dataset. D contains several pathological image WSIs, where each WSI is considered a bag, and each segmented image patch within a bag is considered an instance within that bag. The definition of D is as follows:

[0076] D = {(B1,Y1),…,(B i ,Y i ),…,(B n ,Y n )}

[0077] Among them, B i Let B represent the i-th packet. i It contains m instances of {d1, ...,d2}. j ,…,d m} and j∈[1,m], Y i Indicates assignment to B i The label, B i Each instance within a package has a corresponding real label y; the label of a package depends on the set of instances within it: if all instances are negative, the package is labeled negative; otherwise, the package is labeled positive.

[0078] S200: Using a pre-trained feature learning model, select the i-th bag for instance feature extraction to obtain the instance feature set. The calculation expression is as follows:

[0079]

[0080] in, b represents a collection of instances. j This represents the j-th instance d. jCorresponding instance features; the pre-trained feature learning model It is an offline feature extractor used to extract high-dimensional feature vectors from image patches segmented from whole slice images (WSI). This model This is not a new model created by the present invention, but rather uses an existing deep learning model that has been pre-trained on a large-scale dataset as a feature extraction tool, and belongs to the prior art.

[0081] To illustrate this more clearly, the specific model examples used are clearly listed in the "1. Experimental Content. Implementation Details" section: ResNet50: This is a classic convolutional neural network (CNN) architecture, pre-trained on the ImageNet-1k dataset. ResNet50 is a widely used backbone network in the field of computer vision, and its structure and pre-training methods are existing technologies well-known to those skilled in the art; UNI model: This refers to a general-purpose basic model pre-trained on a large-scale histological image dataset. The UNI model represents the most advanced self-supervised learning model in the current field of pathological image analysis, and its pre-training methods and model architecture are also publicly available technologies.

[0082] Traverse each bag in D to obtain the instance feature set corresponding to each bag. All instance feature sets constitute the training set D'.

[0083] S300: Construct a hard instance mining model M, M including a prototype-based hard instance mining module M PHIM and dual-view adversarial learning module M DVAL The M PHIM Includes a typical negative prototype generation module and a hard instance mining module HIM, the M PHIM Used for mining difficult instances in WSI; the M DVAL It includes two multi-instance learning models (MIL) for outputting classification results; the typical negative prototype generation module includes a diffusion model (DiT), a trainable feature adapter (Adapter), and a feature embedding layer.

[0084] The Diffusion Model DiT and the Trainable Feature Adapter are both existing technologies. The Feature Embedding Layer is a linear layer in the neural network, which is also an existing technology. The Hard Instance Mining Module HIM and the Multi-Instance Learning Model MIL are also existing technologies. The two Multi-Instance Learning Models MIL are used as two classification networks.

[0085] S400: will Enter M PHIM Obtain the set of difficult instances Then As M DVALThe input is the pathological prediction result of the WSI image;

[0086] The set of difficult instances is obtained in S400. The steps are as follows:

[0087] S410: Will Perform pooling operation and incorporate initial noise. The generated noise after adding instance feature information The calculation formula is as follows:

[0088]

[0089] in, This indicates a pooling operation. This process treats average instance features as contextual information specific to the WSI image, incorporating this into the visual prior of the WSI image. The generation process of the negative prototype is a process that... The process begins with a gradual denoising diffusion process; at each step, the diffusion model predicts the pathological image data we need and removes noise to obtain image data that is closer to the original; the diffusion model is existing technology.

[0090] S411: Will As input to the DiT diffusion model, the canonical negative prototype q is calculated, and the calculation expression is as follows:

[0091]

[0092] Where t represents the t-th time step, This indicates the calculation process of DiT;

[0093] S412: Input q into the Adapter and fine-tune it to obtain q′. The calculation formula is as follows:

[0094]

[0095] in, This refers to a trainable feature adapter; in this invention A lightweight multilayer perceptron (MLP) is used, whose main function is to fine-tune and adapt the output features of the diffusion model backbone network that receives the data from the freeze-up to better fit the feature distribution of “canonical negative instances” in the target pathological image data domain.

[0096] After generating the image-specific canonical negative prototype q, it is necessary to ensure that q can accurately simulate the distribution characteristics of canonical negative instances. Considering the limitations of traditional pre-training methods in multi-instance learning scenarios, especially when dealing with the challenges of large-scale model parameters and lack of instance-level labels, this method adds a trainable feature adapter at the end of the frozen diffusion model to fine-tune q without updating the entire backbone network.

[0097] S413: Combining q and q′ yields the diffused and denoised canonical negative prototype q″, expressed as follows:

[0098] q″=q+q′;

[0099] S420: Calculate instance feature b j The similarity score s with q″ j The calculation formula is as follows:

[0100]

[0101] Where ||·|| represents L 2 Norm;

[0102] S421: Repeat S420, calculate... The similarity scores of all instance features and q″ are represented by a set.

[0103] S422: For S i Sort the instances in ascending order and set a hyperparameter θ. Select the instances with the highest similarity scores (top θ percentages) as the set of difficult instances.

[0104] By adopting this strategy, we not only effectively identify difficult instances, but also significantly reduce the number of instances that need to be processed, thus providing a more compact and efficient set of instance features for subsequent pathological image classification tasks.

[0105] S500: The loss function for constructing model M Use D' as input to M and utilize M is trained, and the model parameters of M are updated using the gradient descent backpropagation principle. Training stops when the value no longer decreases or when the maximum number of iterations is reached. At this point, the trained M' is obtained.

[0106] The loss function is constructed in S500. The steps are as follows:

[0107] S510: Input q″ into the feature embedding layer and output the feature embedding vector ε. - (q″), change ε -(q″) is used as the input to the binary classifier to calculate the B... i Prediction results for normal instances The calculation expression is as follows:

[0108]

[0109] Where, ε - (·) represents the feature embedding layer. This indicates a binary classifier head; the binary classifier is existing technology.

[0110] S511: Utilizing And a fine-tuning loss function is constructed using cross-entropy loss. The calculation expression is as follows:

[0111]

[0112] Among them, y nor Labels representing normal portions of a pathological image; the As the loss function of the adapter, it is used to optimize the parameters of the adapter;

[0113] S520: M DVAL The two multi-instance learning models MIL in the model are defined as a normal perspective MIL network and a disease perspective MIL network, respectively. As M DVAL Input, Bag-level embeddings are obtained by embedding and aggregation through the MIL network from a normal perspective. nor , Bag-level embeddings f are obtained by embedding and aggregation of the MIL network from a disease perspective. dis The calculation formula is as follows:

[0114]

[0115] Where, ε nor (·) and ε dis (·) represent the feature embedding layers of the normal perspective network and the disease perspective network, respectively. and These represent the feature aggregators corresponding to the normal perspective network and the disease perspective network, respectively.

[0116] S521: via f dis and Calculate the loss function of a disease-perspective network in binary classification. The calculation formula is as follows:

[0117]

[0118] Where Y represents the image instance type;

[0119] Due to ε nor (·) During the fine-tuning process, canonical negative knowledge is learned, and the normal-view network will focus on aggregating bag-level embeddings f that emphasize normal features. nor Disease-perspective networks, on the other hand, are encouraged to aggregate embeddings that highlight disease-related features. dis To guide these perspective-specific learning processes, this method introduces a binary classification head. To distinguish between disease patterns and normal patterns; the output f of the disease perspective network dis This binary classifier is used for processing, and the corresponding loss function is constructed according to the nature of the task.

[0120] The loss function of the disease perspective network in S521 The categories are as follows:

[0121] For diagnostic tasks, the loss function adapts to the true label y, and the binary classification loss function is expressed as:

[0122]

[0123] For classification tasks other than diagnostic tasks, the loss function uses the disease type label, and the binary classification loss function is expressed as follows:

[0124]

[0125] Among them, y dis The disease type label in the case image is a binary label, so no matter how many types of diseases there are, as long as it is a disease, it is "1", and conversely, the normal label is 0.

[0126] S522: Constructing a binary classification loss function The calculation expression is as follows:

[0127]

[0128] in, y represents the binary classification head, and y represents the bag label; this binary classification head plays a key role in guiding perspective-oriented learning. It forces the disease perspective network to capture disease-specific patterns, while the normal perspective network focuses on normal features through normalization of negative prototypes; this adversarial optimization drives the collaborative work of the two perspective networks, ensuring that each network learns unique perspective-oriented features.

[0129] S530: Integration and Obtain the total loss function of the model The calculation formula is as follows:

[0130]

[0131] Here, γ represents a non-negative coefficient used to balance the two loss terms.

[0132] S600: Select the WSI image X to be predicted, input X into M', and obtain the prediction result of the pathological classification of X.

[0133] Experimental content and results

[0134] 1. Experiment Content

[0135] Dataset

[0136] The model was evaluated using three medical WSI analysis tasks: cancer diagnosis, subtype classification, and survival prediction. For the cancer diagnosis task, the CAMELYON dataset, which is a combination of the CAMELYON-16 and CAMELYON-17 datasets, was used. For the subtype classification task, the TCGA-NSCLC, TCGA-BRCA, and TCGA-BRACS datasets were used. To evaluate the accuracy of survival prediction, the TCGA-LUSC and TCGA-BLCA datasets were used to assess the performance of the survival prediction task.

[0137] Experimental setup

[0138] For diagnosis and typing, the model performance was evaluated using accuracy, AUC, and F1-score during the experiments. For survival prediction, the C-index for all datasets is reported in this experiment. Five-fold cross-validation was used by default, consistent with the baseline method for performance comparison.

[0139] Implementation details

[0140] Following conventional knowledge in this field, this experiment uses ImageNet-1k pre-trained ResNet50 and the state-of-the-art base model UNI as offline feature extractors. UNI is based on a 1×10 6 WSIs' large internal histology dataset 1×10 8 Pre-trained on the patch.

[0141] 2. Experimental Results

[0142] Cancer diagnosis and classification

[0143] Table 1. Comparison of cancer diagnostic performance on the CAMELYON dataset and subtype classification performance on the TCGA-NSCLC dataset.

[0144]

[0145] Table 2 compares the cancer diagnostic performance of the TCGA-BRCA dataset and the subtyping performance of the BRACS-3 dataset.

[0146]

[0147]

[0148] Table 3 shows the performance (AUC metric) of subtype classification using UNI as an offline feature extractor.

[0149] Methods NSCLC BRCA BRACS-3 ABMIL 98.24±0.59 95.62±1.84 92.00±1.71 CLAM 98.27±0.93 96.11±1.89 92.00±2.14 DSMIL 98.18±0.53 96.11±1.57 91.99±1.50 TransMIL 98.37±1.01 96.04±1.62 90.25±1.50 DTFD-MIL 98.06±1.02 96.23±1.65 91.49±2.38 MHIM-MIL 98.40±0.45 96.11±1.77 92.85±3.43 IBMIL 98.37±0.36 95.66±1.65 92.11±2.25 WiKG 98.38±0.80 96.48±1.60 91.91±1.64 <![CDATA[R 2 T-MIL]]> 98.42±0.67 96.32±1.68 92.40±1.45 H-MIL 98.28±0.23 96.15±3.21 91.26±1.79 FR-MIL 98.32±0.66 96.24±1.82 91.96±1.96 <![CDATA[DH 2 IM(AB-MIL)]]> 98.57±0.62 96.73±1.53 92.56±1.47 <![CDATA[DH 2 IM(Trans-MIL)]]> 98.41±0.53 96.59±2.38 93.24±1.78

[0150] Table 1 compares the diagnostic and subtype classification performance of different MIL methods on the CAMELYON and TCGA-NSCLC datasets. The effectiveness of the framework is validated on two representative MIL models, both of which outperform existing MIL methods. Specifically, on the CAMELYON dataset, DH… 2 Compared to the second-best methods other than ours, IM (TransMIL) improves AUC by 1.35% and F1 score by 0.42%; on the TCGA-NSCLC dataset, DH... 2 The IM (AB-MIL) improved accuracy by 1.30%, AUC by 0.48%, and F1 score by 0.60%.

[0151] The subtype classification results of the TCGA-BRCA dataset in Table 2 also support similar observations. DH 2 IM(AB-MIL) outperforms suboptimal methods, achieving a 1.16% improvement in accuracy, a 0.93% improvement in AUC, and a 2.99% improvement in F1 score on this dataset. These significant performance improvements are attributed to the proposed instance mining strategy and the dual-view adversarial learning network, which significantly enhance model performance compared to the baseline: a 2.89% improvement in accuracy, a 2.34% improvement in AUC, and a 5.72% improvement in F1 score. On the BRACS-3 dataset, DH... 2 The IM framework continues to demonstrate its competitive advantage. DH 2 The IM(TransMIL) assay achieved a best AUC result of 89.46%, further confirming the proposed DH... 2 The effectiveness of the IM framework in whole-slice image classification.

[0152] The observations in Tables 1 and 2 also reveal the advantages of image-based hard instance mining strategies compared to traditional dataset-based hard instance mining strategies. MHIM-MIL is a dataset-based instance mining method. More specifically, our method improves the AUC of MHIM-MIL by 1.35%, 0.84%, 1.27%, and 1.10% on the CAMELYON, TCGA-NSCLC, TCGA-BRCA, and BRACS-3 datasets, respectively.

[0153] Table 3 reports the performance of full-slice image classification using UNI features. With the adoption of a more advanced UNI feature extractor, all models achieved significant performance improvements across various datasets, and the performance gap between different methods narrowed due to the improved quality of feature representations. Despite the reduced performance differences, DH... 2 The IM framework still maintains its competitive advantage. These results confirm that even when equipped with state-of-the-art feature extractors, DH... 2 The IM framework remains effective and robust, validating the universality of our instance mining strategy and adversarial learning method across different feature representation spaces.

[0154] Survival prediction

[0155] Table 4 shows the survival prediction results on the two main datasets.

[0156] Methods BLCA LUSC AB-MIL 57.97±5.43 59.11±2.47 CLAM 58.73±3.24 60.71±2.91 DSMIL 58.29±3.53 60.21±3.90 TransMIL 57.00±3.60 57.46±4.69 DTFD-MIL 59.85±4.18 59.03±2.54 MHIM-MIL 59.70±4.06 60.91±4.26 IBMIL 59.09±5.42 58.50±2.13 WiKG 58.04±3.30 58.10±4.95 <![CDATA[R 2 T-MIL]]> 60.98±2.73 60.64±3.83 H-MIL 55.35±2.92 60.14±1.73 FR-MIL 57.66±2.04 59.57±4.74 <![CDATA[DH 2 IM(AB-MIL)]]> 59.52±5.28 61.20±3.44 <![CDATA[DH 2 IM(TransMIL)]]> 62.28±2.40 60.45±4.25 AB-MIL 59.42±5.96 59.25±4.74 CLAM 59.68±4.72 59.69±2.88 DSMIL 60.83±4.08 60.58±4.25 TransMIL 59.14±5.34 58.78±3.54 DTFD-MIL 59.64±6.28 60.27±5.39 MHIM-MIL 60.78±3.71 61.43±3.77 IBMIL 59.33±4.02 61.12±5.10 WiKG 59.53±3.12 60.32±3.25 <![CDATA[R 2 T-MIL]]> 59.42±3.15 61.06±3.85 H-MIL 60.86±2.61 62.21±3.13 FR-MIL 57.66±2.04 59.57±4.74 <![CDATA[DH 2 IM(AB-MIL)]]> 59.97±4.08 60.31±3.53 <![CDATA[DH 2 IM(TransMIL)]]> 61.55±3.18 63.67±2.80

[0157] Table 4 presents the experimental results for the two survival prediction datasets. DH 2 The IM framework achieved excellent performance, reaching a C-index of 62.28% on BLCA and 61.20% on LUSC. These results significantly outperform the comparative methods, improving upon the suboptimal methods by 1.30% and 0.29% on BLCA and LUSC, respectively. Furthermore, when combined with high-quality features extracted from the base model UNI, DH… 2 The IM framework continues to demonstrate strong performance, achieving improvements of 0.69% and 1.46% over suboptimal methods on BLCA and LUSC, respectively.

[0158] 3. Ablation experiment

[0159] Hyperparameter analysis

[0160] like Figure 2As shown, this invention investigates the impact of mining ratio and scaling factor on the CAMELYON dataset. The mining ratio θ determines the proportion of difficult instances selected for training. A too-high ratio reduces the difficulty of mining instances, while a too-low ratio leads to the loss of key information. This experiment shows that θ = 0.5 achieves the optimal balance between instance difficulty and information preservation. The scaling factor γ modulates the balance between diffusion model fine-tuning and two-view adversarial learning. DH based on AB-MIL... 2 The IM model remained stable across different γ values, while the TransMIL-based variant exhibited greater sensitivity, indicating that different backbone architectures possess different optimization characteristics. Based on these observations, γ = 1.0 was set to maintain stable performance across different models.

[0161] Importance analysis of different components

[0162] Table 5 shows the performance impact of different components on Camlyon.

[0163]

[0164] Table 5 illustrates the specific impact of different components in the proposed framework on model performance on the CAMELYON dataset. This validation uses two representative attention-based MIL methods—AB-MIL and TransMIL—as baselines. Based on these models, the heterogeneous hard instance mining (DH) method proposed in this paper is first incorporated. 2 The proposed strategy incorporates Diffusion-based Canonical Negative Prototype Generation (DCNPG) and Prototype-based Hard Instance Mining (PHIM). This strategy improved the AUC of AB-MIL by 0.72% and TransMIL by 1.55% on Camelyon, highlighting the importance of hard instance mining during training and helping mainstream MIL models establish more effective classification boundaries. Furthermore, integrating a Dual-View Adversarial Learning (DVAL) network further improved the AUC of AB-MIL by 0.72% and TransMIL by 0.53%. These improvements are attributed to DVAL's ability to help the model aggregate more discriminative bag-level embedding features.

[0165] Mining Strategy Analysis

[0166] Table 6 Evaluation of Different Mining Strategies

[0167]

[0168]

[0169] To evaluate the effectiveness of the hard instance mining strategy proposed in this study, this experiment compared it with MHIM-MIL and several traditional mining strategies using a mining ratio of 0.5, as shown in Table 6. The max pooling and average pooling prototype mining strategies involve pooling operations across all instances within each WSI bag to obtain prototypes for subsequent instance mining. It is noteworthy that most traditional mining methods negatively impact the performance of the base model, likely because simple strategies have limitations in accurately identifying hard instances beneficial for model training. Table 5 further illustrates the DH... 2 Comparison of the performance of IM and MHIM-MIL on different baselines. Specifically, DH 2 IM outperforms MHIM-MIL and achieves a 1.40% AUC improvement over TransMIL, highlighting the accuracy advantage of the proposed mining strategy in identifying challenging instances.

[0170] Analysis of different criteria

[0171] Table 7 Comparison Strategies for Different Branches

[0172]

[0173] Table 7 presents a comparison of using uniform and heterogeneous criteria in hard instance mining strategies. The table shows the H-DH... 2 IM refers to the DH proposed in this invention. 2 The IM method. This experiment modifies the original DH method by eliminating image-level context to improve the prototype generation for hard instance mining. 2 The prototype generation process in the IM method. Conversely, by fine-tuning the diffusion model at the dataset level through an adapter, image-specific conditional guidance is bypassed during prototype generation, resulting in what is denoted as U-DH in the table. 2 The unified criterion for IM. MHIM-MIL uses the unified criterion to uncover difficult instances in the design, and is therefore represented in the table as U-MHIM-MIL. The performance difference between the unified criterion and the heterogeneous criterion is significant in both cases. For example, H-DH based on TransMIL. 2 The AUC ratio achieved by IM is higher than that of U-DH. 2 The IM score is 0.63% higher, and 1.40% higher than U-MHIM-MIL. These results highlight the superiority of heterogeneous criteria over uniform criteria in improving the performance of hard instance mining.

[0174] Visualization Analysis

[0175] The results in Table 5 demonstrate that two-view adversarial learning helps improve the model's classification performance, which is attributed in the experiments to its ability to aggregate more discriminative bag embedding features. Figure 3The attention scores of the two networks are visualized, with brighter areas representing higher attention scores. It can be observed that dual-view adversarial learning causes the positive network to focus on the lesion area in the WSI, while the negative network focuses on the normal tissue surrounding the lesion. The combination of the two enables the model to make a more comprehensive judgment on the WSI image and enhances the model's generalization ability.

[0176] Figure 4 The t-SNE was used to visualize the uniformity criterion (MHIM-MIL) and the heterogeneity criterion (DH). 2 The mining results for IM). The figure shows all instances in WSI, where circles represent positive instances, the remaining areas represent negative instances, and the areas within squares represent hard instances selected for training. DH 2 IM is more effective at capturing classification-challenged instances, especially positive and negative instances with tightly clustered features in the t-SNE plot; in contrast, MHIM-MIL, which uses a unified criterion, ignores some difficult instances, such as... Figure 4 The area highlighted by the middle circle. These findings confirm that the heterogeneity criterion surpasses the uniform criterion in mining difficult instances.

[0177] exist Figure 5 In this experiment, the baseline model and DH are demonstrated. 2 The IM model generates tumor probability maps for WSI image regions. In these maps, the actual tumor boundaries are outlined with lines, while highlighted areas represent regions predicted by the model to have a high tumor probability. Ideally, the highlighted areas should precisely cover the tumor region within the blue lines. Comparative results show that the MIL model equipped with the instance mining framework in this method can more effectively identify lesion regions during training.

[0178] In summary, this invention proposes DH 2 IM is a novel image-based framework for hard instance mining in full-slice image classification using multi-instance learning. DH 2 IM utilizes a fine-tuned diffusion model to generate a typical negative prototype for each WSI, guided by the unique visual context of each image. This image-specific prototype filters out easily classifiable instances that are highly similar to the prototype, thus focusing on difficult instances that are more informative for training. Considering the imbalance in the distribution of positive and negative instances, the method of this invention further develops a dual-view adversarial learning method, which uses two classification networks, each responsible for embedding and aggregating specific instances with its designated label. This design allows each network to focus on instances of the corresponding category in its own view, thereby generating more discriminative bag-level features and mitigating the negative impact of instance imbalance on WSI classification.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A pathological classification prediction method based on mining difficult instances of diffusion heterogeneity, characterized in that: Includes the following steps: S100: Select a publicly available pathological image dataset. D contains several pathological image WSIs, where each WSI is considered a bag, and each segmented image patch within a bag is considered an instance within that bag. The definition of D is as follows: D={(B1,Y1),…,(B i ,Y i ),…,(B n ,Y n )} Among them, B i Let B represent the i-th packet. i It contains m instances of {d1, ...,d2}. j ,…,d m } and j∈[1,m], Y i Indicates assignment to B i The label, B i Each instance within the instance has a corresponding real label y; S200: Using a pre-trained feature learning model, select the i-th bag for instance feature extraction to obtain the instance feature set. The calculation expression is as follows: in, b represents a collection of instances. j This represents the j-th instance d. j Corresponding instance features; Traverse each bag in D to obtain the instance feature set corresponding to each bag. All instance feature sets constitute the training set D'. S300: Construct a hard instance mining model M, M including a prototype-based hard instance mining module M PHIM and dual-view adversarial learning module M DVAL The M PHIM Includes a typical negative prototype generation module and a hard instance mining module HIM, the M PHIM Used for mining difficult instances in WSI; the M DVAL It includes two multi-instance learning models (MIL) for outputting classification results; the typical negative prototype generation module includes a diffusion model (DiT), a trainable feature adapter (Adapter), and a feature embedding layer. S400: will Enter M PHIM Obtain the set of difficult instances Then As M DVAL The input is the pathological prediction result of the WSI image; S500: The loss function for constructing model M Use D' as input to M and utilize M is trained, and the model parameters of M are updated using the gradient descent backpropagation principle. Training stops when the value no longer decreases or when the maximum number of iterations is reached. At this point, the trained M' is obtained. S600: Select the WSI image X to be predicted, input X into M', and obtain the prediction result of the pathological classification of X.

2. The pathological classification prediction method based on the mining of difficult instances of diffusion heterogeneity as described in claim 1, characterized in that: The set of difficult instances is obtained in S400. The steps are as follows: S410: Will Perform pooling operation and incorporate initial noise. The generated noise after adding instance feature information The calculation formula is as follows: in, This indicates a pooling operation. S411: Will As input to the DiT diffusion model, the canonical negative prototype q is calculated, and the calculation expression is as follows: Where t represents the t-th time step, This indicates the calculation process of DiT; S412: Input q into the Adapter and fine-tune it to obtain q′. The calculation formula is as follows: in, This represents a trainable feature adapter (Adapter). S413: Combining q and q′ yields the diffused and denoised canonical negative prototype q″, expressed as follows: q″=q+q′; S420: Calculate instance feature b j The similarity score s with q″ j The calculation formula is as follows: Where ||·|| represents L 2 Norm; S421: Repeat S420, calculate... The similarity scores of all instance features and q″ are represented by a set. S422: For S i Sort the instances in ascending order and set a hyperparameter θ. Select the instances with the highest similarity scores (top θ percentages) as the set of difficult instances.

3. The pathological classification prediction method based on the mining of difficult instances of diffusion heterogeneity as described in claim 2, characterized in that: The loss function is constructed in S500. The steps are as follows: S510: Input q″ into the feature embedding layer and output the feature embedding vector ε. - (q″), change ε - (q″) is used as the input to the binary classifier to calculate the B... i Prediction results for normal instances The calculation expression is as follows: Where, ε - (·) represents the feature embedding layer. Indicates a binary classification header; S511: Utilizing And a fine-tuning loss function is constructed using cross-entropy loss. The calculation expression is as follows: Among them, y nor Labels indicating normal portions of a pathological image; S520: M DVAL The two multi-instance learning models MIL in the model are defined as a normal perspective MIL network and a disease perspective MIL network, respectively. As M DVAL Input, Bag-level embeddings are obtained by embedding and aggregation through the MIL network from a normal perspective. nor , Bag-level embeddings f are obtained by embedding and aggregation of the MIL network from a disease perspective. dis The calculation formula is as follows: Where, ε nor (·) and ε dis (·) represent the feature embedding layers of the normal perspective network and the disease perspective network, respectively. and These represent the feature aggregators corresponding to the normal perspective network and the disease perspective network, respectively. S521: via f dis and Calculate the loss function of a disease-perspective network in binary classification. The calculation formula is as follows: Where Y represents the image instance type; S522: Constructing a binary classification loss function The calculation expression is as follows: in, This indicates a binary category header, and y represents the package label; S530: Integration and Obtain the total loss function of the model The calculation formula is as follows: Where γ represents a non-negative coefficient.

4. The pathological classification prediction method based on the mining of difficult instances of diffusion heterogeneity as described in claim 3, characterized in that: The loss function of the disease perspective network in S521 The categories are as follows: For diagnostic tasks, the loss function adapts to the true label y, and the binary classification loss function is expressed as: For classification tasks other than diagnostic tasks, the loss function uses the disease type label, and the binary classification loss function is expressed as follows: Among them, y dis This indicates the disease type label in the case image.

Images