Pathological image classification method and system based on online pseudo supervision and dynamic mutual learning
Through the online pseudo-supervision and dynamic mutual learning methods, the problems of inaccurate feature representation and redundant information interference in pathological image classification are solved, and more reliable pseudo-label generation and package decisions are achieved, which improves the robustness and accuracy of the model sex.
Patent Information
- Application Number
- CN202411211635.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-30
AI Technical Summary
Existing pathological image classification methods based on multi-example learning lead to unreliable pseudo-labels and undesirable decisions under inaccurate feature representation and interference from redundant information, which affects model performance.
Adopting the method based on online pseudo-supervision and dynamic mutual learning, through self-progressive feature fusion module, explicit and implicit representation learning module, decision-corrected pseudo-label generation module and dynamic decision-making module, we promote mutual learning of feature representation and pseudo-label generation through self-progressive feature fusion module, explicit and implicit representation learning module, decision-corrected pseudo-label generation module and dynamic decision-making module. , realize more trustworthy package decisions.
Improve the accuracy and reliability of pathological image classification, and enhance the robustness and decision-making ability of the model through refined feature representation and pseudo-label generation. The experimental results are 4.7% higher than the state-of-the-art methods on the CAMELYON16 and TCGA lung cancer datasets. and 6.7%.
Smart Images

Figure CN119068259A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of pathological images, and relates to a pathological image classification method, and specifically to a pathological image classification method and system based on online pseudo-supervision and dynamic mutual learning. Background Art
[0002] The cellular structures provided by histopathology slides are crucial for cancer diagnosis. However, examining the slides using an optical microscope in actual diagnosis is a tedious task for pathologists. Recently, computational pathology has been revolutionized by scanning slides into whole slide images (WSIs). However, due to the gigapixels and diverse microstructures of WSIs, traditional hand-crafted techniques cannot generate satisfactory predictions. In contrast, deep learning (DL) methods have attracted widespread attention due to their superior ability to learn contextual and texture information from WSIs. But collecting fine-grained annotations from irregular tumor tissues and high-resolution WSIs is time-consuming and laborious.
[0003] To address this problem, some researchers have explored the Multi-Instance Learning (MIL) technique. MIL treats each WSI as a "bag", learns the aggregated representation of the sampled image patches (Patches) in the WSI, and predicts the bag labels. According to the introduction of pseudo-labels, MIL-based methods can be roughly divided into two categories: bag label level and pseudo-label level. The former usually depicts a high-level bag representation through all instances for bag label prediction. However, it is not easy to aggregate target-related components from a large number of redundant instances, especially for sparse tumor regions in WSI. In addition, due to only bag label constraints, bag representation inevitably loses fine-grained pathological structures and semantics. These limitations of feature representation greatly reduce the accuracy of the final prediction. To promote feature representation, the latter draws on threshold-based attention, clustering similarity, and teacher models to generate pseudo-labels. Although it promotes model training and performance to a certain extent, over-reliance on the accuracy of pseudo-labels may lead to unforeseen disasters. At the same time, unreliable pseudo-labels may lead to error accumulation, thereby reducing model performance.
[0004] Due to inaccurate and noisy feature representations, existing MIL-based methods often generate unreliable pseudo-labels and inaccurate feature representations, leading to suboptimal decisions. Summary of the invention
[0005] In order to solve the problem of inaccurate feature representation and interference of redundant information in MIL based on packet label and pseudo label learning, which leads to unreliable pseudo labels and unsatisfactory decision-making, the present invention proposes to promote feature representation and pseudo label generation, and further explore their mutual learning to achieve better and more reliable packet decisions, so as to construct a pathological image classification method and system based on online pseudo-supervision and dynamic mutual learning (OODML). The present invention considers the intrinsic relationship between feature representation and pseudo label generation, and combines the advantages of online pseudo-supervision and dynamic mutual learning for WSI classification tasks.
[0006] The objective of the present invention is achieved through the following technical solutions:
[0007] A pathological image classification system based on online pseudo-supervision and dynamic mutual learning includes a tissue pathological image extraction instance and instance feature extraction module, a self-progressive feature fusion (SPFF) module, an explicit and implicit representation learning (EIRL) module, a decision revision pseudo-label (DRPL) generation module and a dynamic decision-making (DDM) module, wherein:
[0008] The tissue pathology image instance extraction and instance feature extraction module is responsible for tissue pathology image instance extraction and instance feature encoding, wherein:
[0009] The specific steps of extracting tissue pathology image instances are as follows: Given a full-slice dataset of pathology images It contains N WSIX i and its corresponding label Y i , X i The whole slice image WSI is regarded as the i-th "package"; the whole slice image WSI is read into the computer at a downsampled resolution and converted from RGB to HSV color space; the non-background tissue area of the WSI is calculated based on the image saturation after median blur and the channel threshold, and the morphological closing operation is combined to fill small gaps and holes as the final tissue area; according to the calculated tissue area, for each WSI X i Crop B at specified magnification i Image patch instances and store, where H and W represent the height and width of an instance, b∈[1,B i ];
[0010] The specific steps of the instance feature encoding are as follows: using a deep neural network pre-trained on ImageNet or a large pathological image dataset as an encoder, first, WSI X i Clipped instance collection Then for each instance x i,b Embedded into a D-dimensional feature vector In the example above, we get a low-dimensional feature vector set
[0011] The SPFF module includes a Transformer module, an Adaptive Memory Bank (AMB), a Multi-Level Feature Fusion (MLFF) module, a Memory-based Cross-Attention (MCA) module and a pseudo-label prediction module, wherein:
[0012] The Transformer module It consists of a multi-head self-attention module and a residual connection. At time t, a set of instances are randomly sampled The corresponding feature vector set M represents the M instance feature vectors contained in the sampled pseudo bag, and T represents the input image X i Dynamic interactive sampling of M-sized pseudo-package execution times, represents the jth instance, is the jth instance The corresponding eigenvector for the sampled Design a Token position vector Let's learn pseudo package Spatial position information in; randomly initialize a Token vector Combined with Transformer module extract The local context information of the instance feature enters the Token In , the above process is expressed as:
[0013]
[0014] The AMB is used to store the most relevant feature expression of the tag. The AMB symbol for time t is: The AMB symbol at time t-1 is AMB uses the enhanced pseudo-packet Token predicted at time t-1 and the corresponding predicted pseudo label probability As input, rearrange To save the key K tokens and their corresponding pseudo-label probabilities:
[0015]
[0016] In the formula, Indicates the kth Token stored, express The corresponding pseudo label probability;
[0017] The MLFF module uses As a hint of the rich semantics of the category, Refined into a more label-related feature In order to further highlight the key contribution, the Softmax function is used to The features in are normalized to 0-1 for strengthening The most significant feature of Quantification to optimal marker To obtain the significance weight Then, is used to transform the discrete discriminant expression from and Aggregate to more discriminative features These operations can be expressed as:
[0018]
[0019] The MCA module has the following features for AMB And the features after MLFF module refinement Perform concatenation operations and map them into K and V. Defined as Q, the following formula makes Extract significant information from K and V and enhance the current pseudo-package Token feature expression as follows:
[0020]
[0021] The pseudo-label prediction module is a multi-layer perceptron (MLP) constructed by a linear layer and a Softmax activation function layer. Used to predict The corresponding pseudo label probability The process is described as:
[0022]
[0023] The EIRL module includes a linear attention module (LAM), a multi-head self-attention module (MSA), a decision revision pseudo-label (DRPL) generation module, an explicit package label prediction module and an implicit package label prediction module, wherein:
[0024] The LAM module is used to explore pseudo-packet markings With explicit package feature expression The single connection relationship between them uses a linear attention score To express, specifically, First, a linear layer Linear and a Softmax activation function are used to obtain the attention weight. Subsequent use right Perform weighted summation to obtain explicit package feature expression The above process is expressed as:
[0025]
[0026] Where (*) represents the weight and vector weighted sum operation;
[0027] The MSA module is equipped with an initialization class marker Let's explore pseudo-package marking With implicit package feature expression Multiple connections and cross-relationships between them, using multi-head attention scores Specifically, first concatenate is a feature map H i , after linear transformation into Q, K and V, and split into multi-head representation, Q and K calculate the attention score and multiply it with V to extract features and merge them into Q. The process can be described as:
[0028]
[0029]
[0030] Among them, W q , W k , W v , W o is the linear transformation matrix, Split the query, key, and value into multiple heads, where Head represents the number of heads, and d k represents the dimension of each head, (*) represents the weight and vector weighted sum operation;
[0031] The explicit packet label prediction module and the implicit packet label prediction module are multi-layer perceptrons constructed by a linear layer and a Softmax activation function layer. and Explicit package feature representation of LAM and MSA module outputs and implicit package feature expression are passed into and and Used to predict and The probability of a packet label and The process is described as:
[0032]
[0033] The DRPL generation module is used to explore the potential relationship between pseudo bag representation and bag label prediction for the positive bag X i In Generate pseudo labels otherwise, will be set to negative label 0. Specifically, the LAM module and the MSA module in the EIRL module are employed to study the complex relationship between pseudo bag labels and their contribution to bag label inference. The bag label probability is used to further calibrate the score to generate more reliable pseudo labels. Used to supervise the pseudo-label prediction probability And the soft cross entropy loss optimization model is introduced. The above process is expressed as:
[0034]
[0035] Among them, the Sigmoid function is used to adjust the pseudo label to 0-1, and τ is set to 1 or 2 to adapt to WSI datasets with different tumor sizes;
[0036] The DDM module is used to predict the packet label, first using and To refine and The predicted attributes are then passed through a linear layer and a Softmax activation function to obtain the exact explicit weights. and implicit weights Final integration and To obtain the final bag label prediction The process is described as:
[0037]
[0038] A pathological image classification method based on online pseudo-supervision and dynamic mutual learning comprises the following steps:
[0039] Step 1: Given a full-slice dataset of pathological images It contains N WSI X i and its corresponding label Y i , X i is regarded as the i-th “bag”, the non-background tissue area of WSI is extracted using the threshold, and B is cropped i Examples Where H and W represent the height and width of an instance, b∈[1,B i ];
[0040] Step 2: Use the pre-trained deep neural network as the encoder to convert x i,b Embedded into a D-dimensional feature vector middle;
[0041] Step 3: At time t, from package X i Randomly sample pseudo packets M represents the M instance feature vectors contained in the sampled pseudo bag, and T represents the input image X i Dynamically interactively sample the number of times a pseudo-packet of size M is executed in order to generate informative pseudo-packet markers '
[0042] Step 4: Pseudo-packet marking The extra valuable information in AMB and the refined feature expression in MLFF module are aggregated into a new tag through the memory bank cross attention MCA module. In order to promote Pseudo label prediction for
[0043] Step 5: Predicted Pseudo-Bag Label Probability Help AMB capture and save the K most relevant tags is the pseudo packet mark at time t+1 Provide additional knowledge;
[0044] Step 6: To provide credible pseudo labels Pseudo bag labels for supervised prediction Exploring pseudo-packet marking using the EIRL module With the bag label prediction ( and ) to construct the DRPL generation module;
[0045] Step 7: The DDM module takes explicit and implicit packet predictions and representations as input and adaptively weights them. and To achieve more robust package decisions
[0046] Step 8. Optimize by coordinating the cross entropy loss of bag labels and pseudo labels to promote better feature representation and pseudo label generation to improve classification performance.
[0047] Compared with the prior art, the present invention has the following advantages:
[0048] 1. This paper proposes to promote feature representation and pseudo-label generation, and further explores their mutual learning to build an online pseudo-supervised and dynamic mutual learning multi-instance learning framework to achieve better and more reliable package decisions.
[0049] 2. To achieve discriminative and compact representation, we design an elegant adaptive repository AMB to collect the most informative components of the current WSI, and further formulate a self-improving feature fusion SPFF module to integrate the label-related historical information and instant semantic changes in AMB to enhance the current pseudo-bag label representation. The ideas behind SPFF include: 1) pseudo-label prediction provides category-level insights to help AMB capture salient labels; 2) AMB combines memory-based cross attention (MCA) and MLFF schemes to provide refined features to promote current labeling; 3) Fine-grained labeling promotes pseudo-label prediction and updates AMB.
[0050] 3. In order to generate reliable pseudo-labels, the present invention designs a decision-corrected pseudo-label DRPL generation module to explore the intrinsic connection between pseudo-package representation and package label prediction, so as to generate more reliable pseudo-labels. Specifically, the present invention proposes an EIRL strategy to study the complex relationship between pseudo-package labels and their contribution to package label reasoning through a linear attention module (LAM) and a multi-head self-attention module (MSA). Since the attention score is inherently noisy, the present invention uses the package label probability to further calibrate the score to generate more reliable pseudo-labels.
[0051] 4. To alleviate redundant and ambiguous representations, the present invention borrows the category prior of pseudo-label prediction to promote label-related feature learning and update AMB, thereby forming dynamic mutual learning and optimization between feature representation and pseudo-label generation.
[0052] 5. The present invention develops a dynamic decision-making DDM module to coordinate the explicit and implicit representations of packet information to achieve more robust decision-making.
[0053] 6. Extensive experiments on two datasets show that the performance of the OODML framework of the present invention on the CAMELYON16 and TCGA lung cancer datasets is 4.7% and 6.7% higher than that of the most advanced BCL method, respectively. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 This is a flow chart of pathological image classification based on online pseudo-supervision and dynamic mutual learning;
[0055] Figure 2 It is a memory-based cross-attention module;
[0056] Figure 3 It is a dynamic decision-making module;
[0057] Figure 4 Correct pseudo-label generation module for decision making. DETAILED DESCRIPTION
[0058] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.
[0059] The present invention provides a pathological image classification system based on online pseudo-supervision and dynamic mutual learning, such as Figure 1 As shown, the system includes a tissue pathology image extraction instance and instance feature extraction module, a SPFF module, an EIRL module, a DRPL generation module and a DDM module.
[0060] In the present invention, the tissue pathology image extraction instance and instance feature extraction module is responsible for tissue pathology image instance extraction and instance feature encoding, wherein:
[0061] The specific steps of extracting tissue pathology image instances are as follows: Given a full-slice dataset of pathology images It contains N WSI X i and its corresponding label Y i , X i The whole slice image WSI is regarded as the i-th "package"; the whole slice image WSI is read into the computer at a downsampled resolution and converted from RGB to HSV color space; the non-background tissue area of the WSI is calculated based on the image saturation after median blur and the channel threshold, and the morphological closing operation is combined to fill small gaps and holes as the final tissue area; according to the calculated tissue area, for each WSI X i Crop B at specified magnification i Image block instance and store, where H and W represent the height and width of an instance, b∈[1,Bi ].
[0062] The specific steps of the instance feature encoding are as follows: using a deep neural network pre-trained on ImageNet or a large pathological image dataset as an encoder, first, WSI X i Clipped instance collection Then for each instance x i,v Embedded into a D-dimensional feature vector In the example above, we get a low-dimensional feature vector set
[0063] In the present invention, in order to better perform feature representation, the SPFF module includes a Transformer module, an Adaptive Memory Bank (AMB), a Multi-Level Feature Fusion (MLFF) module, a memory-based cross-attention (MCA) module, and a pseudo-label prediction module. Specifically, in order to obtain a discriminative and compact feature expression, the SPFF module fuses the historical information and immediate semantic information of the current WSI from the AMB to enhance the current pseudo-bag (Pseudo-bag) token representation. The concepts behind SPFF include: 1) The pseudo-label probabilities corresponding to the predicted pseudo-labels provide category insights to help AMB capture significant labels; 2) AMB integrates memory-based cross-attention (MCA) and multi-level feature fusion (MLFF) schemes to provide refined features to promote the current Token expression; 3) More fine-grained Tokens promote pseudo-label prediction accuracy and update AMB. Among them:
[0064] Transformer module: At time t, randomly sample a set of instances The corresponding feature vector set M represents the M instance feature vectors contained in the sampled pseudo bag, and T represents the input image X i Dynamic interactive sampling of M-sized pseudo-package execution times, represents the jth instance, is the jth instance The corresponding eigenvector for the sampled Design a Token position vector Let's learn pseudo package Spatial position information in; randomly initialize a Token vector Combined with Transformer Module (TRM) extract The local context information of the instance feature enters the Token In the Transformer module It consists of an 8-head self-attention module and residual connections. The above process can be expressed as:
[0065]
[0066] AMB: In order to achieve a compact and discriminative representation, this paper proposes AMB to store the most relevant feature expression of the label by making full use of the pseudo-label prediction prior. The symbolic representation of AMB at time t is The AMB symbol at time t-1 is AMB uses the enhanced pseudo packet token predicted at time t-1 and the corresponding predicted pseudo-label probability As input, rearrange To save the key K tokens and their corresponding pseudo-label probabilities:
[0067]
[0068] In the formula, Indicates the kth Token stored, express The corresponding pseudo label probability.
[0069] MLFF module: In order to extract decision-friendly components, the MLFF proposed in this paper includes two stages of selection and one fusion feature operation. MLFF uses As a hint of the rich semantics of the category, Refined into a more label-related feature In order to further highlight the key contribution, the Softmax function is used to The features in are normalized to 0-1 for strengthening Then Quantification for optimal labeling To obtain the significance weight Then, the precise is used to transform the discrete discriminant expression from and Aggregate to more discriminative features These operations can be expressed as:
[0070]
[0071] MCA: To facilitate pseudo bag label prediction Pseudo-package Token Employ MCA to integrate key knowledge from adaptive memory and multi-level feature fusion modules, such as Figure 2 Specifically, for the features in AMB And the features after MLFF module refinement Perform concatenation operation and map to (key, Key, referred to as K) K and (value, Value, referred to as K) V. Defined as (query, Query, referred to as Q), the following formula makes Extract significant information from K and V and enhance the current pseudo-package Token feature expression as follows:
[0072]
[0073] Pseudo-label prediction module: a more comprehensive pseudo-packet Token feature expression output by the MCA module Passed into a multilayer perceptron consisting of a linear layer and a Softmax activation function layer Used to predict The corresponding pseudo label probability The process is described as:
[0074]
[0075] In order to facilitate pseudo label generation, the present invention introduces an EIRL module to explore pseudo packet labels. With package representation And label prediction The explicit relation (ER) and implicit relation (IR) between them, then and It is transferred into two MLP classifiers consisting of linear layers and Softmax activation functions. and Used to predict package labels and Specifically, in order to generate reliable pseudo labels, the present invention proposes a decision-corrected pseudo label (DRPL) generation scheme to explore the potential relationship between pseudo package representation and package label prediction. More specifically, an explicit-implicit representation learning (EIRL) strategy is proposed to investigate the complex relationship between pseudo package tokens and their contribution to package label prediction through a linear attention module (LAM) and a multi-head self-attention (MSA) module. Since the attention score itself is noisy, the package label probability is used to further calibrate the score to generate reliable pseudo labels. The EIRL module includes a linear attention (LAM) module, a multi-head self-attention (MSA) module, an explicit package label prediction module, and an implicit package label prediction module, wherein:
[0076] LAM module: The present invention uses a LAM module to explore pseudo-packet markings With explicit package feature expression The single connection relationship between , using the linear attention score extracted from LAM Express. First, a linear layer Linear and a Softmax activation function are used to obtain the attention weight. Subsequent use right Perform weighted summation to obtain explicit package feature expression The above process is expressed as:
[0077]
[0078] Where (*) represents the weight and vector weighted sum operation.
[0079] MSA module: The MSA module is equipped with an initialization class marker (Class Token, cls for short) to explore pseudo-package markers With implicit package feature expression The multi-connections and cross-relationships between them are obtained using the multi-head attention scores extracted from MSA. Specifically, first concatenate is a feature map H i , after linear transformation into Q, K and V, and split into multi-head representation, Q and K calculate the attention score and multiply it with V to extract features and merge them into Q. The process can be described as:
[0080]
[0081]
[0082] Among them, W q , W k , Wv , W o is the linear transformation matrix, Split the query, key, and value into multiple heads, where Head represents the number of heads, and d k represents the dimension of each head, (*) represents the weight and vector weighted sum operation.
[0083] Explicit package label prediction module and implicit package label prediction module: explicit package feature representation output by LAM module and MSA module and implicit package feature expression are passed into a multi-layer perceptron consisting of a linear layer and a Softmax activation function layer. and and Used to predict and The probability of a packet label and The process is described as:
[0084]
[0085] In order to further improve the robustness of packet decision making, a dynamic decision making (DDM) module is designed to predict packet labels. Specifically, and To refine and It then passes through a linear layer and a Softmax activation function to obtain the exact explicit weights and implicit weights Finally, integration and To obtain the final bag label prediction like Figure 3 As shown:
[0086]
[0087] In this paper, in order to generate reliable pseudo labels, a DRPL generation module is proposed to explore the potential relationship between pseudo bag representation and bag label prediction. i In Generate pseudo labels otherwise, will be set to negative label 0. Figure 4As shown, specifically, the present invention employs the LAM module and the MSA module in the EIRL module to study the complex relationship between pseudo bag labels and their contribution to bag label inference. Since the attention score itself is noisy, the present invention uses the bag label probability to further calibrate the score to generate more reliable pseudo labels. Used to supervise the pseudo-label prediction probability And the soft cross entropy loss optimization model is introduced. The above process is expressed as:
[0088]
[0089] The Sigmoid function is used to adjust the pseudo-label to 0-1, and τ is set to 1 or 2 to adapt to WSI datasets with different tumor sizes.
[0090] In this paper, in order to achieve better feature representation and pseudo label generation and promote packet decision making, three packet label probabilities ( and ) and pseudo label prediction The actual composite constraint involves two packages of cross entropy loss and DDM cross entropy loss and Pseudo Label Cross Entropy Loss (PLCE) The formulas for these loss functions are as follows:
[0091]
[0092] The final total loss is defined as:
[0093]
[0094] where λ PLCE It is used to adjust the proportion of pseudo-label loss in the total loss, and is adjusted from 0 to 0.5 by integrating the duration and cosine curve to guide the model from coarse to more detailed information exploration.
[0095] The present invention provides a pathological image classification system method based on online pseudo-supervision and dynamic mutual learning, such as Figure 1 As shown, the method comprises the following steps:
[0096] Step 1: Given a full-slice dataset of pathological images It contains N WSI X i and its corresponding label Y i , X i is regarded as the i-th “bag”, the non-background tissue area of WSI is extracted using the threshold, and B is cropped i Examples Where H and W represent the height and width of an instance, b∈[1,Bi ].
[0097] Step 2: Use the pre-trained deep neural network as the encoder to convert x i,b Embedded into a D-dimensional feature vector middle.
[0098] Step 3: To alleviate the problem of incomplete information of a single instance and redundant interference of all instances, at time point t, i Randomly sample pseudo packets M represents the M instance feature vectors contained in the sampled pseudo bag, and T represents the input image X i Dynamic interaction sampling is the number of times a pseudo package of size M is executed. Compared with fixed grouping or Top-K instance strategy, dynamic sampling helps OODML to perceive Semantic and spatial features in order to generate informative pseudo-bag tokens Representation.
[0099] Step 4: Pseudo-package marking to enhance feature representation Equipped with a SPFF module to aggregate the additional valuable information in AMB and the refined feature expressions in the MLFF module into a new marker In order to promote Pseudo label prediction for
[0100] Step 5: Predicted Pseudo-Bag Label Probability Help AMB capture and save the K most relevant tags is the pseudo packet mark at time t+1 Provide additional knowledge.
[0101] Step 6: To provide credible pseudo labels Pseudo bag labels for supervised prediction Introducing the EIRL module to explore pseudo-package marking With the bag label prediction ( and ) to construct the DRPL generation module.
[0102] Step 7: The DDM module takes explicit and implicit packet predictions and representations as input and adaptively weights them. and To achieve more robust package decisions
[0103] Step 8. Optimize OODML by coordinating the cross entropy loss between bag labels and pseudo labels to promote better feature representation and pseudo label generation to improve classification performance.
[0104] In order to verify the OODML of the present invention, the present invention has been extensively experimented with typical representative WSI analysis MIL methods on CAMELYON16 and TCGA lung cancer datasets. These methods include: 1) package label level (MeanPooling, ABMIL, SetTransformer, DeepAttnMIL, TransMIL, MuRCL, MHIM and SSMMIL); 2) pseudo-label level (MaxPooling, RNNMIL, DSMIL, CLAM, DTFD, DGMIL, IAT, BCL and CIMIL). The test indicators are area under the curve (AUC), accuracy (Accuracy), and F1 score (F1) are used for evaluation, and the threshold is set to 0.5. The OODML is trained for 100 iterations, using the AdaMax optimizer with an initial learning rate of 0.0004.
[0105] As can be seen from Table 1, the proposed method OODML is competitive in all indicators, surpassing the pseudo-label-based BCL and packet label-based (SSMMIL and TransMIL) methods by 4.7% and 3.6% in accuracy, respectively.
[0106] Table 1 Comparison of model training results
[0107]
[0108]
Claims
1. A pathological image classification system based on online pseudo-supervision and dynamic mutual learning, characterized by The system includes a tissue pathology image extraction instance and instance feature extraction module, a self-improving feature fusion SPFF module, an explicit and implicit representation learning EIRL module, a decision correction pseudo label DRPL generation module and a dynamic decision DDM module, wherein: The tissue pathology image extraction instance and instance feature extraction module is responsible for tissue pathology image instance extraction and instance feature encoding; The SPFF module includes a Transformer module, an adaptive memory bank AMB, a multi-level feature fusion MLFF module, a memory bank-based cross attention MCA module and a pseudo-label prediction module, wherein: The Transformer module It consists of a multi-head self-attention module and a residual connection. At time t, a set of instances are randomly sampled The corresponding feature vector set M represents the M instance feature vectors contained in the sampled pseudo bag, and T represents the input image X i Dynamic interactive sampling of M-sized pseudo-package execution times, represents the jth instance, is the jth instance The corresponding eigenvector for the sampled Design a Token position vector Let's learn pseudo package Spatial position information in; randomly initialize a Token vector Combined with Transformer module extract The local context information of the instance feature enters the Token In , the above process is expressed as: The AMB is used to store the most relevant feature expression of the tag. The AMB symbol for time t is: The AMB symbol at time t-1 is AMB uses the enhanced pseudo packet token predicted at time t-1 and the corresponding predicted pseudo label probability As input, rearrange To save the key K tokens and their corresponding pseudo-label probabilities: In the formula, Indicates the kth Token stored, express The corresponding pseudo label probability; The MLFF module uses As a hint of the rich semantics of the category, Refined into a more label-related feature In order to further highlight the key contribution, the Softmax function is used to The features in are normalized to 0-1 for strengthening The most significant feature of Quantification to optimal marker To obtain the significance weight Then, is used to transform the discrete discriminant expression from and Aggregate to more discriminative features These operations are expressed as: The MCA module has the following features for AMB And the features after MLFF module refinement Perform concatenation operations and map them into K and V. Defined as Q, the following formula makes Extract significant information from K and V and enhance the current pseudo-package Token feature expression as follows: The pseudo-label prediction module is a multi-layer perceptron constructed by a linear layer and a Softmax activation function layer. Used to predict The corresponding pseudo label probability The process is described as: The EIRL module includes a linear attention LAM module, a multi-head self-attention MSA module, a decision-corrected pseudo-label DRPL generation module, an explicit package label prediction module and an implicit package label prediction module, wherein: The LAM module is used to explore pseudo-packet markings With explicit package feature expression The single connection relationship between them uses a linear attention score To express, specifically, First, a linear layer Linear and a Softmax activation function are used to obtain the attention weight. Subsequent use right Perform weighted summation to obtain explicit package feature expression The above process is expressed as: Where (*) represents the weight and vector weighted sum operation; The MSA module is equipped with an initialization class marker Let's explore pseudo-package marking With implicit package feature expression Multiple connections and cross-relationships between them, using multi-head attention scores Specifically, first concatenate is a feature map H i , after linear transformation into Q, K and V, and split into multi-head representation, Q and K calculate the attention score and multiply it with V to extract features and merge them into Q. The process can be described as: Among them, W q , W k , W v , W o is the linear transformation matrix, Split the query, key, and value into multiple heads, where Head represents the number of heads, and d k represents the dimension of each head, (*) represents the weight and vector weighted sum operation; The explicit packet label prediction module and the implicit packet label prediction module are multi-layer perceptrons constructed by a linear layer and a Softmax activation function layer. and Explicit package feature representation of LAM and MSA module outputs and implicit package feature expression are passed into and and Used to predict and The probability of a packet label and The process is described as: The DRPL generation module is used to explore the potential relationship between pseudo bag representation and bag label prediction for the positive bag X i In Generate pseudo labels otherwise, will be set to negative label 0; The DDM module is used to predict the packet label, first using and To refine and The predicted attributes are then passed through a linear layer and a Softmax activation function to obtain the exact explicit weights. and implicit weights Final integration and To obtain the final bag label prediction The process is described as:
2. The pathological image classification system based on online pseudo-supervision and dynamic mutual learning according to claim 1 is characterized in that The specific steps of extracting tissue pathology image instances are as follows: Given a full-slice dataset of pathology images It contains N WSI X i and its corresponding label Y i , X i The whole slice image WSI is read into the computer at a downsampled resolution and converted from RGB to HSV color space; The non-background tissue area of WSI was calculated based on the image saturation and channel threshold after median blur, and the morphological closing operation was combined to fill small gaps and holes as the final tissue area; According to the calculated tissue area for each WSI X i Crop B at specified magnification i Image patch instances and store, where H and W represent the height and width of an instance, b∈[1,B i ].
3. The pathological image classification system based on online pseudo-supervision and dynamic mutual learning according to claim 2 is characterized in that The specific steps of the instance feature encoding are as follows: using a deep neural network pre-trained on ImageNet or a large pathological image dataset as an encoder, first, WSI X i Clipped instance collection Then for each instance x i,b Embedded into a D-dimensional feature vector In the above example, we get a low-dimensional feature vector set 4. The pathological image classification system based on online pseudo-supervision and dynamic mutual learning according to claim 1 is characterized in that The DRPL generation module employs the LAM module and the MSA module in the EIRL module to study the complex relationship between pseudo bag labels and their contribution to bag label inference, and uses the bag label probability to further calibrate the score to generate more reliable pseudo labels. Used to supervise the pseudo-label prediction probability And the soft cross entropy loss optimization model is introduced. The above process is expressed as: The Sigmoid function is used to adjust the pseudo-label to 0-1, and τ is set to 1 or 2 to adapt to WSI datasets with different tumor sizes.
5. The pathological image classification system based on online pseudo-supervision and dynamic mutual learning according to claim 1 is characterized in that The total loss of the system is defined as: in, and is the packet cross entropy loss, is the DDM cross entropy loss, is the pseudo-label cross entropy loss, λ PLCE Used to adjust the proportion of pseudo label loss in the total loss.
6. A method for pathological image classification based on online pseudo-supervision and dynamic mutual learning using the system according to any one of claims 1 to 5, characterized in that The method comprises the following steps: Step 1: Given a full-slice dataset of pathological images It contains N WSI X i and its corresponding label Y i , X i Considered as the i-th "bag", the non-background tissue area of WSI is extracted using the threshold, and B is cropped i Examples Where H and W represent the height and width of an instance, b∈[1,B i ]; Step 2: Use the pre-trained deep neural network as the encoder to convert x i,b Embedded into a D-dimensional feature vector middle; Step 3: At time t, from package X i Randomly sample pseudo packets M represents the M instance feature vectors contained in the sampled pseudo bag, and T represents the input image X i Dynamically interactively sample the number of times a pseudo-packet of size M is executed in order to generate informative pseudo-packet markers Representation of; Step 4: Pseudo-packet marking The extra valuable information in AMB and the refined feature expression in MLFF module are aggregated into a new tag through the memory bank cross attention MCA module. In order to promote Pseudo label prediction for Step 5: Predicted Pseudo-Bag Label Probability Help AMB capture and save the K most relevant tags is the pseudo packet mark at time t+1 Provide additional knowledge; Step 6: To provide credible pseudo labels Pseudo bag labels for supervised prediction Exploring pseudo-packet marking using the EIRL module With the bag label prediction ( and ) to construct the DRPL generation module; Step 7: The DDM module takes explicit and implicit packet predictions and representations as input and adaptively weights them. and To achieve more robust package decisions Step 8. Optimize by coordinating the cross entropy loss of bag labels and pseudo labels to promote better feature representation and pseudo label generation to improve classification performance.
Citation Information
Patent Citations
Negative sample guided self-distillation pathological image analysis method
CN115482232A
Instance level classifier construction method based on weak supervision self-training
CN115879021A
Pathological image lesion area determination method and device, and model training method and device
CN116958018A
Rapid pathological image analysis method and device based on magnification alignment Transformer
CN117333442A
Biological context for analyzing full slide images
CN118076970A
Cited By
Pathological image classification system and method based on Mama and hybrid expert model
CN119851017A
