A pathological image classification method and system based on online pseudo-supervision and dynamic mutual learning

Abstract

This invention discloses a pathological image classification method and system based on online pseudo-supervision and dynamic mutual learning. To achieve discriminative and compact representations, an adaptive storage library (AMB) is designed to collect the most informative components of the current WSI (Wide Image Substances Indicator). A self-improving feature fusion (SPFF) module is also developed to integrate label-related historical information and real-time semantic changes in the AMB to enhance the current pseudo-packet label representation. A decision-corrected pseudo-label generation (DRPL) module is designed to explore the intrinsic relationship between pseudo-packet representation and packet label prediction, thereby generating more reliable pseudo-labels. To alleviate redundant and ambiguous representations, the category prior of pseudo-label prediction is borrowed to promote label-related feature learning and update the AMB, thus forming a dynamic mutual learning and optimization between feature representation and pseudo-label generation. A dynamic decision-making (DDM) module is developed to coordinate the explicit and implicit representations of packet information, thereby achieving more robust decision-making.

Description

Technical Field

[0001] This invention belongs to the field of pathological image technology and relates to a pathological image classification method, specifically a pathological image classification method and system based on online pseudo-supervision and dynamic mutual learning. Background Technology

[0002] The cellular structure provided by histopathological slides is crucial for cancer diagnosis. However, examining slides using an optical microscope in actual diagnosis is a tedious task for pathologists. Recently, computational pathology has been revolutionized by scanning slides as whole-slide images (WSIs). However, due to the gigapixel size and diverse microstructures of WSIs, traditional hand-crafting techniques cannot generate satisfactory predictions. In contrast, deep learning (DL) methods have attracted widespread attention for their superior ability to learn the context and texture information of WSIs. But collecting fine-grained annotations from irregular tumor tissues and high-resolution WSIs is both time-consuming and laborious.

[0003] To address this issue, some researchers have explored Multi-Instance Learning (MIL) techniques. MIL treats each WSI (Weighted Image Slice) as a "packet," learning an aggregated representation of the sampled image patch (Instance) within the WSI and predicting the packet label. Based on the introduction of pseudo-labels, MIL-based methods can be broadly categorized into two types: packet label level and pseudo-label level. The former typically depicts a high-level packet representation using all instances for packet label prediction. However, aggregating target-related components from a large number of redundant instances is not easy, especially for sparse tumor regions in the WSI. Furthermore, due to the constraint of only packet labels, the packet representation inevitably loses fine-grained pathological structures and semantics. These limitations of feature representation significantly reduce the accuracy of the final prediction. To facilitate feature representation, the latter borrows from techniques such as threshold-based attention, clustering similarity, and teacher models to generate pseudo-labels. While this improves model training and performance to some extent, over-reliance on pseudo-label accuracy can lead to unforeseen disasters. Simultaneously, unreliable pseudo-labels can cause error accumulation, thereby degrading model performance.

[0004] Due to inaccurate feature representations and interference, existing MIL-based methods often generate unreliable pseudo-labels and inaccurate feature representations, leading to undesirable decisions. Summary of the Invention

[0005] To address the problems of inaccurate feature representations and redundant information in MIL (Miscellaneous Learning Based on Packet Labels and Pseudo-labels), which lead to unreliable pseudo-labels and suboptimal decision-making, this invention proposes a method and system for pathological image classification based on Online Pseudo-Supervision and Dynamic Mutual Learning (OODML). This invention considers the intrinsic relationship between feature representation and pseudo-label generation and combines the advantages of online pseudo-supervision and dynamic mutual learning for WSI (Warnings in Situational Image Classification) tasks.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A pathological image classification system based on online pseudo-supervision and dynamic mutual learning includes modules for extracting instances and instance features from histopathological images, a self-progressive feature fusion (SPFF) module, an explicit-implicit representation learning (EIRL) module, a decision revision pseudo-label (DRPL) generation module, and a dynamic decision-making (DDM) module, wherein:

[0008] The histopathological image extraction instance and instance feature extraction module is responsible for histopathological image instance extraction and instance feature encoding, wherein:

[0009] The specific steps for extracting the histopathological image instances are as follows: Given a whole slice dataset of pathological images It contains N WSIX i and its corresponding label Y i X i Consider the i-th "packet"; the full-slice image WSI is read into the computer at downsampling resolution and converted from RGB to HSV color space; the non-background tissue region of WSI is calculated based on the saturation and channel thresholds of the image after median blurring, and morphological closure operations are combined to fill small gaps and holes as the final tissue region; based on the calculated tissue region, for each WSI X i Crop B at a specified magnification. i Image patch instances And store, where H and W represent the height and width of an instance, b∈[1,B] i ];

[0010] The specific steps for encoding the instance features are as follows: Using a deep neural network pre-trained on ImageNet or a large pathological image dataset as the encoder, firstly, WSI X... i A collection of clipping instances Then take each instance x i,b Embedded into a D-dimensional feature vector In this process, a set of low-dimensional feature vectors is obtained.

[0011] The SPFF module includes a Transformer module, an Adaptive Memory Bank (AMB), a Multi-Level Feature Fusion (MLFF) module, a Memory-based Cross-Attention (MCA) module, and a pseudo-label prediction module, wherein:

[0012] The Transformer module It consists of a multi-head self-attention module and residual connections, and at time t, a set of instances is randomly sampled. The corresponding set of feature vectors M represents the number of instance feature vectors contained in the sampled pseudo-packet, and T represents the input image X. i The number of times a pseudo-packet of size M is executed in a dynamic interactive sampling method. This represents the j-th instance. It is the j-th instance. The corresponding feature vector, for the sampled Design a token position vector Come and learn about fake packages Spatial location information; randomly initialize a token vector. Combine with Transformer module extract Local context information of instance features is entered into the Token In the above process, it is described as follows:

[0013]

[0014] The AMB is used to store the most relevant feature representations of the tag. The AMB symbol representation at time t is as follows: The AMB symbol at time t-1 is represented as AMB predicts the enhanced pseudo-packet token at time t-1. and the corresponding predicted pseudo-label probability As input, rearrange To store the key K tokens and their corresponding pseudo-label probabilities:

[0015]

[0016] In the formula, This represents the k-th token stored. express The corresponding pseudo-label probability;

[0017] The MLFF module uses As a semantically rich hint for categories, to store in the dictionary Refined to include more tag-related features To further highlight key contributions, the Softmax function is used to... The features in the data are normalized to 0-1 for reinforcement. The most prominent feature, then Quantization as the best label To obtain significance weights Subsequently, Used to extract dispersed discriminative expressions from and Aggregation to more discriminative features These operations can be expressed as:

[0018]

[0019] The MCA module provides features in the AMB. Features refined from the MLFF module Perform the concatenation operation and map it to K and V. Defined as Q, which is achieved through the following formula Extracting saliency information from K and V enhances the current pseudo-packet token feature representation.

[0020]

[0021] The pseudo-label prediction module is a multi-layer perceptron (MLP) constructed from a linear layer and a softmax activation function layer. Used for prediction Corresponding pseudo-label probability The process is described as follows:

[0022]

[0023] The EIRL module includes a Linear Attention Module (LAM), a Multi-head Self-Attention (MSA) module, a Decision Revision Pseudo-Label (DRPL) generation module, an explicit packet label prediction module, and an implicit packet label prediction module, wherein:

[0024] The LAM module is used to detect fake packet markers. With explicit package feature expression The single connection between them is represented by linear attention scores. Expression, specifically First, attention weights are obtained by passing the material through a linear layer and a softmax activation function. Then use right Perform weighted summation to obtain explicit package feature representation. The above process can be represented as follows:

[0025]

[0026] In the formula, (*) denotes the weight and vector weighted summation operation;

[0027] The MSA module is equipped with an initialization class marker. Explore pseudo-packet tags With implicit bag feature representation Multiple connections and cross relationships between them, using multi-head attention scores Specifically, this means first assembling... For a feature map H i After a linear transformation, the data is transformed into Q, K, and V, and then segmented into multi-head representations. Attention scores are calculated for Q and K, multiplied by V to extract features, and then fused into Q. This process is described as follows:

[0028]

[0029]

[0030] Among them, W q W k W v W o It is a linear transformation matrix. The query, key, and value are split into a multi-head representation, where Head represents the number of multi-heads, and d k Represents the dimension of each head, and (*) indicates the weight and vector weighted summation operation;

[0031] The explicit packet label prediction module and the implicit packet label prediction module are multilayer perceptrons constructed from a linear layer and a Softmax activation function layer. and Explicit packet feature representations output by the LAM and MSA modules and implicit bag feature representation They were passed in separately and and Used to predict and Packet tag probability and The process is described as follows:

[0032]

[0033] The DRPL generation module is used to explore the potential relationship between counterfeit bag characterization and bag label prediction, for positive bag X i In Generate pseudo tags otherwise, The label will be set to negative 0. Specifically, the LAM and MSA modules from the EIRL module will be used to study the complex relationships between pseudo-bag labels and their contribution to bag label inference. The bag label probability will be used to further calibrate the score to generate more reliable pseudo-labels. Used to supervise the prediction probability of pseudo-labels Furthermore, a soft cross-entropy loss optimization model is introduced, and the above process is expressed as follows:

[0034]

[0035] The Sigmoid function is used to adjust the pseudo-labels to 0-1, and τ is set to 1 or 2 to adapt to WSI datasets with different tumor sizes.

[0036] The DDM module is used to predict packet labels, first using... and To refine and The weights depend on the predicted attributes, and are then passed through a linear layer and a softmax activation function to obtain precise explicit weights. and implicit weights Final integration and To obtain the final package label prediction The process is described as follows:

[0037]

[0038] A pathological image classification method based on online pseudo-supervision and dynamic mutual learning includes the following steps:

[0039] Step 1: Given a whole slice dataset of pathological images It contains N WSI X i and its corresponding label Y i X i Treating it as the i-th "packet", the non-background tissue region of WSI is extracted using a threshold, and B is cropped. i An example Where H and W represent the height and width of an instance, b∈[1,B] i ];

[0040] Step 2: Use a pre-trained deep neural network as the encoder to input x. i,b Embedded into a D-dimensional feature vector middle;

[0041] Step 3: At time point t, from package X i Random sampling pseudo-packet M represents the number of instance feature vectors contained in the sampled pseudo-packet, and T represents the input image X. i The number of times a pseudo-packet of size M is dynamically and interactively sampled is performed in order to generate information-rich pseudo-packet tags. The representation of;

[0042] Step 4: Pseudo-packet marking The additional valuable information from the AMB and the refined feature representation from the MLFF module are aggregated into a new label using a memory-based cross-attention MCA module. In order to promote Pseudo-label prediction

[0043] Step 5: Predict the probability of fake packet labels AMB helps capture and save the K most relevant tags. The pseudo-packet label at time t+1 Provide supplementary knowledge;

[0044] Step Six: To provide credible pseudo-labels Supervised prediction of pseudo-package labels Explore pseudo-packet tags using the EIRL module With package label prediction ( and The inherent relationships between these elements are used to construct the DRPL generation module;

[0045] Step 7: The DDM module takes explicit and implicit packet predictions and representations as inputs and adaptively weights them. and To achieve more robust package decision-making

[0046] Step 8: Optimize by coordinating the cross-entropy loss of the package label and pseudo-label to promote better feature representation and pseudo-label generation, thereby improving classification performance.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] 1. This invention proposes to promote feature representation and pseudo-label generation, and further explore their mutual learning, in order to build an online pseudo-supervised and dynamic mutual learning multi-instance learning framework to achieve better and more reliable bag decision-making.

[0049] 2. To achieve discriminative and compact representations, this invention designs an elegant adaptive repository (AMB) to collect the most informative components of the current WSI, and further develops a self-improving feature fusion (SPFF) module to integrate label-related historical information and real-time semantic changes in the AMB to enhance the current pseudo-packet label representation. The ideas behind SPFF include: 1) Pseudo-label prediction provides category-level insights, helping the AMB capture salient labels; 2) The AMB, combined with memory-based cross-attention (MCA) and MLFF schemes, provides refined features to facilitate the current label; 3) Fine-grained labeling facilitates pseudo-label prediction and updates the AMB.

[0050] 3. To generate reliable pseudo-labels, this invention designs a Decision Correction Pseudo-label Generation Module (DRPL) to explore the intrinsic relationship between pseudo-packet representation and packet label prediction, thereby generating more reliable pseudo-labels. Specifically, this invention proposes an EIRL strategy, using a Linear Attention Module (LAM) and a Multi-Head Self-Attention Module (MSA) to study the complex relationship between pseudo-packet labels and their contribution to packet label inference. Since attention scores are inherently noisy, this invention uses packet label probabilities to further calibrate the scores to generate more reliable pseudo-labels.

[0051] 4. To alleviate redundant and ambiguous representations, this invention borrows the category prior of pseudo-label prediction to promote the learning of label-related features and update AMB, thereby forming a dynamic mutual learning and optimization between feature representation and pseudo-label generation.

[0052] 5. This invention develops a dynamic decision-making (DDM) module to coordinate the explicit and implicit representations of packet information, thereby achieving more robust decision-making.

[0053] 6. Extensive experiments on two datasets show that the OODML framework of this invention outperforms the state-of-the-art BCL method by 4.7% and 6.7% on the CAMELYON16 and TCGA lung cancer datasets, respectively. Attached Figure Description

[0054] Figure 1 A flowchart for pathological image classification based on online pseudo-supervision and dynamic mutual learning;

[0055] Figure 2 It is a cross-attention module based on a memory bank;

[0056] Figure 3 This is a dynamic decision-making module;

[0057] Figure 4 The module for generating pseudo-labels is used to correct decisions. Detailed Implementation

[0058] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0059] This invention provides a pathological image classification system based on online pseudo-supervision and dynamic mutual learning, such as... Figure 1 As shown, the system includes a tissue pathology image extraction instance and instance feature extraction module, an SPFF module, an EIRL module, a DRPL generation module, and a DDM module.

[0060] In this invention, the histopathological image extraction instance and instance feature extraction module is responsible for histopathological image instance extraction and instance feature encoding, wherein:

[0061] The specific steps for extracting the histopathological image instances are as follows: Given a whole slice dataset of pathological images It contains N WSI X i and its corresponding label Y i X i Consider the i-th "packet"; the full-slice image WSI is read into the computer at downsampling resolution and converted from RGB to HSV color space; the non-background tissue region of WSI is calculated based on the saturation and channel thresholds of the image after median blurring, and morphological closure operations are combined to fill small gaps and holes as the final tissue region; based on the calculated tissue region, for each WSI X i Crop B at a specified magnification. i Image patch instance And store, where H and W represent the height and width of an instance, b∈[1,B]i ].

[0062] The specific steps for encoding the instance features are as follows: Using a deep neural network pre-trained on ImageNet or a large pathological image dataset as the encoder, firstly, WSI X... i A collection of clipping instances Then take each instance x i,v Embedded into a D-dimensional feature vector In this process, a set of low-dimensional feature vectors is obtained.

[0063] In this invention, to better represent features, the SPFF module includes a Transformer module, an Adaptive Memory Bank (AMB), a Multi-Level Feature Fusion (MLFF) module, a memory-based cross-attention (MCA) module, and a pseudo-label prediction module. Specifically, to obtain discriminative and compact feature representations, the SPFF module fuses historical information and real-time semantic information from the AMB to enhance the current pseudo-bag token representation. The underlying concepts of SPFF include: 1) the predicted pseudo-label probabilities provide category insights, helping the AMB capture significant tokens; 2) the AMB integrates memory-based cross-attention (MCA) and multi-level feature fusion (MLFF) schemes to provide refined features to improve the current token representation; and 3) finer-grained tokens improve pseudo-label prediction accuracy and update the AMB.

[0064] Transformer module: At time t, randomly sample a set of instances. The corresponding set of feature vectors M represents the number of instance feature vectors contained in the sampled pseudo-packet, and T represents the input image X. i The number of times a pseudo-packet of size M is executed in a dynamic interactive sampling method. This represents the j-th instance. It is the j-th instance. The corresponding feature vector, for the sampled Design a token position vector Come and learn about fake packages Spatial location information; randomly initialize a token vector. Combined with the Transformer Module (TRM) extract Local context information of instance features is entered into the Token In the Transformer module It consists of a self-attention module equipped with 8 heads and residual connections. The above process is described as follows:

[0065]

[0066] AMB: To achieve a compact and discriminative representation, this invention proposes AMB to store the most relevant feature representations of the labels by fully utilizing pseudo-label prediction priors. The AMB symbol representation at time t is as follows: The AMB symbol at time t-1 is represented as AMB predicts the enhanced pseudo-packet token at time t-1. and the corresponding predicted pseudo-label probability As input, rearrange To store the key K tokens and their corresponding pseudo-label probabilities:

[0067]

[0068] In the formula, This represents the k-th token stored. express The corresponding pseudo-label probability.

[0069] MLFF Module: To extract decision-friendly components, the MLFF proposed in this invention includes a two-stage selection process and a feature fusion operation. MLFF uses... As a semantically rich hint for categories, to store in the dictionary Refined to include more tag-related features To further highlight key contributions, the Softmax function is used to... The features in the data are normalized to 0-1 for reinforcement. The most prominent feature. Then Quantization as the best label To obtain significance weights Subsequently, precise Used to extract dispersed discriminative expressions from and Aggregation to more discriminative features These operations can be expressed as:

[0070]

[0071] MCA: To facilitate pseudo-packet tag prediction Fake Packet Token Hire MCA to integrate key knowledge from adaptive memory and multi-level feature fusion modules, such as Figure 2 As shown. Specifically, for the features in AMB Features refined from the MLFF module Perform a concatenation operation and map it as (key, key, abbreviated as K)K and (value, value, abbreviated as K)V. Defined as (Query, abbreviated as Q), it is achieved through the following formula. Extracting saliency information from K and V enhances the current pseudo-packet token feature representation.

[0072]

[0073] Pseudo-label prediction module: The MCA module outputs a more comprehensive pseudo-packet token feature representation. It is passed into a multilayer perceptron constructed from a linear layer and a softmax activation function layer. Used to predict Corresponding pseudo-label probability The process is described as follows:

[0074]

[0075] In this invention, to facilitate the generation of pseudo-tags, an EIRL module is introduced to explore pseudo-packet tags. With package and label prediction The explicit relation (ER) and implicit relation (IR) between them, then, and The input is transferred to two MLP classifiers consisting of linear layers and a Softmax activation function. and Used respectively for predicting packet labels and Specifically, to generate reliable pseudo-labels, this invention proposes a Decision Correction Pseudo-Label (DRPL) generation scheme to explore the potential relationship between pseudo-packet representations and packet label predictions. More specifically, an Explicit-Implicit Representation Learning (EIRL) strategy is proposed, which investigates the complex relationships between pseudo-packet tokens and their contribution to packet label prediction through a Linear Attention (LAM) module and a Multi-Head Self-Attention (MSA) module. Since the attention score itself is noisy, the score is further calibrated using the packet label probability to generate reliable pseudo-labels. The EIRL module includes a Linear Attention (LAM) module, a Multi-Head Self-Attention (MSA) module, an explicit packet label prediction module, and an implicit packet label prediction module, wherein:

[0076] LAM Module: This invention uses a LAM module to explore pseudo-packet tags. With explicit package feature expression The single connection between them is represented by linear attention scores extracted from LAM. Express. First, attention weights are obtained by passing the material through a linear layer and a softmax activation function. Then use right Perform weighted summation to obtain explicit package feature representation. The above process can be represented as follows:

[0077]

[0078] In the formula, (*) represents the weight and vector weighted summation operation.

[0079] MSA Module: The MSA module is equipped with an initialization class marker. (Class Token, or cls for short) to explore fake packet markers With implicit bag feature representation Multiple connections and cross relationships between them are analyzed using multi-head attention scores extracted from MSA. This indicates that, specifically, the first step is to assemble... For a feature map H i After a linear transformation, the data is transformed into Q, K, and V, and then segmented into multi-head representations. Attention scores are calculated for Q and K, multiplied by V to extract features, and then fused into Q. This process is described as follows:

[0080]

[0081]

[0082] Among them, W q W k Wv W o It is a linear transformation matrix. The query, key, and value are split into a multi-head representation, where Head represents the number of multi-heads, and d k Represents the dimension of each head, and (*) indicates the weight and vector weighted summation operation.

[0083] Explicit Packet Label Prediction Module and Implicit Packet Label Prediction Module: Explicit Packet Feature Representations Output by LAM and MSA Modules and implicit bag feature representation They are respectively fed into a multilayer perceptron constructed from a linear layer and a softmax activation function layer. and and Used to predict and Packet tag probability and The process is described as follows:

[0084]

[0085] In this invention, to further improve the robustness of packet decision-making, a Dynamic Decision Making (DDM) module is designed to predict packet labels. Specifically, firstly, a... and To refine and The weights depend on the predicted attributes. They are then passed through a linear layer and a softmax activation function to obtain precise explicit weights. and implicit weights Finally, integration and To obtain the final package label prediction like Figure 3 As shown:

[0086]

[0087] In this invention, to generate reliable pseudo-labels, a DRPL generation module is proposed to explore the potential relationship between pseudo-bag characterization and bag label prediction, providing a basis for the generation of reliable pseudo-labels for genuine bags X. i In Generate pseudo tags otherwise, It will be set to a negative label of 0. For example... Figure 4Specifically, this invention employs the LAM and MSA modules from the EIRL module to investigate the complex relationships between pseudo-bag tags and their contribution to bag label inference. Since the attention score itself is noisy, this invention uses bag label probabilities to further calibrate the score to generate more reliable pseudo-tags. Used to supervise the prediction probability of pseudo-labels Furthermore, a soft cross-entropy loss optimization model is introduced. The above process can be expressed as:

[0088]

[0089] The Sigmoid function is used to adjust the pseudo-labels to 0-1, and τ is set to 1 or 2 to adapt to WSI datasets with different tumor sizes.

[0090] In this invention, in order to achieve better feature representation and pseudo-label generation and promote packet decision-making, three packet label probabilities ( and ) and pseudo-label prediction The actual composite constraint involves two packet cross-entropy losses. and DDM Cross-Entropy Loss and pseudo-label cross-entropy loss (PLCE) The formulas for these loss functions are as follows:

[0091]

[0092] The final total loss is defined as:

[0093]

[0094] Where λ PLCE This is used to adjust the proportion of pseudo-label loss in the total loss. By integrating the duration and cosine curve, it is adjusted from 0 to 0.5 to guide the model from coarse to finer information exploration.

[0095] This invention provides a pathological image classification system method based on online pseudo-supervision and dynamic mutual learning, such as... Figure 1 As shown, the method includes the following steps:

[0096] Step 1: Given a whole slice dataset of pathological images It contains N WSI X i and its corresponding label Y i X i Treating it as the i-th "packet", the non-background tissue region of WSI is extracted using a threshold, and B is cropped. i An example Where H and W represent the height and width of an instance, b∈[1,B]i ].

[0097] Step 2: Use a pre-trained deep neural network as the encoder to input x. i,b Embedded into a D-dimensional feature vector middle.

[0098] Step 3: To mitigate the problems of incomplete information for a single instance and redundant interference across all instances, at time point t, from package X... i Random sampling pseudo-packet M represents the number of instance feature vectors contained in the sampled pseudo-packet, and T represents the input image X. i Dynamic interactive sampling is used to count the number of times a pseudo-packet of size M is executed. Compared to fixed grouping or Top-K instance strategies, dynamic sampling helps OODML become more aware of... The semantic and spatial features of the data are used to generate information-rich pseudo-bag tokens. The expression .

[0099] Step 4: To enhance feature representation, pseudo-packet labeling. Equipped with an SPFF module, it aggregates the additional valuable information from the AMB and the refined feature representation from the MLFF module into a new label. In order to promote Pseudo-label prediction

[0100] Step 5: Predict the probability of fake packet labels AMB helps capture and save the K most relevant tags. The pseudo-packet label at time t+1 Provide supplementary knowledge.

[0101] Step Six: To provide credible pseudo-labels Supervised prediction of pseudo-package labels Introducing the EIRL module to explore pseudo-packet tags With package label prediction ( and The inherent relationship between ) is used to construct the DRPL generation module.

[0102] Step 7: The DDM module takes explicit and implicit packet predictions and representations as inputs and adaptively weights them. and To achieve more robust package decision-making

[0103] Step 8: Optimize OODML by coordinating the cross-entropy loss of package labels and pseudo-labels to promote better feature representation and pseudo-label generation, thereby improving classification performance.

[0104] To validate the OODML of this invention, extensive experiments were conducted on the CAMELYON16 and TCGA lung cancer datasets, comparing it with typical representative WSI analysis MIL methods. These methods included: 1) bag label level (MeanPooling, ABMIL, SetTransformer, DeepAttnMIL, TransMIL, MuRCL, MHIM, and SSMMIL); 2) pseudo-label level (MaxPooling, RNNMIL, DSMIL, CLAM, DTFD, DGMIL, IAT, BCL, and CIMIL). The area under the curve (AUC), accuracy, and F1 score were used for evaluation, with a threshold of 0.5. The OODML was trained for 100 iterations using the AdaMax optimizer with an initial learning rate of 0.0004.

[0105] As can be seen from Table 1, the OODML method proposed in this invention is competitive in all metrics, and surpasses the BCL method based on pseudo-labels and the (SSMMIL and TransMIL) method based on packet labels by 4.7% and 3.6% in accuracy, respectively.

[0106] Table 1 Comparison of Model Training Results

[0107]

[0108]

Claims

1. A pathological image classification system based on online pseudo-supervision and dynamic mutual learning, characterized in that... The system includes a histopathological image extraction instance and instance feature extraction module, a self-improving feature fusion (SPFF) module, an explicit and implicit representation learning (EIRL) module, a decision correction pseudo-label (DRPL) generation module, and a dynamic decision (DDM) module, wherein: The tissue pathology image extraction instance and instance feature extraction module is responsible for extracting tissue pathology image instances and encoding instance features. The SPFF module includes a Transformer module, an Adaptive Memory Bank (AMB), a Multi-Level Feature Fusion (MLFF) module, a Memory Bank-Based Cross-Attention (MCA) module, and a pseudo-label prediction module, wherein: The Transformer module It consists of a multi-head self-attention module and residual connections, and at time t, a set of instances is randomly sampled. The corresponding set of feature vectors M represents the number of instance feature vectors contained in the sampled pseudo-packet. Indicates the input image The number of times a pseudo-packet of size M is executed in a dynamic interactive sampling method. Indicates the first One example, It is the first An example The corresponding feature vector, for the sampled Design a token location vector Come and learn about fake packages Spatial location information; randomly initialize a token vector. Combine with Transformer module extract Local context information of instance features is entered into the Token In the above process, it is described as follows: ； The AMB is used to store the most relevant feature representations of the tag. The AMB symbol representation at time t is as follows: The AMB symbol at time t-1 is represented as AMB predicts the enhanced pseudo-packet token at time t-1. and the corresponding predicted pseudo-label probability As input, rearrange To store the key K tokens and their corresponding pseudo-label probabilities: In the formula, This represents the k-th token stored. express The corresponding pseudo-label probability; The MLFF module uses As a semantically rich hint for categories, to store tokens in the dictionary Refined into a token containing more tag-related features To further highlight key contributions, the Softmax function is used to... The features in the data are normalized to 0-1 for reinforcement. The most prominent feature, then Quantization as the best label To obtain significance weights Subsequently, Used to extract dispersed discriminative expressions from and Aggregation to more discriminative features These operations are expressed as: The MCA module provides features in the AMB. Features refined from the MLFF module Perform the concatenation operation and map it to a Key and a Value. Defined as a Query, it is achieved through the following formula. Extracting saliency information from K and V enhances the current pseudo-packet token feature representation. : The pseudo-label prediction module is a multilayer perceptron constructed from a linear layer and a softmax activation function layer. , Used for prediction Corresponding pseudo-label probability The process is described as follows: The EIRL module includes a linear attention (LAM) module, a multi-head self-attention (MSA) module, a decision correction pseudo-label (DRPL) generation module, an explicit packet label prediction module, and an implicit packet label prediction module, wherein: The LAM module is used to detect fake packet markers. With explicit package feature expression The single connection between them is represented by linear attention scores. Expression, specifically First, attention weights are obtained by passing the material through a linear layer and a softmax activation function. Then use right Perform weighted summation to obtain explicit package feature representation. The above process can be represented as follows: In the formula, ( () represents the weighted summation operation of the weights and vectors; The MSA module is equipped with an initialization class marker. Explore pseudo-packet tags With implicit bag feature representation Multiple connections and cross relationships between them, using multi-head attention scores Specifically, this means first assembling... For a feature map The data is linearly transformed into Query, Key, and Value, and then segmented into multi-head representations. Attention scores for Query and Key are calculated and multiplied by Value to extract features, which are then fused into Query. This process is described as follows: ； ； in, , , , It is a linear transformation matrix. The query, key, and value are separated into a multi-head representation, where the head indicates the number of multi-heads. Represents the dimension of each head, ( () represents the weighted summation operation of the weights and vectors; The explicit packet label prediction module and the implicit packet label prediction module are multilayer perceptrons constructed from a linear layer and a Softmax activation function layer. and Explicit packet feature representations output by the LAM and MSA modules and implicit bag feature representation They were passed in separately and , and Used to predict and Packet tag probability and The process is described as follows: The DRPL generation module is used to explore the potential relationship between counterfeit bag characterization and bag label prediction for genuine bags. In Generate pseudo tags ,otherwise, It will be set to a negative label of 0; The DDM module is used to predict packet labels, first using... and To refine and The weights depend on the predicted attributes, and are then passed through a linear layer and a softmax activation function to obtain precise explicit weights. and implicit weights Finally, integration and To obtain the final package label prediction. The process is described as follows: 。 2. The pathological image classification system based on online pseudo-supervision and dynamic mutual learning according to claim 1, characterized in that... The specific steps for extracting the histopathological image instances are as follows: Given a whole slice dataset of pathological images It contains N WSIs and its corresponding tags , Considered as the first Each "package"; the full-slice image WSI is read into the computer at downsampling resolution and converted from RGB to HSV color space; The non-background tissue region of WSI is calculated based on the image saturation and channel threshold after median blurring, and the morphological closure operation is combined to fill small gaps and holes as the final tissue region. For each WSI, based on the calculated tissue region Crop at a specified magnification An example And store, where H and W represent the height and width of an instance, .

3. The pathological image classification system based on online pseudo-supervision and dynamic mutual learning according to claim 2, characterized in that... The specific steps for encoding the instance features are as follows: Using a deep neural network pre-trained on ImageNet or a large pathological image dataset as the encoder, firstly, WSI... A collection of clipping instances Then put each instance Embedded into a D-dimensional feature vector In this process, a set of low-dimensional feature vectors is obtained. .

4. The pathological image classification system based on online pseudo-supervision and dynamic mutual learning according to claim 1, characterized in that... The DRPL generation module employs the LAM and MSA modules from the EIRL module to study the complex relationships between counterfeit bag tags and their contribution to bag label inference, using bag label probabilities to further calibrate scores to generate more reliable counterfeit labels. Used to supervise the prediction probability of pseudo-labels Furthermore, a soft cross-entropy loss optimization model is introduced, and the above process is expressed as follows: The Sigmoid function is used to adjust the pseudo-labels to a value between 0 and 1. Set to 1 or 2 to accommodate WSI datasets with different tumor sizes.

5. The pathological image classification system based on online pseudo-supervision and dynamic mutual learning according to claim 1, characterized in that... The total loss of the system is defined as: in, , It is used to adjust the proportion of false label loss in the total loss.

6. A method for classifying pathological images based on online pseudo-supervision and dynamic mutual learning using the system described in any one of claims 1-5, characterized in that... The method includes the following steps: Step 1: Given a whole slice dataset of pathological images It contains N WSIs and its corresponding tags , Considered as the first Each "package" is used to extract non-background tissue regions of WSI using a threshold, and then cropped. An example Where H and W represent the height and width of an instance, ; Step 2: Use a pre-trained deep neural network as the encoder to... Embedded into a D-dimensional feature vector middle; Step 3: At time point t, from the package Random sampling pseudo-packet M represents the number of instance feature vectors contained in the sampled pseudo-packet. Indicates the input image The number of times a pseudo-packet of size M is dynamically and interactively sampled is performed in order to generate information-rich pseudo-packet tags. The representation of; Step 4: Pseudo-packet marking The additional valuable information from the AMB and the refined feature representation from the MLFF module are aggregated into a new label using a memory-based cross-attention MCA module. In order to promote Pseudo-label prediction ; Step 5: Predict the probability of fake packet labels AMB helps capture and save the K most relevant tags. The pseudo-packet label at time t+1 Provide supplementary knowledge; Step Six: To provide credible pseudo-labels Supervised prediction of pseudo-package labels Explore pseudo-packet markers using the EIRL module With package label prediction ( and The inherent relationships between these elements are used to construct the DRPL generation module; Step 7: The DDM module takes explicit and implicit packet predictions and representations as inputs and adaptively weights them. and To achieve more robust decision-making. ; Step 8: Optimize by coordinating the cross-entropy loss of the package label and pseudo-label to promote better feature representation and pseudo-label generation, thereby improving classification performance.

Images