A whole slide pathological image classification method and device, electronic equipment and medium
By constructing multi-granularity sample representations and performing neighborhood alignment in the RKHS space, the problems of overfitting and poor generalization in pathological image classification are solved, achieving efficient identification of cancerous regions in WSI images and improving the efficiency of pathological image diagnostic analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PEKING UNION MEDICAL COLLEGE HOSPITAL
- Filing Date
- 2026-04-15
- Publication Date
- 2026-08-04
AI Technical Summary
Because public histopathological images and WSI datasets are relatively scarce, deep learning models suffer from overfitting and poor generalization in pathological image classification, making it difficult to effectively identify cancerous regions in WSI.
By acquiring labeled source domain slice sets and unlabeled target domain slice sets, features are extracted using deep convolutional networks and SwingTransformer to construct multi-granularity sample representations. Multi-granularity hierarchical domain alignment is then performed in the RKHS space to optimize the pathological image classification model and generate a lesion identification prediction heatmap.
It improves the accuracy of pathological image classification and recognition, effectively identifies cancerous areas in WSI, enhances diagnostic analysis efficiency, and assists pathologists in annotating cancerous areas.
Smart Images
Figure CN122510620A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image processing technology, and more specifically, to a method, apparatus, electronic device, and medium for classifying pathological images on a whole slide. Background Technology
[0002] With the widespread application of deep learning (DL), deep learning technology has made significant progress in the field of medical image processing. However, due to privacy concerns regarding data sources, publicly available histopathological images and WSI datasets are relatively scarce. Furthermore, small-sized pathological images are relatively easy to annotate precisely, while precise annotation of WSI images requires manual annotation by experienced pathologists, which is time-consuming and labor-intensive. The limited number of annotated datasets cannot meet the data volume requirements of deep neural networks for pathological images, and direct training easily leads to overfitting and poor generalization, thus limiting the performance of classification and recognition models. Summary of the Invention
[0003] In view of this, the purpose of this application is to provide a method, device, electronic device and medium for classifying whole slide pathological images, which can transfer useful information from small-sized histopathological images with relatively easy-to-obtain labels to unlabeled WSI pathological images, realize the identification of cancerous areas of WSI, and improve the recognition accuracy of pathological image classification and recognition models.
[0004] This application provides a method for classifying whole-slide pathological images, the method comprising: A whole-slide pathological image of the target to be identified is acquired, and the whole-slide pathological image of the target is sliced to obtain multiple target slices; The multiple target slices are input into a trained pathological image classification model to determine the predicted category probability of each target slice. The pathological image classification model is trained through the following steps: obtaining a labeled source domain slice set and an unlabeled target domain slice set; extracting sample fusion features from the source domain slice set and the target domain slice set using the feature extraction and fusion module of the pathological image classification model; and constructing a multi-granularity sample representation based on the sample fusion features of the sample slices, including an original sample layer, a granular envelope sample layer, and a class center sample layer. Using the labels of the source domain slice set as conditions, multi-granularity hierarchical neighborhood alignment of the source and target domains is performed in the RKHS space based on the multi-granularity sample representation, and various alignment losses are determined; based on the various alignment losses and the loss of the classifier of the pathological image classification model, the pathological image classification model is optimized to obtain a trained pathological image classification model. Based on the positions of the multiple target slices in the whole slide pathological image, the predicted probabilities of the categories corresponding to each target slice are mapped and synthesized to generate a lesion identification prediction heatmap of the whole slide pathological image.
[0005] In some embodiments, the whole-slide pathological image classification method, the step of obtaining a labeled source domain slice set and an unlabeled target domain slice set includes: Acquire multiple labeled sample pathological images to construct the source domain slice set; Acquire at least one unlabeled whole slide pathology image, and segment the unlabeled whole slide pathology image into multiple sample slices that include effective tissue regions; Randomly select at least a portion of the sample slices that include the effective tissue region to construct the target domain slice set.
[0006] In some embodiments, in the whole-slide pathological image classification method, the feature extraction and fusion module of the pathological image classification model extracts sample fusion features of sample slices from the source domain slice set and the target domain slice set, respectively, including: Local detail features of sample slices in the source domain slice set and the target domain slice set are extracted using deep convolutional networks. Global semantic features of sample slices in the source domain slice set and the target domain slice set are extracted using SwinTransformer. Pooling operations are performed on the local detail features and global semantic features of the sample slices respectively to obtain local detail features and global semantic features with uniform channel dimensions; The local detail features and global semantic features of the channel dimension are weighted and fused to obtain the sample fusion features of the sample slice.
[0007] In some embodiments, the whole-slide pathological image classification method, the construction of a multi-granularity sample representation based on the sample fusion features of the sample slices includes an original sample layer, a particle envelope sample layer, and a class center sample layer, comprising: The sample fusion features of each sample slice in the source domain and the target domain are used as the sample representation of the original sample layer, respectively. For the sample fusion features of each sample slice in the source domain and the target domain, sample slices with similar sample fusion features are clustered into multiple particle envelopes by iterative splitting and purity judgment. The statistical measure of the sample fusion features in each particle envelope is used as the sample representation of the particle envelope sample layer. The mean of the sample fusion features of all sample slices belonging to the same category in the source domain and the target domain are calculated respectively, and the mean is used as the sample representation of the class center sample layer of that category.
[0008] In some embodiments, in the whole-slide pathological image classification method, the step of performing multi-granularity hierarchical neighborhood alignment of the source and target domains in the RKHS space based on the labels of the source domain slice set and the multi-granularity sample representation, and determining various alignment losses, including: In the original sample layer, the global feature distributions of the source and target domains are aligned based on the maximum mean difference of the first-order statistic, and the distribution matching loss is calculated. In the class center sample layer, the source and target domain data distributions conditioned on labels are finely adapted based on the Wasserstein distance of the second-order covariance matrix, and the conditional neighborhood alignment loss is calculated. In the particle envelope sample layer and / or the original sample layer, based on cross-domain contrastive learning, the representation distance of similar samples is narrowed and the representation distance of dissimilar samples is widened, thereby enhancing the semantic discriminativeness of features, and the semantic contrastive loss is calculated.
[0009] In some embodiments, in the whole-slide pathological image classification method, the step of calculating a first alignment loss based on the difference of the conditional covariance operator with the label as the conditional variable at the class center sample layer includes: The sample representations and corresponding labels of the class center sample layers of the source domain and the target domain are mapped to the RKHS space through a kernel function; where the labels of the target domain are pseudo-labels. In the RKHS, the conditional covariance operators for the source domain and the target domain are calculated respectively; The distance between the conditional covariance operators of the source domain and the target domain is calculated and used as the first alignment loss.
[0010] In some embodiments, in the whole-slide pathological image classification method, optimizing the pathological image classification model based on the classifier loss of the multiple alignment losses and the pathological image classification model includes: The various alignment losses and the classifier losses are weighted and fused to determine the total loss function of the pathological image classification model; The pathological image classification model is optimized based on the total loss function.
[0011] In some embodiments, a whole-slide pathological image classification device is also provided, the device comprising: The acquisition module is used to acquire the target whole slide pathological image to be identified, and to perform slicing processing on the target whole slide pathological image to obtain multiple target slices; A determination module is used to input the multiple target slices into a trained pathological image classification model to determine the predicted category probability of each target slice. The pathological image classification model is trained through the following steps: obtaining a labeled source domain slice set and an unlabeled target domain slice set; extracting sample fusion features from the sample slices in the source domain slice set and the target domain slice set using the feature extraction and fusion module of the pathological image classification model; constructing a multi-granularity sample representation including an original sample layer, a granular envelope sample layer, and a class center sample layer based on the sample fusion features of the sample slices; performing multi-granularity hierarchical neighborhood alignment of the source domain and target domain in the RKHS space based on the multi-granularity sample representation, using the labels of the source domain slice set as conditions, and determining various alignment losses; optimizing the pathological image classification model based on the various alignment losses and the classifier loss of the pathological image classification model to obtain a trained pathological image classification model. The generation module is used to map and synthesize the predicted probability of the category corresponding to each target slice according to the position of the multiple target slices in the whole slide pathological image, and generate a lesion identification prediction heatmap of the whole slide pathological image.
[0012] In some embodiments, an electronic device is also provided, including a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the whole slide pathological image classification method are performed.
[0013] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the whole slide pathological image classification method.
[0014] This application provides a method, apparatus, electronic device, and medium for classifying whole-slide pathological images. The method involves acquiring a target whole-slide pathological image to be identified, and performing slice processing on the target whole-slide pathological image to obtain multiple target slices. These multiple target slices are then input into a trained pathological image classification model to determine the predicted category probability of each target slice. The pathological image classification model is trained through the following steps: acquiring a labeled source domain slice set and an unlabeled target domain slice set; extracting sample fusion features from the source domain slice set and the target domain slice set using the feature extraction and fusion module of the pathological image classification model; constructing a multi-granularity sample representation based on the sample fusion features of the sample slices, including an original sample layer, a granular envelope sample layer, and a class center sample layer; and performing multi-granularity hierarchical neighborhood alignment of the source and target domains in the RKHS space based on the labels of the source domain slice set and the multi-granularity sample representation. Multiple alignment losses are determined; based on the multiple alignment losses and the loss of the classifier in the pathological image classification model, the pathological image classification model is optimized to obtain a trained pathological image classification model; according to the position of the multiple target slices in the whole slide pathological image, the predicted probability of the category corresponding to each target slice is mapped and synthesized to generate a lesion identification prediction heatmap of the whole slide pathological image; global and local features are extracted and fused to characterize the fine-grained edge information and coarse-grained semantic information of the pathological image, and the semantic information between the original sample and the category center is mined through multi-granular sample characterization; through multi-dimensional alignment between the source domain and the target domain, the cancer-specific domain invariant representation is further mined to guide the lesion identification prediction of the unlabeled target domain WSI, thereby improving the final effect of cancer region identification prediction in the WSI image. The visualization of the prediction results can assist pathologists in annotating the cancer regions of the pathological image, greatly improving the efficiency of diagnosis and analysis of pathological images. Attached Figure Description
[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 A flowchart of the whole-slide pathological image classification method described in an embodiment of this application is shown; Figure 2 This diagram illustrates the overall framework of the whole-slide pathological image classification method described in the application embodiment; Figure 3A flowchart of the method for generating multi-granularity sample characterization according to an embodiment of this application is shown; Figure 4 A flowchart illustrating the method for determining various alignment losses according to an embodiment of this application is shown; Figure 5 A schematic diagram of the structure of the whole slide pathology image classification device described in an embodiment of this application is shown; Figure 6 A schematic diagram of the structure of the electronic device described in an embodiment of this application is shown. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.
[0018] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0019] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.
[0020] With the widespread application of deep learning (DL), deep learning technology has made significant progress in the field of medical image processing. However, due to privacy concerns regarding data sources, publicly available histopathological images and WSI datasets are relatively scarce. Furthermore, small-sized pathological images are relatively easy to annotate precisely, while precise annotation of WSI images requires manual annotation by experienced pathologists, which is time-consuming and labor-intensive. The limited number of annotated datasets cannot meet the data volume requirements of deep neural networks for pathological images, and direct training easily leads to overfitting and poor generalization, thus limiting the performance of classification and recognition models.
[0021] Based on this, this application provides a method, apparatus, electronic device, and medium for classifying whole-slide pathological images. The method involves acquiring a target whole-slide pathological image to be identified, and performing slice processing on the target whole-slide pathological image to obtain multiple target slices. The multiple target slices are then input into a trained pathological image classification model to determine the predicted category probability of each target slice. The pathological image classification model is trained through the following steps: acquiring a labeled source domain slice set and an unlabeled target domain slice set; extracting sample fusion features from the source domain slice set and the target domain slice set using the feature extraction and fusion module of the pathological image classification model; constructing a multi-granularity sample representation including an original sample layer, a particle envelope sample layer, and a class center sample layer based on the sample fusion features of the sample slices; and performing multi-granularity hierarchical neighborhood pairing of the source domain and target domain in the RKHS space based on the multi-granularity sample representation, using the labels of the source domain slice set as conditions. The process involves determining multiple alignment losses; optimizing the pathological image classification model based on these alignment losses and the classifier loss of the pathological image classification model to obtain a trained pathological image classification model; mapping and synthesizing the predicted probability of each target slice according to its position in the whole slide pathological image to generate a lesion identification prediction heatmap of the whole slide pathological image; extracting and fusing global and local features to characterize the fine-grained edge information and coarse-grained semantic information of the pathological image; mining the semantic information between the original sample and the category center through multi-granularity sample characterization; further mining the cancer-specific domain invariant representation through multi-dimensional alignment between the source domain and the target domain to guide the lesion identification prediction of the unlabeled target domain WSI, thereby improving the final effect of cancer region identification prediction in the WSI image; and visualizing the prediction results to assist pathologists in labeling cancer regions in pathological images, greatly improving the efficiency of diagnostic analysis of pathological images.
[0022] Please refer to Figure 1 , Figure 1 A flowchart of the whole-slide pathological image classification method described in the embodiments of this application is shown; as follows: Figure 1As shown, the whole-slide pathological image classification method includes the following steps S101-S103: S101. Obtain the target whole slide pathological image to be identified, and perform slicing processing on the target whole slide pathological image to obtain multiple target slices. S102. Input the multiple target slices into the trained pathological image classification model to determine the predicted category probability of each target slice; wherein, the pathological image classification model is trained through the following steps: obtaining a labeled source domain slice set and an unlabeled target domain slice set; extracting sample fusion features of sample slices in the source domain slice set and the target domain slice set respectively through the feature extraction and fusion module of the pathological image classification model; constructing a multi-granularity sample representation including an original sample layer, a granular envelope sample layer, and a class center sample layer based on the sample fusion features of the sample slices; performing multi-granularity hierarchical neighborhood alignment of the source domain and target domain in the RKHS space based on the multi-granularity sample representation, using the labels of the source domain slice set as conditions, and determining multiple alignment losses; optimizing the pathological image classification model based on the multiple alignment losses and the classifier loss of the pathological image classification model to obtain the trained pathological image classification model; S103. Based on the positions of the multiple target slices in the whole slide pathological image, the predicted probability of the category corresponding to each target slice is mapped and synthesized to generate a lesion identification prediction heatmap of the whole slide pathological image.
[0023] In step S101, a whole-slide pathological image of the target to be identified is obtained, and the whole-slide pathological image of the target is sliced to obtain multiple target slices.
[0024] Specifically, the target whole-slide pathological image is a WSI image. WSI (Whole Slide Imaging), also known as virtual slide, is a technique that uses a dedicated scanner to scan and stitch together complete pathological glass slides (including tissue / cell samples) into a high-resolution digital image.
[0025] The target whole slide pathological image is sliced, that is, the target whole slide pathological image is divided into slices containing only effective pathological tissue and cells.
[0026] For example, the size of the switch is 224×224.
[0027] In step S102, the multiple target slices are input into a trained pathological image classification model to determine the predicted category probability of each target slice. The pathological image classification model is trained through the following steps: obtaining a labeled source domain slice set and an unlabeled target domain slice set; extracting sample fusion features from the source domain slice set and the target domain slice set using the feature extraction and fusion module of the pathological image classification model; constructing a multi-granularity sample representation based on the sample fusion features of the sample slices, including an original sample layer, a granular envelope sample layer, and a class center sample layer; using the labels of the source domain slice set as conditions, performing multi-granularity hierarchical neighborhood alignment of the source and target domains in the RKHS space based on the multi-granularity sample representation, and determining various alignment losses; optimizing the pathological image classification model based on the various alignment losses and the classifier loss of the pathological image classification model to obtain a trained pathological image classification model.
[0028] Please refer to Figure 2 , Figure 2 A schematic diagram of the overall framework of the whole slide pathological image classification method is shown.
[0029] like Figure 2 As shown, the whole-slide pathological image classification method consists of five sub-modules: (a) Input; the histopathological image as the source domain is cropped into 224×224 slices; the WSI segmentation as the target domain is randomly selected into 224×224 slices containing only valid pathological tissue and cells.
[0030] (b) Feature extraction and fusion module, specifically the CNN-SwinT feature extraction and fusion (CTFF) module. Features from the source and target domain images are extracted using a deep convolutional network BreNet (pre-trained on the ImageNet dataset and transferred to pathological image applications) and a SwinTransformer network, respectively, to obtain effective classification information from different angles and dimensions. Then, the features extracted by the two models are weighted and fused.
[0031] (c) Multi-Granularity Representation (MGR) module. Based on the fused features, samples at three granularity levels are constructed for the source and target domains respectively: original samples, granular envelope samples, and class samples.
[0032] (d) Hierarchical Multi-Granularity Domain Alignment (HMGDA) module: In order to align source and target domain images in an unsupervised manner, thereby using source domain classification features to guide cancer classification prediction of target domain images, alignment and distribution alignment are performed in the Reproducing Kernel Hilbert Space (RKHS) based on the conditional covariance operator of the two domain features, thereby achieving domain alignment of high-dimensional complex features of source and target domain samples.
[0033] (e) Cancerous region identification module: The slice classification probability obtained by the classifier is returned to the corresponding position on the original WSI, and finally a lesion identification prediction heatmap of the entire WSI is generated.
[0034] Among them, the feature extraction and fusion module, the multi-granularity representation module, and the hierarchical multi-granularity domain alignment module are the core model components of the pathological image classification model. They complete the image conversion and knowledge transfer and are the core algorithms for the model to achieve unsupervised domain adaptation. The input module and the cancerous region identification module are the data preprocessing and result postprocessing. They can be integrated with the core model to form an end-to-end black box system, or they can be configured independently (e.g., as independent data preprocessing services and visualization services).
[0035] In some embodiments, obtaining the labeled source domain slice set and the unlabeled target domain slice set includes: Acquire multiple labeled sample pathological images to construct the source domain slice set; Acquire at least one unlabeled whole slide pathology image, and segment the unlabeled whole slide pathology image into multiple sample slices that include effective tissue regions; Randomly select at least a portion of the sample slices that include the effective tissue region to construct the target domain slice set.
[0036] In some embodiments, the feature extraction and fusion module of the pathological image classification model extracts sample fusion features from the source domain slice set and the target domain slice set, respectively, including: Local detail features of sample slices in the source domain slice set and the target domain slice set are extracted using deep convolutional networks. Global semantic features of sample slices in the source domain slice set and the target domain slice set are extracted using SwinTransformer. Pooling operations are performed on the local detail features and global semantic features of the sample slices respectively to obtain local detail features and global semantic features with uniform channel dimensions; The local detail features and global semantic features of the channel dimension are weighted and fused to obtain the sample fusion features of the sample slice.
[0037] Pathological image data is characterized by extremely high resolution and large size. Therefore, a complete pathological image (WSI) contains a large amount of intracellular and interstitial information about the lesion area. Due to the limitation of the input image dimension of deep learning models, pathological images need to be preprocessed and segmented before being input into the model. However, the commonly used feature extraction network—the convolutional network—has a limited receptive field and focuses more on local details. This inevitably leads to the loss of some interstitial information during the feature extraction process.
[0038] In order to fully extract the cancer-specific features contained in the image and obtain equally important fine-grained cellular information and coarse-grained interstitial information, in the process of image feature extraction, in addition to using the necessary disease-specific convolutional network BreNet to extract features, the model also applies Swin Transformer in parallel to obtain feature information from a larger-scale receptive field.
[0039] The BreNet deep convolutional neural network was pre-trained using balloon images from the ImageNet public dataset as the training set before the formal experiments. The input samples had dimensions of 3×224×224. A high-dimensional feature of 512×7×7 was extracted using multiple convolutional units. Then, to match the features extracted by the Swing Transformer... The high-dimensional feature is transformed by global average pooling (GAP) to obtain the final extracted feature. In summary, regarding the sample Features are extracted using a deep convolutional neural network. The process can be described by the following formula (1): (1) in, For the sample Features extracted using a deep convolutional neural network; This indicates that BreNet supports samples Perform feature extraction. This indicates a global average pooling operation.
[0040] To obtain more global and multi-scale features from the samples, a Swin Transformer and a convolutional network are introduced to process the input samples in parallel. This network first divides the input samples into 4×4 patches, then generates corresponding feature sequence tokens through mapping, which are input into the basic unit of the Swin Transformer to finally extract the features. .
[0041] Global average pooling (GAP) is applied to the output of the Swing Transformer to ultimately obtain the same result as... Unified output features in the second channel dimension , means as follows: (2); in, Features output by the Swing Transformer.
[0042] Features were extracted using deep convolutional neural networks BreNet and Swing Transformer. and Through feature weighted fusion, the feature map extracted from the original image slices by this module is finally obtained. In other words, fusion features: (3) in; The fusion features extracted by the feature extraction and fusion module; Features Weight parameters; Features The weight parameters.
[0043] In some embodiments, please refer to Figure 3 , Figure 3 The flowchart of the method for generating multi-granularity sample representations according to an embodiment of this application is shown; in the whole slide pathological image classification method, the construction of multi-granularity sample representations based on the sample fusion features of the sample slices, including the original sample layer, the particle envelope sample layer, and the class center sample layer, includes the following steps S301-S303: S301. The sample fusion features of each sample slice in the source domain and the target domain are respectively used as the sample representation of the original sample layer. S302. For the sample fusion features of each sample slice in the source domain and the target domain respectively, through iterative splitting and purity judgment, the sample slices with similar sample fusion features are clustered into multiple particle envelopes, and the statistical quantity of the sample fusion features in each particle envelope is used as the sample representation of the particle envelope sample layer. S303. Calculate the mean of the sample fusion features of all sample slices belonging to the same category in the source domain and the target domain respectively, and use the mean as the sample representation of the class center sample layer of that category.
[0044] Inspired by the knowledge pyramid theory, the features of the fused source and target domains are described at three different granularities: data layer, information layer, and knowledge layer. For example... Figure 2As shown, the class center sample layer represents the coarsest granularity, reflecting the overall data distribution and used to characterize the "knowledge" of the samples. The original samples represent the finest granularity of "data." Granular envelope samples are used to characterize "information," and the key to this process is performing a granular envelope generation operation to obtain granular envelope samples.
[0045] The particle envelope consists of three important properties: center ,radius and purity ;in, x i The representative sample is n, which is the number of samples; assuming a particle envelope is composed of The composition of objects of a class, that is , express The first in the class The number of objects in the class; the purity of the particle envelope. It can be calculated as The generation process of the particle envelope is conditioned on a set purity threshold. First, the entire dataset is initialized as a single particle envelope, with its initial center being the average of all samples, and the label being... At this point, the purity of the initial particle envelope is at its lowest, failing to meet the set threshold condition; therefore, the particle envelope needs to be split: randomly selected The remaining heterogeneous points are used as new centers, and all samples in the particle envelope are reassigned to these centers. A center, generating A new particle envelope is generated; the purity value of each sub-envelope is calculated. If the purity threshold is not met, the sub-envelope is split again until the requirement is met; finally, all qualified particle envelopes are globally partitioned to ensure that the split point of each particle envelope is closer to its center; at this point, the particle envelope sample located in the information layer can be obtained.
[0046] In some embodiments, please refer to Figure 4 The step of using the labels of the source domain slice set as conditions, and performing multi-granularity hierarchical neighborhood alignment of the source and target domains in the RKHS space based on the multi-granularity sample representation, and determining various alignment losses, includes the following steps S401-S403: S401. In the original sample layer, the global feature distributions of the source domain and the target domain are aligned based on the maximum mean difference of the first-order statistic, and the distribution matching loss is calculated. S402. In the class center sample layer, the source domain and target domain data distributions conditioned on the label are finely adapted based on the Wasserstein distance of the second-order covariance matrix, and the conditional domain alignment loss is calculated. S403. In the particle envelope sample layer and / or the original sample layer, based on cross-domain contrastive learning, the representation distance of similar samples is brought closer and the representation distance of dissimilar samples is widened to enhance the semantic discriminativeness of features, and the semantic contrastive loss is calculated.
[0047] In some embodiments, the step of performing fine-grained adaptation of the source and target domain data distributions conditioned on labels based on the Wasserstein distance of the second-order covariance matrix in the class center sample layer, and calculating the conditional neighborhood alignment loss, includes: mapping the sample representations and corresponding labels of the class center sample layers of the source and target domains to the RKHS space through a kernel function; wherein the labels of the target domain are pseudo-labels; calculating the second-order covariance matrices of the source and target domains conditioned on labels in the RKHS; and calculating the Wasserstein distance between the second-order covariance matrices of the source and target domains as the conditional neighborhood alignment loss.
[0048] In some embodiments, in the whole-slide pathological image classification method, optimizing the pathological image classification model based on the classifier loss of the multiple alignment losses and the pathological image classification model includes: The various alignment losses and the classifier losses are weighted and fused to determine the total loss function of the pathological image classification model; The pathological image classification model is optimized based on the total loss function.
[0049] Unsupervised domain adaptation methods assume that labeled source domains and unlabeled target domains have the same feature space but different feature distributions. Therefore, to preserve the distribution differences between different classes of samples within the domain, the whole-slide pathological image classification method described in this application introduces a conditional covariance operator based on feature representations at three granularities to measure the changes in the relationship between samples in the two domains under given conditions at different granularity levels.
[0050] In the RKHS space (Reproducing Kernel Hilbert Space), the conditional covariance operator can be viewed as a linear operator. Therefore, the label is expressed through a kernel function. Embedded as a condition variable in a high-dimensional RKHS space, it can capture complex nonlinear dependencies between samples while preserving label information.
[0051] In some embodiments, the pathological image classification model uses a Gaussian kernel function in its specific implementation. As an explicit kernel function, its expression is: (4) Wherein, (x, y) are two input sample vectors, specifically the fused feature vector extracted by CNN-SwinT; σ This is the bandwidth parameter of the kernel function.
[0052] Features extracted from source and target domain samples and (pseudo) labels for the source and target domain datasets. Mapping to the reproducing kernel Hilbert space (RKHS), the corresponding implicit mapping matrix is defined as (The latent mapping matrix of the features) and (The implicit mapping matrix of the labels). Its covariance operator can then be written as: (5) in, yes The centralized matrix can... and Zero-mean matrix reduction eliminates the influence of the mean. This indicates the number of labeled source domain features; This indicates the number of unlabeled target domain features; the subscript 's' represents the source domain, and the subscript 't' represents the target domain. Let X represent the covariance operator of features X and labels Y, Φ represent the implicit mapping matrix of features, and Ψ represent the implicit mapping matrix of labels.
[0053] Calculate based on condition variables according to the following formula. Conditional covariance operator : (6) in For regularization parameters, to prevent Irreversible; For conditional covariance operators; The covariance operator that characterizes feature X itself; Covariance operator representing feature X and label Y; The covariance operator that characterizes the label Y itself; The covariance operator characterizes the label Y and the feature X. I represents the identity matrix.
[0054] The conditional covariance operators corresponding to the feature maps of the source and target domains are calculated as follows: and .
[0055] To measure the difference in conditional covariance between two features from different domains in the RKHS high-dimensional space, the two conditional covariance operators are calculated. , The Wasserstein distance between them is used as the loss. The second-order Wasserstein distance is suitable for measuring the similarity of two probability distributions in terms of "shape" and "location". In RKHS, the Wasserstein distance is more geometrically sensitive, reflecting the local geometry of the distributions more accurately. When two probability distributions are identical, their second-order Wasserstein distance is 0, thus minimizing the Conditional Neighborhood Alignment (CDA) loss function... This can achieve feature alignment between two domains; the calculation formula is as follows: (7): (7) in This indicates the calculation of the second-order Wasserstein distance between two variables. .
[0056] To further reduce the distribution differences of source and target domain features in the RKHS space, a loss function based on maximum mean error is used to further align the distributions of features from different domains, corresponding to Sample Domain Alignment (SDA) loss. The calculation is as follows: (8) in The source domain features in this stage of the model, The target domain features in the model at this stage. Let n denote the Gaussian kernel function, where n is the number of features.
[0057] To enhance the discriminative power of domain-invariant features while reducing inter-domain differences, embodiments of this application introduce contrastive learning to narrow the gap between samples of the same type and widen the gap between samples of different categories. Specifically, the features of the i-th sample in the target domain are... As an anchor point, it is compared with similar samples in the source domain. Positive sample pairs are formed, while negative sample pairs are formed with samples from other classes. The cross-class contrastive loss is defined as follows: (9) in This indicates that the source domain has the same pseudo-label as the target anchor point. The sample set; Temperature is a hyperparameter used to adjust the discriminative strength of similarity, thereby controlling the model's attention to difficult negative samples; Finally, the classifier output is normalized using the sigmoid function to obtain the input slice samples. binary classification prediction score For source domain samples According to its corresponding real label and corresponding predicted scores Using cross-entropy loss as the loss for the classifier, the function can be expressed as: (10); Where n is the number of samples; For source domain samples The true label; For source domain samples The predicted score.
[0058] For target domain samples The maximum predicted score is used as the predicted probability of the model for classifying the image slice. and prediction labels , is represented as: (11); in, Characterizing the first j The probability score of a sample being predicted as class 0; Characterizing the first j The probability score of a sample being predicted as category 1.
[0059] In summary, the total loss function of the pathological image classification model in the whole-slide pathological image classification method described in this application embodiment can be defined as: (12) in for L CDA Weight parameters; for L SDA The weight parameters, for L DIS The weight parameters.
[0060] The following section tests and evaluates the whole-slide pathological image classification method described in the embodiments of this application.
[0061] In this embodiment, the small-sized breast cancer histopathological image dataset BreaKHis_100× and the publicly available breast cancer pathology WSI dataset Cmelyon16 were used as the source and target domains, respectively, to evaluate the predictive performance of the method for identifying cancerous regions using WSI. Simultaneously, experiments with cancer data were used to verify the model's generalization ability in the field of pathological medical auxiliary diagnosis.
[0062] The BreaKHis dataset is a public breast cancer histopathology dataset containing 7909 breast tissue pathology sections from 82 patients (including 2480 normal and 5428 cancerous tissue pathology images), all stained with hematoxylin and eosin (H&E). Images at 100x magnification (abbreviated as BreaKHis_100×) were selected in the experiment to match the size of the remaining dataset. The BreaKHis_100× image set contains 644 normal and 1437 cancerous images.
[0063] The Camelyon16 dataset, stained with H&E, includes 170 breast lymph node images (100 normal and 70 cancerous) as the training set and 100 images (60 normal and 40 cancerous) as the test set, used for the detection and classification of breast cancer.
[0064] In the quantitative evaluation, this application employs widely used evaluation metrics, including accuracy (ACC), F1 score, sensitivity, specificity, and the area under the receiver operating characteristic (ROC) curve (AUC), to assess model performance. These metrics can be described as follows: (13) in, , , and This represents the number of samples predicted as true positive, false positive, true negative, and false negative, respectively.
[0065] This application embodiment conducts domain adaptation experiments on small-sized pathological images to WSI images on a breast cancer pathology image dataset to evaluate the performance of the proposed model and component modules. Specifically, the experiment uses the BreaKHis public dataset (abbreviated as BreaKHis_100×) with a magnification of 100× as the source domain and randomly sampled slices of WSI from the Camelyon16 dataset as the target domain for cross-domain experiments.
[0066] First, experiments were conducted to compare the method described in the embodiments of this application with several state-of-the-art unsupervised domain adaptation methods, evaluating the algorithm's ability to predict cancer morphology in WSI slides based on unsupervised domain adaptation technology. The comparison methods included: (1) Confidence-Diffusion InstanceContrastive Learning (CDICL): Constructs an information perception space through instance contrastive learning, thereby achieving unsupervised domain adaptation; (2) Bi-discriminator domain Adversarial neural network with Class-level Gradient alignment (BACG): Unsupervised domain adaptation is achieved through a bi-discriminator domain adversarial neural network with class-level gradient alignment; (3) Robust Spherical Domain Adaptation (RSDA): A reweighted adversarial training strategy is proposed to effectively reduce intra-domain differences within the target domain, thereby optimizing unsupervised domain adaptation methods.
[0067] (4) Risk-Consistent Estimator (RCE): A new risk-consistent training method is proposed for unsupervised domain adaptation.
[0068] The methods and other comparison algorithms described in this application all use Python 3.8 and PyTorch 2.0.1 as the basic implementation environment, equipped with an Intel @ Xeon(R) CPU E5-2620 v4 @ 2.10 GHz × 32 and a GeForce GTX2080Ti. During training, the Adam optimization method is used to optimize the network, and a combination of random search and grid search is used to automatically optimize the hyperparameters in the algorithm. The maximum training epochs and batch size are set to 50 and 24, respectively. The initial learning rate is 0.001 and will be adaptively adjusted according to the training epochs.
[0069] To fairly compare the cancer prediction capabilities of the methods described in this application embodiments with recent advanced UDA methods for breast cancer pathological tissue images, the original parameter settings of the comparison methods were retained during the experiment for training, and all methods were tested 5 times on the same dataset configuration. The mean and standard deviation of each evaluation index in the 5 experiments were recorded in Table 1 for more accurate evaluation.
[0070] According to the table, the method described in this application significantly outperforms other comparative methods in predicting cancerous lesions in pathological images of breast cancer (BreaKHis_100×-Camelyon16). For the breast cancer prediction experiment, the results are shown in Table 1. Compared to the second-best performing RCE method, the classification results of this method show an improvement of approximately 13% in F1 score and accuracy, and an improvement of approximately 20% in AUC, demonstrating the superior overall performance of the method in this application. Furthermore, the specificity indicators of other comparative methods are poor, indicating that other methods have a weaker ability to identify normal samples, which would result in a higher misdiagnosis rate in clinical applications.
[0071] Table 1 Comparison results of different classification methods
[0072] In summary, compared with other comparative UDA methods, the method described in this application demonstrates stability in cancer prediction tasks and exhibits stronger generalization ability, proving the algorithm's potential in practical medical applications. This indicates that the method described in this application is not only suitable for detecting specific cancer types but can also be widely applied to the analysis of various pathological images, providing clinicians with a more reliable auxiliary diagnostic tool.
[0073] The method described in this application achieved a classification accuracy of 90.97% in a domain adaptation experiment from histopathological images of breast cancer to WSI images, far exceeding other unsupervised domain adaptation comparison algorithms, demonstrating the algorithm's powerful performance and good generalization ability.
[0074] This application proposes a whole-slide pathological image classification method. It uses small-sized tissue pathological images as the source domain to extract cancer-specific information, thereby guiding the model to predict cancer in unsupervised WSI slides. First, the pathological image classification model extracts features from samples using a parallel feature extraction and fusion module combining CNN and Swing Transformer. This model combines the high sensitivity of convolutional networks to detailed information with the advantages of Transformer in effectively extracting overall semantic information, extracting high-dimensional features at multiple scales and angles and performing weighted fusion. Then, a conditional-sample domain alignment module is designed to map the fused features to the RKHS space. By using label information as a conditional variable, the second-order Wasserstein distance of the conditional covariance operators in different domains is minimized. Combined with the maximum mean difference between cross-domain features, this minimizes the marginal distribution of features in different domains while preserving the distribution differences of samples of different categories within the domain, effectively achieving distribution alignment between the source and target domains.
[0075] Subsequently, the model performs unsupervised classification and identification of cancerous lesions across species and domains in pathological images of breast and colon cancer. Compared with other UDA and ablation methods, this model achieves the best prediction results for cancerous regions in WSI images, indicating that the algorithm can effectively mine the features of histopathological images and then transfer cancer-specific information from the source domain features for cancer prediction of WSI slices in the target domain. Visualization of the prediction results can assist pathologists in annotating cancerous regions in pathological images, greatly improving the efficiency of diagnostic analysis of pathological images. At the same time, this model performs excellently in breast and colon cancer prediction experiments, further demonstrating the model's generalization ability and potential in clinical applications.
[0076] Based on the same inventive concept, this application also provides a whole slide pathological image classification device corresponding to the whole slide pathological image classification method. Since the principle of the device in this application is similar to the whole slide pathological image classification method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0077] Please refer to Figure 5 , Figure 5 This illustration shows a schematic diagram of the whole-slide pathology image classification device according to an embodiment of this application; the whole-slide pathology image classification device includes: The acquisition module 501 is used to acquire the target whole slide pathological image to be identified, and to perform slicing processing on the target whole slide pathological image to obtain multiple target slices; The determination module 502 is used to input the multiple target slices into a trained pathological image classification model to determine the category prediction probability of each target slice. The pathological image classification model is trained through the following steps: obtaining a labeled source domain slice set and an unlabeled target domain slice set; extracting sample fusion features from the sample slices in the source domain slice set and the target domain slice set using the feature extraction and fusion module of the pathological image classification model; constructing a multi-granularity sample representation including an original sample layer, a granular envelope sample layer, and a class center sample layer based on the sample fusion features of the sample slices; performing multi-granularity hierarchical neighborhood alignment of the source domain and target domain in the RKHS space based on the multi-granularity sample representation, using the labels of the source domain slice set as conditions, and determining various alignment losses; optimizing the pathological image classification model based on the various alignment losses and the classifier loss of the pathological image classification model to obtain a trained pathological image classification model. The generation module 503 is used to map and synthesize the category prediction probability corresponding to each target slice according to the position of the multiple target slices in the whole slide pathological image, and generate a lesion identification prediction heatmap of the whole slide pathological image.
[0078] Based on the same inventive concept, this application also provides an electronic device corresponding to the whole slide pathological image classification method. Since the principle of solving the problem by the electronic device in this application is similar to the whole slide pathological image classification method described above in this application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be described again.
[0079] Please refer to Figure 6 , Figure 6 A schematic diagram of the structure of the electronic device described in this application embodiment is shown. The electronic device 600 includes a processor 602, a memory 601, and a bus. The memory 601 stores machine-readable instructions executable by the processor 602. When the electronic device 600 is running, the processor 602 communicates with the memory 601 via the bus. When the machine-readable instructions are executed by the processor 602, the steps of the whole slide pathological image classification method are performed.
[0080] Based on the same inventive concept, this application also provides a computer-readable storage medium corresponding to the whole slide pathological image classification method. Since the principle of the computer-readable storage medium in this application is similar to the whole slide pathological image classification method described above in this application, the implementation of the computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be described again.
[0081] A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the whole slide pathological image classification method.
[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.
[0083] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0084] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0085] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.
[0086] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for classifying pathological images on a whole slide, characterized in that, The method includes: A whole-slide pathological image of the target to be identified is acquired, and the whole-slide pathological image of the target is sliced to obtain multiple target slices; The multiple target slices are input into a trained pathological image classification model to determine the predicted category probability of each target slice. The pathological image classification model is trained through the following steps: obtaining a labeled source domain slice set and an unlabeled target domain slice set; extracting sample fusion features from the source domain slice set and the target domain slice set using the feature extraction and fusion module of the pathological image classification model; and constructing a multi-granularity sample representation based on the sample fusion features of the sample slices, including an original sample layer, a granular envelope sample layer, and a class center sample layer. Using the labels of the source domain slice set as conditions, multi-granularity hierarchical neighborhood alignment of the source and target domains is performed in the RKHS space based on the multi-granularity sample representation, and various alignment losses are determined; based on the various alignment losses and the loss of the classifier of the pathological image classification model, the pathological image classification model is optimized to obtain a trained pathological image classification model. Based on the positions of the multiple target slices in the whole slide pathological image, the predicted probabilities of the categories corresponding to each target slice are mapped and synthesized to generate a lesion identification prediction heatmap of the whole slide pathological image.
2. The whole-slide pathological image classification method according to claim 1, characterized in that, The process of obtaining the labeled source domain slice set and the unlabeled target domain slice set includes: Acquire multiple labeled sample pathological images to construct the source domain slice set; Acquire at least one unlabeled whole slide pathology image, and segment the unlabeled whole slide pathology image into multiple sample slices that include effective tissue regions; Randomly select at least a portion of the sample slices that include the effective tissue region to construct the target domain slice set.
3. The whole-slide pathological image classification method according to claim 1, characterized in that, The feature extraction and fusion module of the pathological image classification model extracts sample fusion features from sample slices in the source domain slice set and the target domain slice set, respectively, including: Local detail features of sample slices in the source domain slice set and the target domain slice set are extracted using deep convolutional networks. Global semantic features of sample slices in the source domain slice set and the target domain slice set are extracted using SwinTransformer. Pooling operations are performed on the local detail features and global semantic features of the sample slices respectively to obtain local detail features and global semantic features with uniform channel dimensions; The local detail features and global semantic features of the channel dimension are weighted and fused to obtain the sample fusion features of the sample slice.
4. The whole-slide pathological image classification method according to claim 1, characterized in that, The sample fusion feature construction based on the sample slices includes a multi-granularity sample representation comprising an original sample layer, a granular envelope sample layer, and a class center sample layer, including: The sample fusion features of each sample slice in the source domain and the target domain are used as the sample representation of the original sample layer, respectively. For the sample fusion features of each sample slice in the source domain and the target domain, sample slices with similar sample fusion features are clustered into multiple particle envelopes by iterative splitting and purity judgment. The statistical measure of the sample fusion features in each particle envelope is used as the sample representation of the particle envelope sample layer. The mean of the sample fusion features of all sample slices belonging to the same category in the source domain and the target domain are calculated respectively, and the mean is used as the sample representation of the class center sample layer of that category.
5. The whole-slide pathological image classification method according to claim 4, characterized in that, The method uses the labels of the source domain slice set as conditions, and performs multi-granularity hierarchical neighborhood alignment of the source and target domains in the RKHS space based on the multi-granularity sample representation, determining various alignment losses, including: In the original sample layer, the global feature distributions of the source and target domains are aligned based on the maximum mean difference of the first-order statistic, and the distribution matching loss is calculated. In the class center sample layer, the source and target domain data distributions conditioned on labels are finely adapted based on the Wasserstein distance of the second-order covariance matrix, and the conditional neighborhood alignment loss is calculated. In the particle envelope sample layer and / or the original sample layer, based on cross-domain contrastive learning, the representation distance of similar samples is narrowed and the representation distance of dissimilar samples is widened, thereby enhancing the semantic discriminativeness of features, and the semantic contrastive loss is calculated.
6. The whole-slide pathological image classification method according to claim 5, characterized in that, In the class center sample layer, fine-grained adaptation of the source and target domain data distributions conditioned on labels is performed based on the Wasserstein distance of the second-order covariance matrix, and the conditional neighborhood alignment loss is calculated, including: The sample representations and corresponding labels of the class center sample layers of the source domain and the target domain are mapped to the RKHS space through a kernel function; where the labels of the target domain are pseudo-labels. In the RKHS, the second-order covariance matrices of the source domain and the target domain, conditioned on the labels, are calculated respectively; The Wasserstein distance between the second-order covariance matrices of the source domain and the target domain is calculated and used as the conditional neighborhood alignment loss.
7. The whole-slide pathological image classification method according to claim 1, characterized in that, The optimization of the pathological image classification model based on the classifier loss of the various alignment losses and the pathological image classification model includes: The various alignment losses and the classifier losses are weighted and fused to determine the total loss function of the pathological image classification model; The pathological image classification model is optimized based on the total loss function.
8. A whole-slide pathological image classification device, characterized in that, The device includes: The acquisition module is used to acquire the target whole slide pathological image to be identified, and to perform slicing processing on the target whole slide pathological image to obtain multiple target slices; A determination module is used to input the multiple target slices into a trained pathological image classification model to determine the predicted category probability of each target slice. The pathological image classification model is trained through the following steps: obtaining a labeled source domain slice set and an unlabeled target domain slice set; extracting sample fusion features from the sample slices in the source domain slice set and the target domain slice set using the feature extraction and fusion module of the pathological image classification model; constructing a multi-granularity sample representation including an original sample layer, a granular envelope sample layer, and a class center sample layer based on the sample fusion features of the sample slices; performing multi-granularity hierarchical neighborhood alignment of the source domain and target domain in the RKHS space based on the multi-granularity sample representation, using the labels of the source domain slice set as conditions, and determining various alignment losses; optimizing the pathological image classification model based on the various alignment losses and the classifier loss of the pathological image classification model to obtain a trained pathological image classification model. The generation module is used to map and synthesize the predicted probability of the category corresponding to each target slice according to the position of the multiple target slices in the whole slide pathological image, and generate a lesion identification prediction heatmap of the whole slide pathological image.
9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the whole-slide pathological image classification method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the whole slide pathological image classification method as described in any one of claims 1 to 7.