A Histopathological Image Classification Method and System Based on Joint Manifold Alignment Reconstruction

By employing a joint manifold alignment reconstruction method, the problems of overfitting and poor generalization ability in histopathological image classification are solved. High-precision unsupervised adaptive classification is achieved through distribution alignment of sub-domain datasets and cluster center alignment.

CN118674984BActive Publication Date: 2026-11-10CHONGQING UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410791183.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-19
Publication Date
2026-11-10
Estimated Expiration
2044-06-19

AI Technical Summary

Technical Problem

Existing technologies suffer from overfitting and poor generalization in histopathological image classification, especially when the target domain data is unlabeled, making it difficult to effectively extract and utilize low-order features, thus affecting classification accuracy.

Method used

A joint manifold alignment-based reconstruction method is adopted. By constructing sub-domain datasets of the source and target domains, pseudo-labels are generated. Multi-level fusion features are extracted using a feature extraction and fusion module and mapped to the common subspace of the manifold for distribution alignment and cluster center alignment, eliminating information redundancy and finally performing classification in a low-dimensional space.

Benefits of technology

It achieves accurate classification on unlabeled target domain data, improves the classification accuracy of histopathological images, reduces information redundancy, and enhances cross-domain classification capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118674984B_ABST
    Figure CN118674984B_ABST
Patent Text Reader

Abstract

This invention relates to the field of image classification technology, specifically disclosing a method and system for classifying histopathological images based on joint manifold alignment reconstruction. It utilizes multi-layer feature extraction and fusion based on the BreNet deep transfer model pre-trained on the ImageNet natural dataset. The system integrates the manifold space mapping distribution of the fused feature sub-domains with joint alignment of cluster centers, and classifies images based on manifold-reconstructed features. The system and method extract multi-order features through a feature extraction and fusion module and perform global pooling fusion to obtain deeply fused features. These features are then input into the joint manifold alignment reconstruction module, which projects them onto a common subspace of the manifold. Sub-domain distribution alignment and cluster center alignment minimize cross-domain sub-domain distribution differences, achieving domain adaptation. Simultaneously, the features are projected into a low-dimensional space to eliminate information redundancy, retaining key information effective for classification and further improving classification accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image classification technology, and in particular to a method and system for classifying histopathological images based on joint manifold alignment reconstruction. Background Technology

[0002] In recent years, pathological image analysis, led by pathologists, has gradually become an important component of cancer diagnosis and prognosis. Pathological images not only provide diagnostic information, but their rich representational capabilities are also crucial for clinical diagnostic intervention and prediction. Under routine clinical conditions, due to the heavy workload of manually collecting pathological images and the highly subjective nature of pathological analysis, cancer diagnosis based on pathological image analysis is difficult to implement. With the widespread use of digital pathology slide scanners, computer-aided diagnosis (CAD) has become an effective way to help pathologists reduce misdiagnosis rates and shorten diagnostic time. The steps of computer-aided diagnosis of pathological images generally include: image preprocessing, feature extraction, and image classification. Early work mainly relied on empirical knowledge of histopathological images to extract image features; traditional machine learning algorithms, such as support vector machines and random forests, were used to detect or segment regions of interest (ROIs) in histological images, such as cell nuclei and glands.

[0003] With the continuous development of big data analysis and digital pathological images, deep learning has become the mainstream method for computer-aided pathological image analysis. Convolutional Neural Networks (CNNs), due to their ability to extract useful high-order information from raw images, are widely used in research on the classification of natural and even pathological images. Although CNNs can achieve accurate and efficient image classification, this requires a large number of labeled images for training to improve the network's generalization ability and prevent overfitting. Due to issues such as data privacy and high cost, the amount of tissue pathological image data is relatively small, and directly using classic deep neural networks will lead to overfitting and poor generalization ability.

[0004] Transfer learning (TL) can learn information from target domain samples using a limited number of labeled source domain samples, and has therefore received widespread attention in the field of medical image processing. Due to the significant differences between natural and medical images, domain adaptation (DA) algorithms, which consider the similarity between the two domains, should be used for histopathological image classification tasks. Depending on whether the target domain data is labeled, DA algorithms can be further divided into semi-supervised domain adaptation (SSDA) and unsupervised domain adaptation (UDA). SSDA is based on a small number of labeled target domain samples. Compared to SSDA, UDA algorithms are more suitable for pathological image classification tasks because the target domain data does not require any labels, and have thus received wider attention.

[0005] UDA typically processes high-order features extracted by CNNs. However, for pathological image classification tasks, low-order features rich in texture and spatial information are an indispensable part of characterizing cell features and influencing classification accuracy. Therefore, fusing low-order and high-order features of data instances to extract fused features containing global and local information can enhance the similarity between samples of the same category in different domains. Meanwhile, previous DA methods often extract global domain offsets, i.e., extract invariant representations between the source and target domains to minimize the distributional differences between the two domains. However, this can also confuse the classifier and affect the final classification accuracy. Summary of the Invention

[0006] This invention provides a histopathological image classification method and system based on joint manifold alignment reconstruction. The technical problem it solves is: how to extract local domain offset information from sub-domains within the source and target domains for distribution alignment, thereby minimizing the distribution differences between different categories in the two domains.

[0007] To address the above technical problems, this invention provides a histopathological image classification method based on joint manifold alignment reconstruction, comprising the following steps:

[0008] S1. Construct source domain dataset and target domain dataset, wherein the source domain dataset includes multiple labeled source domain histopathological image samples, i.e., source domain samples, and the target domain dataset includes multiple unlabeled target domain histopathological image samples, i.e., target domain samples.

[0009] S2. Generate pseudo-labels for unlabeled target domain samples to obtain labeled target domain samples, and divide the source domain and target domain samples into multiple source domain sub-domain datasets and multiple target domain sub-domain datasets according to the labels.

[0010] S3. Construct an image classification model, which includes a feature extraction and fusion module, a joint manifold alignment and reconstruction module, and a classifier;

[0011] The feature extraction and fusion module is used to extract multi-level fusion features from the input samples. The joint manifold alignment and reconstruction module operates by first mapping the multi-level fusion features to a common subspace of the manifold, and then, within this common subspace, jointly aligning the similar sub-domain distributions of the source and target domains and the inter-domain alignment of sub-domain cluster centers. This minimizes the differences in the distribution of the same category while matching the global distribution, thereby effectively transferring labeled source domain data information to unlabeled target domain data and eliminating information redundancy between multi-level features, ultimately obtaining low-dimensional reconstruction features for cross-domain classification prediction. The classifier is used to output the corresponding classification prediction result based on the low-dimensional reconstruction features of each sample.

[0012] S4. Divide multiple source domain subdomain datasets and multiple target domain subdomain datasets into training sets and test sets, and use the training set to train the image classification model, and use the test set to test the trained image classification model.

[0013] S5. Use the trained image classification model to classify the input target domain histopathological image and output the classification result.

[0014] Furthermore, during training, the total loss function used includes the cross-entropy classification loss calculated based on the labeled source domain samples and their classification prediction probabilities. The sub-domain alignment loss of the fusion features is calculated based on the distribution differences of the fusion features of the sub-domain samples output by the feature extraction and fusion module. and the manifold alignment and reconstruction loss of the joint manifold alignment and reconstruction module.

[0015] Furthermore, manifold alignment reconstruction loss Build as:

[0016]

[0017] in, This represents the manifold reconstruction loss calculated based on the input and output of the joint manifold alignment and reconstruction module. This represents the distribution loss calculated based on the cluster centers of the source and target domain sub-domains. This represents the inter-domain local alignment loss calculated based on the adjacent fusion features in the source domain subdomain, the adjacent fusion features in the target domain subdomain, and the adjacent fusion features in both the source and target domain subdomains. β1 and β2 are... and The weighting coefficients.

[0018] Furthermore, manifold reconstruction loss Calculated by the following formula:

[0019]

[0020] Where, n s And represents n t These represent the number of source domain samples and target domain samples in the training set, respectively. Let i and j represent the low-dimensional reconstruction features of the i-th and j-th source domain samples obtained by the joint manifold alignment and reconstruction module, respectively, where i ≠ j. Let W represent the low-dimensional reconstruction features of the a-th target domain sample and the j-th target domain sample obtained by the joint manifold alignment and reconstruction module, respectively, where a≠b. ij F represents the multi-level fusion feature of the i-th source domain sample and the j-th source domain sample in the original feature space. i and F j The intrinsic nearest neighbor matrix between them, W ab F represents the multi-level fusion features of the a-th target domain sample and the b-th target domain sample in the original feature space. a and F b The intrinsic nearest neighbor matrix between them, ||·||2 represents the calculation of Euclidean distance;

[0021] The low-dimensional reconstruction feature F of any source domain sample or target domain sample r Calculated by the following formula:

[0022]

[0023] Among them, W P For the characteristic projection matrix, It represents the multi-level fusion feature of any source domain sample or target domain sample.

[0024] Furthermore, inter-domain local alignment loss Calculated by the following formula:

[0025]

[0026] Where C represents the number of classifications, i.e., the number of subdomains of the source or target domain, and c∈{0,1,2,...,C-1} represents the class labels of the C classification task, n sc n tc These represent the number of source domain samples and target domain samples of class c in the training set, respectively. They represent the i-th class of class c in the training set, respectively. c The source domain sample and the j-th sample c Multi-level fusion features of source domain samples, i c ≠jc , Let a represent the multi-level fusion features of the ac-th and bc-th target domain samples of class c in the training set, respectively. c ≠b c κ(·) represents the Gaussian kernel function.

[0027] Furthermore, distributed loss Calculated by the following formula:

[0028]

[0029] in, and These represent the cluster centers of subdomain c within the source and target domains, respectively;

[0030] Initial cluster centers of the source or target domain Calculated by the following formula:

[0031]

[0032] Where N represents the number of samples in category c from the source or target domain. This represents the nth sample in category c from the source or target domain.

[0033] During the training process, m c Updated by the following formula:

[0034]

[0035] in, Let α and β represent the cluster centers of the source or target subdomains with label c updated in round p and round p-1, respectively, and let α∈[0,1] represent the average moving coefficient.

[0036] Furthermore, the total loss function used during training... Designed as follows:

[0037]

[0038] Where λ1 and λ2 are the alignment losses of the fused feature sub-domains, respectively. and manifold alignment reconstruction loss The weights;

[0039] Cross-entropy classification loss Calculated by the following formula:

[0040]

[0041] in, The output z of the image classification model represents s The predicted probability.

[0042] Furthermore, the steps for training and testing the image classification model include:

[0043] 1) Pre-train the BreNet network using the ImageNet dataset;

[0044] 2) Transfer the feature extraction layer and its corresponding parameters of the BreNet network, add a feature fusion layer to construct a feature extraction and fusion module, establish a classifier and the feature extraction and fusion module to construct an image classification model, and then initialize the fully connected layer of the classifier;

[0045] 3) Divide the source domain dataset and the target domain dataset into training and test sets according to the label categories respectively;

[0046] 4) Initialize the cluster centers of the source domain and target domain sub-domain samples for each label in the training set;

[0047] 5) Enter the iteration; in each iteration, execute:

[0048] ① The feature extraction and fusion module is used to extract multi-level fusion features from each subdomain of the input samples;

[0049] ② Calculate the sub-domain alignment loss of the fusion features based on the calculated multi-level fusion features.

[0050] ③ Map the fusion features of the source and target domains onto the common subspace of the manifold;

[0051] ④ Calculate and update the cluster centers for each subdomain;

[0052] ⑤ Align the updated class centers of each subdomain and align the distribution of each subdomain based on the local maximum mean difference mechanism;

[0053] ⑥ Based on the feature projection matrix W P The low-dimensional reconstruction features of each source domain sample and target domain sample are calculated in step ①, along with the multi-level fusion features calculated in step ①.

[0054] ⑦ Calculate manifold alignment reconstruction loss based on computational multi-level fusion features, low-dimensional reconstruction features, and cluster centers of various sub-domains.

[0055] ⑧ Feed the low-dimensional reconstructed features of each source domain sample and target domain sample into the classifier; calculate the cross-entropy classification loss based on the classifier's output.

[0056] ⑨ Alignment loss based on weights λ1, λ2 and the calculated fusion feature sub-domain. Manifold alignment reconstruction loss Cross-entropy classification loss Calculate total loss

[0057] ⑩ Based on the calculated total loss Backpropagation updates network parameters;

[0058] 6) Use the test set as input to perform classification tests on the trained image classification model.

[0059] Furthermore, the operation of the feature extraction and fusion module is represented as follows:

[0060]

[0061] Among them, s k (x i ) represents sample x i Input the feature extraction operation of the k-th layer of the neural network; Indicates sample x i The number of channels for feature extraction in the k-th layer; GAP(·) represents the global average pooling operation, and Ω represents the set of network layers selected for high, medium, and low-order features fusion. This indicates a cascade operation on k∈Ω-order features.

[0062] The present invention also provides a histopathological image classification system based on semantic relevance clustering, the key of which is: it is equipped with an image classification model generated by the above-mentioned histopathological image classification method based on joint manifold alignment reconstruction.

[0063] This invention provides a histopathological image classification method and system based on joint manifold alignment reconstruction. It utilizes multi-layer feature extraction and fusion based on the BreNet deep transfer model pre-trained on the ImageNet natural dataset. The system integrates the manifold space mapping distribution of the fused feature sub-domains with joint alignment of cluster centers, and classifies images based on manifold-reconstructed features. The system and method extract multi-order features through a feature extraction and fusion module and perform global pooling fusion to obtain deeply fused features. These features are then input into the joint manifold alignment reconstruction module, which projects them onto a common manifold subspace. Sub-domain distribution alignment and cluster center alignment minimize cross-domain sub-domain distribution differences, achieving domain adaptation. Simultaneously, the features are projected into a low-dimensional space to eliminate information redundancy, retaining key information effective for classification and further improving classification accuracy. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the histopathological image classification method based on joint manifold alignment reconstruction provided in the embodiments of the present invention;

[0065] Figure 2 This is a schematic diagram of the feature extraction and fusion module provided in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the joint manifold alignment and reconstruction module provided in an embodiment of the present invention. Detailed Implementation

[0067] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0068] A category within a domain can be abstracted as a subdomain. By extracting local domain offset information from the subdomains in both the source and target domains and aligning their distributions, fine-grained information useful for classification can be preserved, minimizing the distribution differences between different categories in the two domains. Simultaneously, the global distributions of the source and target domains are matched, thus achieving accurate classification of the target domain data. Therefore, this invention proposes a histopathological image classification method based on joint manifold alignment reconstruction, or an unsupervised subdomain adaptive algorithm based on joint manifold alignment reconstruction (JMSA-UDA), for classifying histopathological images.

[0069] The histopathological image classification method based on joint manifold alignment reconstruction provided in this invention includes the following steps:

[0070] S1. Construct source domain dataset and target domain dataset, wherein the source domain dataset includes multiple labeled source domain histopathological image samples, i.e., source domain samples, and the target domain dataset includes multiple unlabeled target domain histopathological image samples, i.e., target domain samples.

[0071] S2. Generate pseudo-labels for unlabeled target domain samples to obtain labeled target domain samples, and divide the source domain and target domain samples into multiple source domain sub-domain datasets and multiple target domain sub-domain datasets according to the labels.

[0072] S3. Construct an image classification model, which includes a feature extraction and fusion module, a joint manifold alignment and reconstruction module, and a classifier.

[0073] S4. Divide multiple source domain sub-domain datasets and multiple target domain sub-domain datasets into training sets and test sets, and use the training set to train the image classification model and use the test set to test the trained image classification model.

[0074] S5. Use the trained image classification model to classify the input target domain histopathological image and output the classification result.

[0075] (1) Steps S1 and S2: Constructing the dataset

[0076] In unsupervised subdomain adaptation, n s The source domain of a labeled sample (in For the sample (tags) and containing n t The target domain of an unlabeled sample Based on the category, the source domain and target domain are respectively divided into subdomains. Where c∈{0,1,2,...,C-1} represents the class label for a classification task, such as the source domain in a binary classification task. Divided into subdomains and target domain Divided into subdomains and Data instances in the source and target domains are respectively drawn from the source domain sample set {X} s ,Y s} and the target domain sample set {X t The sample was obtained from}, among which This represents a tagged source domain instance. This represents the label matrix corresponding to the source domain data instance. This represents the unlabeled target domain data matrix. d represents the data dimension, and C represents the number of classes.

[0077] (2) Step S3: Construct an image classification model

[0078] like Figure 1 As shown, the image classification model (JMSA-UDA model) includes a feature extraction and fusion module, a joint manifold alignment and reconstruction module, and a classifier. The feature extraction and fusion module extracts multi-level fusion features from the input samples. The joint manifold alignment and reconstruction module first maps the multi-level fusion features to a common subspace of the manifold. Then, within this common subspace, it performs joint alignment of similar sub-domain distributions of the source and target domains and inter-domain alignment of sub-domain cluster centers. This minimizes the differences in distributions of the same class while matching the global distribution, effectively transferring labeled source domain data information to unlabeled target domain data and eliminating information redundancy between multi-level features. Finally, it obtains low-dimensional reconstructed features for cross-domain classification prediction. The classifier outputs the corresponding classification prediction result based on the low-dimensional reconstructed features of each sample.

[0079] During feature extraction, global average pooling (GAP) is performed on the multi-level features extracted from different layers of the convolutional neural network to match features from different layers. For example, for a set of feature maps extracted from the k-th layer of the network... (Height × Width is h × w, number of channels is d) k After global average pooling, the generated feature map is... Therefore, after the feature extraction and fusion module extracts high-order, mid-order, and low-order features, it performs global average pooling and fuses the features through feature concatenation. Then, sample x... i Corresponding fusion features It can be represented as:

[0080]

[0081] Among them, s k (x i ) represents sample x i Input the feature extraction operation of the k-th layer of the neural network; Indicates sample x i The number of channels for feature extraction in the k-th layer; GAP(·) represents the global average pooling operation, and Ω represents the set of network layers selected for high, medium, and low-order features fusion. This indicates a cascade operation on features of order k ∈ Ω. Since the selected features at each level are extracted through the same network model and possess a certain degree of similarity, and higher-order features can represent more deep semantic information while lower-order features contain rich low-level structural information, the fused features can retain rich global and local information of the original image, enhancing its representational ability. A schematic diagram of the feature extraction and fusion module is shown below. Figure 2 As shown.

[0082] For samples with labeled source domains Its corresponding tag is For unlabeled target domain samples Based on the output z of the model network, the output probability p = soft max(z) can be obtained through the softmax output unit, and the pseudo-label of the target domain sample can be defined. Then, the target domain samples are matched one by one with the pseudo-labels to obtain...

[0083] In the joint manifold alignment and reconstruction module, the fused features are mapped to a common subspace of the manifold. Within this common subspace, the distribution alignment of similar sub-domains in the source and target domains and the inter-domain alignment of sub-domain cluster centers are performed. This minimizes the distribution differences of the same category while matching the global distribution, thereby effectively transferring labeled source domain data information to unlabeled target domain data and eliminating information redundancy between multi-level features. Finally, low-dimensional reconstructed features for cross-domain classification prediction are obtained.

[0084] Firstly, regarding the fusion features in the source and target domains and By projecting the feature onto the common subspace of the manifold feature through a manifold projection layer, a low-dimensional reconstructed feature that preserves the optimal neighborhood structure is obtained. and Its transformation can be expressed as:

[0085]

[0086] Where F r Represents low-dimensional reconstructed features, which are multi-level fusion features derived from any source domain sample or target domain sample. Obtained through manifold projection reconstruction; W P is the characteristic projection matrix; the superscript T indicates matrix transpose.

[0087] (3) Step S4: Training and testing the image classification model

[0088] For data from the same domain, initialize the sub-domain cluster centers; then update the cluster centers of each category according to the average moving average criterion. If the same domain contains C sub-domains, the process of generating and updating its cluster centers can be represented as follows:

[0089]

[0090] Where N represents the number of samples in category c from the source or target domain. This represents the nth sample in category c from the source or target domain. This represents the initial cluster centers generated from the training samples in each round of training. Let α represent the sub-domain cluster center with label c updated in the p-th round, and let α∈[0,1] represent the average moving coefficient.

[0091] Using cluster centers as representations of the distribution of samples within sub-domains of the source and target domains, if the cluster centers of related categories in the source and target domains are brought closer together, samples within the corresponding sub-domains will also be drawn closer together. Therefore, the Euclidean distance between the cluster centers of related sub-domains in the two domains is used as the loss.

[0092]

[0093] in, and represents the cluster centers of the sub-domains labeled c (c = 0, 1, ..., C-1) within the source and target domains, respectively, and ||·||2 represents the calculation of Euclidean distance.

[0094] To compare the distribution differences of fused features among samples from different sub-domains in the source and target domains within the fused feature space, the fused features are embedded into the Reproducing Kernel Hilbert Space (RKHS). Local domain adaptation is performed within this space based on the maximum mean difference criterion to align the features. The fused feature sub-domain alignment loss can be expressed as:

[0095]

[0096] Where C represents the number of classifications, i.e., the number of subdomains of the source or target domain, and c∈{0,1,2,...,C-1} represents the class labels of the C classification task, n sc n tc These represent the number of source domain samples and target domain samples of class c in the training set, respectively. They represent the i-th class of class c in the training set, respectively. c The source domain sample and the j-th sample c Multi-level fusion features of source domain samples, i c ≠j c , They represent the a-th class of class c in the training set, respectively. c The multi-level fusion features of the b-th target domain sample and the bc-th target domain sample, a c ≠b c κ(·) represents the Gaussian kernel function. κ(·) can be expressed as:

[0097]

[0098] Where x and y represent any two samples, and σ > 0 represents the bandwidth of the Gaussian kernel.

[0099] Assume F i and F j The module inputs sample features that have two adjacent relationships in the feature space. Based on the local discriminant projection preservation mechanism, to ensure that the low-dimensional sample features after projection retain this adjacency relationship, a manifold projection regularization constraint is set. The loss of the manifold reconstruction module is then the loss of the manifold reconstruction module. The loss is represented as:

[0100]

[0101] Let i and j represent the low-dimensional reconstruction features of the i-th and j-th source domain samples obtained by the joint manifold alignment reconstruction module, respectively, where i ≠ j. Let W represent the low-dimensional reconstruction features of the a-th and j-th target domain samples obtained by the joint manifold alignment reconstruction module, respectively, where a≠b. ijF represents the multi-level fusion feature of the i-th source domain sample and the j-th source domain sample in the original feature space. i and F j The intrinsic nearest neighbor matrix between them, W ab F represents the multi-level fusion features of the a-th target domain sample and the b-th target domain sample in the original feature space. a and F b The inherent nearest neighbor matrix between them.

[0102] W ij It can be represented as:

[0103]

[0104] Where t represents the thermonuclear parameter.

[0105] W ab The same formula as equation (8) is used for calculation.

[0106] Inter-domain local alignment loss of sub-neighborhood samples in the common feature subspace of the joint manifold The loss of establishing the subdomain manifold joint alignment reconstruction module is:

[0107]

[0108] The principle of the subdomain joint manifold alignment and reconstruction module is as follows: Figure 3 As shown. Figure 3 middle, They represent subdomains from the source domain within the manifold space, respectively. and The fusion feature projection. They represent subdomains from the source domain within the manifold space, respectively. and The fusion feature projection. F sr and F tr These represent the low-dimensional reconstructed feature instances from the source and target domains, respectively, after manifold projection.

[0109] Because of the alignment of sub-domain cluster centers and distributions, related sub-domains of the same category in the source and target domains are drawn closer together. Simultaneously, through the manifold reconstruction layer, information redundancy is eliminated while preserving the domain structure of features. Therefore, sample instances in the target domain can be guided by the labels of sample instances in the source domain for classification, thus achieving effective classification of unlabeled source domain data. To train the network's classification ability, for source domain data, based on its own labels... and the predicted probability of network output The cross-entropy classification loss is established and expressed as:

[0110]

[0111] Therefore, the overall loss function of the proposed algorithm can be written as:

[0112]

[0113] Where λ1 and λ2 are the alignment losses of the fused feature sub-domains, respectively. and manifold alignment reconstruction loss The weight.

[0114] Since the pathological tissue image is segmented into multiple slices before being input into the network, the classification results of all image slices are summed by probability to reconstruct the classification result of the entire pathological image, which can be expressed as:

[0115]

[0116] Where Y pic This represents the classification prediction result for the entire histopathological image, where M represents the number of slices into which an image is divided. represents the predicted probability of the m-th slice, and c represents the class label.

[0117] The specific algorithm flow of JSMA-UDA is shown in Table 1 below.

[0118] Table 1

[0119]

[0120]

[0121] Based on Table 1, the steps for training and testing the image classification model can be summarized as follows:

[0122] 1) Pre-train the BreNet network using the ImageNet dataset;

[0123] 2) Transfer the feature extraction layer and its corresponding parameters of the BreNet network, add a feature fusion layer to build a feature extraction fusion module, build a classifier and feature extraction fusion module to construct an image classification model, and then initialize the fully connected layer of the classifier;

[0124] 3) Divide the source domain dataset and the target domain dataset into training and test sets according to the label categories respectively;

[0125] 4) Initialize the cluster centers of the source domain and target domain sub-domain samples for each label in the training set;

[0126] 5) Enter the iteration; in each iteration, execute:

[0127] ① The feature extraction and fusion module is used to extract multi-level fusion features from each subdomain of the input samples;

[0128] ② Calculate the sub-domain alignment loss of the fusion features based on the calculated multi-level fusion features.

[0129] ③ Map the fusion features of the source and target domains onto the common subspace of the manifold;

[0130] ④ Calculate and update the cluster centers for each subdomain;

[0131] ⑤ Align the updated class centers of each subdomain and align the distribution of each subdomain based on the local maximum mean difference mechanism;

[0132] ⑥ Based on the feature projection matrix W P The low-dimensional reconstruction features of each source domain sample and target domain sample are calculated in step ①, along with the multi-level fusion features calculated in step ①.

[0133] ⑦ Calculate manifold alignment reconstruction loss based on computational multi-level fusion features, low-dimensional reconstruction features, and cluster centers of various sub-domains.

[0134] ⑧ Feed the low-dimensional reconstructed features of each source domain sample and target domain sample into the classifier; calculate the cross-entropy classification loss based on the classifier's output.

[0135] ⑨ Alignment loss based on weights λ1, λ2 and the calculated fusion feature sub-domain. Manifold alignment reconstruction loss Cross-entropy classification loss Calculate total loss

[0136] ⑩ Based on the calculated total loss Backpropagation updates network parameters;

[0137] 6) Use the test set as input to perform classification tests on the trained image classification model.

[0138] This embodiment also provides a histopathological image classification system based on semantic relevance clustering, which is equipped with an image classification model generated by the above-mentioned histopathological image classification method based on joint manifold alignment reconstruction, for classifying any input histopathological image.

[0139] The following is an experimental analysis.

[0140] The network model built in the experiment was based on the hardware conditions of Intel@Xeon(R) CPU E5-2620 v4@2.10GHz×32 and GeForce GTX 2080Ti, and was implemented in the programming environment of Python 3.8 and PyTorch 2.0.1.

[0141] During model training, the number of iterations was set to 50, the batch size of the training dataset was set to 23, and adaptive moment estimation (Adam) was used to optimize the network parameters. ReLU was used as the activation function, and the dropout ratio of the fully connected layers in the classifier was set to 0.5. The initial learning rate lr was 0.0001 and will be adaptively adjusted as the number of training iterations changes, lr ← lr × γ. p , where p represents the number of training iterations, and γ = 0.95 is the training learning rate iteration parameter.

[0142] During the experiment, two breast cancer histopathology image datasets, the public dataset BreakHis and the private dataset SNL, were used to evaluate the proposed JMSA-UDA method. The public dataset BreakHis contains 7909 breast tissue pathology sections, which consist of four magnifications. To match the image size of the SNL dataset, we selected images with a magnification of 200× from the BreakHis dataset (recorded as BreakHis_200×). BreakHis_200× contains 623 benign images and 1390 malignant images. All images were prepared by staining breast tissue biopsy sections with hematoxylin and eosin (H&E) stains, and the tissue sections were manually observed and analyzed under a microscope by pathologists to obtain image category labels. Detailed information on the number of images is shown in Table 2.

[0143] Table 2. Basic information about the dataset used in the experiment.

[0144]

[0145] To verify the effectiveness of the JMSA-UDA algorithm, this example conducted domain adaptation experiments in two directions: BreakHis_200× to SNL and SNL to BreakHis_200×. The experimental results are shown in Table 3. As can be seen from Table 3, the accuracy reached 88.80% in the BreakHis_200× to SNL direction, and 83.13% in the SNL to BreakHis_200× direction. The algorithm achieved good classification results in both directions.

[0146] Table 3 Experimental Results

[0147]

[0148] In summary, the histopathological image classification method and system based on joint manifold alignment reconstruction provided by this invention utilizes multi-layer feature extraction and fusion based on the BreNet deep transfer model pre-trained on the ImageNet natural dataset. It integrates the manifold space mapping distribution of the fused feature sub-domains with joint alignment of cluster centers, and performs image classification based on manifold reconstruction features. The system and method extract multi-order features through a feature extraction and fusion module and perform global pooling fusion to obtain deeply fused features. These features are then input into the joint manifold alignment reconstruction module, which projects them onto the common subspace of the manifold. Sub-domain distribution alignment and cluster center alignment minimize cross-domain sub-domain distribution differences, achieving domain adaptation. Simultaneously, the features are projected into a low-dimensional space to eliminate information redundancy, retaining key information effective for classification and further improving classification accuracy.

[0149] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A histopathological image classification method based on joint manifold alignment reconstruction, characterized in that, Including the following steps: S1. Construct source domain dataset and target domain dataset, wherein the source domain dataset includes multiple labeled source domain histopathological image samples, i.e., source domain samples, and the target domain dataset includes multiple unlabeled target domain histopathological image samples, i.e., target domain samples. S2. Generate pseudo-labels for unlabeled target domain samples to obtain labeled target domain samples, and divide the source domain and target domain samples into multiple source domain sub-domain datasets and multiple target domain sub-domain datasets according to the labels. S3. Construct an image classification model, which includes a feature extraction and fusion module, a joint manifold alignment and reconstruction module, and a classifier; The feature extraction and fusion module is used to extract multi-level fusion features from the input samples. The joint manifold alignment and reconstruction module operates by first mapping the multi-level fusion features to a common subspace of the manifold, and then, within this common subspace, jointly aligning the distributions of similar sub-domains of the source and target domains and aligning the sub-domain cluster centers between the domains. This minimizes the differences in the distributions of the same category while matching the global distribution, thereby effectively transferring labeled source domain data information to unlabeled target domain data and eliminating information redundancy between multi-level features, ultimately obtaining low-dimensional reconstructed features for cross-domain classification prediction. The classifier is used to output the corresponding classification prediction result based on the low-dimensional reconstructed features of each sample; S4. Divide multiple source domain subdomain datasets and multiple target domain subdomain datasets into training sets and test sets, and use the training set to train the image classification model, and use the test set to test the trained image classification model. S5. Use the trained image classification model to classify the input target domain histopathological image and output the classification result.

2. The histopathological image classification method based on joint manifold alignment reconstruction according to claim 1, characterized in that: During training, the total loss function used includes the cross-entropy classification loss calculated based on the labeled source domain samples and their classification prediction probabilities. The sub-domain alignment loss of the fusion features is calculated based on the distribution differences of the fusion features of the sub-domain samples output by the feature extraction and fusion module. and the manifold alignment reconstruction loss of the joint manifold alignment reconstruction module. .

3. The histopathological image classification method based on joint manifold alignment reconstruction according to claim 2, characterized in that, Manifold alignment reconstruction loss Build as: , in, This represents the manifold reconstruction loss calculated based on the input and output of the joint manifold alignment and reconstruction module. This represents the distribution loss calculated based on the cluster centers of the source and target domain sub-domains. This represents the sub-domain alignment loss of the fusion features calculated based on adjacent fusion features in the source domain sub-domain, adjacent fusion features in the target domain sub-domain, and adjacent fusion features in both the source and target domain sub-domains. and for and The weighting coefficients.

4. The histopathological image classification method based on joint manifold alignment reconstruction according to claim 3, characterized in that, Manifold Reconstruction Loss Calculated by the following formula: , in, and These represent the number of source domain samples and target domain samples in the training set, respectively. , These represent the first and second parts obtained by the joint manifold alignment and reconstruction module, respectively. The source domain sample and the first Low-dimensional reconstruction features of each source domain sample , These represent the first and second parts obtained by the joint manifold alignment and reconstruction module, respectively. The target domain sample and the first Low-dimensional reconstruction features of each target domain sample , Represents the first in the original feature space The source domain sample and the first In the multi-level fusion features of source domain samples and The intrinsic nearest neighbor matrix between them Represents the first in the original feature space The target domain sample and the first In the multi-level fusion features of target domain samples and The intrinsic nearest neighbor matrix between them This indicates the calculation of Euclidean distance; Low-dimensional reconstruction features of any source domain sample or target domain sample Calculated by the following formula: , in, For the characteristic projection matrix, It represents the multi-level fusion feature of any source domain sample or target domain sample.

5. The histopathological image classification method based on joint manifold alignment reconstruction according to claim 4, characterized in that, Inter-domain local alignment loss Calculated by the following formula: , in, The number of categories represents the number of subdomains of the source or target domain, where express Category labels for categorized tasks. , Representing the categories in the training set The number of source domain samples and target domain samples, , Representing the categories in the training set The The source domain sample and the first Multi-level fusion features of source domain samples , Representing the categories in the training set The The target domain sample and the first Multi-level fusion features of target domain samples , This represents the Gaussian kernel function.

6. The histopathological image classification method based on joint manifold alignment reconstruction according to claim 5, characterized in that, Distributed loss Calculated by the following formula: , in, and Representing categories within the source and target domains respectively. Cluster centers; Initial cluster centers of the source or target domain Calculated by the following formula: , in, Indicates the source or target domain in the category The number of samples on Indicates the source or target domain in the category The first One sample; During the training process, Updated by the following formula: , in, , They represent the first Wheel, First The updated labels are categories. Cluster centers of the source or target domain subdomains, This represents the average moving average.

7. The histopathological image classification method based on joint manifold alignment reconstruction according to claim 6, characterized in that, The total loss function used during training Designed as follows: , in, and The corresponding feature sub-domain alignment loss is as follows: and manifold alignment reconstruction loss The weights; Cross-entropy classification loss Calculated by the following formula: , in, The output of the image classification model represents the value of the image classification model. The predicted probability.

8. The histopathological image classification method based on joint manifold alignment reconstruction according to claim 7, characterized in that, The steps for training and testing the image classification model include: 1) Pre-train the BreNet network using the ImageNet dataset; 2) Transfer the feature extraction layer and its corresponding parameters of the BreNet network, add a feature fusion layer to construct a feature extraction and fusion module, establish a classifier and the feature extraction and fusion module to construct an image classification model, and then initialize the fully connected layer of the classifier; 3) Divide the source domain dataset and the target domain dataset into training and test sets according to the label categories respectively; 4) Initialize the cluster centers of the source domain and target domain sub-domain samples for each label in the training set; 5) Enter the iteration; in each iteration, execute: ① The feature extraction and fusion module is used to extract multi-level fusion features from each subdomain of the input samples; ② Calculate the sub-domain alignment loss of the fusion features based on the calculated multi-level fusion features. ; ③ Map the fusion features of the source and target domains onto the common subspace of the manifold; ④ Calculate and update the cluster centers for each subdomain; ⑤ Align the updated class centers of each subdomain and align the distribution of each subdomain based on the local maximum mean difference mechanism; ⑥ Based on the feature projection matrix The low-dimensional reconstruction features of each source domain sample and target domain sample are calculated in step ①, along with the multi-level fusion features calculated in step ①. ⑦ Calculate manifold alignment reconstruction loss based on computational multi-level fusion features, low-dimensional reconstruction features, and cluster centers of various sub-domains. ; ⑧ Feed the low-dimensional reconstructed features of each source domain sample and target domain sample into the classifier; calculate the cross-entropy classification loss based on the classifier's output. ; ⑨ Based on weight , And the calculated fusion feature sub-domain alignment loss Manifold alignment reconstruction loss Cross-entropy classification loss Calculate total loss ; ⑩ Based on the calculated total loss Backpropagation updates network parameters; 6) Use the test set as input to perform classification tests on the trained image classification model.

9. The histopathological image classification method based on joint manifold alignment reconstruction according to any one of claims 1 to 8, characterized in that, The operation of the feature extraction and fusion module is represented as follows: , in, Indicates sample Input neural network Feature extraction operations of layers; Indicates sample The The number of channels for feature extraction from a layer; This indicates a global average pooling operation. This represents the set of network layers selected for feature fusion, consisting of high, medium, and low-order features. Indicates to Cascade operations are performed on the first-order features.

10. A histopathological image classification system based on semantic relevance clustering, characterized in that: It is equipped with an image classification model generated using the histopathological image classification method based on joint manifold alignment reconstruction as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Cross-domain image classification method and system based on fine-grained domain adaptation

    CN111259941A

  • Histopathological image classification method based on depth class-sample double-layer cross-domain adaptation

    CN118154966A