Pathological image form classification method and equipment based on optimal transmission prototype network

Abstract

The invention relates to a pathological image morphological subtype classification method based on an optimal transmission prototype network, and belongs to the technical field of medical image processing. The method comprises the steps of constructing a pathological image data set, performing preliminary grouping by using an unsupervised clustering method, and performing cooperation with a pathologist to optimize the data set; preprocessing the image through a data enhancement strategy; performing feature extraction on the enhanced image through a pre-training model; the method comprises the following steps of: constructing and training an OT-ProtoNet model; setting a prototype, determining dimensions and quantity, realizing a category specific prototype based on a soft distribution mechanism, calculating similarity based on an OT distance, setting training parameters and selecting an optimization algorithm to train a model, and ensuring that the model effectively learns key features in data; and carrying out image morphological subtype classification on the basis of an OT-ProtoNet model. According to the pathological image morphological subtype classification method, the pre-training features can be fully utilized, the classification accuracy can be improved, and interpretability can be kept.

Description

Technical Field

[0001] The present invention relates to a pathological image morphological subtype classification method and device based on an optimal transport (OT) distance prototype network (OT-Prototype Network, OT-ProtoNet) model, belonging to the technical field of medical image processing. Background Art

[0002] In recent years, with the rapid development of deep learning technology, feature extraction methods based on large-scale pathology image pre-training models have been widely used in the field of pathology image classification. These pre-training models can effectively extract rich features from high-resolution whole slide images (WSI). However, when using these pre-trained features for downstream classification tasks, existing classification models still face the following limitations: 1. Insufficient feature utilization efficiency: Although traditional machine learning methods (such as logistic regression) are computationally efficient, they are difficult to fully utilize the expressive power of deep features. Although deep neural network models (such as multilayer perceptrons) can learn complex nonlinear relationships, they are prone to overfitting when training data is limited. 2. Limitations of distance metrics: Although prototype methods based on Euclidean distance (such as ProtoNet) have certain interpretability, they may not accurately capture complex relationships in high-dimensional feature space based on Euclidean distance, resulting in decreased classification accuracy. 3. Insufficient interpretability and generalization ability: Existing methods often lack sufficient interpretability and are difficult to directly associate with the diagnostic experience of pathology experts, especially when dealing with rare or difficult-to-distinguish tumor subtypes. The classification accuracy of these methods still needs to be improved. Therefore, there is an urgent need in this technical field to develop a method for classifying pathological image morphological subtypes that can fully utilize pre-trained features, improve classification accuracy, and maintain interpretability. Summary of the invention

[0003] The purpose of the present invention is to solve the technical problem of how to obtain a pathological image morphological subtype classification method that can fully utilize pre-training features, improve classification accuracy, and maintain interpretability.

[0004] In order to achieve the purpose of solving the above problems, the technical solution adopted by the present invention is to provide a pathological image morphological subtype classification method based on the optimal transmission prototype network OT-ProtoNet, comprising the following steps:

[0005] Step 1: Construct a pathology image dataset; this includes preliminary grouping using unsupervised clustering methods and collaborating with pathologists to optimize the dataset to ensure clinical relevance and morphological consistency of the dataset;

[0006] Step 2: Data preprocessing and enhancement: Preprocess the image by adopting data enhancement strategy and extract features through pre-trained model to obtain high-quality image representation;

[0007] Step 3: Construct the OT-ProtoNet model, including setting up prototypes, determining dimensions and numbers, implementing category-specific prototypes based on the soft assignment mechanism, and calculating similarity based on OT distance;

[0008] Step 4: Train the OT-ProtoNet model and set key training parameters; select the optimization algorithm and loss function to ensure that the model effectively learns the key features in the data;

[0009] Step 5: Pathological images are classified into morphological subtypes based on the OT-ProtoNet model.

[0010] Preferably, the data enhancement strategy in step 2 includes random image flipping, color jittering, random image erasing and / or Mixup strategy.

[0011] Preferably, constructing the OT-ProtoNet model in step 3 includes learning shared prototypes shared_prototypes, denoted as sproto, and specific weights class_weights for each class, denoted as w;

[0012] The model includes the following parameters: the number of categories cn, the number of sampling points for each category sn, each sampling point is represented as a patch token, the feature dimension of each sampling point and the dimension of each prototype are dim, the number of shared prototypes proto, the shared prototype dimension is proto*dim, the category-specific prototype cproto is calculated by soft allocation of the shared prototype sproto, the category-specific prototype dimension is cn*sn*dim, and the category-specific prototype weight w is calculated by the weight dimension cn*sn*proto and the shared prototype. The calculation formula is:

[0013]

[0014] In the formula, p represents the shared prototype index, from 1 to the number of shared prototypes proto.

[0015] Preferably, the training parameters of the model training in step 4 are set as follows: the number of training rounds is 150 rounds, the batch size is 64 tiles, and the learning rate is 1e-3; the model training algorithm adopts the Adam optimization algorithm; and the loss function uses the negative log-likelihood loss.

[0016] Preferably, during the model training in step 4, in order to ensure balanced learning of the model in various categories, a random balanced sampling strategy is adopted for each round of training samples; during the training process, a certain number of spatial features are randomly selected for each image, and the number of sampling points is sn patch tokens for data enhancement.

[0017] Preferably, in the model training of step 4, in order to enhance the generalization ability of the model, a variety of data enhancement strategies are adopted, and the enhancement strategies include random image flipping, color jittering, random image erasing and / or Mixup strategy.

[0018] Preferably, when calculating the similarity in step 3, in order to capture the complex relationship in the high-dimensional feature space, the Sinkhorn distance, that is, an optimal transmission OT distance, is used to measure the similarity between each block and the category prototype; the calculated OT distance from the image to each category prototype is used as the output, and the category with the smallest distance is selected as the model classification result for subsequent classification decisions and loss calculations.

[0019] The present invention provides a storage device, which stores a plurality of instructions, and the instructions are used to execute the image classification work of the above-mentioned pathological image morphological subtype classification method based on the optimal transmission prototype network.

[0020] The present invention provides an embedded device, comprising:

[0021] A storage device for storing a computer program;

[0022] A processor is used to execute the computer program to implement the image classification work of the above-mentioned pathological image morphological subtype classification method based on the optimal transmission prototype network.

[0023] The present invention provides an application of a pathological image morphological subtype classification method based on an optimal transmission prototype network, and the application includes implementing an image classification work of the pathological image morphological subtype classification method based on an optimal transmission prototype network as described above.

[0024] The purpose of the present invention is to provide a prototype network (OT-Prototype Network, OT-ProtoNet) model based on optimal transport (OT) distance for morphological pattern classification of pathological images. It includes:

[0025] Prototype learning model based on optimal transmission distance: Improves classification accuracy by calculating the similarity between samples and prototypes using Sinkhorn distance;

[0026] Soft assignment mechanism for category-specific prototypes: The softmax function is used to ensure that the sum of the prototype weights of each category is 1, thereby achieving flexible category prototype generation;

[0027] Similarity calculation based on OT distance: OT distance is used to capture complex relationships in high-dimensional feature space and enhance the generalization ability and interpretability of the model.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. Improve feature utilization efficiency: The method of the present invention significantly improves feature utilization efficiency and can better capture subtle morphological information in the deep features extracted by the pre-trained model. By optimizing the feature extraction process, the model can effectively identify subtle changes and complex structures in the image, which is particularly important for pathological image analysis. This feature enables the model to more accurately identify and classify different tumor subtypes when processing complex pathological images, thereby providing a more reliable basis for clinical diagnosis.

[0030] 2. Enhance the generalization ability of the model: The similarity calculation based on OT distance enhances the generalization ability of the model, especially in the classification accuracy of rare or difficult-to-distinguish tumor subtypes. Compared with traditional methods, the present invention can effectively reduce the classification error rate and ensure that the model can still maintain efficient classification performance when facing unseen samples. This improvement in generalization ability enables the model to better adapt to different pathological image data in practical applications and reduces the risk of overfitting.

[0031] 3. Improve interpretability: The prototype-based learning method provides an intuitive similarity metric, which enhances the interpretability of the model. This advantage enables the output of the model to be better combined with the diagnostic experience of pathologists, thereby improving the credibility of clinical applications. By visualizing the similarity metric, pathologists can more easily understand the decision-making process of the model and make more accurate judgments in actual diagnosis.

[0032] 4. Reduced reliance on labeled data: The method of the present invention reduces the reliance on a large amount of labeled data while maintaining high classification performance through a soft allocation mechanism and an effective data enhancement strategy. This feature makes the method more adaptable and practical in practical applications, especially in areas where labeled data is scarce. By effectively utilizing unlabeled data, the model can still achieve good classification results with fewer labeled samples, reducing the cost and time of data annotation.

[0033] 5. Flexible adjustment and broad application prospects: The method of the present invention can be flexibly adjusted according to different pathological image data sets and classification tasks, and has broad application prospects. The data set construction method optimized by unsupervised clustering and cooperation with pathologists ensures the quality of experimental data and provides a reliable basis for model training and evaluation. This flexibility enables the present invention to adapt to different clinical needs and promote its application in multiple fields such as medical image analysis and tumor classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the implementation process of the present invention.

[0035] Figure 2 This is a diagram of the test results of the OT-ProtoNet model of the present invention.

[0036] Figure 3 This is a comparison chart of the OT-ProtoNet model of the present invention and other benchmark methods. DETAILED DESCRIPTION

[0037] In order to make the present invention more clearly understood, a preferred embodiment is described in detail with reference to the accompanying drawings as follows:

[0038] The present invention aims to solve the deficiencies in the prior art and provide a method for pathological image classification based on a prototype network with optimal transmission distance to achieve the following objectives:

[0039] 1. Improve feature utilization efficiency: By introducing the optimal transmission distance, the ability to capture subtle morphological information in the deep features extracted by the pre-trained model is enhanced.

[0040] 2. Enhance the generalization ability of the model: By optimizing the prototype learning mechanism, the classification accuracy of the model when dealing with rare or difficult-to-distinguish tumor subtypes can be improved, ensuring the stable performance of the model on different data sets.

[0041] 3. Reduce dependence on large amounts of labeled data: While maintaining high classification performance, through effective feature extraction and prototype generation methods, reduce the need for large amounts of labeled data and improve the adaptability of the model in practical applications.

[0042] To achieve the above object, the present invention provides a method for pathological image classification based on a prototype network (OT-PrototypeNetwork, OT-ProtoNet) based on an optimal transmission distance. The method mainly includes the following key steps:

[0043] 1. Feature extraction:

[0044] In the initial stage of model construction, the pathology image pre-trained deep learning model UNI based on the Vision Transformer (ViT) model architecture is used to extract high-dimensional features of the input pathology image. The model architecture of UNI is ViT-Large, the patch_size of ViT is set to 16, and the extracted feature dimension is set to 1024. The input image is a whole slice image (WSI) cut into non-overlapping blocks of 256x256 pixels at 10x magnification.

[0045] 2. Model construction:

[0046] The core of building the OT-ProtoNet model is to learn the shared prototypes shared_prototypes, denoted as sproto, and the specific weights class_weights for each category, denoted as w.

[0047] The model includes the following parameters: the number of categories cn, the number of sampling points for each category sn, each sampling point is represented as a patch token, the feature dimension of each sampling point and the dimension of each prototype are dim, the number of shared prototypes proto, the shared prototype dimension is proto*dim, the category-specific prototype cproto is calculated by soft allocation of the shared prototype sproto, the category-specific prototype dimension is cn*sn*dim, and is calculated by the category-specific prototype weight w (the weight dimension is cn*sn*proto) and the shared prototype. The calculation formula is:

[0048]

[0049] In the formula, p represents the shared prototype index (from 1 to the number of shared prototypes proto)

[0050] 3. Training settings:

[0051] During the model training process, a series of effective strategies were adopted. The training parameters were set to 150 rounds, the batch size was 64 tiles, the learning rate was 1e-3, and the Adam optimization algorithm was used. The loss function used was the negative log-likelihood loss, which aims to minimize the difference between the model prediction and the true label.

[0052] 3.1. Random balanced sampling:

[0053] To ensure balanced learning of the model across categories, a random balanced sampling strategy is used for each round of training samples. Specifically, in each batch, the same number of samples are randomly selected from each category to avoid the impact of category imbalance on model training.

[0054] 3.2.Data enhancement method:

[0055] In order to enhance the generalization ability of the model, a variety of data augmentation strategies are implemented, including: random image flipping (horizontally and vertically), color jittering (adjusting brightness, contrast, saturation and hue), random image erasing (randomly blocking part of the image), and Mixup strategy (linearly combining two images and their labels). Specifically, the Mixup strategy enhances the robustness of the model to inter-sample variation by linearly combining the features and labels of different samples. The specific implementation is: given a pair of samples (x a ,y a ) and (x b ,y b ), generate a mixture ratio λ through Beta distribution, and calculate the mixed sample as:

[0056] x mixed =λx a +(1-λ)x b

[0057] 3.3. Randomly select features:

[0058] During the training process, a certain number of spatial features (sampling points sn patchtokens) are randomly selected for each image for data augmentation. In the specific implementation, a certain number of features are randomly selected from the extracted high-dimensional features to increase the diversity and complexity of training. In the inference stage, in order to make full use of all the information learned by the model during training, all available spatial features are selected for processing.

[0059] 4. Distance Metrics:

[0060] Before calculating the distance, the input features and the category prototype are first L2 normalized to ensure that they are compared on the same scale. In the similarity calculation, in order to capture the complex relationship in the high-dimensional feature space, the Sinkhorn distance, an optimal transport (OT) distance, is innovatively used to measure the similarity between each tile and the category prototype. For the feature matrix, the Sinkhorn distance is calculated by the following steps:

[0061] 1. Cost matrix calculation. The cost matrix C is calculated based on the distance between the feature matrix of the tile and the category prototype. For a tile M and a category prototype P, the kth token of M and the lth token of P, the cost C kl is the Euclidean distance between their eigenvectors:

[0062] C kl =|M k -P l | 2

[0063] 2. OT distance calculation. Using the cost matrix C and the regularization parameter ∈, the OT distance is calculated by the Sinkhorn-Knopp method. The goal is to find a transportation matrix π that minimizes the regularized transportation cost:

[0064]

[0065] where U(r,c) represents the set of transport matrices with row and column sums equal to r and c respectively, chosen to be uniformly distributed. The regularization parameter ∈ is set to 0.05.

[0066] The calculated OT distance from the image to each category prototype is used as the output, and the category with the smallest distance is selected as the model classification result for subsequent classification decisions and loss calculations.

[0067] 5. Loss function calculation:

[0068] During the model training process, the loss function is calculated as classification loss, which is specifically implemented by using negative log-likelihood loss (NLL Loss) to measure the difference between the model prediction and the true label. Combined with the Mixup strategy to generate mixed samples,

[0069] x mixed =λx a +(1-λ)x b

[0070] , the specific calculation process is as follows:

[0071] log_p = log(softmax(-d))

[0072] Loss = λ·NLL_L(log_p,y a )+(1-λ)·NLL_L(log_p,y b )

[0073] Where d represents the OT distance from the calculated output image to each category prototype, log_p represents the log probability, NLL_L represents the negative log likelihood loss, and λ represents the mixing ratio.

[0074] 6. Model Evaluation and Comparison

[0075] The present invention uses a carefully constructed pathology image dataset for experimental validation. This dataset was obtained through unsupervised clustering and optimization process in collaboration with pathologists, ensuring clinical relevance and morphological consistency, providing a reliable basis for model training and evaluation.

[0076] This paper compares OT-ProtoNet with a variety of benchmark methods, including voting method (Vote@5), traditional ProtoNet, logistic regression (LR), and multi-layer perceptron classifier (MLP+LR). These comparative experiments fully demonstrate the superiority of OT-ProtoNet in pathological image classification tasks. Figure 3 .

[0077] Example

[0078] The present invention provides a pathological image morphological subtype classification method based on an optimal transmission prototype network, such as Figure 1-3 As shown, the method comprises the following steps:

[0079] 1. Construct a pathology image dataset, including initial grouping using unsupervised clustering methods, followed by optimization in collaboration with pathologists to ensure the clinical relevance and morphological consistency of the dataset.

[0080] 2. Preprocess and enhance the training set data. A variety of data enhancement strategies are used, including random image flipping, color jittering, random image erasing, and Mixup strategies. The enhanced images are subjected to feature extraction using a large-scale pathology image pre-training model to obtain high-quality image representation.

[0081] 3. Build the OT-ProtoNet model. This includes setting the prototype dimension and number, implementing a soft assignment mechanism for category-specific prototypes, and a similarity calculation method based on OT distance. These designs are intended to improve the model’s expressiveness and classification performance.

[0082] 4. During the model training phase, set key training parameters, such as the number of training rounds, batch size, and learning rate. Select appropriate optimization algorithms and loss functions to ensure that the model can effectively learn the key features in the data.

[0083] 5. Model evaluation and comparison. OT-ProtoNet is compared with multiple baseline methods, including Vote@5, traditional ProtoNet, logistic regression (LR), and multi-layer perceptron classifier (MLP+LR). Through these comparisons, the performance advantages of OT-ProtoNet in pathological image classification tasks are fully evaluated. Figure 3 shown.

[0084] The above is only a preferred embodiment of the present invention, and is not any formal or substantial limitation of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the present invention, and these improvements and supplements should also be regarded as the protection scope of the present invention. Any technician familiar with this profession, without departing from the spirit and scope of the present invention, can make some changes, modifications and equivalent changes made by using the technical content disclosed above, which are equivalent embodiments of the present invention; at the same time, any changes, modifications and evolutions of any equivalent changes made to the above embodiments based on the essential technology of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A pathological image morphological subtype classification method based on an optimal transmission prototype network, characterized in that: The following steps are involved: Step 1: Construct a pathology image dataset; this includes preliminary grouping using unsupervised clustering methods and collaborating with pathologists to optimize the dataset to ensure clinical relevance and morphological consistency of the dataset; Step 2: Data preprocessing and enhancement: Preprocess the image by adopting data enhancement strategy and extract features through pre-trained model; Step 3: Construct the OT-ProtoNet model, including setting up prototypes, determining dimensions and numbers, implementing category-specific prototypes based on the soft assignment mechanism, and calculating similarity based on OT distance; Step 4: Train the OT-ProtoNet model and set key training parameters; select the optimization algorithm and loss function to ensure that the model effectively learns the key features in the data; Step 5: Pathological images are classified into morphological subtypes based on the OT-ProtoNet model.

2. According to the method for classifying pathological image morphological subtypes based on the optimal transmission prototype network according to claim 1, it is characterized in that: The data enhancement strategy in step 2 includes random image flipping, color jittering, random image erasing and / or Mixup strategy.

3. The method for pathological image morphological subtype classification based on an optimal transmission prototype network according to claim 1, characterized in that: The construction of the OT-ProtoNet model in step 3 includes learning shared prototypes shared_prototypes, denoted as sproto, and specific weights class_weights for each category, denoted as w; The model includes the following parameters: the number of categories cn, the number of sampling points for each category sn, each sampling point is represented as a patch token, the feature dimension of each sampling point and the dimension of each prototype are dim, the number of shared prototypes proto, the shared prototype dimension is proto*dim, the category-specific prototype cproto is calculated by soft allocation of the shared prototype sproto, the category-specific prototype dimension is cn*sn*dim, and the category-specific prototype weight w is calculated by the weight dimension cn*sn*proto and the shared prototype. The calculation formula is: In the formula, p represents the shared prototype index, from 1 to the number of shared prototypes proto.

4. The method for pathological image morphological subtype classification based on an optimal transmission prototype network according to claim 1, characterized in that: The training parameters of the model training in step 4 are set as follows: the number of training rounds is 150 rounds, the batch size is 64 tiles, and the learning rate is 1e-3; the model training algorithm adopts the Adam optimization algorithm; and the loss function selects the negative log-likelihood loss.

5. The method for classifying pathological image morphological subtypes based on an optimal transmission prototype network according to claim 1, characterized in that: During the model training in step 4, in order to ensure balanced learning of the model in each category, a random balanced sampling strategy is adopted for each round of training samples; during the training process, a certain number of spatial features are randomly selected for each image, and the number of sampling points is sn patch tokens for data enhancement.

6. The method for classifying pathological image morphological subtypes based on an optimal transmission prototype network according to claim 1, characterized in that: In the step 4, a variety of data enhancement strategies are used in the model training to enhance the generalization ability of the model. The enhancement strategies include random image flipping, color jittering, random image erasing and / or Mixup strategy.

7. The method for classifying pathological image morphological subtypes based on an optimal transmission prototype network according to claim 1, characterized in that: When calculating the similarity in step 3, in order to capture the complex relationship in the high-dimensional feature space, the Sinkhorn distance, that is, an optimal transmission OT distance, is used to measure the similarity between each block and the category prototype; the calculated OT distance from the image to each category prototype is used as the output, and the category with the smallest distance is selected as the model classification result for subsequent classification decisions and loss calculations.

8. A storage device, characterized in that: The storage device stores a plurality of instructions, and the instructions are used to execute the image classification work of a pathological image morphological subtype classification method based on an optimal transmission prototype network as described in any one of claims 1-7.

9. An embedded device, characterized in that: include: A storage device for storing a computer program; A processor is used to execute the computer program to implement the image classification work of the pathological image morphological subtype classification method based on the optimal transmission prototype network as described in any one of claims 1-7.

10. An application of a pathological image morphological subtype classification method based on an optimal transmission prototype network, characterized in that: The application includes implementing an image classification work of a pathological image morphological subtype classification method based on an optimal transmission prototype network as described in any one of claims 1-7.

Images