Automatic diagnosis method for thyroid pathology image

Abstract

The invention discloses an automatic diagnosis method of a thyroid pathological image, and belongs to the technical field of medical pathological image auxiliary intelligent diagnosis, and the method comprises the following steps: constructing a thyroid tumor pathological image data set, preprocessing a pathological image, carrying out multi-scale fusion, carrying out feature extraction, carrying out feature fusion, inputting a CLAM-SB multi-instance learning model, and generating a visual heat map. The invention provides a thyroid pathology image automatic diagnosis method based on weak supervision multi-instance learning and multi-scale feature fusion, which can effectively improve the accuracy of pathology tissue region segmentation and pathology image classification, reduce the workload of pathologists and improve the diagnosis efficiency and reliability.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical pathology image-assisted intelligent diagnosis, and in particular relates to an automatic diagnosis method for thyroid pathology images. Background Art

[0002] The incidence of thyroid cancer has rapidly increased in recent years, making early and accurate diagnosis crucial for patient treatment selection and prognosis assessment. Pathological diagnosis, the gold standard for diagnosing thyroid cancer, remains a risk of missed and misdiagnosed cases due to factors such as a shortage of pathologists at primary care hospitals and varying film preparation techniques.

[0003] Furthermore, traditional pathology image analysis methods rely heavily on manual experience, which can be highly subjective, inefficient, and poorly reproducible. Therefore, there is an urgent need for an intelligent diagnostic method that can assist pathologists in diagnosis and improve the efficiency and accuracy of pathology image analysis.

[0004] In recent years, deep learning technology has been widely used in the field of pathology image analysis and has made significant progress in the automatic classification of whole slide pathology digital images (WSI). However, existing technologies still have the following problems and limitations:

[0005] (1) Difficulty in labeling pathological images: Pathological images have high resolution and complex tissue structures. Accurate labeling is time-consuming and labor-intensive, and requires reliance on pathology experts, making it difficult to obtain training data.

[0006] (2) Traditional tissue region segmentation methods are not effective: Traditional rule-based or traditional machine learning tissue region segmentation methods have limited generalization capabilities in complex pathological images, making it difficult to accurately segment pathological tissue regions, affecting the reliability of subsequent analysis.

[0007] (3) Limitations of single-scale scanning: Existing pathology image analysis models usually perform feature extraction based on a single magnification, which limits the ability to capture subtle features at different scales and affects classification performance.

[0008] (4) Limitations of single feature extraction models: Current deep learning methods mainly rely on a single feature extractor, while pathological images are rich in information. A single model cannot fully utilize multi-level and cross-domain image features, resulting in insufficient generalization and discrimination capabilities of the model. Summary of the Invention

[0009] In order to solve the above problems, the present invention provides an automatic diagnosis method for thyroid pathology images based on weakly supervised multi-instance learning and multi-scale feature fusion, which can effectively improve the accuracy of pathological tissue region segmentation and pathological image classification, reduce the workload of pathologists, and improve diagnostic efficiency and reliability.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] An automatic diagnosis method for thyroid pathology images comprises the following steps:

[0012] S1. Construct a thyroid tumor pathology image dataset;

[0013] S2. Obtain several pathological images from the thyroid tumor pathology image dataset and preprocess the pathological images. The preprocessing includes tissue region segmentation, staining normalization, and data enhancement.

[0014] S3, using feature pyramid to perform multi-scale fusion on pre-processed pathological images;

[0015] S4. Use RetCCL, iBOT, and DINO feature extractors to extract features from multi-scale fused pathological images;

[0016] S5. For the pathological image after feature extraction, feature fusion is performed using a parallel fusion method;

[0017] S6. Input the pathological image after feature fusion into the CLAM-SB multi-instance learning model, classify the pathological image into benign or malignant, and generate a visual heat map.

[0018] Furthermore, in step S2, the method for segmenting the tissue region includes the following steps:

[0019] (1) Construct a dataset for training a tissue region segmentation model based on the AFFormer architecture;

[0020] (2) Pixel enhancement and spatial enhancement: Based on the MedAugment strategy, pixel enhancement and spatial enhancement are performed on the pathological images of the constructed dataset, and a hyperparameter mapping relationship is introduced;

[0021] (3) Model training: The data set after data augmentation is divided into a training set and a validation set. The image blocks of set pixels are cut out from the pathological image through the image cutting training method. The image blocks in the boundary area are selected and input into the tissue area segmentation model for training. At the same time, a balanced sampling strategy is adopted to strengthen the sampling of image blocks near the boundary.

[0022] Furthermore, in step S2, constructing a dataset when segmenting tissue regions includes the following steps:

[0023] A. Apply OTSU algorithm to perform tissue region segmentation on pathological images;

[0024] B. After manually eliminating images with unsatisfactory segmentation results, the remaining images were re-segmented using the HistoQC algorithm and manually screened again;

[0025] C. PaddleSeg is used to manually annotate the tissue areas of the above images with poor segmentation effects, thus forming a dataset consisting of segmented masks and WSI original images.

[0026] Furthermore, in step S2, the loss function of the tissue region segmentation model used in model training is composed of the superposition of label smoothed cross entropy loss and Diceloss. The specific calculation formula is as follows:

[0027] l loss =l ls-ce +2l dice ;

[0028] The label smoothed cross entropy loss is calculated as follows:

[0029]

[0030] Among them, ce is the cross entropy loss, ε represents a small positive number, and β is the weight used to adjust the label smoothed cross entropy loss as a hyperparameter;

[0031] The calculation formula of DiceLoss is as follows:

[0032]

[0033] Among them, |X∩Y| represents the intersection between the true result X and the predicted result Y, and |X| and |Y| represent the number of elements in X and Y respectively.

[0034] Furthermore, in step S2, an automatic color vector positioning algorithm is used in the color normalization process, and image processing is performed in combination with a color deconvolution method. The specific steps are as follows:

[0035] (1) Convert the RGB color space to the optical density (OD) space and standardize the OD value to the range of [0,1]. The calculation formula of OD is: OD = -log 10 I, where I is an RGB color vector;

[0036] (2) Delete the OD values ​​less than the threshold β, remove the transparent white background, and obtain the main tissue image;

[0037] (3) Find the direction of the geodesic path, perform SVD decomposition based on the OD value, and obtain the singular value. The calculation formula is: OD = V*S, where V and S are the matrix of the coloring vector and the saturation of each coloring, respectively;

[0038] (4) The vectors corresponding to the two largest singular values ​​of the OD transformed pixel are decomposed by SVD to form an evaluation plane. All optical density values ​​are projected onto the plane and normalized.

[0039] (5) Calculate the angle between each point on the curve and the first SVD direction, and map the direction on the plane into a scalar;

[0040] (6) Determine the boundary value of the staining vector by calculating the minimum and maximum percentiles of the staining agent;

[0041] (7) Use OD values ​​to determine the staining vector of each stain and normalize the staining intensity;

[0042] (8) Use the staining concentration matrix to recreate the source image so that it has staining intensity similar to the target image.

[0043] Furthermore, in step S2, during the data enhancement process, any one of the five data enhancement methods, image compression, random rotation, random gamma correction, color adjustment, and gradient brightness, is randomly selected for processing, and each image enhancement method has the same probability of being selected.

[0044] Furthermore, in step S3, a feature pyramid method is used to splice and fuse the features extracted from the original 20-fold magnification image block and the features extracted from the corresponding 10-fold scaled 256*256 image block.

[0045] Furthermore, in step S4, during feature extraction, feature extractors iBOT and DINO are used to extract the feature vector of each image block, and the 1536-dimensional feature vector of feature extractor iBOT and the 768-dimensional feature vector of feature extractor DINO are expanded to 4096 dimensions by applying the method of padding with 0, so that they have the same feature length as the 4096-dimensional feature vector extracted by feature extractor RetCCL.

[0046] Furthermore, in step S5, during feature fusion, the three sets of feature vectors corresponding to each image block that have undergone feature extraction processing are fused into a single feature vector through a concat method.

[0047] Furthermore, step S6 specifically includes the following steps:

[0048] (1) Downsampling and tissue segmentation: After downsampling and tissue region segmentation, the input image is cut into small image blocks of 256*256;

[0049] (2) Use the ResNet50 model pre-trained on ImageNet to convert each small image patch into a 1024-dimensional feature vector and compress it to 512 dimensions through a fully connected layer;

[0050] (3) Attention pooling: Based on the gated attention mechanism, attention scores are calculated for each small image patch, and features are aggregated to generate full-slice-level representations;

[0051] (4) Clustering constraints: a binary clustering layer is introduced to optimize the feature space using pseudo labels to ensure that positive and negative features are linearly separable.

[0052] (5) Optimize the loss function by combining the image-level cross entropy loss and the image block-level smoothing support vector machine loss;

[0053] (6) Generate and output a visualization heat map.

[0054] Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0055] 1. This invention can effectively reduce manual labeling costs and improve training efficiency: through the weakly supervised multi-instance learning method, only image-level labels are used for training, which reduces the dependence on finely labeled data and significantly reduces data labeling costs.

[0056] 2. The present invention can effectively improve the accuracy of pathological tissue region segmentation: by building a self-built pathological tissue region segmentation dataset and combining it with automatic data enhancement technology for medical images, a new lightweight semantic segmentation algorithm was successfully trained. Unlike traditional methods of tissue region segmentation, this algorithm does not require tedious manual parameter adjustment and can effectively eliminate stain interference in the image. It achieves the ability to quickly and accurately locate and segment tissue regions in thyroid tumor pathology image datasets, providing strong support for subsequent analysis and application.

[0057] 3. The present invention can effectively enhance the multi-scale feature extraction capability of the classification model: by fusing multi-scale pathological image information through a pyramid cascade strategy, the model can learn the characteristics of the lesion area more comprehensively and improve the classification accuracy.

[0058] 4. This invention breaks through the limitations of a single feature extraction model: it integrates multiple in-domain feature extractors to achieve effective combination of multi-level information, enhance the model's discrimination and generalization capabilities, and improve the pathological image classification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a technical flow chart of the present invention;

[0060] Figure 2 is a flow chart of constructing a data set during the process of segmenting tissue regions of the present invention;

[0061] Figure 3 This is the effect diagram after image enhancement during the process of segmenting tissue regions in the present invention;

[0062] Figure 4This is a block training visual field diagram during the process of segmenting tissue regions in the present invention; wherein, Figure 4 (a) is the 2.5x field of view, Figure 4 (b) is 5x field of view, Figure 4 (c) is the 10x field of view, Figure 4 (d) is the 20x field of view;

[0063] Figure 5 Schematic diagram of the retraining process of the tissue region segmentation method based on AFFormer of the present invention;

[0064] Figure 6 This is a comparison between the tissue region segmentation effect based on AFFormer of the present invention and the segmentation effect of the traditional OTSU algorithm, where: Figure 6 (a) is the segmentation effect diagram of the traditional OTSU algorithm. Figure 6 (b) is a diagram showing the segmentation effect of the present invention;

[0065] Figure 7 Schematic diagram of the dyeing normalization processing results of the present invention; wherein, Figure 7 (a) is the source image, Figure 7 (b) is the target image, Figure 7 (c) is the staining normalization result diagram;

[0066] Figure 8 It is a schematic diagram of the multi-scale fusion process of the present invention;

[0067] Figure 9 This is the RetCCL network architecture diagram of the present invention;

[0068] Figure 10 Schematic diagram of the feature fusion mechanism of the present invention;

[0069] Figure 11 is a comparison between the heat map generated by the present invention and the pathologist's annotation results; wherein, Figure 11 (a) is the result of the pathologist's annotation of benign tumor. Figure 11 (b) is a benign tumor heat map generated by the present invention, Figure 11 (c) is the result of the pathologist’s annotation of malignant tumors. Figure 11 (d) is a heat map of malignant tumors generated by the present invention. DETAILED DESCRIPTION

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0071] Example 1

[0072] like Figure 1 As shown, a method for automatic diagnosis of thyroid pathology images includes steps S1-S6.

[0073] S1. Construct a thyroid tumor pathology image dataset.

[0074] S2. Obtain several pathological images from the thyroid tumor pathology image dataset and preprocess the pathological images. The preprocessing includes tissue region segmentation, staining normalization, and data enhancement.

[0075] Since full-slice pathology images are large in size and not suitable for direct processing, block sampling technology is used to extract small-size images for training. Pathology images contain a large number of background areas, and the redundant images generated by these areas not only increase training time, but may also reduce the accuracy of the classification model. Therefore, it is necessary to segment the pathology images, remove background blank areas, and identify tissue areas. In order to improve the accuracy of the model, the present invention combines a new lightweight semantic segmentation method AFFormer, which adopts a parallel architecture to express specific learnable local descriptions through prototype representation to replace the role of the decoder, and can maintain rich image semantic information on high-resolution features while reducing computational costs. On this basis, the present invention adds thyroid pathology images that cannot be solved by existing models as training data, and re-fine-tunes the model. Among them, the method of segmenting tissue areas includes the following steps:

[0076] (1) Constructing a dataset for training a tissue region segmentation model based on the AFFormer architecture. In this embodiment, 100 pathological images were selected from a thyroid tumor pathology image dataset, and combined with their corresponding tissue region masks, a dataset for training a tissue region segmentation model based on the AFFormer architecture was constructed.

[0077] like Figure 2 As shown, dataset construction includes the following steps:

[0078] A. Apply the OTSU algorithm to perform tissue region segmentation on pathological images.

[0079] B. After manually excluding images with unsatisfactory segmentation results, the remaining images were re-segmented using the HistoQC algorithm and manually screened again.

[0080] C. PaddleSeg is used to manually annotate the tissue areas of the above images with poor segmentation effects, thus forming a dataset consisting of segmented masks and WSI original images.

[0081] (2) Pixel enhancement and spatial enhancement. When performing medical image analysis through deep learning, data enhancement is usually used to alleviate the data shortage problem. Based on the MedAugment strategy, this paper performs pixel enhancement and spatial enhancement on the pathological images of the constructed data set, and introduces a hyperparameter mapping relationship to achieve full controllability, effectively solving the difference between natural images and medical images. The enhancement effect is shown in the figure below. Figure 3 shown.

[0082] (3) Model training: The data set after data augmentation is divided into a training set and a validation set. The image blocks of set pixels are cut out from the pathological image through the image cutting training method. The image blocks in the boundary area are selected and input into the tissue area segmentation model for training. At the same time, a balanced sampling strategy is adopted to strengthen the sampling of image blocks near the boundary.

[0083] In this embodiment, the data set is divided into a training set and a validation set according to a ratio of 8:2, wherein the training set has 80 WSIs and the validation set has 20 WSIs. In the data preprocessing stage, the ratio of the training tissue area to the non-tissue area is calculated, and the pixel ratio of the non-tissue area to the tissue area is approximately 1.538:1. Since the main goal is to segment the tissue area, a 1024*1024 image patch is cut out from each WSI image according to the field of view size of 2.5x, 5x, 10x, and 20x through the image cutting training method. Figure 4 As shown in the figure, the green boxes represent the boundary region image patches under different fields of view. Due to the large number of 20x image patches, only the boundary region image patches were selected for training. In addition, to further balance the sample distribution, a balanced sampling strategy was adopted to increase the sampling of image patches near the boundary, thereby increasing the number of tissue region image patches and boundary region image patches.

[0084] In the present invention, the loss function of the tissue region segmentation model used in model training is composed of the superposition of label smoothed cross entropy loss and Diceloss. The specific calculation formula is as follows:

[0085] l loss =l ls-ce +2l dice ;

[0086] The label smoothed cross entropy loss is calculated as follows:

[0087]

[0088] Among them, ce is the cross entropy loss, ε represents a small positive number, and β is the weight used to adjust the label smoothed cross entropy loss as a hyperparameter;

[0089] The calculation formula of DiceLoss is as follows:

[0090]

[0091] Among them, |X∩Y| represents the intersection between the true result X and the predicted result Y, and |X| and |Y| represent the number of elements in X and Y respectively.

[0092] By superimposing these two loss functions, this paper constructs a more robust and efficient model for complex medical image segmentation tasks. Label smoothing regularization is incorporated into the traditional cross-entropy loss function to further improve the model's generalization performance. DiceLoss emphasizes region overlap, which improves the model's segmentation accuracy for small target regions. A smaller DiceLoss indicates a higher similarity between the predicted and true segmented images, and thus better model performance.

[0093] The loss function during training is as follows Figure 5 As shown in the figure, the weight of the best DCIE in the validation set is finally selected from the 100 rounds of training weights as the weight for subsequent tissue region mask prediction. It is determined that the best weight is the 73rd round weight, and the corresponding DICE is 0.989.

[0094] Depend on Figure 6 As can be seen, compared to the traditional OTSU algorithm, the tissue region segmentation method trained by this invention based on AFFormer demonstrates higher accuracy, more accurately defining tissue regions and effectively avoiding missegmentation. Furthermore, on a P40 graphics card, this algorithm extracts a WSI tissue region mask in 5-30 seconds, significantly improving efficiency compared to traditional methods such as the OTSU algorithm (which takes over 5 minutes).

[0095] By building a self-built pathological tissue region segmentation dataset and combining it with automatic data enhancement techniques for medical images, the present invention successfully trained a novel lightweight semantic segmentation algorithm. Unlike traditional methods for tissue region segmentation, this model eliminates the need for tedious manual parameter adjustments and effectively eliminates image artifacts. After training, the model demonstrated rapid and accurate localization and segmentation of tissue regions in a dataset of thyroid tumor pathology images, providing strong support for subsequent analysis and application.

[0096] Differences in the pathological tissue sectioning process can lead to differences in the appearance of H&E-stained histological samples. The digitized pathological images can also affect the accuracy and robustness of the classification model due to uneven and inconsistent staining. The present invention uses an automatic staining vector localization algorithm in the staining normalization process and combines it with a color deconvolution method for image processing. The specific steps are as follows:

[0097] (1) Convert the RGB color space to the optical density (OD) space and standardize the OD value to the range of [0,1]. The calculation formula of OD is: OD = -log 10 I, where I is an RGB color vector.

[0098] (2) Delete the OD values ​​less than the threshold β and remove the transparent white background to obtain the main tissue image.

[0099] (3) Find the direction of the geodesic path, perform SVD decomposition based on the OD value, and obtain the singular value. The calculation formula is: OD = V*S, where V and S are the matrix of the coloring vector and the saturation of each coloring, respectively.

[0100] (4) The vectors corresponding to the two largest singular values ​​of the OD transformed pixel are decomposed by SVD to form an evaluation plane. All optical density values ​​are projected onto the plane and normalized.

[0101] (5) Calculate the angle between each point on the curve and the first SVD direction, and map the direction on the plane to a scalar.

[0102] (6) Determine the boundary values ​​of the staining vector by calculating the minimum and maximum percentiles of the staining agent.

[0103] (7) The OD value was used to determine the staining vector of each stain and the staining intensity was normalized.

[0104] (8) Use the staining concentration matrix to recreate the source image so that it has staining intensity similar to the target image.

[0105] Among them, the comparison of the source image, target image and normalization result is shown in Figure 7, from which it can be seen that the algorithm can reduce color differences in the normalization process while retaining more image details.

[0106] To increase the diversity of training data and improve the performance and generalization ability of the model, this paper performs data augmentation on the cut image blocks. During the data augmentation process, one of the five data augmentation methods, image compression, random rotation, random gamma correction, color adjustment, and gradient brightness, is randomly selected for processing. Each image augmentation method has an equal probability of being selected. The specific parameter settings are as follows:

[0107] (1) Image compression: The quality_range parameter is set to 85-100.

[0108] (2) Random rotation: The rotation angle is 90 degrees, which is achieved through the RandomRotate90() function.

[0109] (3) Random Gamma Correction: Use the RandomGamma() function to randomly adjust the Gamma value of the image, thereby changing the brightness of the image.

[0110] (4) Color adjustment: Parameters are set using the ColorJitter() function, where the brightness is adjusted to 0.1, the contrast is adjusted to 0.05, the saturation is adjusted to 0.02, and the hue is adjusted to 0.02.

[0111] (5) Gradient brightness change: This is achieved through the GradualBrightness() function, and the gradient brightness change range is set to (0.8, 0.99).

[0112] Through the above image enhancement operations, the diversity of training data can be effectively increased, and the performance and generalization ability of the model can be improved.

[0113] S3. Perform multi-scale fusion on the preprocessed pathology images using a feature pyramid approach. This approach combines the features extracted from the original 20x magnification image block with the features extracted from its corresponding 10x scaled 256*256 image block. This step aims to fuse feature information at different magnifications to more comprehensively describe the image's characteristics, thereby improving classification accuracy.

[0114] The initial WSI is 20 times magnification, and the fusion process is as follows Figure 8 As shown in the figure, the preprocessed image is first sliced ​​to obtain a 256*256 patch. This patch is then centered and enlarged to a 10x magnification at a 1:1 ratio, resulting in a 512*512 patch. These patches are then scaled to 256*256 image blocks. RetCCL is then used to extract features from the 10x and 20x magnification images, followed by feature fusion.

[0115] S4. Perform feature extraction on the multi-scale fused pathology images using the RetCCL, iBOT, and DINO feature extractors. During feature extraction, the iBOT and DINO feature extractors extract feature vectors for each image block. Using zero padding, the 1536-dimensional feature vectors from the iBOT and DINO feature extractors are expanded to 4096 dimensions, ensuring they have the same feature length as the 4096-dimensional feature vectors extracted by the RetCCL feature extractor.

[0116] RetCCL is a clustering-guided contrastive learning framework for extracting WSI-level image features. By integrating public datasets such as TCGA and PAIR, it has built a massive histopathology image database covering a wide range of cell types, tissue types, cancer diagnoses, and organs. It performs well in various downstream tasks and is suitable for a variety of histopathology image applications.

[0117] In addition to using the original RetCCL feature extractor, this paper also introduces the iBOT (image BERT pre-training with Online Tokenizer) feature extractor and the DINO (self-distillation with no labels) feature extractor based on self-supervised learning. All three feature extractors are pre-trained on large-scale pathology image datasets.

[0118] S5. For the pathological image after feature extraction, feature fusion is performed by using a parallel fusion method. During feature fusion, the three sets of feature vectors corresponding to each image block after feature extraction are fused into a single feature vector by a concat method.

[0119] During the fusion process, no information in the original feature vectors is changed, but they are simply juxtaposed together. This feature fusion method realizes the fusion of features extracted by different feature extractors, which can effectively improve the performance of image classification tasks.

[0120] S6. Input the pathological image after feature fusion into the CLAM-SB multi-instance learning model to classify the pathological image into benign or malignant and generate a visual heat map. The specific steps include:

[0121] (1) Downsampling and tissue segmentation: After downsampling and tissue region segmentation, the input image is cut into small image blocks of 256*256.

[0122] (2) Use the ResNet50 model pre-trained on ImageNet to convert each small image block into a 1024-dimensional feature vector and compress it to 512 dimensions through a fully connected layer.

[0123] (3) Attention pooling: Based on the gated attention mechanism, attention scores are calculated for each small image block, and features are aggregated to generate full-slice level representations.

[0124] (4) Clustering constraint: a binary clustering layer is introduced to optimize the feature space using pseudo labels to ensure that positive and negative features are linearly separable.

[0125] (5) The loss function is optimized by combining image-level cross entropy loss and image block-level smooth support vector machine loss.

[0126] (6) Generate and output a visualization heat map.

[0127] Clustering-constrained-attention Multiple-instance Learning (CLAM) is a weakly supervised method based on deep learning. Through attention-based learning, the model generates explanatory heat maps to help clinicians visualize the relative contribution and importance of various tissue regions to model predictions on full slides. At the same time, instance-level clustering can be used on the identified representative regions to constrain and refine the feature space.

[0128] The following is a further explanation of the steps of the CLAM multi-instance learning model:

[0129] First, the image is read in using downsampling techniques and tissue region segmentation is performed. The segmented mask is then cut into small 256x256 image patches. Using a ResNet50 model pre-trained on ImageNet, each small patch is converted into a low-dimensional 1024-dimensional feature vector. This achieves data dimensionality reduction while preserving key information, significantly improving subsequent processing speed and reducing computational costs.

[0130] To reduce data redundancy and retain sufficient information, these feature vectors are further compressed into 512-dimensional vectors through the first fully connected layer. During the aggregation stage, a gated attention-based pooling mechanism overcomes the limitations of traditional max pooling methods. This mechanism calculates an attention score for each patch. The results for all patches are aggregated based on these scores, not only taking into account the importance of each patch but also avoiding the high false positive rate that can be caused by max pooling.

[0131] The attention score a of the k-th patch of the i-th category after attention pooling i,k The formula is:

[0132]

[0133] Where i and j represent the numbers of the small image blocks in an instance package, and k is 0 or 1 in the binary classification task. (W a, 1W a,2 ......W a,i )∈R 1*256 Indicates that the attention network is divided into i parallel attention branches, tanh() represents the hyperbolic tangent function, V a ∈R 256*512 and U a ∈R 256*512There are two attention layers, sigm() represents the Sigmoid function, and ⊙ represents multiplication.

[0134] Then, the patch-level information is aggregated into a full-scan image-level representation h through an attention network. slide,i , the calculation formula is:

[0135] Corresponding to the unnormalized total score S under the i-th category slide,i The calculation formula is: S slide,i =W c,i h slide,i Among them, W c,i ∈R 1*256 is the classification layer.

[0136] Finally, the score is calculated through the SoftMax function to predict the classification result.

[0137] In order to assist the final image classification task, the present invention introduces a binary clustering layer in the network design. This layer is located after the first fully connected layer. Its function is to sort the small patches processed by the attention layer in ascending order according to their attention scores. The output of the attention network is then used to generate pseudo labels for each WSI in each training iteration, that is, the M patches with the highest attention scores and the M patches with the lowest scores are selected, and positive and negative pseudo labels are assigned to them respectively. These pseudo labels play an important supervisory role in the subsequent clustering process, ensuring that the clustering process can be carried out more accurately. The clustering assignment score P predicted by the kth patch is i,k The calculation formula is as follows:

[0138] P i,k =W inst,i h k ;

[0139] Where W inst,i Represents the weight of the clustering layer corresponding to the i-th class.

[0140] In the clustering algorithm, pseudo-labels are assigned based on the predictions of the attention branch. This prediction process is always supervised by the image-level label Y. Therefore, patches with high attention scores are more likely to be assigned positive labels, while patches with low attention scores are more likely to be assigned negative labels. This process essentially optimizes the image block-level feature space to achieve linear separability between the positive and negative features of each category in the feature space. Since a WSI may contain both positive and negative image patches, when the predicted category of the attention branch is inconsistent with the true label, to avoid misleading the clustering process, the 2M small image patches currently in focus will not be included in the clustering range, thereby ensuring the accuracy and reliability of the clustering results.

[0141] The error is calculated by the image level loss (l slide ) and image block level loss (l patch ). The image-level loss uses the standard cross-entropy loss, which is obtained by comparing the true WSI labels with the normalized predicted classification results. The image-block-level loss uses the smoothed support vector machine loss, which is obtained by comparing the instance label of the binary classifier with the classification results.

[0142] l total =c1l slide +c2l patch ;

[0143] where c1 and c2 are scaling parameters.

[0144] In addition, the goal of constructing the heat map in this process is not to obtain pixel-level annotations of WSI, but to serve as a visualization tool to assist clinicians in making judgments, so only approximate areas are included.

[0145] The present invention mainly includes the following innovative features:

[0146] 1. Intelligent classification method based on weakly supervised multi-instance learning: To address the high resolution and labeling challenges of pathological images, this invention utilizes a weakly supervised multi-instance learning method, relying solely on pathological image-level labels for training, thereby reducing the workload of manual labeling. By optimizing the multi-instance learning framework, the model automatically learns key features of localized lesions, improving classification performance.

[0147] 2. Pathological Tissue Region Segmentation Based on a Lightweight Deep Learning Model: To address the ineffectiveness of traditional tissue region segmentation methods for thyroid tumor pathology images, this paper constructs a lightweight deep learning segmentation model and trains it on a self-built pathological tissue region segmentation dataset. Furthermore, by combining automatic data augmentation techniques for medical images, the model learns more representative pathological tissue region features, improving segmentation accuracy.

[0148] 3. Multi-scale feature fusion method based on a pyramid cascade strategy: To address the problem that single-scale scans cannot fully capture pathological image information, this paper proposes a multi-scale feature fusion method based on a pyramid cascade strategy, building on the weakly supervised multi-instance learning framework. This method combines image information at 10× and 20× magnifications, and through cascading fusion of features at different scales, improves the model's ability to identify lesions at different scales and further enhances classification accuracy.

[0149] 4. Pathological image classification method integrating multiple feature extractors: In view of the limitations of a single feature extraction model, the present invention proposes a pathological image classification method integrating multiple feature extractors. By integrating multiple in-domain feature extractors (RetCCL, iBOT, DINO) trained based on large-scale pathological image datasets, comprehensive capture of multi-level information in pathological images is achieved. These feature extractors have been pre-trained on different pathological datasets and can extract information at different levels such as texture, structure, and semantics. Through feature fusion, the classification model of the present invention has been significantly improved in both discrimination and generalization capabilities.

[0150] The present invention can effectively reduce the cost of manual labeling, improve training efficiency, enhance the accuracy of pathological tissue region segmentation, and enhance the multi-scale feature extraction capability of the classification model, thus breaking through the limitations of a single feature extraction model.

[0151] In summary, the present invention addresses the key issues of intelligent diagnosis of thyroid cancer pathological images and proposes an intelligent diagnosis method that integrates multiple advanced technologies. It can effectively alleviate the workload of pathologists, improve the accuracy and efficiency of pathological image analysis, and provide strong support for the early detection and precise treatment of thyroid cancer.

[0152] The above description is a detailed description of the preferred embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. Any equivalent changes or modifications completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A method for automatic diagnosis of thyroid pathological images, characterized in that: The following steps are involved: S1. Construct a thyroid tumor pathology image dataset; S2. Obtain several pathological images from the thyroid tumor pathology image dataset and preprocess the pathological images. The preprocessing includes tissue region segmentation, staining normalization, and data enhancement. S3, using feature pyramid to perform multi-scale fusion on pre-processed pathological images; S4. Use RetCCL, iBOT, and DINO feature extractors to extract features from multi-scale fused pathological images; S5. For the pathological image after feature extraction, feature fusion is performed using a parallel fusion method; S6. Input the pathological image after feature fusion into the CLAM-SB multi-instance learning model, classify the pathological image into benign or malignant, and generate a visual heat map.

2. The automatic diagnosis method of thyroid pathology images according to claim 1, characterized in that: In step S2, the method for segmenting tissue regions includes the following steps: (1) Construct a dataset for training a tissue region segmentation model based on the AFFormer architecture; (2) Pixel enhancement and spatial enhancement: Based on the MedAugment strategy, pixel enhancement and spatial enhancement are performed on the pathological images of the constructed dataset, and a hyperparameter mapping relationship is introduced; (3) Model training: The data set after data augmentation is divided into a training set and a validation set. The image blocks of set pixels are cut out from the pathological image through the image cutting training method. The image blocks in the boundary area are selected and input into the tissue area segmentation model for training. At the same time, a balanced sampling strategy is adopted to strengthen the sampling of image blocks near the boundary.

3. The automatic diagnosis method of thyroid pathology images according to claim 2, characterized in that: In step S2, constructing a dataset when segmenting tissue regions includes the following steps: A. Apply OTSU algorithm to perform tissue region segmentation on pathological images; B. After manually eliminating images with unsatisfactory segmentation results, the remaining images were re-segmented using the HistoQC algorithm and manually screened again; C. PaddleSeg is used to manually annotate the tissue areas of the above images with poor segmentation effects, thus forming a dataset consisting of segmented masks and WSI original images.

4. The automatic diagnosis method of thyroid pathology images according to claim 3, characterized in that: In step S2, the loss function of the tissue region segmentation model used in model training is composed of the superposition of label smoothed cross entropy loss and Diceloss. The specific calculation formula is as follows: The label smoothed cross entropy loss is calculated as follows: Among them, ce is the cross entropy loss, ε represents a small positive number, and β is the weight used to adjust the label smoothed cross entropy loss as a hyperparameter; The calculation formula of DiceLoss is as follows: Among them, |X∩Y| represents the intersection between the true result X and the predicted result Y, and |X| and |Y| represent the number of elements in X and Y respectively.

5. The automatic diagnosis method of thyroid pathology images according to claim 1, characterized in that: In step S2, the automatic color vector positioning algorithm is used in the color normalization process, and the image processing is performed in combination with the color deconvolution method. The specific steps are as follows: (1) Convert the RGB color space to the optical density (OD) space and standardize the OD value to the range of [0,1]. The calculation formula of OD is: OD = -log 10 I, where I is an RGB color vector; (2) Delete the OD values ​​less than the threshold β, remove the transparent white background, and obtain the main tissue image; (3) Find the direction of the geodesic path, perform SVD decomposition based on the OD value, and obtain the singular value. The calculation formula is: OD = V*S, where V and S are the matrix of the coloring vector and the saturation of each coloring, respectively; (4) The vectors corresponding to the two largest singular values ​​of the OD transformed pixel are decomposed by SVD to form an evaluation plane. All optical density values ​​are projected onto the plane and normalized. (5) Calculate the angle between each point on the curve and the first SVD direction, and map the direction on the plane into a scalar; (6) Determine the boundary value of the staining vector by calculating the minimum and maximum percentiles of the staining agent; (7) Use OD values ​​to determine the staining vector of each stain and normalize the staining intensity; (8) Use the staining concentration matrix to recreate the source image so that it has staining intensity similar to the target image.

6. The automatic diagnosis method of thyroid pathology images according to claim 1, characterized in that: In step S2, during the data enhancement process, any one of the five data enhancement methods, image compression, random rotation, random gamma correction, color adjustment, and gradient brightness, is randomly selected for processing, and each image enhancement method has the same probability of being selected.

7. The automatic diagnosis method of thyroid pathology images according to claim 1, characterized in that: In step S3, a feature pyramid method is used to splice and fuse the features extracted from the original 20-fold magnification image block and the features extracted from the corresponding 10-fold scaled 256*256 image block.

8. The automatic diagnosis method of thyroid pathology images according to claim 7, characterized in that: In step S4, during feature extraction, feature extractors iBOT and DINO are used to extract the feature vector of each image block. The 1536-dimensional feature vector of feature extractor iBOT and the 768-dimensional feature vector of feature extractor DINO are expanded to 4096 dimensions by padding with zeros, so that they have the same feature length as the 4096-dimensional feature vector extracted by feature extractor RetCCL.

9. The automatic diagnosis method of thyroid pathology images according to claim 8, characterized in that: In step S5, during feature fusion, the three sets of feature vectors corresponding to each image block and subjected to feature extraction processing are fused into a single feature vector through a concat method.

10. The automatic diagnosis method of thyroid pathology images according to claim 9, characterized in that: Step S6 specifically includes the following steps: (1) Downsampling and tissue segmentation: After downsampling and tissue region segmentation, the input image is cut into small image blocks of 256*256; (2) Use the ResNet50 model pre-trained on ImageNet to convert each small image patch into a 1024-dimensional feature vector and compress it to 512 dimensions through a fully connected layer; (3) Attention pooling: Based on the gated attention mechanism, attention scores are calculated for each small image patch, and features are aggregated to generate full-slice-level representations; (4) Clustering constraints: a binary clustering layer is introduced to optimize the feature space using pseudo labels to ensure that positive and negative features are linearly separable. (5) Optimize the loss function by combining the image-level cross entropy loss and the image block-level smoothing support vector machine loss; (6) Generate and output a visualization heat map.

Images