An automatic prostate Gleason grading method based on multi-level visual basic model feature fusion

Abstract

The present invention provides a method for automatic Gleason grading of prostate based on the fusion of multi-level visual basic model features, comprising: step 1, data collection and preprocessing; step 2, training the basic model; step 3, using the trained model to extract image features at multiple resolutions, and using the single-cell morphology and topology atlas framework sc-MTOP to extract single-cell features; step 4, using the Gleason grading prediction model to complete the automatic Gleason grading of prostate. The present invention achieves more accurate Gleason grading of prostate cancer pathology images by fusing multi-resolution image features with single-cell features. By utilizing multi-center data collection and self-supervised training, the adaptability and generalization ability of the model to different data sources are improved. A cross-attention module is introduced to fully integrate multi-level features and improve grading accuracy. Combining expert correction feedback with the pre-trained basic model, the utilization efficiency of unlabeled data and the model performance are further optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a prostate Gleason automatic grading method based on multi-level visual basic model feature fusion. Background Art

[0002] Prostate adenocarcinoma (PCa) is one of the most common malignant tumors in men worldwide. Prostate cancer develops slowly and is highly heterogeneous, so accurate grading assessment is required for diagnosis and treatment. Currently, Gleason Grade (GG) is an important standard for prostate cancer assessment, which can provide clinicians with a reference for tumor aggressiveness in order to formulate appropriate treatment strategies. By evaluating the microscopic and histological features of prostatectomy specimens and recurrent / routine biopsy tissue specimens, pathologists can detect malignant tumors and stratify patient management based on GG.

[0003] The Gleason grading assessment process is highly specialized and often requires pathologists to spend a significant amount of time observing and judging under the microscope. However, this process is somewhat subjective, and diagnostic results may differ between pathologists, leading to the risk of misdiagnosis or overdiagnosis. As the number of pathology images increases, improving the efficiency and consistency of assessment becomes an urgent issue. Therefore, developing an automated Gleason grading method is of great significance. It can provide decision support for pathologists, reduce their workload, and improve the accuracy of prostate cancer diagnosis and treatment.

[0004] Automated assessment methods for prostate pathology images based on artificial intelligence (AI) technology offer a viable technical solution to this problem. In recent years, deep learning models have made significant progress in image recognition, particularly with the application of deep learning models, which have made it possible to extract latent features from large-scale data sets. By constructing AI models based on prostate pathology images, it is possible to automatically identify histological features under a microscope, enabling rapid and accurate prediction of Gleason grade. This type of technology not only significantly shortens assessment time but also improves the stability and consistency of diagnostic results by eliminating human error, providing strong support for clinical decision-making. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to construct a basic model for prostate cancer pathology image feature extraction, extract key histological features and single-cell features from prostate pathology slice images, and automatically evaluate the aggressiveness of the tumor based on these features, thereby ensuring the stability, accuracy and efficiency of the Gleason grading of prostate cancer in multi-center pathology images.

[0006] Technical solution: The present invention specifically provides a prostate Gleason automatic grading method based on multi-level visual basic model feature fusion, comprising the following steps:

[0007] Step 1, Data Collection and Preprocessing: Collect and normalize data, construct unlabeled and supervised training queues based on the presence or absence of labels, and perform foreground and background segmentation on panoramic slice images, cutting the foreground into image blocks of different resolutions;

[0008] Step 2: Train the basic model: retrain the pre-trained basic model using image patches of different resolutions.

[0009] Step 3: Use the trained model to extract image features at multiple resolutions, and use the single cell morphological and topological profiling framework sc-MTOP (Single Cell Morphological and Topological Profiling) to extract single cell features;

[0010] Step 4: Using the features extracted in step 3 to perform interactive fusion between features, a Gleason grading prediction model that integrates image block level and slice level is constructed, and the Gleason grading prediction model is used to complete the automatic Gleason grading of the prostate.

[0011] In step 1, the data collection and collection scope includes public data sets PANDA Challenge, Gallo, Tolkach and SICAP-MIL, among which PANDA Challenge data are annotated by pathology experts to outline five grading areas: interstitial, benign, and Gleason's scoring areas. First, based on the general empirical method of foreground and background segmentation, image processing methods such as Otsu's threshold segmentation, inversion, hole filling, image dilation, image erosion and small target removal are used to achieve foreground and background segmentation of panoramic slice images. Subsequently, image blocks of a specified size are extracted from the foreground area for use in subsequent steps. Finally, a data queue index is formulated to index and queue the panoramic slice images, foreground and background segmentation masks, image blocks and corresponding labels.

[0012] In step 2, the pre-trained basic model is the universal pathology model UNI.

[0013] In step 2, the collected data is used to retrain the publicly available pre-trained basic model, the Universal Pathology Model UNI, to enhance the feature extraction capabilities of prostate pathology images. The training process uses two identical network structures: a teacher model and a student model. The model framework is the Vision Transformer Large (ViT-Large) model. The training process uses the training strategy of the Universal Pathology Model UNI. First, two images of the same size are randomly cropped from the image block and two types of random data augmentation are performed on each. The two images of the same size are used as the global image. One of the global images is further cropped into two or more local images, and the global image is randomly partially masked. The masked global image and the local images are used as the input of the student model; the other global image is used as the input of the teacher model.

[0014] In step 2, the same self-supervised training method as the UNI model is used, that is, the existing improved training method based on the second version of the enhanced denoising anchor point detection transformer DINOv2 (DETR with Improved deNoising anchor boxes), and the loss functions used include the DINO loss function and the online token bidirectional encoding transformer iBOT (Image BERT Pre-Training with Online Tokenizer) loss function;

[0015] In the DINO loss function, the DINO head is added to the student model and the teacher model respectively to perform loss function calculation. For the student model, the class token of each output is obtained and outputted through a multi-layer perceptron (MLP) layer to obtain the student prototype score. The student prototype score is then normalized by the exponential function softmax calculation to obtain the student score p. s ;

[0016] In order to avoid the degeneration of the model, that is, the output features tend to be the same, the output branch of the teacher model is modified. For the teacher model, the class token output by the teacher model is obtained and the teacher prototype score is obtained through the multi-layer perceptron MLP layer. The Sinkhorn-Knopp centralization process is added after the teacher prototype score, and then the normalized exponential function softmax is used to calculate the teacher score p. t The Sinkhorn-Knopp SK centralization process is a method for normalizing a matrix. In one iteration, normalization of rows and columns is performed separately. After two or more iterations, the matrix will be approximately a doubly random matrix. The specific process includes:

[0017] Input: teacher prototype score Z, trainable clustering parameters θ C , number of iterations P;

[0018] Initialization: The teacher prototype score Z0 = Z × C in the 0th iteration;

[0019] The iterative steps include:

[0020] For p, it is from 1 to P cycles:

[0021] Temporary value Z1 of the p+1th iteration p+1 for

[0022] in is a diagonal matrix, and the diagonal elements are The reciprocal of the sum of each row;

[0023] The Z value of the p+1th iteration is

[0024] in is a diagonal matrix, and the diagonal elements are The reciprocal of the sum of each column;

[0025] Output the final teacher prototype score Z K .

[0026] In step 2, the softmax normalized exponential function in the output headers of the student and teacher models introduces a sharpening operation, dividing the input data by a temperature constant set to the square root of the dimension to widen the gap between the features of different channels in the model. By combining centering and sharpening, the model output features are prevented from degrading.

[0027] Finally, for the student score p s With the teacher score p t Use the cross entropy CE (Cross Entropy) loss function to obtain the DINO loss function L DINO :

[0028] L DINo =CE(p t ,p s )=-∑p t logp s (1)

[0029] During the training process, the student model performs normal backpropagation to update the model parameters, while the teacher model does not perform backpropagation. The parameters of the teacher model are updated with momentum using the parameters of the student model.

[0030] In step 2, in the iBOT loss function L iBOtIn the process, the loss function is calculated by adding a header to the student model and the teacher model. The input image of the student model randomly masks the area. In the student model, the iBOT header is used for the token corresponding to the masked area, and in the teacher model, the iBOT header is applied to the token at the same position. Subsequently, the iBOT header adopts the same loss function L as the DINO loss function. DINO The same normalized exponential function softmax calculation and Sinkhorn-Knop SK centralization process are used, and the iBOT loss function L is obtained using the cross entropy CE loss function. iBOT :

[0031]

[0032] Among them, p ti is the teacher score for the ith covered region, p si is the student score of the i-th covered area; the parameters of the student model are trained and the parameters of the teacher model are adjusted through the momentum update mechanism.

[0033] In step 2, the model also uses KoLeo regularization during training to encourage the output features to be evenly distributed within a batch: given a set of network output features, the network output features are first regularized using the 2-norm, and then the feature x vector (x1, x2, ..., x n ), define the KoLeo loss function L koleo :

[0034]

[0035] in is the sth feature x s The minimum distance to other features in the batch, n is the total number of features, x n represents the nth feature.

[0036] Step 3 includes: using the student model to extract features of all image blocks and storing the results in a normalized manner, wherein the normalized storage includes image features, image block starting pixel coordinates, size, relative coordinates, panoramic slice image source, and magnification;

[0037] The single-cell morphology and topology atlas framework sc-MTOP is used in image blocks to extract cell features and construct a cell nucleus map: cell segmentation is performed on the image blocks, a cell nucleus map is constructed based on spatial coordinate vectors and the nearest neighbor KNN (K-nearest neighbor), and three types of manual features of single cells, texture, morphology, and topology, are extracted. In the cell nucleus map, the manual features of each cell are spliced ​​with the corresponding spatial position code as a node feature.

[0038] Step 4 includes: based on multi-instance learning, constructing a Gleason grade prediction model that takes multi-scale and cell nucleus images as input and outputs slice-level and image-block-level outputs through slice branches and image-block branches;

[0039] In terms of input, multi-resolution images and cell nucleus maps are used as input. The high-resolution images provide image details, the low-resolution images provide a wider receptive field, and the cell nucleus maps provide information on the distribution and arrangement of cells in the image.

[0040] In terms of output, slice-level output can obtain patient-level diagnostic results, and image block level output can obtain the spatial distribution of each grade;

[0041] The Gleason grading prediction model uses the improved attention multiple instance learning method Attention-MIL (Attention Multiple Instance Learning) for multi-instance learning. That is, a multi-layer perceptron (MLP) model is added to the original attention multiple instance Attention-MIL as an instance classifier, acting as an image block branch, and the multi-layer perceptron output of the attention multiple instance Attention-MIL is used as a slice classifier, acting as a slice branch. Specifically, the Gleason grading prediction model needs to input image features at the image block level and cell-level nuclear map features at two different resolutions at the same time. For the image block-level image features, the multi-resolution image features extracted by the trained model in step 3 are directly used; for the cell-level features, the graph neural network TransformerGraph model is first used to update the node features of the nuclear map, and deep features with contextual information are obtained from the manual features by connecting neighboring nodes. Then, the cell-level nuclear map features are obtained through a transformer with a class token. After feature extraction, the image features at the image block level and the cell-level nuclear map features are spliced ​​as the features of the high-resolution image, namely, the instance features.

[0042] The improved attention multi-instance network is trained simultaneously through the two output branches of the image block branch instance classifier and the slice classifier of the slice branch during the training process. There are M feature numbers before the network calculation. The network first calculates the feature number based on the 1×M dimensional real vector space. The kth instance feature of Calculate the kth instance attention coefficient a k , calculated as:

[0043]

[0044] The network generates L features during calculation, and the model divides the results into C categories. V and U are real vector spaces in L×M dimensions. The learnable parameters of W is a real vector space in L×C dimensions The learnable parameters of Then use the attention coefficient to train the instance classifier and slice classifier separately;

[0045] For instance classifiers: The training of instance classifiers is divided into two cases: one with image block classification labels and the other with only slice classification labels. Different classification heads are used for the two cases: When only slice classification is used, the classification head output is in the 2×C real vector space. Rating That is, the binary classification score of the instance for each category, which is used to indicate whether the instance belongs to the category; when there is image block classification, the classification head output is in the 1×C real vector space Rating That is, the score of the instance output for each category; let the classification probability of category i given by the instance classifier for instance k be y i,k ;

[0046] When the sample only has a slice label, the priority method Top-K attention coefficient is used to extract the K highest and lowest instances and generate pseudo labels for instance classifier training: for the category label c of the input slice, the instance classifier model of the image block branch selects the top K instances with the highest attention coefficient and the top K instances with the lowest attention coefficient from the attention map of the corresponding category label c, and generates corresponding pseudo classification labels 1 and 0 respectively; for other categories, only the K highest attention instances of other categories are generated with pseudo label 0; finally, the instance and pseudo label pairs are spliced ​​and used for instance classifier training, and the training uses the cross entropy loss function L inst :

[0047]

[0048] Among them, Top(a i ) is the largest K image blocks in the attention of category i, Top(-a i ) is the smallest K image patches in the attention of category i, CE(1,y) is the cross entropy calculation between the prediction result y and the pseudo label 1;

[0049] When the sample has an image block label, the instance classifier is trained directly using the image block level label: first, according to the doctor's segmentation label, the classification label is set for the image block at the corresponding position. For image block k, the number of pixels in image block k that are marked as a certain category is counted. When the number exceeds the empirically set threshold of 75%, the image block is determined to be marked as a category. Then, according to the image block classification, the attention coefficient is set for the image block. For classification c, it is set that there are image blocks in the slice. For the set of image blocks classified as c in the slice, the c-class attention label value of image block k is Set as Then, the cross entropy CE loss function is used for image block classification, and the mean square error MSE loss function is used for attention coefficient training:

[0050]

[0051] in is a set of labeled instances, is the set of instances with label 0, is a set of instances with label C, MSE represents the mean square error function, Prediction result y for the image block k With mark Cross entropy calculation of ;

[0052] For the slice classifier: the native Attention-MIL performs weighted addition of all instance features through the attention coefficient to obtain slice-level features, obtains slice classification through the fully connected layer FC (Fully Connected), and then uses the cross entropy loss function CE for training. The instance feature aggregation formula and training loss function L slide for:

[0053]

[0054]

[0055] in is the slice-level label, a k is the kth instance attention coefficient, h k is the kth instance feature, the fully connected layer FC is a linear mapping; y represents the output of the fully connected layer FC;

[0056] Finally, through two loss functions L slide and L inst The instance classifier and slice classifier are trained simultaneously by superposition;

[0057] During the testing phase, the classification head outputs the category of each patch, generating spatial predictions for each level. The final slice-level prediction is then derived by combining the slice classifier output with the patch-level instance classifier. The trained model is validated using the test set. Evaluation includes comparing the trained base model with the native base model, comparing multi-task multi-instance learning with traditional single-task models, and analyzing the consistency between patch-level and slice-level predictions. Accuracy, sensitivity, specificity, precision, F1 score, and AUC (Area Under the Curve) are used to assess the model's predictive performance. First, the predicted and true results on the test set are counted to obtain a confusion matrix, including true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). The various metrics are then calculated based on the confusion matrix. For accuracy, there is (TP+TN) / (TP+TN+FP+FN); for sensitivity, there is TP / (TP+FN); for specificity, there is TN / (TN+FP); for precision, there is TP / (TP+FP); for F1 score, there is 2*(Precision*Recall) / (Precision+Recall); for AUC, first change the classification threshold, draw the ROC (Receiver Operating Characteristic Curve) curve with 1-specificity as the horizontal axis and sensitivity as the vertical axis, and then calculate the area under the curve, which is the AUC.

[0058] Beneficial Effects: This invention achieves more accurate Gleason grading of prostate cancer pathology images by fusing multi-resolution image features with single-cell features. Leveraging multi-center data collection and self-supervised training, the model's adaptability and generalization capabilities to diverse data sources are enhanced. The introduction of a cross-attention module fully integrates multi-level features, improving grading accuracy. Combining expert correction feedback with a pre-trained base model further optimizes the utilization efficiency of unlabeled data and model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0060] Figure 1 Schematic diagram of the entire process of the artificial intelligence method for evaluating Gleason grade based on the basic model features of prostate pathology images.

[0061] Figure 2 This is a flowchart for training the basic model for prostate diseases.

[0062] Figure 3 Flowchart for extracting multi-resolution image patch features and cell nucleus features.

[0063] Figure 4 Flowchart for training Gleason grading prediction models at the image block level and slice level based on image features and graph features at multiple resolutions. DETAILED DESCRIPTION

[0064] like Figure 1 As shown, an embodiment of the present invention provides a prostate Gleason automatic grading method based on multi-level visual basic model feature fusion, comprising the following steps:

[0065] Step 1, Data Collection and Preprocessing: Collect and normalize data, construct unlabeled and supervised training queues based on the presence or absence of labels, and perform foreground and background segmentation on panoramic slice images, cutting the foreground into image blocks of different resolutions;

[0066] Step 2: Train the basic model: retrain the pre-trained basic model using image patches of different resolutions.

[0067] Step 3: Use the trained model to extract image features at multiple resolutions, and use the single cell morphological and topological profiling framework sc-MTOP (Single Cell Morphological and Topological Profiling) to extract single cell features;

[0068] Step 4: Using the features extracted in step 3 to perform interactive fusion between features, a Gleason grading prediction model that integrates image block level and slice level is constructed, and the Gleason grading prediction model is used to complete the automatic Gleason grading of the prostate.

[0069] In step 1, the data collection scope includes the public datasets PANDA Challenge, Gallo, Tolkach and SICAP-MIL. The PANDA Challenge data are annotated by pathologists to delineate five grading areas: stromal, benign, and Gleason's scoring areas. After completing the data collection, Figure 1As shown in the data preparation, we first implement foreground and background segmentation for panoramic slice images based on common empirical methods for foreground and background segmentation. This is accomplished using image processing methods such as Otsu's threshold segmentation, inversion, hole filling, image dilation, image erosion, and small object removal. We then extract image patches of a specified size from the foreground region for use in subsequent steps. Finally, we develop a data queue index to index and queue the panoramic slice images, foreground and background segmentation masks, image patches, and corresponding labels.

[0070] In step 2, the pre-trained basic model is the universal pathology model UNI.

[0071] In step 2, the whole process is Figure 1 The collected data is used to train the public pre-trained base model, using Figure 2 The universal pathology model UNI is retrained in the prostate disease-specific basic model training process (the basic model training process includes a teacher network and a student network. The student network input is the global image and local image obtained by cropping and randomly enhancing the original image, and the teacher network input is the global image cropped from the original image. Then the two networks obtain the features of each image, use the iBOT and DINO loss functions to calculate the differences between the features, and use gradient back propagation to train the two networks. The student network Pros-UNI is used for subsequent feature extraction), obtaining a prostate-specific basic model and enhancing the feature extraction capability of prostate pathology images. Two identical network structures are used in the training process, namely the teacher model and the student model. The model framework is the vision transformer model ViT-Large (Vision Transformer Large). The training process uses the training strategy of the universal pathology model UNI. First, two pictures of the same size are randomly cropped from the image block and two types of random data augmentation are performed respectively. The two pictures of the same size are used as the global image. One of the global images is further cropped into two or more local images, and the global image is randomly partially masked. The masked global image and the local image are used as the input of the student model together; the other global image is used as the input of the teacher model.

[0072] In step 2, the same self-supervised training method as the UNI model is used, that is, the existing improved training method based on the second version of the enhanced denoising anchor detection transformer DINOv2 (DETR with Improved deNoising anchor boxes), and the loss function used includes Figure 2 The DINO loss function and the online token bidirectional encoding transformer iBOT (Image BERTPre-Training with Online Tokenizer) loss function in ;

[0073] In the DINO loss function, the DINO head is added to the student model and the teacher model respectively to perform loss function calculation. For the student model, the class token of each output is obtained and outputted through a multi-layer perceptron (MLP) layer to obtain the student prototype score. The student prototype score is then normalized by the exponential function softmax calculation to obtain the student score p. s ;

[0074] In order to avoid the degeneration of the model, that is, the output features tend to be the same, the output branch of the teacher model is modified. For the teacher model, the class token output by the teacher model is obtained and the teacher prototype score is obtained through the multi-layer perceptron MLP layer. The Sinkhorn-Knopp centralization process is added after the teacher prototype score, and then the normalized exponential function softmax is used to calculate the teacher score p. t The Sinkhorn-Knopp SK centralization process is a method for normalizing a matrix. In one iteration, normalization of rows and columns is performed separately. After two or more iterations, the matrix will be approximately a doubly random matrix. The specific process includes:

[0075] Input: teacher prototype score Z, trainable clustering parameters θ C , number of iterations P;

[0076] Initialization: The teacher prototype score Z0 = Z × C in the 0th iteration;

[0077] The iterative steps include:

[0078] For p, it is from 1 to P cycles:

[0079] Temporary value Z1 of the p+1th iteration p+1 for

[0080] in is a diagonal matrix, and the diagonal elements are The reciprocal of the sum of each row;

[0081] The Z value of the p+1th iteration is

[0082] in is a diagonal matrix, and the diagonal elements are The reciprocal of the sum of each column;

[0083] Output the final teacher prototype score Z K .

[0084] In step 2, the softmax normalized exponential function in the output headers of the student and teacher models introduces a sharpening operation, dividing the input data by a temperature constant set to the square root of the dimension to widen the gap between the features of different channels in the model. By combining centering and sharpening, the model output features are prevented from degrading.

[0085] Finally, for the student score p s With the teacher score p t Use the cross entropy CE (Cross Entropy) loss function to obtain the DINO loss function L DINO :

[0086] L DINO =CE(p t ,p s )=-∑p t logp s (1)

[0087] During the training process, Figure 1 Basic model training and Figure 2 As shown in Figure 1, the student model performs normal back propagation to update the model parameters, the teacher model does not perform back propagation, and the parameters of the teacher model are updated with momentum using the parameters of the student model.

[0088] In step 2, in the iBOT loss function L iBOT In the process, the loss function is calculated by adding a header to the student model and the teacher model. The input image of the student model randomly masks the area. In the student model, the iBOT header is used for the token corresponding to the masked area, and in the teacher model, the iBOT header is applied to the token at the same position. Subsequently, the iBOT header adopts the same loss function L as the DINO loss function. DINo The same normalized exponential function softmax calculation and Sinkhorn-Knop SK centralization process are used, and the iBOT loss function L is obtained using the cross entropy CE loss function. iBOT :

[0089]

[0090] Among them, p ti is the teacher score for the ith covered region, p si is the student score of the i-th covered area; the parameters of the student model are trained and the parameters of the teacher model are adjusted through the momentum update mechanism.

[0091] In step 2, the model also uses KoLeo regularization during training to encourage the output features to be evenly distributed within a batch: given a set of network output features, the network output features are first regularized using the 2-norm, and then the feature x vector (x1, x2, ..., x n ), define the KoLeo loss function L koleo :

[0092]

[0093] in is the sth feature x s The minimum distance to other features in the batch, n is the total number of features, x n represents the nth feature.

[0094] Step 3 includes: using the student model to extract features of all image blocks and storing the results in a normalized manner, wherein the normalized storage includes image features, image block starting pixel coordinates, size, relative coordinates, panoramic slice image source, and magnification;

[0095] like Figure 1 Feature extraction and Figure 3 As shown in the figure, the single-cell morphology and topology atlas framework sc-MTOP is used in the image block to extract cell features and construct a cell nucleus map: cell segmentation is performed on the image block, the cell nucleus map is constructed based on the spatial coordinate vector and the nearest neighbor KNN (K-nearst neighbor), and three types of manual features of single cells are extracted: texture, morphology, and topology; in the cell nucleus map, the manual features of each cell are spliced ​​with the corresponding spatial position code as the node feature.

[0096] Step 4 includes: based on multi-instance learning, constructing a Gleason grade prediction model that takes multi-scale and cell nucleus images as input and outputs slice-level and image-block-level outputs through slice branches and image-block branches;

[0097] like Figure 1 Prediction model training and Figure 4 As shown in the figure, in terms of input, multi-resolution images and cell nuclear maps are used as input simultaneously, where high-resolution images provide image details, low-resolution images provide a wider receptive field, and cell nuclear maps provide information on the distribution and arrangement of cells within the image; in terms of output, slice-level output can obtain patient-level diagnostic results, and image block level output can obtain the spatial distribution of each grade;

[0098] The Gleason grading prediction model uses the improved attention multiple instance method Attention-MIL (Attention Multiple Instance Learning) for multi-instance learning, that is, a multi-layer perceptron MLP model is added to the original attention multiple instance Attention-MIL as an instance classifier, acting as an image block branch, and the multi-layer perceptron output of the attention multiple instance Attention-MIL is used as a slice classifier, acting as a slice branch. Specifically, the Gleason grading prediction model needs to input two image block-level image features and cell-level nuclear map features of different resolutions at the same time. For the image block-level image features, the multi-resolution image features extracted by the trained model in step 3 are directly used; for the cell-level features, the graph neural network graph transformer TransformerGraph model is first used to update the node features of the nuclear map, and the deep features with contextual information are obtained from the manual features by connecting the neighboring nodes. Then, the nuclear map features of the cell level are obtained through a transformer with a class token class token; after the feature extraction is completed, the image features of the image block level of two different resolutions and the nuclear map features of the cell level are spliced ​​as the features of the high-resolution image, that is, the instance features, such as Figure 1 Prediction model training and Figure 4 As shown in the splicing features;

[0099] The improved attention multi-instance network is trained simultaneously through the two output branches of the image block branch instance classifier and the slice classifier of the slice branch during the training process, such as Figure 1 Prediction model training and Figure 4 As shown in the Slice Classifier and Instance Classifier section. There are M features before the network calculation. The network first calculates the features according to the 1×M dimensional real vector space. The kth instance feature of Calculate the kth instance attention coefficient a k , calculated as:

[0100]

[0101] The network generates L features during calculation, and the model divides the results into C categories. V and U are real vector spaces in L×M dimensions. The learnable parameters of W is a real vector space in L×C dimensions The learnable parameters of Then use the attention coefficient to train the instance classifier and slice classifier separately;

[0102] For instance classifiers: The training of instance classifiers is divided into two cases: one with image block classification labels and the other with only slice classification labels. Different classification heads are used for the two cases: When only slice classification is used, the classification head output is in the 2×C real vector space. Rating That is, the binary classification score of the instance for each category, which is used to indicate whether the instance belongs to the category; when there is image block classification, the classification head output is in the 1×C real vector space Rating That is, the score of the instance output for each category; let the classification probability of category i given by the instance classifier for instance k be y i,k ;

[0103] When the sample only has a slice label, the priority method Top-K attention coefficient is used to extract the K highest and lowest instances and generate pseudo labels for instance classifier training: for the category label c of the input slice, the instance classifier model of the image block branch selects the top K instances with the highest attention coefficient and the top K instances with the lowest attention coefficient from the attention map of the corresponding category label c, and generates corresponding pseudo classification labels 1 and 0 respectively; for other categories, only the K highest attention instances of other categories are generated with pseudo label 0; finally, the instance and pseudo label pairs are spliced ​​and used for instance classifier training, and the training uses the cross entropy loss function L inst :

[0104]

[0105] Among them, Top(a i ) is the largest K image blocks in the attention of category i, Top(-a i ) is the smallest K image patches in the attention of category i, CE(1,y) is the cross entropy calculation between the prediction result y and the pseudo label 1;

[0106] When the sample has an image block label, the instance classifier is trained directly using the image block level label: first, according to the doctor's segmentation label, the classification label is set for the image block at the corresponding position. For image block k, the number of pixels in image block k that are marked as a certain category is counted. When the number exceeds the empirically set threshold of 75%, the image block is determined to be marked as a category. Then, according to the image block classification, the attention coefficient is set for the image block. For classification c, it is set that there are image blocks in the slice. For the set of image blocks classified as c in the slice, the c-class attention label value of image block k is Set as Then, the cross entropy CE loss function is used for image block classification, and the mean square error MSE loss function is used for attention coefficient training:

[0107]

[0108] in is a set of labeled instances, is the set of instances with label 0, is a set of instances with label C, MSE represents the mean square error function, Prediction result y for the image block k With mark Cross entropy calculation of ;

[0109] For the slice classifier: the native Attention-MIL performs weighted addition of all instance features through the attention coefficient to obtain slice-level features, obtains slice classification through the fully connected layer FC (Fully Connected), and then uses the cross entropy loss function CE for training. The instance feature aggregation formula and training loss function L slide for:

[0110]

[0111]

[0112] in is the slice-level label, a k is the kth instance attention coefficient, h k is the kth instance feature, the fully connected layer FC is a linear mapping; y represents the output of the fully connected layer FC;

[0113] Finally, through two loss functions L slide and L inst The instance classifier and slice classifier are trained simultaneously by superposition;

[0114] During the testing phase, the classification head outputs the category of each patch, generating spatial predictions for each level. The final slice-level prediction is then derived by combining the slice classifier output with the patch-level instance classifier. The trained model is validated using the test set. Evaluation includes comparing the trained base model with the native base model, comparing multi-task multi-instance learning with traditional single-task models, and analyzing the consistency between patch-level and slice-level predictions. Accuracy, sensitivity, specificity, precision, F1 score, and AUC (Area Under the Curve) are used to assess the model's predictive performance. First, the predicted and true results on the test set are counted to obtain a confusion matrix, including true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). The various metrics are then calculated based on the confusion matrix. For accuracy, there is (TP+TN) / (TP+TN+FP+FN); for sensitivity, there is TP / (TP+FN); for specificity, there is TN / (TN+FP); for precision, there is TP / (TP+FP); for F1 score, there is 2*(Precision*Recall) / (Precision+Recall); for AUC, first change the classification threshold, draw the ROC (Receiver Operating Characteristic Curve) curve with 1-specificity as the horizontal axis and sensitivity as the vertical axis, and then calculate the area under the curve, which is the AUC.

[0115] In another specific embodiment of the present invention, a method for automatic prostate Gleason grading based on multi-level visual basic model feature fusion is provided, comprising the following steps:

[0116] Step 1: The dataset used in the embodiment of the present invention contains a total of 11,853 slice samples, which were digitized using multiple different scanners to obtain panoramic slice images, including 3DHistech Pannoramic Flash II 250s, Hamamatsu C9600-12, and Aperio ScanScope AT2 scanners. The corresponding labels were annotated by multiple pathologists using the scanner's corresponding film reading software. Cohorts were constructed based on the presence or absence of labels. Among them, the unlabeled cohort had 887 unlabeled data, the supervised training cohort had 10,616 slice-level labeled data, and 350 had both slice-level and image block-level labeled data. At the same time, a validation cohort was constructed at an 8:2 ratio, with 280 slice-level labeled data and 8,492 slice-level and image block-level labeled data.

[0117] After collecting the data, foreground and background segmentation was performed on 2.5x hematoxylin and eosin (HE) staining images (16x downsampling). Foreground and background segmentation was achieved using a traditional image processing algorithm, including the following steps: 1. The binarization threshold was calculated using the Otsu method, and a binarized image was obtained based on the threshold. 2. An inversion operation was used to change the foreground area to white and the background to black. 3. Closed small holes were removed. 4. A closing operation, including an expansion operation with a radius of 5 and an erosion operation with a radius of 3, was used to eliminate small black holes that were not closed. 5. Closed small holes were removed again. The closing operation closes some holes to form new holes. This step is used to eliminate newly generated small holes. 6. Small targets were removed. This step removed useless areas outside the tissue, such as contamination and useless tissue fragments generated during the slicing process. The image foreground was obtained after the above steps. Finally, an image block of 224×224 pixels was extracted from the foreground area at a magnification of 20 times (about 0.5 μm per pixel), and an image block of 224×224 pixels was extracted with a magnification of 10 times (about 0.25 μm per pixel) centered on the image block.

[0118] Step 2: The network uses the Visual Transformer ViT Large, with an input size of 224×224. The input images are regularized with mean [0.485, 0.456, 0.406] and variance [0.229, 0.224, 0.225]. The network patches are 16×16. Training is performed using all the image patches from the dataset. Hyperparameter settings include the following: the AdamW optimizer, a cosine schedule from 0.04 to 0.2 for weight decay, a learning rate warmup of 3.5e-4 before 50,000 iterations, LayerScale, a cosine schedule from 0.994 to 1 for the teacher model momentum, a dropout of 0.4, and float16 half-precision computations for all computations except the loss function header. KoLeo regularization is added to the loss function with a weight of 0.1. Training is performed on two A100 cards with a batch size of 96.

[0119] Step 3: Extract features. For image features, the input image is regularized with a mean of [0.485, 0.456, 0.406] and a variance of [0.229, 0.224, 0.225]. The output feature is the visual token class token output by the last layer transformer module. For graph features, the cell visual transformer CellViT network is first used for cell segmentation and classification. Then, the nearest neighbor KNN algorithm is used to connect the 8 cells closest to each cell to construct a cell nucleus map. The single-cell morphology and topology atlas framework sc-MTOP is then used to extract cell manual features from the cell segmentation mask, cell image, and cell nucleus map. These cell manual features will be used as node features of the cell nucleus map. The following features are extracted for a cell:

[0120] Morphological features: area, bounding box area, eccentricity, cell-bounding box area ratio, major axis length, minor axis length, perimeter, solidity, elongation, roundness, mean, variance, maximum, and minimum values ​​of outline curvature;

[0121] Texture features: angular second moment, contrast, correlation, dissimilarity, energy, entropy, homogeneity, mean and variance of co-occurrence sum, mean, variance, maximum and minimum of cell pixel intensity;

[0122] Topological features: number of nodes in the subgraph, degree, coreness, clustering coefficient, eccentricity, normalized eccentricity, closeness centrality, betweenness centrality, normalized betweenness centrality, harmonic centrality, minimum edge length, and mean edge length.

[0123] Step 4: Use multi-instance learning for training. In this step, a 20x image is used as an instance, and the high-resolution image features of the 20x image block centered on the instance, the low-resolution image features of the 10x image block, and the cell nucleus map of the instance are obtained respectively. First, a graph transformer TransformerGraph and a transformer with a class token classtoken are used to obtain the graph representation of the cell nucleus map, and the graph representation, high-resolution features, and low-resolution features are spliced ​​as instance features. This step uses a double-hidden-layer multi-layer perceptron MLP as an instance classifier, and sets two different output layers as classification heads, one of which outputs the instance category score. For sample prediction with only slice-level labels, the first dimension of the output is normalized exponential function Softmax calculation, and its score is expressed as a binary classification score for each category; the other output slice category score For training with image patch labels, the second dimension is normalized by the exponential function Softmax calculation, and its score is expressed as the probability of each category. During training, only instances with pseudo labels or image patch labels are trained, that is, the instances k∈Top(a specified in formulas (5) and (6) i )and The hyperparameter selection strategy for this training process is mainly to avoid overfitting, including: the optimizer AdamW, weight decay using a cosine schedule from 0.04 to 0.2, parameter dropout rate of 0.4, and learning rate of 1e-3.

[0124] After the training is completed, the model effect is evaluated. In the experiment comparing the effects of the trained basic model and the native basic model, the image features extracted by the trained basic model and the image features extracted by the native basic model are used respectively, and used in the subsequent Gleason grading task. In the indicators of Accuracy and AUC, the basic model trained with prostate images improved by 7.24% and 5.41% respectively compared with the native basic model, which means that through training on specific prostate organ images, the model has improved performance in related downstream tasks. In the comparison between multi-task multi-instance learning and traditional single-task models, several classic single-task models are used for comparison under different tasks. For the image block classification task, under the same features, the instance classifiers of the multi-layer perceptron MLP model, support vector machine SVM model, random forest model and clustering-constrained attention multiple instance learning CLAM (Clustering-constrained Attention Multiple Instance Learning) model are used for comparison. For the slice-level classification task, using the same features, attention multi-instance learning (AttentionMIL), clustering-constrained attention multi-instance learning (CLAM), dual-stream multiple instance learning (DS-MIL), and variance multi-instance learning (VarianceMIL) were compared. Ablation experiments showed that the multi-branch model predictions were significantly higher than those of the single-branch model, indicating that the different branches in multi-task learning play a role in mutual guidance, resulting in higher performance. Finally, the consistency of the results between the multi-task branches was evaluated. For the instance classifier branch, the slice-level predictions with the largest classification exceeding 10% of the total number of image patches were used as the instance branch predictions. For the slice classifier branch, the predictions of the classifier were directly used, and the accuracy of the predictions between the two branches was calculated. The experiments found that the classification performance of the single-branch models was similar, indicating that the different branches of the model have consistent classification capabilities, which enhances the confidence of the diagnosis.

[0125] The present invention provides a method for automatic prostate Gleason grading based on the fusion of multi-level visual foundation model features. There are many methods and approaches for implementing this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A prostate Gleason automatic grading method based on multi-level visual basic model feature fusion, characterized by: The following steps are involved: Step 1, Data Collection and Preprocessing: Collect and normalize data, construct unlabeled and supervised training queues based on the presence or absence of labels, and perform foreground and background segmentation on panoramic slice images, cutting the foreground into image blocks of different resolutions; Step 2: Train the basic model: retrain the pre-trained basic model using image patches of different resolutions. Step 3: Use the trained model to extract image features at multiple resolutions, and use the single-cell morphology and topology atlas framework sc-MTOP to extract single-cell features and construct a cell nucleus map; Step 4: Update the node features of the cell nucleus map in step 3 and merge them with the multi-resolution image features in step 3 to construct a Gleason grading prediction model that integrates the image block level and the slice level. Use the Gleason grading prediction model to complete the automatic Gleason grading of the prostate.

2. The method according to claim 1, characterized in that In step 1, a data queue index is formulated to index and queue the panoramic slice image, foreground and background segmentation mask, image blocks and corresponding tags.

3. The method according to claim 2, characterized in that In step 2, the pre-trained basic model is the universal pathology model UNI.

4. The method according to claim 3, characterized in that In step 2, two identical network structures are used in the training process, namely the teacher model and the student model. The model framework is the visual transformer model ViT-Large. The training process uses the training strategy of the universal pathology model UNI. First, two pictures of the same size are randomly cropped from the image block and two types of random data enhancement are performed respectively. The two pictures of the same size are used as global images, one of which is further cropped into two or more local images, and the global image is randomly partially masked. The masked global image and the local image are used as the input of the student model together; the other global image is used as the input of the teacher model.

5. The method according to claim 4, characterized in that In step 2, the same self-supervised training method as the UNI model is used, that is, the existing improved training method based on the second version of the enhanced denoising anchor detection transformer DINOv2. The loss functions used include the DINO loss function and the online token bidirectional encoding transformer iBOT loss function. In the DINO loss function, the DINO head is added to the student model and the teacher model respectively to calculate the loss function. For the student model, the class token of each output is obtained and the student prototype score is output through a multi-layer perceptron MLP layer. The student prototype score is then normalized by the exponential function softmax to obtain the student score p. s ; For the teacher model, obtain the class token output by the teacher model and obtain the teacher prototype score through the multi-layer perceptron MLP layer. Add the Sinkhorn-Knop SK centralization process after the teacher prototype score, and then use the normalized exponential function softmax to calculate the teacher score p t The Sinkhorn-Knopp SK centralization process is a method for normalizing a matrix. In one iteration, normalization of rows and columns is performed separately. After two or more iterations, the matrix will be approximately a doubly random matrix. The specific process includes: Input: teacher prototype score Z, trainable clustering parameters θ C , number of iterations P; Initialization: The teacher prototype score Z0 = Z × C in the 0th iteration; The iterative steps include: For p, it is from 1 to P cycles: Temporary value Z1 of the p+1th iteration p+1 for in is a diagonal matrix, and the diagonal elements are The reciprocal of the sum of each row; The Z value of the p+1th iteration is in is a diagonal matrix, and the diagonal elements are The reciprocal of the sum of each column; Output the final teacher prototype score Z K .

6. The method according to claim 5, characterized in that In step 2, the normalized exponential function Softmax in the output heads of the student model and the teacher model introduces a sharpening operation, dividing the input data by a temperature constant, which is set to the square root of the dimension; Finally, for the student score p s With the teacher score p t Use the cross entropy CE loss function to obtain the DINO loss function L DINO : IT DINo =CE(p t ,p s )=-∑p t logp s (1) During the training process, the student model performs normal backpropagation to update the model parameters, while the teacher model does not perform backpropagation. The parameters of the teacher model are updated with momentum using the parameters of the student model.

7. The method according to claim 6, characterized in that In step 2, in the iBOT loss function L iBoT In the process, the loss function is calculated by adding a header to the student model and the teacher model. The input image of the student model randomly masks the area. In the student model, the iBOT header is used for the token corresponding to the masked area, and in the teacher model, the iBOT header is applied to the token at the same position. Subsequently, the iBOT header adopts the same loss function L as the DINO loss function. DIBO The same normalized exponential function softmax calculation and Sinkhorn-Knop SK centralization process are used, and the iBOT loss function L is obtained using the cross entropy CE loss function. iBOT : Among them, p ti is the teacher score for the ith covered region, p si is the student score of the i-th covered area; the parameters of the student model are trained and the parameters of the teacher model are adjusted through the momentum update mechanism.

8. The method according to claim 7, characterized in that In step 2, the model also uses KoLeo regularization during training to encourage the output features to be evenly distributed within a batch: given a set of network output features, the network output features are first regularized using the 2-norm, and then the feature x vector (x1, x2, ..., x n ), define the KoLeo loss function L koleo : in is the sth feature x s The minimum distance to other features in the batch, n is the total number of features, x n represents the nth feature.

9. The method according to claim 8, characterized in that Step 3 includes: using the student model to extract features of all image blocks and storing the results in a normalized manner, wherein the normalized storage includes image features, image block starting pixel coordinates, size, relative coordinates, panoramic slice image source, and magnification; The single-cell morphology and topology atlas framework sc-MTOP is used in image blocks to extract cell features and construct a cell nuclear map: cell segmentation is performed on the image blocks, the cell nuclear map is constructed based on spatial coordinate vectors and nearest neighbor KNN, and three types of manual features of single cells, texture, morphology, and topology, are extracted; in the cell nuclear map, the manual features of each cell are spliced ​​with the corresponding spatial position code as node features.

10. The method according to claim 9, characterized in that Step 4 includes: based on multi-instance learning, constructing a Gleason grade prediction model that takes multi-scale and cell nucleus images as input and outputs slice-level and image-block-level outputs through slice branches and image-block branches; In terms of input, multi-resolution images and cell nucleus maps are used as input. The high-resolution images provide image details, the low-resolution images provide a wider receptive field, and the cell nucleus maps provide information on the distribution and arrangement of cells in the image. In terms of output, slice-level output can obtain patient-level diagnostic results, and image block level output can obtain the spatial distribution of each grade; The Gleason grading prediction model uses the improved attention multi-instance method Attention-MIL for multi-instance learning, that is, a multi-layer perceptron (MLP) model is added to the original attention multi-instance Attention-MIL as an instance classifier, acting as an image block branch, and the multi-layer perceptron output of the attention multi-instance Attention-MIL is used as a slice classifier, acting as a slice branch. Specifically, the Gleason grading prediction model needs to simultaneously input image features at the image block level and cell-level nuclear map features at the cell level with two different resolutions. For the image features at the image block level, the multi-resolution image features extracted by the trained model in step 3 are directly used; for the cell-level features, the graph neural network graph transformer TransformerGraph model is first used to update the node features of the nuclear map, and deep features with contextual information are obtained by connecting neighboring nodes with manual features. Then, the nuclear map features at the cell level are obtained through a transformer with a class token class token; after feature extraction, the image features at the image block level and the nuclear map features at the cell level with two different resolutions are spliced ​​as the features of the high-resolution image, that is, the instance features; The improved attention multi-instance network is trained simultaneously through the two output branches of the image block branch instance classifier and the slice classifier of the slice branch during the training process. There are M feature numbers before the network calculation. The network first calculates the feature number based on the 1×M dimensional real vector space. The kth instance feature of Calculate the kth instance attention coefficient a k , calculated as: The network generates L features during calculation, and the model divides the results into C categories. V and U are real vector spaces in L×M dimensions. The learnable parameters of W is a real vector space in L×C dimensions The learnable parameters of Then use the attention coefficient to train the instance classifier and slice classifier separately; For instance classifiers: The training of instance classifiers is divided into two cases: one with image block classification labels and the other with only slice classification labels. Different classification heads are used for the two cases: When only slice classification is used, the classification head output is in the 2×C real vector space. Rating That is, the binary classification score of the instance for each category, which is used to indicate whether the instance belongs to the category; when there is image block classification, the classification head output is in the 1×C real vector space Rating That is, the score of the instance output for each category; let the classification probability of category i given by the instance classifier for instance k be y i,k ; When the sample only has a slice label, the priority method Top-K attention coefficient is used to extract the K highest and lowest instances and generate pseudo labels for instance classifier training: for the category label c of the input slice, the instance classifier model of the image block branch selects the top K instances with the highest attention coefficient and the top K instances with the lowest attention coefficient from the attention map of the corresponding category label c, and generates corresponding pseudo classification labels 1 and 0 respectively; for other categories, only the K highest attention instances of other categories are generated with pseudo label 0; finally, the instance and pseudo label pairs are spliced ​​and used for instance classifier training, and the training uses the cross entropy loss function L inst : Among them, Top(a i ) is the largest K image blocks in the attention of category i, Top(-a i ) is the smallest K image patches in the attention of category i, CE(1,y) is the cross entropy calculation between the prediction result y and the pseudo label 1; When the sample has an image block label, the instance classifier is trained directly using the image block level label: first, according to the doctor's segmentation label, the classification label is set for the image block at the corresponding position. For image block k, the number of pixels in image block k that are marked as a certain category is counted. When the number exceeds the threshold set by experience, the label of the image block is determined to be set as the category. Then, according to the image block classification, the attention coefficient is set for the image block. For classification c, it is set that there are image blocks in the slice. For the set of image blocks classified as c in the slice, the c-class attention label value of image block k is Set as Then, the cross entropy CE loss function is used for image block classification, and the mean square error MSE loss function is used for attention coefficient training: in is a set of labeled instances, is the set of instances with label 0, is a set of instances with label C, MSE represents the mean square error function, Prediction result y for the image block k With mark Cross entropy meter of ; For the slice classifier: the native Attention-MIL performs weighted addition of all instance features through the attention coefficient to obtain slice-level features, obtains slice classification through the fully connected layer FC, and then uses the cross entropy loss function CE for training. The instance feature aggregation formula and training loss function L slide for: in is the slice-level label, a k is the kth instance attention coefficient, h k is the kth instance feature, the fully connected layer FC is a linear mapping; y represents the output of the fully connected layer FC; Finally, through two loss functions L slide and L inst The instance classifier and slice classifier are trained simultaneously by superposition; In the testing phase, the classification head outputs the category of each image block to obtain the spatial prediction of each level, and the final slice-level prediction is obtained by combining the slice classifier output and the image block-level instance classifier.

Images