Prostate Glison automatic grading method based on multi-level visual basic model feature fusion

Abstract

The invention provides a prostate Glison automatic grading method based on multi-level visual basic model feature fusion. The prostate Glison automatic grading method comprises the following steps: step 1, collecting and preprocessing data; 2, training a basic model; 3, extracting image features under multiple resolutions by using the trained model, and extracting single cell features by using a single cell morphology and topology atlas framework sc-MTOP; and step 4, finishing automatic grading of the prostate Glison by adopting the Glison grading prediction model. According to the method, the multi-resolution image features and the single cell features are fused, so that more accurate Glison grading of the prostatic cancer pathological image is realized. By means of multi-center data collection and self-supervised training, the adaptability and generalization ability of the model to different data sources are improved. A cross attention module is introduced, multi-level features are fully integrated, and the classification accuracy is improved. And in combination with expert correction feedback and a pre-training basic model, the utilization efficiency and the model performance of the label-free data are further optimized.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a prostate Gleason automatic grading method based on multi-level visual basic model feature fusion. Background Art

[0002] Prostate cancer (PCa) is one of the most common malignancies in men worldwide, accounting for 15.1% of all male cancer diagnoses in 2020. Prostate cancer develops slowly and is highly heterogeneous, so accurate grading assessment is required for diagnosis and treatment. Currently, Gleason Grade (GG) is an important criterion for prostate cancer assessment, which can provide clinicians with a reference for tumor aggressiveness in order to formulate appropriate treatment strategies. By evaluating the microscopic and histological features of prostatectomy specimens and recurrent / routine biopsy tissue specimens, pathologists can detect malignant tumors and stratify patient management based on GG.

[0003] The evaluation process of Gleason grading is highly professional and often requires pathologists to spend a lot of time observing and judging under the microscope. However, this process is somewhat subjective, and the diagnostic results of different pathologists may differ, leading to the risk of misdiagnosis or overdiagnosis. With the increase in the number of pathological images, how to improve the efficiency and consistency of evaluation has become an urgent problem to be solved. Therefore, it is of great significance to develop an automated Gleason grading method, which can provide decision support for pathologists, reduce their workload, and improve the accuracy of prostate cancer diagnosis and treatment.

[0004] The automatic evaluation method of prostate pathology images based on artificial intelligence technology provides a feasible technical path to solve this problem. In recent years, deep learning models have made significant progress in the field of image recognition, especially the application of basic models, which makes it possible to extract potential features from large-scale data. By building an AI model based on prostate pathology images, it is possible to automatically identify histological features under the microscope, so as to quickly and accurately predict the Gleason grade. This type of technology can not only significantly shorten the evaluation time, but also improve the stability and consistency of diagnostic results by eliminating artificial differences, providing strong support for clinical decision-making. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to construct a basic model for feature extraction of prostate cancer pathology images, extract key histological features and single-cell features from prostate pathology slice images, and automatically evaluate the aggressiveness of the tumor based on these features, thereby ensuring the stability, accuracy and efficiency of the Gleason grading of prostate cancer in multi-center pathology images.

[0006] Technical solution: The present invention specifically provides a prostate Gleason automatic grading method based on multi-level visual basic model feature fusion, comprising the following steps:

[0007] Step 1, data collection and preprocessing: collect and normalize data, build unlabeled queues and supervised training queues according to the presence or absence of labels, and perform foreground and background segmentation on panoramic slice images, cutting the foreground into image blocks of different resolutions;

[0008] Step 2: Train the basic model: retrain the pre-trained basic model using image patches of different resolutions;

[0009] Step 3: Use the trained model to extract image features at multiple resolutions, and use the single cell morphological and topological profiling framework sc-MTOP (Single Cell Morphological and Topological Profiling) to extract single cell features;

[0010] Step 4, using the features extracted in step 3 to perform interactive fusion between features, construct a Gleason grading prediction model that integrates the image block level and the slice level, and use the Gleason grading prediction model to complete the automatic Gleason grading of the prostate.

[0011] In step 1, the data acquisition and collection scope includes public data sets PANDA Challenge, Gallo, Tolkach and SICAP-MIL, among which PANDA Challenge data are annotated by pathologists to outline the score areas of interstitial, benign, and Gleason, a total of five grading areas. First, according to the general empirical method of foreground and background segmentation, based on image processing methods such as Otsu threshold segmentation, inversion, hole filling, image expansion, image corrosion and small target removal, the foreground and background segmentation of panoramic slice images is realized. Then, image blocks of specified sizes are extracted in the foreground area for use in subsequent steps. Finally, a data queue index is formulated to index and queue the panoramic slice images, foreground and background segmentation masks, image blocks and corresponding tags.

[0012] In step 2, the pre-trained basic model is the universal pathology model UNI.

[0013] In step 2, the collected data is used to retrain the public pre-trained basic model universal pathology model UNI to enhance the feature extraction capability of prostate pathology images. Two identical network structures are used in the training process, namely the teacher model and the student model. The model framework is the vision transformer model ViT-Large (Vision Transformer Large). The training process uses the training strategy of the universal pathology model UNI. First, two pictures of the same size are randomly cut from the image block and two kinds of random data enhancement are performed respectively. The two pictures of the same size are used as the global image, one of which is further cut into more than two local images, and the global image is randomly partially masked. The masked global image and the local image are used as the input of the student model together; the other global image is used as the input of the teacher model.

[0014] In step 2, the same self-supervised training method as the UNI model is used, that is, the existing training improvement method based on the second version of the enhanced denoising anchor detection transformer DINOv2 (DETR with Improved deNoising anchor boxes), and the loss functions used include the DINO loss function and the online token bidirectional encoding transformer iBOT (Image BERT Pre-Training with Online Tokenizer) loss function;

[0015] In the DINO loss function, DINO heads are added to the student model and the teacher model respectively to calculate the loss function. For the student model, each output class token is obtained, and the student prototype score is output through a multilayer perceptron (MLP) layer. Then, the student prototype score is normalized by the exponential function softmax to obtain the student score p s ;

[0016] In order to avoid the degeneration of the model, that is, the output features tend to be the same, the output branch of the teacher model is modified. For the teacher model, the class token output by the teacher model is obtained and the teacher prototype score is obtained through the multi-layer perceptron MLP layer. The Sinkhorn-Knopp centralization process is added after the teacher prototype score, and then the normalized exponential function softmax is used to calculate the teacher score p t , wherein the Sinkhorn-Knopp SK centralization process is a method for normalizing a matrix. In one iteration, normalized rows and normalized columns are performed separately. After two or more iterations, the matrix will be approximately a double random matrix. The specific process includes:

[0017] Input: teacher prototype score Z, trainable clustering parameter θ C , number of iterations P;

[0018] Initialization: The teacher prototype score Z0 = Z × C in the 0th iteration;

[0019] The iteration steps include:

[0020] For p, from 1 to P cycles:

[0021] The temporary value Z1 of the p+1th iteration p+1 for

[0022] in is a diagonal matrix with diagonal elements The reciprocal of the sum of each row;

[0023] The Z value of the p+1th iteration is

[0024] in is a diagonal matrix with diagonal elements The reciprocal of the sum of each column;

[0025] Output the final teacher prototype score Z K .

[0026] In step 2, the normalized exponential function Softmax in the output head of the student model and the teacher model introduces a sharpening operation, dividing the input data by a temperature constant, which is set to the square root of the dimension to widen the gap between the features of different channels of the model. By combining centering and sharpening, the features of the model output are prevented from being degraded.

[0027] Finally, for the student score p s With the teacher score p t Use the cross entropy CE (Cross Entropy) loss function to obtain the DINO loss function L DINO :

[0028] L dINo =CE(p t ,p s )=-Σp t logp s (1)

[0029] During the training process, the student model performs normal back-propagation to update the model parameters, while the teacher model does not perform back-propagation. The parameters of the teacher model are updated with momentum using the parameters of the student model.

[0030] In step 2, in the iBOT loss function L iBOTIn the above, we add headers to the student model and the teacher model to calculate the loss function. We randomly mask areas in the input image of the student model. In the student model, we use the iBOT header for the tokens corresponding to the masked areas, and in the teacher model, we use the iBOT header for the tokens at the same position. Then, we use the same loss function L as the DINO loss function. DINO The same normalized exponential function softmax calculation and Sinkhorn-Knopp SK centralization process, and the cross entropy CE loss function is used to obtain the iBOT loss function L iBOT :

[0031]

[0032] Among them, p ti is the teacher score for the ith covered region, p si is the student score of the i-th covered region; the parameters of the student model are trained and the parameters of the teacher model are adjusted through the momentum update mechanism.

[0033] In step 2, the model also uses KoLeo regularization during training to encourage the output features to be evenly distributed within a batch: given a set of network output features, the network output features are first regularized using the 2-norm, and then the feature x vector (x1, x2, …, x n ), define the KoLeo loss function L koleo :

[0034]

[0035] where d s =min t≠s ‖x s -x t ‖ is the sth feature x s The minimum distance to other features in the batch, n is the total number of features, x n represents the nth feature.

[0036] Step 3 includes: using the student model to extract features of all image blocks, and storing the results in a normalized manner, wherein the normalized storage includes image features, image block starting pixel coordinates, size, relative coordinates, panoramic slice image source, and magnification;

[0037] The single-cell morphology and topology atlas framework sc-MTOP is used in the image block to extract cell features and construct a cell nucleus map: cell segmentation is performed on the image block, the cell nucleus map is constructed based on the spatial coordinate vector and the nearest neighbor KNN (K-nearst neighbor), and three types of manual features of single cells, texture, morphology and topology, are extracted; in the cell nucleus map, the manual features of each cell are spliced ​​with the corresponding spatial position code as the node feature.

[0038] Step 4 includes: based on multi-instance learning, constructing a Gleason grading prediction model that takes multi-scale and cell nucleus images as inputs and outputs slice-level and image-block-level outputs by slice branches and image-block branches;

[0039] In terms of input, multi-resolution images and cell nucleus maps are used as input at the same time, where high-resolution images provide image details, low-resolution images provide a wider receptive field, and cell nucleus maps provide information on the distribution and arrangement of cells in the image;

[0040] In terms of output, the slice-level output can obtain the patient-level diagnosis results, and the image block level can obtain the spatial distribution of each grade;

[0041] The Gleason grading prediction model uses the improved attention multiple instance method Attention-MIL (Attention Multiple Instance Learning) for multiple instance learning, that is, a multi-layer perceptron MLP model is added as an instance classifier on the basis of the original attention multiple instance Attention-MIL, acting as an image block branch, and the multi-layer perceptron output of the attention multiple instance Attention-MIL is used as a slice classifier, acting as a slice branch, specifically including: the Gleason grading prediction model needs to simultaneously input image features at the image block level and cell nucleus map features at the cell level with two different resolutions. For the image features at the image block level, the multi-resolution image features extracted by the trained model in step 3 are directly used; for the cell level features, the graph neural network graph transformer TransformerGraph model is first used to update the node features of the cell nucleus map, and the deep features with contextual information are obtained from the manual feature contact neighbor nodes, and then the cell level cell nucleus map features are obtained through a transformer with a class token class token; after the feature extraction is completed, the image features at the image block level of two different resolutions and the cell level cell nucleus map features are spliced ​​as the features of the high-resolution image, that is, the instance features;

[0042] The improved attention multi-instance network is trained through two output branches, the instance classifier of the image block branch and the slice classifier of the slice branch, during the training process. There are M feature numbers before the network calculation. The network first calculates the feature number according to the 1×M dimensional real vector space. The kth instance feature of Calculate the kth instance attention coefficient a k , calculated as:

[0043]

[0044] The network generates L feature numbers during calculation, and the model divides the results into C categories. V and U are real vector spaces in L×M dimensions. The learnable parameters are W is a real vector space in L×C dimensions. The learnable parameters are Then the instance classifier and slice classifier are trained separately using the attention coefficient;

[0045] For instance classifier: The training of instance classifier is divided into two cases: one with image block classification labels and one with slice classification labels only. Different classification heads are used for the two cases: When only slice classification is performed, the classification head output is in the 2×C real vector space. Rating That is, the binary classification score of the instance for each category, which is used to indicate whether the instance belongs to the category; when there is image block classification, the classification head output is in the 1×C real vector space Rating That is, the score of the instance output for each category; let the probability of classifying category i given by the instance classifier for instance k be y i,k ;

[0046] When the sample only has a slice label, the priority method Top-K attention coefficient is used to extract the K highest and lowest instances and generate pseudo labels for instance classifier training: for the category label c of the input slice, the instance classifier model of the image block branch selects the top K instances with the highest attention coefficient and the top K instances with the lowest attention coefficient from the attention map of the corresponding category label c, and generates corresponding pseudo classification labels 1 and 0 respectively; for other categories, only the pseudo label 0 is generated for the K highest attention instances of other categories; finally, the instance and pseudo label pairs are concatenated and used for instance classifier training, and the training uses the cross entropy loss function L inst :

[0047]

[0048] Among them, Top(a i) is the largest K image blocks in the attention of category i, Top(-a i ) is the smallest K image patches in the attention of category i, CE(1,y) is the cross entropy calculation between the prediction result k and the pseudo label 1;

[0049] When the sample has an image block label, the instance classifier is trained directly using the image block level label: first, according to the doctor's segmentation label, the classification label is set for the image block at the corresponding position. For image block k, the number of pixels marked as a certain category in image block k is counted. When the number exceeds the empirically set threshold of 75%, the label of the image block is determined to be set as the category. Then, the attention coefficient is set for the image block according to the image block classification. For category c, it is set that there are image blocks in the slice. is the set of image blocks classified as c in the slice, then the c-class attention label value of image block k is Set as Then, the cross entropy CE loss function is used for image block classification, and the mean square error MSE loss function is used for attention coefficient training:

[0050]

[0051] in is a set of labeled instances, is the set of instances with label 0, is a set of instances with label C, MSE represents the mean square error function, Prediction result y for the image block k With mark Calculate the cross entropy of ;

[0052] For the slice classifier: the native Attention-MIL performs weighted addition of the features of all instances through the attention coefficient to obtain slice-level features, obtains slice classification through the fully connected layer FC (Fully Connected), and then uses the cross entropy loss function CE for training. The instance feature aggregation formula and training loss function L slide for:

[0053]

[0054] in is the slice-level label, a k is the kth instance attention coefficient, h k is the kth instance feature, the fully connected layer FC is a linear mapping; y represents the output of the fully connected layer FC;

[0055] Finally, through two loss functions L slide and Linst The instance classifier and the slice classifier are trained simultaneously by superposition;

[0056] In the test phase, the classification head outputs the category of each image block, obtains the spatial prediction of each level, and combines the slice classifier output and the instance classifier at the image block level to obtain the final slice-level prediction. The slice classifier output and the instance classifier at the image block level are combined to obtain the final slice-level prediction. The training model is verified using the test set. The evaluation content includes the comparison of the effect of the trained basic model and the native basic model, the comparison of multi-task multi-instance learning and the traditional single-task model, and the consistency analysis of the image block-level prediction and the slice-level prediction. The evaluation indicators use accuracy, sensitivity, specificity, precision, F1 score F1Score and AUC (Area Under the Curve) to evaluate the prediction performance of the model. First, count the predicted results and true results of the test set to obtain the confusion matrix, including true positive TP (True Positive), true negative TN (True Negative), false positive FP (False Positive), false negative FN (False Negative), and then calculate each indicator based on the confusion matrix. For accuracy, there is (TP+TN) / (TP+TN+FP+FN); for sensitivity, there is TP / (TP+FN); for specificity, there is TN / (TN+FP); for precision, there is TP / (TP+FP); for F1 score, there is 2*(Precision*Recall) / (Precision+Recall); for AUC, first change the classification threshold, draw the ROC (Receiver Operating Characteristic Curve) curve with 1-specificity as the horizontal axis and sensitivity as the vertical axis, and then calculate the area under the curve, which is the AUC.

[0057] Beneficial effects: The present invention achieves more accurate Gleason grading of prostate cancer pathology images by fusing multi-resolution image features with single-cell features. Multi-center data collection and self-supervised training are used to improve the adaptability and generalization ability of the model to different data sources. The cross-attention module is introduced to fully integrate multi-level features and improve grading accuracy. Combining expert correction feedback with pre-trained basic models, the utilization efficiency of unlabeled data and model performance are further optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0059] Figure 1 Schematic diagram of the entire process of the artificial intelligence method for evaluating Gleason grading based on the basic model features of prostate pathology images.

[0060] Figure 2 This is a flowchart for training the basic model for prostate diseases.

[0061] Figure 3 Flowchart for extracting multi-resolution image block features and cell nucleus features.

[0062] Figure 4 The flowchart for training the Gleason grading prediction model at the image block level and slice level based on image features and graph features at multiple resolutions. DETAILED DESCRIPTION

[0063] like Figure 1 As shown, the embodiment of the present invention provides a prostate Gleason automatic grading method based on multi-level visual basic model feature fusion, comprising the following steps:

[0064] Step 1, data collection and preprocessing: collect and normalize data, build unlabeled queues and supervised training queues according to the presence or absence of labels, and perform foreground and background segmentation on panoramic slice images, cutting the foreground into image blocks of different resolutions;

[0065] Step 2: Train the basic model: retrain the pre-trained basic model using image patches of different resolutions;

[0066] Step 3: Use the trained model to extract image features at multiple resolutions, and use the single cell morphological and topological profiling framework sc-MTOP (Single Cell Morphological and Topological Profiling) to extract single cell features;

[0067] Step 4, using the features extracted in step 3 to perform interactive fusion between features, construct a Gleason grading prediction model that integrates the image block level and the slice level, and use the Gleason grading prediction model to complete the automatic Gleason grading of the prostate.

[0068] In step 1, the data collection includes the public data sets PANDA Challenge, Gallo, Tolkach and SICAP-MIL. The PANDA Challenge data is annotated by pathologists to delineate five grading areas: interstitial, benign, and Gleason's scoring areas. Figure 1As shown in the data preparation in , firstly, according to the general empirical method of foreground and background segmentation, the foreground and background segmentation of the panoramic slice image is realized based on image processing methods such as Otsu threshold segmentation, inversion, hole filling, image dilation, image erosion and small target removal. Then, image blocks of specified size are extracted in the foreground area for use in subsequent steps. Finally, a data queue index is formulated to index and queue the panoramic slice image, foreground and background segmentation mask, image blocks and corresponding tags.

[0069] In step 2, the pre-trained basic model is the universal pathology model UNI.

[0070] In step 2, the whole process is Figure 1 The collected data is used to train the public pre-trained base model using Figure 2 The universal pathology model UNI is trained again according to the prostate disease-specific basic model training process (the basic model training process includes a teacher network and a student network. The student network input is the global image and local image obtained by cropping and randomly enhancing the original image, and the teacher network input is the global image cropped from the original image. Then the two networks obtain the features of each image, use the iBOT and DINO loss functions to calculate the differences between the features, and use gradient back propagation to train the two networks. The student network Pros-UNI is used for subsequent feature extraction) to obtain a prostate-specific basic model and enhance the feature extraction capability of prostate pathology images. Two identical network structures are used in the training process, namely the teacher model and the student model. The model framework is the vision transformer model ViT-Large (Vision Transformer Large). The training process uses the training strategy of the universal pathology model UNI. First, two pictures of the same size are randomly cropped from the image block and two kinds of random data enhancement are performed respectively. The two pictures of the same size are used as the global image, one of which is further cropped into more than two local images, and the global image is randomly partially masked. The masked global image and the local image are used as the input of the student model together; the other global image is used as the input of the teacher model.

[0071] In step 2, the same self-supervised training method as the UNI model is used, that is, the existing training improvement method based on the second version of the enhanced denoising anchor point detection transformer DINOv2 (DETR with Improved deNoising anchor boxes), and the loss function used includes Figure 2 The DINO loss function in and the iBOT (Image BERTPre-Training with Online Tokenizer) loss function;

[0072] In the DINO loss function, DINO heads are added to the student model and the teacher model respectively to calculate the loss function. For the student model, each output class token is obtained, and the student prototype score is output through a multilayer perceptron (MLP) layer. Then, the student prototype score is normalized by the exponential function softmax to obtain the student score p s ;

[0073] In order to avoid the degeneration of the model, that is, the output features tend to be the same, the output branch of the teacher model is modified. For the teacher model, the class token output by the teacher model is obtained and the teacher prototype score is obtained through the multi-layer perceptron MLP layer. The Sinkhorn-Knopp centralization process is added after the teacher prototype score, and then the normalized exponential function softmax is used to calculate the teacher score p t , wherein the Sinkhorn-Knopp SK centralization process is a method for normalizing a matrix. In one iteration, normalized rows and normalized columns are performed separately. After two or more iterations, the matrix will be approximately a double random matrix. The specific process includes:

[0074] Input: teacher prototype score Z, trainable clustering parameter θ C , number of iterations P;

[0075] Initialization: The teacher prototype score Z0 = Z × C in the 0th iteration;

[0076] The iteration steps include:

[0077] For p, from 1 to P cycles:

[0078] The temporary value Z1 of the p+1th iteration p+1 for

[0079] in is a diagonal matrix with diagonal elements The reciprocal of the sum of each row;

[0080] The Z value of the p+1th iteration is

[0081] in is a diagonal matrix with diagonal elements The reciprocal of the sum of each column;

[0082] Output the final teacher prototype score Z K .

[0083] In step 2, the normalized exponential function Softmax in the output head of the student model and the teacher model introduces a sharpening operation, dividing the input data by a temperature constant, which is set to the square root of the dimension to widen the gap between the features of different channels of the model. By combining centering and sharpening, the features of the model output are prevented from being degraded.

[0084] Finally, for the student score p s With the teacher score p t Use the cross entropy CE (Cross Entropy) loss function to obtain the DINO loss function L DINO :

[0085] L DINO =CE(p t ,p s )=-∑p t logp s (1)

[0086] During the training process, Figure 1 Basic model training and Figure 2 As shown, the student model performs normal back propagation to update the model parameters, the teacher model does not perform back propagation, and the parameters of the teacher model are updated with momentum using the parameters of the student model.

[0087] In step 2, in the iBOT loss function L iBOT In the above, we add headers to the student model and the teacher model to calculate the loss function. We randomly mask areas in the input image of the student model. In the student model, we use the iBOT header for the tokens corresponding to the masked areas, and in the teacher model, we use the iBOT header for the tokens at the same position. Then, we use the same loss function L as the DINO loss function. DINO The same normalized exponential function softmax calculation and Sinkhorn-Knopp SK centralization process, and the cross entropy CE loss function is used to obtain the iBOT loss function L iBOT :

[0088]

[0089] Among them, p ti is the teacher score for the ith covered region, p si is the student score of the i-th covered region; the parameters of the student model are trained and the parameters of the teacher model are adjusted through the momentum update mechanism.

[0090] In step 2, the model also uses KoLeo regularization during training to encourage the output features to be evenly distributed within a batch: given a set of network output features, the network output features are first regularized using the 2-norm, and then the feature x vector (x1, x2, …, x n ), define the KoLeo loss function L koleo :

[0091]

[0092] where d s =min t≠s ‖x s -x t ‖ is the sth feature x s The minimum distance to other features in the batch, n is the total number of features, x n represents the nth feature.

[0093] Step 3 includes: using the student model to extract features of all image blocks, and storing the results in a normalized manner, wherein the normalized storage includes image features, image block starting pixel coordinates, size, relative coordinates, panoramic slice image source, and magnification;

[0094] like Figure 1 Feature extraction and Figure 3 As shown, the single-cell morphology and topology atlas framework sc-MTOP is used in the image block to extract cell features and construct a cell nucleus map: cell segmentation is performed on the image block, the cell nucleus map is constructed based on the spatial coordinate vector and the nearest neighbor KNN (K-nearst neighbor), and three types of manual features of single cell texture, morphology and topology are extracted; in the cell nucleus map, the manual features of each cell are spliced ​​with the corresponding spatial position code as the node feature.

[0095] Step 4 includes: based on multi-instance learning, constructing a Gleason grading prediction model that takes multi-scale and cell nucleus images as inputs and outputs slice-level and image-block-level outputs by slice branches and image-block branches;

[0096] like Figure 1 Prediction model training and Figure 4 As shown in the figure, in terms of input, multi-resolution images and cell nucleus maps are used as input at the same time, where high-resolution images provide image details, low-resolution images provide a wider receptive field, and cell nucleus maps provide information on the distribution and arrangement of cells in the image; in terms of output, the slice-level output can obtain the diagnosis results at the patient level, and the image block level can obtain the spatial distribution of each grade;

[0097] The Gleason grading prediction model uses the improved attention multiple instance method Attention-MIL (Attention Multiple Instance Learning) for multi-instance learning, that is, a multi-layer perceptron MLP model is added on the basis of the original attention multiple instance Attention-MIL as an instance classifier, acting as an image block branch, and the multi-layer perceptron output of the attention multiple instance Attention-MIL is used as a slice classifier, acting as a slice branch. Specifically, the Gleason grading prediction model needs to input image features at the image block level and cell nucleus map features at the cell level at two different resolutions at the same time. For the image features at the image block level, the multi-resolution image features extracted by the trained model in step 3 are directly used; for the cell level features, the graph neural network graph transformer TransformerGraph model is first used to update the node features of the cell nucleus map, and the deep features with contextual information are obtained from the manual features to contact the neighboring nodes, and then the cell level cell nucleus map features are obtained through a transformer with a class token class token; after the feature extraction is completed, the image features at the image block level of two different resolutions and the cell level cell nucleus map features are spliced ​​as the features of the high-resolution image, that is, the instance features, such as Figure 1 Prediction model training and Figure 4 As shown in the splicing features;

[0098] The improved attention multi-instance network is trained simultaneously through the two output branches of the image block branch instance classifier and the slice classifier of the slice branch during the training process, such as Figure 1 Prediction model training and Figure 4 As shown in the Slice Classifier and Instance Classifier section in . There are M features before the network calculation. The network first calculates the features according to the 1×M dimensional real vector space. The kth instance feature of Calculate the kth instance attention coefficient a k , calculated as:

[0099]

[0100] The network generates L feature numbers during calculation, and the model divides the results into C categories. V and U are real vector spaces in L×M dimensions. The learnable parameters are W is a real vector space in L×C dimensions. The learnable parameters are Then the instance classifier and slice classifier are trained separately using the attention coefficient;

[0101] For instance classifier: The training of instance classifier is divided into two cases: one with image block classification labels and one with slice classification labels only. Different classification heads are used for the two cases: When only slice classification is performed, the classification head output is in the 2×C real vector space. Rating That is, the binary classification score of the instance for each category, which is used to indicate whether the instance belongs to the category; when there is image block classification, the classification head output is in the 1×C real vector space Rating That is, the score of the instance output for each category; let the classification probability of category i given by the instance classifier for instance K be y i,k ;

[0102] When the sample only has a slice label, the priority method Top-K attention coefficient is used to extract the K highest and lowest instances and generate pseudo labels for instance classifier training: for the category label c of the input slice, the instance classifier model of the image block branch selects the top K instances with the highest attention coefficient and the top K instances with the lowest attention coefficient from the attention map of the corresponding category label c, and generates corresponding pseudo classification labels 1 and 0 respectively; for other categories, only the pseudo label 0 is generated for the K highest attention instances of other categories; finally, the instance and pseudo label pairs are concatenated and used for instance classifier training, and the training uses the cross entropy loss function L inst :

[0103]

[0104] Among them, Top(a i ) is the largest K image blocks in the attention of category i, Top(-a i ) is the smallest K image patches in the attention of category i, CE(1,y) is the cross entropy calculation between the prediction result y and the pseudo label 1;

[0105] When the sample has an image block label, the instance classifier is trained directly using the image block level label: first, according to the doctor's segmentation label, the classification label is set for the image block at the corresponding position. For image block k, the number of pixels marked as a certain category in image block k is counted. When the number exceeds the empirically set threshold of 75%, the label of the image block is determined to be set as the category. Then, the attention coefficient is set for the image block according to the image block classification. For category c, it is set that there are image blocks in the slice. is the set of image blocks classified as c in the slice, then the c-class attention label value of image block k is Set as Then, the cross entropy CE loss function is used for image block classification, and the mean square error MSE loss function is used for attention coefficient training:

[0106]

[0107] in is a set of labeled instances, is the set of instances with label 0, is a set of instances with label C, MSE represents the mean square error function, Prediction result y for the image block k With mark Calculate the cross entropy of ;

[0108] For the slice classifier: the native Attention-MIL performs weighted addition of the features of all instances through the attention coefficient to obtain slice-level features, obtains slice classification through the fully connected layer FC (Fully Connected), and then uses the cross entropy loss function CE for training. The instance feature aggregation formula and training loss function L slide for:

[0109]

[0110] in is the slice-level label, a k is the kth instance attention coefficient, h k is the kth instance feature, the fully connected layer FC is a linear mapping; y represents the output of the fully connected layer FC;

[0111] Finally, through two loss functions L slide and L inst The instance classifier and the slice classifier are trained simultaneously by superposition;

[0112] In the test phase, the classification head outputs the category of each image block, obtains the spatial prediction of each level, and combines the slice classifier output and the instance classifier at the image block level to obtain the final slice-level prediction. The slice classifier output and the instance classifier at the image block level are combined to obtain the final slice-level prediction. The training model is verified using the test set. The evaluation content includes the comparison of the effect of the trained basic model and the native basic model, the comparison of multi-task multi-instance learning and the traditional single-task model, and the consistency analysis of the image block-level prediction and the slice-level prediction. The evaluation indicators use accuracy, sensitivity, specificity, precision, F1 score F1Score and AUC (Area Under the Curve) to evaluate the prediction performance of the model. First, count the predicted results and true results of the test set to obtain the confusion matrix, including true positive TP (True Positive), true negative TN (True Negative), false positive FP (False Positive), false negative FN (False Negative), and then calculate each indicator based on the confusion matrix. For accuracy, there is (TP+TN) / (TP+TN+FP+FN); for sensitivity, there is TP / (TP+FN); for specificity, there is TN / (TN+FP); for precision, there is TP / (TP+FP); for F1 score, there is 2*(Precision*Recall) / (Precision+Recall); for AUC, first change the classification threshold, draw the ROC (Receiver Operating Characteristic Curve) curve with 1-specificity as the horizontal axis and sensitivity as the vertical axis, and then calculate the area under the curve, which is the AUC.

[0113] In another specific embodiment of the present invention, a method for automatic prostate Gleason grading based on multi-level visual basic model feature fusion is provided, comprising the following steps:

[0114] Step 1: The data set used in the embodiment of the present invention has a total of 11,853 slice samples, and panoramic slice images are obtained by digitizing them through multiple different scanners, including 3DHistech Pannoramic Flash II 250s, Hamamatsu C9600-12 and Aperio ScanScope AT2 scanners, and the corresponding labels are annotated by multiple pathologists using the corresponding film reading software of the scanner. A queue is constructed according to the presence or absence of labels, of which the unlabeled queue has 887 unlabeled data, the supervised training queue has 10,616 slice-level label data, and 350 slice-level label data and image block-level label data. At the same time, a verification queue is constructed according to the ratio of 8:2, with 280 slice-level label data and 8,492 slice-level label data and image block-level label data.

[0115] After collecting the data, foreground and background segmentation was performed on 2.5x hematoxylin and eosin (HE) staining images (16x downsampling). Foreground and background segmentation was achieved using a traditional image processing algorithm, including the following steps: 1. The binarization threshold was calculated using the Otsu method, and a binarized image was obtained based on the threshold. 2. The inversion operation was used to make the foreground area white and the background black. 3. Closed small holes were removed. 4. A closing operation was used, including a dilation operation with a radius of 5 and an erosion operation with a radius of 3, to eliminate small black holes that were not closed. 5. Closed small holes were removed again. The closing operation closed some holes to form new holes. This step was used to eliminate newly generated small holes. 6. Small targets were removed. This step removed useless areas outside the tissue, such as contamination and useless tissue fragments generated during the slicing process. The image foreground was obtained after the above steps. Finally, an image block of 224×224 pixels was extracted from the foreground area at 20 times (about 0.5 μm per pixel), and an image block of 224×224 pixels was extracted with 10 times magnification (about 0.25 μm per pixel) centered on the image block.

[0116] Step 2: The network uses the Vision Transformer ViT Large, the input size is 224×224, the input image is regularized with mean [0.485, 0.456, 0.406] and variance [0.229, 0.224, 0.225], and the network image patch size is 16×16. Use image patches of all data for training. The specific hyperparameters are set as follows: use the optimizer AdamW, weight decay Weight Decay uses a cosine schedule from 0.04 to 0.2, warm up the learning rate before 50,000 iterations, the learning rate is 3.5e-4, use the network layer parameter scaling LayerScale, the teacher model momentum uses a cosine schedule from 0.994 to 1, dropout uses 0.4, and float16 half-precision calculations are used for calculations except for the loss function head. KoLeo regularization is added to the loss function with a weight of 0.1. Training is performed on 2 A100 cards, and the batch size is set to 96.

[0117] Step 3: Extract features. For image features, the input image is regularized with mean [0.485, 0.456, 0.406] and variance [0.229, 0.224, 0.225]. The output feature is the visual token class token output by the last layer transformer module. For graph features, the cell visual transformer CellViT network is first used for cell segmentation and classification. Then, the nearest neighbor KNN algorithm is used to connect the 8 cells closest to each cell to construct a cell nucleus map. Then, the single-cell morphology and topology atlas framework sc-MTOP is used to extract cell manual features from the cell segmentation mask, cell image and cell nucleus map. These cell manual features will be used as node features of the cell nucleus map. The following features are extracted for a cell:

[0118] Morphological features: mean, variance, maximum, and minimum of area, bounding box area, eccentricity, cell-bounding box area ratio, major axis length, minor axis length, perimeter, solidity, elongation, roundness, and contour curvature;

[0119] Texture features: angular second moment, contrast, correlation, dissimilarity, energy, entropy, homogeneity, mean, variance of co-occurrence, mean, variance, maximum and minimum of cell pixel intensity;

[0120] Topological features: number of nodes in the subgraph, degree, coreness, clustering coefficient, eccentricity, standardized eccentricity, closeness centrality, betweenness centrality, standardized betweenness centrality, harmonic centrality, minimum edge length, and mean edge length.

[0121] Step 4: Use multi-instance learning for training. In this step, the 20x image is taken as an instance, and the high-resolution image features of the 20x image block centered on the instance, the low-resolution image features of the 10x image block, and the cell nucleus map of the instance are obtained respectively. First, a graph transformer TransformerGraph and a transformer Transformer with a class token classtoken are used to obtain the graph representation of the cell nucleus map, and the graph representation, high-resolution features, and low-resolution features are concatenated as instance features. This step uses a double-hidden-layer multi-layer perceptron MLP as an instance classifier, and sets two different output layers as classification heads, one of which outputs the instance category score. For sample prediction with only slice-level labels, the first dimension of the output is normalized exponential function Softmax calculation, and its score is expressed as a binary classification score for each category; the other output slice category score For training with image patch labels, the normalized exponential function Softmax is calculated for the second dimension, and its score is expressed as the probability of each category. During training, only instances with pseudo labels or image patch labels are trained, that is, the instances k∈Top(a specified in formulas (5) and (6) i )and The hyperparameter selection strategy of this training process is mainly to avoid overfitting, including: optimizer AdamW, weight decay Weight Decay uses a cosine schedule from 0.04 to 0.2, parameter dropout rate Dropout is 0.4, and learning rate is 1e-3.

[0122] After the training is completed, the model effect is evaluated. In the experiment comparing the effects of the trained basic model and the native basic model, the image features extracted by the trained basic model and the native basic model are used respectively, and then used in the subsequent Gleason grading task. In the indicators of Accuracy and AUC, the basic model trained with prostate images improved by 7.24% and 5.41% respectively compared with the native basic model, which means that through specific prostate organ image training, the model has improved performance in related downstream tasks. In the comparison between multi-task multi-instance learning and traditional single-task models, several classic single-task models are used for comparison under different tasks. For the image block classification task, under the same features, the instance classifiers of the multi-layer perceptron MLP model, support vector machine SVM model, random forest model and clustering-constrained attention multiple instance learning CLAM (Clustering-constrained Attention Multiple Instance Learning) model are used for comparison. For the slice-level classification task, under the same features, attention multi-instance learning AttentionMIL, clustering-constrained attention multi-instance learning CLAM, dual-stream multiple instance learning DS-MIL (Dual-stream multiple instance learning) and variance multi-instance learning Variance MIL were used for comparison. In the ablation experiment, it was found that the prediction results of the multi-branch model were significantly higher than the single-branch results, which means that different branches in multi-task learning play a role of mutual prompting, thereby obtaining higher performance. Finally, the consistency of the results between multi-task branches was evaluated. For the instance classifier branch, the maximum classification exceeding 10% of the total number of image blocks was used as the slice-level prediction of the instance branch prediction. For the slice classifier branch, the prediction results of the classifier were directly taken, and the accuracy of the prediction results between the two branches was calculated. In the experiment, it was found that the classification performance of the single-branch model was close, which means that different branches of the model have consistent classification capabilities, which enhances the diagnostic credibility.

[0123] The present invention provides a prostate Gleason automatic grading method based on multi-level visual basic model feature fusion. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A prostate Gleason automatic grading method based on multi-level visual basic model feature fusion, characterized in that: The following steps are involved: Step 1, data collection and preprocessing: collect and normalize data, build unlabeled queues and supervised training queues according to the presence or absence of labels, and perform foreground and background segmentation on panoramic slice images, cutting the foreground into image blocks of different resolutions; Step 2: Train the basic model: retrain the pre-trained basic model using image patches of different resolutions; Step 3: Use the trained model to extract image features at multiple resolutions, and use the single-cell morphology and topology atlas framework sc-MTOP to extract single-cell features; Step 4, using the features extracted in step 3 to perform interactive fusion between features, constructing a Gleason grading prediction model that integrates the image block level and the slice level, and using the Gleason grading prediction model to complete the automatic Gleason grading of the prostate.

2. The method according to claim 1, characterized in that In step 1, a data queue index is formulated to index and queue the panoramic slice image, the foreground and background segmentation mask, the image block and the corresponding mark.

3. The method according to claim 2, characterized in that In step 2, the pre-trained basic model is the universal pathology model UNI.

4. The method according to claim 3, characterized in that In step 2, two identical network structures are used in the training process, namely the teacher model and the student model. The model framework is the visual transformer model ViT-Large. The training process uses the training strategy of the universal pathology model UNI. First, two pictures of the same size are randomly cropped from the image block and two kinds of random data enhancement are performed respectively. The two pictures of the same size are used as global images, one of which is further cropped into more than two local images, and the global image is randomly partially masked. The masked global image and the local image are used as the input of the student model together; the other global image is used as the input of the teacher model.

5. The method according to claim 4, characterized in that In step 2, the same self-supervised training method as the UNI model is used, that is, the existing improved training method based on the second version of the enhanced denoising anchor point detection transformer DINOv2, and the loss functions used include the DINO loss function and the online token bidirectional encoding transformer iBOT loss function; In the DINO loss function, DINO heads are added to the student model and the teacher model respectively to calculate the loss function. For the student model, each output class token is obtained, and the student prototype score is output through a multi-layer perceptron MLP layer. Then, the student prototype score prototype score is normalized and the exponential function softmax is calculated to obtain the student score p s ; For the teacher model, obtain the class token output by the teacher model and get the teacher prototype score through the multi-layer perceptron MLP layer. Add the Sinkhorn-Knopp SK centralization process after the teacher prototype score, and then use the normalized exponential function softmax to calculate the teacher score p t , wherein the Sinkhorn-Knopp SK centralization process is a method for normalizing a matrix, wherein normalized rows and normalized columns are performed in one iteration, and after two or more iterations, the matrix will be approximately a double random matrix, and the specific process includes: Input: teacher prototype score Z, trainable clustering parameter θ C , number of iterations P; Initialization: The teacher prototype score Z0 = Z × C in the 0th iteration; The iteration steps include: For p, from 1 to P cycles: The temporary value Z1 of the p+1th iteration p+1 for in is a diagonal matrix with diagonal elements The reciprocal of the sum of each row; The Z value of the p+1th iteration is in is a diagonal matrix with diagonal elements The reciprocal of the sum of each column; Output the final teacher prototype score Z K .

6. The method according to claim 5, characterized in that In step 2, the normalized exponential function Softmax in the output head of the student model and the teacher model introduces a sharpening operation, dividing the input data by a temperature constant, which is set to the square root of the dimension; Finally, for the student score p s With the teacher score p t Use the cross entropy CE loss function to obtain the DINO loss function L DINO : IT DINO =CE(p t ,p s )=-∑p t logp s (1) During the training process, the student model performs normal back-propagation to update the model parameters, while the teacher model does not perform back-propagation. The parameters of the teacher model are updated with momentum using the parameters of the student model.

7. The method according to claim 6, characterized in that In step 2, in the iBOT loss function L iBOT In the above, we add headers to the student model and the teacher model to calculate the loss function. We randomly mask areas in the input image of the student model. In the student model, we use the iBOT header for the tokens corresponding to the masked areas, and in the teacher model, we use the iBOT header for the tokens at the same position. Then, we use the same loss function L as the DINO loss function. DINO The same normalized exponential function softmax calculation and Sinkhorn-Knopp SK centralization process, and the cross entropy CE loss function is used to obtain the iBOT loss function L iBOT : Among them, p ti is the teacher score for the ith covered region, p si is the student score of the i-th covered region; the parameters of the student model are trained and the parameters of the teacher model are adjusted through the momentum update mechanism.

8. The method according to claim 7, characterized in that In step 2, the model also uses KoLeo regularization during training to encourage the output features to be evenly distributed within a batch: given a set of network output features, the network output features are first regularized using the 2-norm, and then the feature x vector (x1, x2, …, x n ), define the KoLeo loss function L koleo : where d s =min t≠s ‖x s -x t ‖ is the sth feature x s The minimum distance to other features in the batch, n is the total number of features, x n represents the nth feature.

9. The method according to claim 8, characterized in that Step 3 includes: using the student model to extract features of all image blocks, and storing the results in a normalized manner, wherein the normalized storage includes image features, image block starting pixel coordinates, size, relative coordinates, panoramic slice image source, and magnification; The single-cell morphology and topology atlas framework sc-MTOP is used in the image blocks to extract cell features and construct the cell nucleus map: cell segmentation is performed on the image blocks, the cell nucleus map is constructed based on the spatial coordinate vector and the nearest neighbor KNN, and three types of manual features of single cells, texture, morphology and topology, are extracted; in the cell nucleus map, the manual features of each cell are spliced ​​with the corresponding spatial position code as the node feature.

10. The method according to claim 9, characterized in that Step 4 includes: based on multi-instance learning, constructing a Gleason grading prediction model that takes multi-scale and cell nucleus images as inputs and outputs slice-level and image-block-level outputs by slice branches and image-block branches; In terms of input, multi-resolution images and cell nucleus maps are used as input at the same time, where high-resolution images provide image details, low-resolution images provide a wider receptive field, and cell nucleus maps provide information on the distribution and arrangement of cells in the image; In terms of output, the slice-level output can obtain the patient-level diagnosis results, and the image block level can obtain the spatial distribution of each grade; The Gleason grading prediction model uses the improved attention multi-instance method Attention-MIL for multi-instance learning, that is, a multi-layer perceptron MLP model is added as an instance classifier on the basis of the original attention multi-instance Attention-MIL, acting as an image block branch, and the multi-layer perceptron output of the attention multi-instance Attention-MIL is used as a slice classifier, acting as a slice branch. Specifically, the Gleason grading prediction model needs to input image features at the image block level and cell nucleus map features at the cell level at two different resolutions at the same time. For the image features at the image block level, the multi-resolution image features extracted by the trained model in step 3 are directly used; for the cell level features, the graph neural network graph transformer TransformerGraph model is first used to update the node features of the cell nucleus map, and the deep features with contextual information are obtained from the manual features to contact the neighboring nodes, and then the cell level cell nucleus map features are obtained through a transformer with a class token class token; after the feature extraction is completed, the image features at the image block level of two different resolutions and the cell level cell nucleus map features are spliced ​​as the features of the high-resolution image, that is, the instance features; The improved attention multi-instance network is trained through two output branches, the instance classifier of the image block branch and the slice classifier of the slice branch, during the training process. There are M feature numbers before the network calculation. The network first calculates the feature number according to the 1×M dimensional real vector space. The kth instance feature of Calculate the kth instance attention coefficient a k , calculated as: The network generates L feature numbers during calculation, and the model divides the results into C categories. V and U are real vector spaces in L×M dimensions. The learnable parameters are W is a real vector space in L×C dimensions. The learnable parameters are Then the instance classifier and slice classifier are trained separately using the attention coefficient; For instance classifier: The training of instance classifier is divided into two cases: one with image block classification labels and one with slice classification labels only. Different classification heads are used for the two cases: When only slice classification is performed, the classification head output is in the 2×C real vector space. Rating That is, the binary classification score of the instance for each category, which is used to indicate whether the instance belongs to the category; when there is image block classification, the classification head output is in the 1×C real vector space Rating That is, the score of the instance output for each category; let the probability of classifying category i given by the instance classifier for instance k be y i,k ; When the sample only has a slice label, the priority method Top-K attention coefficient is used to extract the K highest and lowest instances and generate pseudo labels for instance classifier training: for the category label c of the input slice, the instance classifier model of the image block branch selects the top K instances with the highest attention coefficient and the top K instances with the lowest attention coefficient from the attention map of the corresponding category label c, and generates corresponding pseudo classification labels 1 and 0 respectively; for other categories, only the pseudo label 0 is generated for the K highest attention instances of other categories; finally, the instance and pseudo label pairs are concatenated and used for instance classifier training, and the training uses the cross entropy loss function L inst : Among them, Top(a i ) is the largest K image blocks in the attention of category i, Top(-a i ) is the smallest K image patches in the attention of category i, CE(1,y) is the cross entropy calculation between the prediction result k and the pseudo label 1; When the sample has an image block label, the instance classifier is trained directly using the image block level label: first, according to the doctor's segmentation label, the classification label is set for the image block at the corresponding position. For image block k, the number of pixels marked as a certain category in image block k is counted. When the number exceeds the threshold set by experience, the label of the image block is determined to be set as the category. Then, the attention coefficient is set for the image block according to the image block classification. For category c, it is set that there are image blocks in the slice. is the set of image blocks classified as c in the slice, then the c-class attention label value of image block k is Set as Then, the cross entropy CE loss function is used for image block classification, and the mean square error MSE loss function is used for attention coefficient training: in is a set of labeled instances, is the set of instances with label 0, is a set of instances with label C, MSE represents the mean square error function, Prediction result y for the image block k With mark Cross entropy calculation of ; For the slice classifier: the native Attention-MIL performs weighted addition of the features of all instances through the attention coefficient to obtain slice-level features, obtains slice classification through the fully connected layer FC, and then uses the cross entropy loss function CE for training, the instance feature aggregation formula and the training loss function L slide for: in is the slice-level label, a k is the kth instance attention coefficient, h k is the kth instance feature, the fully connected layer FC is a linear mapping; y represents the output of the fully connected layer FC; Finally, through two loss functions L slide and l inst The instance classifier and the slice classifier are trained simultaneously by superposition; In the testing phase, the classification head outputs the category of each image block to obtain the spatial prediction of each level, and the final slice-level prediction is obtained by combining the slice classifier output and the instance classifier at the image block level.

Images