A method, system and storage medium for predicting the efficacy of ustekinumab based on pathological images

Abstract

The present invention belongs to the field of computer vision technology, and in particular, relates to a method, system and storage medium for predicting the efficacy of ustekinumab based on pathological images, comprising the following steps: obtaining a full-slice biopsy pathological image of a Crohn's disease patient to be tested before medication; cropping the full-slice biopsy pathological image into image blocks; and obtaining the response results of Crohn's disease patients after using ustekinumab based on the cropped image blocks using a pre-trained ustekinumab efficacy prediction model. This can further improve the ability to predict the efficacy of ustekinumab on Crohn's disease, guide clinical medication selection, promote disease remission in patients, and reduce related medical and health costs.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method, system and storage medium for predicting the efficacy of ustekinumab based on pathological images. Background Art

[0002] Crohn's disease is a chronic, relapsing inflammatory bowel disease. It can occur anywhere from the mouth to the anus, with lesions that jump and can affect all layers of the intestinal wall, especially the terminal ileum and colon. The disease can occur at any age, and more than 80% of Crohn's patients are diagnosed before the age of 40. The treatment of Crohn's disease is mainly based on a combination of nutrition and drug therapy. In some cases, when drug therapy cannot effectively control the disease or serious complications occur, surgical treatment is used as an auxiliary.

[0003] Ustekinumab is a new type of biological agent that has appeared in recent years. It is a humanized monoclonal antibody used to treat Crohn's disease. Ustekinumab is widely used in the treatment of inflammatory bowel disease in clinical practice and has been proven to have good efficacy and safety in alleviating symptoms, improving intestinal inflammation and improving quality of life. The efficacy of ustekinumab in the treatment of Crohn's disease varies from person to person. Therefore, it is necessary to predict and analyze the response of ustekinumab to patients to provide guidance for individualized precision treatment.

[0004] The widespread application of artificial intelligence technology in the biomedical field is having a huge impact on the field. In particular, progress in the field of tumors in histopathology is the most obvious. The application of artificial intelligence methods in clinical practice and clinical trials still faces many challenges. These include the acquisition of relatively large and well-labeled training data sets, uneven image quality and specimen collection methods that require extensive data standardization and normalization, hardware limitations encountered when processing large high-resolution images, and ethical and privacy issues caused by interaction with human subjects. The application of artificial intelligence in gastroenterology is growing, most of which use endoscopic data. Remarkable success has been achieved in the detection of intestinal polyps and gastrointestinal malignancies. Therefore, it provides technical feasibility in predicting drug efficacy. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method, system and storage medium for predicting the efficacy of ustekinumab based on pathological images, which can further improve the ability to predict the efficacy of ustekinumab for Crohn's disease, guide clinical drug selection, promote disease remission in patients, and reduce related medical and health costs.

[0006] The technical solution provided by the present invention is as follows:

[0007] The present invention provides a method for predicting the efficacy of ustekinumab based on pathological images, comprising the following steps: Obtain the full biopsy slice pathological images of the Crohn's disease patients to be tested before medication; Crop the biopsy full-slice pathology image into image blocks; Based on the cropped image blocks, the pre-trained ustekinumab efficacy prediction model was used to obtain the response results of Crohn's disease patients after using ustekinumab; The process of processing the image block by the ustekinumab efficacy prediction model includes: Using the Resnet50 network model and the K-means clustering algorithm, valid image blocks are screened out from all image blocks; wherein, valid image blocks are image blocks that are effective in responding to ustekinumab; The Densenet121 network model is used to predict the probability of valid image blocks, and the predicted probabilities are aggregated to obtain the feature set of the biopsy full-slice pathology image; A machine learning method based on support vector product was used to classify the feature set of biopsy whole-slice pathological images to obtain the response results of Crohn's disease patients after using ustekinumab.

[0008] Furthermore, the biopsy full-slice pathology image is cropped into image blocks, including: The foreground of the biopsy whole-slice pathology image was extracted by the maximum intra-variance method; At the maximum magnification, the full biopsy slice pathology image after the panoramic view is taken out is cropped into image blocks of a preset size; Before using the ustekinumab efficacy prediction model to process the cropped image blocks, the steps of performing preliminary screening and standardization processing on the image blocks in sequence are also included, and the steps include: Select image blocks where the tissue area exceeds a threshold; Vahadane is used to perform color normalization on the filtered image blocks.

[0009] Furthermore, the Resnet50 network model and K-means clustering algorithm are used to filter out valid image blocks from all image blocks, including: Use the Resnet50 network model to extract the deep features of image blocks; Clustering image patches based on deep features; Based on the clustering results, image patches in specific clusters were selected for the ustekinumab efficacy prediction model.

[0010] Furthermore, in the ustekinumab efficacy prediction model, before screening out effective image blocks, a step of expanding the image blocks is also included, which step includes: Z-score normalization is used on the RGB channel to obtain image patches whose intensities conform to the standard normal distribution, and data enhancement is used to expand the image patches.

[0011] Furthermore, when training the ustekinumab efficacy prediction model, when the number of training times reaches The cosine decay learning rate algorithm is performed on the backbone learning rate. The formula of the learning rate is as follows: ; in, .

[0012] Furthermore, when training the ustekinumab efficacy prediction model, the Densenet121 network model was used to predict the probability of the screened image blocks that were most effective in responding to the drug. The optimizer used stochastic gradient descent SGD, the loss function used softmax cross entropy, and the batch size was 256.

[0013] Furthermore, the predicted probabilities are aggregated to obtain the feature set of the biopsy full-slice pathology image, including: The probability hierarchy histogram PLH is used to calculate the histogram features of probability; Use the Bag of Words model BoW algorithm to extract the word frequency-inverse document frequency TF-IDF feature of the probability to obtain the TF-IDF feature; The histogram features and TF-IDF features are concatenated, and the concatenated features are selected to obtain a feature set; A machine learning method based on support vector product was used to classify the feature set of biopsy whole-slice pathological images to obtain the response results of Crohn's disease patients after using ustekinumab.

[0014] Furthermore, the selection of the concatenated features includes: Filter out the feature subset that meets the Pearson correlation coefficient; The feature subset after preliminary screening is reselected using the lasso model to obtain the feature set.

[0015] The present invention also provides a prediction system for the efficacy of ustekinumab based on pathological images, comprising a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps according to the above method.

[0016] The present invention also provides a storage medium on which a computer program is stored, and the computer program implements the steps of the above method when executed by a processor.

[0017] Beneficial Effects

[0018] Crohn's disease is a chronic disease that is still incurable. Biological agents such as ustekinumab are currently the main therapeutic drugs, but the efficacy of different biological agents for individual patients varies. In the past, relevant predictions were mostly established based on laboratory tests and imaging data, and the image data information of biopsy pathology was rarely used. However, the biopsy pathological manifestations can just reflect the characteristics and severity of the disease from a microscopic perspective. Therefore, the present invention provides a prediction method and system and storage medium for predicting the efficacy of ustekinumab based on biopsy pathological images of Crohn's patients before medication. This will further improve the predictive ability of ustekinumab for Crohn's disease, guide clinical drug selection, promote disease remission in patients, and reduce related medical and health costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a method for predicting the efficacy of ustekinumab based on pathological images of the present invention;

[0020] Figure 2 A schematic diagram of the results of image block clustering and image block selection of the present invention;

[0021] Figure 3 The prediction probability map and prediction heat map of the image block level of the present invention;

[0022] Figure 4 A schematic diagram of the last convolution result of the image block predicted in the deep learning model of the present invention;

[0023] Figure 5 It is a schematic diagram of the prediction evaluation results of the prediction model for the efficacy of ustekinumab at the whole-slice level of the present invention. DETAILED DESCRIPTION

[0024] Example 1

[0025] The present invention provides a model for predicting the efficacy of ustekinumab, and the process of constructing the model includes the following steps: Step 1: Data collection: collect biopsy pathological tissues of Crohn's disease patients before medication and digitize them using a digital scanner for subsequent image data analysis; Step 2: Perform data preprocessing on the digital pathology images, mainly to control the image content and quality, and divide the tissue areas in the digital pathology images into image blocks to form the training and testing data sets of the model; Step 3: Screen the image blocks and use an integrated prediction model based on clustering of image blocks; Step 4: training the screened image blocks to construct an image block-level ustenuzumab efficacy prediction model; Step 5: Use multi-instance fusion to fuse the prediction results at the image block level to construct pathological features, and then use feature selection methods to select features related to the prediction results to construct a patient-level ustenumab efficacy prediction model.

[0026] Among them, the data collection in step 1 includes:

[0027] First, patients with Crohn's disease were collected and those treated with ustenumab were screened, and then endoscopic examination was performed to determine whether the patients were effective in the ustenumab treatment. Secondly, tissue samples before medication were selected and made into slices, and finally the slices were digitized using a digital scanner.

[0028] Step 2: Data preprocessing of the digitized pathological image is mainly divided into image content, quality control and color standardization of image blocks. Since the size of the whole-slice digital pathological image (WSI) is very large, its storage level usually reaches the GB level, so it is very difficult to process it directly. The present invention adopts the proposed image preprocessing. The present invention takes out the foreground of the image by the maximum intravariant variance method, and preprocesses the whole-slice image WSI (Whole Slide Imaging) into an image block of 256×256 pixel size at the maximum magnification of WSI, and selects images with more than 50% of the tissue area in each image. Therefore, after processing all WSIs, the present invention forms a queue of millions of image blocks. The present invention uses HE-stained pathological tissue images. Since the image itself has problems such as uneven staining and fading, this article uses the Vahadane color standardization method for processing. Compared with the Reinhard and Macenko methods, the Vahadane method has the best standardization result, but the processed image is longer.

[0029] Step 3: In order to select valid image blocks from the image blocks generated by WSI, the present invention clusters the image blocks according to their phenotypes and distinguishes more discriminative image blocks. First, N samples of Crohn's disease are randomly selected from the training cohort, including WSIs of response samples and non-response samples, thereby obtaining n image blocks, and then the k-means algorithm is used for training to cluster the n image blocks to generate k clusters. Considering that the original image may have a drug response phenotype that is not related to the type, the present invention chooses to cluster the image blocks in the feature domain. The Resnet50 model trained based on the Imagenet dataset is used as a feature extractor by transfer learning, and all image blocks are converted into 2048-dimensional structured data through the average pooling layer of the pre-trained Resnet50 model. The present invention divides these image blocks into k clusters to develop a k-means clustering algorithm, in which the Calinski-Harabasz index is used to evaluate the clustering quality, and the image blocks in different clusters are considered to have discriminative imaging patterns related to drug response.

[0030] The main purpose of screening image blocks is to select the image blocks that are most effective in responding to drugs. The present invention uses the established k-means clustering algorithm to divide all image blocks of each WSI in the training cohort into k clusters. Next, the image block clustering results of all patients in the training cohort are trained separately for k independent patch-level CNN classifiers, where ResNet-50 is used as the CNN architecture, and the training parameters are the same as the full patch classifier. When training the patch-level CNN classifier, we use Z-score normalization on the RGB channel to obtain the standard normal distribution of image intensity as the input of the model. Data enhancement functions such as horizontal and vertical random flipping are used in model training. The k clusters obtained in the validation cohort are used to optimize the k corresponding classifiers. Classifiers based on k clusters may have different capabilities in classifying whether drugs respond. Here, the present invention uses the classification results of all image blocks as performance benchmarks. For each WSI, a cluster with a classification accuracy higher than the benchmark is selected for further analysis. After the dimensionality reduction by principal component analysis PCA (Principal Component Analysis), it can be well distinguished in two-dimensional space.

[0031] Step 4: After the image blocks are screened, the present invention selects the image blocks in the cluster with higher performance than the benchmark from all samples for constructing the response model of Ustekinumab to Crohn's disease. Among them, for the labels of all image blocks, this paper uses the WSI-level labels as the labels of the image blocks for constructing the patch-level response model of Ustekinumab to Crohn's disease.

[0032] For image block level prediction, the present invention evaluates multiple deep learning neural networks, such as mobilenet_v2, Resnet18, Resnet50, Densenet121, and Vison Transformer (ViT). These are currently effective network structures for image processing. Our goal is to establish the probability that each patch corresponds to the label of the respective WSI to which it belongs. The models have similar prediction performance at the patch level, so the present invention uses the prediction results of the Densenet121 model.

[0033] To improve the generalization of the model across heterogeneous cohorts, we implemented transfer learning. This entailed initializing the model’s parameters using pre-trained weights from the ImageNet dataset, while retaining the weights of the patch-level classifier. Afterwards, we fine-tuned the entire model using a limited dataset that had been particularly weakly annotated for the task. Through transfer learning, we successfully leveraged the knowledge gained from ImageNet and tailored it to fit the requirements of the classification problem.

[0034] In order to improve generalization, the cosine decay learning rate algorithm is used in the present invention to carefully set the learning rate. The learning rate is expressed as: ; in, , , Respectively represent the minimum learning rate, maximum learning rate and number of model iterations, , the learning rate of the current task at time t.

[0035] The huge dataset of the present invention contains more than one million training image blocks, so the model can achieve relatively good results in neural networks with different structures. In addition, the transfer learning algorithm is used to ensure the optimal model fit. By loading the pre-trained model to initialize the parameters of Backbone, the parameters of the current task can be effectively fine-tuned. When the number of training times reaches The learning rate of the backbone component is defined as follows: ; in, , the configuration of other hyperparameters is as follows: the optimizer uses SGD, the loss function uses softmax cross entropy, and the batch size is 256.

[0036] Step 5: The image patch-level prediction model predicts labels and probabilities for all patches. A classifier is then used to aggregate these patch probabilities to produce WSI-level predictions. To collect patch likelihoods, two different machine learning methods are used: Probability Level Histogram PLH (Probability Level Histogram): In this method, a histogram is used to represent the distribution of patch likelihoods within the WSI. By discretizing the likelihood, the distribution of likelihoods is effectively captured as a representation of the WSI. Bag of Words BoW (Words Bag) BoW: The BoW pipeline is built on histogram-based and vocabulary-based techniques, using a term frequency-inverse document frequency TF-IDF (Term Frequency-Inverse Document Frequency) mapping for each patch to produce a TF-IDF feature vector representing the WSI. These feature vectors are then used to train traditional machine learning classifiers to predict the state in each WSI.

[0037] By deploying two independent pipelines, we successfully merge the initially scattered patch-level predictions to generate WSI-level features. These features provide important information for subsequent analysis operations.

[0038] In the feature concatenation built based on the two pipelines of PLH and BoW, due to the complex distribution of features, in order to control the complexity of the model, improve the generalization ability of the model, and enhance the stability of the model, this article will use the z-score regularization method for the features, and use regularization to change the data to a mean of 0 and a variance of 1. The regularization formula is as follows: ; in is the value after regularization, is the original value, The mean of a feature in a certain dimension, The standard deviation of a feature.

[0039] Since the features of the present invention have a higher dimension, the Pearson correlation coefficient is used for feature selection. After using the Pearson correlation coefficient to calculate the correlation between the regularized features and the prediction target of this article, the features with a correlation coefficient greater than 0.9 will be selected. After feature selection based on the Pearson correlation coefficient, the present invention uses a cross-validation method to train the lasso model, and uses the mean square error to evaluate the model performance of the lasso model. The penalty coefficient of cross-validation is used as the basis for model training. Afterwards, the selected features are screened again based on the construction of the lasso model. Thereby forming a feature data set of all samples.

[0040] The present invention uses an SVM classifier to construct a response prediction model of Ustekinumab for Crohn's disease patients. It is found that different machine learning algorithms have certain differences in judging whether there is a response. LR, SVM, and MLP classifiers have better results, while KNN, RandomForest, and ExtraTrees classifier models have poor effects. Among them, the comprehensive comparison of SVM performance results is the best. Logistic regression LR (Logistic Regression), support vector machine SVM (Support Vector Machine) and multi-layer perceptron MLP (Multi-Layer Perceptron) are traditional supervised learning classifiers, while K-nearest neighbors KNN (K-Nearest Neighbors), random forest Random Forest and extreme random trees Extra Trees are classifiers in ensemble learning.

[0041] LR: LR is a linear classifier suitable for handling binary classification problems. The prediction result is obtained by multiplying the features with weights and passing through an activation function (such as sigmoid). SVM: SVM is a classifier based on margin maximization, which can handle linear and nonlinear classification problems. By mapping the data into a high-dimensional space, the optimal hyperplane is found to separate samples of different categories. MLP: MLP is an artificial neural network with multiple hidden layers that can handle complex nonlinear relationships. The network is trained through forward propagation and backpropagation to achieve classification tasks. KNN: KNN is an instance-based learning method that classifies by calculating the distance between the sample to be classified and the sample in the training set. KNN has no explicit training process and is a lazy learning method. Random Forest: Random Forest is an ensemble learning method consisting of multiple decision trees that are classified through a voting mechanism. Each tree is trained on a different random subset to improve generalization ability. Extra Trees: Extra Trees is a variant of Random Forest, which is similar to Random Forest, but uses more randomness when splitting nodes to enhance the diversity of the model.

[0042] Example 2

[0043] The embodiment of the present invention provides a method for predicting the efficacy of ustekinumab based on pathological images, comprising the following steps: Obtain the full biopsy slice pathological images of the Crohn's disease patients to be tested before medication; Crop the biopsy full-slice pathology image into image blocks; Based on the cropped image blocks, the pre-trained ustekinumab efficacy prediction model was used to obtain the response results of Crohn's disease patients after using ustekinumab; The process of processing the image block by the ustekinumab efficacy prediction model includes: Using the Resnet50 network model and the K-means clustering algorithm, valid image blocks are screened out from all image blocks; wherein, valid image blocks are image blocks that are effective in responding to ustekinumab; The Densenet121 network model is used to predict the probability of valid image blocks, and the predicted probabilities are aggregated to obtain the feature set of the biopsy full-slice pathology image; A machine learning method based on support vector product was used to classify the feature set of biopsy whole-slice pathological images to obtain the response results of Crohn's disease patients after using ustekinumab.

[0044] In this embodiment, the biopsy full-slice pathology image is cropped into image blocks, including: The foreground of the biopsy whole-slice pathology image was extracted by the maximum intra-variance method; At the maximum magnification, the full biopsy slice pathology image after the panoramic view is taken out is cropped into image blocks of a preset size; Before using the ustekinumab efficacy prediction model to process the cropped image blocks, the steps of performing preliminary screening and standardization processing on the image blocks in sequence are also included, and the steps include: Select image blocks where the tissue area exceeds a threshold; Vahadane is used to perform color normalization on the filtered image blocks.

[0045] In this embodiment, the Resnet50 network model and the K-means clustering algorithm are used to filter out valid image blocks from all image blocks, including: Use the Resnet50 network model to extract the deep features of image blocks; Clustering image patches based on deep features; Based on the clustering results, image patches in specific clusters were selected for the ustekinumab efficacy prediction model.

[0046] In this embodiment, the ustekinumab efficacy prediction model further includes a step of expanding the image blocks before screening out the effective image blocks, and the step includes: Z-score normalization is used on the RGB channel to obtain image patches whose intensities conform to the standard normal distribution, and data enhancement is used to expand the image patches.

[0047] In this embodiment, when training the ustekinumab efficacy prediction model, when the number of training times reaches The cosine decay learning rate algorithm is performed on the backbone learning rate. The formula of the learning rate is as follows: ; in, .

[0048] In this embodiment, when training the ustekinumab efficacy prediction model, the Densenet121 network model is used to predict the probability of the screened image blocks that are most effective in responding to the drug. The optimizer uses stochastic gradient descent SGD, the loss function uses softmax cross entropy, and the batch size is 256.

[0049] In this embodiment, the predicted probabilities are aggregated to obtain a feature set of the biopsy full-slice pathological image, including: The probability hierarchy histogram PLH is used to calculate the histogram features of probability; Use the Bag of Words model BoW algorithm to extract the word frequency-inverse document frequency TF-IDF feature of the probability to obtain the TF-IDF feature; The histogram features and TF-IDF features are concatenated, and the concatenated features are selected to obtain a feature set; A machine learning method based on support vector product was used to classify the feature set of biopsy whole-slice pathological images to obtain the response results of Crohn's disease patients after using ustekinumab.

[0050] In this embodiment, selecting the spliced ​​features includes: Filter out the feature subset that meets the Pearson correlation coefficient; The feature subset after preliminary screening is reselected using the lasso model to obtain the feature set.

[0051] Example 3

[0052] The embodiment of the present invention will specifically illustrate the construction process of the ustekinumab efficacy prediction model and the ustekinumab efficacy prediction method based on pathological images, including the following steps:

[0053] Step 1: The data set used in the present invention has a total of 402 tissue samples, and each tissue sample has a tissue slice. After the tissue slices are digitized, the WSI is preprocessed into image blocks of 256×256 pixels at the maximum magnification. Among them, 236 data have treatment response, and 166 data have no treatment response.

[0054] All WSIs were digitized by a Leica GT450 scanner at a pixel resolution of 263 nm / pixel.

[0055] In the process of predicting whether a drug responds, the present invention randomly divides the data set into two groups (train:test=8:2), wherein the training set has 321 tissue samples and the test set has 81 tissue samples.

[0056] Step 2: After extracting the foreground area of ​​the WSI image, the present invention processes the foreground area into 256×256 pixel size image blocks, and selects image blocks with more than 50% of the tissue area in each WSI image. Therefore, after processing all WSI images, there are more than 2.6 million images in the training set and more than 590,000 images in the test set.

[0057] Step 3: Preprocess all image blocks. First, use Vahadane to perform color standardization. During the model input process, use Z-score normalization on the RGB channel to obtain the standard normal distribution of image intensity. During the model training process, randomly crop, flip, rotate and other data enhancement methods are performed on the image blocks to expand the data set training model.

[0058] In this embodiment, the image after the preprocessing operation is normalized to adjust the image size to 256×256; the image is randomly cropped to adjust the image size to 224×224. The specific process is as follows Figure 1 (a) shown.

[0059] Step 4: In order to select effective image patches from the image patches generated by WSI, this embodiment clusters the image patches according to their phenotypes and distinguishes more discriminative image patches.

[0060] The main process of this method is as follows Figure 1 As shown in (b), first, 50 WSI samples are randomly selected from the training set, of which 29 are WSI samples with responses and 21 are WSI samples with no responses. The 329,045 image patches obtained are used to train the k-means algorithm.

[0061] Considering that the original image may have drug response phenotypes that are not related to the type, the present invention chooses to cluster the image blocks in the feature domain. Through transfer learning, the Resnet50 model trained based on the Imagenet dataset is used as a feature extractor, and all image blocks are converted into 2048-dimensional structured data through the average pooling layer of the pre-trained Resnet50 model.

[0062] Based on the deep learning features of 329045 image blocks, this embodiment divides these image blocks into K clusters to develop a K-means clustering algorithm, in which the Calinski-Harabasz index is used to evaluate the clustering quality. Image blocks in different clusters are considered to have discriminative imaging patterns related to drug response, and finally K corresponding to the maximum value of the silhouette coefficient is selected as 3. Figure 2 (a) shows the evaluation results of k-means clustering, Figure 2 (b) Visualization results showing clustering into three clusters.

[0063] Step 5: First, use the k-means optimal clustering model to divide all image blocks, and the data set is divided into 3 clusters. Then, use the CNN model to build an image block level classifier, so that 3 independent image block level classifiers are obtained accordingly.

[0064] Based on the CNN model to train the image block level classifier, the benchmark accuracy of the full block classifier of this embodiment reaches 0.72, and the accuracies of the three clusters cluster1, cluster2, and cluster3 are 0.68, 0.76, and 0.79 respectively. Therefore, the embodiment of the present invention will select images in cluster2 and cluster3 to build the response model.

[0065] The accuracy of image block level classification for whether there is a response is as follows: Figure 2 (c) shows that the accuracy of the classifiers trained on cluster2 and cluster3 is higher than that of the benchmark classifier (such as Figure 2 c). Therefore, the selected training set contains nearly 2 million image patches to construct the ustekinumab response model. The clustering results of 3 representative patients are shown in Figure 2 (d) shown.

[0066] Step 6: Predict the response at the image block level. This example uses the image block level ROC curve to compare and characterize the model and evaluate the accuracy of the pathology model in region recognition. This example uses mobilenet_v2, Resnet18, Resnet50, Densenet121, and Vison Transformer (ViT). The model performance is evaluated at the same time.

[0067] The Densenet121 model used in this example has an image block-level AUC of 0.866 (95% CI: 0.865-0.867) and an accuracy of 0.774 for diagnosing whether Crohn's disease responds to ustekinumab in the test set.

[0068] To further visualize and explain the performance of the deep learning model in diagnosing whether Crohn's disease responds to ustekinumab at the image patch level, Figure 3 As shown, the predicted labels and probability heat maps are displayed to assist in the evaluation. The visual result diagram shows that the pathology prediction model of the present invention achieves a high accuracy in evaluating image blocks.

[0069] To further explain the model’s predictions, we use Grad-CAM to visualize the gradients of the last convolutional layer of the network to generate a graph showing class localization. This preserves class-specific spatial information. Notably, it does not require any modifications to the model architecture or additional training.

[0070] Figure 4 Illustrated using Grad-CAM in displaying the activations of the last convolutional layer for diagnostic class evaluation. This depiction highlights regions of the input image that contribute significantly to the prediction, thus providing valuable insights into the model’s decision-making process.

[0071] Step 7: The features constructed based on the two pipelines of PLH and BoW are concatenated into 202 dimensions. Therefore, too high a dimension is not conducive to the training of machine learning models. This study first uses the z-score method to regularize the feature set, then uses the Pearson correlation coefficient method to calculate the correlation coefficient of each dimension of the feature, and then selects the features with a correlation coefficient greater than 0.9, obtaining a total of 45 dimensions of features.

[0072] In addition, the selected 45-dimensional features are used for feature selection again using the lasso model. The cross-validation method is used to select the weights of the features under different lambdas, and the performance of the lasso model is evaluated at the same time.

[0073] Then, the features with feature weight of 0 are removed, and the selected features are BoW_10, BoW_099, BoW_075, BoW_071, BoW_063, BoW_061, hist-0.37, hist-0.34, hist-0.35, hist-0.31, hist-0.36, hist-0.4, and BoW_001.

[0074] Based on the selected features, six types of machine learning models were verified and trained using a five-fold cross-validation method in the training set, namely LR, SVM, KNN, RandomForest, ExtraTrees and MLP.

[0075] According to the results of the five-fold cross-validation, we found that not all machine learning models are suitable for the current task. Among them, LR, SVM and MLP have better performance. In the five-fold cross-validation, all evaluation indicators of the three models of LR, SVM and MLP are relatively high in the training set, and the accuracy rate reaches 0.997. In the test set, the accuracy rates of the three models reach 0.877, 0.889 and 0.852.

[0076] The three models of KNN, RandomForest and ExtraTrees are not suitable for the current task, have poor sensitivity, and cannot predict the response samples. Their ROC curves and AUC values ​​are as follows: Figure 5 (a) shown.

[0077] According to all the performance evaluation indicators of the test set, the SVM model achieved the best performance in the current task, with an Acc of 0.889, an AUC of 0.938 (95% CI: 0.879-0.996), a sensitivity of 0.951, a specificity of 0.825, and an F1 value of 0.897.

[0078] The performance of the LR model is close to that of the SVM model, and its specificity is slightly worse than that of the SVM model. Compared with the performance of the SVM model, although the AUC value of the MLP model is higher than that of the SVM model, its sensitivity is 0.756, which is lower than that of the SVM, and its accuracy is also lower than that of the SVM. DCA is also an important indicator for evaluating model performance. The present invention further evaluates whether the model predicts whether the sample has a response in the test set by using the decision curve (DCA), such as Figure 5 As shown in (b), it can be seen from the DCA curve that the models performed well and showed good clinical application value.

[0079] The probability bar graph of whether Crohn's disease responds to ustekinumab is predicted by the SVM model for all test set samples. Figure 5 As shown in (c), 7 non-response samples were predicted to be responses, and 1 response sample was predicted to be non-response. Figure 5 (d) The confusion matrix is ​​used to visualize the prediction results of the test samples.

[0080] Example 4

[0081] An embodiment of the present invention provides a prediction system for the efficacy of ustekinumab based on pathological images, characterized in that it includes a processor and a storage medium;

[0082] The storage medium is used to store instructions;

[0083] The processor is used to operate according to the instructions to execute the steps of the method described in implementation case 2.

[0084] Example 5

[0085] An embodiment of the present invention provides a storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method described in Embodiment 2 when executed by a processor.

Claims

1. A method for predicting the efficacy of ustekinumab based on pathological images, characterized in that: The following steps are involved: Obtain the full biopsy slice pathological images of the Crohn's disease patients to be tested before medication; Crop the biopsy full-slice pathology image into image blocks; Based on the cropped image blocks, the pre-trained ustekinumab efficacy prediction model was used to obtain the response results of Crohn's disease patients after using ustekinumab; The process of processing the image block by the ustekinumab efficacy prediction model includes: Using the Resnet50 network model and the K-means clustering algorithm, valid image blocks are screened out from all image blocks; wherein, valid image blocks are image blocks that are effective in responding to ustekinumab; The Densenet121 network model is used to predict the probability of valid image blocks, and the predicted probabilities are aggregated to obtain the feature set of the biopsy full-slice pathology image; A machine learning method based on support vector product was used to classify the feature set of biopsy whole-slice pathological images to obtain the response results of Crohn's disease patients after using ustekinumab.

2. The method for predicting the efficacy of ustekinumab based on pathological images according to claim 1, characterized in that: Crop the biopsy full-slice pathology image into image blocks, including: The foreground of the biopsy whole-slice pathology image was extracted by the maximum intra-variance method; At the maximum magnification, the full biopsy slice pathology image after the panoramic view is taken out is cropped into image blocks of a preset size; Before using the ustekinumab efficacy prediction model to process the cropped image blocks, the steps of performing preliminary screening and standardization processing on the image blocks in sequence are also included, and the steps include: Select image blocks where the tissue area exceeds a threshold; Vahadane is used to perform color normalization on the filtered image blocks.

3. The method for predicting the efficacy of ustekinumab based on pathological images according to claim 1, characterized in that: Using the Resnet50 network model and K-means clustering algorithm, valid image blocks are screened out from all image blocks, including: Use the Resnet50 network model to extract the deep features of image blocks; Clustering image patches based on deep features; Based on the clustering results, image patches in specific clusters were selected for the ustekinumab efficacy prediction model.

4. The method for predicting the efficacy of ustekinumab based on pathological images according to claim 3, characterized in that: In the ustekinumab efficacy prediction model, before screening out effective image blocks, there is also a step of expanding the image blocks, which includes: Z-score normalization is used on the RGB channel to obtain image patches whose intensities conform to the standard normal distribution, and data enhancement is used to expand the image patches.

5. The method for predicting the efficacy of ustekinumab based on pathological images according to claim 1, characterized in that: When training the ustekinumab efficacy prediction model, when the number of training times reaches The cosine decay learning rate algorithm is performed on the backbone learning rate. The formula of the learning rate is as follows: ； in, .

6. The method for predicting the efficacy of ustekinumab based on pathological images according to claim 1, characterized in that: When training the ustekinumab efficacy prediction model, the Densenet121 network model was used to predict the probability of the screened image blocks that were most effective in responding to the drug. The optimizer used stochastic gradient descent SGD, the loss function used softmax cross entropy, and the batch size was 256.

7. The method for predicting the efficacy of ustekinumab based on pathological images according to claim 1, characterized in that: The predicted probabilities are aggregated to obtain the feature set of the biopsy full-slice pathology image, including: The probability hierarchy histogram PLH is used to calculate the histogram features of probability; Use the Bag of Words model BoW algorithm to extract the word frequency-inverse document frequency TF-IDF feature of the probability to obtain the TF-IDF feature; The histogram features and TF-IDF features are concatenated, and the concatenated features are selected to obtain a feature set; A machine learning method based on support vector product was used to classify the feature set of biopsy whole-slice pathological images to obtain the response results of Crohn's disease patients after using ustekinumab.

8. The method for predicting the efficacy of ustekinumab based on pathological images according to claim 7, characterized in that: The selection of the concatenated features includes: Filter out the feature subset that meets the Pearson correlation coefficient; The feature subset after preliminary screening is reselected using the lasso model to obtain the feature set.

9. A prediction system for the efficacy of ustekinumab based on pathological images, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images