Immunotherapy prediction model training method, calling method, device and equipment

Abstract

The invention discloses an immunotherapy prediction model training method, calling method, device and equipment, and the method comprises the steps: obtaining a clinical pathology image and clinical pathology information, the clinical pathology image is a pathology image of an immunotherapy patient before treatment, and the clinical pathology information is a pathology image of the immunotherapy patient before treatment; the clinical pathological information comprises pre-treatment clinical information and post-treatment prognosis information of the immunotherapy patient; marking the clinical pathology image to obtain a marked image, and performing feature extraction on the marked image based on the clinical pathology information to obtain pathology omics features; and carrying out model training by adopting the pathological omics characteristics to obtain an immunotherapy prediction model. The image features are accurately extracted through the pathological information, then the image features are used for model training, the precision of the model can be improved, and the error of the model is reduced.

Description

Background Art

[0002] Gastric cancer is a common malignancy and the third leading cause of cancer-related death worldwide. Immune checkpoint inhibitors targeting PD-(L)1 have become the first choice for gastric cancer patients who lack targeted treatment options. However, due to the complexity of anti-tumor immune regulation, these molecular markers cannot accurately identify patients who benefit from anti-PD-(L)1 immune checkpoint inhibitors. Therefore, it is necessary to test the therapeutic efficacy of anti-PD-(L)1 immune checkpoint inhibitors.

[0003] In order to simplify the processing flow of efficacy testing, artificial intelligence technology has been introduced in this field. The commonly used method is to collect the patient's HE full-field pathological slice image (WSI), use the collected images for model training, and obtain an image recognition model. The image recognition model is then used to identify the patient's treatment area, allowing professional and experienced pathologists to view the treatment area and make a diagnosis and differential diagnosis to determine the efficacy of the treatment.

[0004] However, the commonly used methods currently have the following technical problems: the color, shape and contour of normal tissue and pathological tissue in HE full-field pathology slice images are similar. The image recognition model obtained by only using HE full-field pathology slice images for model training has low recognition accuracy and large errors, which is difficult to meet existing application needs. Summary of the invention

[0005] The present invention proposes a training method, calling method, device and equipment for an immunotherapy prediction model, which can solve one or more of the above-mentioned technical problems.

[0006] A first aspect of an embodiment of the present invention provides a method for training an immunotherapy prediction model, the method comprising:

[0007] Acquiring clinical pathology images and clinical pathology information, wherein the clinical pathology images are pathology images of immunotherapy patients before treatment, and the clinical pathology information includes clinical information of immunotherapy patients before treatment and prognosis information after treatment;

[0008] After labeling the clinical pathology image to obtain a labeled image, feature extraction is performed on the labeled image based on the clinical pathology information to obtain a pathology omics feature;

[0009] The pathological omics features are used to perform model training to obtain an immunotherapy prediction model.

[0010] In a possible implementation manner of the first aspect, extracting features from the labeled image based on the clinical pathology information to obtain pathological omics features includes:

[0011] A preset feature extraction model is called to extract pathological genomic features in the marked area of ​​the marked image according to the clinical pathological information, wherein the pathological genomic features include pathological cell nucleus features, pathological microenvironment features and pathological spatial distribution features.

[0012] In a possible implementation of the first aspect, the using the pathological genomics features to perform model training to obtain an immunotherapy prediction model includes:

[0013] After constructing a plurality of base models, each of the base models is trained and optimized using the pathological omics features to obtain a plurality of training models;

[0014] After integrating several of the training models using the voting method of ensemble learning, each of the training models is fitted using the pathological genomics features to obtain an immunotherapy prediction model.

[0015] In a possible implementation manner of the first aspect, marking the clinical pathology image to obtain a marked image includes:

[0016] Acquiring a marked area, wherein the marked area is an area where the medical staff outlines the tumor site in the image after showing the clinical pathology image to the medical staff;

[0017] The mark of the marked area is added to the clinical pathology image to obtain a marked image.

[0018] In a possible implementation manner of the first aspect, acquiring a clinical pathology image includes:

[0019] Acquiring a full-field pathological image of the patient, wherein the full-field pathological image is an image of a HE-stained section of the immunotherapy patient before receiving an anti-PD-(L)1 immune checkpoint inhibitor;

[0020] After digital scanning and screening of the full-field pathological image, a clinical pathological image is obtained.

[0021] In a possible implementation of the first aspect, after the step of using the pathological genomics features to perform model training to obtain an immunotherapy prediction model, the method further includes:

[0022] Using the roc-curve function to construct a characteristic curve of the immunotherapy prediction model, and calculating the curve area corresponding to the characteristic curve to obtain a characteristic area value;

[0023] If the characteristic area value is smaller than the preset area value, the immunotherapy prediction model is optimized.

[0024] A second aspect of an embodiment of the present invention provides a method for calling an immunotherapy prediction model, the method comprising:

[0025] Obtain real-time pathological images of patients;

[0026] The immunotherapy prediction model trained by the immunotherapy prediction model training method as described above is called to calculate the real-time pathological image to obtain the efficacy probability value.

[0027] A third aspect of an embodiment of the present invention provides a training device for an immunotherapy prediction model, the device comprising:

[0028] An image and information acquisition module is used to acquire clinical pathology images and clinical pathology information, wherein the clinical pathology images are pathology images of immunotherapy patients before treatment, and the clinical pathology information includes clinical information of immunotherapy patients before treatment and prognosis information after treatment;

[0029] A feature extraction module is used to extract features from the labeled image based on the clinical pathology information after labeling the clinical pathology image to obtain a labeled image, so as to obtain a pathology omics feature;

[0030] The model training module is used to use the pathological group characteristics to perform model training to obtain an immunotherapy prediction model.

[0031] In a possible implementation manner of the third aspect, extracting features from the labeled image based on the clinical pathology information to obtain pathological omics features includes:

[0032] A preset feature extraction model is called to extract pathological genomic features in the marked area of ​​the marked image according to the clinical pathological information, wherein the pathological genomic features include pathological cell nucleus features, pathological microenvironment features and pathological spatial distribution features.

[0033] In a possible implementation of the third aspect, the using the pathological genomics features to perform model training to obtain an immunotherapy prediction model includes:

[0034] After constructing a plurality of base models, each of the base models is trained and optimized using the pathological omics features to obtain a plurality of training models;

[0035] After integrating several of the training models using the voting method of ensemble learning, each of the training models is fitted using the pathological omics features to obtain an immunotherapy prediction model.

[0036] In a possible implementation manner of the third aspect, marking the clinical pathology image to obtain a marked image includes:

[0037] Acquiring a marked area, wherein the marked area is an area where the medical staff outlines the tumor site in the image after showing the clinical pathology image to the medical staff;

[0038] The mark of the marked area is added to the clinical pathology image to obtain a marked image.

[0039] In a possible implementation manner of the third aspect, acquiring the clinical pathology image includes:

[0040] Acquiring a full-field pathological image of the patient, wherein the full-field pathological image is an image of a HE-stained section of the immunotherapy patient before receiving an anti-PD-(L)1 immune checkpoint inhibitor;

[0041] After digital scanning and screening of the full-field pathological image, a clinical pathological image is obtained.

[0042] In a possible implementation manner of the third aspect, the device further includes:

[0043] an area calculation module, for constructing a characteristic curve of the immunotherapy prediction model using the roc-curve function after the step of using the pathological group characteristics to perform model training to obtain the immunotherapy prediction model, and calculating the curve area corresponding to the characteristic curve to obtain a characteristic area value;

[0044] The optimization module is used to optimize the immunotherapy prediction model if the characteristic area value is less than a preset area value.

[0045] A fourth aspect of an embodiment of the present invention provides a device for calling an immunotherapy prediction model, the device comprising:

[0046] A real-time image acquisition module is used to acquire real-time pathological images of patients;

[0047] The model calling module is used to call the immunotherapy prediction model trained by the immunotherapy prediction model training method as described above to calculate the real-time pathological image to obtain the efficacy probability value.

[0048] Compared with the prior art, the training method, calling method, device and equipment of an immunotherapy prediction model provided by the embodiments of the present invention have the following beneficial effects: the present invention can obtain clinical pathology images and clinical pathology information; after marking the clinical pathology images to obtain marked images, feature extraction is performed on the marked images based on the clinical pathology information to obtain pathological genomic features; model training is performed using pathological genomic features to obtain an immunotherapy prediction model; image features are accurately extracted through pathology information, and then the image features are used for model training, which can improve the accuracy of the model and reduce the error of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flowchart of a method for training an immunotherapy prediction model provided by one embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the effect of the characteristic curve provided by an embodiment of the present invention. Figure 1 ;

[0051] Figure 3 This is a schematic diagram of the effect of the characteristic curve provided by an embodiment of the present invention. Figure 2 ;

[0052] Figure 4 is an operational flow chart of a method for training an immunotherapy prediction model provided by one embodiment of the present invention;

[0053] Figure 5 It is a flowchart of a method for calling an immunotherapy prediction model provided by one embodiment of the present invention;

[0054] Figure 6 It is a structural schematic diagram of a training device for an immunotherapy prediction model provided by one embodiment of the present invention;

[0055] Figure 7 It is a structural schematic diagram of a device for calling an immunotherapy prediction model provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Gastric cancer is a common malignancy and the third leading cause of cancer-related death worldwide. Immune checkpoint inhibitors targeting PD-(L)1 have become the first choice for gastric cancer patients who lack targeted treatment options. However, due to the complexity of anti-tumor immune regulation, these molecular markers cannot accurately identify patients who benefit from anti-PD-(L)1 immune checkpoint inhibitors. Therefore, it is necessary to test the therapeutic efficacy of anti-PD-(L)1 immune checkpoint inhibitors.

[0058] In order to simplify the processing flow of efficacy testing, artificial intelligence technology has been introduced in this field. The commonly used method is to collect the patient's HE full-field pathological slice image (WSI), use the collected images for model training, and obtain an image recognition model. The image recognition model is then used to identify the patient's treatment area, allowing professional and experienced pathologists to view the treatment area and make a diagnosis and differential diagnosis to determine the efficacy of the treatment.

[0059] However, the commonly used methods currently have the following technical problems: the color, shape and contour of normal tissue and pathological tissue in HE full-field pathology slice images are similar. The image recognition model obtained by only using HE full-field pathology slice images for model training has low recognition accuracy and large errors, which is difficult to meet existing application needs.

[0060] In order to solve the above problems, the training method, calling method, device and equipment of an immunotherapy prediction model provided in the embodiments of the present application will be introduced and explained in detail through the following specific embodiments.

[0061] Reference Figure 1 , showing a flow chart of a method for training an immunotherapy prediction model provided by an embodiment of the present invention.

[0062] In one embodiment, the method is applicable to a computer or an intelligent system, which can perform model training.

[0063] Wherein, as an example, the training method of the immunotherapy prediction model may include:

[0064] S11. Acquire clinical pathological images and clinical pathological information, wherein the clinical pathological images are pathological images of immunotherapy patients before treatment, and the clinical pathological information includes clinical information of immunotherapy patients before treatment and prognosis information after treatment.

[0065] In one embodiment, clinical pathology images and clinical pathology information may be acquired separately, wherein the clinical pathology images are pathology images of immunotherapy patients before treatment, and the clinical pathology information includes clinical information of immunotherapy patients before treatment and prognosis information after treatment.

[0066] In one operation mode, the clinical pathology images and clinical pathology information can be uploaded separately on the computer by a technician.

[0067] Wherein, as an example, the obtaining of clinical pathology images may include the following sub-steps:

[0068] S111. Acquire a full-field pathological image of the patient, wherein the full-field pathological image is an image of a HE-stained section of an immunotherapy patient before receiving an anti-PD-(L)1 immune checkpoint inhibitor.

[0069] S112, digitally scanning and screening the full-field pathological image to obtain a clinical pathological image.

[0070] In the specific operation, technicians can upload the information of patients who received anti-PD-(L)1 immune checkpoint inhibitors between January 20xx and December 20xx, with HE-stained sections before treatment, and complete treatment information and follow-up information. Then obtain high-quality HE-stained sections before receiving anti-PD-(L)1 immune checkpoint inhibitors, and then use a pathology full-slice scanner to digitally scan the patient's HE-stained sections under a 40x objective lens to obtain full-field pathology images.

[0071] In an optional embodiment, in order to determine the category of the full-field pathology image, the technician can evaluate the therapeutic effect of the immune checkpoint inhibitor according to the RECIST (v.1.1) standard, and divide the response of the full-field pathology image into progressive disease (PD), stable disease (SD), partial remission (PR) or complete remission (CR). Further, according to the patient's response to immunotherapy, they are divided into two groups: immunotherapy effective (CR / PR) and immunotherapy ineffective (SD / PD).

[0072] In another optional embodiment, in order to save various image data, the image can be digitally saved in an SVS format file after scanning. Specifically, a pathology full-slice scanner can be called to digitally scan the patient's HE-stained slice image under a 40x objective lens. After scanning, the full-field pathology image is digitally saved in an SVS format file.

[0073] After obtaining the full-field pathology images, in order to ensure the quality of the scanned full-field pathology images, the Aperio ImageScope software can be called to manage and view the full-field pathology images. At the same time, all full-field pathology images can be checked and screened to exclude full-field pathology images such as those with contamination on the slices, unfocused scans, and unclear scans, so as to obtain clinical pathology images.

[0074] S12: After labeling the clinical pathology image to obtain a labeled image, feature extraction is performed on the labeled image based on the clinical pathology information to obtain pathology omics features.

[0075] In one embodiment, the clinical pathology image may be labeled by marking the lesion area, treatment area, or rehabilitation area of ​​the image. Since the normal tissue and the pathological tissue in the image may be similar in color, shape, and outline in the image, the above-mentioned various areas can be distinguished by marking, and subsequent training can improve the recognition and classification accuracy of the model. After obtaining the labeled image, the labeled image can be feature extracted based on the clinical pathology information to obtain the pathology omics feature.

[0076] Specifically, feature extraction can be performed on the labeled image according to the patient's clinical pathological information, and various features of cells in the pathological tissue can be extracted to obtain pathological omics features.

[0077] In an optional embodiment, in order to accurately distinguish normal tissue from pathological tissue, as an example, the step of marking the clinical pathological image to obtain a marked image may include the following sub-steps:

[0078] S21, obtaining a marked area, wherein the marked area is an area where the medical staff outlines the tumor site in the image after showing the clinical pathology image to the medical staff.

[0079] S22. Add the mark of the marked area to the clinical pathology image to obtain a marked image.

[0080] In an actual operation mode, after obtaining high-quality clinical pathology images of patients before immunotherapy, Aperio ImageScope software can be called to show the clinical pathology images to medical staff.

[0081] After viewing the clinical pathology images, medical staff can determine the area of ​​the tumor site, the area of ​​normal tissue or the area of ​​recovery change, or the area of ​​personal interest in the clinical pathology images.

[0082] Then, the medical staff can outline the image, and the Aperio ImageScope software can receive the trajectory of the medical staff sliding in the image, determine the area to be marked based on the sliding trajectory, and obtain the marked area.

[0083] Finally, the marked area can be cropped out, or the marked area can be added to the image to obtain a marked image.

[0084] In one embodiment, subsequent model training may require various cell features. In order to perform model training through various cell features to improve the accuracy of the model, as an example, the feature extraction of the labeled image based on the clinical pathological information to obtain the pathological genomics feature may include the following sub-steps:

[0085] S31, calling a preset feature extraction model to extract pathological genomic features in the marked area of ​​the marked image according to the clinical pathological information, wherein the pathological genomic features include: pathological cell nucleus features, pathological microenvironment features and pathological spatial distribution features.

[0086] In order to cover complete clinical pathology information, a preset feature extraction model can be called to extract pathology genomics features in the marked area of ​​the marked image according to the clinical pathology information.

[0087] In one embodiment, three types of quantitative pathological omics features can be extracted, including pathological cell nuclear features, pathological microenvironment features, and pathological spatial distribution features. These three categories cover information about single cell morphology, cell spatial distribution, and overall microenvironment, providing a comprehensive understanding of the tumor microenvironment.

[0088] In an actual operation method, when extracting pathological cell nuclear features, the preset HoverNet model can be called according to the clinical pathological information to perform tumor cell nucleus segmentation on each marked area (such as the region of interest) of the marked image, and then the "MeasureObjectIntensity", "MeasureObjectSizeShape" and "MeasureTexture" modules of the CellProfiler platform are used to calculate the intensity feature values ​​of the cell nucleus, the morphological feature values ​​such as the cell nucleus circumference and size, and the statistical values ​​related to the cell nucleus texture, and finally 105 original values ​​of the cell nucleus features are obtained.

[0089] Furthermore, the mean, median, standard deviation, 25th percentile and 75th percentile of the original values ​​of the cell nuclear features obtained above in the full-field pathological images of each gastric cancer patient were calculated to form cell nuclear features, and finally 525 pathological cell nuclear features were obtained.

[0090] In a practical operation, when extracting pathological microenvironment features, the Python-public Openslide library can be used to segment the region of interest of the full-view pathological image into 256x 256 pixel image sub-blocks without overlap. Subsequently, using the ResNet50 model pre-trained on ImageNet, each 256x256 pixel image sub-block is converted into a 1024-dimensional feature vector by performing spatial average pooling after the third residual block as the pathological omics deep microenvironment feature.

[0091] In a practical operation, when extracting the spatial distribution features of pathology, the HoverNet model pre-trained on the Pannuke dataset can be used to segment and classify cells in the region of interest of the full-view pathology image and mark them as tumor cells, lymphocytes, stromal cells, dead cells or non-tumor epithelial cells. 2Each tumor cell is marked as a red dot, lymphocytes are marked as green dots, and stromal cells are marked as red and blue dots on the grid, and the number of tumor cells, lymphocytes, and stromal cells is calculated. Thus, the cell adjacency structure in the pathological image is simplified into an RGB format image composed of three color dot maps. The RGB format image contains the adjacent position information and density information of tumor cells, lymphocytes, and stromal cells. Finally, the pre-trained ResNet50 model for extracting deep microenvironment features of pathological genomics is applied to the RGB format image to extract the 1024-dimensional feature vector and obtain the pathological spatial distribution features.

[0092] S13. Using the pathological genomics features to perform model training, and obtain an immunotherapy prediction model.

[0093] After obtaining the pathological genomics features, the pathological genomics features can be used to train the model and optimize the model to obtain an immunotherapy prediction model.

[0094] In one of the embodiments, in order to improve the prediction accuracy of the model, it is necessary to perform integrated learning on the model, wherein, as an example, the model training using the pathological genomics features to obtain an immunotherapy prediction model may include the following sub-steps:

[0095] S131. After constructing several base models, each base model is trained and optimized using the pathological genomics features to obtain several training models.

[0096] S132. After integrating several of the training models using the voting method of ensemble learning, each of the training models is fitted using the pathological genomics features to obtain an immunotherapy prediction model.

[0097] After extracting the pathological omics features of the three categories, the pathological omics features can be used as input and the therapeutic efficacy of the patient can be used as output, which can be a classification result of the efficacy or a probability value. If it is a classification result, it can include a classification result of effective immunotherapy and a classification result of ineffective immunotherapy. If it is a probability value, whether it is effective can be determined based on the size of the probability value. For example, if the probability value is greater than the xx preset value, it is effective. On the contrary, if the probability value is less than the xx preset value, it is invalid.

[0098] Among them, the immunotherapy prediction model can be an integrated learning model for predicting the therapeutic efficacy of anti-PD-(L)1 immune checkpoint inhibitors for gastric cancer.

[0099] In specific operations, the pathological omics features can be divided into a training set and a validation set, and an integrated learning model for predicting the efficacy of gastric cancer immunotherapy can be trained based on the training set. The training includes: least absolute shrinkage and selection of several base models. In specific operations, the four base models of least absolute shrinkage and selection operator (LASSO), k-nearest neighbor algorithm (KNN), decision tree, and random forest can be further integrated using the voting method of integrated learning.

[0100] Specifically, each of the four base models is first trained and optimized using the training dataset to obtain four training models. This will include tuning the parameters of each model to perform best on the training data.

[0101] Next, use each training model to predict the test data set and obtain the prediction results of each training model. After obtaining the prediction results of each training model, the voting method can be used to combine these results. That is, the category label with the most votes is selected as the final prediction result. By combining different training models, the ensemble learning model can improve the accuracy and robustness of the model by integrating the advantages of different models.

[0102] It should be noted that the training of the base model and the ensemble learning model is implemented based on the Python machine learning library scikit-learn. The voting method is an ensemble learning model that follows the principle of majority rule. It reduces variance by integrating multiple models, thereby improving the robustness of the model.

[0103] After the model is trained, the accuracy of the model needs to be calculated. If the accuracy is low, the model needs to be optimized. As an example, the method may also include:

[0104] S14. Use the roc-curve function to construct a characteristic curve of the immunotherapy prediction model, and calculate the curve area corresponding to the characteristic curve to obtain a characteristic area value.

[0105] S15. If the characteristic area value is smaller than the preset area value, the immunotherapy prediction model is optimized.

[0106] Specifically, after the immunotherapy prediction model is trained, the immunotherapy prediction model can be verified and evaluated.

[0107] Reference Figure 2-3 , respectively showing the effect of the characteristic curve provided by an embodiment of the present invention Figure 1 The effect of the characteristic curve provided by an embodiment of the present invention is shown in FIG. Figure 2 .

[0108] The above-analyzed validation set can be used to validate the immunotherapy prediction model. The roc_curve function of the python machine learning library scikit-learn can be called to generate the receiver operating characteristic curve (ROC), as shown in the following example: Figure 2-3 As shown. Then calculate the area under the curve (AUC) to get the curve area corresponding to the characteristic curve.

[0109] The characteristic curve is a comprehensive indicator that reflects the sensitivity and specificity of continuous variables. It uses the composition method to reveal the relationship between sensitivity and specificity. It calculates a series of sensitivities and specificities by setting multiple different critical values ​​for continuous variables. The ROC curve is a curve drawn based on a series of different binary classification methods (cutoff values ​​or decision thresholds), with the True Positive Rate (TPR, that is, sensitivity) as the ordinate and the False Positive Rate (FPR, 1-specificity) as the abscissa. For different ROC curves, the greater the curvature / curvature, that is, the farther away from the 45-degree diagonal of accidental events (the diagonal represents the subject's discrimination of 0), the better the model performance.

[0110] After the characteristic curve is drawn, the area under the curve (AUC) is calculated using the sklearn.metrics.roc_curve function. The area under the curve is defined as the area under the ROC curve and the coordinate axis, so the value range of AUC is between 0.5 and 1. The closer the AUC is to 1.0, the higher the accuracy of the model prediction; when it is equal to 0.5, the prediction accuracy is the lowest and there is no application value.

[0111] Specifically, 0.5 can be used as the preset area value. If the characteristic area value is smaller than the preset area value, it means that the prediction accuracy of the immunotherapy prediction model is low, and the immunotherapy prediction model can be optimized.

[0112] Reference Figure 4 , shows an operational flow chart of a method for training an immunotherapy prediction model provided by one embodiment of the present invention.

[0113] Specifically, the training method of the immunotherapy prediction model may include the following steps:

[0114] The first step is to obtain pathological images of patients before immunotherapy.

[0115] The second step is pathological image processing.

[0116] The third step is pathological feature extraction.

[0117] The fourth step is to construct an integrated learning model to predict the efficacy of gastric cancer immunotherapy based on pathological characteristics.

[0118] The fifth step is to verify the integrated learning model for predicting the efficacy of gastric cancer immunotherapy based on pathological characteristics.

[0119] Introducing artificial intelligence technology into pathological image analysis can fully tap the value of pathological sections, realize automatic prediction of immunotherapy efficacy, improve the efficiency of pathologists, and effectively save medical resources. It can assist clinicians in accurately identifying patients who benefit from immunotherapy, help clinicians make decisions, and choose more reasonable treatment plans, so that patients can benefit from survival, reduce treatment side effects, and improve the quality of life of gastric cancer patients.

[0120] In this embodiment, the embodiment of the present invention provides a training method for an immunotherapy prediction model, and its beneficial effects are: the present invention can obtain clinical pathology images and clinical pathology information; after marking the clinical pathology image to obtain a marked image, feature extraction is performed on the marked image based on the clinical pathology information to obtain pathological genomics features; model training is performed using the pathological genomics features to obtain an immunotherapy prediction model; image features are accurately extracted through pathological information, and then the image features are used for model training, which can improve the accuracy of the model and reduce the error of the model.

[0121] Reference Figure 5 , showing a flow chart of a method for calling an immunotherapy prediction model provided by an embodiment of the present invention.

[0122] Wherein, as an example, the calling method of the immunotherapy prediction model may include:

[0123] S41. Acquire a real-time pathological image of the patient.

[0124] S42, calling the immunotherapy prediction model trained by the training method of the immunotherapy prediction model described in the above embodiment to calculate the real-time pathological image to obtain a therapeutic effect probability value.

[0125] In one embodiment, the real-time pathological image of the patient can be directly obtained, and then the real-time pathological image is input into the immunotherapy prediction model trained by the training method of the immunotherapy prediction model described in the above embodiment, and the real-time pathological image is calculated by the immunotherapy prediction model to obtain the efficacy probability value.

[0126] If the probability value is greater than 0.5, the treatment is effective. Otherwise, if the probability value is less than 0.5, the treatment is ineffective.

[0127] In this embodiment, the embodiment of the present invention provides a method for calling an immunotherapy prediction model, and its beneficial effect is that the present invention can improve the accuracy of efficacy evaluation and calculation and reduce calculation errors by performing calculations through the immunotherapy prediction model.

[0128] The present invention also provides a training device for an immunotherapy prediction model. Figure 6 , showing a schematic structural diagram of a training device for an immunotherapy prediction model provided by an embodiment of the present invention.

[0129] Wherein, as an example, the training device of the immunotherapy prediction model may include:

[0130] An image and information acquisition module 601 is used to acquire clinical pathology images and clinical pathology information, wherein the clinical pathology images are pathology images of immunotherapy patients before treatment, and the clinical pathology information includes clinical information of immunotherapy patients before treatment and prognosis information after treatment;

[0131] A feature extraction module 602 is used to extract features from the labeled image based on the clinical pathology information to obtain pathological genomic features after labeling the clinical pathology image to obtain a labeled image;

[0132] The model training module 603 is used to perform model training using the pathological genomics features to obtain an immunotherapy prediction model.

[0133] Optionally, extracting features from the labeled image based on the clinical pathological information to obtain pathological omics features includes:

[0134] A preset feature extraction model is called to extract pathological genomic features in the marked area of ​​the marked image according to the clinical pathological information, wherein the pathological genomic features include pathological cell nucleus features, pathological microenvironment features and pathological spatial distribution features.

[0135] Optionally, the method of using the pathological omics features to perform model training to obtain an immunotherapy prediction model comprises:

[0136] After constructing a plurality of base models, each of the base models is trained and optimized using the pathological omics features to obtain a plurality of training models;

[0137] After integrating several of the training models using the voting method of ensemble learning, each of the training models is fitted using the pathological omics features to obtain an immunotherapy prediction model.

[0138] Optionally, the step of marking the clinical pathology image to obtain a marked image includes:

[0139] Acquiring a marked area, wherein the marked area is an area where the medical staff outlines the tumor site in the image after showing the clinical pathology image to the medical staff;

[0140] The mark of the marked area is added to the clinical pathology image to obtain a marked image.

[0141] Optionally, acquiring clinical pathological images includes:

[0142] Acquiring a full-field pathological image of the patient, wherein the full-field pathological image is an image of a HE-stained section of the immunotherapy patient before receiving an anti-PD-(L)1 immune checkpoint inhibitor;

[0143] After digital scanning and screening of the full-field pathological image, a clinical pathological image is obtained.

[0144] Optionally, the device further comprises:

[0145] an area calculation module, for constructing a characteristic curve of the immunotherapy prediction model using the roc-curve function after the step of using the pathological group characteristics to perform model training to obtain the immunotherapy prediction model, and calculating the curve area corresponding to the characteristic curve to obtain a characteristic area value;

[0146] The optimization module is used to optimize the immunotherapy prediction model if the characteristic area value is less than a preset area value.

[0147] The present invention also provides a device for calling an immunotherapy prediction model, see Figure 7 , showing a schematic diagram of the structure of a device for calling an immunotherapy prediction model provided by an embodiment of the present invention.

[0148] Wherein, as an example, the calling device of the immunotherapy prediction model may include:

[0149] A real-time image acquisition module 701 is used to acquire a real-time pathological image of a patient;

[0150] The model calling module 702 is used to call the immunotherapy prediction model trained by the immunotherapy prediction model training method as described above to calculate the real-time pathological image to obtain the efficacy probability value.

[0151] Those skilled in the art can clearly understand that, for the sake of convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0152] Furthermore, an embodiment of the present application also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training method of the immunotherapy prediction model as described in the above embodiment or the calling method of the immunotherapy prediction model as described in the above embodiment is implemented.

[0153] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer-executable program, and the computer-executable program is used to enable a computer to execute the training method of the immunotherapy prediction model as described in the above embodiment or the calling method of the immunotherapy prediction model as described in the above embodiment.

[0154] It should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. When an element such as a layer, region or substrate is referred to as being "on" or "above" another element, it can be directly on the other element, or there can also be an intermediate element. On the contrary, when an element is referred to as "directly on" or "above" another element, there is no intermediate element. It should also be understood that when an element is referred to as being "under" or "below" another element, it can be directly under or below the other element, or there can also be an intermediate element. On the contrary, when an element is referred to as being "directly under" or "below" another element, there is no intermediate element. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0155] Those skilled in the art will appreciate that the embodiments of the present application may also provide computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes.

[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), apparatuses, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0159] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for training an immunotherapy prediction model, characterized in that: The method comprises: Acquiring clinical pathology images and clinical pathology information, wherein the clinical pathology images are pathology images of immunotherapy patients before treatment, and the clinical pathology information includes clinical information of immunotherapy patients before treatment and prognosis information after treatment; After labeling the clinical pathology image to obtain a labeled image, feature extraction is performed on the labeled image based on the clinical pathology information to obtain a pathology omics feature; The pathological omics features are used to perform model training to obtain an immunotherapy prediction model.

2. The method for training an immunotherapy prediction model according to claim 1, characterized in that: The step of extracting features from the labeled image based on the clinical pathological information to obtain pathological omics features includes: A preset feature extraction model is called to extract pathological genomic features in the marked area of ​​the marked image according to the clinical pathological information, wherein the pathological genomic features include pathological cell nucleus features, pathological microenvironment features and pathological spatial distribution features.

3. The method for training an immunotherapy prediction model according to claim 1, characterized in that: The method of using the pathological genomics features to perform model training to obtain an immunotherapy prediction model comprises: After constructing a plurality of base models, each of the base models is trained and optimized using the pathological omics features to obtain a plurality of training models; After integrating several of the training models using the voting method of ensemble learning, each of the training models is fitted using the pathological genomics features to obtain an immunotherapy prediction model.

4. The method for training an immunotherapy prediction model according to claim 1, characterized in that: The step of marking the clinical pathology image to obtain a marked image includes: Acquiring a marked area, wherein the marked area is an area where the medical staff outlines the tumor site in the image after showing the clinical pathology image to the medical staff; The mark of the marked area is added to the clinical pathology image to obtain a marked image.

5. The method for training an immunotherapy prediction model according to claim 1, characterized in that: The obtaining of clinical pathological images comprises: Acquiring a full-field pathological image of the patient, wherein the full-field pathological image is an image of a HE-stained section of the immunotherapy patient before receiving an anti-PD-(L)1 immune checkpoint inhibitor; After digital scanning and screening of the full-field pathological image, a clinical pathological image is obtained.

6. The method for training an immunotherapy prediction model according to any one of claims 1 to 5, characterized in that: After the step of using the pathological genomics features to perform model training to obtain an immunotherapy prediction model, the method further comprises: Using the roc-curve function to construct a characteristic curve of the immunotherapy prediction model, and calculating the curve area corresponding to the characteristic curve to obtain a characteristic area value; If the characteristic area value is smaller than the preset area value, the immunotherapy prediction model is optimized.

7. A method for calling an immunotherapy prediction model, characterized in that: The method comprises: Obtain real-time pathological images of patients; The immunotherapy prediction model trained by the training method of the immunotherapy prediction model as described in any one of claims 1 to 6 is called to calculate the real-time pathological image to obtain a efficacy probability value.

8. A training device for an immunotherapy prediction model, characterized in that: The device comprises: An image and information acquisition module is used to acquire clinical pathology images and clinical pathology information, wherein the clinical pathology images are pathology images of immunotherapy patients before treatment, and the clinical pathology information includes clinical information of immunotherapy patients before treatment and prognosis information after treatment; A feature extraction module is used to extract features from the labeled image based on the clinical pathology information after labeling the clinical pathology image to obtain a labeled image, so as to obtain a pathology omics feature; The model training module is used to use the pathological group characteristics to perform model training to obtain an immunotherapy prediction model.

9. A device for calling an immunotherapy prediction model, characterized in that: The device comprises: A real-time image acquisition module is used to acquire real-time pathological images of patients; The model calling module is used to call the immunotherapy prediction model trained by the training method of the immunotherapy prediction model as described in any one of claims 1 to 6 to calculate the real-time pathological image to obtain the efficacy probability value.

10. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method for the immunotherapy prediction model as described in any one of claims 1 to 6 or the calling method for the immunotherapy prediction model as described in claim 7 is implemented.

Images