Tumor efficacy prediction method and system based on image and pathology interactive network

Abstract

The present invention belongs to the field of medical detection technology, and provides a method and system for predicting tumor efficacy based on an image and pathology interaction network, including extracting features from acquired image data and pathology images to obtain an image feature matrix and a pathology feature matrix; constructing an image and pathology interaction network based on the image feature matrix and the pathology feature matrix to extract the interaction information between the image features and the pathology features; combining the interaction information between the image features and the pathology features with multimodal features to output the efficacy prediction result; defining a loss function, optimizing model parameters, and obtaining a trained efficacy prediction model. The present invention characterizes individualized biological characteristics by learning the macroscopic and microscopic interaction information between CT images and HE-stained pathology modalities, thereby predicting individual treatment efficacy before surgery and providing guidance for clinical treatment decisions for tumor patients.

Description

Technical Field

[0001] The present disclosure relates to the field of medical detection technology, and in particular to a method and system for predicting tumor efficacy based on an image and pathology interactive network. Background Art

[0002] Lung cancer is the cancer with the highest morbidity and mortality worldwide. Non-small cell lung cancer (NSCLC) is the main type of lung cancer, and about 75% of NSCLC are in the advanced stage when first diagnosed. Radiotherapy and chemotherapy are the main treatments for patients with advanced NSCLC, but the efficacy is poor. The five-year survival rate was less than 25.4% between 2013 and 2019. Immune checkpoint inhibitors (ICIs) are the most commonly used immunotherapy regimen, and have made breakthrough progress in the treatment of advanced NSCLC. However, due to the heterogeneity of tumors and the complexity of the tumor immune microenvironment, the overall efficacy of ICIs treatment is low. Predicting the efficacy of ICIs in NSCLC patients before treatment can screen out the potential beneficiaries of ICIs, avoid ineffective treatment, and reduce the risk of adverse reactions. Therefore, how to predict the efficacy of immunotherapy is crucial for the choice of treatment for NSCLC patients.

[0003] CT images and pathological images before treatment can reflect the biological characteristics of patients from different dimensions. The construction of artificial intelligence models based on CT images and pathological multimodal data shows great potential in lung cancer diagnosis and efficacy prediction. CT can provide macroscopic structural information of tumors, such as size, morphology and spatial heterogeneity. Intelligent analysis models based on medical images are effective tools for lung cancer prognosis prediction. Pathological images present cell morphology, tumor microenvironment and specific protein expression information. Artificial intelligence research has found that these pathological microscopic information is related to tumor prognosis. Recent studies have combined the global macroscopic information of tumor areas on medical images with the local microscopic information on pathological images, achieving better performance in cancer prognosis prediction. However, such methods usually simply combine the features of the two for joint modeling. No research has yet effectively integrated the interactive information between the modalities of imaging and pathology to build an efficacy prediction model for NSCLC immunotherapy. Summary of the invention

[0004] In response to the above-mentioned defects, the present invention proposes a tumor efficacy prediction method and system based on an image and pathology interaction network. By learning the macroscopic and microscopic interaction information between CT images and HE staining pathological modalities, individualized biological characteristics are characterized, thereby predicting the efficacy of immunotherapy before surgery and improving the precision and accuracy of efficacy prediction.

[0005] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0006] The first aspect of the present disclosure provides a method for predicting tumor efficacy based on an image and pathology interaction network, comprising the following steps:

[0007] Perform feature extraction on the acquired image data and pathological images to obtain an image feature matrix and a pathological feature matrix;

[0008] Based on the image feature matrix and the pathological feature matrix, an image-pathology interaction network is constructed to extract the interaction information between the image features and the pathological features;

[0009] Combine the interactive information between image features and pathological features with multimodal features to output therapeutic effect prediction results;

[0010] Define the loss function, optimize the model parameters, and obtain a trained efficacy prediction model.

[0011] As a further implementation method, the image and pathology interaction network includes a static interaction network and a dynamic interaction network, which are used to extract static interaction information and dynamic interaction information between image features and pathology features, respectively.

[0012] As a further implementation method, the dynamic interaction network is dynamically optimized according to the prediction results during the model training process, and the static interaction network does not change with the model training.

[0013] As a further implementation method, the Pearson correlation coefficient matrix is ​​used to represent the static interaction network between image features and pathological features, and the Transformer self-attention network is used to represent the dynamic interaction network between image features and pathological features.

[0014] As a further implementation method, the interactive information between the image features and the pathological features is combined with the multimodal features to output the therapeutic effect prediction results, specifically:

[0015] Through Transformer, the macroscopic global characteristics of the tumor, cell morphology and microenvironment characteristics expressed by CT images, pathological images and clinical information, as well as individualized diagnosis and treatment data are integrated, and the interactive information between image features and pathological features is combined. The combined information is converted into efficacy prediction results through a linear layer.

[0016] As a further implementation, the multimodal features include imaging features, pathological features and clinical features, which provide macroscopic, microscopic and clinical information respectively.

[0017] As a further implementation method, before extracting features from the image data and the pathological image, the image data and the pathological image are also preprocessed.

[0018] The second aspect of the present disclosure provides a tumor efficacy prediction system based on an image and pathology interactive network, comprising:

[0019] The feature extraction module is configured to: extract features from the acquired image data and pathological images to obtain an image feature matrix and a pathological feature matrix;

[0020] The interactive information extraction module is configured to: construct an image and pathology interactive network based on the image feature matrix and the pathology feature matrix, and extract interactive information between the image features and the pathology features;

[0021] The joint prediction module is configured to: combine the interactive information between the image features and the pathological features with the multimodal features to output the therapeutic effect prediction result;

[0022] The model training module is configured to: define the loss function, optimize the model parameters, and obtain a trained efficacy prediction model.

[0023] The third aspect of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps of the tumor efficacy prediction method based on an image and pathology interactive network described in the first aspect of the present disclosure are implemented.

[0024] The fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a method for predicting tumor efficacy based on an image and pathology interactive network described in the first aspect of the present disclosure are implemented.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The tumor efficacy prediction method and system based on the image and pathology interaction network proposed in the present disclosure adopts a combination of static interaction network and dynamic interaction network, and uses the Pearson correlation coefficient matrix to represent the static interaction information between image features and pathology features, and uses the Transformer self-attention network to represent the dynamic interaction information between image features and pathology features. The static and dynamic interaction information are further integrated through the Transformer to construct an "image and pathology interaction network" to characterize the interaction relationship between modalities. Then, the macro, micro and clinical features of the three modalities and the "image-pathology interaction network" information are integrated through the Transformer network to construct a multimodal efficacy prediction model for the prediction of immunotherapy efficacy, so that the model has achieved better performance in prediction and improved the accuracy and precision of the prediction.

[0027] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0029] Figure 1 is a data processing flow chart of the tumor efficacy prediction method based on the image and pathology interactive network disclosed in the present invention;

[0030] Figure 2 is a schematic diagram of CT image feature extraction disclosed in the present invention;

[0031] Figure 3 is a schematic diagram of HE pathology image feature extraction disclosed in the present invention;

[0032] Figure 4 It is a framework diagram of the efficacy prediction model disclosed in the present invention. DETAILED DESCRIPTION

[0033] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0035] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0036] Embodiment 1

[0037] like Figure 1 As shown, the first embodiment of the present disclosure provides a method for predicting tumor efficacy based on an image and pathology interaction network, which mainly includes the following steps:

[0038] S1 extracts features from the acquired image data and pathological images to obtain an image feature matrix and a pathological feature matrix;

[0039] S2 builds an image-pathology interaction network based on the image feature matrix and the pathology feature matrix to extract the interaction information between the image features and the pathology features;

[0040] S3 combines the interactive information between image features and pathological features with multimodal features to output therapeutic effect prediction results;

[0041] S4 defines the loss function, optimizes the model parameters, and obtains a trained efficacy prediction model.

[0042] Before extracting features from image data and pathological images, they need to be preprocessed. Specifically:

[0043] The regions of interest were marked on CT images and HE pathology images, and the tumor area and lungs were marked on CT images, and the tumor area was marked on pathology images. The CT data were resampled to reduce the impact of image data differences between patients on the results, and then the grayscale value range of the image data was discretized. At a magnification of 2.5 times, the tumor area of ​​the pathology image was cropped into a 256×256 pathology sub-image, and the coordinate range of each sub-image was mapped to images with 5 times, 10 times, 20 times, and 40 times magnification, and 4, 16, 64, and 256 256×256 sub-images were cropped from the mapped area, respectively. Then, the Vahadane method was used to perform color standardization on all pathology sub-images to reduce the impact of staining differences on the results.

[0044] After data preprocessing, the PyRadiomics tool and the pre-trained deep learning network ResNet50 were used to extract image features from each region of interest on CT, and the features of different regions of interest were combined into an image feature matrix, such as Figure 2 At the same time, the pre-trained ResNet50 is used to extract pathological features from each pathological image, and then the features of all images at each magnification are aggregated into a set of feature vectors by using mean aggregation, and the feature vectors at different magnifications are combined into a pathological feature matrix, as shown in Figure 3 shown.

[0045] An image-pathology interaction network is constructed based on the image feature matrix and the pathology feature matrix to extract the interaction information between the image features and the pathology features.

[0046] The image and pathology interaction network was constructed by combining static interaction network with dynamic interaction network. The static interaction relationship between CT features and HE pathology features was characterized by Pearson correlation coefficient matrix, which was calculated as follows:

[0047]

[0048] in, For the i The sample m Image features, For the i The sample k pathological features, and Respectively m image features and k The sample means of the pathological characteristics.

[0049] The dynamic interaction network between CT features and HE pathological features is represented by the self-attention network of Transformer, and the calculation method is:

[0050] in, R m and P k For all samples, m image features and k pathological features, CONCAT [ R m ; P k ] is the combination of the two. Q is the query matrix, K is the keyword matrix, V is a numerical matrix, W Q , W K and W V They are Q , K and V The weight parameters are automatically updated during the training process. d k For use QK T Hyperparameters for scaling.

[0051] The interaction relationship of the dynamic interaction network is dynamically optimized according to the prediction results during the model training process. The static interaction network does not change with the model training and can provide a stable interaction relationship. The combination of the two can effectively avoid the overfitting problem during the model training process.

[0052] The interactive information between image features and pathological features is combined with multimodal features to output the therapeutic effect prediction results. Multimodal features include image features, pathological features, and clinical features, which provide macroscopic, microscopic, and clinical information respectively.

[0053] Based on the image and pathology interaction network, combined with multimodal features, a therapeutic effect prediction model is constructed based on Transformer, such as Figure 4Specifically, the macroscopic global characteristics of the tumor, cell morphology and microenvironment characteristics expressed by CT images, HE pathological images and clinical information, as well as individualized diagnosis and treatment data are integrated through Transformer, and the modal interaction information of the image and pathological interaction network is further combined. The calculation method is:

[0054]

[0055] in, C For clinical information, R , P , S and D They are all the imaging features, pathological features, static interaction networks, and dynamic interaction networks of all samples. CONCAT [ R ; P ; S ; D ] is the concatenation of four sets of features.

[0056] The combined information is transformed into efficacy prediction results through a linear layer.

[0057] Define the loss function, optimize the model parameters, and obtain a trained efficacy prediction model.

[0058] The data of all patients were divided into training set, test set and validation set in a 3:1:1 manner. The training set was used for training and optimization of the multimodal model, the test set was used to select the best hyperparameters of the model, and the validation set was used to verify the predictive performance of the model. The receiver operating characteristic (ROC) curve, accuracy, sensitivity, specificity and F1 score were used to evaluate the predictive performance of the model.

[0059] This paper innovatively proposes an "image and pathology interaction network" to characterize the interaction between different modal data. This method calculates the static interaction network and the dynamic interaction network through the Pearson correlation matrix and the self-attention mechanism of Transformer, respectively, and deeply explores the correlation between different modal data from different angles. This network not only reveals the mechanism of action between CT macroscopic features, tumor cell morphology and its microenvironment and clinical information, but also provides a new perspective for understanding the mutual influence of these factors.

[0060] This paper introduces the interaction between modalities for the first time in the multimodal efficacy prediction task. Using the Transformer network, the modal features and the interaction between modalities are combined to integrate the global macro information of CT image features, the cell morphology and microenvironment characteristics of HE pathological images, clinical information, and the interaction between these information, which jointly improves the prediction performance of the efficacy of NSCLC immunotherapy. This method not only enriches the dimension of efficacy prediction, but also provides more accurate support for individualized treatment strategies.

[0061] Embodiment 2

[0062] The second embodiment of the present disclosure provides a tumor efficacy prediction system based on an image and pathology interactive network, including:

[0063] The feature extraction module is configured to: extract features from the acquired image data and pathological images to obtain an image feature matrix and a pathological feature matrix;

[0064] The interactive information extraction module is configured to: construct an image and pathology interactive network based on the image feature matrix and the pathology feature matrix, and extract interactive information between the image features and the pathology features;

[0065] The joint prediction module is configured to: combine the interactive information between the image features and the pathological features with the multimodal features to output the therapeutic effect prediction result;

[0066] The model training module is configured to: define the loss function, optimize the model parameters, and obtain a trained efficacy prediction model.

[0067] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0068] Embodiment 3

[0069] Embodiment 3 of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the method for predicting tumor efficacy based on an image and pathology interactive network as described in Embodiment 1 of the present disclosure are implemented.

[0070] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0071] Embodiment 4

[0072] Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in a method for predicting tumor efficacy based on an image and pathology interactive network as described in Embodiment 1 of the present disclosure are implemented.

[0073] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0074] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A tumor efficacy prediction method based on an image and pathology interaction network, characterized in that: The following steps are involved: Feature extraction is performed on the acquired image data and pathological images to obtain the image feature matrix and pathological feature matrix, specifically: The PyRadiomics tool and the pre-trained deep learning network ResNet50 were used to extract image features from each region of interest on CT, and the features of different regions of interest were combined into an image feature matrix. The pre-trained ResNet50 was used to extract pathological features from each pathological image. Then, the features of all the images at each magnification were aggregated into a set of feature vectors using mean aggregation, and the feature vectors at different magnifications were combined into a pathological feature matrix. Based on the image feature matrix and the pathological feature matrix, an image-pathology interaction network is constructed to extract the interaction information between the image features and the pathological features; The image and pathology interaction network includes a static interaction network and a dynamic interaction network, which are used to extract the static interaction information and dynamic interaction information between image features and pathology features respectively; The Pearson correlation coefficient matrix is ​​used to represent the static interaction network between image features and pathological features, and the Transformer self-attention network is used to represent the dynamic interaction network between image features and pathological features. The static interaction network is calculated as: in, For the i The sample m Image features, For the i The sample k pathological features, and Respectively m image features and k The sample mean of each pathological feature; The dynamic interaction network is calculated as: in, R m and P k For all samples, m image features and k pathological features, CONCAT [ R m ; P k ] is the combination of the two. Q is the query matrix, K is the keyword matrix, V is a numerical matrix, W Q , W K and W V They are Q , K and V The weight parameters are automatically updated during the training process. d k For use QK T Hyperparameters for scaling; The dynamic interaction network is dynamically optimized according to the prediction results during the model training process, while the static interaction network does not change with the model training; Combine the interactive information between image features and pathological features with multimodal features to output therapeutic effect prediction results; The interactive information between image features and pathological features is combined with multimodal features to output the efficacy prediction results, specifically: Through Transformer, the macroscopic global features of the tumor, cell morphology and microenvironmental features, and individualized diagnosis and treatment data expressed by CT images, pathological images, and clinical information are integrated, and the interactive information between image features and pathological features is combined. The combined information is converted into efficacy prediction results through a linear layer; The multimodal features include imaging features, pathological features, and clinical features, which provide macroscopic, microscopic, and clinical information, respectively; Define the loss function, optimize the model parameters, and obtain a trained efficacy prediction model.

2. The method for predicting tumor efficacy based on image and pathology interaction network as claimed in claim 1, characterized in that: Before extracting features from the image data and the pathological image, the image data and the pathological image are also preprocessed.

3. A tumor efficacy prediction system based on an image and pathology interactive network, characterized in that: include: The feature extraction module is configured to: extract features from the acquired image data and pathological images to obtain an image feature matrix and a pathological feature matrix, specifically: The PyRadiomics tool and the pre-trained deep learning network ResNet50 were used to extract image features from each region of interest on CT, and the features of different regions of interest were combined into an image feature matrix. The pre-trained ResNet50 was used to extract pathological features from each pathological image. Then, the features of all the images at each magnification were aggregated into a set of feature vectors using mean aggregation, and the feature vectors at different magnifications were combined into a pathological feature matrix. The interactive information extraction module is configured to: construct an image and pathology interactive network based on the image feature matrix and the pathology feature matrix, and extract interactive information between the image features and the pathology features; The image and pathology interaction network includes a static interaction network and a dynamic interaction network, which are used to extract the static interaction information and dynamic interaction information between image features and pathology features respectively; The Pearson correlation coefficient matrix is ​​used to represent the static interaction network between image features and pathological features, and the Transformer self-attention network is used to represent the dynamic interaction network between image features and pathological features. The static interaction network is calculated as: in, For the i The sample m Image features, For the i The sample k pathological features, and Respectively m image features and k The sample mean of each pathological feature; The dynamic interaction network is calculated as: in, R m and P k For all samples, m image features and k pathological features, CONCAT [ R m ; P k ] is the combination of the two. Q is the query matrix, K is the keyword matrix, V is a numerical matrix, W Q , W K and W V They are Q , K and V The weight parameters are automatically updated during the training process. d k For use QK T Hyperparameters for scaling; The dynamic interaction network is dynamically optimized according to the prediction results during the model training process, while the static interaction network does not change with the model training; The joint prediction module is configured to: combine the interactive information between the image features and the pathological features with the multimodal features to output the therapeutic effect prediction result; The interactive information between image features and pathological features is combined with multimodal features to output the efficacy prediction results, specifically: Through Transformer, the macroscopic global features of the tumor, cell morphology and microenvironmental features, and individualized diagnosis and treatment data expressed by CT images, pathological images, and clinical information are integrated, and the interactive information between image features and pathological features is combined. The combined information is converted into efficacy prediction results through a linear layer; The multimodal features include imaging features, pathological features, and clinical features, which provide macroscopic, microscopic, and clinical information, respectively; The model training module is configured to: define the loss function, optimize the model parameters, and obtain a trained efficacy prediction model.

4. A medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for predicting tumor efficacy based on an image and pathology interactive network as described in any one of claims 1 to 2 are implemented.

5. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the tumor efficacy prediction method based on the image and pathology interaction network as described in any one of claims 1-2 are implemented.

Images