Lung cancer immunotherapy efficacy prediction method and system based on dual CT images

Abstract

The present invention proposes a method and system for predicting the efficacy of lung cancer immunotherapy based on dual CT images, belonging to the technical field of medical image processing. First, the tumor regions of the baseline CT image and the puncture CT image are labeled. The puncture position in the puncture CT image is registered to the baseline CT image, and the baseline CT image is divided into multiple tumor sub-regions. Based on a spatial attention deep learning model, the PD-L1 expression level at the position of each tumor sub-region is predicted and a PD-L1 map is generated. A dual-task prediction model is constructed, and the clinical information of the patient, the baseline CT image, and the PD-L1 map are input into the dual-task prediction model to predict the efficacy of lung cancer immunotherapy. The present invention can achieve accurate prediction of the efficacy of lung cancer immunotherapy for non-small cell lung cancer patients on the basis of ensuring effective image-pathological spatial matching of non-small cell lung cancer.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and particularly relates to a method and system for predicting the efficacy of lung cancer immunotherapy based on dual CT images. Background Art

[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] Lung cancer is the cancer with the highest incidence and mortality worldwide; among them, non-small cell lung cancer (NSCLC) is the main type of lung cancer. Approximately 75% of NSCLC patients are in the advanced stage at the time of initial diagnosis. Radiotherapy and chemotherapy are the main treatment methods for advanced NSCLC patients, but the curative effect is poor, and the five-year survival rate was less than 25.4% between 2013 and 2019; improving the long-term survival rate is the key issue in the treatment of advanced NSCLC.

[0004] Immunotherapy based on programmed death-ligand 1 (PD-L1) has made breakthrough progress in the treatment of NSCLC and can effectively extend the survival time of advanced NSCLC patients. However, due to the tumor heterogeneity and the complexity of the tumor immune microenvironment, the overall effective rate of immunotherapy is low, and the objective response rate of using PD-L1 inhibitors in the advanced NSCLC population is only about 20%. For patients who cannot benefit, there is a risk of serious adverse reactions due to treatment-related toxicity and missing the best treatment opportunity. Therefore, predicting the efficacy of immunotherapy and screening potential beneficiary populations before treatment are crucial for the treatment decision-making of advanced NSCLC patients.

[0005] Clinically, the expression level of PD-L1 is used as a biomarker to predict the effect of NSCLC immunotherapy. High PD-L1 expression is often associated with a better immunotherapy response. However, the performance of predicting the efficacy of NSCLC immunotherapy based on PD-L1 is not ideal, and some patients with low or even negative PD-L1 expression can also benefit from immunotherapy. Tumor spatial heterogeneity is one of the main influencing factors. PD-L1 is mainly detected by immunohistochemistry using specific antibodies on pathological tissues obtained by puncture biopsy; however, biopsy can only obtain a small range of local tumor tissues, and the detection result only represents the local PD-L1 expression level of the tumor, making it difficult to evaluate the overall PD-L1 expression of the tumor, which may lead to an underestimation of the PD-L1 expression level in some patients. Therefore, comprehensively evaluating the overall PD-L1 expression of the tumor is the key issue to improve the accuracy of predicting the efficacy of NSCLC immunotherapy.

[0006] Recent studies have shown that CT images combined with artificial intelligence algorithms can also predict the efficacy of immunotherapy for advanced NSCLC. These studies generally use a single CT image as the analysis object and directly use convolutional neural network algorithms to predict the expression level of PD-L1 and / or the efficacy of immunotherapy. There are still some technical problems in dealing with the relationship among CT, PD-L1 expression, and the efficacy of immunotherapy for advanced NSCLC. For example:

[0007] (1) In studies using the PD-L1 expression level as the model input and the efficacy of immunotherapy for advanced NSCLC as the prediction target, only the individualized treatment efficacy of patients can be predicted based on the local PD-L1 expression level obtained from percutaneous biopsy, lacking an assessment of the spatial distribution of the global PD-L1 expression in the tumor.

[0008] (2) In studies using CT images as the model input and the PD-L1 expression level as the prediction target, only the global imaging features of the tumor are used to predict the local PD-L1 expression level, without matching the spatial positions of the imaging and pathology, and lacking attention and emphasis on the local CT image features at the PD-L1 detection position. Therefore, the prediction accuracy of this method for the efficacy of NSCLC immunotherapy is not ideal.

[0009] (3) In studies using CT images or CT combined with the PD-L1 expression level as the model input and the efficacy of immunotherapy for advanced NSCLC as the prediction target, it is impossible to perform cross-modal spatial mapping between CT images and PD-L1, and the association between CT features and the efficacy of immunotherapy for advanced NSCLC lacks interpretability. Summary of the Invention

[0010] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a method and system for predicting the efficacy of lung cancer immunotherapy based on dual CT images, which can accurately predict the efficacy of lung cancer immunotherapy for non-small cell lung cancer patients on the basis of ensuring effective matching and mapping of the imaging-pathology space of non-small cell lung cancer.

[0011] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:

[0012] The first aspect of the present invention provides a method for predicting the efficacy of lung cancer immunotherapy based on dual CT images.

[0013] The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images includes:

[0014] Obtain the baseline CT image and the puncture CT image of a non-small cell lung cancer patient, and label the tumor regions in the baseline CT image and the puncture CT image respectively;

[0015] The tissue puncture position in the puncture CT image is registered to the baseline CT image by using an image deformation registration method, and the baseline CT image is divided into multiple tumor sub-regions;

[0016] The entire tumor region of the labeled baseline CT image and each tumor sub-region are jointly input into a spatial attention deep learning model to predict the PD-L1 expression level at the position of each tumor sub-region; according to the obtained PD-L1 expression level, a PD-L1 map is generated;

[0017] A dual-task prediction model is constructed, taking the clinical information of non-small cell lung cancer patients, the baseline CT image, and the obtained PD-L1 map as the input of the dual-task prediction model to predict the efficacy of lung cancer immunotherapy.

[0018] Furthermore, the baseline CT image is divided into multiple tumor sub-regions. Specifically, the tumor region marked on the baseline CT image is divided into several sub-regions with the same size as the puncture region.

[0019] Furthermore, the spatial attention deep learning model includes a CNN network module and a spatial attention module; among them, the spatial attention module consists of two parallel fully connected layers, a cross-attention module, and two serial fully connected layers.

[0020] Furthermore, based on the CNN network module, the deep image features of the entire tumor region on the baseline CT image are extracted, and the radiomics features of the puncture position on the baseline CT image are calculated; based on the spatial attention module, the obtained deep image features and radiomics features are jointly analyzed to construct and optimize the PD-L1 prediction model to predict the PD-L1 expression level at the position of each tumor sub-region.

[0021] Furthermore, the dual-task prediction model consists of two CNN network modules and a Transformer module integrated with an attention mechanism.

[0022] Furthermore, the two CNN network modules are used to extract deep features. Specifically: in the two CNN network modules, one is used to extract the deep features of the PD-L1 map, and the other is used to extract the deep features of the baseline CT image; the Transformer module is used to integrate and analyze the obtained two types of deep features and the clinical information of non-small cell lung cancer patients.

[0023] Furthermore, before sending the clinical information of non-small cell lung cancer patients into the dual-task prediction model, the clinical information is digitally encoded.

[0024] The second aspect of the present invention provides a lung cancer immunotherapy efficacy prediction system based on dual CT images.

[0025] A lung cancer immunotherapy efficacy prediction system based on dual CT images, comprising:

[0026] An image annotation module, configured to: obtain the baseline CT image and the puncture CT image of a non-small cell lung cancer patient, and respectively annotate the tumor regions in the baseline CT image and the puncture CT image;

[0027] An imaging pathology spatial matching module, configured to: register the tissue puncture position in the puncture CT image to the baseline CT image by using an image deformation registration method, and divide the baseline CT image into multiple tumor sub-regions;

[0028] A PD-L1 map construction module, configured to: jointly input the entire tumor region of the annotated baseline CT image and each tumor sub-region into a spatial attention deep learning model to predict the PD-L1 expression level at the position of each tumor sub-region; generate a PD-L1 map according to the obtained PD-L1 expression level;

[0029] An efficacy prediction module, configured to: construct a dual-task prediction model, and use the clinical information of the non-small cell lung cancer patient, the baseline CT image, and the obtained PD-L1 map as the input of the dual-task prediction model to predict the efficacy of lung cancer immunotherapy.

[0030] The third aspect of the present invention provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in the first aspect of the present invention are implemented.

[0031] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored on the memory and executable on the processor, and when the processor executes the program, the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in the first aspect of the present invention are implemented.

[0032] The above one or more technical solutions have the following beneficial effects:

[0033] (1) The present invention first obtains the dual CT images (baseline CT image and puncture CT image) of NSCLC patients, and respectively labels the tumor regions in the dual CT images; then adopts the image deformation registration method to register the tissue puncture position in the puncture CT image to the baseline CT image, so as to realize the local spatial matching of CT images and PD-L1 pathology, and divides the baseline CT image into multiple tumor sub-regions. On this basis, the entire tumor region of the labeled baseline CT image and the puncture position are jointly input into the spatial attention deep learning model for PD-L1 expression prediction, rather than only predicting the local PD-L1 expression based on the global image features of the tumor. Based on this model, the PD-L1 expression is evaluated for each tumor sub-region, thereby realizing the evaluation of the spatial distribution of the global PD-L1 expression of the tumor. Therefore, the present invention can realize the cross-modal spatial mapping from CT to PD-L1 on the basis of ensuring the effective spatial matching of NSCLC image-pathology, and on this basis, realize the accurate prediction of the efficacy of lung cancer immunotherapy for NSCLC patients.

[0034] (2) The present invention combines the image deformation registration method and the tissue puncture position on the puncture CT image to realize local spatial matching; and divides multiple tumor sub-regions with the same size as the puncture region on the baseline CT, predicts the PD-L1 expression for each sub-region, and can accurately realize the all-volume three-dimensional cross-modal spatial mapping of NSCLC image-pathology; at the same time, the present invention is based on a dual-task prediction model, and combines the clinical information of NSCLC patients, the baseline CT image and the obtained PD-L1 map as the input of the dual-task prediction model to predict the efficacy of lung cancer immunotherapy, which can further ensure the accuracy of the treatment efficacy prediction.

[0035] The advantages of the additional aspects of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The specification drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0037] Figure 1 It is a flowchart of the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images in Embodiment 1 of the present invention.

[0038] Figure 2 It is a flowchart of registering the tissue puncture position in the puncture CT image to the baseline CT image in Embodiment 1 of the present invention.

[0039] Figure 3 It is a flowchart of generating the PD-L1 map in Embodiment 1 of the present invention.

[0040] Figure 4 This is the flowchart for predicting the treatment efficacy based on the dual-task prediction model in the first embodiment of the present invention. Detailed implementation manners

[0041] It should be noted that the following detailed description is exemplary and is intended to provide further illustration of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0042] It should be noted that the terms used herein are only for describing the specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention.

[0043] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0044] The overall idea proposed by the present invention: The present invention provides a method for predicting the efficacy of lung cancer immunotherapy based on dual CT images. First, the baseline CT image and the puncture CT image are registered through an image deformation registration algorithm, and a spatially attentive PD-L1 expression prediction model (i.e., a spatially attentive deep learning model) is constructed by combining the tumor region and the puncture region of the baseline CT image; then, this model is used to estimate the PD-L1 map based on the baseline CT image, and the immunotherapy efficacy of NSCLC patients is comprehensively predicted by combining the baseline CT image and clinical information; finally, potential factors related to the efficacy of NSCLC immunotherapy are analyzed through model visualization means.

[0045] Embodiment 1

[0046] This embodiment discloses a method for predicting the efficacy of lung cancer immunotherapy based on dual CT images.

[0047] As Figure 1 shown, the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images includes:

[0048] Step S1: Obtain the baseline CT image and the puncture CT image of a non-small cell lung cancer patient, and label the tumor regions in the baseline CT image and the puncture CT image respectively;

[0049] Step S2: Register the tissue puncture position in the puncture CT image to the baseline CT image by using an image deformation registration method, and divide the baseline CT image into multiple tumor sub-regions;

[0050] Step S3: Jointly input the entire tumor region of the labeled baseline CT image and each tumor sub-region into the spatial attention deep learning model to predict the PD-L1 expression level at the location of each tumor sub-region; generate a PD-L1 map based on the obtained PD-L1 expression levels.

[0051] Step S4: Construct a dual-task prediction model, using the clinical information of NSCLC patients, the baseline CT image, and the obtained PD-L1 map as the input of the dual-task prediction model to predict the efficacy of lung cancer immunotherapy.

[0052] Based on the above process, the present invention can accurately predict the efficacy of lung cancer immunotherapy for NSCLC patients on the basis of ensuring the effective spatial matching of imaging and pathology of NSCLC. For the convenience of understanding the technical solution of the present invention, the following further explains and illustrates the specific implementation steps in the technical solution of the present invention.

[0053] Step S1: Obtain the baseline CT image and the puncture CT image of NSCLC patients, and label the tumor regions in the baseline CT image and the puncture CT image respectively.

[0054] Step S1-1: Obtain the baseline CT image and the puncture CT image of NSCLC patients.

[0055] It is difficult to achieve three-dimensional matching of imaging and pathology. However, when performing percutaneous biopsy on NSCLC patients in clinical practice, puncture CT images are collected to record the puncture location; and PD-L1 detection is performed on the punctured tissue. Therefore, the present invention intends to collect dual images (baseline CT image and puncture CT image) of NSCLC patients for subsequent three-dimensional matching operations in the imaging-pathology space. Specifically:

[0056] In this embodiment, a total of 1000 NSCLC patients who received PD-L1 detection and immunotherapy in three hospitals are used as the collection objects, and the baseline CT images, puncture CT images, PD-L1 tumor proportion score (TPS), and clinical data of these patients are collected.

[0057] Furthermore, all selected collection subjects must meet the following conditions: ① Age > 18 years old; ② Advanced NSCLC patients, with clinical stage from stage IIIB to stage VIB (refer to the 8th edition of the lung cancer TNM staging standard); ③ Treated with immune checkpoint inhibitors, including patients treated with single-agent immunotherapy, immune combination chemotherapy, and immune combination targeted therapy; ④ Have complete baseline clinical data before immunotherapy; ⑤ Have undergone chest CT examination (baseline CT images) within 1 month before immunotherapy; ⑥ Have pre-treatment puncture CT images; ⑦ The time interval between the acquisition of the baseline CT image and the puncture CT image does not exceed 2 weeks; ⑧ Perform PD-L1 detection based on the puncture pathological tissue before treatment. Collection subjects with the following conditions will not be adopted: ① Patients with a history of more than one primary malignant tumor; ② Patients treated with neoadjuvant or adjuvant immune checkpoint inhibitors before surgery; ③ Patients whose treatment response cannot be evaluated, including those without follow-up data or with inmeasurable primary lesions (lesion long diameter < 10 mm); ④ Poor image quality of any one of the two CT images; ⑤ Patients with irregular follow-up.

[0058] Step S1-2: Mark the tumor regions in the baseline CT image and the puncture CT image.

[0059] The data annotation work is carried out by experienced imaging experts. Specifically, the imaging experts carefully mark the tumor regions in the baseline CT image and the puncture CT image; confirm the relative relationship between the puncture needle and the punctured tissue in the puncture CT image through pathological records, and at the same time evaluate the size of the punctured tissue, and the imaging experts mark the puncture region on the puncture CT image.

[0060] Step S2: Use the image deformation registration method to register the tissue puncture position in the puncture CT image to the baseline CT image, and divide the baseline CT image into multiple tumor sub-regions.

[0061] As Figure 2 shown, use the image deformation registration method based on the tumor region contour to register the baseline CT image and the puncture CT image, and map the puncture region marked on the puncture CT image to the baseline CT image through the deformation vector field of the image during registration. In the actual implementation process, first, resample the puncture CT image to ensure the consistency of the spatial resolution between the puncture CT image and the baseline CT image. Then, the imaging experts compare the shape and anatomical position similarity of the two CT images, and select the best matching layer of the two CT images for subsequent registration. During the registration process, the baseline CT is used as the fixed image, and the puncture CT is used as the floating image. At different rotation angles and deformation degrees extract and compare the circumscribed rectangles and contours of the tumor regions on the two CT images, and for the tumor regions on the two CT images DiceThe coefficient (Days similarity coefficient) is maximized as the optimization condition, and the Dice The rotation angle of the puncture CT image with the largest coefficient And the degree of deformation , to achieve accurate matching of the tumor area. Assume that the baseline CT and puncture CT images are I f and I m The tumor areas of the two were T f and T m , the binary mask images of the tumor areas of the two are M f and M m , then the loss function of the registration algorithm is:

[0062] ;

[0063] in, D for Dice Coefficient; rotation angle The value range is ,by is the interval; the degree of deformation Including front and back direction The degree of deformation and left and right directions The degree of deformation , the value range of both is 0- and 0- , with 1 voxel as interval, and They are and The maximum deformation in the direction is calculated as:

[0064] ;

[0065] in, are image directions, representing and .

[0066] After the registration is completed, based on the optimal rotation angle And the degree of deformation , the puncture area marked on the puncture CT image is mapped to the baseline CT image through the deformation vector field between the puncture CT image and the baseline CT image during registration. Finally, the imaging experts are asked to confirm the registration results and remove the samples with poor registration effects.

[0067] Step S3: Input the entire tumor region of the labeled baseline CT image and each tumor sub-region into the spatial attention deep learning model respectively to predict the PD-L1 expression level at the location of each tumor sub-region; generate a PD-L1 map according to the obtained PD-L1 expression level.

[0068] Step S3-1: Construct a spatial attention deep learning model and perform model training.

[0069] As Figure 3 shown, the spatial attention deep learning model includes a CNN network module and a spatial attention module; among them, the network model used in the CNN network module is the ResNET50 network, which consists of an input layer, a convolutional layer, multiple residual blocks, a global average pooling layer, and a fully connected layer; the spatial attention module consists of two parallel fully connected layers, a cross-attention module, and two serial fully connected layers.

[0070] When performing model training, first, in the way of transfer learning, use the ResNET50 network pre-trained on the ImageNet dataset to extract the deep image features of the entire tumor region on the baseline CT image. This network captures the global anatomical structure characteristics of the entire tumor region through a series of convolutional and pooling operations, forming a deep feature representation of the entire tumor region; among them, the ImageNet dataset is a large image database. Then, use the PyRadiomics tool to calculate the radiomics features of the puncture region in the baseline CT image, including average gray value, entropy value, and texture features, etc. These features reflect the local anatomical structure characteristics of the puncture region; among them, the PyRadiomics tool is an open-source tool based on Python. After feature extraction and calculation, the deep image features of the tumor region and the radiomics features of the puncture region are jointly analyzed. The spatial attention module consists of two parallel fully connected layers, a cross-attention module, and two serial fully connected layers. The two parallel fully connected layers respectively convert and into and , making the two types of features have the same dimension; subsequently, the cross-attention module deeply fuses the two types of features, transforms the puncture position feature into a query matrix Q , transforms the whole tumor feature into key and value matrices K and V , and the attention weights A are:

[0071] ;

[0072] Among them, , and are parameters automatically learned during training; is used to perform scale transformation to prevent the gradient of the Softmax function from vanishing when the numerical value is too large.

[0073] The fused feature is:

[0074] ;

[0075] Among them, is element-wise multiplication, is an adjustment coefficient that can adjust the weights of the two features. This feature fusion method can amplify key features and suppress non-key features. The fused feature is integrated and compressed through two serial fully connected layers to identify important information related to PD-L1 expression, so as to predict the PD-L1 expression level at the puncture position (three classifications: TPS < 1%, 1% ≤ TPS < 50%, TPS ≥ 50%).

[0076] Based on the global features of the tumor, this model strengthens the attention to the features of the CT images in the puncture area, and improves the prediction accuracy of the PD-L1 expression level through spatial emphasis. During the training process, the two feature extraction parts are fixed, and only the parameters of the spatial attention module are optimized to avoid overfitting and improve the generalization of the model.

[0077] Step S3-2: Generate a PD-L1 map according to the obtained PD-L1 expression level.

[0078] After completing the training of the spatial attention deep learning model, a sub-region prediction strategy is adopted, and only the baseline CT image is used to generate the spatial distribution map of PD-L1 expression. Specifically, the tumor region marked on the baseline CT image is divided into several sub-regions with the same size as the puncture area, and the interval between the central points of the sub-regions is 1 voxel. Then, radiomics features are extracted from each sub-region and input into the trained spatial attention deep learning model together with the depth features of the whole tumor region, so as to predict the PD-L1 expression level. Finally, the prediction results of all sub-regions are integrated, and the PD-L1 map of the whole tumor is generated according to their position coordinates on the CT, so as to perform a detailed spatial estimation of the PD-L1 expression situation inside the tumor.

[0079] Step S4: Construct a dual-task prediction model, and use the clinical information of non-small cell lung cancer patients, the baseline CT image, and the obtained PD-L1 map as the input of the dual-task prediction model to predict the efficacy of lung cancer immunotherapy.

[0080] Step S4-1: Construct a dual-task prediction model.

[0081] As Figure 4 shown, the dual-task prediction model consists of two CNN network modules and a Transformer module integrated with an attention mechanism. Among them, the two CNN network modules have the same structure and also adopt the ResNET50 network. Further, the two CNN network modules are used to extract deep features to quantify the texture distribution patterns of the baseline CT images and PD-L1 maps. Specifically, in the two CNN network modules, one is used to extract the deep features of the PD-L1 map, and the other is used to extract the deep features of the baseline CT images. The Transformer module integrated with the attention mechanism is used to learn the interaction relationships between different modalities, and integrate and analyze the two obtained deep features and the clinical information of NSCLC patients for accurate prediction of immunotherapy efficacy. In addition, before sending the clinical information of NSCLC patients into the dual-task prediction model, the clinical information needs to be digitally encoded. Continuous clinical information such as age, height, weight, and test results is encoded with continuous numerical values, and categorical clinical information such as gender, tumor grade, and pathological type is encoded with categorical numerical values (1, 2,...) for easy reading by the Transformer module. Specifically:

[0082] Step S4-2: Predict the efficacy of lung cancer immunotherapy.

[0083] First, two pre-trained ResNET50 networks are used to extract the deep features of the baseline CT images of the tumor region and the PD-L1 expression distribution map respectively, in order to quantify the anatomical structure of the tumor and the spatial distribution of PD-L1 expression. Then, the attention mechanism in the Transformer module is adopted to analyze the interaction relationships between different modalities. The attention mechanism of the Transformer model can effectively integrate and correlate clinical information and the multi-modal features extracted by the ResNET50 network, and deeply analyze the complex relationships among clinical data, imaging features, and PD-L1 distribution. The combination of the ResNET50 network and the Transformer can not only enhance the comprehensive analysis ability of the model, but also improve the prediction accuracy of immunotherapy response. Finally, based on the sharing of underlying feature extraction, this dual-task prediction model realizes the synchronous prediction of overall survival (OS) and progression-free survival (PFS) through the output layers specific to the dual tasks, so as to output the predicted values of OS and PFS, and further predict the efficacy of lung cancer immunotherapy; among them, OS and PFS are the evaluation criteria for the efficacy of lung cancer immunotherapy in this embodiment. OS represents the time from the start of immunotherapy to death of lung cancer patients, and PFS represents the time from the start of immunotherapy to tumor progression or death of lung cancer patients. The output layers of OS and PFS respectively output the logical risk scores of the two. In each output layer, based on the logical risk scores at different time intervals of each task, the survival probability distribution function of each sample is calculated using the survival function of the Cox proportional hazards model. p The difference between the calculated survival probability distribution of the output and the true survival situation is calculated through the negative log-likelihood loss function of the discrete-time Cox model, and the calculation method of the loss function is: p The difference between the calculated survival probability distribution of the output and the true survival situation is calculated through the negative log-likelihood loss function of the discrete-time Cox model, and the calculation method of the loss function is:

[0084] ;

[0085] where i is the patient index; j is the time interval index; represents the survival probability of the individual i in the j th year output by the model; represents the number of individuals who actually died / progressed in the j th year; represents the number of individuals who were actually at risk at the start of the j th year, that is, the number of individuals who did not experience an event and were not lost to follow-up before the start of the j th year.

[0086] The combined loss function for the OS and PFS dual tasks is:

[0087] ;

[0088] Among them, and are the loss functions of OS and PFS respectively, and are hyperparameters used to adjust the relative importance of the losses of the two tasks.

[0089] Using the joint loss function of the two tasks to optimize and update the model parameters can effectively distinguish the survival risks of different patients and synchronously improve the prediction performance of the dual tasks.

[0090] Furthermore, model visualization techniques such as gradient-weighted class activation mapping (Grad-CAM) can also be used to analyze the key tumor regions related to the efficacy of immunotherapy and the estimated values of PD-L1 expression; among them, Grad-CAM can visualize the tumor regions that the model focuses on during prediction, revealing the potential relationship between imaging features and PD-L1 expression distribution and the effect of immunotherapy. This kind of visual analysis can help explain the decision-making process of the model, discover potential risk factors related to the efficacy of immunotherapy, and provide intuitive evidence for understanding the prediction results.

[0091] In summary, the present invention provides a method for predicting the efficacy of lung cancer immunotherapy based on dual CT images. By registering the baseline CT image and the puncture CT image, the problem of spatial correspondence between imaging and pathological data is solved. Moreover, a spatial attention deep learning model is constructed to learn the mapping relationship between the features of the baseline CT image and the PD-L1 expression level. By combining global features and local features, the prediction accuracy of the local PD-L1 expression level is improved. After training, the spatial attention deep learning model can predict the PD-L1 expression level for the tumor sub-regions of each baseline CT image, thereby estimating the spatial distribution of the PD-L1 expression in the tumor region, realizing a more detailed tumor feature mapping, and providing a reference for pre-treatment risk assessment. Therefore, the present invention provides a new method that is low-cost and easy to implement for the spatial matching of cross-modal data and the evaluation of the overall molecular expression distribution of tumors. In addition, the present invention first introduces the PD-L1 atlas in the efficacy prediction task, combines it with clinical information and baseline CT images to jointly predict the efficacy of immunotherapy, which not only enriches the dimensions of efficacy prediction, but also provides more accurate support for individualized treatment strategies. At the same time, by exploring the spatial positions related to the efficacy, the relationship between the PD-L1 distribution and the efficacy of immunotherapy can be analyzed, thereby guiding the selection of the puncture position and the treatment strategy, revealing potential risk factors related to immunotherapy, and comprehensively improving the prognosis of patients. Therefore, the present invention also provides a new idea for exploring immune-related risk factors in NSCLC and a new perspective for image analysis and efficacy prediction.

[0092] Example Two

[0093] This example discloses a system for predicting the efficacy of lung cancer immunotherapy based on dual CT images.

[0094] A system for predicting the efficacy of lung cancer immunotherapy based on dual CT images includes:

[0095] An image annotation module, configured to: obtain the baseline CT image and the puncture CT image of a non-small cell lung cancer patient, and respectively annotate the tumor regions in the baseline CT image and the puncture CT image;

[0096] An imaging-pathology spatial matching module, configured to: register the tissue puncture position in the puncture CT image to the baseline CT image by using an image deformation registration method, and divide the baseline CT image into multiple tumor sub-regions;

[0097] A PD-L1 atlas construction module, configured to: jointly input the entire tumor region of the annotated baseline CT image and each tumor sub-region into a spatial attention deep learning model to predict the PD-L1 expression level at the position of each tumor sub-region; generate a PD-L1 atlas according to the obtained PD-L1 expression level;

[0098] The efficacy prediction module is configured to: construct a dual-task prediction model, use the clinical information of non-small cell lung cancer patients, baseline CT images, and the obtained PD-L1 atlas as the input of the dual-task prediction model, and predict the efficacy of lung cancer immunotherapy.

[0099] Example 3

[0100] The purpose of this example is to provide a computer-readable storage medium.

[0101] A computer-readable storage medium stores a computer program, and when the program is executed by a processor, it implements the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in Example 1 of the present disclosure.

[0102] Example 4

[0103] The purpose of this example is to provide an electronic device.

[0104] An electronic device includes a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in Example 1 of the present disclosure.

[0105] The steps involved in the devices in the above Examples 2, 3, and 4 correspond to those in Method Example 1. For specific implementation manners, reference may be made to the relevant description part of Example 1. The term "computer-readable storage medium" should be understood to include a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0106] Those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0107] Although the specific implementation manners of the present invention are described above in conjunction with the accompanying drawings, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions of the present invention, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present invention.

Claims

1. A method for predicting the efficacy of lung cancer immunotherapy based on dual CT images, characterized in that: include: Obtain baseline CT images and puncture CT images of patients with non-small cell lung cancer, and annotate the tumor areas in the baseline CT images and the puncture CT images respectively; Using an image deformation registration method to register the tissue puncture position in the puncture CT image to the baseline CT image, and dividing the baseline CT image into multiple tumor sub-areas; The entire tumor region of the annotated baseline CT image and each tumor sub-region are respectively input into the spatial attention deep learning model to predict the PD-L1 expression level at the location of each tumor sub-region; and a PD-L1 atlas is generated based on the obtained PD-L1 expression level; A dual-task prediction model was constructed, and the clinical information, baseline CT images and obtained PD-L1 maps of patients with non-small cell lung cancer were used as the input of the dual-task prediction model to predict the efficacy of immunotherapy for lung cancer.

2. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 1, characterized in that: The baseline CT image is divided into a plurality of tumor sub-regions. Specifically, the tumor region marked on the baseline CT image is divided into a plurality of sub-regions having the same size as the puncture region.

3. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 1, characterized in that: The spatial attention deep learning model includes a CNN network module and a spatial attention module; wherein the spatial attention module consists of two parallel fully connected layers, a cross attention module and two serial fully connected layers.

4. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 3, characterized in that: Based on the CNN network module, the deep imaging features of the entire tumor area on the baseline CT image are extracted, and the imaging features of the puncture position on the baseline CT image are calculated; based on the spatial attention module, the obtained deep imaging features and imaging features are jointly analyzed to predict the PD-L1 expression level at each tumor sub-region location.

5. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 1, characterized in that: The dual-task prediction model consists of two CNN network modules and a Transformer module with an integrated attention mechanism.

6. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 5, characterized in that: The two CNN network modules are used to extract deep features. Specifically, one of the two CNN network modules is used to extract deep features of the PD-L1 map, and the other is used to extract deep features of the baseline CT image. The Transformer module is used to integrate and analyze the two obtained deep features and the clinical information of patients with non-small cell lung cancer.

7. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 1, characterized in that: Before feeding the clinical information of non-small cell lung cancer patients into the dual-task prediction model, the clinical information is digitally encoded.

8. A lung cancer immunotherapy efficacy prediction system based on dual CT images, characterized in that: include: The image annotation module is configured to: obtain a baseline CT image and a puncture CT image of a patient with non-small cell lung cancer, and respectively annotate tumor regions in the baseline CT image and the puncture CT image; The image pathology space matching module is configured to: register the tissue puncture position in the puncture CT image to the baseline CT image using an image deformation registration method, and divide the baseline CT image into multiple tumor sub-regions; The PD-L1 atlas construction module is configured to: input the entire tumor region of the annotated baseline CT image and each tumor sub-region into the spatial attention deep learning model to predict the PD-L1 expression level at the location of each tumor sub-region; generate a PD-L1 atlas based on the obtained PD-L1 expression level; The efficacy prediction module is configured to: construct a dual-task prediction model, use the clinical information, baseline CT images and obtained PD-L1 map of patients with non-small cell lung cancer as the input of the dual-task prediction model, and predict the efficacy of lung cancer immunotherapy.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in any one of claims 1 to 7 are implemented.

Images