Lung cancer immunotherapy curative effect prediction method and system based on double CT images

Abstract

The invention provides a lung cancer immunotherapy curative effect prediction method and system based on double CT images, and belongs to the technical field of medical image processing. Firstly, tumor areas of a baseline CT image and a puncture CT image are marked; registering a puncture position in the puncture CT image to a baseline CT image, and dividing the baseline CT image into a plurality of tumor sub-regions; predicting the PD-L1 expression quantity at each tumor sub-region position based on a spatial attention deep learning model, and generating a PD-L1 map; and constructing a double-task prediction model, inputting the clinical information, the baseline CT image and the PD-L1 atlas of the patient into the double-task prediction model, and predicting the curative effect of lung cancer immunotherapy. According to the invention, on the basis of ensuring the effective matching of the image-pathological space of the non-small cell lung cancer, the accurate prediction of the lung cancer immunotherapy curative effect of the non-small cell lung cancer patient can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a method and system for predicting the efficacy of lung cancer immunotherapy based on dual CT images. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Lung cancer is the cancer with the highest morbidity and mortality worldwide; among them, non-small cell lung cancer (NSCLC) is the main type of lung cancer. About 75% of NSCLC patients are in the advanced stage when first diagnosed. Radiotherapy and chemotherapy are the main treatments for patients with advanced NSCLC, but the efficacy is poor. The five-year survival rate was less than 25.4% between 2013 and 2019. Improving long-term survival rate is the key issue in the treatment of advanced NSCLC.

[0004] Immunotherapy based on programmed death-ligand 1 (PD-L1) has made breakthrough progress in the treatment of NSCLC and can effectively prolong the survival of patients with advanced NSCLC. However, due to the heterogeneity of tumors and the complexity of the tumor immune microenvironment, the overall efficacy of immunotherapy is low, and the objective response rate of PD-L1 inhibitors in the advanced NSCLC population is only about 20%. For patients who cannot benefit, there is a risk of serious adverse reactions due to treatment-related toxicity and missing the best treatment opportunity. Therefore, predicting the efficacy of immunotherapy and screening potential beneficiaries before treatment are crucial for treatment decisions for patients with advanced NSCLC.

[0005] Clinically, the expression level of PD-L1 is used as a biomarker to predict the effect of immunotherapy for NSCLC, and high PD-L1 expression is often associated with better immunotherapy response. However, the performance of PD-L1-based NSCLC immunotherapy efficacy prediction is not ideal, and some patients with low or even negative PD-L1 expression can also benefit from immunotherapy. The spatial heterogeneity of the tumor is one of its most important influencing factors. PD-L1 is mainly detected by immunohistochemistry using specific antibodies on pathological tissues obtained by puncture biopsy; however, biopsy can only obtain a small area of ​​local tumor tissue, and the test results only represent the PD-L1 expression level of the local tumor, making it difficult to evaluate the overall PD-L1 expression of the tumor, which may lead to an underestimated PD-L1 expression in some patients. Therefore, a comprehensive assessment of the overall PD-L1 expression of the tumor is a key issue in improving the accuracy of predicting the efficacy of NSCLC immunotherapy.

[0006] Recent studies have shown that CT images combined with artificial intelligence algorithms can also predict the efficacy of immunotherapy for advanced NSCLC. These studies generally use a single CT image as the analysis object and directly use convolutional neural network algorithms to predict PD-L1 expression and / or immunotherapy efficacy. Existing methods still have some technical problems in dealing with the relationship between CT, PD-L1 expression, and the efficacy of immunotherapy for advanced NSCLC, such as: (1) Studies that use PD-L1 expression as model input and immunotherapy efficacy of advanced NSCLC as a prediction target can only predict the individualized treatment efficacy of patients based on the local PD-L1 expression in the tumor obtained by puncture biopsy, and lack an assessment of the spatial distribution of global PD-L1 expression in the tumor.

[0007] (2) Studies that use CT images as model input and PD-L1 expression as a prediction target only predict local PD-L1 expression based on the global imaging features of the tumor, without matching the spatial location of the image with the pathology, and lack attention and emphasis on the local CT image features at the PD-L1 detection location. Therefore, this method is not ideal for predicting the efficacy of immunotherapy for NSCLC.

[0008] (3) Studies that used CT images or CT combined with PD-L1 expression as model input and the efficacy of immunotherapy for advanced NSCLC as the prediction target were unable to perform cross-modal spatial mapping of CT images and PD-L1, and the association between CT features and the efficacy of immunotherapy for advanced NSCLC lacked explainability. Summary of the invention

[0009] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a method and system for predicting the efficacy of lung cancer immunotherapy based on dual CT images, which can accurately predict the efficacy of lung cancer immunotherapy for patients with non-small cell lung cancer while ensuring effective matching and mapping of the image-pathology space of non-small cell lung cancer.

[0010] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of the present invention provides a method for predicting the efficacy of lung cancer immunotherapy based on dual CT images.

[0011] The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images includes: Obtain baseline CT images and puncture CT images of patients with non-small cell lung cancer, and annotate the tumor areas in the baseline CT images and the puncture CT images respectively; Using an image deformation registration method to register the tissue puncture position in the puncture CT image to the baseline CT image, and dividing the baseline CT image into multiple tumor sub-areas; The entire tumor region of the annotated baseline CT image and each tumor sub-region are respectively input into the spatial attention deep learning model to predict the PD-L1 expression level at the location of each tumor sub-region; and a PD-L1 atlas is generated based on the obtained PD-L1 expression level; A dual-task prediction model was constructed, and the clinical information, baseline CT images and obtained PD-L1 maps of patients with non-small cell lung cancer were used as the input of the dual-task prediction model to predict the efficacy of immunotherapy for lung cancer.

[0012] Furthermore, the baseline CT image is divided into a plurality of tumor sub-regions. Specifically, the tumor region marked on the baseline CT image is divided into a plurality of sub-regions having the same size as the puncture region.

[0013] Furthermore, the spatial attention deep learning model includes a CNN network module and a spatial attention module; wherein the spatial attention module consists of two parallel fully connected layers, a cross attention module and two serial fully connected layers.

[0014] Furthermore, based on the CNN network module, the deep imaging features of the entire tumor area on the baseline CT image are extracted, and the imaging features of the puncture position on the baseline CT image are calculated; based on the spatial attention module, the obtained deep imaging features and imaging features are jointly analyzed to construct and optimize the PD-L1 prediction model to predict the PD-L1 expression level at each tumor sub-region location.

[0015] Furthermore, the dual-task prediction model consists of two CNN network modules and a Transformer module integrated with an attention mechanism.

[0016] Furthermore, the two CNN network modules are used to extract deep features. Specifically, one of the two CNN network modules is used to extract deep features of the PD-L1 map, and the other is used to extract deep features of the baseline CT image; the Transformer module is used to integrate and analyze the two obtained deep features and the clinical information of patients with non-small cell lung cancer.

[0017] Furthermore, before the clinical information of the non-small cell lung cancer patients is fed into the dual-task prediction model, the clinical information is digitally encoded.

[0018] The second aspect of the present invention provides a lung cancer immunotherapy efficacy prediction system based on dual CT images.

[0019] The lung cancer immunotherapy efficacy prediction system based on dual CT images includes: The image annotation module is configured to: obtain a baseline CT image and a puncture CT image of a patient with non-small cell lung cancer, and respectively annotate tumor regions in the baseline CT image and the puncture CT image; The image pathology space matching module is configured to: register the tissue puncture position in the puncture CT image to the baseline CT image using an image deformation registration method, and divide the baseline CT image into multiple tumor sub-regions; The PD-L1 atlas construction module is configured to: input the entire tumor region of the annotated baseline CT image and each tumor sub-region into the spatial attention deep learning model to predict the PD-L1 expression level at the location of each tumor sub-region; generate a PD-L1 atlas based on the obtained PD-L1 expression level; The efficacy prediction module is configured to: construct a dual-task prediction model, use the clinical information, baseline CT images and obtained PD-L1 map of patients with non-small cell lung cancer as the input of the dual-task prediction model, and predict the efficacy of lung cancer immunotherapy. The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in the first aspect of the present invention.

[0020] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in the first aspect of the present invention are implemented.

[0021] One or more of the above technical solutions have the following beneficial effects: (1) The present invention first obtains dual CT images (baseline CT images and puncture CT images) of NSCLC patients, and annotates the tumor areas in the dual CT images respectively; then, the image deformation registration method is used to register the tissue puncture position in the puncture CT image to the baseline CT image, so as to achieve local spatial matching of the CT image and the PD-L1 pathology, and divide the baseline CT image into multiple tumor sub-regions. On this basis, the entire tumor area and the puncture position of the annotated baseline CT image are jointly input into the spatial attention deep learning model to predict the PD-L1 expression, instead of predicting the local PD-L1 expression based only on the global image features of the tumor. Based on this model, the PD-L1 expression is evaluated for each tumor sub-region respectively, thereby realizing the evaluation of the spatial distribution of the global PD-L1 expression of the tumor. Therefore, the present invention can realize the cross-modal spatial mapping from CT to PD-L1 on the basis of ensuring the effective matching of the NSCLC image-pathology space, and on this basis, realize the accurate prediction of the efficacy of lung cancer immunotherapy for NSCLC patients.

[0022] (2) The present invention combines the image deformation registration method and the tissue puncture position on the puncture CT image to achieve local spatial matching; and divides the baseline CT into multiple tumor sub-regions with the same size as the puncture area, and predicts the PD-L1 expression for each sub-region, which can accurately achieve full-volume three-dimensional cross-modal spatial mapping of NSCLC image-pathology; at the same time, based on the dual-task prediction model, the present invention combines the clinical information of non-small cell lung cancer patients, baseline CT images and the obtained PD-L1 map as the input of the dual-task prediction model to predict the efficacy of lung cancer immunotherapy, which can provide further guarantee for the accuracy of the prediction of treatment efficacy.

[0023] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is a flow chart of a method for predicting the efficacy of lung cancer immunotherapy based on dual CT images in Example 1 of the present invention.

[0026] Figure 2 The present invention is a flowchart of registering the tissue puncture position in the puncture CT image to the baseline CT image in the first embodiment of the present invention.

[0027] Figure 3 This is a flow chart of generating a PD-L1 map in Example 1 of the present invention.

[0028] Figure 4 This is a flow chart of predicting treatment efficacy based on a dual-task prediction model in Example 1 of the present invention. DETAILED DESCRIPTION

[0029] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0030] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0031] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0032] The overall idea proposed by the present invention is as follows: The present invention provides a method for predicting the efficacy of lung cancer immunotherapy based on dual CT images. First, the baseline CT image and the puncture CT image are registered by an image deformation registration algorithm, and a spatially focused PD-L1 expression prediction model (i.e., a spatially focused deep learning model) is constructed in combination with the tumor area and the puncture area of ​​the baseline CT image; then, the model is used to estimate the PD-L1 atlas based on the baseline CT image, and the baseline CT image and clinical information are combined to comprehensively predict the efficacy of immunotherapy for NSCLC patients; finally, potential factors related to the efficacy of NSCLC immunotherapy are analyzed by model visualization.

[0033] Embodiment 1 This embodiment discloses a method for predicting the efficacy of lung cancer immunotherapy based on dual CT images.

[0034] like Figure 1 As shown, the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images includes: Step S1, obtaining a baseline CT image and a puncture CT image of a patient with non-small cell lung cancer, and marking tumor areas in the baseline CT image and the puncture CT image respectively; Step S2, registering the tissue puncture position in the puncture CT image to the baseline CT image using an image deformation registration method, and dividing the baseline CT image into multiple tumor sub-regions; Step S3, the entire tumor region of the annotated baseline CT image and each tumor sub-region are respectively input into the spatial attention deep learning model to predict the PD-L1 expression level at the position of each tumor sub-region; and a PD-L1 atlas is generated according to the obtained PD-L1 expression level; Step S4: construct a dual-task prediction model, using the clinical information, baseline CT images and obtained PD-L1 map of patients with non-small cell lung cancer as input of the dual-task prediction model to predict the efficacy of lung cancer immunotherapy.

[0035] Based on the above process, the present invention can achieve accurate prediction of the efficacy of lung cancer immunotherapy for patients with non-small cell lung cancer on the basis of ensuring effective matching of the image-pathology space of non-small cell lung cancer. To facilitate the understanding of the technical solution of the present invention, the specific implementation steps in the technical solution of the present invention are further explained and illustrated below.

[0036] Step S1, obtaining a baseline CT image and a puncture CT image of a NSCLC patient, and marking tumor areas in the baseline CT image and the puncture CT image respectively.

[0037] Step S1-1, obtaining a baseline CT image and a puncture CT image of a NSCLC patient.

[0038] The three-dimensional matching of images and pathology is difficult to achieve, but in clinical practice, when NSCLC patients undergo puncture biopsy, puncture CT images are collected to record the puncture location; and PD-L1 detection is performed on the punctured tissue. Therefore, the present invention intends to collect dual images (baseline CT images and puncture CT images) of NSCLC patients for subsequent three-dimensional matching operations of image-pathology space, specifically: In this example, a total of 1,000 NSCLC patients who received PD-L1 testing and immunotherapy in three hospitals were selected as the subjects for collection, and their baseline CT images, puncture CT images, PD-L1 tumor proportion scores (TPS) and clinical data were collected.

[0039] Furthermore, the selected subjects need to meet the following conditions: ① Age > 18 years old; ② Advanced NSCLC patients, clinical stage IIIB to VIB (refer to the 8th edition of the TNM staging standard for lung cancer); ③ Patients treated with immune checkpoint inhibitors, including immune monotherapy, immune combined chemotherapy, and immune combined targeted therapy; ④ Complete baseline clinical data before immunotherapy; ⑤ Chest CT examination within 1 month before immunotherapy (baseline CT image); ⑥ Pre-treatment puncture CT image; ⑦ The interval between the baseline CT image and the puncture CT image is no more than 2 weeks; ⑧ PD-L1 detection based on puncture pathological tissue before treatment. The selected subjects with the following conditions will not be used: ① Patients with a history of more than one primary malignant tumor; ② Patients treated with preoperative neoadjuvant or postoperative adjuvant immune checkpoint inhibitors; ③ Unable to evaluate the treatment response, including patients with no follow-up data or unmeasurable primary lesions (lesion long diameter < 10 mm); ④ Poor image quality of any CT image in the dual CT images; ⑤ Patients with irregular follow-up.

[0040] Step S1-2: marking the tumor area in the baseline CT image and the puncture CT image.

[0041] The data annotation work is carried out by experienced imaging experts. Specifically, the imaging experts annotate the tumor areas in the baseline CT images and puncture CT images in detail; confirm the relative relationship between the puncture needle and the punctured tissue in the puncture CT images through pathological records, evaluate the size of the punctured tissue, and annotate the puncture area on the puncture CT images by the imaging experts.

[0042] Step S2: using an image deformation registration method to register the tissue puncture position in the puncture CT image to the baseline CT image, and dividing the baseline CT image into multiple tumor sub-regions.

[0043] like Figure 2As shown, the baseline CT image and the puncture CT image are registered using an image deformation registration method based on the contour of the tumor area, and the puncture area marked on the puncture CT image is mapped to the baseline CT image through the deformation vector field of the image during registration. In the actual implementation process, first, the puncture CT image is resampled to ensure the consistency of the spatial resolution between the puncture CT image and the baseline CT image. Then, imaging experts compare the shapes and anatomical position similarities of the two CT images, and select the best matching layers of the two CT images for subsequent registration. During the registration process, the baseline CT is used as a fixed image and the puncture CT is used as a floating image at different rotation angles. and degree of deformation The bounding rectangles and contours of the tumor regions on the two CT images are extracted and compared to obtain the bounding rectangles and contours of the tumor regions on the two CT images. Dice The coefficient (Days similarity coefficient) is maximized as the optimization condition, and the Dice The rotation angle of the puncture CT image with the largest coefficient And the degree of deformation , to achieve accurate matching of the tumor area. Assume that the baseline CT and puncture CT images are I f and I m The tumor areas of the two were T f and T m , the binary mask images of the tumor areas of the two are M f and M m , then the loss function of the registration algorithm is: ; in, D for Dice Coefficient; rotation angle The value range is ,by is the interval; the degree of deformation Including front and back direction The degree of deformation and left and right directions The degree of deformation , the value range of both is 0- and 0- , with 1 voxel as interval, and They are and The maximum deformation in the direction is calculated as: ; in, are image directions, representing and .

[0044] After the registration is completed, based on the optimal rotation angle And the degree of deformation , the puncture area marked on the puncture CT image is mapped to the baseline CT image through the deformation vector field between the puncture CT image and the baseline CT image during registration. Finally, the imaging experts are asked to confirm the registration results and remove the samples with poor registration effects.

[0045] Step S3, the entire tumor region of the annotated baseline CT image and each tumor sub-region are jointly input into the spatial attention deep learning model to predict the PD-L1 expression level at the position of each tumor sub-region; and a PD-L1 atlas is generated based on the obtained PD-L1 expression level.

[0046] Step S3-1: Build a spatial attention deep learning model and perform model training.

[0047] like Figure 3 As shown in the figure, the spatial attention deep learning model includes a CNN network module and a spatial attention module; the network model used in the CNN network module is the ResNET50 network, which consists of an input layer, a convolutional layer, multiple residual blocks, a global average pooling layer and a fully connected layer; the spatial attention module consists of two parallel fully connected layers, a cross attention module and two serial fully connected layers.

[0048] When training the model, first, the transfer learning method is used to use the ResNET50 network pre-trained on the ImageNet dataset to extract the deep imaging features of the entire tumor area on the baseline CT image. The network captures the global anatomical structural characteristics of the entire tumor area through a series of convolution and pooling operations, forming a deep feature representation of the entire tumor area. Among them, the ImageNet dataset is a large image database. Next, the PyRadiomics tool is used to calculate the imaging features of the puncture area in the baseline CT image, including the average grayscale value, entropy value, and texture features. These features reflect the local anatomical structural characteristics of the puncture area. Among them, the PyRadiomics tool is an open source tool based on Python. After feature extraction and calculation, the deep imaging features of the tumor area are transferred through the spatial attention module. Radiomics characteristics of the puncture area The spatial attention module consists of two parallel fully connected layers, a cross attention module, and two serial fully connected layers. The two parallel fully connected layers respectively and Convert to and , so that the two features have the same dimension; then the cross attention module deeply fuses the two features and integrates the puncture position feature Transformed into query matrix Q , the whole tumor characteristics Transform into a matrix of keys and values K and V , the attention weights of the two A for: ; in, , and are parameters learned automatically during training; Used for Perform a scale transformation to prevent When the value is too large, the gradient of the Softmax function disappears.

[0049] The fused features are: ; in, is element-wise multiplication, is the adjustment coefficient, which can adjust the weights of the two features. This feature fusion method can amplify key features and suppress non-key features. The fused features Information integration and compression were performed through two serial fully connected layers to identify important information related to PD-L1 expression to predict the PD-L1 expression level at the puncture site (three categories: TPS < 1%, 1% ≤ TPS < 50%, TPS ≥ 50%).

[0050] Based on the global features of the tumor, the model strengthens the focus on the CT image features of the puncture area, and improves the prediction accuracy of the PD-L1 expression level by spatial emphasis. During the training process, the two feature extraction parts are fixed, and only the parameters of the spatial attention module are optimized to avoid overfitting and improve the generalization of the model.

[0051] Step S3-2: Generate a PD-L1 map based on the obtained PD-L1 expression level.

[0052] After completing the training of the spatial attention deep learning model, a sub-region prediction strategy is adopted to generate a spatial distribution map of PD-L1 expression using only the baseline CT image. Specifically, the tumor area marked on the baseline CT image is divided into several sub-regions of the same size as the puncture area, and the interval between the center points of the sub-regions is 1 voxel. Then, the imaging genomics features are extracted from each sub-region and input into the trained spatial attention deep learning model together with the deep features of the overall tumor area to predict the PD-L1 expression level. Finally, the prediction results of all sub-regions are combined and combined according to their position coordinates on CT to generate a PD-L1 map of the entire tumor, thereby performing a detailed spatial estimation of the PD-L1 expression inside the tumor.

[0053] Step S4: construct a dual-task prediction model, using the clinical information, baseline CT images and obtained PD-L1 map of patients with non-small cell lung cancer as input of the dual-task prediction model to predict the efficacy of lung cancer immunotherapy.

[0054] Step S4-1, constructing a dual-task prediction model.

[0055] like Figure 4 As shown in the figure, the dual-task prediction model consists of two CNN network modules and a Transformer module integrated with an attention mechanism; the two CNN network modules have the same structure and also use the ResNET50 network. Furthermore, the two CNN network modules are used to extract deep features to quantify the texture distribution pattern of the baseline CT image and the PD-L1 atlas. Specifically: one of the two CNN network modules is used to extract the deep features of the PD-L1 atlas, and the other is used to extract the deep features of the baseline CT image; the Transformer module integrated with the attention mechanism is used to learn the interaction between different modalities, and integrate and analyze the obtained two deep features and the clinical information of NSCLC patients for accurate prediction of the efficacy of immunotherapy. In addition, before the clinical information of NSCLC patients is sent to the dual-task prediction model, the clinical information needs to be digitally encoded, and continuous clinical information such as age, height, weight and test results are encoded with continuous values, and categorical clinical information such as gender, tumor grade, pathological type, etc. is encoded with categorical values ​​(1, 2...) to facilitate the reading of the Transformer module. Specifically: Step S4-2: Prediction of the efficacy of immunotherapy for lung cancer.

[0056] First, two pre-trained ResNET50 networks were used to extract deep features of the baseline CT images and PD-L1 expression distribution maps of the tumor area, respectively, to quantify the anatomical structure of the tumor and the spatial distribution of PD-L1 expression. Then, the attention mechanism in the Transformer module was used to analyze the interaction between different modalities. The attention mechanism of the Transformer model can effectively integrate and associate clinical information with the multimodal features extracted by the ResNET50 network, and deeply analyze the complex relationship between clinical data, imaging features, and PD-L1 distribution. The combination of the ResNET50 network and the Transformer can not only enhance the comprehensive analysis capabilities of the model, but also improve the prediction accuracy of immunotherapy response. Finally, based on the shared underlying feature extraction, the dual-task prediction model realizes the simultaneous prediction of OS and progression-free survival (PFS) through the dual-task specific output layer to output the predicted values ​​of OS and PFS, and then predict the efficacy of lung cancer immunotherapy; wherein, OS and PFS are the evaluation criteria for the efficacy of lung cancer immunotherapy in this embodiment, OS represents the time from the start of immunotherapy to death of lung cancer patients, and PFS represents the time from the start of immunotherapy to tumor progression or death of lung cancer patients. The output layers of OS and PFS output the logical risk scores of the two respectively. In each output layer, based on the logical risk scores of different time intervals of each task, the survival probability distribution function of each sample is calculated using the survival function of the Cox proportional hazard model. p The output survival probability distribution is calculated by the negative log-likelihood loss function of the discrete-time Cox model p The difference between the predicted survival and the actual survival, the loss function is calculated as: ; in, i indexing patients; j Index for time intervals; Individuals representing the output of the model i In the j Probability of survival within a year; Indicated in j the number of individuals who actually died / progressed during the year; Indicated in j The number of individuals who are actually in the risk set at the beginning of the year, i.e., those j The number of individuals who did not have an event and were not lost to follow-up before the beginning of the year.

[0057] The joint loss function of OS and PFS dual tasks is: ; in, and are the loss functions for OS and PFS respectively, and is a hyperparameter used to adjust the relative importance of the two task losses.

[0058] Use a joint loss function for both tasks To optimize and update the model parameters, the survival risks of different patients can be effectively distinguished and the dual-task prediction performance can be improved simultaneously.

[0059] Furthermore, model visualization techniques such as gradient-weighted class activation mapping (Grad-CAM) can be used to analyze key tumor regions and estimated values ​​of PD-L1 expression related to the efficacy of immunotherapy; among them, Grad-CAM can visualize the tumor regions that the model focuses on when making predictions, revealing the potential relationship between imaging features and PD-L1 expression distribution and immunotherapy effects. This visual analysis can help explain the decision-making process of the model, discover potential risk factors related to the efficacy of immunotherapy, and provide intuitive evidence for understanding the prediction results.

[0060] In summary, the present invention proposes a method for predicting the efficacy of lung cancer immunotherapy based on dual CT images, which solves the problem of spatial correspondence between images and pathological data by registering baseline CT images with puncture CT images. In addition, a spatial attention deep learning model is constructed to learn the mapping relationship between baseline CT image features and PD-L1 expression levels, and the prediction accuracy of local PD-L1 expression levels is improved by combining global features with local features; after training, the spatial attention deep learning model can predict the PD-L1 expression level of each tumor sub-region of the baseline CT image, thereby estimating the spatial distribution of PD-L1 expression in the tumor region, achieving more detailed tumor feature mapping, and providing a reference for pre-treatment risk assessment. Therefore, the present invention provides a low-cost and easy-to-implement new method for spatial matching of cross-modal data and evaluation of the overall molecular expression distribution of tumors. In addition, the present invention introduces the PD-L1 atlas in the efficacy prediction task for the first time, and combines it with clinical information and baseline CT images to jointly predict the efficacy of immunotherapy, which not only enriches the dimension of efficacy prediction, but also provides more accurate support for individualized treatment strategies. At the same time, by exploring the spatial location related to the efficacy, the relationship between PD-L1 distribution and the efficacy of immunotherapy can be analyzed, thereby guiding the selection of puncture locations and treatment strategies, revealing potential risk factors related to immunotherapy, and comprehensively improving patient prognosis. Therefore, the present invention also provides a new idea for the exploration of immune-related risk factors for NSCLC and a new perspective for image analysis and efficacy prediction.

[0061] Embodiment 2 This embodiment discloses a lung cancer immunotherapy efficacy prediction system based on dual CT images.

[0062] The lung cancer immunotherapy efficacy prediction system based on dual CT images includes: The image annotation module is configured to: obtain a baseline CT image and a puncture CT image of a patient with non-small cell lung cancer, and respectively annotate tumor regions in the baseline CT image and the puncture CT image; The image pathology space matching module is configured to: register the tissue puncture position in the puncture CT image to the baseline CT image using an image deformation registration method, and divide the baseline CT image into multiple tumor sub-regions; The PD-L1 atlas construction module is configured to: input the entire tumor region of the annotated baseline CT image and each tumor sub-region into the spatial attention deep learning model to predict the PD-L1 expression level at the location of each tumor sub-region; generate a PD-L1 atlas based on the obtained PD-L1 expression level; The efficacy prediction module is configured to: construct a dual-task prediction model, use the clinical information, baseline CT images and obtained PD-L1 map of patients with non-small cell lung cancer as the input of the dual-task prediction model, and predict the efficacy of lung cancer immunotherapy. Embodiment 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0063] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in the first embodiment of the present disclosure.

[0064] Embodiment 4 The purpose of this embodiment is to provide an electronic device.

[0065] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in the first embodiment of the present disclosure are implemented.

[0066] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0067] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0068] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for predicting the efficacy of lung cancer immunotherapy based on dual CT images, characterized in that: include: Obtain baseline CT images and puncture CT images of patients with non-small cell lung cancer, and annotate the tumor areas in the baseline CT images and the puncture CT images respectively; Using an image deformation registration method to register the tissue puncture position in the puncture CT image to the baseline CT image, and dividing the baseline CT image into multiple tumor sub-areas; The entire tumor region of the annotated baseline CT image and each tumor sub-region are respectively input into the spatial attention deep learning model to predict the PD-L1 expression level at the location of each tumor sub-region; and a PD-L1 atlas is generated based on the obtained PD-L1 expression level; A dual-task prediction model was constructed, and the clinical information, baseline CT images and obtained PD-L1 maps of patients with non-small cell lung cancer were used as the input of the dual-task prediction model to predict the efficacy of immunotherapy for lung cancer.

2. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 1, characterized in that: The baseline CT image is divided into a plurality of tumor sub-regions. Specifically, the tumor region marked on the baseline CT image is divided into a plurality of sub-regions having the same size as the puncture region.

3. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 1, characterized in that: The spatial attention deep learning model includes a CNN network module and a spatial attention module; wherein the spatial attention module consists of two parallel fully connected layers, a cross attention module and two serial fully connected layers.

4. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 3, characterized in that: Based on the CNN network module, the deep imaging features of the entire tumor area on the baseline CT image are extracted, and the imaging features of the puncture position on the baseline CT image are calculated; based on the spatial attention module, the obtained deep imaging features and imaging features are jointly analyzed to predict the PD-L1 expression level at each tumor sub-region location.

5. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 1, characterized in that: The dual-task prediction model consists of two CNN network modules and a Transformer module with an integrated attention mechanism.

6. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 5, characterized in that: The two CNN network modules are used to extract deep features. Specifically, one of the two CNN network modules is used to extract deep features of the PD-L1 map, and the other is used to extract deep features of the baseline CT image. The Transformer module is used to integrate and analyze the two obtained deep features and the clinical information of patients with non-small cell lung cancer.

7. The method for predicting the efficacy of lung cancer immunotherapy based on dual CT images according to claim 1, characterized in that: Before feeding the clinical information of non-small cell lung cancer patients into the dual-task prediction model, the clinical information is digitally encoded.

8. A lung cancer immunotherapy efficacy prediction system based on dual CT images, characterized in that: include: The image annotation module is configured to: obtain a baseline CT image and a puncture CT image of a patient with non-small cell lung cancer, and respectively annotate tumor regions in the baseline CT image and the puncture CT image; The image pathology space matching module is configured to: register the tissue puncture position in the puncture CT image to the baseline CT image using an image deformation registration method, and divide the baseline CT image into multiple tumor sub-regions; The PD-L1 atlas construction module is configured to: input the entire tumor region of the annotated baseline CT image and each tumor sub-region into the spatial attention deep learning model to predict the PD-L1 expression level at the location of each tumor sub-region; generate a PD-L1 atlas based on the obtained PD-L1 expression level; The efficacy prediction module is configured to: construct a dual-task prediction model, use the clinical information, baseline CT images and obtained PD-L1 map of patients with non-small cell lung cancer as the input of the dual-task prediction model, and predict the efficacy of lung cancer immunotherapy.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the method for predicting the efficacy of lung cancer immunotherapy based on dual CT images as described in any one of claims 1 to 7 are implemented.

Images