Non-small cell lung cancer efficacy prediction method based on pet-ct deep learning

Abstract

The application relates to the technical field of deep learning, and particularly discloses a non-small cell lung cancer efficacy prediction method based on PET-CT deep learning. The method acquires the pre-treatment positron emission tomography and computer tomography of the same subject, performs registration, resampling and intensity standardization, generates a plurality of lesion candidate regions according to metabolic uptake distribution and anatomical boundary distribution, respectively extracts metabolic features and anatomical features of the lesion candidate regions, forms lesion-level joint representation through metabolic-anatomical cross-gating fusion, combines lesion spatial position and anatomical attribution to construct a patient-level lesion relationship graph, and inputs the graph attention deep learning network to output an efficacy prediction result. The method can reduce the interference of non-tumor metabolic uptake, boundary fuzzy structure and local noise on the prediction features, retain the correlation information among primary lesions, lymph node candidates and metastatic lesion candidates, and improve the stability of efficacy prediction in a multi-lesion heterogeneous scene.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and specifically discloses a method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning. Background Technology

[0002] Predicting the treatment efficacy of non-small cell lung cancer typically relies on combined imaging from positron emission tomography (PET) and computed tomography (CT). PET can display the metabolic uptake status of tumor tissue, while CT can show the lesion morphology, density, boundaries, and relationship with surrounding tissues. Current conventional methods often use manual delineation or semi-automatic segmentation to determine the primary lesion region, then extract indicators such as maximum metabolic value, average metabolic value, tumor metabolic volume, total glycolysis amount, major axis, minor axis, mean density, and texture features. These indicators are then input into statistical models, machine learning models, or convolutional neural networks for efficacy prediction. Some methods use pre-treatment images as input, while others register pre- and post-treatment images and calculate the difference features. The technical process of these methods is relatively fixed, mainly relying on a single major lesion or manually selected regions to represent the patient's overall condition. The model learning object is mainly the global pooling features of local image blocks or the entire image, lacking a unified modeling approach for the spatial distribution, metabolic differences, and anatomical attribution among multiple lesions.

[0003] Current deep learning prediction schemes typically spatially register PET and CT images before inputting them into a convolutional neural network in a dual-channel or multi-channel manner. The network automatically extracts deep features and outputs the efficacy category. To reduce manual feature design, some schemes perform standardization, cropping, resampling, and data augmentation before image input, then extract metabolic and anatomical textures through convolutional layers, and obtain the prediction result through feature stitching, fully connected layers, or attention layers. While such schemes can utilize deep learning for end-to-end modeling, they often treat the two images as parallel inputs, lacking mutual correction between metabolic information and anatomical boundaries. PET images contain inflammatory uptake, physiological uptake, low uptake of necrosis, and registration offsets; CT images contain blurred boundaries, lung atelectasis obstruction, and areas of similar soft tissue density. Simple stitching is insufficient to distinguish between true tumor activity and non-tumor interference. For patients with coexisting primary lesions, lymph node metastases, whole-image pooling can also weaken the expression of small-volume, high-metabolic lesions.

[0004] In the prediction of treatment efficacy for non-small cell lung cancer based on PET and CT, conventional deep learning models lack joint constraints on lesion-level metabolic features, anatomical features, and relationships between lesions. This leads to the model's tendency to use non-tumor metabolic uptake, poorly defined anatomical structures, or local noise as criteria for efficacy judgment. The technical reason for this problem is that existing solutions typically input the entire image into the network or extract features only around the largest lesion, failing to model the primary lesion, lymph node candidates, and metastatic lesion candidates as related objects with different anatomical affiliations and metabolic responses. Simultaneously, the metabolic sensitivity of PET is not constrained by CT boundaries, and the structural information from computed tomography is not guided by metabolic activity. Therefore, when there is significant lesion heterogeneity, inflammatory uptake is present, or multiple lesion responses are inconsistent, the model's efficacy prediction results are prone to bias. Summary of the Invention

[0005] The purpose of this invention is to provide a method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning, which can effectively solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Methods for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning include: Acquire PET and CT images of the same subject, and perform spatial registration, voxel resampling, and intensity standardization on the PET and CT images; Multiple lesion candidate regions are generated based on the metabolic uptake distribution in the PET images and the anatomical boundary distribution in the CT images. PET metabolic features and CT anatomical features were extracted from the candidate lesion regions, and a joint lesion-level characterization was obtained by metabolic-anatomical cross-gated fusion. A patient-level lesion relationship diagram is constructed based on the aforementioned lesion-level joint characterization, lesion spatial location, and lesion anatomical attribution. The patient-level lesion relationship graph is input into a graph attention deep learning network to obtain the efficacy prediction results for non-small cell lung cancer.

[0007] Preferably, generating the plurality of candidate lesion regions includes: extracting the lung parenchyma region, mediastinal region, and soft tissue boundary region from the CT image; extracting the metabolic uptake connected region from the PET image; and performing spatial cross-matching between the soft tissue boundary region and the metabolic uptake connected region. A candidate set containing primary lesion candidates, lymph node candidates, and metastatic lesion candidates is established for the matched region; For each candidate region in the candidate set, a center region, an edge region, and a neighboring background region are labeled, and the labeling results are used as a spatial index for subsequent feature extraction.

[0008] Preferably, obtaining the lesion-level joint characterization includes: learning anatomical features from CT images, where the listed density distribution, boundary morphology, internal necrosis, and contact relationship with adjacent tissues are the meaning of the features; and learning metabolic features from PET images, where the listed metabolic intensity, metabolic gradient, metabolic dispersion, and low uptake distribution are the meaning of the features. Structural constraint weights are generated based on the anatomical features, and activity guiding weights are generated based on the metabolic features. The two types of features are then cross-gated and fused using the structural constraint weights and the activity guiding weights.

[0009] Preferably, constructing the patient-level lesion relationship graph includes: using the joint representation of each lesion level as a graph node feature, and encoding the spatial distance, anatomical partition relationship, metabolic similarity, volume level difference and positional relationship relative to the primary lesion between different lesion candidate regions as graph edge features; The graph node features and graph edge features are jointly embedded to form graph structure data that includes lesion self-attributes, lesion association attributes and patient-level lesion distribution attributes, and the graph structure data is used as the input of the graph attention deep learning network.

[0010] Preferably, marking the central region, edge region, and adjacent background region for each candidate region in the candidate set includes: determining the central region based on the internal uptake peak position of the metabolic uptake connected region, determining the edge region based on the soft tissue boundary and lung parenchyma contact boundary in the CT image, and determining the adjacent background region based on the tissue region located outside the edge region and spatially adjacent to the lesion candidate region. The central region, the edge region, and the adjacent background region are mapped to PET feature maps and CT feature maps, respectively, and a partitioned feature sequence is generated.

[0011] Preferably, the cross-gated fusion includes: scaling the boundary confidence map output by the CT coding branch with the metabolic response map output by the PET coding branch; Feature suppression weights are applied to non-boundary consistent regions in the metabolic response map based on the boundary confidence map, and feature enhancement weights are applied to active residue regions in the boundary confidence map based on the metabolic response map. The weighted metabolic features, weighted anatomical features, and regional feature sequences are recombined into channels, and the lesion-level joint characterization is obtained by attention normalization.

[0012] Preferably, the encoding of the graph edge features includes: converting the three-dimensional center distance, ipsilateral or contralateral lung lobe affiliation, mediastinal station affiliation, metabolic uptake distribution similarity, metabolic difference between the central area and the edge area, and lesion volume level difference between any two graph nodes into edge attribute vectors respectively. The edge attribute vector is bidirectionally spliced ​​with the joint lesion-level representation of the two graph nodes to form a direction-related lesion association embedding, and hierarchical graph connections are established according to the lesion category, from primary lesion to lymph node candidates, from primary lesion to metastatic lesion candidates, and between candidate lesions.

[0013] Preferably, inputting the patient-level lesion relationship graph into the graph attention deep learning network includes: setting node attention weights for the graph node features, setting edge attention weights for the graph edge features, and inputting the node attention weights and edge attention weights together into the message passing layer; A patient-level pooling layer is set after the message passing layer. The patient-level pooling layer aggregates lesion characteristics according to the hierarchical map connection of primary lesion candidates, lymph node candidates, and metastatic lesion candidates, and inputs the aggregation results into the efficacy classification layer to generate the efficacy prediction results of non-small cell lung cancer.

[0014] Preferably, the attention normalization includes: calculating metabolic response attention and anatomical boundary attention for the central region, the edge region, and the adjacent background region, respectively; The metabolic response attention of the adjacent background region is used as a background reference, and the anatomical boundary attention of the edge region is used as a structural reference. The metabolic response attention of the central region is then differentially normalized. The normalized central region features, edge region features, and adjacent background region features are rearranged according to the spatial index of the lesion candidate region and input into the graph node feature generation process.

[0015] Preferably, the training of the graph attention deep learning network includes: using patient-level efficacy labels, lesion-level response labels, and PET-CT cross-modal reconstruction labels as supervisory data; A patient-level classification loss is set for the efficacy classification layer, a lesion response consistency loss is set for the message passing layer, and a cross-modal reconstruction loss is set for the lesion-level joint representation after cross-gated fusion. The parameters of the CT coding branch, the PET coding branch, and the graph attention deep learning network are updated based on the patient-level classification loss, the lesion response consistency loss, and the cross-modal reconstruction loss.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention combines the metabolic uptake distribution from PET and the anatomical boundary distribution from CT for lesion candidate region generation. Metabolic and anatomical features are extracted for each lesion candidate region, and then fused through metabolic-anatomical cross-gating to form a joint lesion-level characterization. The boundary, density, necrosis, and adjacent tissue contact relationships provided by computed tomography (CT) can constrain high-uptake areas in positron emission tomography (PET), reducing inflammatory uptake, physiological uptake, and registration shifts from entering the lesion characterization. The metabolic intensity, metabolic gradient, and metabolic dispersion information provided by PET can guide activity in areas with unclear boundaries or insignificant volume changes in CT. After cross-gating, these two types of features no longer enter the prediction process through simple splicing, but rather complete mutual screening and compensation at the lesion level. This makes the features used for efficacy prediction more concentrated in areas with tumor metabolic activity and supported by anatomical structures, thereby addressing the technical problem of conventional models easily incorporating non-tumor interference features.

[0017] 2. This invention further uses lesion-level joint representation as graph node features, encoding lesion spatial distance, anatomical partitioning relationships, metabolic similarity, volume level differences, and positional relationships relative to the primary lesion as graph edge features to construct a patient-level lesion relationship graph. Efficacy prediction is then performed using a graph attention deep learning network. Primary lesion candidates, lymph node candidates, and metastatic lesion candidates are hierarchically connected within the same graph structure, preserving spatial associations, anatomical affiliation associations, and metabolic response associations between different lesions, avoiding the dilution of small-volume, high-metabolic lesions or asynchronously responding lesions into background features by whole-graph pooling. The partitioning features of the central area, peripheral area, and adjacent background area also preserve the differential information of the tumor interior, invasion boundaries, and surrounding tissues in the lesion representation. During training, patient-level efficacy labels, lesion-level response labels, and cross-modal reconstruction labels are introduced to ensure consistency between the graph attention network, the metabolic-anatomical fusion process, and the lesion response expression, thereby providing a more complete basis for patient-level prediction results in terms of lesion relationships. Attached Figure Description

[0018] Figure 1 This is the overall flowchart of the non-small cell lung cancer efficacy prediction based on PET-CT deep learning according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please refer to Figure 1This embodiment provides a method for predicting the efficacy of treatment for non-small cell lung cancer based on PET-CT deep learning, including: Acquire PET and CT images of the same subject, and perform spatial registration, voxel resampling, and intensity standardization on the PET and CT images; Multiple lesion candidate regions are generated based on the metabolic uptake distribution in the PET images and the anatomical boundary distribution in the CT images. PET metabolic features and CT anatomical features were extracted from the candidate lesion regions, and a joint lesion-level characterization was obtained by metabolic-anatomical cross-gated fusion. A patient-level lesion relationship diagram is constructed based on the aforementioned lesion-level joint characterization, lesion spatial location, and lesion anatomical attribution. The patient-level lesion relationship graph is input into a graph attention deep learning network to obtain the efficacy prediction results for non-small cell lung cancer.

[0021] In this embodiment, two types of images from the same subject are first transformed to a unified coordinate system. PET images express lesion activity using metabolic uptake distribution, while CT images express anatomical structures using grayscale density and boundary gradient. After registration, they form a voxel-by-voxel data basis. Voxel resampling uses a unified voxel grid to ensure consistency in spatial indexing between the two types of images. Intensity standardization maps metabolic uptake values ​​and CT grayscale to numerical ranges suitable for deep learning network input. Lesion candidate regions are not based on a single largest lesion as the sole input, but rather on the intersection, adjacency, and inclusion relationships between areas with abnormal metabolic uptake and suspicious anatomical boundary regions. Multiple candidate blocks are formed, each covering at least the primary lesion, suspicious lymph nodes, and suspicious metastatic lesions. Metabolic and anatomical features are extracted from each candidate block, and lesion-level fusion is achieved through cross-gating. The fused representation is then used as a graph node in the patient-level lesion relationship graph. The graph attention deep learning network transmits association information between lesion nodes and outputs the efficacy prediction result. This embodiment achieves efficacy prediction by using the same-layer synergy of metabolic information, anatomical information, and lesion relationship information, so that the efficacy prediction no longer depends on whole-graph pooling or single lesion features. It can form a more stable patient-level judgment basis when there is heterogeneity of multiple lesions and non-tumor uptake interference.

[0022] In this embodiment, spatial registration uses CT images as the reference coordinates. Initial alignment is achieved by first reading the DICOM spatial orientation, slice spacing, and coordinate origin. For data with positional or respiratory misalignment, the lung parenchyma, mediastinal soft tissue, and thoracic boundary are used as stable anatomical references. PET images are mapped to the reference coordinate system through rigid transformation and local deformation transformation. Rigid transformation eliminates overall translational and rotational differences, while local deformation transformation corrects local misalignment caused by respiratory motion and examination position differences. After registration, both types of images are cropped and resampled at the same scale to ensure that each candidate region has the same 3D index range in both types of images. Intensity normalization includes normalizing the in vivo reference region and truncating abnormal high values ​​in PET images, and jointly normalizing lung windows and soft tissue windows in CT images while preserving grayscale gradient changes. The deep learning input tensor consists of a metabolic channel, an anatomical grayscale channel, and an anatomical gradient channel.

[0023] In a preferred embodiment, to provide a clear quantitative basis for the distribution of metabolic uptake and anatomical boundaries during the candidate region generation stage, voxels in three-dimensional space are denoted as... The standardized metabolic uptake value of PET is denoted as The normalized gray-scale gradient amplitude of CT is denoted as The confidence level of metabolic connectivity is denoted as The confidence level of the anatomical boundary is denoted as The confidence level of the candidate lesion is recorded as . ,in A function that maps real numbers to the interval [0,1]. , and These are the fusion weights for metabolic uptake, anatomical boundaries, and spatial adjacency, respectively. voxels The neighborhood voxel set, and the candidate confidence score are calculated according to the following formula:

[0024] When a voxel is located in a region of high metabolic uptake and there are continuous anatomical boundaries in its neighborhood. and At the same time, a higher value is obtained, and the neighborhood term increases with the increase of metabolic-anatomical consistency of the surrounding voxels, and the candidate confidence is then included in the lesion candidate set; when the voxel only shows isolated high metabolic value without anatomical boundary support, the neighborhood term and anatomical boundary term are limited, and the candidate confidence is suppressed; when the voxel has a clear anatomical boundary but no abnormal metabolic uptake, the metabolic term is limited, and the candidate confidence is also difficult to enter the lesion candidate set. This embodiment provides reproducible numerical rules for candidate region generation through this calculation method, and makes PET and CT mutually constrain each other in the candidate screening stage.

[0025] Preferably, generating the plurality of candidate lesion regions includes: extracting the lung parenchyma region, mediastinal region, and soft tissue boundary region from the CT images; extracting metabolic uptake connectivity regions from the PET images; and performing spatial cross-matching between the soft tissue boundary regions and the metabolic uptake connectivity regions. A candidate set containing primary lesion candidates, lymph node candidates, and metastatic lesion candidates is established for the matched regions. Each candidate region in the candidate set is labeled with a central region, a marginal region, and a neighboring background region, and the labeling results are used as a spatial index for subsequent feature extraction. In this embodiment, the lung parenchyma region is determined by CT grayscale range, lung field connectivity, and intrathoracic boundary; the mediastinal region is determined by a region located between the two lungs with continuous soft tissue grayscale; the soft tissue boundary region is determined by grayscale gradient changes and local density mutations; and the metabolic uptake connectivity region is determined by standardization. After determining the connectivity of the metabolic uptake voxels, the spatial cross-matching does not directly take the intersection of the two, but establishes a local search domain centered on the metabolic connectivity region. In the search domain, it is checked whether the soft tissue boundary region has a relationship of inclusion, overlap, attachment or short-distance adjacency with the metabolic region. Regions that meet the conditions are included in the candidate set. Primary lesion candidates are mainly located in the lung parenchyma or attached to the lung parenchyma boundary. Lymph node candidates are mainly located in the soft tissue region of the mediastinum or hilum. Metastatic lesion candidates are classified according to different anatomical partitions and metabolic connectivity features from the primary lesion. The central region is used to express the main metabolism and internal density of the lesion, the edge region is used to express the infiltration boundary and metabolic gradient, and the adjacent background region is used to express the surrounding tissue reference. In this embodiment, the candidate region generation process retains metabolic abnormalities, anatomical attribution and local background at the same time, so that the subsequent deep learning features are not limited by a single uptake threshold or a single anatomical segmentation.

[0026] Further, for each candidate region in the candidate set, a central region, an edge region, and a neighboring background region are labeled, including: determining the central region based on the internal uptake peak position of the metabolic uptake connected region; determining the edge region based on the soft tissue boundary and lung parenchyma contact boundary in the CT image; and determining the neighboring background region based on the tissue region located outside the edge region and spatially adjacent to the lesion candidate region. The central region, the edge region, and the neighboring background region are mapped to PET feature maps and CT feature maps, respectively, and a partitioned feature sequence is generated. In this embodiment, the internal uptake peak position is not limited to a single voxel point, but is jointly determined by the geometric center and metabolic weighting center of the high uptake voxel cluster, avoiding single-point noise as the basis for center localization; the central region is generated around the metabolic weighting center in the three-dimensional candidate block and is restricted to the area surrounded or attached by the soft tissue boundary; the edge region is jointly determined by the outer contour of the candidate block, the CT gradient band, and the lung parenchyma contact surface, and can cover solid boundaries, spiculated boundaries, and pleural boundaries; the neighboring background region is selected from local tissue outside the edge region, excluding areas that overlap with other candidate lesions. When the three types of partitions are mapped to the deep feature map, they maintain the same spatial index. The partition feature sequence is encoded in the order of central region, edge region, and adjacent background region. Each feature consists of metabolic statistical features, anatomical texture features, and location embedding. In this embodiment, the activity, invasion boundary, and surrounding tissue reference within the lesion are expressed separately, so that subsequent cross-gated fusion can screen the lesion characteristics that truly participate in efficacy prediction based on the differences between the partitions.

[0027] In one embodiment, obtaining the lesion-level joint characterization includes: learning anatomical features from CT images, where density distribution, boundary morphology, internal necrosis, and contact relationships with adjacent tissues are the semantic representations of these features; learning metabolic features from PET images, where metabolic intensity, metabolic gradient, metabolic dispersion, and low uptake distribution are the semantic representations of these features; generating structural constraint weights based on the anatomical features; generating activity-guided weights based on the metabolic features; and performing cross-gating fusion of the two types of features using the structural constraint weights and the activity-guided weights. In this embodiment, the CT coding branch uses three-dimensional convolutional layers and local attention layers to progressively extract grayscale texture, edge direction, and tissue contact patterns within candidate blocks. Density distribution is used to distinguish solid areas, low-density necrotic areas, and surrounding lung tissue, and boundary morphology is used to express clear boundaries. Fuzzy boundaries, lobulated boundaries, and spiculated boundaries are used to represent the correspondence between low-density areas and low-uptake metabolic areas. The contact relationship between adjacent tissues is used to represent the spatial attachment of the lesion to the pleura, blood vessels, mediastinal soft tissue, or lung parenchyma. The PET coding branch uses a shared-scale three-dimensional convolutional layer to extract high-value metabolic areas, low-value metabolic areas, uneven metabolic areas, and central low-uptake areas. Metabolic intensity characterizes the size of lesion activity, metabolic gradient characterizes the change from the center to the edge, metabolic dispersion characterizes internal heterogeneity, and low-uptake distribution characterizes necrotic or low-activity areas. The structural constraint weight is generated by the anatomical boundary and tissue contact relationship, and the activity guiding weight is generated by the metabolic gradient and high-value metabolic clusters. The two types of weights cross-act on the output of the other branch. This embodiment avoids metabolic features and anatomical features participating in prediction independently after splicing, so that the two types of features complete mutual correction at the lesion level.

[0028] Preferably, the cross-gated fusion includes: scaling the boundary confidence map output by the CT coding branch and the metabolic response map output by the PET coding branch; applying feature suppression weights to non-boundary consistent regions in the metabolic response map based on the boundary confidence map, and applying feature enhancement weights to active residual regions in the boundary confidence map based on the metabolic response map; performing channel recombination on the weighted metabolic features, weighted anatomical features, and regional feature sequences, and obtaining the lesion-level joint representation through attention normalization; in this embodiment, the boundary confidence map is obtained from the last layer features of the CT branch through channel compression and spatial normalization, the metabolic response map is obtained from the last layer features of the PET branch through metabolic activation mapping, and scale alignment is achieved through three lines. After interpolation and index pruning are completed, the boundary confidence map and the metabolic response map have consistent grid positions in the central region, edge region, and adjacent background region. Non-consistent boundary regions refer to regions with high metabolic response but low anatomical boundary confidence and not adjacent to the edge of candidate lesions. Active residual regions refer to regions with high boundary confidence and continuous distribution of metabolic response above the local background. During channel recombination, the weighted metabolic channels, weighted anatomical channels, and partition feature embedded channels are spliced ​​and linearly mapped according to the same spatial index. Attention normalization assigns representation weights among different partitions and different channels. This embodiment makes cross-gating have reproducible spatial constraints and channel constraints, reducing model confusion in inflammatory uptake, vascular proximity uptake, and soft tissue boundary proximity regions.

[0029] In a preferred embodiment, the anatomical feature vector output by the CT coding branch is denoted as... The metabolic feature vector output by the PET encoding branch is denoted as... The structural constraint weights generated from anatomical features are denoted as... The activity-guided weights generated by metabolic characteristics are denoted as The combined lesion-level characteristics after cross-gated control are denoted as ,Will This is denoted as element-wise multiplication in the same dimension. Let be the trainable mapping matrix, and If denoted as a nonlinear transformation, then cross-gating fusion is performed according to the following formula:

[0030] in, For the first The convergence vector of partition features of candidate lesion regions is such that when the structural constraint weight is low at a certain spatial position, the corresponding metabolic features are suppressed in the element-wise multiplication. When the activity guiding weight is high at a certain boundary position, the corresponding anatomical features are preserved or enhanced in the element-wise multiplication. If a region has both high metabolic response and clear anatomical boundary, the corresponding elements enter the dominant channel of the trainable mapping matrix after splicing. If a region has only a single modality abnormality, the corresponding elements are weakened in the fusion vector. In this embodiment, the activity information of PET and the structural information of CT are unified into the same lesion-level representation in a mutually gating manner through this formula.

[0031] Further, the attention normalization includes: calculating metabolic response attention and anatomical boundary attention for the central region, the edge region, and the adjacent background region respectively; using the metabolic response attention of the adjacent background region as a background reference term and the anatomical boundary attention of the edge region as a structural reference term, and performing differential normalization processing on the metabolic response attention of the central region. The normalized features of the central region, edge region, and adjacent background region are then rearranged according to the spatial index of the lesion candidate region and input into the graph node feature generation process.

[0032] In the formula, For the normalized central region metabolic response attention, Attention is required for metabolic response in the central region. Attention is given to the average metabolic response of the neighboring background region and used as a background reference. For the set of voxels in the adjacent background region, The anatomical boundary of the marginal region is used for attention and as a structural reference. When the metabolic response in the central region exceeds the background reference and the marginal region has structural boundary support, A high value was achieved.

[0033] In this embodiment, the metabolic response attention in the central region is jointly determined by the local mean, local peak value, and continuity of the metabolic response map within the central region. The anatomical boundary attention in the edge region is jointly determined by the gradient intensity, boundary direction consistency, and closure relationship with the central region within the edge region. The metabolic response attention in the adjacent background region is used as a reference for the uptake of surrounding tissues. Differential normalization processing compares the metabolic response in the central region with the uptake of the adjacent background region, while simultaneously referencing the structural boundary of the edge region. This ensures that high uptake in the central region only results in a higher node input weight when it exceeds the background reference and has structural boundary support. During rearrangement, the sequential index of the three types of partitions in the original three-dimensional space is maintained to avoid losing spatial position after feature convergence. In this embodiment, the interference of local background uptake on the metabolic expression in the central region is eliminated before the formation of lesion nodes, while retaining the constraint of edge region structural information on the determination of central region activity.

[0034] In this embodiment, to ensure that the central region, edge region, and adjacent background region of the lesion candidate region have a unified data definition during feature extraction, gating fusion, and graph node generation, the data organization method shown in Table 1 below is used to generate partition features for each lesion candidate region. The fields in Table 1 are jointly generated from the registered 3D image, metabolic response map, anatomical boundary map, and candidate region index, and the field order is kept consistent during the deep learning training and inference stages. Table 1. Description of Three-Part Feature Configuration of PET-CT Lesions Central area Metabolic uptake peak within the connected region, surrounding voxels, and the region bounded by the CT boundary. Metabolic intensity, metabolic continuity, internal density, low uptake distribution Central area features are incorporated into lesion-level combined characterization. edge area CT soft tissue boundaries, lung parenchyma contact boundaries, and metabolic gradient change zones Anatomical boundary confidence, metabolic gradient, edge density, contact relationship Edge region features participate in cross-gating and attention normalization Adjacent background area Local tissue located outside the candidate lesion area and spatially adjacent to the marginal zone Background metabolic response, background density, tissue adjacency category, local texture Background reference features are used in differential normalization processing. In this embodiment, the central region features in Table 1 mainly consist of the lesion's metabolic activity and internal anatomical state, the edge region features mainly consist of the structural boundary and metabolic gradient, and the adjacent background region features mainly consist of the uptake and density reference of surrounding tissues. After the candidate region is generated, the three types of features are bound to the same lesion index. When the PET coding branch and CT coding branch output features, they all read the corresponding partition based on the lesion index. The cross-gating fusion does not directly perform mean pooling on the entire candidate region, but processes the three types of features separately before merging them into the lesion-level joint representation. When the graph node is generated, the lesion-level joint representation is the main body, and the convergence result of the partition features is retained as the internal structural information of the node. This embodiment decomposes the internal structure of the candidate region into stable data fields, which enhances the reproducibility of the implementation process and makes the metabolic-anatomical cross-gating have a unified input form between different subjects and different lesion types.

[0035] In one embodiment, constructing the patient-level lesion relationship graph includes: using each lesion-level joint representation as a graph node feature; encoding the spatial distance, anatomical partitioning relationship, metabolic similarity, volume level difference, and positional relationship relative to the primary lesion between different lesion candidate regions as graph edge features; jointly embedding the graph node features and the graph edge features to form graph structure data containing lesion self-attributes, lesion association attributes, and patient-level lesion distribution attributes; and using the graph structure data as input to the graph attention deep learning network. In this embodiment, each lesion candidate region corresponds to a graph node, and the node features consist of lesion-level joint representation, lesion category embedding, partitioning feature convergence vector, and spatial location embedding. The spatial distance is determined by the three-dimensional center coordinates of different lesion candidate regions. The anatomical zoning relationship is determined by the lung lobe affiliation, mediastinal station affiliation, pleural adjacency status, and whether it is located in the same lateral anatomical region. The metabolic similarity is determined by the distance between the metabolic intensity distribution, metabolic gradient distribution, and metabolic dispersion distribution of different lesions. The volume level difference is calculated by the number of voxels contained in the candidate region and the volume enclosed by the anatomical boundary. The positional relationship relative to the primary lesion is jointly expressed by the direction vector, distance, and anatomical path between the center of the candidate region and the center of the primary lesion. During joint embedding, node features and edge features are mapped to the same feature dimension, enabling the graph attention deep learning network to simultaneously read the information of the lesion itself and the information between lesions. In this embodiment, the multi-lesion efficacy prediction is transformed from whole-graph compression into computable relational reasoning, preserving the independent contribution of heterogeneous lesions in patient-level prediction.

[0036] Preferably, the encoding of the graph edge features includes: converting the three-dimensional center distance, ipsilateral or contralateral lung lobe affiliation, mediastinal station affiliation, metabolic uptake distribution similarity, metabolic differences between the central and peripheral areas, and lesion volume level differences between any two graph nodes into edge attribute vectors; bidirectionally concatenating the edge attribute vectors with the lesion-level joint representation of the two graph nodes to form a direction-related lesion association embedding, and establishing hierarchical graph connections between primary lesions and lymph node candidates, primary lesions and metastatic lesion candidates, and between candidate lesions according to lesion categories; in this embodiment, the three-dimensional center distance is calculated from the geometric center of the voxel coordinates of the candidate region, the lung lobe affiliation is determined according to the lung lobe where the center of the candidate region is located or the nearest lung lobe, and the mediastinal station affiliation is determined according to the candidate lesion category. The location of the selected region in the mediastinal region is determined by encoding. The similarity of metabolic uptake distribution is obtained by comparing the distribution vectors of the metabolic response map. The metabolic difference between the central and peripheral regions is obtained by the statistical difference of the metabolic response of the two regions. The difference in lesion volume level is obtained by comparing the number of voxels and the bounded volume of the two candidate regions. During bidirectional splicing, lesion association embeddings are formed from node i to node j and from node j to node i, respectively. The direction-related expression makes the influence path of the primary lesion on the lymph node candidate and the feedback path of the lymph node candidate on the primary lesion have different parameter representations. The hierarchical graph connection places the primary lesion, lymph node candidate and metastatic lesion candidate in different relation layers. In this embodiment, the graph edge no longer only expresses distance, but also carries anatomical affiliation, metabolic similarity and lesion hierarchical relationship.

[0037] In a preferred embodiment, the first The lesion node and the first The edge attribute vector between each lesion node is denoted as . Let the three-dimensional center distance between the two be denoted as The anatomical partition matching vector is denoted as The similarity of metabolic uptake distribution is denoted as The difference in volume grades is denoted as The positional relationship relative to the primary lesion is coded as follows: ,Will Let be the trainable mapping matrix, and If denoted as a nonlinear mapping, then the edge attribute vector is formed according to the following formula:

[0038] in, It is obtained from the Euclidean distance between the center coordinates of the two candidate regions. Different codes are used when two candidate regions belong to the same lobe, contralateral lobe, the same mediastinal station, or different mediastinal stations. When the metabolic response distributions of the two lesions are similar, the higher value is taken. Used to differentiate between the primary lesion and small-volume candidate lesions. The algorithm expresses the directional and hierarchical relationship between candidate lesions and the primary lesion. When two lesions have a close spatial distance, the same anatomical path, and similar metabolic distribution, the edge attribute vectors form a strong connection representation in the corresponding dimension. When two lesions are far apart but are in the hierarchical connection from the primary lesion to the metastatic lesion, the positional relationship encoding can still retain this long-distance relationship. This embodiment uses this formula to make the graph edge features have computable generation rules and enable the relationship between lesions to be directly used by the graph attention deep learning network.

[0039] Furthermore, inputting the patient-level lesion relationship graph into the graph attention deep learning network includes: setting node attention weights for the graph node features, setting edge attention weights for the graph edge features, and inputting the node attention weights and edge attention weights together into the message passing layer. After the message passing layer, a patient-level pooling layer is set. This patient-level pooling layer aggregates lesion representations according to the hierarchical graph connections of primary lesion candidates, lymph node candidates, and metastatic lesion candidates, and inputs the aggregation results into the efficacy classification layer to generate the non-small cell lung cancer efficacy prediction results. In this embodiment, node attention weights are jointly calculated by metabolic-anatomical consistency, partition feature strength, and lesion category embedding in the lesion-level joint representation. Edge attention weights are jointly calculated by spatial distance, anatomical relationship, metabolic similarity, and positional relationship relative to the primary lesion in the edge attribute vector. The message passing layer receives information from adjacent lesion nodes at each lesion node and updates it in combination with the edge attention weights. The primary lesion candidate layer aggregates the main lesion representation, the lymph node candidate layer aggregates mediastinal and hilar related candidate representations, and the metastatic lesion candidate layer aggregates candidate representations far from the primary lesion. The patient-level pooling layer maps the aggregation results of different levels to patient-level vectors according to trainable weights. The efficacy classification layer maps patient-level vectors to efficacy categories. This embodiment enables the patient-level prediction results to be jointly generated by the internal lesion representation and the relationship between lesions, avoiding the dominance of a single high-weight region in the global pooling of all predictions.

[0040] In this embodiment, a method is provided for implementing message passing in a graph attention deep learning network where nodes and edges jointly participate. The node features of a graph node in layer l are denoted as follows: Let the set of adjacent nodes be denoted as Let the edge attention coefficient be denoted as The node update feature is denoted as ,Will and Let them be the node mapping matrix and message mapping matrix of the l-th layer, respectively. This indicates normalization by node j within the set of adjacent nodes. If non-linear activation is indicated, message passing is performed according to the following formula:

[0041] When adjacent lesion nodes and the target node have similar metabolic distributions and reasonable anatomical paths, the edge attribute vectors are included in the attention calculation to make the corresponding... When adjacent nodes are close in distance but lack metabolic-anatomical consistency, the edge attribute vector limits their message contribution. After multiple layers of message passing, the node features contain their own lesion representation and information about their relationship with other lesions. This embodiment clarifies how the graph attention deep learning network uses node features and edge features to jointly update the lesion representation.

[0042] In a preferred embodiment, the patient-level pooling layer of the graph attention deep learning network employs hierarchical pooling for different lesion levels. The candidate sets for primary lesions, lymph node candidates, and metastatic lesions are respectively denoted as... , and The patient-level vector is formed by concatenating the weighted convergence results of the three sets. For any lesion candidate set... The weighting coefficients of each node within the set are calculated using a trainable attention vector based on the final layer features of the node. When a set is empty, an empty set mask is used to make the pooling vector corresponding to that set output a zero vector without generating lesion information. The efficacy classification layer consists of a fully connected mapping, a normalization layer, and a category output layer. Its input is a patient-level vector obtained through lesion-level fusion, graph-edge encoding, message passing, and hierarchical pooling. During training, efficacy categories are encoded according to preset efficacy evaluation labels; during inference, the efficacy classification layer outputs the predicted score or predicted probability of each efficacy category. This embodiment can adapt to cases with different numbers of lesion candidates and maintain hierarchical distinction between primary lesion candidates, lymph node candidates, and metastatic lesion candidates in the patient-level output. The relevant formulas are as follows:

[0043] when :

[0044] For patient-level feature vectors, For the first Pooling and aggregation characteristics of lesion-like lesions These represent the pooling aggregation features of the candidate sets for primary lesions, lymph nodes, and metastatic lesions, respectively. Indicates the first The node characteristics obtained from the last layer of message passing for each lesion node Indicates the first The node characteristics are obtained from the last layer of message passing for each lesion node.

[0045] when :

[0046] In the formula, This represents a trainable attention vector. Indicates the first The normalized weighted coefficients of each lesion node within its respective set. This represents a zero vector with the same dimension as the node's features. The three pooling vectors occupy fixed concatenation positions in the patient-level vectors, thus maintaining the corresponding lesion category position even when a set is empty.

[0047] Furthermore, the training of the graph attention deep learning network includes: using patient-level efficacy labels, lesion-level response labels, and PET-CT cross-modal reconstruction labels as supervisory data; setting a patient-level classification loss for the efficacy classification layer, a lesion response consistency loss for the message passing layer, and a cross-modal reconstruction loss for the lesion-level joint representation after cross-gated fusion; updating the parameters of the CT coding branch, the PET coding branch, and the graph attention deep learning network based on the patient-level classification loss, the lesion response consistency loss, and the cross-modal reconstruction loss; in this embodiment, patient-level efficacy labels are used to supervise the final efficacy category, and lesion-level response labels are used to constrain different lesion nodes in the elimination... The consistency or difference in response direction after information transmission is considered. Cross-modal reconstruction labels are used to constrain the joint representation at the lesion level to retain mutually interpretable information between the two types of images. Patient-level classification loss is applied to the output of the efficacy classification layer, lesion response consistency loss is applied to the node representation output of the message transmission layer, and cross-modal reconstruction loss is applied to the metabolic response map and anatomical boundary map reconstructed from the joint representation at the lesion level. During training, the above losses are merged with fixed or adaptive weights. Backpropagation updates the encoding branches, cross-gating parameters, edge attribute mapping parameters, and graph attention network parameters of the two types of images simultaneously. In this embodiment, the model training is not only centered around the patient-level labels, but also maintains the consistency between the lesion representation and the image source through lesion-level and cross-modal constraints.

[0048] In this embodiment, the patient-level efficacy classification loss is denoted as... The loss of consistency in lesion response is denoted as The cross-modal reconstruction loss is denoted as The total training loss is denoted as The unique hot code for the true category of patient-level efficacy is denoted as The output probability of the efficacy classification layer is denoted as The response marker of the i-th lesion node is denoted as... The predicted lesion response after message transmission is denoted as The reconstructed metabolic response map and anatomical boundary map are denoted as follows: and The corresponding supervision markers are recorded as follows: and ,Will and Let these be the loss weights, then the training objective is set according to the following formula:

[0049] in, This represents the therapeutic efficacy category index, where N represents the number of candidate lesion regions. This represents the squared error. The sum of absolute errors indicates that when the output of the efficacy classification layer is inconsistent with the patient-level efficacy label, the patient-level classification loss increases; when the lesion node response prediction is inconsistent with the lesion-level response label, the lesion response consistency loss increases; and when the reconstructed metabolic response map or anatomical boundary map deviates from the supervision label, the cross-modal reconstruction loss increases. The training process uses this joint objective to simultaneously constrain the final classification, lesion relationship, and cross-modal representation. This embodiment provides an executable parameter update objective to support the deep learning model in learning lesion-level metabolic-anatomical co-representation under multi-task supervision.

[0050] In one embodiment, lesion-level response labels can originate from changes in candidate lesion regions in pre- and post-treatment images, or from expert markings indicating whether a lesion is a primary responding lesion. If only patient-level efficacy labels are available without lesion-specific labels, weak response labels are generated using patient-level labels and lesion metabolic-anatomical consistency. These weak response labels do not directly replace manual markings but are incorporated as soft constraints into the lesion response consistency loss during the initial training phase. Specifically, for training samples with patient-level efficacy of remission, primary lesion candidates and lymph node candidates with high metabolic uptake and located within well-defined anatomical boundaries are assigned higher response attention values. For training samples with patient-level efficacy of progression, newly added candidate lesions or candidate lesions with increased metabolism are assigned higher response attention values. To prevent representation collapse caused by the network's own output determining the supervision target, weak response labels remain fixed during a single model training process and are not updated synchronously with the real-time output of the graph attention deep learning network. When weak response labels participate as target values ​​in the lesion response consistency loss, gradient truncation is performed, and gradients are not backpropagated to the generation process of weak response labels. The loss weights corresponding to weak response labels are set to low values ​​in the early stages of training and gradually increased with each training round. Cross-modal reconstruction loss independently constrains the lesion-level joint representation, preserving PET metabolic uptake information and CT anatomical boundary information, thereby reducing the possibility of lesion response prediction deviating from imaging evidence. If artificial lesion-level response labels exist, the corresponding lesion node is preferentially supervised using artificial labels, and weak response labels are not included in the loss calculation for that node. This embodiment utilizes patient-level labels and image consistency to provide auxiliary supervision when lesion-by-lesion labeling is insufficient, while avoiding mutual definition between weak response labels and network output.

[0051] In a preferred embodiment, the cross-modal reconstruction marker consists of a metabolic response map and an anatomical boundary map. The metabolic response map is obtained by local smoothing and background subtraction within the lesion candidate region from a standardized PET image. The anatomical boundary map is obtained by gradient calculation, edge continuity screening, and soft tissue boundary enhancement within the lesion candidate region from a CT image. The lesion-level joint representation after cross-gated fusion is reconstructed into the metabolic response map and the anatomical boundary map through two decoding branches, respectively. The two decoding branches share the lesion spatial index with the PET coding branch and the CT coding branch but do not share the output parameters. The metabolic reconstruction branch requires the lesion-level joint representation to retain the metabolic response distribution of the central and peripheral regions, while the anatomical reconstruction branch requires the lesion-level joint representation to retain the structural boundary distribution of the peripheral and adjacent background regions. If the cross-gated fusion relies excessively on one modality and loses the features of the other modality, the corresponding reconstruction error will enter the joint training objective, and the model parameters will be corrected with backpropagation. This embodiment establishes a trainable correspondence between the fused representation and the original image source, reducing the modal bias generated after simple stitching of the two types of images.

[0052] Furthermore, in the inference phase, after inputting PET and CT images of the same subject, the same spatial registration, voxel resampling, intensity standardization, lesion candidate region generation, partition feature generation, metabolic-anatomical cross-gated fusion, patient-level lesion relationship graph construction, and graph attention deep learning network propagation process as in the training phase are executed. The model parameters are not updated in the inference phase. All candidate regions generate lesion-level joint representations and form nodes in sequence. The edge attribute vectors are directly calculated from the spatial distance between candidate regions, anatomical partitions, metabolic similarity, volume level differences, and positional relationship relative to the primary lesion. After message passing, the patient-level pooling layer outputs patient-level vectors, and the efficacy classification layer outputs the predicted probabilities of different efficacy categories. The system can simultaneously save node attention weights, edge attention weights, partition feature weights, and candidate region indices to form a lesion contribution record corresponding to the prediction results. In this embodiment, the inference process is consistent with the training process in terms of image preprocessing, feature generation, and relationship modeling, and can output intermediate data traceable to lesion candidate regions.

[0053] In this embodiment, to prevent an excessive number of candidate regions from resulting in a large number of irrelevant nodes in the graph structure, candidate merging and candidate retention are performed before candidate regions enter the patient-level lesion relationship graph. Candidate merging is performed based on the three-dimensional spatial overlap ratio, metabolic uptake connectivity, and anatomical boundary overlap. If two candidate regions highly overlap and have the same anatomical affiliation, the region with higher candidate confidence is retained, and the partitioning features of the other candidate region are incorporated into the adjacency features of the retained region. If two candidate regions are spatially adjacent but located in different anatomical affiliations, they are not merged, but edge connections are established in the graph structure. Candidate retention is performed based on candidate confidence, metabolic-anatomical consistency, and lesion category coverage requirements. Specifically, primary lesion candidates are retained first, while lymph node candidates and metastatic lesion candidates retain a number of representative regions according to the layered graph connection requirements. The number of graph nodes after candidate retention is still allowed to vary among different subjects. This embodiment reduces the interference of isolated noise regions on graph attention propagation and maintains the coverage of different lesion categories in the patient-level graph structure.

[0054] In a preferred embodiment, the lesion category embedding is generated jointly by the anatomical attribution and spatial location of the candidate region. Candidate regions located within the lung parenchyma and possessing soft tissue boundaries and metabolic uptake are encoded as primary lesion candidate types; candidate regions located in the mediastinum or hilum region and exhibiting local metabolic uptake connectivity are encoded as lymph node candidate types; and suspicious metabolic-anatomical matching regions located far from the primary lesion and discontinuous with it are encoded as metastatic lesion candidate types. If the same candidate region simultaneously satisfies multiple category conditions, a weighted sum is generated based on its spatial relationship with the primary lesion candidate, its anatomical region, and boundary morphology. The mixed category embedding is entered into the graph node features as a continuous vector, without forcing a single-category hard classification. The graph attention deep learning network can identify different hierarchical relationships based on the category embedding during message passing. This embodiment adapts to situations where the anatomical attribution of lesions is unclear in complex cases and avoids premature misclassification that could disrupt the structure of the lesion relationship graph.

[0055] Furthermore, the graph-structured data undergoes graph normalization before entering the graph attention deep learning network. Graph normalization includes node feature normalization, edge attribute vector normalization, and hierarchical connection mask generation. Node feature normalization transforms the joint lesion-level representation of each lesion node to a uniform numerical scale according to the statistical distribution of the training set; edge attribute vector normalization maps spatial distance, metabolic similarity, and volumetric differences to the same numerical range; and hierarchical connection masks embed unreasonable message passing paths based on lesion categories. For example, direct strong connections between candidate nodes that clearly belong to the nearby background and distant candidate nodes are masked, while connections between primary lesion candidates and lymph node and metastatic lesion candidates are preserved. The mask serves as an additional input for edge attention calculation, enabling the graph attention deep learning network to maintain learnable connection weights while adhering to basic anatomical hierarchies. This embodiment unifies the data scale and connection relationships of the lesion relationship graph to a trainable range, avoiding the impact of numerical scale differences or irrelevant connections on message passing.

[0056] In one embodiment, the metabolic-anatomical cross-gated fusion and graph attention deep learning network can be trained jointly or pre-trained and then fine-tuned. The joint training method directly trains the PET coding branch, CT coding branch, cross-gated layer, graph-edge mapping layer, message passing layer, and efficacy classification layer from the registered candidate regions. The pre-trained and then fine-tuned method first trains the two coding branches and the cross-gated layer using cross-modal reconstruction labels, enabling the lesion-level joint representation to reconstruct the metabolic response map and anatomical boundary map. Then, it trains the graph attention deep learning network using patient-level efficacy labels and lesion-level response labels. During the fine-tuning phase, the cross-gated layer and the graph attention network are allowed to update together, but the range of change in the underlying coding branches is limited to prevent excessive bias towards patient-level labels when the sample size is small. This embodiment can select different training strategies based on the training data annotation and ensures that the core process always revolves around lesion-level metabolic-anatomical fusion and patient-level relationship graph reasoning.

[0057] In this embodiment, for training samples with pre-treatment images and efficacy assessment labels, the model can directly predict subsequent efficacy using pre-treatment images. For samples with pre- and post-treatment images, post-treatment images are not required input for inference but are used to form lesion-level response markers or weak response markers during the training phase. Specifically, candidate regions before and after treatment are established through spatial registration and lesion index matching. Metabolic changes in the central area, anatomical changes in the marginal area, and changes in the adjacent background are calculated, and the direction of change is mapped to lesion-level response supervision. During the inference phase, only PET and CT images at the time point to be predicted are input to output the efficacy prediction results. This embodiment uses longitudinal images to enhance training supervision but does not increase the necessary input for the clinical inference phase, thereby maintaining the basic input range of the method.

[0058] In a preferred embodiment, a local consistency check can be added before the lesion-level joint representation enters the graph node. The local consistency check includes checking the relative relationship between the metabolic response in the central region, the anatomical boundary in the edge region, and the reference response in the adjacent background region. If the metabolic response in the central region is higher than that in the adjacent background and there is a continuous anatomical boundary in the edge region, the corresponding node retains its complete features. If the metabolic response in the central region is less different from that in the adjacent background but the anatomical boundary in the edge region is obvious, the anatomical features are retained and the weight of the metabolic features is reduced. If the metabolic response in the central region is high but the anatomical boundary in the edge region is missing and the candidate region is located in the vicinity of common physiological uptake, the initial attention weight of the node is reduced. The result of the local consistency check is not used to directly delete the node, but is embedded into the graph attention deep learning network as a consistency feature in the node. This embodiment preserves the model's learning space for suspicious lesions and records whether metabolism and anatomy are consistent at the node level.

[0059] Furthermore, the efficacy prediction result can be represented as a probability distribution of multiple efficacy categories. Efficacy categories include at least two of remission, stable disease, and progression. The number of categories is set according to the training labels, which can be set based on the Remedial Evaluation Criteria for Solid Tumors (RECIST), the Percutaneous Emission Tomography-Computed Tomography Criteria for Solid Tumors (PERCIST), pathological remission results, or clinical follow-up assessment results. The classification rules for remission, stable disease, and progression under different evaluation criteria can be uniformly implemented with reference to the corresponding clinical guidelines. The efficacy classification layer outputs category scores after linearly mapping the patient-level vectors and converts them into probability distributions through a normalization function. If a binary prediction is required, remission and non-remission are output as two categories. If a multi-category prediction is required, the probabilities corresponding to remission, stable disease, and progression are output separately. The parameters of the efficacy classification layer are updated by the joint training loss. During inference, the category with the highest probability is used as the prediction result, while the probability distribution is retained for subsequent result verification. This embodiment adapts to efficacy prediction tasks under different annotation systems without changing the lesion candidate generation, cross-gating fusion, and lesion relationship graph construction processes.

[0060] In one embodiment, the intermediate attention information output by the graph attention deep learning network can be used to generate lesion contribution data. The lesion contribution data includes the node attention weight of each node, the edge attention weight between each node and other nodes, the central area feature weight, the edge area feature weight, and the adjacent background area feature weight. The lesion contribution data is stored together with the candidate region index, so that each contribution data can be mapped back to the three-dimensional position in the original PET and CT images. No additional hardware acquisition is performed on the original images during generation. Records are formed only based on the model's existing intermediate tensors and lesion spatial index. A higher node contribution indicates that the node has a higher aggregation weight before patient-level pooling. A higher edge contribution indicates that the corresponding lesion relationship participates more in message passing. This embodiment makes the key lesions and key relationships in the prediction process have localizable data, which facilitates the technical verification of whether the model depends on metabolically-anatomically consistent lesion regions.

[0061] In this embodiment, sample consistency processing is performed before the training data enters the model. Sample consistency processing includes unifying image orientation, unifying voxel coordinate origin, checking whether PET images and CT images belong to the same patient and the same examination, removing registration failure samples, correcting image cropping range, and recording the spatial transformation matrix. The spatial transformation matrix is ​​used to map the candidate region index back to the original image coordinates during inference output. If the image is missing key layers or has severe artifacts, the sample will not enter the training set. If local artifacts are located far from the lung and mediastinal regions, they will be naturally excluded through the candidate region generation process. Sample consistency processing does not change the core algorithm of the lesion relationship graph; it only provides a consistent data entry point for registration, candidate region generation, and graph node construction. This embodiment reduces the impact of inconsistent input data on deep learning training and inference and ensures the traceability of candidate region coordinates.

[0062] In a preferred embodiment, model parameter updates employ mini-batch training. Each batch contains graph structure data from multiple subjects. Since the number of lesion candidates varies for each subject, a graph batch index is used during batch processing to distinguish nodes and edges from different subjects. Message passing occurs only within the node set of the same subject and does not cross subject boundaries. The patient-level pooling layer aggregates primary lesion candidates, lymph node candidates, and metastatic lesion candidates from the same subject based on the graph batch index. The loss function is calculated at each subject level and then summed or averaged. Backpropagation updates all trainable parameters batch by batch. If a subject lacks a certain type of candidate lesion, the pooling input for that type is masked. The mask does not generate false node supervision during loss calculation. This embodiment enables patient-level graph structures with a variable number of nodes to stably enter the deep learning training process and avoids the erroneous mixing of lesion relationships between different subjects.

[0063] Furthermore, when saving the model, the image preprocessing configuration, candidate region generation rules, partition feature definitions, cross-gating parameters, graph edge encoding parameters, graph attention deep learning network parameters, and efficacy classification layer parameters are all saved simultaneously. During inference deployment, the same data processing flow is executed according to the saved configuration. If the voxel spacing or intensity distribution of the input image differs from the training configuration, resampling and standardization are performed using the preprocessing configuration. If the number of candidate regions in the input image exceeds the common range during the training phase, the candidate retention process retains representative nodes based on candidate confidence, lesion category coverage, and graph connectivity requirements. The index relationship between the retained nodes and the merged nodes is recorded in the candidate region index. This embodiment ensures that the parameters obtained from training are consistent with the inference data processing chain and allows multi-lesion cases to continue operating according to the same graph inference mechanism even when the number of nodes changes.

[0064] In this embodiment, the core working principle of the non-small cell lung cancer efficacy prediction method is as follows: PET images provide metabolic uptake distribution, and CT images provide anatomical boundary distribution. After registration and standardization, the two are used to jointly generate lesion candidate regions. The lesion candidate regions are divided into central regions, edge regions, and adjacent background regions. Deep learning coding branches extract metabolic features and anatomical features respectively. Cross-gated fusion constrains metabolic abnormalities with anatomical boundaries and guides anatomical structures with metabolic activity to form a lesion-level joint representation. The patient-level lesion relationship graph uses multiple lesion-level joint representations as nodes and uses the spatial distance between lesions, anatomical partitions, metabolic similarity, volume difference, and positional relationship relative to the primary lesion as edges. The graph attention deep learning network transmits lesion relationship information through node attention and edge attention. The patient-level pooling layer aggregates according to the lesion level, and the efficacy classification layer outputs the prediction results. This embodiment makes image input, lesion-level fusion, relationship graph construction, deep learning inference, and training constraints form a continuous technical chain, covering the complete implementation process from data preprocessing to efficacy output.

Claims

1. A method for predicting the efficacy of treatment in non-small cell lung cancer based on PET-CT deep learning, characterized in that, include: Acquire PET and CT images of the same subject, and perform spatial registration, voxel resampling, and intensity standardization on the PET and CT images; Multiple lesion candidate regions are generated based on the metabolic uptake distribution in the PET images and the anatomical boundary distribution in the CT images. PET metabolic features and CT anatomical features were extracted from the candidate lesion regions, and a joint lesion-level characterization was obtained by metabolic-anatomical cross-gated fusion. A patient-level lesion relationship diagram is constructed based on the aforementioned lesion-level joint characterization, lesion spatial location, and lesion anatomical attribution. The patient-level lesion relationship graph is input into a graph attention deep learning network to obtain the efficacy prediction results for non-small cell lung cancer.

2. The method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning according to claim 1, characterized in that, Generating the multiple candidate lesion regions includes: extracting the lung parenchyma region, mediastinal region, and soft tissue boundary region from the CT image; extracting metabolic uptake connected regions from the PET image; and performing spatial cross-matching between the soft tissue boundary region and the metabolic uptake connected regions. A candidate set containing primary lesion candidates, lymph node candidates, and metastatic lesion candidates is established for the matched region; For each candidate region in the candidate set, a center region, an edge region, and a neighboring background region are labeled, and the labeling results are used as a spatial index for subsequent feature extraction.

3. The method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning according to claim 2, characterized in that, Learning anatomical features from CT images, such as density distribution, boundary morphology, internal necrosis, and contact relationship with adjacent tissues, reflects the meaning of these features. Learning metabolic features from PET images, such as metabolic intensity, metabolic gradient, metabolic dispersion, and low uptake distribution, reflects the meaning of these features. Structural constraint weights are generated based on the anatomical features, and activity guiding weights are generated based on the metabolic features. The two types of features are then cross-gated and fused using the structural constraint weights and the activity guiding weights.

4. The method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning according to claim 3, characterized in that, Constructing the patient-level lesion relationship graph includes: using the joint representation of each lesion level as a graph node feature, and encoding the spatial distance, anatomical partition relationship, metabolic similarity, volume level difference and positional relationship relative to the primary lesion between different lesion candidate regions as graph edge features; The graph node features and graph edge features are jointly embedded to form graph structure data that includes lesion self-attributes, lesion association attributes and patient-level lesion distribution attributes, and the graph structure data is used as the input of the graph attention deep learning network.

5. The method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning according to claim 4, characterized in that, Marking the central region, edge region, and adjacent background region for each candidate region in the candidate set includes: determining the central region based on the internal uptake peak position of the metabolic uptake connected region; determining the edge region based on the soft tissue boundary and lung parenchyma contact boundary in the CT image; and determining the adjacent background region based on the tissue region located outside the edge region and spatially adjacent to the lesion candidate region. The central region, the edge region, and the adjacent background region are mapped to PET feature maps and CT feature maps, respectively, and a partitioned feature sequence is generated.

6. The method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning according to claim 5, characterized in that, The cross-gated fusion includes: scaling the boundary confidence map output by the CT coding branch with the metabolic response map output by the PET coding branch; Feature suppression weights are applied to non-boundary consistent regions in the metabolic response map based on the boundary confidence map, and feature enhancement weights are applied to active residue regions in the boundary confidence map based on the metabolic response map. The weighted metabolic features, weighted anatomical features, and regional feature sequences are recombined into channels, and the lesion-level joint characterization is obtained by attention normalization.

7. The method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning according to claim 4, characterized in that, The encoding of the graph edge features includes: converting the three-dimensional center distance, ipsilateral or contralateral lung lobe affiliation, mediastinal station affiliation, metabolic uptake distribution similarity, metabolic difference between the central area and the edge area, and lesion volume level difference between any two graph nodes into edge attribute vectors respectively. The edge attribute vector is bidirectionally spliced ​​with the joint lesion-level representation of the two graph nodes to form a direction-related lesion association embedding, and hierarchical graph connections are established according to the lesion category, from primary lesion to lymph node candidates, from primary lesion to metastatic lesion candidates, and between candidate lesions.

8. The method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning according to claim 4, characterized in that, Inputting the patient-level lesion relationship graph into a graph attention deep learning network includes: setting node attention weights for the graph node features, setting edge attention weights for the graph edge features, and inputting the node attention weights and edge attention weights together into a message passing layer; A patient-level pooling layer is set after the message passing layer. The patient-level pooling layer aggregates lesion characteristics according to the hierarchical map connection of primary lesion candidates, lymph node candidates, and metastatic lesion candidates, and inputs the aggregation results into the efficacy classification layer to generate the efficacy prediction results of non-small cell lung cancer.

9. The method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning according to claim 6, characterized in that, The attention normalization includes: calculating metabolic response attention and anatomical boundary attention for the central region, the edge region, and the adjacent background region, respectively; The metabolic response attention of the adjacent background region is used as a background reference, and the anatomical boundary attention of the edge region is used as a structural reference. The metabolic response attention of the central region is then differentially normalized. The normalized central region features, edge region features, and adjacent background region features are rearranged according to the spatial index of the lesion candidate region and input into the graph node feature generation process.

10. The method for predicting the efficacy of non-small cell lung cancer treatment based on PET-CT deep learning according to claim 8, characterized in that, The training of the graph attention deep learning network includes using patient-level efficacy labels, lesion-level response labels, and PET-CT cross-modal reconstruction labels as supervisory data. A patient-level classification loss is set for the efficacy classification layer, a lesion response consistency loss is set for the message passing layer, and a cross-modal reconstruction loss is set for the lesion-level joint representation after cross-gated fusion. The parameters of the CT coding branch, the PET coding branch, and the graph attention deep learning network are updated based on the patient-level classification loss, the lesion response consistency loss, and the cross-modal reconstruction loss.

Images