Image heterogeneity region fusion technology based on graph network for efficacy prediction method

Abstract

The present application relates to a kind of based on image network's image heterogeneity area fusion technology to curative effect prediction method and device, belong to intelligent medical treatment and medical image analysis technical field.The method first divides the tumor and peritumoral region in medical image into multiple non-connected sub-regions as graph node, and generates the comprehensive feature representation that fused local image feature and node type embedding vector for each node;Subsequently, based on the edge between nodes defined by spatial distance threshold, a graph structure containing intratumoral, peritumoral and cross-border connection is constructed;Finally, the graph structure is input into graph neural network model for training, and a curative effect prediction model is obtained.The present application explicitly models the complex spatial topological relationship between tumor heterogeneity regions through graph structure, solves the core problems of spatial information loss and insufficient heterogeneity modeling in the prior art, significantly improves the accuracy and robustness of treatment efficacy prediction.

Description

Technical Field

[0001] This invention belongs to the field of smart healthcare and medical image analysis technology, specifically relating to a method and system for intelligent tumor prognosis assessment based on graph models and tumor heterogeneity technology. This invention integrates computer vision, deep learning, and radiomics analysis methods, particularly a graph structure construction technique for spatial relationship modeling of heterogeneous tumor regions, and a multi-task prognostic prediction model based on graph neural networks. Background Technology

[0002] Precise prognostic assessment of malignant tumors (such as lymph node metastasis, pathological complete remission, and disease progression) is crucial for guiding clinical treatment decisions and achieving personalized diagnosis and treatment. Radiomics provides a powerful technical means for non-invasively and quantitatively characterizing tumor heterogeneity by extracting quantitative features from medical images (such as CT, MRI, and PET) in high throughput. However, existing radiomics-based prognostic models still face significant challenges, the core issue being insufficient ability to model the spatial complexity of tumors.

[0003] The current mainstream methods can be mainly divided into two categories:

[0004] (1) "Black box" model based on global features: This type of method usually treats the entire tumor region as a uniform whole, extracts its macroscopic radiomics features (such as first-order statistics, texture features, etc.), and directly inputs them into a machine learning classifier for prediction. This method completely ignores the significant differences in biological behavior between different subregions within the tumor (such as necrotic core, active region, edema zone, etc.) and the complex spatial interactions between these subregions. The crude "averaging" of heterogeneous information causes the model to lose key biological signals, thus limiting the upper limit of its predictive performance.

[0005] (2) Feature-based pre-fusion methods: To characterize tumor heterogeneity, some advanced studies have attempted to first segment the tumor region, identify different habitat subregions, then extract features from each subregion, and finally simply concatenate these feature vectors to form a long feature vector, which is then input into the downstream model. Although this method introduces heterogeneity information to some extent, it has a fundamental flaw:

[0006] a) Loss of spatial relationships: Simple feature splicing completely destroys the physical proximity, connectivity, and topological relationships between different habitat subregions. The model cannot know whether the "active region" is adjacent to the "necrotic core," nor can it capture the interaction information of the "tumor-peritumor boundary," which are precisely the key to revealing biological processes such as tumor invasion and immune response.

[0007] b) Rigidity modeling of heterogeneity: This method flattens complex spatial structural relationships into an unordered list of features, which cannot effectively express high-order, non-local dependencies such as "region A is indirectly related to region C through region B".

[0008] c) Insufficient integration of the peritumoral microenvironment: The biological behavior of tumors depends not only on their internal structure but also on the surrounding peritumoral microenvironment. Existing methods struggle to organically integrate the features of the peritumoral region with those of the intratumoral subregions in a structured and explicit manner; they are usually still simple splicing and cannot model cross-regional interactions at their boundaries.

[0009] In summary, the bottleneck of existing technologies lies in the lack of an effective means to explicitly model the complex spatial topological relationships between heterogeneous regions within a tumor, as well as between the tumor and its peritumoral microenvironment. This makes it difficult for models to fully extract the deep prognostic information contained in medical images, and the predictive accuracy and robustness are insufficient to meet the needs of precision medicine in clinical practice. Summary of the Invention

[0010] This invention proposes a habitat region fusion method based on a graph model, aiming to solve the core problems of spatial relationship loss and insufficient heterogeneity modeling in existing technologies, and to provide a more powerful and accurate analytical tool for tumor prognosis prediction. The technical solution is as follows:

[0011] A method for predicting treatment efficacy using image heterogeneity region fusion technology based on graph networks, characterized by the following steps:

[0012] Obtain the division results of tumor regions and their peritumoral regions in medical images, with each region corresponding to a non-connected sub-region;

[0013] Based on the partitioning results, a graph structure is constructed, where nodes represent intratumoral subregions or peritumoral regions, and edges are defined based on spatial distance thresholds.

[0014] A comprehensive feature representation is generated for each node, including local image features and node type embedding features;

[0015] The graph structure is input into a graph neural network model for training to obtain a treatment efficacy prediction model.

[0016] The predictive model is used to predict the therapeutic effects of new medical images.

[0017] Preferably, the construction graph structure includes: defining a set of nodes. Each node This corresponds to a habitat subregion or peritumoral region;

[0018] Define an edge set E based on the minimum Euclidean voxel distance between any two nodes. With distance threshold In comparison, if Then, establish an edge. .

[0019] Preferably, the type embedding feature of the node is obtained through embedding encoding, specifically:

[0020] Node type ∈ {0,1}, where 0 represents nodes in the intratumoral subregion and 1 represents nodes around the tumor; node type embedding vector ,in Indicates the node type (0 indicates intratumoral subregion, 1 indicates peritumoral node). Embed vectors for node types. This is for the embedding dimension. This process enables the model to capture the potential differences and semantic relationships between nodes at different spatial levels; it integrates node feature representations. , where the symbol " " indicates a vector concatenation operation. For the first The comprehensive characteristics of each node.

[0021] Preferably, the edges include the following types: intratumoral subregion edges: connecting different habitat subregions within the tumor; peritumoral region edges: connecting different peritumoral regions; intratumoral-peritumoral interaction edges: connecting intratumoral subregion nodes and peritumoral nodes.

[0022] Preferably, the training of the graph neural network model adopts an end-to-end optimization strategy, using the Adam optimizer and combining it with a weight decay mechanism.

[0023] Preferably, the learning rate in the training adopts a cosine annealing decay strategy, and its learning rate... At the maximum value and minimum value Between based on the current training cycle With total cycle Make dynamic adjustments.

[0024] Preferably, the classification task in the training uses Focal Loss as the loss function, and the Focal Loss includes a balancing parameter α for balancing the weights of positive and negative samples and a modulation factor γ for adjusting the weights of easy and difficult samples.

[0025] Preferably, the training employs an early stopping strategy, terminating training when the performance metrics of the validation set show no improvement over multiple consecutive training rounds.

[0026] Preferably, the output layer of the prediction model includes a fully connected network and a Softmax function to transform the logit value into a probability distribution.

[0027] This invention also discloses a device for predicting treatment efficacy using image heterogeneity region fusion technology based on graph networks, characterized in that it includes:

[0028] The graph construction module is used to construct a graph structure based on the division results of the tumor region and its peritumoral region in medical images, where nodes represent intratumoral subregions or peritumoral regions, and edges are defined based on spatial distance thresholds.

[0029] The feature fusion module is used to generate a comprehensive feature representation for each node, including local image features and node type embedding features;

[0030] The model training module is used to train the graph neural network model by inputting the graph structure into it, so as to obtain the treatment efficacy prediction model.

[0031] The prediction module is used to predict the therapeutic effect of new medical images using the prediction model.

[0032] The present invention also discloses a non-volatile storage medium, characterized in that the non-volatile storage medium includes a stored program, wherein the program, when running, controls the device where the non-volatile storage medium is located to execute the above-described method.

[0033] The present invention also discloses a terminal device, characterized in that the terminal device includes: a processor, a memory, a communication interface, and a bus; the processor, the memory, and the communication interface are connected through the bus and communicate with each other; the memory stores executable program code; the processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to execute the method described above.

[0034] Beneficial effects

[0035] (1) Achieved accurate quantitative modeling of tumor spatial heterogeneity: By defining intratumoral heterogeneous subregions and peritumoral regions as graph nodes and constructing edges based on spatial Euclidean distance thresholds, this invention transforms the complex spatial topology of the tumor and its microenvironment into graph data. This enables the model to explicitly capture the spatial adjacency and interaction relationships between different regions, overcoming the spatial information loss problem caused by existing "black box" models or simple feature splicing methods.

[0036] (2) Improved the model's ability to distinguish and fuse regional semantic differences: By introducing node type embedding and stitching it with local image features, the model can automatically learn and distinguish the potential differences in function and biological behavior between the "intratumoral subregion" and the "peritumoral region". This fusion representation method enables the features input to the graph neural network to simultaneously include the physical characteristics and spatial semantic information of the region, laying the foundation for subsequent accurate analysis.

[0037] (3) Enhanced accuracy and robustness of prognostic prediction models: The heterogeneous region fusion graph model framework constructed based on the aforementioned technical means can deeply integrate the interaction information between intratumoral heterogeneity and the peritumoral microenvironment through the message passing mechanism of graph neural networks. This enables the model to uncover deep imaging patterns closely related to treatment efficacy, thus demonstrating higher predictive accuracy and generalization ability (i.e., robustness) compared to traditional methods in clinical prognostic prediction tasks (such as efficacy assessment). Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall structure of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The specific embodiments described herein are merely illustrative and are not intended to limit the scope of the invention.

[0040] Due to the unique advantages of graph models, graph-structured data is naturally suitable for modeling non-Euclidean spatial relationships. By defining each habitat subregion and peritumoral region as a graph node, and their spatial adjacency and interactions as edges, a mathematical representation that fully preserves the spatial topology of the tumor can be constructed. Based on this graph structure, graph neural networks can efficiently learn the spatial dependencies and functional interactions between nodes through message passing and feature aggregation mechanisms, thereby achieving a deep analysis of tumor heterogeneity and microenvironment interactions.

[0041] Prerequisites for this invention: The intratumoral heterogeneous regions and peritumoral regions of the image have already been generated using existing methods (e.g., superpixel segmentation, K-means clustering, or deep learning segmentation models). Each region corresponds to a disconnected sub-region, which serves as a graph node, and the corresponding features have been extracted. We denote... It includes first-order statistical features (intensity distribution), morphological features (geometric structure), and texture features (grayscale patterns and spatial organization), which are used to describe the local image features and tissue microstructure of the region.

[0042] See Figure 1 As shown, this invention discloses a method for predicting therapeutic efficacy using image heterogeneity region fusion technology based on graph networks, comprising the following steps:

[0043] (1) Obtain the division results of the tumor region and its peritumoral region in medical images, with each region corresponding to a non-connected sub-region;

[0044] Based on the habitat segmentation results, a graph structure model is constructed with habitat subregions and peritumoral regions as nodes and spatial interactions as edges. This graph structure can capture the spatial dependencies and functional interactions between different regions while preserving local image features, providing structured input for subsequent graph convolutional learning.

[0045] First, the tumor ROI and its peritumoral extension area are divided into multiple disconnected sub-regions, each of which is treated as an independent node in the graph. Let the set of nodes be...

[0046] (1)

[0047] Each node Each node corresponds to a local habitat unit. Each node not only contains radiomics features within the region, but also carries its semantic category information (i.e., node type).

[0048] To achieve a unified representation of different source regions (intratumoral subregion and peritumoral region) in the feature space, node type information is embedded and encoded, and then jointly represented with the image features of the region to form a comprehensive feature representation of the node.

[0049] • Node Type Embedding:

[0050] Node types are used to identify the spatial origin of the region, and are divided into two categories: intratumoral subregion nodes and peritumoral nodes. To enable the model to automatically learn the semantic differences between different node types during training, a learnable embedding vector is assigned to each node type:

[0051] (2)

[0052] in Indicates the node type (0 indicates intratumoral subregion, 1 indicates peritumoral node). Embed vectors for node types. This is the embedding dimension. This process enables the model to capture the potential differences and semantic relationships between nodes at different spatial levels.

[0053] • Radiomic Features:

[0054] The local feature vector of each node is composed of the radiomics features of the region, denoted as:

[0055] (3)

[0056] • Combined Node Representation:

[0057] To fuse the semantic information of node types with regional image features, the two are concatenated and standardized to obtain the final comprehensive feature representation:

[0058] (4)

[0059] The symbol " " indicates a vector concatenation operation. For the first The comprehensive characteristics of each node.

[0060] (2) Based on the division results, construct a graph structure, where nodes represent intratumoral subregions or peritumoral regions, and edges are defined based on spatial distance thresholds.

[0061] The edge relationships between nodes are established based on the assumptions of spatial proximity and organizational interaction. A spatial distance threshold is set. If the nearest voxel distance between any two disconnected regions is less than (For example If individual elements are considered, then a potential biological connection is assumed between them, thus establishing an undirected edge between them. The formal definition is as follows:

[0062] (14)

[0063] in Represents a node and The minimum Euclidean voxel distance. Based on the spatial type of the nodes, edges can be classified into three categories:

[0064] Intra-tumoral edges: Spatial neighborhoods connecting different habitat subregions within a tumor, used to characterize the continuity and internal dependence of heterogeneous structures within the tumor.

[0065] Peritumoral edges: connect different peritumoral rings or adjacent regions, reflecting the spatial coupling relationship of the tissue environment surrounding the tumor.

[0066] Intratumoral-peritumoral cross-boundary edges: connect intratumoral subregion nodes with adjacent peritumoral nodes, and are used to capture cross-boundary information exchange and microenvironmental interactions in the tumor edge region.

[0067] (3) The graph structure is input into the graph neural network model for training to obtain the treatment efficacy prediction model.

[0068] Based on the above definitions of nodes and edges, construct a complete graph structure:

[0069] (15)

[0070] in, For a set of nodes, Let be the set of edges. This is a node attribute matrix. This graph not only preserves the spatial structural features of tumor images, but also makes the topological relationships between heterogeneous regions explicit.

[0071] (4) Training and evaluation of efficacy prediction model: The graph structure is input into the graph neural network model for training to obtain the efficacy prediction model; the prediction model is used to predict the efficacy of new medical images.

[0072] Output and prediction logic:

[0073] Prognostic efficacy is assessed using features output from a graphical model, with a fully connected network (MLP) introduced into the climate data, corresponding to specific treatment efficacy tasks. The output of each branch is transformed into a probability distribution using a softmax function.

[0074] (16)

[0075] in For the sample Category The logit value is used, and the classification loss function is defined as cross-entropy loss.

[0076] (17)

[0077] in For a specific task, we'll take efficacy prediction as an example. A multi-task model could also be introduced. Indicates task The number of samples.

[0078] Model training strategy:

[0079] The model training employs an end-to-end optimization strategy. The optimizer used is Adam (β1=0.9, β2=0.999), combined with a weight decay mechanism to prevent overfitting and improve generalization performance. The learning rate is dynamically adjusted using a cosine annealing strategy, ensuring rapid convergence in the early stages of training and smooth fine-tuning in the later stages. The specific formula is as follows:

[0080] (18)

[0081] in , , A complete learning rate cycle.

[0082] To address the issue of class imbalance in multi-task scenarios, the classification task employs Focal Loss as the loss function, assigning higher weights to hard-to-classify samples to improve model robustness.

[0083] (19)

[0084] in , , This represents the predicted probability of the sample. The survival task still uses Cox partial likelihood loss.

[0085] The training batch size was set to 32, and 5-fold cross-validation was used to evaluate generalization performance in each iteration. Dropout (0.3) was introduced in both the multi-head attention and feedforward layers of the Transformer to suppress overfitting. To avoid the model getting overtrained, an Early Stopping strategy was applied: training was terminated early when the validation set AUC showed no significant improvement for 10 consecutive iterations, ensuring optimal generalization performance.

[0086] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for predicting therapeutic efficacy using image heterogeneity region fusion technology based on graph networks, aiming to solve the core problems of lost spatial relationships and insufficient heterogeneity modeling, characterized in that... Includes the following steps: Obtain the division results of tumor regions and their peritumoral regions in medical images, with each region corresponding to a non-connected sub-region; Based on the partitioning results, a graph structure is constructed, where nodes represent intratumoral subregions or peritumoral regions, and edges are defined based on spatial distance thresholds. A comprehensive feature representation is generated for each node, including local image features and node type embedding features; The graph structure is input into a graph neural network model for training to obtain a treatment efficacy prediction model. The predictive model is used to predict the therapeutic effect of new medical images. The construction graph structure includes: Define a set of nodes Each node This corresponds to a habitat subregion or peritumoral region; Define an edge set E based on the minimum Euclidean voxel distance between any two nodes. With distance threshold In comparison, if Then, establish an edge. ; The type embedding feature of the node is obtained through embedding encoding, specifically: Node type ∈ {0,1}, where 0 represents nodes in the intratumoral subregion and 1 represents nodes around the tumor; Node type embedding vector ,in Indicates the node type: 0 represents an intratumoral subregion, and 1 represents a peritumoral node. Embed vectors for node types. This is an embedding dimension; the process enables the model to capture the potential differences and semantic relationships between nodes at different spatial levels. Comprehensive node feature representation , where the symbol " " indicates a vector concatenation operation. For the first The comprehensive characteristics of each node; The edges include the following types: Intratumoral subregion boundaries: Spatial neighborhoods connecting different habitat subregions within the tumor, used to characterize the continuity and internal dependence of heterogeneous structures within the tumor; Peritumoral region boundary: connects different peritumoral rings or adjacent regions, reflecting the spatial coupling relationship of the tissue environment surrounding the tumor; Intratumoral-peritumoral interaction edge: connects intratumoral subregion nodes with adjacent peritumoral nodes, used to capture cross-boundary information exchange and microenvironmental interactions in the tumor edge region.

2. The method according to claim 1, characterized in that, The training of the graph neural network model adopts an end-to-end optimization strategy, using the Adam optimizer and combining it with a weight decay mechanism.

3. The method according to claim 2, characterized in that, The training process employs a cosine annealing decay strategy for the learning rate. At the maximum value and minimum value Between based on the current training cycle With total cycle Make dynamic adjustments.

4. The method according to claim 3, characterized in that, The classification task in the training uses Focal Loss as the loss function, which includes a balancing parameter α for balancing the weights of positive and negative samples and a modulation factor γ for adjusting the weights of easy and difficult samples.

5. The method according to claim 4, characterized in that, The training employs an early stopping strategy, terminating training when the validation set performance metrics show no improvement over multiple consecutive training rounds.

6. The method according to claim 1, characterized in that, The output layer of the prediction model includes a fully connected network and a Softmax function, which is used to transform the logit value into a probability distribution.

7. A device for predicting treatment efficacy using image heterogeneity region fusion technology based on graph networks, wherein the device applies the method described in claim 1, characterized in that, include: The graph construction module is used to construct a graph structure based on the division results of the tumor region and its peritumoral region in medical images, where nodes represent intratumoral subregions or peritumoral regions, and edges are defined based on spatial distance thresholds. The feature fusion module is used to generate a comprehensive feature representation for each node, including local image features and node type embedding features; The model training module is used to train the graph neural network model by inputting the graph structure into it, so as to obtain the treatment efficacy prediction model. The prediction module is used to predict the therapeutic effect of new medical images using the prediction model.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein the program, when executed, controls the device where the non-volatile storage medium is located to perform the method described in any one of claims 1 to 6.

9. A terminal device, characterized in that, The terminal device includes: a processor, a memory, a communication interface, and a bus; the processor, the memory, and the communication interface are connected through the bus and communicate with each other; the memory stores executable program code; the processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to perform the method as described in any one of claims 1-6 above.

Images