Multi-modal medical image deep learning enhanced segmentation system and method thereof

Abstract

The invention relates to a multi-modal medical image deep learning enhanced segmentation system and a method thereof, belongs to the field of medical image processing, and aims to solve the problem of inaccurate brain glioma target region boundary segmentation. According to the technical scheme, the method comprises the steps that a metabolic domain PET image and a structural domain MRI-T2 image are obtained, and image registration, noise suppression and intensity standardization preprocessing are carried out; glycolysis features and texture anisotropy features are extracted through a double-domain feature decoupler; constructing a graph neural network to generate a metabolism-structure interaction graph; calculating a pathological infiltration confidence coefficient thermodynamic diagram based on the atlas; the thermodynamic diagram is used for self-adaptive radiotherapy dose segmentation, and the dose gradient is dynamically adjusted; and finally outputting the tumor target boundary.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and in particular to a deep learning-enhanced segmentation system and method for multimodal medical images. Background Technology

[0002] Gliomas are common malignant tumors of the central nervous system. Their highly invasive growth characteristics make precise delineation of the radiotherapy target area (including the tumor core and surrounding infiltration zone) a key challenge and a determinant of efficacy in clinical treatment. Due to the blurred boundaries between the tumor and normal brain tissue, traditional imaging methods (such as MRI-T2 and PET) struggle to accurately distinguish between the tumor infiltration area and the edema area, leading to a significant discrepancy between radiotherapy target planning and dose distribution. Furthermore, the lack of quantitative tools to characterize the dynamic relationship between metabolic activity and anatomical structure further limits the development of individualized treatment plans.

[0003] In existing technologies, multimodal medical image segmentation methods (such as multimodal 3D convolutional neural networks and superpixel segmentation algorithms) have been widely used for the automatic identification and boundary delineation of gliomas. However, these methods generally rely on local feature extraction from single-modal images or simple multimodal data fusion strategies, failing to effectively separate the independent feature spaces of the metabolic domain (such as FDG uptake values ​​from PET) and the structural domain (such as texture features from MRI-T2). For example, while U-Net-based segmentation models can combine T1 and T2 weighted image information, they lack a cross-domain mapping mechanism for metabolically active regions and structurally abnormal regions; while attempts based on graph neural networks (GNNs) struggle to generate high-confidence pathological invasion probability distributions due to unclear node attribute definitions or inaccurate adjacency relationship construction. Furthermore, existing radiotherapy dose planning methods often use static segmentation results as input, neglecting the dynamic characteristics of metabolic-structural conflict regions, leading to a mismatch between the dose gradient and the actual tumor invasion range.

[0004] The aforementioned technical deficiencies directly lead to key problems in clinical diagnosis and treatment:

[0005] First, the separation of metabolic and structural domain information makes it impossible to quantify the pathological differences between tumor infiltration and edema, resulting in blurred target area boundaries.

[0006] In addition, existing technologies have failed to establish an accurate model for the transfer of metabolic features to the structural domain space, which makes it impossible to accurately assess the probability of overlap between metabolically active regions and structurally abnormal regions, leading to contradictions in radiotherapy dose distribution. Summary of the Invention

[0007] This invention provides a multimodal medical image deep learning-enhanced segmentation system and method, and the technical problem to be solved is: how to improve the accurate segmentation of the target area boundary of glioma.

[0008] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0009] On the one hand, multimodal medical image deep learning-enhanced segmentation systems,

[0010] The data acquisition module is used to acquire metabolic domain images and structural domain images of the target region;

[0011] The preprocessing module is used to perform image registration, noise suppression, and intensity normalization on the metabolic domain image and the structural domain image. It uses a non-rigid registration algorithm to generate the registered metabolic domain image and the structural domain image.

[0012] The dual-domain feature decoupling module is used to extract metabolic and structural features from the registered metabolic and structural domain images using a dual-domain feature decoupling device, and to align the metabolic and structural features using a cross-attention mechanism.

[0013] The inter-domain infiltration probability transfer module is used to construct an inter-domain infiltration probability transfer model. It uses a graph neural network (GNN) to map metabolic features to structural features in the structural domain, generating a metabolic-structural interaction graph. The node attributes are the cluster centers of metabolic features, and the spatial neighborhood similarity of the structural domain image is used as the edge weight.

[0014] The heatmap generation module is used to generate a pathological infiltration confidence heatmap based on the metabolism-structure interaction map. The heatmap outputs a tumor infiltration probability distribution by quantifying the overlap probability between metabolically active regions and structurally abnormal regions.

[0015] The radiotherapy dose planning module is used to take the heat map as input for adaptive radiotherapy dose segmentation and dynamically adjust the dose gradient, wherein high probability areas correspond to high dose prescriptions and low probability areas correspond to low dose prescriptions.

[0016] The results output module is used to output the final segmentation results, including the tumor target area boundary, metabolic-structural conflict region marking, and dose distribution scheme.

[0017] Furthermore, the dual-domain feature decoupling module includes:

[0018] S3a: Metabolic feature branch, which uses 3DCNN to extract metabolic features from the registered metabolic domain image;

[0019] S3b: Structural feature branch, which uses the U-Net architecture to extract structural features from the registered structural domain image MRI-T2;

[0020] S3c: Cross-modal attention module, which aligns metabolic and structural features through a cross-attention mechanism, as shown in the formula:

[0021]

[0022] In this context, the query matrix Q is derived from metabolic features, while the key matrix K and value matrix V are derived from structural features. It is the dimension of K.

[0023] Furthermore, the inter-domain infiltration probability migration module includes:

[0024] Graph structure building unit, used to define graph structure, where nodes are cluster centers of metabolic features in metabolic domain images divided by K-means clustering, and spatial neighborhood similarity of MRI-T2 in structural domain images is used as edge weight;

[0025] The feature propagation unit is used to propagate features of the interaction graph using a graph convolutional network (GCN), and to aggregate the PET metabolic features of nodes by weighting the edge weights to generate new node features that integrate local spatial neighborhood information.

[0026] The consistency constraint unit is used to introduce a cross-domain consistency constraint term into the loss function, and its formula is as follows:

[0027]

[0028] in, Metabolic feature nodes, For structural feature nodes, This represents the weighting of the metabolism-structure association.

[0029] Furthermore, the heatmap generation module includes:

[0030] The probability calculation unit is used to calculate the metabolic-structural overlap probability of each voxel based on the metabolic-structural interaction map, using the following formula:

[0031]

[0032] In the formula: Pinvasion(v) is the tumor invasion probability of voxel v; SUV(v) is the standardized uptake value of voxel v, reflecting the metabolic activity level, obtained from PET images; FA(v) is the anisotropy fraction of voxel v, characterizing tissue structural integrity, obtained based on MRI diffusion tensor technology; SUVmax and FAmax are the global maximum standardized uptake value and the global maximum anisotropy fraction, respectively, used for probability normalization.

[0033] The partitioning unit is used to divide the probability distribution into high, medium, and low risk areas through threshold segmentation to obtain a heat map; the threshold is dynamically adjusted based on clinical pathology data.

[0034] The visualization unit is used to fuse heatmaps with raw image data to generate three-dimensional visualization results for doctors to interactively correct.

[0035] Furthermore, the radiotherapy dose planning module includes:

[0036] The dose calculation unit is used to set the dose gradient function based on the high-probability regions of the heatmap. ,in As the baseline dose, The attenuation coefficient;

[0037] An optimization unit is used to introduce Monte Carlo simulation optimization in low-probability regions and adjust the dose distribution based on the tumor edge diffusion model.

[0038] The validation unit is used to output a dose distribution scheme that meets the ICRU dose constraint standard and is physically feasible through Monte Carlo validation.

[0039] Furthermore, the data acquisition module includes:

[0040] The PET acquisition unit is used to acquire FDG images through a PET device;

[0041] The MRI acquisition unit is used to acquire T2-weighted sequences using a 3T MRI device.

[0042] Furthermore, the preprocessing module includes:

[0043] The registration unit is used to spatially align PET and MRI images using a non-rigid registration algorithm;

[0044] The intensity correction unit is used to apply N4ITK field correction to PET images and perform histogram equalization on MRI images.

[0045] The denoising unit is used to suppress random noise in PET images by wavelet thresholding while preserving metabolic hotspot details.

[0046] Secondly, the method for a multimodal medical image deep learning-enhanced segmentation system includes the following steps:

[0047] Acquire medical imaging data of the target region, including metabolic domain images (PET) and structural domain images (MRI-T2).

[0048] Image registration, noise suppression, and intensity normalization were performed on PET and MRI-T2 images to generate registered metabolic and structural domain images.

[0049] Metabolic and structural features are extracted using a dual-domain feature decoupler, where metabolic features include glycolysis features and structural features include texture anisotropy features.

[0050] A probabilistic transfer model for interdomain infiltration was constructed. A graph neural network (GNN) was used to map metabolic features to the structural domain space to generate a metabolic-structural interaction map, in which the node attributes are metabolic features and the adjacency relationship is based on the spatial neighborhood topology of MRI-T2 images.

[0051] Based on the interactive atlas, a pathological invasion confidence heatmap is generated. The heatmap outputs the tumor invasion probability distribution by quantifying the overlap probability between metabolically active regions and structurally abnormal regions.

[0052] The heatmap is used as input for adaptive radiotherapy dose segmentation to dynamically adjust the dose gradient, where high-probability regions correspond to high-dose prescriptions and low-probability regions correspond to low-dose prescriptions.

[0053] The final segmentation results are output, including tumor target boundary, metabolic-structural conflict region marking, and dose distribution scheme.

[0054] The beneficial effects of this invention are as follows:

[0055] 1. Improved target boundary segmentation: A dual-domain feature decoupler extracts metabolic features from PET (such as glycolysis features SUV, MTV, TLG) and structural features from MRI-T2 (such as texture anisotropy features FA value, gray-level co-occurrence matrix parameters), and uses a cross-attention mechanism to align the dual-domain feature spaces, avoiding the separation of metabolic and structural domain information. A graph neural network (GNN) constructs an interactive graph with metabolic features as nodes and MRI-T2 spatial topology as adjacency relationships. Cross-domain consistency constraints and multi-hop aggregation strategies achieve deep fusion of cross-domain features. Combined with the pathological invasion confidence heatmap to quantify the probability of metabolic-structural overlap (such as normalization calculation based on SUV and FA), the tumor invasion area and edema area can be accurately distinguished. Then, through 3D visualization and interactive correction with doctors, the accuracy and clarity of tumor target boundary segmentation are significantly improved, solving the problem of blurred target boundaries in traditional methods.

[0056] 2. Optimize the distribution of radiotherapy dose: Using the pathological invasion confidence heatmap as input, a dose gradient function is used to dynamically adjust the dose distribution so that high-probability tumor invasion areas correspond to high doses and low-probability areas correspond to low doses, thus matching the dose gradient with the actual tumor invasion range. In low-probability areas, Monte Carlo simulation optimization is introduced, and the dose distribution is adjusted in combination with the tumor edge diffusion model. At the same time, the ICRU dose constraint standard is met and the physical feasibility is ensured through Monte Carlo verification. This not only ensures dose coverage in high-risk areas, but also reduces the radiation dose to normal tissues, effectively solving the contradiction between dose distribution and tumor invasion range in traditional static dose planning, and optimizing the accuracy and safety of radiotherapy dose. Attached Figure Description

[0057] Figure 1This is a logical schematic diagram of the method of the present invention;

[0058] Figure 2 This is a technical architecture diagram of the system in this application;

[0059] Figure 3 This is a schematic diagram of the preprocessing process;

[0060] Figure 4 This is a schematic diagram of a dual-domain feature decoupling module. Detailed Implementation

[0061] In order to make the content of the present invention easier to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described.

[0062] Example 1

[0063] A deep learning-based augmentation segmentation method for multimodal medical images, characterized by the following steps:

[0064] S1: Acquire medical imaging data of the target area, including metabolic domain image PET (i.e., PET image) and structural domain image MRI-T2 (i.e., MRI-T2 image).

[0065] Specifically, metabolic domain imaging (PET) is used to characterize metabolic activity, while structural domain imaging (MRI-T2) is used to characterize anatomical structures. Metabolic domain imaging (PET) captures the glycolytic activity of tumor cells using radioactive tracers (such as FDG), with high-signal areas reflecting metabolically active tumor tissue; structural domain imaging (MRI-T2) displays the range of abnormal signals in brain tissue through the diffusion properties of water molecules.

[0066] S2: Preprocess the metabolic domain image (PET) and structural domain image (MRI-T2), including image registration, noise suppression and intensity normalization, and generate registered metabolic domain image and structural domain image using a non-rigid registration algorithm.

[0067] The image preprocessing in S2 includes the following steps:

[0068] S21: Spatial alignment of metabolic domain images (PET) and structural domain images (MRI) is performed using non-rigid registration algorithms (such as ANTs);

[0069] S22: Apply N4ITK field correction to PET images and perform histogram equalization on MRI images;

[0070] S23: Suppress random noise in PET images by using wavelet thresholding to preserve metabolic hotspot details.

[0071] The preprocessing steps ensure the spatial and intensity comparability of the metabolic domain image and the structural domain image, providing high-quality input for subsequent feature extraction.

[0072] Specifically, in step S2, the process of preprocessing the metabolic domain image (PET) and structural domain image (MRI-T2) to generate registered metabolic domain and structural domain images is as follows:

[0073] First, spatial discrepancies between PET and MRI-T2 images need to be eliminated using image registration techniques. The non-rigid registration algorithm ANTs is used during registration. After registration, the PET image is adjusted to a spatial coordinate system consistent with the MRI-T2 image, ensuring complete spatial alignment between the two.

[0074] Next, noise suppression processing is required for PET and MRI-T2 images. For PET images, wavelet thresholding denoising is used to reduce the impact of random noise by decomposing high-frequency noise components and preserving low-frequency metabolic hotspot features. For MRI-T2 images, nonlocal mean filtering is employed to smooth noise using redundant information from similar regions in the image, while preserving edge details of anatomical structures. Simultaneously, to address intensity inhomogeneities in MRI-T2 images (such as grayscale drift caused by magnetic field inhomogeneities), the N4ITK bias correction tool is used to model and remove intensity biases, thereby improving image contrast.

[0075] Finally, intensity normalization is required for PET and MRI-T2 images to eliminate inter-device differences. For PET images, Z-score normalization is typically used to adjust the distribution of metabolic features to a standard normal distribution range, facilitating subsequent feature extraction. For MRI-T2 images, Min-Max normalization is used to map gray values ​​to a fixed interval (e.g., 0 to 1) to ensure the comparability of texture features (e.g., gray-level co-occurrence matrix contrast). After completing these steps, the final registered metabolic and structural domain images will possess consistent spatial alignment, low noise levels, and a uniform intensity range, providing high-quality input for subsequent dual-domain feature extraction and cross-modal interactive modeling.

[0076] The entire preprocessing workflow addresses the spatial, intensity, and noise level differences between PET and MRI-T2 images through the synergistic effect of multimodal registration, noise suppression, and intensity normalization, ensuring that subsequent analysis can be based on consistent medical imaging data.

[0077] S3: Metabolic and structural features are extracted from the registered metabolic domain image (PET) and structural domain image (MRI-T2) using a dual-domain feature decoupler, and the metabolic and structural features are aligned using a cross-attention mechanism. The metabolic features include glycolysis features (FDG uptake value, metabolic volume MTV), and the structural features include texture anisotropy features (gray-level co-occurrence matrix contrast, diffusion tensor imaging FA value).

[0078] The dual-domain feature decoupler further includes:

[0079] S3a: Metabolic feature branch, which uses 3DCNN to extract metabolic features from the registered metabolic domain image;

[0080] S3b: Structural feature branch, which uses the U-Net architecture to extract structural features from the registered structural domain image MRI-T2;

[0081] S3c: Cross-modal attention module, which aligns metabolic and structural features through a cross-attention mechanism, the formula of which is:

[0082]

[0083] The query matrix Q is derived from metabolic features, while the key matrix K and value matrix V are derived from structural features. It is the dimension of K.

[0084] The dual-domain feature decoupler achieves collaborative representation of metabolic and structural features through a dual-branch network (metabolic feature branch and structural feature branch) and an attention mechanism.

[0085] S4: Construct an inter-domain infiltration probability transfer model, using a graph neural network to map metabolic features to structural features in the structural domain, generating a metabolism-structure interaction map, where node attributes are the cluster centers of metabolic features, and the spatial neighborhood similarity of the structural domain image MRI-T2 is used as the edge weight.

[0086] Specifically, the construction of the graph neural network (GNN) in S4 includes the following steps:

[0087] S41: Define a graph structure where nodes are cluster centers of metabolic features in the metabolic domain image divided by K-means clustering, and the spatial neighborhood similarity of the structural domain image MRI-T2 is used as edge weights; where nodes are divided into metabolic hotspot regions by K-means clustering, and edge weights quantify spatial correlation by the neighborhood similarity of MRI-T2 (such as Euclidean distance or mutual information), thus constructing the basic topology of the metabolism-structure interaction graph.

[0088] S42: A graph convolutional network (GCN) is used to propagate features of the interaction graph. The PET metabolic features of the nodes are weighted and aggregated by edge weights to generate new node features that integrate local spatial neighborhood information.

[0089] S43: Introduce a cross-domain consistency constraint term into the loss function, the formula is:

[0090]

[0091] in, Metabolic feature nodes, For structural feature nodes, Representing metabolic feature nodes eigenvectors, Represents structural feature nodes eigenvectors, The edge weights are calculated based on the spatial neighborhood similarity of MRI-T2. The cross-domain consistency constraint term ensures that the interaction map output by the graph neural network conforms to the biological laws of tumor invasion, enhancing the model's generalization ability.

[0092] S5: Generate a pathological infiltration confidence heatmap based on the metabolism-structure interaction map. The heatmap generates the tumor infiltration probability by quantifying the overlap probability between metabolically active regions and structurally abnormal regions.

[0093] Specifically, the generation of the pathological infiltration confidence heatmap in S5 includes the following steps:

[0094] S51: Based on the metabolism-structure interaction map, calculate the metabolism-structure overlap probability of each voxel, using the following formula:

[0095]

[0096] In the formula: voxels The probability of tumor invasion; voxels Standardized uptake values, reflecting metabolic activity levels, are obtained from PET images; voxels The anisotropy fraction, characterizing tissue structure integrity, was obtained using diffusion tensor technology based on MRI. and These are the global maximum standardized uptake value and the global maximum anisotropy score, respectively, used for probability normalization.

[0097] S52: The probability distribution is divided into high, medium and low risk areas by threshold segmentation to obtain a heat map; the threshold is dynamically adjusted based on clinical pathological data (such as Ki-67 index); wherein, the threshold is dynamically generated by training a regression model with clinical pathological data (such as Ki-67 proliferation index) to ensure the clinical relevance of the heat map.

[0098] S53: The heatmap is fused with the original image data to generate a visual result for interactive correction by doctors. The heatmap generation process combines metabolic activity and structural heterogeneity, avoiding the bias of single-modal segmentation.

[0099] S6: The heat map is used as the input for adaptive radiotherapy dose segmentation to dynamically adjust the dose gradient, wherein high probability regions correspond to high dose prescriptions and low probability regions correspond to low dose prescriptions.

[0100] The adaptive radiotherapy dose segmentation in S6 includes the following steps:

[0101] S61: Set the dose gradient function based on the heat map. ,in As the baseline dose, The attenuation coefficient is denoted as . The dose gradient function follows an exponential attenuation model ( ). To achieve dose-priority coverage in high-probability areas, for example... At that time, the high probability area ( The dosage is Low probability area ( The dosage is To balance target control with toxicity risks.

[0102] S62: Monte Carlo simulation optimization is introduced in low-probability regions to adjust the dose distribution based on the tumor edge diffusion model. Specifically, Monte Carlo simulation uses a particle tracking algorithm to simulate the interaction between photons and matter, optimizing the dose distribution in low-probability regions. For example, in the tumor edge diffusion model, the dose gradient is adjusted through a probability diffusion equation.

[0103] S63: Output dose distribution scheme to meet ICRU dose constraint standards and ensure physical feasibility through Monte Carlo verification. The ICRU standard requires target area coverage of at least 95% of the prescribed dose, and the dose received by normal tissues not exceeding tolerance limits. Monte Carlo verification verifies the physical feasibility of the dose distribution by simulating the radiation transmission process, avoiding errors caused by model simplification. In practice, the integrated dose distribution is first verified according to ICRU dose constraint standards (such as the ICRU-83 report): target area dose coverage must meet the requirement of ≥95% of the target volume receiving the prescribed dose (e.g., 60 Gy in the high-probability region), and the maximum dose in the target area must not exceed 110% of the prescribed dose; normal tissues must meet tolerance dose limits. If the constraints are not met, return to S62 to readjust the dose in the low-probability region. After the verification is passed, Monte Carlo verification is performed: using the same particle transport algorithm as the treatment planning system (such as EGSnrc or FLUKA), the final dose distribution is independently simulated and calculated, and the difference between the simulated dose and the planned dose is compared. The voxel dose deviation is required to be ≤2%, and the dose distribution is ensured to be physically achievable (such as hardware support such as X-ray beam modulation and multi-leaf collimator movement).

[0104] S7: Determine the tumor target area boundary by the intersection of high-probability regions in the heatmap and the MRI-T2 structural domain;

[0105] The final segmentation results are output, including tumor target boundary, metabolic-structural conflict region marking, and dose distribution scheme.

[0106] In the method, the dual-domain feature decoupler extracts deep features from PET and MRI-T2 images respectively through a convolutional neural network (CNN) and uses an attention mechanism to align the dual-domain feature space; the graph neural network realizes cross-domain feature interaction through a hierarchical aggregation strategy (such as GraphSAGE), and its loss function includes a metabolic-structural correlation constraint term.

[0107] Example 2

[0108] Based on the same inventive concept as the multimodal medical image deep learning enhanced segmentation method in the foregoing embodiments, Embodiment 2 of this application provides a multimodal medical image deep learning enhanced segmentation system.

[0109] Multimodal medical image deep learning augmentation segmentation system

[0110] The data acquisition module is used to acquire metabolic domain images and structural domain images of the target region;

[0111] The preprocessing module is used to perform image registration, noise suppression, and intensity normalization on the metabolic domain image and the structural domain image. It uses a non-rigid registration algorithm to generate the registered metabolic domain image and the structural domain image.

[0112] The dual-domain feature decoupling module is used to extract metabolic and structural features from the registered metabolic and structural domain images using a dual-domain feature decoupling device, and to align the metabolic and structural features using a cross-attention mechanism.

[0113] The inter-domain infiltration probability transfer module is used to construct an inter-domain infiltration probability transfer model. It uses a graph neural network (GNN) to map metabolic features to structural features in the structural domain, generating a metabolic-structural interaction graph. The node attributes are the cluster centers of metabolic features, and the spatial neighborhood similarity of the structural domain image is used as the edge weight.

[0114] The heatmap generation module is used to generate a pathological infiltration confidence heatmap based on the metabolism-structure interaction map. The heatmap outputs a tumor infiltration probability distribution by quantifying the overlap probability between metabolically active regions and structurally abnormal regions.

[0115] The radiotherapy dose planning module is used to take the heat map as input for adaptive radiotherapy dose segmentation and dynamically adjust the dose gradient, wherein high probability areas correspond to high dose prescriptions and low probability areas correspond to low dose prescriptions.

[0116] The results output module is used to output the final segmentation results, including the tumor target area boundary, metabolic-structural conflict region marking, and dose distribution scheme.

[0117] Preferably, the dual-domain feature decoupling module includes:

[0118] S3a: Metabolic feature branch, which uses 3DCNN to extract metabolic features from the registered metabolic domain image;

[0119] S3b: Structural feature branch, which uses the U-Net architecture to extract structural features from the registered structural domain image MRI-T2;

[0120] S3c: Cross-modal attention module, which aligns metabolic and structural features through a cross-attention mechanism, as shown in the formula:

[0121]

[0122] In this context, the query matrix Q is derived from metabolic features, while the key matrix K and value matrix V are derived from structural features. It is the dimension of K.

[0123] Preferably, the inter-domain infiltration probability migration module includes:

[0124] Graph structure building unit, used to define graph structure, where nodes are cluster centers of metabolic features in metabolic domain images divided by K-means clustering, and spatial neighborhood similarity of MRI-T2 in structural domain images is used as edge weight;

[0125] The feature propagation unit is used to propagate features of the interaction graph using a graph convolutional network (GCN), and to aggregate the PET metabolic features of nodes by weighting the edge weights to generate new node features that integrate local spatial neighborhood information.

[0126] The consistency constraint unit is used to introduce a cross-domain consistency constraint term into the loss function, and its formula is as follows:

[0127]

[0128] in, Metabolic feature nodes, For structural feature nodes, This represents the weighting of the metabolism-structure association.

[0129] Preferably, the heatmap generation module includes:

[0130] The probability calculation unit is used to calculate the metabolic-structural overlap probability of each voxel based on the metabolic-structural interaction map, using the following formula:

[0131]

[0132] In the formula: Pinvasion(v) is the tumor invasion probability of voxel v; SUV(v) is the standardized uptake value of voxel v, reflecting the metabolic activity level, obtained from PET images; FA(v) is the anisotropy fraction of voxel v, characterizing tissue structural integrity, obtained based on MRI diffusion tensor technology; SUVmax and FAmax are the global maximum standardized uptake value and the global maximum anisotropy fraction, respectively, used for probability normalization.

[0133] The partitioning unit is used to divide the probability distribution into high, medium, and low risk areas through threshold segmentation to obtain a heat map; the threshold is dynamically adjusted based on clinical pathology data.

[0134] The visualization unit is used to fuse heatmaps with raw image data to generate three-dimensional visualization results for doctors to interactively correct.

[0135] Preferably, the radiotherapy dose planning module includes:

[0136] The dose calculation unit is used to set the dose gradient function based on the high-probability regions of the heatmap. ,in As the baseline dose, The attenuation coefficient;

[0137] An optimization unit is used to introduce Monte Carlo simulation optimization in low-probability regions and adjust the dose distribution based on the tumor edge diffusion model.

[0138] The validation unit is used to output a dose distribution scheme that meets the ICRU dose constraint standard and is physically feasible through Monte Carlo validation.

[0139] Preferably, the data acquisition module includes:

[0140] The PET acquisition unit is used to acquire FDG images through a PET device;

[0141] The MRI acquisition unit is used to acquire T2-weighted sequences using a 3T MRI device.

[0142] Preferably, the preprocessing module includes:

[0143] The registration unit is used to spatially align PET and MRI images using a non-rigid registration algorithm;

[0144] The intensity correction unit is used to apply N4ITK field correction to PET images and perform histogram equalization on MRI images.

[0145] The denoising unit is used to suppress random noise in PET images by wavelet thresholding while preserving metabolic hotspot details.

[0146] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal medical image deep learning-enhanced segmentation system, characterized in that, The data acquisition module is used to acquire metabolic domain images and structural domain images of the target region; The preprocessing module is used to perform image registration, noise suppression, and intensity normalization on the metabolic domain image and the structural domain image. It uses a non-rigid registration algorithm to generate the registered metabolic domain image and the structural domain image. The dual-domain feature decoupling module is used to extract metabolic and structural features from the registered metabolic and structural domain images using a dual-domain feature decoupling device, and to align the metabolic and structural features using a cross-attention mechanism. The inter-domain infiltration probability transfer module is used to construct an inter-domain infiltration probability transfer model. It uses a graph neural network (GNN) to map metabolic features to structural features in the structural domain, generating a metabolic-structural interaction graph. The node attributes are the cluster centers of metabolic features, and the spatial neighborhood similarity of the structural domain image is used as the edge weight. The heatmap generation module is used to generate a pathological infiltration confidence heatmap based on the metabolism-structure interaction map. The heatmap outputs a tumor infiltration probability distribution by quantifying the overlap probability between metabolically active regions and structurally abnormal regions. The radiotherapy dose planning module is used to take the heat map as input for adaptive radiotherapy dose segmentation and dynamically adjust the dose gradient, wherein high probability areas correspond to high dose prescriptions and low probability areas correspond to low dose prescriptions. The results output module is used to output the final segmentation results, including the tumor target area boundary, metabolic-structural conflict region marking, and dose distribution scheme.

2. The multimodal medical image deep learning enhancement segmentation system according to claim 1, characterized in that, The dual-domain feature decoupling module includes: S3a: Metabolic feature branch, which uses 3DCNN to extract metabolic features from the registered metabolic domain image; S3b: Structural feature branch, which uses the U-Net architecture to extract structural features from the registered structural domain image MRI-T2; S3c: Cross-modal attention module, which aligns metabolic and structural features through a cross-attention mechanism, as shown in the formula: In this context, the query matrix Q is derived from metabolic features, while the key matrix K and value matrix V are derived from structural features. It is the dimension of K.

3. The multimodal medical image deep learning-enhanced segmentation system according to claim 1, characterized in that, The inter-domain infiltration probability transfer module includes: Graph structure building unit, used to define graph structure, where nodes are cluster centers of metabolic features in metabolic domain images divided by K-means clustering, and spatial neighborhood similarity of MRI-T2 in structural domain images is used as edge weight; The feature propagation unit is used to propagate features of the interaction graph using a graph convolutional network (GCN), and to aggregate the PET metabolic features of nodes by weighting the edge weights to generate new node features that integrate local spatial neighborhood information. The consistency constraint unit is used to introduce a cross-domain consistency constraint term into the loss function, and its formula is as follows: in, Metabolic feature nodes, For structural feature nodes, This represents the weighting of the metabolism-structure association.

4. The multimodal medical image deep learning-enhanced segmentation system according to claim 1, characterized in that, The heatmap generation module includes: The probability calculation unit is used to calculate the metabolic-structural overlap probability of each voxel based on the metabolic-structural interaction map, using the following formula: In the formula: Pinvasion(v) is the tumor invasion probability of voxel v; SUV(v) is the standardized uptake value of voxel v, reflecting the metabolic activity level, obtained from PET images; FA(v) is the anisotropy fraction of voxel v, characterizing tissue structural integrity, obtained based on MRI diffusion tensor technology; SUVmax and FAmax are the global maximum standardized uptake value and the global maximum anisotropy fraction, respectively, used for probability normalization. The partitioning unit is used to divide the probability distribution into high, medium, and low risk areas through threshold segmentation to obtain a heat map; the threshold is dynamically adjusted based on clinical pathology data. The visualization unit is used to fuse heatmaps with raw image data to generate three-dimensional visualization results for doctors to interactively correct.

5. The multimodal medical image deep learning-enhanced segmentation system according to claim 1, characterized in that, The radiotherapy dose planning module includes: The dose calculation unit is used to set the dose gradient function based on the high-probability regions of the heatmap. ,in As the baseline dose, The attenuation coefficient; An optimization unit is used to introduce Monte Carlo simulation optimization in low-probability regions and adjust the dose distribution based on the tumor edge diffusion model. The validation unit is used to output a dose distribution scheme that meets the ICRU dose constraint standard and is physically feasible through Monte Carlo validation.

6. The multimodal medical image deep learning-enhanced segmentation system according to claim 1, characterized in that, The data acquisition module includes: The PET acquisition unit is used to acquire FDG images through a PET device; The MRI acquisition unit is used to acquire T2-weighted sequences using a 3T MRI device.

7. The multimodal medical image deep learning-enhanced segmentation system according to claim 1, characterized in that, The preprocessing module includes: The registration unit is used to spatially align PET and MRI images using a non-rigid registration algorithm; The intensity correction unit is used to apply N4ITK field correction to PET images and perform histogram equalization on MRI images. The denoising unit is used to suppress random noise in PET images by wavelet thresholding, while preserving metabolic hotspot details.

8. A method for a multimodal medical image deep learning-enhanced segmentation system, characterized in that, Includes the following steps: Acquire medical imaging data of the target region, including metabolic domain images (PET) and structural domain images (MRI-T2). Image registration, noise suppression, and intensity normalization were performed on PET and MRI-T2 images to generate registered metabolic and structural domain images. Metabolic and structural features are extracted using a dual-domain feature decoupler, where metabolic features include glycolysis features and structural features include texture anisotropy features. A probabilistic transfer model for interdomain infiltration was constructed. A graph neural network (GNN) was used to map metabolic features to the structural domain space to generate a metabolic-structural interaction map, in which the node attributes are metabolic features and the adjacency relationship is based on the spatial neighborhood topology of MRI-T2 images. Based on the interactive atlas, a pathological invasion confidence heatmap is generated. The heatmap outputs the tumor invasion probability distribution by quantifying the overlap probability between metabolically active regions and structurally abnormal regions. The heatmap is used as input for adaptive radiotherapy dose segmentation to dynamically adjust the dose gradient, where high-probability regions correspond to high-dose prescriptions and low-probability regions correspond to low-dose prescriptions. The final segmentation results are output, including tumor target boundary, metabolic-structural conflict region marking, and dose distribution scheme.

Images