Tumor space-occupying brain network neural image alignment method based on multi-modal fusion

Abstract

The invention discloses a tumor space-occupying brain network neural image alignment method based on multi-modal fusion, and belongs to the technical field of medical image processing and artificial intelligence crossing. The method comprises the following core steps of multi-modal image heterogeneous feature decoupling, tumor occupation deformation field modeling, functional network topological structure maintenance, cross-modal feature adversarial alignment, dynamic deformation constraint optimization and clinical interpretability verification, and construction of a three-dimensional non-rigid registration network based on a double attention mechanism. And differential homeomorphic mapping of a tumor focus area and normal brain tissue is realized through the cascaded spatial transformation module. Aiming at the problems of insufficient multi-modal feature alignment and brain network topology distortion in the prior art, the invention provides a function connection constrained cross-modal fusion strategy, a graph convolution network is adopted to encode resting state function connection features, and network node displacement caused by tumor occupation is dynamically corrected in combination with deformable convolution and a bidirectional feature competition mechanism; a space consistency loss function based on white matter fiber bundle tracing is designed, and through diffusion tensor imaging feature guide structure-function bimodal joint optimization, the problems of insufficient registration precision in a focus area and whole brain network connection distortion of a traditional method are solved. Experiments show that the registration precision of the method in glioma cases reaches 0.82 mm and is improved by 37% compared with that of a traditional method, and dissection-function consistency of functional network reconstruction around tumors is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of intersection of artificial intelligence and medical image processing technology, and specifically relates to a tumor-occupying brain network neural image alignment method based on multimodal fusion, which is used to solve the spatial registration problem of multimodal brain images (such as structural MRI, functional MRI, and DTI) under the tumor space-occupying effect, and provide an accurate image alignment basis for brain tumor surgical planning and functional protection. Background Art

[0002] 1. In the diagnosis and treatment of brain tumors, precise alignment of multimodal neuroimaging (such as T1 / T2 structural MRI, fMRI functional activation areas, and DTI white matter fiber tracts) is a prerequisite for locating the relationship between tumors and key brain functional networks. However, existing technologies have the following problems: 1. Insufficient multimodal feature fusion: Traditional registration methods (such as affine transformation and elastic registration) do not consider local deformation caused by tumor space occupation and lack dynamic weighting of feature differences between different modalities (such as structural-functional imaging). This can lead to misalignment between functional activation areas and anatomical structures after registration (error > 3mm).

[0003] 2. Lack of compensation for tumor deformation: Existing algorithms (such as SyN and Demons) directly align to the healthy brain template and do not model the brain tissue displacement effect caused by tumor growth, resulting in deviations in the positioning of functional areas (for example, the mismatch rate of the precentral gyrus displacement reaches 62%).

[0004] 3. Real-time and robustness defects: Traditional methods rely on iterative optimization, with a single-case registration time of >5 minutes. They are also sensitive to heterogeneous areas such as edema and necrotic cores, with a registration failure rate as high as 28%.

[0005] 4. Weak clinical validation: There is a lack of collaborative validation mechanisms with DTI fiber tracking and intraoperative electrophysiology, which cannot guarantee the reliability of the registration results in surgical navigation.

[0006] To address the above problems, the present invention proposes an image alignment method that integrates dynamic selection of multimodal features, tumor deformation field prediction and adversarial boundary optimization. Summary of the Invention

[0007] A method for aligning tumor-occupying brain network neuroimaging images based on multimodal fusion, comprising the following steps:

[0008] S1. Multimodal image preprocessing and feature extraction

[0009] Input structural MRI (T1 / T2 / FLAIR), functional MRI (BOLD), and DTI data, and simulate the brain tissue displacement field under the tumor space effect through finite element analysis;

[0010] 3D ResNet-18 is used to extract deep features of each modality and generate a multi-scale feature pyramid (resolution 1mm 3 Up to 4mm 3 ).

[0011] S2. Dynamic Weight Multimodal Fusion

[0012] Construct a channel-spatial dual attention module (DCSAM) to dynamically calculate the weights of each modality feature:

[0013] Channel attention: Generate channel masks based on the tumor edema area features of the FLAIR sequence;

[0014] Spatial attention: Screening key spatial areas through the functional connectivity strength distribution of BOLD signals;

[0015] Output the fused multimodal feature tensor.

[0016] S3. Tumor deformation field prediction and compensation

[0017] Design a deformation field generation network (DF-Net), which inputs the fused features and tumor segmentation mask to predict the local deformation vector field caused by the tumor growth direction;

[0018] Combined with biomechanical models to constrain the physical rationality of the deformation field (such as the continuity of white matter fiber bundles).

[0019] S4. Adversarial Registration Optimization

[0020] Using a conditional generative adversarial network (cGAN), the discriminator verifies the registration quality from three scales:

[0021] Macroscopically: examining the alignment of whole-brain anatomy (e.g., ventricular morphology);

[0022] Mesoscopic scale: verifying the spatial consistency of functional activation areas with anatomical landmarks (such as the central sulcus);

[0023] Microscale: Constraining the continuity of white matter fiber bundles through the FA value gradient of DTI.

[0024] S5. Multimodal Registration and Clinical Validation

[0025] Apply the deformation field to the target modality (e.g., fMRI→T1) to generate a registered image.

[0026] Two-factor authentication:

[0027] DTI verification: distance threshold between the fiber tracking path and the tumor boundary after registration (≥2 mm);

[0028] Functional verification: Pearson correlation coefficient of BOLD signal connectivity strength (≥0.9).

[0029] S6, real-time interaction and output

[0030] Deploy a lightweight registration model (parameter size < 50MB) and support GPU acceleration (single-instance execution time < 30 seconds);

[0031] Output registered images and deformation field thermal maps, and integrate with the neuronavigation system through the DICOM interface.

[0032] Preferred solution

[0033] In step S2, the channel attention weight is dynamically adjusted according to the tumor volume ratio, and the formula is:

[0034] wc=σ(α·Vtumor+β·Cedge)

[0035] in

[0036] Vtumor is the tumor volume, and Cedge is the edge complexity.

[0037] The deformation field generation network in step S3 uses a U-Net++ architecture and embeds a residual deformation module (RDM) to handle large displacements. The discriminator in step S5 introduces a topological loss function to constrain the global connectivity properties of the functional brain network after registration (such as the small-world network metric).

[0038] Technical effects and advantages of the present invention:

[0039] 1. Multimodal dynamic fusion: Adaptively fuses structural and functional features through a dual-attention mechanism, reducing registration error by 42% compared to traditional methods (average 1.2mm vs 2.1mm).

[0040] 2. Tumor deformation compensation: The deformation field predicted by DF-Net is 89% consistent with the displacement measured during surgery, significantly improving the accuracy of functional area positioning.

[0041] 3. Real-time and robustness: The lightweight model supports real-time intraoperative registration (30 seconds / case), and the registration success rate for edema areas is increased to 95%.

[0042] 4. Clinical practicality: Integrated DTI / BOLD dual verification and seamless integration with neuronavigation systems have increased the clinical adoption rate to 78%. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the workflow of the present invention. DETAILED DESCRIPTION

[0044] The present invention discloses a method for aligning tumor-occupying brain network neuroimaging images based on multimodal fusion, comprising the following collaborative steps:

[0045] S1. Dynamic registration of multimodal data:

[0046] Structural MRI (T1 / T2 / FLAIR), resting-state functional MRI (rs-fMRI), and diffusion spectrum imaging (DSI) data were simultaneously acquired to construct a three-dimensional spatial coordinate system that includes tumor mass effect and functional network topology;

[0047] A hierarchical registration strategy was adopted: first, 6-DOF rigid registration was performed using the ANTs toolkit (taking ≤ 5 seconds), and then the brain tissue deformation field caused by tumor occupation was simulated based on finite element analysis (grid size 0.5 mm 3 ), and finally the SyN nonlinear registration algorithm was used to compensate for the functional area displacement (accuracy 0.2mm);

[0048] S2. Cross-modal feature coupling:

[0049] Build a two-stream feature extraction network:

[0050] Anatomical flow: 3D Swin Transformer was used to hierarchically extract gray matter-white matter interface features at the tumor edge (window size 4×4×4, shift window mechanism);

[0051] Functional flow: Dynamic graph neural networks (DGNNs) were used to model the dynamic characteristics of functional connectivity in rs-fMRI time series (nodes were AAL90 brain regions, and edge weights were dynamically updated by sliding window Pearson correlation coefficients).

[0052] Introducing a cross-modal feature coupling layer: Based on the principle of maximizing mutual information, the joint probability distribution of anatomical and functional features is calculated to generate feature embeddings with spatial consistency;

[0053] S3. Prediction of tumor-induced deformation field:

[0054] Building a biomechanics-deep learning hybrid model:

[0055] Input layer: fused tumor volume, location (Talairach coordinates) and Young's modulus parameters (gray matter 3kPa / white matter 7kPa / tumor 15kPa);

[0056] Hidden layer: 64-channel 3D convolutional network in conjunction with the COMSOL Multiphysics finite element solver;

[0057] Output layer: Generates a vector matrix of the predicted deformation field (resolution 1mm 3 ), and the anatomical continuity of the pyramidal tract was verified by DTI fiber tracking (FA value decrease rate ≤ 15%);

[0058] S4. Multi-scale alignment optimization:

[0059] Macroscopically, a deformable convolutional network (DCN) was used to correct for ventricular system deviation caused by the overall tumor size.

[0060] Mesoscopic scale: using graph matching algorithms (Gromov-Wasserstein distance) to align modular partitions of functional networks (such as the default mode network DMN);

[0061] Microscale: Optimize the functional connectivity weights of individual voxels based on non-local means filtering, preserving the temporal correlation of BOLD signals (correlation coefficient variation ≤ 0.1);

[0062] S5. Adversarial deformation verification:

[0063] Constructing a three-level conditional generative adversarial network (cGAN):

[0064] Generator: Inputs the tumor mask and the original image, and outputs a deformation-corrected multimodal image;

[0065] Discriminator:

[0066] Level 1: Verify the overall sulcal and gyral topology (using curvature energy map loss);

[0067] Level 2: Detect the distance constraint between the motor function area and the tumor boundary (protection radius of the hand knob area in the precentral gyrus ≥ 5 mm);

[0068] Level 3: Analysis of the consistency of white matter fiber tract orientation (quantification of fiber tract anisotropy fraction FA gradient using DSI Studio);

[0069] S6. Dynamic loss function design: Define the composite loss function: L_total = λ1*L_anatomy+λ2*L_function+λ3*L_deformation, where:

[0070] L_anatomy: anatomical alignment loss based on Dice coefficient and Hausdorff distance;

[0071] L_function: functional network attribute loss (including graph theory indicators such as node degree centrality and clustering coefficient);

[0072] L_deformation: deformation field smoothness constraint (using Laplacian regularization);

[0073] Dynamic adjustment mechanism: Automatically adjust weight according to tumor volume (when volume > 4cm 3 When λ3 is increased by 50%);

[0074] S7. Clinical Real-Time Interaction System: Deployment of a WebAssembly-based online alignment platform that supports real-time drag-and-drop adjustment of DICOM images and deformation field visualization (using Three.js volume rendering); integrated incremental learning module that records the doctor's manual correction trajectory and updates the local deformation model through online backpropagation; output multimodal fusion report including functional network displacement vector map, surgical approach risk heat map (transparency map 0.2-0.8), and fiber tract integrity score (0-5 levels);

[0075] Preferred embodiment details

[0076] 1. DGNN design in step S2:

[0077] Node features: Contains 6-dimensional time series features such as BOLD signal mean, variance, and Hurst exponent;

[0078] Dynamic edge update: Sliding window width 30 seconds, step size 5 seconds, thresholding to retain the top 10% strong connections;

[0079] Coupling layer implementation: The correlation matrix between anatomical voxels and functional nodes is calculated through a cross-attention mechanism (8 heads), and 128-dimensional fusion features are generated after Softmax normalization;

[0080] 2. Graph matching optimization in step S4:

[0081] Construct a functional network hypergraph: Power-264 partitions are used as supernodes and functional connectivity strengths are used as hyperedge weights;

[0082] We use an entropy-regularized optimal transfer algorithm (Sinkhorn iteration, ε = 0.1) to achieve a matching speed of 2000 nodes / second on a GPU.

[0083] 3. Generator architecture of step S5:

[0084] Encoder: 4 layers of 3D convolution (kernel size 3×3×3, stride 2), each followed by IN-ReLU;

[0085] Deformation field prediction: 9 residual blocks (including biomechanical constraint layer), outputting a 3-channel displacement field;

[0086] Discriminator: Multi-scale PatchGAN (70×70 / 140×140 / full image) with adversarial loss weighting of 1:2:1; 4. Real-time interactive system performance:

[0087] The TensorRT engine is deployed on the NVIDIA A100, achieving inference latency of less than 50ms per case. The system supports concurrent operation by 8 or more doctors and synchronizes DICOM-RT data with the StealthStation navigation system via WebSocket (latency less than 200ms).

[0088] This method achieves submillimeter-level precision in tumor-occupying brain network alignment by coupling anatomical-functional multimodal features with biomechanically constrained deformation field prediction, reducing functional area positioning error by 62% (p < 0.001, paired t-test) compared with traditional methods (such as FLIRT).

[0089] Example 1: Multimodal elastic registration and functional network reconstruction

[0090] Scenario: Left parietal lobe metastasis (volume 3.8cm 3 ) leads to displacement of the default mode network (DMN)

[0091] Implementation steps:

[0092] 1.Synchronous acquisition of multimodal data:

[0093] Using GE Discovery MR750w 3.0T for simultaneous acquisition:

[0094] Resting-state fMRI (TR = 800 ms, voxel size 2.5 × 2.5 × 3 mm) 3 , 15 minutes)

[0095] High-resolution T2-FLAIR (resolution 0.5×0.5×1mm 3 )

[0096] Diffusion spectrum imaging DSI (257 directions, bmax = 5000s / mm 2 )

[0097] 2. Nonlinear registration:

[0098] Construct a layered B-spline deformation field:

[0099] Initial rigid registration (6 degrees of freedom, MI similarity)

[0100] Multi-resolution elastic registration (3-level pyramid, the highest level control point spacing is 4mm)

[0101] Add tumor-specific constraints: Poisson ratio of 0.45 for tumor area and 0.35 for gray matter

[0102] 3. Network node alignment:

[0103] Extract DMN nodes based on independent component analysis (ICA):

[0104] Calculate the functional connectivity matrix using the PROFUMO algorithm

[0105] The healthy template network (Yeo-7 network) is mapped to the disease space through the geodesic flow kernel function

[0106] After registration, the position error of the DMN core node (posterior cingulate gyrus) was reduced from 4.1mm to 1.2mm

[0107] 4. Verification method:

[0108] Intraoperative O-arm scan verification: registration error 1.3mm (traditional method 3.7mm)

[0109] Postoperative fMRI review: DMN functional connectivity strength recovered to 85% of the healthy side (62% before surgery)

[0110] Example 2: Dynamic Manifold Alignment and White Matter Fiber Tracking

[0111] Scenario: Right frontal lobe glioma (volume 6.5cm 3 ) causes distortion of the corpus callosum fiber bundles

[0112] Implementation steps:

[0113] 1. Dynamic Manifold Learning:

[0114] Construct the diffusion map embedding space:

[0115] Fusion of T1 structural features (cortical thickness), DTI features (FA / MD value), and fMRI features (ReHo value)

[0116] Using the graph Laplacian operator to build a manifold distance matrix

[0117] Set the tumor area attenuation factor λ to 0.3

[0118] 2. Fiber bundle redirection:

[0119] Improved FACT tracking algorithm:

[0120] Computing fiber orientation probability fields in manifold space

[0121] Add anisotropic constraints caused by tumor compression (stiffness coefficient 0.6)

[0122] After registration, the angle deviation of the fibers at the genu of the corpus callosum was reduced from 28° to 9°

[0123] 3. Multi-atlas fusion:

[0124] Integration of HCP-MMP1.0 atlas and JHU white matter atlas:

[0125] Calculate the deformation field using the Log-Euclidean mean

[0126] Adaptive weighting (α=0.7) is used for the 3mm area around the tumor

[0127] 4. Effect verification:

[0128] Intraoperative electrophysiological monitoring: The deviation between the motor evoked potential site and the predicted position was 1.1 mm

[0129] Postoperative DTI review: corpus callosum FA value correlation r = 0.91 (preoperative r = 0.68) Example 3: Cross-modal attention registration and surgical navigation scenario: thalamic glioma (volume 2.7cm 3 ) leads to bilateral basal ganglia network disconnection

[0130] Implementation steps:

[0131] 1. Attention mechanism registration:

[0132] Construct a dual-path registration network:

[0133] Structural path: Processing T1 / T2 weighted images (3D convolution kernel 7×7×7)

[0134] Functional pathway: processing rs-fMRI time series (64 LSTM units)

[0135] Cross-modal attention gating: focusing on the tumor-basal ganglia connection region (threshold > 0.75)

[0136] 2. Network topology preservation:

[0137] Applying graph matching algorithm:

[0138] Extracting node degree centrality features of functional networks

[0139] Preserving network topology using Gromov-Wasserstein distance

[0140] After registration, the global efficiency error of the network is reduced from 0.15 to 0.04

[0141] 3.Surgical navigation integration:

[0142] Generate a heatmap of registration error:

[0143] Tumor core area error <0.5mm

[0144] Edge area error <1.2mm

[0145] Transmitted to the Brainlab navigation system via the OpenIGTLink protocol

[0146] 4. Clinical validation:

[0147] Language testing during awake surgery: Network node prediction accuracy of 94%

[0148] Follow-up 3 months after surgery: basal ganglia functional connectivity recovered to 92% of normal levels

[0149] Example 4: Multi-phase dynamic registration and radiotherapy planning scenario: recurrent glioma (volume 5.1cm 3 ) Monitoring of fiber bundle changes after radiotherapy

[0150] Implementation steps:

[0151] 1. Temporal feature extraction:

[0152] Four-dimensional registration framework (3D+Δt):

[0153] Obtain MR sequences before radiotherapy, 20Gy, 40Gy, and after surgery

[0154] Calculate Jacobian determinant map (30-day interval)

[0155] Tumor growth rate limit: ≤1.2mm / week

[0156] 2. Dose deformation modeling:

[0157] Establishing radiation-tissue deformation model:

[0158] Dose cumulative effect coefficient γ=0.85

[0159] White matter fiber bundle deformation threshold: absorbed dose > 15Gy area

[0160] Predicted cone beam displacement: 2.3 mm (direction: posteromedial)

[0161] 3. Adaptive registration:

[0162] Update registration parameters online:

[0163] The deformation field is recalculated after every 10 Gy dose.

[0164] Use Kalman filter to predict the next stage deformation

[0165] Final registration error: 0.7mm (2.4mm for traditional method)

[0166] 4. Verification method:

[0167] Fiber bundle anatomy verification: intraoperative ultrasound elastography matching degree 89%

[0168] Functional prognosis assessment: correlation between motor function score and prediction after 6 months: r = 0.88

[0169] Finally, a few points need to be noted: first, the image registration algorithm involved in this method can be extended to other modalities such as PET-MRI fusion and optical coherence tomography;

[0170] Secondly, the elastic deformation model parameters can be adjusted according to different tumor pathological types;

[0171] Finally, the network node alignment results can be exported as a DICOM-RT structure set for radiotherapy planning.

Claims

1. A method for alignment of tumor-occupying brain network neuroimaging images based on multimodal fusion, characterized by It includes the following steps that work together: S1. Dynamic registration of multimodal data: Synchronously acquire preoperative structural MRI (T1 / T2 / FLAIR), functional MRI (fMRI), and diffusion tensor imaging (DTI) data from tumor patients to construct a three-dimensional spatial mapping matrix that includes tumor space deformation and functional connectivity networks; S2. Cross-modal feature coupling alignment: Using an improved VoxelMorph network architecture and integrating a multi-scale feature competition mechanism, we extract the anatomical boundary features of T1-weighted images and the functional connectivity features of the fMRI BOLD signal through dual-branch convolution, and establish a cross-modal feature similarity measurement function. S3. Elastic matching of network nodes: Based on graph neural networks (GNNs), we model the whole-brain functional network nodes, use a dynamic edge weight allocation algorithm to align the functional connectivity topology differences between healthy templates and individual patients, and generate key functional area matching anchors through spectral clustering. S4. Bidirectional prediction of space-occupying deformation field: Construct a bidirectional LSTM deformation field generator to forward predict brain tissue displacement vectors caused by tumor growth, reversely reconstruct the functional area positions in the original anatomical space, and verify the continuity constraints of white matter pathways in combination with DTI fiber tracking; S5. Multimodal adversarial optimization: Design a generative adversarial network with three discriminators. The macro discriminator verifies the overall sulcus-gyrus registration accuracy, the mesoscopic discriminator detects the relative position error between the central sulcus and the motor area, and the microscopic discriminator evaluates the registration consistency of FA values ​​at the single voxel level. S6. Dynamic weight loss function: Construct a composite loss function L_total = λ1*L_similarity + λ2*L_smooth + λ3*L_topology, where λ1, λ2, and λ3 are dynamically adjusted based on the distance between the tumor volume and the functional area, and the connection strength gradient is introduced as a topological constraint term; S7, Cascade registration strategy: In the first stage, rigid registration is used to eliminate head motion artifacts and coarsely register to the MNI standard space. In the second stage, nonlinear registration is used to compensate for the functional area displacement caused by the tumor space effect. S8. Intraoperative real-time update: The spatial coordinate conversion module of the integrated optical navigation system maps the registration results to the intraoperative cortical electrical stimulation sites in real time, and updates the newly acquired fMRI data during the operation through incremental learning.

2. The method for aligning tumor-occupying brain network neuroimaging images based on multimodal fusion according to claim 1, characterized in that: The cross-modal feature coupling alignment described in step S2 adopts a channel-space dual attention mechanism, specifically including: implementing spatial weighting based on tumor boundaries for structural MRI feature channels, adopting graph attention aggregation of functional connection strength for fMRI feature channels, and finally achieving spatial alignment of heterogeneous features through deformable convolution.

3. The method for aligning tumor-occupying brain network neuroimaging images based on multimodal fusion according to claim 1, characterized in that: The network node elasticity matching in step S3 includes two-fold verification: the first verification layer verifies the anatomical connection integrity between the motor area and the posterior limb of the internal capsule by DTI fiber tracking, and the second verification layer uses resting-state fMRI to calculate the Frobenius norm difference of the functional connectivity matrix (threshold ≤ 0.15).

4. The method for aligning tumor-occupying brain network neuroimaging images based on multimodal fusion according to claim 1, characterized in that: The deformation field generator in step S4 contains biomechanical prior constraints, simulates the changes in the cerebrospinal fluid flow field caused by tumor occupation through finite element analysis, and incorporates it into the deformation field prediction as a regularization term.

5. The method for aligning tumor-occupying brain network neuroimaging images based on multimodal fusion according to claim 1, characterized in that: The microscopic discriminator in step S5 uses a tensor invariance detection module to ensure the geometric invariance of the white matter fiber orientation by comparing the covariance matrix similarity of the DTI eigenvalues ​​(including FA / MD / AD / RD) before and after registration.

6. The method for aligning tumor-occupying brain network neuroimaging images based on multimodal fusion according to claim 1, characterized in that: The dynamic weight adjustment strategy in step S6 includes calculation of the tumor infiltration index, which automatically increases the λ3 topological constraint weight of the adjacent functional area based on the inverse relationship between the volume of the high signal area and the distance to the functional area in the FLAIR sequence.

7. The method for aligning tumor-occupying brain network neuroimaging images based on multimodal fusion according to claim 1, characterized in that: The cascade registration in step S7 uses a multi-resolution pyramid structure to 3 Perform global affine transformation at a resolution of 0.5 mm 3 Local nonlinear registration based on a fluid dynamics model is implemented at high resolution.

8. The method for aligning tumor-occupying brain network neuroimaging images based on multimodal fusion according to claim 1, characterized in that: The final output includes a multimodal registration quality report, including: functional network node alignment heat map, tumor-functional area minimum Euclidean distance measurement value, white matter fiber bundle integrity index after registration, and real-time monitoring curve of intraoperative navigation coordinate conversion error.

Images