Liver segmentation, modeling and operation simulation system based on cascade neural network and multi-scale reconstruction technology

Abstract

The invention relates to the technical field of medical image processing and surgical navigation, in particular to a liver segmentation, modeling and surgical simulation system based on a cascade neural network and a multi-scale reconstruction technology. The multi-scale dynamic registration module is used for carrying out coarse segmentation and fine segmentation on liver and focus areas of the medical image and is used for non-rigid registration and respiratory motion compensation of the multi-modal medical image; the physical enhancement modeling engine is used for constructing a liver vessel tree three-dimensional model containing hemodynamic characteristics; the operation navigation and risk assessment module is used for performing real-time navigation and operation path planning in the operation; and the self-adaptive computing framework provides underlying computing resource support for the system. And high-precision segmentation: through a cascade recursive network and a boundary sensitive loss function, the liver tumor segmentation Dice coefficient reaches 94-96%, which is increased by 6-12% compared with the traditional method.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and surgical navigation technology, specifically relating to a liver segmentation, modeling and surgical simulation system and method based on cascaded neural networks and multi-scale reconstruction technology. Background Technology

[0002] The success of liver surgery highly depends on the accuracy of preoperative planning and the real-time nature of intraoperative navigation. Current liver image segmentation methods (such as region growing and cascade segmentation) are insensitive to low-contrast tumors, small lesions (<5mm in diameter), and regions with blurred boundaries, especially showing significant errors in the context of cirrhosis. Although deep learning models (such as U-Net) improve basic segmentation capabilities, insufficient multi-scale feature fusion results in Dice coefficients generally below 90%. In multimodal image registration, CT / MRI image registration typically relies on manually labeled reference points, with registration errors often exceeding 3mm, and poor adaptability to dynamic factors such as respiratory movements and changes in body position. While recursive cascade networks can progressively optimize registration, their global deformation modeling capabilities are weak, easily losing vascular topological details. In the field of 3D reconstruction, existing reconstruction models (such as the surface rendering MC algorithm) struggle to recreate the physiological branching angles of the hepatic artery and portal vein (e.g., the bifurcation angle error of the right hepatic vein reaches 10°–15%), affecting the realism of hemodynamic simulations. Furthermore, existing virtual surgery systems (such as the FreeForm Modeling System) lack intraoperative dynamic calibration, resulting in a mismatch between the preoperative model and intraoperative organ deformation (such as liver displacement), with navigation path errors reaching 5–8 mm. Therefore, there is an urgent need for an integrated solution capable of achieving high-precision liver segmentation, multimodal registration, physically realistic modeling, and real-time surgical navigation. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a liver surgery simulation system and method with high segmentation accuracy, small registration error, realistic vascular topology restoration, and support for real-time navigation.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a liver segmentation, modeling, and surgical simulation system based on cascaded neural networks and multi-scale reconstruction technology, comprising:

[0005] A cascaded recursive segmentation network module is used for coarse and fine segmentation of liver and lesion regions in medical images;

[0006] A multi-scale dynamic registration module is used for non-rigid registration and respiratory motion compensation in multimodal medical images;

[0007] A physical enhancement modeling engine for building 3D models of liver vascular trees that include hemodynamic properties;

[0008] The surgical navigation and risk assessment module is used for real-time intraoperative navigation and surgical path planning;

[0009] Generative data augmentation module, used to synthesize and expand training datasets;

[0010] An adaptive computing framework for the dynamic scheduling and management of distributed computing resources;

[0011] The cascaded recursive segmentation network module, the multi-scale dynamic registration module, and the physical enhancement modeling engine are connected in sequence.

[0012] The surgical navigation and risk assessment module has a bidirectional data connection with the physical augmentation modeling engine;

[0013] The generative data augmentation module is connected to the training data input end of the cascaded recursive segmentation network module.

[0014] The adaptive computing framework provides underlying computing resource support for the system.

[0015] Preferably, the cascaded recursive partitioning network module includes:

[0016] The coarse segmentation unit uses a U-Net++ network with a feature pyramid structure for initial liver segmentation;

[0017] The fine segmentation unit employs a recurrent neural network with attention gating to fine-tune the tumor boundary;

[0018] The loss function optimization unit combines Dice loss and boundary-sensitive loss function for model training.

[0019] Preferably, the multi-scale dynamic registration module includes:

[0020] A multimodal fusion unit is used to achieve non-rigid registration of CT / MRI images based on a recursive cascaded registration network.

[0021] The motion compensation unit uses an LSTM network to predict liver displacement during the respiratory cycle;

[0022] The deformation field optimization unit reduces deformation folding through cyclic consistency loss, and the registration error is controlled within 1.5mm.

[0023] Preferably, the physical augmentation modeling engine includes:

[0024] A vascular topology reconstruction unit was used to reconstruct the three-dimensional hepatic vascular network based on an improved Marching Cubes algorithm.

[0025] The hemodynamic simulation unit embeds the Navier-Stokes equation to simulate the hepatic vein pressure gradient;

[0026] The force feedback modeling unit calculates the stress distribution of surgical instruments on liver tissue through finite element analysis.

[0027] Preferably, the generative data augmentation module includes:

[0028] Conditional generative adversarial networks are used to generate liver pathological images with specific attributes.

[0029] A diffusion model unit is used to synthesize high-fidelity liver tumor images;

[0030] The quality control system includes automated quality assessment indicators and a human-machine collaborative review mechanism.

[0031] Preferably, the adaptive computing framework includes:

[0032] A unified query interface supports federated queries based on the ANSI SQL standard.

[0033] Intelligent routing decision unit, based on SQL characteristics and historical load prediction optimal calculation engine;

[0034] Edge-cloud collaborative units enable dynamic offloading of computing tasks and elastic scaling of resources.

[0035] Preferably, the surgical navigation and risk assessment module includes:

[0036] The AR navigation unit overlays a 3D model onto the real field of vision during surgery using a head-mounted display device;

[0037] The risk warning unit dynamically calculates the risk coefficient based on the shortest distance between the tumor and blood vessels;

[0038] The real-time calibration unit updates the registration field using intraoperative images, resulting in a navigation latency of less than 100ms.

[0039] A method for liver segmentation, modeling, and surgical simulation based on cascaded neural networks and multi-scale reconstruction technology includes the following steps:

[0040] Step S1: Acquire the patient's multimodal medical imaging data, including CT arterial phase, portal venous phase, and MRI sequences;

[0041] Step S2: Segment the liver parenchyma and lesions in the image using a cascaded recursive segmentation network to generate a segmentation mask;

[0042] Step S3: Use the multi-scale dynamic registration module to achieve accurate registration of multi-phase images and compensate for respiratory motion;

[0043] Step S4: Construct three-dimensional models of the hepatic artery, portal vein, and hepatic vein based on a physical augmentation modeling engine;

[0044] Step S5: Plan the surgical path based on hemodynamic parameters and assess the risk level of different options;

[0045] Step S6: Intraoperative real-time registration and model update, providing AR navigation and force feedback simulation.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] High-precision segmentation: Through cascaded recursive networks and boundary-sensitive loss functions, the Dice coefficient for liver tumor segmentation reaches 94-96%, which is 6-12 percentage points higher than traditional methods.

[0048] Precise registration: Multi-scale dynamic registration combined with respiratory motion compensation, multimodal image registration error ≤1.5mm, which is more than 50% lower than the error of existing technologies.

[0049] Realistic modeling: Based on the fluid dynamics equations, the topology of blood vessels is reconstructed, and the error of the branch angle of blood vessels is less than 3°, which significantly improves the realism of hemodynamic simulation.

[0050] Real-time navigation: LSTM prediction compensation and edge-cloud collaborative computing architecture enable intraoperative navigation latency of <100ms, meeting clinical real-time requirements.

[0051] Strong generalization ability: The generative data augmentation module synthesizes diverse pathological samples to solve the problem of data scarcity and improve the model's generalization ability on rare cases. Attached Figure Description

[0052] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the overall system architecture of the present invention;

[0054] Figure 2 This is a two-stage processing flowchart of the cascaded recursive partitioning network of the present invention;

[0055] Figure 3 This is a step diagram illustrating the multimodal image registration and respiratory motion compensation of the present invention;

[0056] Figure 4 This diagram illustrates the process of generating a physically enhanced 3D model from the segmentation results of this invention.

[0057] Figure 5This invention provides a rigorous quality control process for the generated data from its generation to its final use.

[0058] Figure 6 This is a resource scheduling logic diagram of the adaptive computing framework of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Please see Figures 1 to 6 The present invention provides a technical solution:

[0061] A liver segmentation, modeling, and surgical simulation system based on cascaded neural networks and multi-scale reconstruction technology includes:

[0062] A cascaded recursive segmentation network module is used for coarse and fine segmentation of liver and lesion regions in medical images;

[0063] A multi-scale dynamic registration module is used for non-rigid registration and respiratory motion compensation in multimodal medical images;

[0064] A physical enhancement modeling engine for building 3D models of liver vascular trees that include hemodynamic properties;

[0065] The surgical navigation and risk assessment module is used for real-time intraoperative navigation and surgical path planning;

[0066] Generative data augmentation module, used to synthesize and expand training datasets;

[0067] An adaptive computing framework for the dynamic scheduling and management of distributed computing resources;

[0068] The cascaded recursive segmentation network module, the multi-scale dynamic registration module, and the physical enhancement modeling engine are connected in sequence.

[0069] The surgical navigation and risk assessment module has a bidirectional data connection with the physical augmentation modeling engine;

[0070] The generative data augmentation module is connected to the training data input end of the cascaded recursive segmentation network module.

[0071] The adaptive computing framework provides underlying computing resource support for the system.

[0072] Specifically, the cascaded recursive segmentation network module includes:

[0073] The coarse segmentation unit uses a U-Net++ network with a feature pyramid structure for initial liver segmentation;

[0074] The fine segmentation unit employs a recurrent neural network with attention gating to fine-tune the tumor boundary;

[0075] The loss function optimization unit combines Dice loss and boundary-sensitive loss function for model training.

[0076] Specifically, the multi-scale dynamic registration module includes:

[0077] A multimodal fusion unit is used to achieve non-rigid registration of CT / MRI images based on a recursive cascaded registration network.

[0078] The motion compensation unit uses an LSTM network to predict liver displacement during the respiratory cycle;

[0079] The deformation field optimization unit reduces deformation folding through cyclic consistency loss, and the registration error is controlled within 1.5mm.

[0080] Specifically, the physical augmentation modeling engine includes:

[0081] A vascular topology reconstruction unit was used to reconstruct the three-dimensional hepatic vascular network based on an improved Marching Cubes algorithm.

[0082] The hemodynamic simulation unit embeds the Navier-Stokes equation to simulate the hepatic vein pressure gradient;

[0083] The force feedback modeling unit calculates the stress distribution of surgical instruments on liver tissue through finite element analysis.

[0084] Specifically, the generative data augmentation module includes:

[0085] Conditional generative adversarial networks are used to generate liver pathological images with specific attributes.

[0086] A diffusion model unit is used to synthesize high-fidelity liver tumor images;

[0087] The quality control system includes automated quality assessment indicators and a human-machine collaborative review mechanism.

[0088] Specifically, the adaptive computing framework includes:

[0089] A unified query interface supports federated queries based on the ANSI SQL standard.

[0090] Intelligent routing decision unit, based on SQL characteristics and historical load prediction optimal calculation engine;

[0091] Edge-cloud collaborative units enable dynamic offloading of computing tasks and elastic scaling of resources.

[0092] Specifically, the surgical navigation and risk assessment module includes:

[0093] The AR navigation unit overlays a 3D model onto the real field of vision during surgery using a head-mounted display device;

[0094] The risk warning unit dynamically calculates the risk coefficient based on the shortest distance between the tumor and blood vessels;

[0095] The real-time calibration unit updates the registration field using intraoperative images, resulting in a navigation latency of less than 100ms.

[0096] A method for liver segmentation, modeling, and surgical simulation based on cascaded neural networks and multi-scale reconstruction technology includes the following steps:

[0097] Step S1: Acquire the patient's multimodal medical imaging data, including CT arterial phase, portal venous phase, and MRI sequences;

[0098] Step S2: Segment the liver parenchyma and lesions in the image using a cascaded recursive segmentation network to generate a segmentation mask;

[0099] Step S3: Use the multi-scale dynamic registration module to achieve accurate registration of multi-phase images and compensate for respiratory motion;

[0100] Step S4: Construct three-dimensional models of the hepatic artery, portal vein, and hepatic vein based on a physical augmentation modeling engine;

[0101] Step S5: Plan the surgical path based on hemodynamic parameters and assess the risk level of different options;

[0102] Step S6: Intraoperative real-time registration and model update, providing AR navigation and force feedback simulation.

[0103] The liver surgery simulation system based on cascaded neural networks and multi-scale reconstruction provided by this invention includes the following core modules:

[0104] The cascaded recursive segmentation network module employs a two-stage cascaded structure. The first stage uses a U-Net++ network with multi-scale feature pyramids for coarse liver segmentation. The second stage introduces a recursive sub-network with an attention gating mechanism to fine-tune the coarse segmentation results, particularly enhancing feature extraction for low-contrast regions. The loss function combines Dice loss and boundary-sensitive loss, improving the boundary intersection-union ratio by more than 12%.

[0105] Multi-scale dynamic registration module: Based on unsupervised learning, a recursive cascaded registration network aligns CT arterial / portal venous phase images with MRI-T2 sequences, reducing deformation and folding through cycle consistency loss. An integrated LSTM network predicts liver displacement during the respiratory cycle, dynamically updating the registration field, with registration error controlled within 1.5 mm.

[0106] The physical enhancement modeling engine embeds the Navier-Stokes fluid dynamics equations into vascular network reconstruction to simulate hepatic vein pressure gradients and optimize vascular branch angles and diameter ratios. By combining PHANTOM force feedback equipment with finite element analysis, it calculates the stress distribution of the scalpel on liver tissue in real time with an accuracy of 0.1N.

[0107] Generative data augmentation module: Generates liver images with specific attributes using conditional generative adversarial networks; synthesizes high-fidelity images through a diffusion model via a "stepwise noise addition-reverse noise reduction" process; establishes a quality control pipeline, including automated quality assessment indicators and a human-machine collaborative review mechanism.

[0108] Adaptive computing framework: Provides a unified SQL entry point to support federated computing; an optimizer based on historical load predicts the optimal execution engine; enables edge-cloud collaborative computing, dynamically offloading computationally intensive tasks to the cloud and processing real-time tasks at the edge.

[0109] Example 1: Implementation of a Cascaded Recursive Partition Network Module

[0110] The coarse segmentation unit uses a U-Net++ network structure, with the encoder using a ResNet-50 backbone, and the input is a 512×512 pixel abdominal CT image. The fine segmentation unit, based on the coarse segmentation, employs a GRU network with attention gating for three rounds of recursive optimization, with the attention weights dynamically adjusted according to the tumor boundary contrast.

[0111] The loss function is a combination of weighted Dice loss and boundary-sensitive loss: L = α·L_Dice + β·L_Boundary;

[0112] Where α=0.7, β=0.3, and the boundary-sensitive loss focuses on the pixel classification accuracy within a 5mm range at the junction of tumor and normal liver tissue;

[0113] Example 2: Implementation of the multi-scale dynamic registration module

[0114] The multimodal fusion unit employs a VoxelMorph network architecture, taking CT arterial phase and MRI T2-weighted images as input, and outputting a deformation field through an encoder-decoder structure. The motion compensation unit uses a three-layer LSTM network, taking diaphragmatic motion signals from 10 consecutive respiratory cycles as input, and outputting a predicted liver displacement for the next 2 seconds.

[0115] Example 3: Implementation of the Generative Data Augmentation Module

[0116] A conditional generative adversarial network (GAN) takes tumor size, type, and location as input to generate 256×256 pixel liver CT images. The diffusion model uses a DDPM architecture, guiding the generation process with textual descriptions, such as "well-defined 3 cm hepatocellular carcinoma enhanced CT image." The quality control system uses the FID (Fréchet Inception Distance) metric to evaluate the quality of the generated images, with a threshold set at 15. Images below this threshold are subject to manual review.

[0117] Example 4: Implementation of an Adaptive Computing Framework

[0118] The unified query interface supports standard SQL syntax, enabling federated queries across data sources such as Hive and MySQL. The intelligent routing decision unit, based on the random forest algorithm, selects the optimal computing engine (PyTorch, TensorFlow, or Monai) according to query complexity, data volume, and real-time requirements. The edge-cloud collaboration unit employs Kubernetes container orchestration technology to achieve elastic scaling of computing nodes.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A liver segmentation, modeling, and surgical simulation system based on cascaded neural networks and multi-scale reconstruction technology, characterized in that, include: A cascaded recursive segmentation network module is used for coarse and fine segmentation of liver and lesion regions in medical images; A multi-scale dynamic registration module is used for non-rigid registration and respiratory motion compensation in multimodal medical images; A physical enhancement modeling engine for building 3D models of liver vascular trees that include hemodynamic properties; The surgical navigation and risk assessment module is used for real-time intraoperative navigation and surgical path planning; Generative data augmentation module, used to synthesize and expand training datasets; An adaptive computing framework for the dynamic scheduling and management of distributed computing resources; The cascaded recursive segmentation network module, the multi-scale dynamic registration module, and the physical enhancement modeling engine are connected in sequence. The surgical navigation and risk assessment module has a bidirectional data connection with the physical augmentation modeling engine; The generative data augmentation module is connected to the training data input end of the cascaded recursive segmentation network module. The adaptive computing framework provides underlying computing resource support for the system.

2. The liver segmentation, modeling, and surgical simulation system based on cascaded neural networks and multi-scale reconstruction technology according to claim 1, characterized in that: The cascaded recursive segmentation network module includes: The coarse segmentation unit uses a U-Net++ network with a feature pyramid structure for initial liver segmentation; The fine segmentation unit employs a recurrent neural network with attention gating to fine-tune the tumor boundary; The loss function optimization unit combines Dice loss and boundary-sensitive loss function for model training.

3. The liver segmentation, modeling, and surgical simulation system based on cascaded neural networks and multi-scale reconstruction technology according to claim 1, characterized in that: The multi-scale dynamic registration module includes: A multimodal fusion unit is used to achieve non-rigid registration of CT / MRI images based on a recursive cascaded registration network. The motion compensation unit uses an LSTM network to predict liver displacement during the respiratory cycle; The deformation field optimization unit reduces deformation folding through cyclic consistency loss, and the registration error is controlled within 1.5mm.

4. The liver segmentation, modeling, and surgical simulation system based on cascaded neural networks and multi-scale reconstruction technology according to claim 1, characterized in that: The physical augmentation modeling engine includes: A vascular topology reconstruction unit was used to reconstruct the three-dimensional hepatic vascular network based on an improved Marching Cubes algorithm. The hemodynamic simulation unit embeds the Navier-Stokes equation to simulate the hepatic vein pressure gradient; The force feedback modeling unit calculates the stress distribution of surgical instruments on liver tissue through finite element analysis.

5. The liver segmentation, modeling, and surgical simulation system based on cascaded neural networks and multi-scale reconstruction technology according to claim 1, characterized in that: The generative data augmentation module includes: Conditional generative adversarial networks are used to generate liver pathological images with specific attributes. A diffusion model unit is used to synthesize high-fidelity liver tumor images; The quality control system includes automated quality assessment indicators and a human-machine collaborative review mechanism.

6. The liver segmentation, modeling, and surgical simulation system based on cascaded neural networks and multi-scale reconstruction technology according to claim 1, characterized in that: The adaptive computing framework includes: A unified query interface supports federated queries based on the ANSI SQL standard. Intelligent routing decision unit, based on SQL characteristics and historical load prediction optimal calculation engine; Edge-cloud collaborative units enable dynamic offloading of computing tasks and elastic scaling of resources.

7. The liver segmentation, modeling, and surgical simulation system based on cascaded neural networks and multi-scale reconstruction technology according to claim 1, characterized in that: The surgical navigation and risk assessment module includes: The AR navigation unit overlays a 3D model onto the real field of vision during surgery using a head-mounted display device; The risk warning unit dynamically calculates the risk coefficient based on the shortest distance between the tumor and blood vessels; The real-time calibration unit updates the registration field using intraoperative images, resulting in a navigation latency of less than 100ms.

8. A method for liver segmentation, modeling, and surgical simulation based on cascaded neural networks and multi-scale reconstruction technology, characterized in that: Includes the following steps: Step S1: Acquire the patient's multimodal medical imaging data, including CT arterial phase, portal venous phase, and MRI sequences; Step S2: Segment the liver parenchyma and lesions in the image using a cascaded recursive segmentation network to generate a segmentation mask; Step S3: Use the multi-scale dynamic registration module to achieve accurate registration of multi-phase images and compensate for respiratory motion; Step S4: Construct three-dimensional models of the hepatic artery, portal vein, and hepatic vein based on a physical augmentation modeling engine; Step S5: Combine hemodynamic parameters to plan the surgical path and assess the risk level of different options; Step S6: Intraoperative real-time registration and model update, providing AR navigation and force feedback simulation.

Images