[0074] The technical solutions of the present invention will be further described in detail through specific embodiments in conjunction with the accompanying drawings.
[0075] The present invention provides a method of using implicit surface algorithm to simulate soft tissue surgery path planning. Here, liver tissue is taken as an example of soft tissue. The steps of the method are as follows:
[0076] S1: Obtain medical imaging data of the liver of a specific case;
[0077] S2: Establish a three-dimensional geometric model, specifically, use a visualization algorithm to generate a mesh model of the liver tissue structure, or use the volume rendering data of the medical imaging software as the volume element model;
[0078] For example, algorithms such as Marching Cubes are often used for 3D model reconstruction of medical image data. Many open source tools such as VTK and commercial software that comes with clinical imaging equipment have these functions;
[0079] The volume rendering data comes from medical imaging equipment. Many open source tools such as VTK and commercial software that comes with clinical imaging equipment have these functions. The application purpose of the present invention is surgical planning, and the initial data is from the case images of the surgical target (for example, CT/MRI).
[0080] S3: Obtain the characteristic data of the device for resection, including the corresponding relationship between the configuration of the mechanical structure of the device and the related parameter settings of the device power and the amount of tissue resection or ablation. The acquisition method is actual measurement or provided by the device manufacturer;
[0081] S4: Based on the human-computer interaction interface design of the integrated surgical simulator, the operation of the liver resection surgical instrument or model is tracked and located, the relative position of the hepatectomy surgical instrument or model is calculated, and its relative position to the anatomical model is calculated, and the amount of resection instrument operation at each moment is detected;
[0082] The operation simulator designed by computer simulation has tracking and positioning functions. At the same time, the surgical robot system also has the function of positioning and detecting surgical instruments. Surgery planning function modules can be integrated into surgical robot systems or computer-assisted surgery systems, as well as surgical simulators and medical imaging diagnostic equipment;
[0083] S5: Based on the implicit surface algorithm, apply the operation amount to the liver tissue model, and calculate the effect amount corresponding to the current operation amount according to the characteristics of the instrument. The effect amount includes the effective ablation area and ablation contour of the tissue by the surgical instrument;
[0084] Specifically, according to the mechanical contour of the surgical instrument and the distribution characteristics of the ultrasonic energy in the blade, the amount of ablation and resection of the liver tissue and the amount of damage to the blood vessels and related tissues are calculated. Using the implicit surface algorithm, doctors can generate different wound contours corresponding to the different effects of each operation, and continuously modify and update the overall resection path very reliably;
[0085] S6: Edit and modify the tissue model according to the calculated amount of action, and display the dynamic changes including the path extension of the cut surface during the operation and the extension of the wound boundary contour;
[0086] The boundary contour changes are automatically generated by the above simulation method: new wounds are generated by using the implicit surface algorithm and integrated with the existing 3D model; the location of the newly added wounds or the existing wounds that have changed The position of is determined by step S4; the shape of the boundary contour and the size of the corresponding resection amount are determined by step S5.
[0087] S7: Based on other simulation functions that have been realized by the integrated surgical simulator, including soft tissue deformation and bleeding hemostasis, the resection process of a specific case can be displayed in real time by means of human-computer interaction.
[0088] The present invention can be integrated with a conventional computer surgery simulator, and the main purpose is to use the human-computer interaction interface of the computer surgery simulator. Requirements: The system needs to have real-time/interactive computing capabilities.
[0089] Specifically, due to product design considerations such as cost, if the computational efficiency of a computer cannot meet the standard of real-time response, secondary simulation functions can be removed: for example, ignoring special effects such as bleeding and simplifying the simulation of tissue deformation ( Quoting some existing fast algorithms), the software module only realizes the function of surgical path planning, and the hardware provides a man-machine interactive operation interface for it.
[0090] The core content of surgical path planning is to realize the gradual generation and change process of the wound on the case model. In addition, regarding the design and implementation of simulation functions of virtual surgery scenes, and the integration of these functions with the resection path simulation calculation module, the existing technology has provided us with feasible examples.
[0091] In step S1, the medical image data is X-ray tomography (CT) or magnetic resonance tomography (MRI) data.
[0092] The system flow diagram based on the method of the present invention: medical imaging data (CT tomography, magnetic resonance), using medical visualization techniques such as image segmentation to reconstruct the 3D anatomical model of the body element (voxel) of the organs of the surgical site, or The volume element model uses visual 3D reconstruction technology to generate a mesh model, uses surgical simulation modeling technology to establish an interactive model, and uses computer graphics and rendering technology in the field of visualization to dynamically display the interactive model on the display device;
[0093] The interactive model, separation resection simulation and the generated surgical path are all part of the main program of the real-time simulation simulation system (also includes other common technical components, please refer to the review literature on the surgical simulator or virtual surgical system); real-time simulation The simulation updates the interactive model based on the operation volume detected and collected by the human-computer interaction interface, and the surgical path that is gradually generated and extended;
[0094] An illustration of the separation and resection simulation based on the method of the present invention: the separation and resection data can be obtained by experimental surgery or estimated by Molecular Dynamics Simulations. (Note: Molecular dynamics simulation cannot reach the computational efficiency of real-time simulation, and the results obtained are the same as experimental data, which can be used as preliminary reference data.) According to the amount of operation, after fitting, the amount of action of separation resection is obtained, and the The wound module gradually calculates the new wound formed by the operation at the current time based on the amount of action and the interaction model at the previous moment, and updates the surgical path in a gradual manner.
[0095] In the present invention, the connotations of related technical components are as follows:
[0096] 1. Medical image data: Obtain medical image data of the surgical site of a specific case. Preferably, it may be an image such as clinical X-ray tomography (CT) or magnetic resonance (MRI).
[0097] 2. Visualization: The image segmentation technology in medical visualization is mainly used here to extract and reconstruct the three-dimensional anatomical model of the organs and tissues of the surgical site, and determine the boundary, internal structure, and relative position relationship between the models.
[0098] 3. Body element model: a three-dimensional model of organ organization composed of many voxels.
[0099] 4. Visualization: Visualization technology is used here to generate a mesh model from the volume element model. Implicit surface algorithm is a type of algorithm in visualization technology, which can extract, display, and modify isosurface from volume elements.
[0100] 5. Mesh model: a two-dimensional or three-dimensional geometric model composed of polygons or polyhedrons, and its surface texture, material rendering properties and other settings.
[0101] 6. Modeling: According to the characteristics of the morphological structure and physical properties of the organ tissue, define its response and change rules during the operation, and use a series of computer simulation algorithms to establish a solution mechanism on the three-dimensional model (refer to related Literature review of surgical simulation simulation). The morphological structure can be defined by a mesh model or a volume element model.
[0102] 7. Interactive model: According to the operator's input during the operation, a simulation model that simulates, updates and displays the corresponding dynamic changes of organs in real time.
[0103] 8. Separation resection data: Separation resection data reflects the distribution of ablation resection energy and the amount of action in the target tissue. The data can be obtained by experiments or calculated by simulations such as Molecular Dynamics Simulations. Our application area is mainly precise path planning for minimally invasive resection. Different from mechanical resection tools such as scalpel scissors in traditional open surgery, in modern minimally invasive surgery, controllable energy ablation resection instruments are generally used for important separation and resection operations, such as applying ultrasound, laser and other energy to local tissues. Produce liquefaction, gasification and other ablation effects, and realize fine separation and resection operations.
[0104] 8. Separation resection simulation: Fit the separation resection data, calculate the amount of separation resection by the operator's operation, and generate the wound surface.
[0105] 9. Surgical path: As the surgical operation progresses, the generated wound surface on the interactive model gradually forms an overall separation resection boundary contour. The surgical path includes the marks left by the separation and resection of the organs and tissues, as well as the operation path information.
[0106] 10. Rendering: Use computer graphics display card products, and call and write 3D rendering programs, or call and write volume rendering programs in visualization technology to show the appearance and structure of the 3D model.
[0107] 11. Display: The display can use a flat-panel display, a three-dimensional holographic display, or a head-mounted display (HMD) such as virtual reality (VR) and augmented display (AR) as needed.
[0108] 12. Human-computer interaction interface: a user interface device for users to interact with virtual surgical scenes, generally mainly composed of surgical instrument models, and motion detection sensors or motion tracking devices.
[0109] 13. Operation volume: the measurement of surgical operations through sensors or motion tracking equipment.
[0110] 14. Fitting: Based on the known separation resection data, calculate the change of the interaction model from the currently measured operation volume to make it match the known data as much as possible.
[0111] 15. Amount of action: the changes in the current interaction model.
[0112] 16. Generate wounds: According to the calculated effects of the current separation and resection operation, edit and modify the interactive model, and add new surgical wounds on it.
[0113] Surgical operation volume tracking detection -> Calculation of the amount of ablation resection -> Separation cutting algorithm editing (explicit or implicit surface) -> Generate excision wound -> Update the surgical path.
[0114] The present invention is the first to propose the use of artificial intelligence machine learning technology to perform separated resection data fitting-from the known separated resection data, calculate the amount of action currently measured on the interaction model to make it consistent with the known reference data .
[0115] Objective: To estimate the volume of the resected tissue and its spatial distribution under the action of a given ablation resection instrument power, relative position/distance to the tissue and other variables.
[0116] Compared with existing methods, this method can accurately determine the contour of the ablation resection boundary and the surgical path formed by it.
[0117] When the boundary changes gradually and progressively, the amount of separation and resection in each period can be defined as the difference between the existing boundary and the new boundary. In a three-dimensional space, the total amount of this difference is the volume. In order to completely determine the amount of separation and resection, we must not only determine the total amount, but also determine the distribution of this volume on the existing boundary. Therefore, we define the “effect” here as the distribution of volume along the boundary surface, that is, the volume volume corresponding to each point on the boundary surface.
[0118] Two common methods of machine learning, such as support vector machine (SVM) method and neural network method, can be applied to this problem of the present invention.
[0119] 1.1 Data and characteristics of machine learning
[0120] Data: The separation resection data reflects the distribution of the ablation resection energy and the amount of action in the target tissue. The data can be obtained by experiments or calculated by simulations such as Molecular Dynamics Simulations.
[0121] The energy is emitted from the device and spread to the surroundings. The tissues located in the effective area are affected by energy, and their physical state changes.
[0122] The experiment can measure the effect of energy on tissue specimens. For example, after the laser energy is absorbed by the tissue, heat is generated in the tissue, and some solid tissue is vaporized; after the ultrasonic energy is absorbed by the tissue, some solid tissue is emulsified. After the specimens are processed by inverting molds and slicing, the range (position and volume) of the tissue's gaseous and liquid phase changes can be measured.
[0123] Simulation can calculate the effect of energy on the tissue model. For example, molecular dynamics simulation can calculate the distribution of energy, temperature and other values in the tissue, and can calculate the range of solid tissues to reach gas and liquid phase transition conditions.
[0124] feature:
[0125] power
[0126] Coordinates/distance
[0127] Action time
[0128] Organizational differences
[0129] Goal: To determine and estimate the location/range of tissue gasification and ablation and other phase changes.
[0130] Problem: This fitting problem can be defined as a regression problem, or it can be defined as a classification problem.
[0131] Solving the regression problem is to determine the distribution of ablation energy in the tissue based on experimental results or molecular dynamics simulation, and then determine the range of tissue liquefaction or vaporization from the energy distribution results.
[0132] Solving the classification problem is to directly divide the scope of liquefaction or gasification based on experimental or molecular dynamics simulation data. For example, using the direct results of molecular dynamics simulations, we can classify the virtual tissues of the interactive model. One is the tissue that has not been separated and resected but is still remaining, and the other is the tissue that has been separated and resected in the current period. In this way, the problem of solving the effect is transformed into a binary classification problem in machine learning.
[0133] The geometric model of the 3D model is usually a mesh structure, such as a polygonal mesh surface or a polyhedral mesh, and volumetric rendering (volumetric rendering) can also be directly performed on the voxel of the medical image data.
[0134] For a long time, computer 3D graphics technology and equipment have been designed based on the surface rendering method of mesh models.
[0135] At present, the surface rendering effect of the mesh model has more special effect options and is faster. Volume rendering requires preprocessing of the volume element model, which currently cannot achieve the fidelity of the mesh model surface rendering.
[0136] The volume rendering model more completely includes the structural information of the 3D solid model, such as the anatomical structure inside the model boundary. When constructing complex structures, the volume element model is more concise and reliable than the mesh model.
[0137] This technical solution uses a separation and resection algorithm based on an implicit surface, and can use either a mesh model or a body element model composed of medical image data. The work of reconstructing the geometric model of the mesh becomes optional rather than necessary. This technical solution can realize path planning with interactive surgery simulation simulation on deformable soft tissue models such as liver.
[0138] The process of reconstructing the grid model and then building an interactive simulation model is complicated. Many computer simulation algorithms have strict requirements on the quality of the grid. Therefore, professionals are often required to manually proofread and optimize the grid reconstruction and splitting results, as well as test verification. . The reconstruction of the grid model for each patient-specific patient-specific cannot be completed automatically, and the process is complicated; the process of establishing a dynamic simulation model of the virtual patient's organs in the surgical simulator is even more complicated. These two main technical difficulties restrict the feasibility and practicability of the idea of using a surgical simulation simulator as a surgical planning system.
[0139] The two methods of mesh model reconstruction and volume element model volume rendering have been commonly used in the medical image application fields of clinical diagnosis such as radiology; the separation resection module in the existing surgical simulation technology products generally uses the mesh reconstruction model, The body element model is only used for "rigid body" models such as dental orthopedics. The main technical problem that needs to be solved when applied to the soft tissue model is to use the body element model to simulate the separation and deformation of the soft tissue.
[0140] The early results of the inventor of the present application and other prior art examples show that the implicit surface algorithm can be used to realize a soft tissue surgery simulation simulation system.
[0141] This technical solution further proposes to directly use the body element model to construct a simulation and path planning system for soft tissue separation and resection, and provides the option of directly jumping from the body element model to the interactive model.
[0142] The implicit surface algorithm of this application can edit and modify body element model data.
[0143] Volume element data, or volume data, is the data corresponding to the 8 vertices (sampling points) of the cube occupied by each voxel in the three-dimensional sampling space where the volume element model is located. These data values collected by medical imaging equipment usually reflect some of the physical values and tissue properties corresponding to the vertex positions, such as magnetic resonance amplitude, X-ray transmission distribution, ultrasonic transmission distribution, as well as the tissue density and biomaterial properties derived from transformation. These data are modified and updated by the separation and resection simulation module in the system.
[0144] As a visualization algorithm, the implicit surface algorithm of this application can dynamically extract isosurfaces from these volume element volume data. The isosurfaces can be constructed from a mesh surface model, which is then transferred to the processing flow of the mesh model. Model simulation and path planning options. You can also skip the construction steps of the grid model mentioned above and jump to modules 3, 6, and 7 in the system, namely the volume element and volume rendering options.
[0145] This application provides the following related technical solutions and answers to the new design based on volume rendering: The problem of too high resolution of volume model:
[0146] If the original data resolution of some medical images is too high compared to the real-time simulation load of soft tissue model deformation and separation and resection, common optimization techniques such as down sampling and adaptive resolution can be used. , Reduce the amount of calculation of the real-time module of the system.
[0147] Whether the resolution of the volume element model is too low, the volume rendering does not have an intuitive boundary surface in the mesh model, and whether the clarity of the mesh model can be achieved.
[0148] Body element models and volume rendering have been widely used in clinical medical imaging diagnostic equipment, including some surgical planning software for keyboard and mouse stroke operations, which seem to meet the requirements of clinical work and have practical value.
[0149] With the volume element and volume rendering options, is it possible to realize automatic surgical path planning or automatic surgical simulation for any specific case?
[0150] The original volume element image data is still in the medical tomography format, and the reconstruction of the three-dimensional volume element model requires visualization processing such as image segmentation. The current reconstruction technology is generally in a semi-automated stage.
[0151] The main advantages of the volume rendering option: significantly reduce the complexity and time cost of the modeling process, and improve the modeling efficiency and practicality of preoperative path planning.
[0152] figure 1 It is the existing liver surgery planning software of reference [11]-draw a line to display the cutting results, Figure 2 is the cut surface effect in the existing liver surgery simulator of reference [8]-display subdivision method, image 3 Yes Reference [9] The cut surface effect in the existing liver surgery simulator-display subdivision method, Figure 4 It is the linear cutting effect of the existing liver surgery planning VR system in reference [5-7]-explicit subdivision method. Figure 5 is the effect of the present invention using the implicit ablation resection method to simulate the liver and other tissues-the resection path boundary can be control. The present invention uses man-machine interactive surgery simulation to plan soft tissue surgery paths. Compared with traditional surgical planning software, it can give users a more comprehensive and realistic pre-operative preview experience. On the basis of the existing minimally invasive surgery simulator, the present invention uses an implicit method to generate a cut surface, which is more accurate than the existing soft tissue surgery simulator using an explicit method, and can meet precise planning, analysis and rehearsal surgery. The actual needs of the path have practical value. The present invention can not only be used for the grid model data generated by visualization software, but also can omit this modeling link, and be directly used for the specific case data provided by medical imaging equipment, which is significant for the application of surgical planning technology in clinical practice. Save time and cost.
[0153] The above-mentioned embodiment is only a preferred solution of the present invention, and does not impose any formal restriction on the present invention. There are other variations and modifications without exceeding the technical solution described in the claims.
