Real-time abdominal puncture virtual simulation training method and device

Abstract

A real-time virtual simulation training method and device for abdominal puncture, the method comprising: (1) obtaining a CT image of a target patient and generating a corresponding three-dimensional virtual facet model; (2) tetrahedronizing the model, changing the hardness and softness of different regions of the model according to physiological characteristics, obtaining the boundary state of different regions, and organically merging the model with a triangular facet mesh to synchronize visual changes with force changes; (3) traversing detection to obtain the position information of the collision point between the surgical tool and the organ tissue, calculating the speed, force, and damping at different times, and transmitting the information to a force feedback device; (4) converting the patient's CT into ultrasound, overlapping and stacking the ultrasound image and the model at the same position, attaching slices to the virtual ultrasound probe, and cutting the slices into the stacked ultrasound image at different angles; (5) marking the needle placement points planned before the operation, and the operator during the operation fits the previously planned landmark points based on real-time ultrasound navigation, and comprehensively scoring the operation points based on the gap between the operation points and the landmark points and the operation time.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a real-time abdominal puncture virtual simulation training method and a real-time abdominal puncture virtual simulation training device. Background Art

[0002] Virtual reality medical training technology has gradually matured and is being used by a variety of surgical professionals. Its primary purpose is to provide doctors with a realistic, immersive, and risk-free surgical training environment, shortening their learning cycle. With the development of biomedicine, numerous medical tools have become available, among which paracentesis has become a routine treatment method. However, training with this complex instrument often requires extensive training and a long familiarization process to master it. Medical students with undeveloped skills face many ethical issues when practicing clinically, and the amount of training required is limited by the number of cases. Therefore, paracentesis simulation technology, supported by virtual reality technology, has gradually come into people's attention, shortening doctors' training cycles without compromising medical ethics.

[0003] Therefore, this invention focuses on real-time abdominal puncture simulation with force feedback. While existing abdominal puncture teaching simulations have become popular, they still present many challenges: 1. Ultrasound guidance is typically used during a real abdominal puncture, but this is often not addressed in virtual simulations. 2. When puncturing the same organ, the tactile sensation often changes with the depth of the puncture, but existing puncture devices do not adequately simulate these changes in force feedback.

[0004] Chinese patent CN201610863627.9 produced a force feedback puncture simulation system with kinetic energy provided by three DC motors, but was unable to provide force feedback effects for puncturing the same organ at different depths; secondly, for example, Chinese patent CN201710977489.1 produced a device for simulating puncture operations, but it did not mention virtual ultrasonic real-time navigation in the virtual space; another example is patent CN202111206308.8, which produced a visual simulation teaching system for a puncture surgery robot, which has key modules such as preoperative planning, intraoperative execution, and postoperative evaluation consistent with the present invention, but did not mention the function of customizing force feedback settings. Summary of the Invention

[0005] In order to overcome the defects of the existing technology, the technical problem to be solved by the present invention is to provide a real-time abdominal puncture virtual simulation training method, which highly restores the real surgical environment in the virtual simulation process, has simulated real-time ultrasound guidance, and is planned according to the real surgical process. It has preoperative planning, intraoperative navigation execution, postoperative evaluation and other contents, adopts a mature force feedback system device and is developed in a mature system. It has strong scalability and adaptability, and adopts CTFigure 1 Compared with the reconstructed virtual model, the structure is clear and the details are complete, and the new results can be reconstructed quickly based on the CT image.

[0006] The technical solution of the present invention is: this real-time abdominal puncture virtual simulation training method comprises the following steps:

[0007] (1) First, obtain the CT image of the target patient, and then generate the corresponding 3D virtual surface model based on semi-automatic threshold segmentation;

[0008] (2) After obtaining the patient-based face model, tetrahedronize it, change the softness and hardness of different regions of the model according to physiological characteristics, obtain the boundary state of different regions, and then organically merge it with the triangular face mesh to synchronize the visual changes with the force perception changes;

[0009] (3) Obtain the collision point information between the surgical tool and the organ tissue through traversal detection, calculate the physical information such as speed, force, and damping at different times, and transmit it to the force feedback device so that the user can feel the mechanical properties of the model in real time;

[0010] (4) Convert the patient's CT scan to ultrasound, overlap the ultrasound image and the model at the same position, and then attach slices to the virtual ultrasound probe, and cut the slices into the stacked ultrasound image at different angles;

[0011] (5) First, mark the needle placement points planned before the operation. During the operation, the operator fits the previously planned landmark points based on real-time ultrasound navigation, and then comprehensively scores based on the difference between the actual operation point and the planned landmark point and the operation time.

[0012] The present invention first obtains a CT image of the target patient, and then generates a corresponding three-dimensional virtual patch model based on semi-automatic threshold segmentation; after obtaining the patch model based on the patient, it is tetrahedronized, and the softness and hardness of different areas of the model are changed according to physiological characteristics, the boundary state of different areas is obtained, and then it is organically merged with the triangular patch grid to synchronize its visual changes with force changes; traversal detection is performed to obtain the position information of the collision point between the surgical tool and the organ tissue, and physical information such as speed, force, and damping at different times is calculated, and transmitted to the force feedback device so that the user can feel the mechanical properties of the model in real time; the patient's CT is converted into ultrasound, and the ultrasound image and the model are re-imaged at the same position. The slices are stacked together, and then slices are tied to the virtual ultrasound probe, and the slices are cut into the stacked ultrasound images at different angles; the needle placement points planned before the operation are marked first, and the intraoperative operator fits the previously planned landmark points according to the real-time ultrasound navigation, and then comprehensively scores according to the gap between the actual operation point and the planned landmark point and the operation time; therefore, the present invention highly restores the real surgical environment in the virtual simulation process, has simulated real-time ultrasound guidance, and is planned according to the real surgical process. It has preoperative planning, intraoperative navigation execution, postoperative evaluation and other contents, adopts mature force feedback system devices, and is developed in mature systems, has strong ductility and adaptability, and adopts CT Figure 1 Compared with the reconstructed virtual model, the structure is clear and the details are complete, and the new results can be reconstructed quickly based on the CT image.

[0013] Also provided is a real-time abdominal puncture virtual simulation training device, which includes:

[0014] The 3D modeling module first obtains the CT image of the target patient and then generates the corresponding 3D virtual surface model based on semi-automatic threshold segmentation;

[0015] The soft tissue modeling module obtains a patient-based face model and then tetrahedronizes it. It then changes the softness and hardness of different regions of the model according to physiological characteristics, obtains the boundary state of different regions, and then organically merges it with the triangular facet mesh to synchronize visual changes with force perception.

[0016] The collision detection and force feedback module detects and obtains the collision point information between the surgical tool and the organ tissue, calculates the physical information such as speed, force, and damping at different times, and transmits it to the force feedback device so that the user can feel the mechanical properties of the model in real time;

[0017] The ultrasound navigation module converts the patient's CT scan into ultrasound, overlays the ultrasound image with the model at the same location, and then attaches slices to the virtual ultrasound probe, which cut into the stacked ultrasound image at different angles.

[0018] In the comprehensive evaluation module, the preoperatively planned needle placement points are first marked. During the operation, the operator fits the previously planned landmark points based on real-time ultrasound navigation, and then comprehensively scores the patient based on the gap between the actual operation point and the planned landmark point and the operation time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention is a flowchart of a real-time virtual simulation training method for abdominal puncture. DETAILED DESCRIPTION

[0020] like Figure 1 As shown, this real-time abdominal puncture virtual simulation training method includes the following steps:

[0021] (1) First, obtain the CT image of the target patient, and then generate the corresponding 3D virtual surface model based on semi-automatic threshold segmentation;

[0022] (2) After obtaining the patient-based face model, tetrahedronize it, change the softness and hardness of different regions of the model according to physiological characteristics, obtain the boundary state of different regions, and then organically merge it with the triangular face mesh to synchronize the visual changes with the force perception changes;

[0023] (3) Obtain the collision point information between the surgical tool and the organ tissue through traversal detection, calculate the physical information such as speed, force, and damping at different times, and transmit it to the force feedback device so that the user can feel the mechanical properties of the model in real time;

[0024] (4) Convert the patient's CT scan to ultrasound, overlap the ultrasound image and the model at the same position, and then attach slices to the virtual ultrasound probe, and cut the slices into the stacked ultrasound image at different angles;

[0025] (5) First, mark the needle placement points planned before the operation. During the operation, the operator fits the previously planned landmark points based on real-time ultrasound navigation, and then comprehensively scores based on the difference between the actual operation point and the planned landmark point and the operation time.

[0026] The present invention first obtains a CT image of the target patient, and then generates a corresponding three-dimensional virtual patch model based on semi-automatic threshold segmentation; after obtaining the patch model based on the patient, it is tetrahedronized, and the softness and hardness of different areas of the model are changed according to physiological characteristics, the boundary state of different areas is obtained, and then it is organically merged with the triangular patch grid to synchronize its visual changes with force changes; traversal detection is performed to obtain the position information of the collision point between the surgical tool and the organ tissue, and physical information such as speed, force, and damping at different times is calculated, and transmitted to the force feedback device so that the user can feel the mechanical properties of the model in real time; the patient's CT is converted into ultrasound, and the ultrasound image and the model are re-imaged at the same position. The slices are stacked together, and then slices are tied to the virtual ultrasound probe, and the slices are cut into the stacked ultrasound images at different angles; the needle placement points planned before the operation are marked first, and the intraoperative operator fits the previously planned landmark points according to the real-time ultrasound navigation, and then comprehensively scores according to the gap between the actual operation point and the planned landmark point and the operation time; therefore, the present invention highly restores the real surgical environment in the virtual simulation process, has simulated real-time ultrasound guidance, and is planned according to the real surgical process. It has preoperative planning, intraoperative navigation execution, postoperative evaluation and other contents, adopts mature force feedback system devices, and is developed in mature systems, has strong ductility and adaptability, and adopts CT Figure 1 Compared with the reconstructed virtual model, the structure is clear and the details are complete, and the new results can be reconstructed quickly based on the CT image.

[0027] Those skilled in the art will appreciate that all or part of the steps in the above-described embodiment method can be implemented by instructing the relevant hardware through a program. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the above-described embodiment method. The storage medium can be ROM / RAM, a magnetic disk, an optical disk, a memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a real-time abdominal puncture virtual simulation training device, which is generally represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0028] The 3D modeling module first obtains the CT image of the target patient and then generates the corresponding 3D virtual surface model based on semi-automatic threshold segmentation;

[0029] The soft tissue modeling module obtains a patient-based face model and then tetrahedronizes it. It then changes the softness and hardness of different regions of the model according to physiological characteristics, obtains the boundary state of different regions, and then organically merges it with the triangular facet mesh to synchronize visual changes with force perception.

[0030] The collision detection and force feedback module detects and obtains the collision point information between the surgical tool and the organ tissue, calculates the physical information such as speed, force, and damping at different times, and transmits it to the force feedback device so that the user can feel the mechanical properties of the model in real time;

[0031] The ultrasound navigation module converts the patient's CT scan into ultrasound, overlays the ultrasound image with the model at the same location, and then attaches slices to the virtual ultrasound probe, which cut into the stacked ultrasound image at different angles.

[0032] In the comprehensive evaluation module, the preoperatively planned needle placement points are first marked. During the operation, the operator fits the previously planned landmark points based on real-time ultrasound navigation, and then comprehensively scores the patient based on the gap between the actual operation point and the planned landmark point and the operation time.

[0033] Preferably, the collision detection and force feedback module has peripheral tools, which can be customized, disassembled and assembled by laser 3D printing according to different needs, so that the device is suitable for a variety of different surgical tools and procedures.

[0034] Preferably, the ultrasound navigation module uses a deep learning method to convert the CT image into an ultrasound image.

[0035] Preferably, in the ultrasound navigation module, pixels are rendered onto slices, and then images of the slices are displayed.

[0036] Preferably, the three-dimensional modeling module is modeled using CT data or MRI in DICOM format, wherein the CT data is shot with a layer thickness of 1.00 mm, the object is the abdominal cavity, and is stored in DICOM format.

[0037] Preferably, in the soft tissue modeling module, based on the MITK library, threshold screening and segmentation are performed according to the different grayscale ranges of different tissues. After obtaining the preliminary model, the mesh is refined for key parts based on the opinions of professional doctors, and the mesh is deleted for non-critical parts, and stretching and rotation are used to make the final model conform to the physiological structure characteristics, and finally saved as a triangular patch file in stl format.

[0038] Preferably, the soft tissue modeling module sets specific flexibility coefficients for different areas of the model, and merges the soft force feedback model with the visual model to synchronize force feedback with vision; the internal structure of the abdominal cavity divides tissues into four categories: skin, muscle, fat, and bone, and assigns different biomechanical parameters and constraints to different types of models under external force conditions, and the biomechanical parameters of different parts of the same organ can be adjusted arbitrarily.

[0039] Preferably, the collision detection and force feedback module determines the collision position and collision physical parameters between the virtual surgical instrument and the virtual model, so as to guide the input of the model deformation and force feedback value; the virtual surgical instrument model is a matrix produced by 3D printing based on a 1:1 ratio of the real surgical instrument model, and the collision detection model is given an AABB algorithm to determine whether there is a collision. In each frame, the collision point between the surgical tool and the physical model is cyclically detected to obtain its collision point information. The collision detection box of all instruments is larger than its own diameter by 0.1 mm; after obtaining the collision speed, direction, force magnitude, and momentum information of certain points at a certain moment, the numerical value is input into the force feedback model to complete the physical simulation of force feedback; the force feedback device module uses a 3D-system Touch pen device, which is connected to the computer via a USB interface and has the open source OpenHaptics driver toolkit installed.

[0040] Preferably, the intraoperative navigation module simulates the actual surgical process, and obtains ultrasound images of different parts through a real-time ultrasound system to play the role of intraoperative guidance and positioning; after obtaining the CT data, the initial ultrasound image is obtained through deep learning style transfer, refined, and then stacked into the system for one-to-one position matching with the virtual model, and then a plane is bound to the front end of the virtual ultrasound probe. The plane obtains the pixel point information of the corresponding position that overlaps with it and copies it, so that when the plane cuts into the stacked body generated by ultrasound, the plane can obtain ultrasound slices at different angles, and finally upload this slice to the UI interface; the postoperative evaluation module, after obtaining the positioning information planned before surgery, stores the positioning, and compares and matches it with the operation in the surgical simulation, and calculates the final score through time, position deviation, angle deviation, and false touch accumulation.

[0041] The above-mentioned force feedback device can be designed in appearance according to different needs, and its hardware peripherals can be modified to produce a force feedback peripheral that is consistent with real surgical tools.

[0042] In some embodiments of the present application, the model and its scene may be rendered with color, texture, and lighting, and updated in real time when the model is deformed or displaced.

[0043] The embodiments of the present application provide a real-time abdominal puncture virtual simulation device and method with force feedback. This device aims to transform the mechanical properties of a human biomodel into a virtual simulated physical model by organizing the key steps and key points of the surgical operation, and then use a force feedback device to perform virtual surgical operations. The virtual surgical simulation device described in the embodiments of the present application has the advantages of high authenticity and real-time performance. Furthermore, the entire device is a modular combination, and different surgical tools and surgical models can be quickly customized according to the doctor's needs, which is highly flexible and convenient.

[0044] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of protection of the technical solution of the present invention.

Claims

1. Real-time abdominal puncture virtual simulation training device, characterized by: It includes: The 3D modeling module first obtains the CT image of the target patient and then generates the corresponding 3D virtual surface model based on semi-automatic threshold segmentation; The soft tissue modeling module obtains a patient-based face model and then tetrahedronizes it. It then changes the softness and hardness of different regions of the model according to physiological characteristics, obtains the boundary state of different regions, and then organically merges it with the triangular facet mesh to synchronize visual changes with force perception. The collision detection and force feedback module detects and obtains the collision point information between the surgical tool and the organ tissue, calculates the physical information such as speed, force, and damping at different times, and transmits it to the force feedback device so that the user can feel the mechanical properties of the model in real time; The ultrasound navigation module converts the patient's CT scan into ultrasound, overlays the ultrasound image with the model at the same location, and then attaches slices to the virtual ultrasound probe, which cut into the stacked ultrasound image at different angles. The comprehensive evaluation module first marks the preoperatively planned needle placement points. During the operation, the operator uses real-time ultrasound navigation to fit the previously planned landmarks. The operator then provides a comprehensive score based on the difference between the actual operation point and the planned landmarks and the operation time. The collision detection and force feedback module determines the collision location and physical parameters between the virtual surgical instrument and the virtual model to guide the input of model deformation and force feedback values. The virtual surgical instrument model is a matrix produced by 3D printing based on a 1:1 ratio of the real surgical instrument model. The collision detection model uses an AABB algorithm to determine whether there is a collision. In each frame, the collision point between the surgical tool and the physical model is cyclically detected to obtain its collision point information. The collision detection box of all instruments is larger than 0.1 mm in diameter. After obtaining the collision speed, direction, force magnitude, and momentum information of certain points at a certain moment, the numerical value is input into the force feedback model to complete the physical simulation of force feedback. The force feedback device module uses a 3D-system touch pen device, which is connected to the computer via a USB interface and has the open source OpenHaptics driver toolkit installed. The intraoperative navigation module simulates the real surgical process and obtains ultrasound images of different parts through a real-time ultrasound system to play the role of intraoperative guidance and positioning; after obtaining the CT data, the initial ultrasound image is obtained through deep learning style transfer, refined, and then stacked into the system for one-to-one position matching with the virtual model. Then, a plane is bound to the front end of the virtual ultrasound probe. The plane obtains the pixel point information of the corresponding position that overlaps with it and copies it. In this way, when the plane cuts into the stacked body generated by ultrasound, the plane can obtain ultrasound slices at different angles, and finally upload this slice to the UI interface; the postoperative evaluation module, after obtaining the positioning information of the preoperative plan, stores the positioning, and compares and matches it with the operations in the surgical simulation, and calculates the final score through time, position deviation, angle deviation, and accumulated false touches.

2. The real-time abdominal puncture virtual simulation training device according to claim 1 is characterized in that: The collision detection and force feedback module has peripheral tools that can be customized, disassembled, and assembled by laser 3D printing according to different needs, making the device suitable for a variety of different surgical tools and procedures.

3. The real-time abdominal puncture virtual simulation training device according to claim 2 is characterized in that: The ultrasound navigation module uses a deep learning method to convert CT images into ultrasound images.

4. The real-time abdominal puncture virtual simulation training device according to claim 3 is characterized by: In the ultrasound navigation module, pixels are rendered onto slices, and then images of the slices are displayed.

5. The real-time abdominal puncture virtual simulation training device according to claim 4 is characterized in that: The three-dimensional modeling module is modeled using CT data or MRI in DICOM format, wherein the CT data is shot with a layer thickness of 1.00 mm, the object is the abdominal cavity, and is stored in DICOM format.

6. The real-time abdominal puncture virtual simulation training device according to claim 5, characterized in that: In the soft tissue modeling module, based on the MITK library, threshold screening and segmentation are performed according to the different grayscale ranges of different tissues. After obtaining a preliminary model, the mesh is refined for key parts based on the opinions of professional doctors, and the mesh is deleted for non-critical parts. Stretching and rotation are used to make the final model conform to the physiological structure characteristics, and finally it is saved as a triangular patch file in STL format.

7. The real-time abdominal puncture virtual simulation training device according to claim 6, characterized in that: The soft tissue modeling module sets specific flexibility coefficients for different areas of the model and merges the soft body force feedback model with the visual model to synchronize force feedback with vision. The internal structure of the abdominal cavity divides tissues into four categories: skin, muscle, fat, and bone. Different types of models are given different biomechanical parameters and constraints under external force. Different parts of the same organ can arbitrarily adjust their biomechanical parameters.

Images