Tricuspid valve interventional operation image simulation method based on Unity engine

Abstract

The invention discloses a tricuspid valve interventional operation image simulation method based on a Unity engine, and the method comprises the steps: building an X-ray simulation system of a cubic model which comprises a heart and a surrounding region; establishing an ultrasonic simulation system of a heart model, wherein the heart model comprises a heart internal structure; and superposing the X-ray simulation system and the ultrasonic simulation system on a UI canvas. According to the invention, highly realistic X-ray images and ultrasonic images are provided, and the system has real-time interaction capability and is suitable for multiple scenes such as medical teaching, preoperative planning, surgical skill training and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image simulation, and in particular to a tricuspid valve interventional surgery image simulation method based on a Unity engine. Background Art

[0002] Apical regurgitation is a common valvular heart disease, and interventional surgery is an effective treatment. However, interventional procedures are complex, requiring doctors to rely on X-rays and ultrasound images. Traditional training methods rely on a realistic surgical environment, exposing doctors to prolonged X-ray radiation and posing health risks. Furthermore, existing surgical simulators lack image simulation and interactive experience, failing to provide a highly realistic surgical environment.

[0003] Currently, there are several surgical simulators based on game engines, such as Unity-based laparoscopic and orthopedic surgery simulators. However, these simulators primarily focus on simulating general surgical procedures, lacking specialized simulation systems for cardiac interventional surgery. Existing cardiac interventional surgery simulators lack image simulation, interactive experience, and real-time performance. For example, an interactive cardiac interventional surgery simulator can only provide 3D images and corresponding 2D sections of the heart within the engine, but cannot provide the X-ray and ultrasound images actually used during surgery. A preoperative planning and virtual reality simulation method for minimally invasive cardiac surgery, although developed based on a game engine, only provides 3D reconstructions generated from 2D images, but cannot provide the X-ray and ultrasound images actually used during surgery. Existing cardiac interventional surgery simulators only provide 3D images and 2D sections, failing to simulate the X-ray / ultrasound images used in actual surgery. This results in a lack of immersion in the real surgical environment for physicians during training. Furthermore, real-time rendering of ultrasound images based on game engines consumes significant computing resources, resulting in poor real-time performance and a lack of consideration for physical phenomena such as scattering and refraction, resulting in a low level of simulation.

[0004] Therefore, technicians in this field are committed to providing a tricuspid valve interventional surgery image simulation method based on the Unity engine to achieve a highly simulated, high-efficiency, and low-cost surgical training environment. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is how to provide a tricuspid valve interventional surgery image simulation method that can be used for interventional cardiac surgery training.

[0006] To achieve the above objectives, the present invention provides a tricuspid valve interventional surgery image simulation method based on the Unity engine, comprising:

[0007] An X-ray simulation system is established with a cubic model, wherein the cubic model includes the heart and surrounding areas;

[0008] Establishing an ultrasound simulation system for a heart model, wherein the heart model includes an internal structure of the heart;

[0009] The X-ray simulation system and the ultrasound simulation system are superimposed on the UI canvas.

[0010] Furthermore, the X-ray simulation system for establishing a cube model includes:

[0011] adding pre-processed CT data to the cube model;

[0012] Using a movable X-ray emitting point as a starting point, an X-ray is generated between the back surface of the cube model and the starting point;

[0013] Color mixing is performed along the X-ray line to form an X-ray simulation image on the back side of the cube model.

[0014] Furthermore, the preprocessing of the CT data includes generating a three-dimensional texture of grayscale values ​​and gradient values ​​according to the CT data, and then generating a two-dimensional texture according to X-ray sampling points of a transfer function.

[0015] Preferably, the upper point of the transfer function controls the transparency value corresponding to the grayscale value, and the lower point controls the color value corresponding to the grayscale value.

[0016] Furthermore, the forming of the X-ray simulation image includes reading the grayscale gradient texture value of the X-ray sampling point, obtaining the color value of the X-ray sampling point, mixing the colors along the X-ray line, and drawing the accumulated colors into the X-ray simulation image.

[0017] Furthermore, the ultrasound simulation system for establishing a heart model includes:

[0018] Disposing the heart model and the probe model inside the cube model;

[0019] An ultrasonic emission point is arranged at the end of the probe to generate ultrasonic light;

[0020] An ultrasound simulation image is generated based on the light intensity on the ultrasound beam.

[0021] Preferably, ultrasonic sampling points are evenly distributed on the ultrasonic light, and the ultrasonic sampling points acquire the intensity of the ultrasonic light.

[0022] Furthermore, when generating the ultrasonic simulation image, the reflection process is simulated according to the collision detection of ultrasonic light in the forward and reverse directions.

[0023] Preferably, a binary tree structure is used to record information of the collision point and determine the position of the ultrasonic sampling point.

[0024] Furthermore, superimposing the X-ray simulation system and the ultrasound simulation system includes controlling the X-ray emission point and the ultrasound emission point, and displaying the emission point position, the current position of the surgical instrument, and the guide position on the same interface.

[0025] The present invention has at least the following beneficial technical effects:

[0026] The Unity engine-based tricuspid valve interventional surgery imaging simulation method, developed by the present invention, utilizes volume rendering technology, a ray casting algorithm, and binary tree-based ray collision detection to achieve a highly realistic, efficient, and low-cost surgical training environment. This method not only provides highly realistic X-ray and ultrasound images but also offers real-time interactive capabilities, making it suitable for a variety of scenarios, including medical education, preoperative planning, and surgical skills training.

[0027] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of a tricuspid valve interventional surgery image simulation method based on the Unity engine according to an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of an X-ray simulation process according to an embodiment of the present invention;

[0030] Figure 3 is a schematic diagram of an X-ray rendering transfer function according to an embodiment of the present invention;

[0031] Figure 4 is a schematic diagram of a ray casting algorithm according to an embodiment of the present invention;

[0032] Figure 5 is a schematic diagram of the ultrasonic imaging process according to an embodiment of the present invention;

[0033] Figure 6 This is a schematic diagram of the collision point detection and sequence generation principle of an embodiment of the present invention;

[0034] Figure 7 This is a flow chart of sampling point determination based on a binary tree according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following describes preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0036] In the drawings, components with identical structures are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrary and are not limited by the present invention. For clarity, the thickness of components in some places in the drawings is appropriately exaggerated.

[0037] The present invention provides a tricuspid valve interventional surgery image simulation method based on the Unity engine. By introducing volume rendering technology, ray casting algorithm and ray collision detection based on a binary tree structure, a highly simulated, high-efficiency and low-cost surgical training environment is achieved.

[0038] like Figure 1 As shown, the tricuspid valve interventional surgery image simulation method based on the Unity engine of this embodiment includes the following processes: establishing an X-ray simulation system of a cube model, wherein the cube model includes the heart and the surrounding area; establishing an ultrasound simulation system of a heart model, wherein the heart model includes structural information inside the heart; and superimposing the X-ray simulation system and the ultrasound simulation system on the UI canvas.

[0039] For an X-ray simulation system, we first need CT data of the heart and surrounding areas. We then use an import program to generate three-dimensional grayscale and gradient textures for Unity volume rendering based on this data. We then need to set the transfer function sampling points to control the display effect, and an X-ray image can be formed in the scene.

[0040] like Figure 2 As shown, the specific process of establishing the X-ray simulation system of the cube model in this embodiment is as follows: adding the preprocessed CT data to the cube model; using the movable X-ray emission point as the starting point, generating X-rays between the back of the cube model and the starting point; performing color mixing along the X-rays, and forming an X-ray simulation image on the back of the cube model.

[0041] In this example, we first generate a three-dimensional texture of grayscale values ​​and gradient values ​​based on the CT data, and then generate a two-dimensional texture of the transfer function based on the sampling points of the transfer function. The design of the transfer function is as follows: Figure 3 As shown in the figure, the upper point is used to control the transparency value corresponding to different grayscale values, setting the transparency value of the heart and nearby tissues to a higher value; the lower point is used to control the color value corresponding to different grayscale values, setting the heart and bone areas to bright white; this information is passed to the material to prepare for shader rendering. Use the shader to render the cube, calculate the starting point and end point of each ray based on the bounding box principle, and generate sampling points in between. The method for calculating the end point and sampling points is as follows Figure 4 As shown, the grayscale gradient texture value of the sampling point is read, the color value of the sampling point is obtained according to the transfer function texture, the colors are mixed along the light, and the final accumulated color is drawn into an X-ray image.

[0042] For ultrasound simulation, it is necessary to import a heart STL model containing complete internal information, then set mesh collision detection and physical properties based on the program, and add an imaging plane to the scene to form a heart ultrasound image.

[0043] like Figure 5 As shown, the specific process of the ultrasound simulation system for establishing a heart model in this embodiment is as follows: the heart model and probe model are placed inside a cube model; an ultrasound emission point is configured at the end of the probe to generate an ultrasound beam; and an ultrasound simulation image is generated based on the intensity of the ultrasound beam. Ultrasonic sampling points are evenly distributed along the ultrasound beam, and the ultrasound beam intensity is obtained at the ultrasound sampling points.

[0044] In this embodiment, the detection plane and sampling point position are first generated according to the position and rotation angle of the emission origin, and the collision detection of light in the forward and reverse directions is used to simulate the reflection process in ultrasound imaging. Two positive and negative rays are constructed between the starting point and the end point of each ray. If the light collides with the model, the RaycastHit structure at the collision point and the order in which it collides on the ray are recorded. Then, a new ray is generated with a starting point that is offset by a small amount in the positive direction of projection based on the collision point, and the positive direction is the same as the original ray direction, until the distance between the collision point and the emission origin exceeds the sound field range. The principles of forward and reverse collision point detection and order generation are as follows: Figure 6 As shown. The collision order of all collision bodies in forward and reverse collisions and the RaycastHit structure are combined to form a new structure inputHits, whose members include the collision order Index and the collision point position calculated according to the RaycastHit structure.

[0045] This embodiment uses a binary tree structure to determine the position of the collision point. Specifically, the cosine value of the collision point angle is calculated based on the collision point model normal and the incident light direction of the RaycastHit structure, and these structures are formed into an array for constructing a binary tree. Each time, the collision point in the middle of the array is selected as the root node, and the binary tree is recursively constructed. The distinction between left and right nodes is based on their distance from the ultrasound emission origin. The ultrasound probe is always located in the -y direction of the heart model. If the y value of a collision point is greater than the y value of the root node, the collision point is far away and is assigned to the left subtree. If the y value of the collision point is less than the y value of the root node, the collision point is nearby. After the binary tree is established, each sampling point on the scan line is detected, and the two collision points in the front and back directions closest to it are determined based on its position in the world coordinate system. The detection logic is as follows: Figure 7As shown, the purpose of the test is to determine whether each sampling point is inside the model. If so, the depth and angle of the sampling point can be calculated based on the position and angle information of the nearest collision points. This allows the reflected sound intensity at that point to be determined by combining Lambert's law and the exponential decay law. Furthermore, the sound intensity is recalculated for each reflection to simulate the attenuation process. After obtaining the reflected sound intensity at each point, it is plotted on a two-dimensional plane. Gaussian blur and random noise are added to the image to simulate refraction and scattering.

[0046] After obtaining the real-time generated X-ray and ultrasound images, they are added to the same canvas in the Unity system's built-in UI system. The X-ray and ultrasound emission points are controlled by the program to simulate the position adjustment process of the imaging equipment during the operation. Auxiliary information such as the emission point position, the current position of the surgical instrument, and the guide position are displayed on the same interface, thus realizing the tricuspid valve interventional surgery image simulation process based on the Unity engine.

[0047] The present invention's Unity engine-based tricuspid valve interventional surgery image simulation method, in X-ray image simulation, uses CT data-based volume rendering technology and transfer function design to truly restore the visual effects of DSA images; in ultrasound image simulation, through a ray collision detection algorithm based on a binary tree structure, combined with the reflection, refraction, and attenuation principles in ultrasound propagation, it significantly improves image quality and operating efficiency, achieving high-speed rendering of only about 34 milliseconds per frame, greatly improving efficiency. This simulation method is built on the Unity engine, has good cross-platform compatibility and scalability, and can support further integration of virtual reality equipment and artificial intelligence evaluation modules; the fully virtualized training environment effectively avoids the health risks of doctors being exposed to radiation for a long time in traditional training, and comprehensively improves the safety and practicality of medical training.

[0048] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A tricuspid valve interventional surgery image simulation method based on Unity engine, characterized in that: include: An X-ray simulation system is established with a cubic model, wherein the cubic model includes the heart and surrounding areas; Establishing an ultrasound simulation system for a heart model, wherein the heart model includes an internal structure of the heart; The X-ray simulation system and the ultrasound simulation system are superimposed on the UI canvas.

2. The tricuspid valve interventional surgery image simulation method based on the Unity engine according to claim 1, characterized in that: The X-ray simulation system for establishing a cube model includes: adding pre-processed CT data to the cube model; Using a movable X-ray emitting point as a starting point, an X-ray is generated between the back surface of the cube model and the starting point; Color mixing is performed along the X-ray line to form an X-ray simulation image on the back side of the cube model.

3. The tricuspid valve interventional surgery image simulation method based on the Unity engine according to claim 2, characterized in that: The preprocessing of the CT data includes generating a three-dimensional texture of grayscale values ​​and gradient values ​​according to the CT data, and then generating a two-dimensional texture according to X-ray sampling of a transfer function.

4. The tricuspid valve interventional surgery image simulation method based on the Unity engine according to claim 3, characterized in that: The upper point of the transfer function controls the transparency value corresponding to the grayscale value, and the lower point controls the color value corresponding to the grayscale value.

5. The tricuspid valve interventional surgery image simulation method based on Unity engine according to claim 4, characterized in that: The forming of the X-ray simulation image includes reading the grayscale gradient texture value of the X-ray sampling point, obtaining the color value of the X-ray sampling point, mixing the colors along the X-ray line, and drawing the accumulated colors into the X-ray simulation image.

6. The tricuspid valve interventional surgery image simulation method based on Unity engine according to claim 1, characterized in that: The ultrasound simulation system for establishing a heart model includes: Disposing the heart model and the probe model inside the cube model; An ultrasonic emission point is arranged at the end of the probe to generate ultrasonic light; An ultrasound simulation image is generated based on the light intensity on the ultrasound beam.

7. The tricuspid valve interventional surgery image simulation method based on Unity engine according to claim 6, characterized in that: Ultrasonic sampling points are evenly distributed on the ultrasonic light, and the ultrasonic sampling points acquire the intensity of the ultrasonic light.

8. The tricuspid valve interventional surgery image simulation method based on Unity engine according to claim 7, characterized in that: When generating the ultrasonic simulation image, the reflection process is simulated according to the ultrasonic light collision detection in the forward and reverse directions.

9. The tricuspid valve interventional surgery image simulation method based on Unity engine according to claim 8, characterized in that: A binary tree structure is used to record the information of the collision point and determine the location of the ultrasonic sampling point.

10. The tricuspid valve interventional surgery image simulation method based on Unity engine according to claim 1, characterized in that: Superimposing the X-ray simulation system and the ultrasound simulation system includes controlling the X-ray emission point and the ultrasound emission point, and displaying the emission point position, the current position of the surgical instrument, and the guide position on the same interface.

Images