Simulator, simulation method, simulation program, and simulator system
The simulator system addresses the complexity of simulating bleeding and smoke in surgical training by using CPU and GPU to model heat propagation and fluid dynamics, offering realistic graphical effects for enhanced training realism.
Patent Information
- Application Number
- PCT/JP2024/016001
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-18
- Filing Date
- 2024-04-24
- Publication Date
- 2024-11-21
AI Technical Summary
Conventional surgical simulation techniques often omit or simplify graphical effects like bleeding and smoke from heated surgical instruments, due to high computational costs and complexity in combining different simulation models.
A simulator system that models heat generation and propagation in living tissue using a CPU and GPU, simulating bleeding with Position Based Fluids dynamics and smoke using smoke particles, allowing for realistic rendering of bleeding and smoking during surgical procedures.
Enables real-time simulation of surgeries involving heated surgical tools, providing accurate and realistic graphical effects of bleeding and smoke, improving the realism of surgical training simulations.
Smart Images

Figure JP2024016001_21112024_PF_FP_ABST
Abstract
Description
Simulator, simulation method, simulation program, and simulator system
[0001] The present disclosure relates to a simulator, a simulation method, a simulation program, and a simulator system.
[0002] For example, as disclosed in Patent Document 1, various techniques relating to the simulation of cutting a three-dimensional object have been proposed.
[0003] Japanese Patent Application Laid-Open No. 2003-141566
[0004] When biological tissue is cut during surgery, bleeding may occur. When a heated surgical tool such as an electric scalpel is used, smoking may also occur. However, simulations that include such bleeding and smoking are not specifically considered in Patent Document 1.
[0005] One aspect of the present disclosure allows for the simulation of surgery, including bleeding and smoking.
[0006] A simulator according to one aspect of the present disclosure includes a processing unit including at least one processor, and the processing unit simulates at least one of bleeding and smoking from biological tissue based on heat generation in the biological tissue caused by an energy device and heat propagation within the biological tissue.
[0007] A simulation method according to one aspect of the present disclosure includes simulating at least one of bleeding and smoking from biological tissue based on heat generated in the biological tissue by an energy device and heat propagation within the biological tissue.
[0008] A simulation program according to one aspect of the present disclosure causes a computer to execute a process of simulating at least one of bleeding and smoking from biological tissue based on heat generated in the biological tissue by an energy device and heat propagation within the biological tissue.
[0009] A simulator system according to one aspect of the present disclosure includes an input device that accepts user operations to operate an energy device, and a display device that displays at least one of bleeding and smoking from the biological tissue simulated based on heat generated by the energy device and heat propagation within the biological tissue.
[0010] 1 is a diagram illustrating an example of a schematic configuration of a simulator system 100 according to an embodiment. FIG. 1 is a diagram illustrating an example of functional blocks of the simulator 1. FIG. 1 is a diagram illustrating an example of roles of a CPU 61 and a GPU 62 in a processing unit 6. FIG. 1 is a diagram illustrating an example of particles 20 and rigid bodies 30. FIG. 2 is a diagram illustrating an example of constraints. FIG. 3 is a diagram illustrating an example of constraints. FIG. 4 is a flowchart illustrating an example of processing (simulation method) executed in the simulator 1. FIG. 5 is a flowchart illustrating a detailed example of step S210. FIG. 6 is a diagram illustrating an example of collision pair detection. FIG. 7 is a flowchart illustrating a detailed example of steps S220 and S230. FIG. 8 is a flowchart illustrating an example of constraint solving. FIG. 9 is a flowchart illustrating a detailed example of step S240. FIG. 10 is a flowchart illustrating a detailed example of steps S250 and S260. FIG. 11 is a flowchart illustrating a detailed example of step S270. FIG. 12 is a diagram illustrating an overlap check. FIG. 13 is a flowchart illustrating an example of processing (simulation method) executed in the simulator 1. FIG. 14 is a diagram illustrating steps S410 to S470. FIG. 15 is a flowchart illustrating an example of processing (simulation method) executed in the simulator 1. FIG. 16 is a diagram illustrating step S500. FIG. 17 is a diagram illustrating an example of the presence or absence of bleeding. FIG. 18 is a flowchart illustrating a detailed example of step S710. FIG. 19 is a diagram illustrating an example of spatial hash calculation. Fig. 10 is a flowchart showing a detailed example of step S720 and step S730. Fig. 11 is a diagram showing an example of density constraints. Fig. 12 is a flowchart showing a detailed example of step S740 and step S750. Fig. 13 is a flowchart showing a detailed example of step S760. Fig. 14 is a diagram showing a modified example. Fig. 15 is a diagram showing a modified example. Fig. 16 is a diagram showing an example of the hardware configuration of the device.
[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same elements are designated by the same reference numerals, and redundant description will be omitted.
[0012] The present disclosure will be described in the following order: 0. Introduction 1. Embodiments 1.1. Modeling and Dynamics Simulation of Biological Tissue and Surgical Tools 1.2. Heat Transfer Simulation and Smoke Generation Simulation 1.3. Fluid Simulation 2. Modifications 3. Example of Hardware Configuration 4. Conclusion
[0013] 0. Introduction In conventional surgical simulation technology, graphical effects (also called visual effects), such as bleeding and smoke, that can occur when using heated surgical tools such as electric scalpels, have been omitted or limited to very simplified effects. This is because achieving such graphical effects requires the use of different simulation models and algorithms, which results in high computational costs and makes it difficult to combine them. Rendering realistic blood and smoke also traditionally requires computational costs.
[0014] According to the disclosed technology, for example, to simulate surgery using a heating tool such as an electric scalpel, the heating tool is modeled as an energy device that heats biological tissue, and the heating of biological tissue by the energy device and heat propagation within the biological tissue are simulated. Bleeding is simulated using position-based fluid (PBF) dynamics. Smoke is modeled using smoke particles, which are rendered and realistically represented.
[0015] In one embodiment, a central processing unit (CPU) and a graphics processing unit (GPU) are used in combination to provide a real-time simulation of the cutting of biological tissue with a surgical tool, as well as the resulting bleeding and smoking.
[0016] 1. Embodiment Fig. 1 is a diagram showing an example of a schematic configuration of a simulator system 100 according to an embodiment. The simulator system 100 includes a simulator 1, a haptic device 8, and a display 9. A user of the simulator system 100 is referred to as a user U. In Fig. 1, only the hand portion of the user U is schematically shown.
[0017] The simulator 1 executes a simulation in accordance with a simulation scenario 10. One example of a simulation is a surgical simulation, which is illustrated in FIG. 1. The simulation scenario 10 includes several virtual objects related to the surgery. As objects, FIG. 1 illustrates a soft object, a rigid object, a robot arm 4, and a camera 5. When there is no need to distinguish between soft objects and rigid objects, they are simply referred to as objects.
[0018] An example of a soft object is biological tissue such as an organ, which is illustrated as biological tissue 2. An example of a rigid object is a surgical tool, which is illustrated as surgical tool 3. Examples of the surgical tool 3 include a scalpel, forceps, etc., and the scalpel may be a heating surgical tool such as an electric scalpel or an ultrasonic scalpel, i.e., an energy device. Unless otherwise specified, the surgical tool 3 is considered to include an energy device. To the extent that there is no contradiction, the surgical tool 3 may be appropriately interpreted as an energy device.
[0019] The robot arm 4 is a surgical robot arm that supports a camera 5. The camera 5 is a surgical camera that captures an image of a surgical field including the biological tissue 2 and the surgical tool 3. The field of view of the camera 5 is illustrated as a field of view 5f. Note that the term "image" may be interpreted to include video, and these terms may be interpreted interchangeably as appropriate within the scope of no contradiction.
[0020] The haptic device 8 (tactile device) is an input device that accepts operation of the surgical tool 3. The user U operates the haptic device 8, for example, by touching it with their hands, in order to move the surgical tool 3. When the user U operates the haptic device 8 to move the surgical tool 3, the surgical tool 3 treats the biological tissue 2, for example, by clamping or cutting, thereby progressing the surgical simulation. When the surgical tool 3 collides (contacts) with (comes into) contact with the biological tissue 2, a change in the position or orientation of the biological tissue 2 is simulated. The force (which can also be called a contact force or a reaction force) generated by this interaction is reflected in the operation of the haptic device 8 and is fed back to the user U. Note that, within the scope of no contradiction, the terms "collision" and "contact" may be interpreted as appropriate.
[0021] The display 9 displays an image within the field of view 5f of the camera 5, i.e., an image including at least a portion of the biological tissue 2 and the surgical tool 3. The image also includes simulation results (described below), such as bleeding and smoking (described below) that may occur due to treatment of the biological tissue 2 with the surgical tool 3. The user U operates the haptic device 8 while viewing the image displayed on the display 9. The position, angle, etc. of the camera 5 may also be operated by the user U. The operation unit for the camera 5 may be incorporated into the haptic device 8 or may be provided separately from the haptic device 8. The operation of the camera 5 may be performed automatically. For example, the function of a control unit that automatically operates the camera 5 may be included in the simulation scenario 10.
[0022] 2 is a diagram showing an example of functional blocks of the simulator 1. The simulator 1 includes a processing unit 6 and a storage unit 7.
[0023] The processing unit 6 includes at least one processor. In FIG. 2, a CPU 61 and a GPU 62 are illustrated as examples of the processor.
[0024] The storage unit 7 stores information used by the simulator 1. An example of information stored in the storage unit 7 is a program 70. The program 70 is a simulation program (software, application program) for causing a computer to function as the simulator 1. The storage unit 7 also includes a CPU memory 71 and a GPU memory 72. The CPU memory 71 is memory used by the CPU 61 when executing various processes. The GPU memory 72 is memory used by the GPU 62 when executing various processes.
[0025] Motion is input to the processing unit 6 via the haptic device 8. The motion is information such as position, posture, and force. The processing unit 6 corresponds the motion of the surgical tool 3 to the motion input via the haptic device 8. The user U operates the surgical tool 3 via the haptic device 8. The processing unit 6 also calculates the contact force generated by the collision between the biological tissue 2 and the surgical tool 3. The contact force is sent from the processing unit 6 to the haptic device 8 and fed back to the user U via the haptic device 8.
[0026] As explained above, the simulation scenario 10 includes the camera 5, and the captured image is displayed on the display 9. The image captured by the camera 5 is sent from the processing unit 6 to the display 9 and displayed thereon.
[0027] FIG. 3 is a diagram showing an example of the roles of the CPU 61 and the GPU 62 in the processing unit 6.
[0028] The CPU 61 models the biological tissue 2 and the surgical tool 3 in accordance with the simulation scenario 10. As will be described in detail later, the CPU 61 models the biological tissue 2 using a plurality of particles 20. The CPU 61 also models the surgical tool 3 using rigid bodies 30 having zero or more collision shapes 31. The surgical tool 3 may include a plurality of rigid bodies 30, and in this case, the rigid bodies 30 may be connected to each other by kinematic joints J. The tip (or tip portion) of the surgical tool 3 is illustrated and referred to as tip 3T.
[0029] Various data (simulation data) relating to the biological tissue 2 and the surgical tool 3 modeled by the CPU 61 are copied (transferred) from the CPU 61 to the GPU 62, more specifically from the CPU memory 71 to the GPU memory 72. The simulation data can be described in an array, as will be described later.
[0030] The GPU 62 calculates the changes in the positions of the particles 20 in the biological tissue 2, and the changes in the position and orientation of the rigid body 30 in the surgical tool 3, which occur when the modeled biological tissue 2 and the surgical tool 3 collide, so as to satisfy the given constraints. A solver for solving the constraints is implemented on the GPU 62 (a parallel solver for soft and rigid body dynamics).
[0031] Although details will be described later, the solver performs various calculations using data related to the particles 20, constraints, rigid bodies 30, collision shapes 31, kinematic joints J, etc. as input data. As a result, various data such as the particles 20, rigid bodies 30, kinematic joints J, and trigger shapes (described later) are output as output data. The output data is copied back from the GPU 62 to the CPU 61, more specifically, from the GPU memory 72 to the CPU memory 71.
[0032] As described above, the surgical tool 3 includes an energy device such as an electric scalpel. When the surgical tool 3 comes into contact with the biological tissue 2 during treatment of the biological tissue 2, the biological tissue 2 generates heat and is cut or otherwise subjected to other treatments.
[0033] The processing unit 6 simulates at least one of bleeding and smoking from the biological tissue 2 based on bleeding and smoking caused by treatment of the biological tissue 2 with the surgical tool 3, more specifically, based on heat generated in the biological tissue 2 by the surgical tool 3 and heat propagation within the biological tissue 2. An overview will be given here with reference to Fig. 3 , and details will be described later. Note that in the simulation described below, the surgical tool 3 (energy device) is treated as a heat emitter, and heat transfer from the surgical tool 3 to the biological tissue 2 causes heat generation in the biological tissue 2.
[0034] In the simulation, heat (thermal energy) is applied to particles 20 in the biological tissue 2 that are in thermal contact with the surgical tool 3, causing the temperature of the particles 20 to rise. The particles 20 to which heat has been applied are referred to as heated particles 21 and are shown in the figure.
[0035] To simulate smoking, the CPU 61 removes particles 20 that have reached the vaporization temperature, i.e., heated particles 21 in an overheated state, from the biological tissue 2. The vaporization temperature may be set arbitrarily, and an example thereof is 100° C. Together with the heated particles 21, polyhedron elements (e.g., tetrahedron elements described below) connected thereto may also be removed.
[0036] To simulate smoking, the CPU 61 uses the smoke particles 22 to model smoke generated from the biological tissue 2 at a portion where the heated particles 21 that have reached the vaporization temperature have been removed. To obtain a realistic graphical effect of smoke, the smoke particles 22 may be rendered using a known graphics engine such as UnrealEngine®.
[0037] The bleeding simulation is executed in the GPU 62. For this purpose, data (simulation data) including the biological tissue 2 after the heating particles 21 have been removed is copied from the CPU memory 71 to the GPU memory 72.
[0038] To simulate bleeding, the GPU 62 uses a plurality of fluid particles 24 to model blood flowing out from a portion of the biological tissue 2 where the heated particles 21 that have reached their vaporization temperature have been removed. The GPU 62 calculates the positions, velocity changes, etc. of the plurality of fluid particles 24 so as to satisfy density constraints (density conservation constraints) of the plurality of fluid particles 24. A solver for solving the density constraints is implemented on the GPU 62 (a fluid dynamics parallel solver). The density constraints include, for example, a constraint that the plurality of fluid particles 24 maintain a rest density.
[0039] More specifically, prior to processing by the GPU 62, the CPU 61 samples the plurality of boundary particles 23 located on the surface of the biological tissue 2 more densely than the plurality of fluid particles 24 contained in the biological tissue 2. Data including this is copied from the CPU memory 71 to the GPU memory 72 as described above. The GPU 62 calculates the positions of the plurality of fluid particles 24 so as to satisfy density constraints on the plurality of fluid particles 24 and the plurality of boundary particles 23. The density constraint includes, for example, a constraint that a fluid particle 24 that is close to a boundary particle 23 (for example, located within an influence radius) among the plurality of fluid particles 24 maintains a rest distance from the boundary particle 23.
[0040] The GPU 62 also calculates changes in the velocities of the plurality of fluid particles 24 so as to satisfy the velocity constraints of the plurality of fluid particles 24. Examples of velocity constraints include a velocity constraint due to viscosity, a velocity constraint due to vorticity, a velocity constraint due to adhesion, a velocity constraint due to absorption, a velocity constraint due to friction, a velocity constraint due to cohesion, and a velocity constraint due to surface tension.
[0041] Data indicating the results of the calculations performed by the GPU 62 described above, such as data including the positions of each fluid particle 24 , is copied back from the GPU memory 72 to the CPU memory 71 .
[0042] The CPU 61 transmits image data including blood modeled by a plurality of fluid particles 24 and smoke modeled by the smoke particles 22 to the display 9. The blood and smoke may be rendered by a known graphics engine and then displayed. Realistic graphics of bleeding and smoke during surgery can be displayed on the display 9.
[0043] The above explanation of Fig. 3 is an overview of the simulation. The specific procedure of the simulation will now be explained.
[0044] 1.1. Modeling of Biological Tissue and Surgical Tool and Dynamics Simulation Modeling of the biological tissue 2 and surgical tool 3 and simulation of soft body and rigid body dynamics by the CPU 61 are performed based on the XPBD (extended position-based dynamics) method. First, the CPU 61 models the biological tissue 2 using a plurality of particles 20, and models the surgical tool 3 using rigid bodies 30 having zero or more collision shapes 31. The particles 20 and rigid bodies 30 will be described with reference to FIGS. 4 and 5.
[0045] 4 is a diagram showing an example of the particle 20 and the rigid body 30. The coordinates common to the particle 20 and the rigid body 30 (and more specifically, the biological tissue 2 and the surgical tool 3 including them) are referred to as global coordinates C global It is called.
[0046] For ease of understanding, the particle 20 is illustrated schematically as a sphere having a constant radius r. The particle 20 may also be a point particle. The particle 20 is described using various physical quantities (physical properties). As physical quantities that describe the particle 20, mass m, position p, and velocity v are exemplified in FIG. 4 .
[0047] The rigid body 30 is also described using various physical quantities. Examples of physical quantities that describe the rigid body 30 include mass m, position p, orientation q, velocity v, angular velocity ω, and inertia I, as shown in FIG. 4 . The position and velocity of the rigid body 30 may be the central position and central velocity of the rigid body 30.
[0048] If the particles 20 and the rigid body 30 are treated as static, their masses may be set to zero. Also, if the rotation of the rigid body 30 is fixed, the inertia I of the rigid body 30 may be set to zero.
[0049] Particles 20 may collide with each other, and rigid bodies 30 may also collide with each other. Furthermore, particles 20 and rigid bodies 30 may collide with each other. In a collision involving a rigid body 30, the collision shape 31 possessed by the rigid body 30 is taken into consideration. Zero or more collision shapes 31 are attached to the rigid body 30. By combining multiple collision shapes 31, a variety of collision shapes can be attached to the rigid body 30 as a whole.
[0050] When the rigid body 30 has a plurality of collision shapes 31, each collision shape 31 may have a different basic shape (geometric shape), size, orientation, etc. The orientation of the collision shape 31 is expressed in, for example, the global coordinate system C global Alternatively, the relative posture (relative position and orientation) may be defined in local coordinates based on the center of mass of the rigid body 30.
[0051] 4 illustrates three types of collision shapes 31 having different basic shapes. To distinguish between the collision shapes 31, they are referred to as collision shape 31-1, collision shape 31-2, and collision shape 31-3. When no particular distinction is made between them, they are simply referred to as collision shapes 31.
[0052] The basic shape of the collision shape 31-1 is a sphere. The collision shape 31 having a spherical shape is also simply called a "sphere."
[0053] The basic shape of the collision shape 31-2 is a capsule shape, which can be said to be a cylindrical shape with hemispheres on both ends, an elliptical shape, etc. The collision shape 31 having a capsule shape is also simply called a "capsule."
[0054] The basic shape of the collision shape 31-3 is a box shape, or in other words, a rectangular parallelepiped shape, etc. The collision shape 31 having a box shape is also simply called a "box."
[0055] For example, the shape of the rigid body 30 in relation to a collision is formed by attaching a collision shape 31 having various basic shapes as described above to the rigid body 30. The collision shape 31 of the rigid body 30 may collide with the collision shape 31 of another rigid body 30 or with a particle 20. Note that a rigid body 30 that does not have a collision shape 31 does not collide with another rigid body 30, but the movement and momentum of such a collision shape 31 caused by an external force are also calculated.
[0056] The particles 20 and rigid bodies 30 are described using individual arrays for each physical quantity (each characteristic), and are stored in the storage unit 7, more specifically, the CPU memory 71. For example, quaternions are used for rotation representation, and the particles 20 and rigid bodies 30 are stored as vectors of four elements qa={qa_w, qa_x, qa_y, qa_z} (qa_w is the real part). The collision shape 31 is stored in association with a list of indices (indicators for identifying the rigid bodies 30) indicating which rigid bodies 30 the collision shape 31 is attached to, a list of flags indicating the type of basic shape, a list of vectors indicating the size of the collision shape 31, etc.
[0057] For example, complex biological tissue 2 such as an organ, or an articulated surgical tool 3 such as a surgical instrument, is modeled by combining particles 20 and rigid bodies 30 under given constraints. The constraints are objective functions given to the positions of the particles 20 and the positions and orientations (postures) of the rigid bodies 30, and in simulations, the positions of the particles 20 and the positions and orientations of the rigid bodies 30 are modified to minimize these errors. The constraints are given flexibility by setting a compliance parameter, and the constraints are made very (infinitely) stiff by setting the flexibility to zero (setting rigidity). The constraints may include constraints in XPBD (extended position-based dynamics). Some examples of constraints will be described with reference to FIGS. 5 and 6 .
[0058] 5 and 6 are diagrams showing examples of constraints. The constraints shown in FIG. 5 are inter-particle constraints that limit the relative movement of particles 20. The movement of each particle 20 in a particle group is limited, and the whole is maintained to have a constant structure. As inter-particle constraints, distance constraints, FEM2D (two-dimensional) constraints, FEM3D (three-dimensional) constraints, and volume maintenance constraints are schematically shown in FIGS. 5A to 5D.
[0059] 5A, the distance constraint is a constraint on two particles 20. The distance constraint functions, for example, as a linear spring that maintains the distance between the particles 20 to some extent.
[0060] 5B, the FEM2D constraint is a constraint on three particles 20, each of which is located at a vertex and forms a triangular element, and is defined as the plane strain tensor of the triangular element. For example, the FEM2D constraint is constructed (meshed) along a 3D triangular mesh topology and is used to model thin, soft structures such as membranes.
[0061] 5C, the FEM3D constraint is a constraint on four particles 20, each of which is located at a vertex and forms a tetrahedral element, and is defined as the strain tensor of the tetrahedral element. The FEM3D constraint is constructed, for example, along an input tetrahedral mesh and is used to model biological tissue 2 having a volume such as an organ.
[0062] 5D, the volume maintenance constraint is a constraint on a plurality of particles 20 constituting a closed triangle mesh, each of which is located at a vertex, and maintains the volume of the mesh constant. The volume maintenance constraint is used, for example, to model a balloon-shaped biological tissue 2, such as a gallbladder, which is an elastic membrane filled with an incompressible fluid.
[0063] 6 are inter-rigid body constraints that restrict the relative movement between the rigid bodies 30, and inter-rigid particle constraints that restrict the relative movement between the rigid bodies 30 and the particles 20. For simplicity, the rigid bodies 30 are depicted as cubes, and their collision shapes 31 are not shown.
[0064] Like kinematic constraints, rigid body constraints limit the relative movement between rigid bodies 30. As rigid body constraints, fixed joint constraints, distance joint constraints, hinge or revolute joint constraints, slider or prismatic joint constraints, cylindrical joint constraints, spherical joint constraints, and 6 DOF (Degree Of Freedom) joint constraints are shown schematically in Figures 6A to 6G.
[0065] 6A, a fixed joint constraint maintains the relative pose of rigid bodies 30. A fixed joint constraint is used, for example, to connect two rigid bodies 30 so that they behave as one rigid body.
[0066] As shown in Figure 6B, a distance joint constraint maintains the distance between the attachment points of rigid bodies 30. The two rigid bodies 30 move as if connected by a linear spring. A distance joint constraint is used, for example, to elastically attach two rigid bodies 30.
[0067] As shown in Figure 6C, the rotational joint constraint limits the rigid body 30 to one rotational degree of freedom (1 DOF). The rigid body 30 cannot translate, but can rotate about an axis. The rotational joint is driven according to a target angle and acts as a 1 DOF rotational spring. The rotational joint constraint is used, for example, to model joints in robotic mechanisms or to model the opening and closing of the grip of forceps (e.g., surgical tool 3 in Figures 1 and 3).
[0068] As shown in Figure 6D, the slider joint constraint limits the rigid body 30 to one linear DOF. The rigid body 30 cannot rotate, but can slide along one axis. The slider joint is driven according to a desired linear displacement and acts as a linear spring. The slider joint constraint is used, for example, to simulate a linear actuator in a mechanism.
[0069] As shown in Figure 6E, a cylindrical joint constraint is a combination of a revolute joint constraint and a slider joint constraint. The rigid body 30 can slide and rotate about an axis. The cylindrical joint constraint is used, for example, to model mechanisms that are more complex than a revolute joint or a slider joint alone.
[0070] As shown in Figure 6F, the spherical joint constraint restricts the linear motion of the rigid body 30 to the attachment point. The rigid body 30 can rotate with three degrees of freedom (3 DOF). The spherical joint is driven according to the desired orientation and acts as a 3 DOF rotational spring. The spherical joint constraint is used to model joints such as the shoulder.
[0071] As shown in FIG. 6G, the 6DOF joint constraint is a combination of a linear spring and a ramp spring. The 6DOF joint acts as an elastic 6DOF attachment between rigid bodies 30. The 6DOF joint constraint is used, for example, to attach to a static rigid body 30 that does not have a collision shape 31, to make the rigid body 30 follow a target pose. For example, as described above, the virtual tool (surgical implement 3) can be moved by motion input from the haptic device 8.
[0072] 6H is a special kinematic joint constraint used to connect a rigid body 30 to one or more particles 20. Rigid body-particle joint constraints are used, for example, to maintain the relative positions of a particle 20 and an attached rigid body 30, and to combine a rigid body 30 with a soft body made up of particles 20.
[0073] The various constraints described above are also described using, for example, individual arrays for each physical quantity to be constrained, and are stored in the storage unit 7, more specifically, the CPU memory 71. For example, to represent the distance between particles 20, a list of pairs of indices indicating connected particles 20, a list of static distances, a list of flexibility values, etc. are defined. These may differ depending on the constraint. For example, in a 2D FEM constraint, three particle 20 indices are used for each constraint, and in a 3D FEM constraint, four particle 20 indices are used.
[0074] 1 , the simulation proceeds based on simulation data including information on the modeled particles 20, rigid bodies 30, constraints, etc., as described above. One of the purposes of the simulation is to calculate the changes in the positions of the particles 20, the changes in the positions of the rigid bodies 30, and the changes in the orientation of the rigid bodies 30 caused by collisions between modeled objects, so as to satisfy the given constraints.
[0075] Conventional XPBD uses a sequential iterative solver to process constraints sequentially, updating the position and orientation immediately after each constraint is processed. However, this approach suffers from the problem that processing slows as the number of constraints increases. Other dynamics engines are based on heterogeneous computing architectures, for example. That is, the portion of the solver that is essentially time-series is implemented on the CPU, and the parallelizable algorithm is implemented on the GPU. In contrast, in this embodiment, fully parallel solvers for both particles 20 and rigid bodies 30 are implemented on the GPU 62. This will be described with reference to Figure 7 and subsequent figures.
[0076] A dynamics solver for a soft body such as the biological tissue 2 and a rigid body such as the surgical tool 3 will be described with reference to FIGS.
[0077] 7 is a flowchart showing an example of processing (simulation method) executed in the simulator 1. Simulation data such as that shown on the left side of FIG.
[0078] In step S100, the processing unit 6 copies (transfers) data from the CPU memory 71 to the GPU memory 72. The data here is simulation data including data on the biological tissue 2 and the surgical tool 3 modeled in the CPU memory 71, data on constraints, etc., as described above, and serves as input data to the GPU 62. Examples of input data are as follows, and are shown before step S100 in FIG. 7 : Data on the particles 20, such as data indicating the particles 20 and their positions, masses, and velocities; Data on constraints, such as the content of the constraint, the index of the particle 20 subject to the constraint (an index for identifying the particle 20), and data indicating the rigidity provided by the constraint; Data on the rigid body 30, such as data indicating the rigid body 30 and its attitude, velocity, angular velocity, mass, and inertia; Data on the collision shape 31, such as data indicating the index of the rigid body 30 to which the collision shape 31 is attached, the relative position, relative orientation, type of basic shape, and size of the collision shape 31. Data relating to kinematic joints, such as data indicating the index of the rigid bodies 30 connected by the kinematic joints, the relative attitudes of the rigid bodies 30, the target attitude, type, stiffness, and damping.
[0079] The data may be described as an array (for example, a two-dimensional array (plain array)), which allows the data to be efficiently copied without reformatting, thereby contributing to efficient processing.
[0080] In step S200, a solver implemented on the GPU 62, more specifically, a parallel solver for soft and rigid body dynamics, executes various internal processes required for the simulation. This internal process corresponds to the processing for one time step Δt in the simulation. Data indicating the results obtained by the solver's processing is stored in the GPU memory 72.
[0081] In step S300, data is copied from the GPU memory 72 to the CPU memory 71. This data can also be considered output data of the GPU 62. Examples of output data are as follows, and are shown after step S300 in FIG. 7 : Data related to the particles 20, for example, data indicating the particles 20 and their positions and velocities. Data related to the rigid bodies 30, for example, data indicating the rigid bodies 30 and their attitudes (positions, orientations), velocities, angular velocities, and reaction forces. Data related to kinematic joints, for example, data indicating kinematic joints and their reaction forces. Data related to trigger collision shape pairs (described below), for example, data indicating the trigger collision shape pair and the index of that shape pair (an index for identifying that trigger collision shape pair).
[0082] The above-described processing of steps S100, S200, and S300 is repeatedly executed for each time step Δt to progress the simulation. The processing by the solver on the GPU 62 in step S200 will be further described. The solver on the GPU 62 may also be simply referred to as the solver. Within the scope of no contradiction, the solver may be appropriately interpreted as the GPU 62.
[0083] In FIG. 7, the process of step S200 is further divided into steps S210 to S270.
[0084] In step S210, the solver finds collision pairs, as will be explained with reference to Figures 8 and 9.
[0085] 8 is a flowchart showing a detailed example of step S210. In steps S211 to S214, the solver finds overlapping pairs of particles 20, overlapping pairs of collision shapes 31, and overlapping and potentially overlapping pairs of particles 20 and collision shapes 31. To this end, an axis-aligned bounding box (AABB) is calculated. Note that, unless otherwise specified, the AABB is assumed to be the AABB in global coordinates.
[0086] Specifically, the solver calculates the AABB of the particle 20 and the AABB of the collision shape 31 in parallel. The AABB of the particle 20 is calculated based on data such as the position, velocity, and diameter of the particle 20 (steps S211 and S212). The AABB of the collision shape 31 is calculated based on the position, velocity, and index of the rigid body 30, the relative orientation of the collision shape 31, and the AABB in local coordinates (steps S213 and S214). This will be described with reference to FIG. 9 as well.
[0087] 9 is a diagram showing an example of detecting a collision pair, in which the velocity vectors of two collision shapes 31 are schematically shown by arrows.
[0088] In the example shown in Figure 9A, two collision shapes 31 are predicted to overlap during time step Δt, as shown virtually by the dashed-dotted lines. AABB is extruded a distance (= vΔt) calculated by multiplying the velocity of the collision shape 31 by the time step Δt, where v is the velocity of the collision shape 31. If the movement of the collision shape 31 is a small rotation within the time step Δt, the velocity of the collision shape 31 is the same as the velocity of the rigid body 30 to which the collision shape 31 is attached. As shown in Figure 9B, by extruding AABB using a velocity vector, it is possible to predict the overlap between the collision shapes 31 and further predict collisions that will occur within one time step Δt.
[0089] In one embodiment, there may be special collision shapes 31 that do not respond to collisions. Such collision shapes 31 may optionally be tagged with a "trigger" flag. An overlap may also be predicted for this collision shape 31, and the result may be signaled at the end of time step Δt. This functionality is useful for event implementation of trigger shape tests, as described below.
[0090] Returning to FIG. 8 , in steps S215 to S217, the solver finds possible collision pairs between particles 20, collision pairs between collision shapes 31, and collision pairs between particles 20 and collision shapes 31 for the actual overlaps. In step S215, all possible AABB pairs of particles 20 are tested, resulting in a list of collision pairs between particles 20. In step S216, all possible AABB pairs of collision shapes 31 are tested. In step S217, all possible pairs of particles 20 and collision shapes 31 are tested. These three processes, steps S215 to S217, are executed in parallel. That is, the solver finds collision pairs by testing overlaps between particles 20, overlaps between collision shapes 31, and overlaps between particles 20 and collision shapes 31 in parallel.
[0091] In the above-described steps S216 and S217, collision pairs including the collision shape 31 are discovered. In the subsequent steps S218 and S219, the solver filters and classifies all collision pairs (collision pairs before filtering) discovered in steps S216 and S217 according to the basic shape of the collision shape 31 and further according to the presence or absence of a trigger flag, and lists them.
[0092] Specifically, collision pairs between collision shapes 31 among collision pairs without a trigger flag are filtered, and are classified and listed as collision pairs between spheres, collision pairs between a sphere and a capsule, collision pairs between a sphere and a box, collision pairs between capsules, collision pairs between a capsule and a box, or collision pairs between boxes.
[0093] Also, collision pairs of particles 20 and collision shapes 31 are filtered, and they are classified and listed as collision pairs of particles 20 and spheres, collision pairs of particles 20 and capsules, or collision pairs of particles 20 and boxes.
[0094] Also, collision pairs with trigger flags (collision pairs between collision shapes 31) are filtered and classified and listed as trigger pairs between spheres, trigger pairs between spheres and capsules, trigger pairs between spheres and boxes, trigger pairs between capsules, trigger pairs between capsules and boxes, or trigger pairs between boxes.
[0095] The above-described listing is effective for efficient parallel processing, for example, when collisions occur between the same type of collision shapes 31 (for example, between spheres). The two processes of step S218 and step S219 are also executed in parallel. As a result of the processes of step S216 to step S219, lists of collision pairs between collision shapes 31, trigger pairs between collision shapes 31, and collision pairs between particles 20 and collision shapes 31 are obtained.
[0096] The change in position of the particle 20 included in the collision pair found here, the change in position of the rigid body 30, and the change in orientation of the rigid body 30 are calculated later by the solver.
[0097] 7, the processes of steps S220 to S260 are executed as an inner loop in which one time step Δt in the simulation is divided into n substeps (sub-time step h=Δt / n). The processes of steps S220 to S260 are repeatedly executed a predetermined number of times (n times).
[0098] In steps S220 and S230, the solver predicts the position and orientation and generates contact points. The contact points are constraints between the particles 20 and the rigid body 30, and move them so that they do not interpenetrate. This will also be described with reference to FIG. 10.
[0099] FIG. 10 is a flowchart showing a detailed example of step S220 and step S230.
[0100] In step S221, the solver predicts (calculates) the next position of the particle 20 based on the position, mass, velocity, gravity, etc. of the particle 20. In this calculation, for example, the current velocity of the particle 20, (the influence of) gravity over the sub-time step h, etc. are integrated. Note that the position of the particle 20 before prediction is temporarily stored as a backup, for example, in the GPU memory 72, so that it can be used for subsequent velocity calculation of the particle 20.
[0101] In step S222, the solver predicts the next position and orientation (posture) of the rigid body 30 based on the position, orientation, mass, inertia, velocity, angular velocity, weight, etc. of the rigid body 30. In this calculation, for example, the current angular velocity of the particle 20, gravity over the sub-time step h, etc. are integrated. The next position and orientation of the rigid body 30 before the prediction is backed up so that it can be used for subsequent velocity calculation of the rigid body 30.
[0102] In step S223, the solver updates the orientation of the collision shape 31. The orientation of the collision shape 31 of the rigid body 30 is updated so that the rigid body 30 has the position, orientation, etc. calculated in the previous step S222. The updated global orientation of the collision shape 31 is obtained.
[0103] After the particle 20 moves to the next position and the collision shape 31 moves to the next posture as described above, in steps S231 to S233, the solver checks whether an actual collision between the particle 20 and the collision shape 31 will occur, and generates contact points. Contact points are special constraints that are generated separately from collisions, and are generated in each substep for moving the particle 20 and the rigid body 30. Note that the expected collision pairs have already been discovered in the previous step S210 (FIGS. 7 and 8), and for efficiency, collisions are checked only for the discovered collision pairs.
[0104] In step S231, the solver identifies actual collisions of particles 20. The solver also generates contact points of particles 20 for the identified collisions.
[0105] In steps S232 and S233, the solver checks for collisions involving the collision shape 31 and generates contact points. In step S232, the solver checks for actual collisions of the collision shape 31 and generates contact points for the collision shape 31 for the confirmed collisions. In step S233, the solver checks for actual collisions of the particles 20 and the collision shape 31 and generates contact points for the confirmed collisions of the particles 20 and the collision shape 31. In this example, in both steps S232 and S233, the solver checks for actual collisions in parallel for each basic shape of the collision shape 31 (sphere, capsule, box).
[0106] The changes in the positions of the particles 20, the positions of the rigid bodies 30, and the orientations of the rigid bodies 30 involved in the collisions confirmed (discovered) here are calculated later by a solver.
[0107] The contact force (reaction force) caused by contact between objects is also calculated. For example, in the case of contact between the biological tissue 2 and the surgical tool 3, the forces acting on all particles 20 that collide with the collision shape 31 of the surgical tool 3 are calculated in step S240, which solves the constraints described below, and these forces are accumulated as contact forces within the rigid body 30. The sum of the contact forces of each rigid body 30 becomes the overall contact force of the surgical tool 3.
[0108] Returning to FIG. 7 , in step S240, the solver solves the given constraints. The solver calculates changes in the positions of the particles 20, the positions of the rigid bodies 30, the orientation of the rigid bodies 30, etc. so as to satisfy the constraints. The changes calculated here may be changes caused by collisions between objects (biological tissue 2, surgical tool 3). The solver corrects the positions of the particles 20 and the positions and orientations of the rigid bodies 30 based on the calculated changes in the positions of the particles 20, the positions and orientations of the rigid bodies 30, etc.
[0109] A constraint affects a number of particles 20 or rigid bodies 30 determined depending on the type of constraint. When multiple constraints are given, for calculation efficiency, the constraints may be grouped into constraints of the same type, such as constraints on the positions of the particles 20, constraints on the positions of the rigid bodies 30, and constraints on the orientation of the rigid bodies 30, and solved in parallel. The solver calculates in parallel the changes in the positions of the particles 20, the changes in the positions of the rigid bodies 30, the changes in the orientation of the rigid bodies 30, etc., for each constraint of the same type, so as to satisfy the multiple given constraints.
[0110] Unless otherwise specified, the Jacobi method is used for the constraint solving method, which is advantageous for parallel processing. The following description will be made with reference to FIGS. 11 and 12.
[0111] 11 is a diagram showing an example of a constraint solving method. Constraint C 1 , constraint C 2 and constraint C 3 These are all position constraints. 1 is the position p 1 and position p 2 is a constraint on C 1 (p 1 , p 2 ) Constraint C 2 is the position p 1 , position p 2 and position p 3 is a constraint on C 2 (p 1 , p 2 , p 3 ) Constraint C 3 is the position p 2 , position p 3 , position p 4 and position p 5 is a constraint on C 3 (p 2 , p 3 , p 4 , p 5 )
[0112] The position p calculated to satisfy each constraint 1 ~Position p 5 The change in position Δp 1 ~Position change Δp5 The solver calculates the position change Δp 1 ~Position change Δp 5 The solver accumulates these in separate buffers and tracks the number of constraints that affected each one. Specifically, the solver initializes the buffers to zero and then uses atomic addition to sum the position changes for the same position calculated to satisfy each different constraint. The solver also counts the number of position changes, or in other words, the number of constraints used in calculating that position change (count c). The solver averages the summed positions by the count and adds it to the current position to calculate the latest (modified) position.
[0113] The above is an example of a method for calculating the position of the particle 20 or the rigid body 30, and a similar method can also be used to calculate the orientation (or rotation) of the rigid body 30. Note that if the constraint is a kinematic joint constraint of the rigid body 30 or a contact point of the collision shape 31, the constraint force or contact force obtained by the calculation result may also be stored in a buffer.
[0114] FIG. 12 is a flowchart showing a detailed example of step S240.
[0115] In step S241, the solver solves the constraints of the particle 20 based on the constraints (e.g., FEM-based constraints, contact, etc.) given to the particle 20 and the predicted particle position. As described above with reference to FIG. 11, the sum of the position changes of the particle 20, the count number, is obtained.
[0116] In step S242, the solver solves the constraints of the rigid body 30 (or the particle 20 and the rigid body 30) based on the constraints (e.g., kinematic joint constraints, contact, etc.) given to the rigid body 30 (or the particle 20 and the rigid body 30) and the predicted position and orientation of the rigid body. The total value of the position change and orientation change of the rigid body 30, the count number, and the contact force are obtained. Constraint forces are also obtained.
[0117] In step S243, the solver averages the position changes of the particles 20. As previously described with reference to FIG. 11 , the sum of the position changes of the particles 20 is averaged by the count number. The solver calculates the corrected (next) position of the particle 20 by adding the averaged position change of the particle 20 to the current position of the particle 20.
[0118] In step S244, the solver averages the positional changes and orientation changes of the rigid body 30. The sum of the positional changes and orientation changes of the rigid body 30 is averaged by the count number. The solver calculates the corrected (next) position and orientation of the rigid body 30 by adding the averaged positional changes and orientation changes of the rigid body 30 to the current position and orientation of the rigid body 30.
[0119] Since a correct solution is not necessarily obtained by performing the above averaging only once, the processes of steps S241 to S244 are repeated a fixed number of times (position iterations).
[0120] Returning to Fig. 7, in steps S250 and S260, the solver updates the velocities and solves the updated velocities. For example, the next velocities of the particles 20 and the rigid bodies 30 are calculated in parallel, and the velocities are modified to satisfy the velocity constraints. In the case of the rigid bodies 30, the angular velocity is also calculated and modified. See also Fig. 13 for an explanation.
[0121] FIG. 13 is a flowchart showing a detailed example of step S250 and step S260.
[0122] In step S251, the solver updates the velocity of the particle 20 based on the corrected position of the particle 20 and the previous position of the particle 20. The corrected position of the particle 20 here is the position of the particle 20 after correction in the previous step 240, and corresponds to the latest position of the particle 20. The previous position of the particle 20 is the position of the particle 20 backed up in the previous step S220.
[0123] In step S252, the solver updates the velocity and angular velocity of the rigid body 30 based on the corrected position and orientation of the rigid body 30 and the previous position and orientation of the rigid body 30. The corrected position and orientation of the rigid body 30 here refers to the position and orientation of the rigid body 30 after correction in the previous step S240, and corresponds to the latest position and orientation of the rigid body 30. The previous position and orientation of the rigid body 30 refers to the position and orientation of the rigid body 30 backed up in the previous step S220.
[0124] In step S261, the solver solves the velocity constraints of the particles 20 based on the velocities of the particles 20 and the velocity constraints of the particles 20. The velocity constraints are constraints on the velocity due to, for example, friction (static friction, kinetic friction), contact, damping at kinematic joints, etc. The velocity constraints are used to model, for example, friction and rebound (elasticity) of the collision shape 31 and to add damping to kinematic joints. The constraint solving method is similar to that described above and includes calculations such as summing and counting the velocity changes of the particles 20.
[0125] In step S262, the solver solves the velocity constraints of the rigid body 30 (or of the particles 20 and the rigid body 30) based on the velocity and angular velocity of the rigid body 30 and the velocity constraints of the rigid body 30 (or of the particles 20 and the rigid body 30). The constraint solving method is similar to that described above and includes calculations such as summing and counting the velocity changes and angular velocity changes of the rigid body 30 (or the particles 20 and the rigid body 30).
[0126] In step S263, the solver averages the velocity changes of the particles 20. The solver also calculates the corrected (next) velocity of the particle 20 by adding the averaged velocity change of the particle 20 to the current velocity of the particle 20.
[0127] In step S264, the solver averages the velocity changes and angular velocity changes of the rigid body 30. The solver also calculates the corrected (next) velocity and angular velocity of the rigid body 30 by adding the averaged velocity changes and angular velocity changes of the rigid body 30 to the current velocity and angular velocity of the rigid body 30.
[0128] Since a correct solution is not necessarily obtained by performing the above averaging only once, the processes of steps S251 to S264 are repeated a fixed number of times (rate: iterations).
[0129] Returning to FIG. 7, after the processes of steps S220 to S260 are repeatedly executed n times as one sub-step process, the process proceeds to step S270.
[0130] In step S270, the solver tests the trigger shapes (described below), including a trigger shape overlap test, as described with reference to Figures 14 and 15.
[0131] FIG. 14 is a flowchart showing a detailed example of step S270.
[0132] In step S271, the solver updates the orientation of the collision shape 31 based on the position and orientation of the rigid body 30 and the relative orientation of the collision shape 31. The global orientation of the collision shape 31 is obtained.
[0133] In step S272, the solver checks for overlap of the collision shape 31 based on the global orientation of the collision shape 31 and the trigger pair for each basic shape of the collision shape 31 (for example, finds a collision shape trigger pair, which will be described later). The trigger pair is obtained in step S210, which was previously described, and therefore the description will not be repeated. The overlap check will be described with reference to FIG. 15 as well.
[0134] FIG. 15 is a diagram illustrating overlap checking. The illustrated collision shape 31 is a capsule. A shape that may overlap the collision shape 31 is referred to as a trigger shape 32 and is illustrated. If a trigger shape 32 actually overlaps the collision shape 31, the solver discovers the pair of collision shapes 31 as a collision shape trigger pair. In the example shown in FIG. 15A, the trigger shape 32 of one collision shape 31 does not overlap the other collision shape 31, and no collision shape trigger pair is discovered. In the example shown in FIG. 15B, the trigger shape 32 of one collision shape 31 overlaps the other collision shape 31, and the collision shapes 31 are discovered as a collision shape trigger pair.
[0135] 14, in step S262 illustrated, the solver checks for overlap in parallel for each combination of basic shapes of the collision shape 31. The indexes (indexes for identifying collision trigger pairs) of the confirmed (discovered) collision shape trigger pairs are added to a list.
[0136] 7, after the process of step S200, i.e., the series of processes by the solver on the GPU 62, is completed, data indicating the results is copied back from the GPU memory 72 to the CPU memory 71, as described above. This result data includes the new positions of the particles 20, the new positions, orientations (postures) and contact forces of the rigid bodies 30, the kinematic joint constraints, the trigger shape pairs, etc.
[0137] The position, deformation, etc. of the biological tissue 2 are updated based on the new position, etc. of the particle 20. The position, orientation, deformation, etc. of the surgical tool 3 are updated based on the new position, orientation, etc. of the rigid body 30. Images of these are displayed on the display 9. In addition, the contact force is fed back to the user U via the haptic device 8 (tactile feedback). A series of processes is repeated to progress the simulation. Bidirectional dynamics simulation of the biological tissue 2 and the surgical tool 3 is possible.
[0138] The use of a fully parallel dynamics solver implemented on the GPU 62 improves the real-time performance of simulation and haptic feedback. In addition, the accuracy of calculations of deformation of the biological tissue 2, haptic forces, etc. using FEM-based constraints is improved, allowing for real-time simulation of more complex surgical scenarios, etc.
[0139] Based on the XPBD method, interactive simulation (based on two-way interaction) of the biological tissue 2 and surgical tool 3 is possible (integrated dynamics solver). By having the surgical tool 3 affect the surgical tool 3 or the biological tissue 2, a more realistic interactive surgical simulation is possible.
[0140] Collision detection and the like can also be processed in parallel by a solver implemented on the GPU 62, making it possible to process calculations relating to a very large number (e.g., thousands) of particles 20 and rigid bodies 30, as well as constraints, in real time.
[0141] The contact force obtained by the solver from the interaction between the biological tissue 2 and the surgical tool 3 is consistent with the mass, rigidity, and other properties of the virtual objects, such as the biological tissue 2 and the surgical tool 3. Stable and accurate tactile feedback can be obtained.
[0142] Referring again to FIG. 3, for example, the biological tissue 2 and the surgical tool 3 are modeled as described above, and their collision, deformation, etc. are simulated.
[0143] 1.2 Simulation of Heat Transfer and Smoke Generation Based on the results of the simulation including the contact between the biological tissue 2 and the surgical tool 3 as described above, the CPU 61 simulates heat transfer from the surgical tool 3 to the biological tissue 2 and also simulates smoke generation. This will be described with reference to FIGS. 16 and 17.
[0144] Fig. 16 is a flowchart showing an example of processing (simulation method) executed in the simulator 1. Simulation data such as that shown on the left side of Fig. 16 is prepared and stored in the CPU memory 71. Examples of data are as follows, and are shown before step S400 in Fig. 16: Data related to the heating particles 21, such as data indicating their index, heat (thermal energy), and temperature Data related to the thermal spring, such as data indicating its weight and node index Data indicating the heat transfer coefficient Data related to the particles 20 that model the biological tissue 2, such as data indicating their position Data related to the constraints of the particles 20, such as data indicating the index of the particles 20 Data indicating the position of the tip 3T of the surgical tool 3
[0145] The thermal springs are parts (elements) that connect the heating particles 21 together according to a mesh model of polyhedral elements of the biological tissue 2. The weight of the thermal spring indicates the strength of the connection, and the node index corresponds to the particle index of the connecting heating particle 21. The heat transfer coefficient is, for example, the heat transfer coefficient between the heating particles 21, and can define the heat transfer via the thermal spring.
[0146] In step S400, a solver implemented on the CPU 61, more specifically a heat transfer solver, executes various internal processes required for the simulation. Data indicating the results obtained by the solver's processing is stored in the CPU memory 71. Examples of the data are as follows, and are shown after step S400 in FIG. 16: Data related to the new heating particle 21, such as data indicating its index, heat, and temperature; Data related to the new heat spring, such as data indicating its weight and node index; Data indicating the smoke particle 22
[0147] After the soft-body / rigid-body dynamics simulation described above, the position of the tip 3T of the surgical tool 3 is determined. As described above, the surgical tool 3 is modeled using a rigid body 30 to which a collision shape 31 is attached.
[0148] 16, the process of step S400 is further divided into steps S410 to S470. The following description will be made with reference to FIG.
[0149] FIG. 17 is a diagram showing steps S410 to S470 in a schematic manner.
[0150] In step S410 of FIG. 16, the solver finds the closest particle 20. Specifically, the closest particle 20 is found among the particles 20 located within a given influence radius from the tip T of the surgical tool 3. The influence radius is referred to as influence radius h'. As shown in FIG. 17A, the particle 20 surrounded by a dashed line is found as the closest particle 20. Note that the lines connecting the particles 20 in the figure schematically indicate their constraints.
[0151] In step S420 of Fig. 16, the solver applies heat to the nearest particle 20 that it finds. Fig. 17(B) shows heated particles 21 formed by heating particles 20 in the biological tissue 2. The lines connecting the heated particles 21 schematically represent thermal springs. Heat is transferred between the heated particles 21 via the thermal springs (thermal propagation within the biological tissue 2). Heat is applied to particles 20 that are thermally connected to the surgical tool 3 or thermally connected via other heated particles 21 (interacting with the surgical tool 3), and these particles become heated particles 21.
[0152] In steps S430 and S440 of FIG. 16 , the solver solves the heat transfer model to calculate the temperature change of the heating particles 21 and checks (confirms) the temperatures of the heating particles 21. As shown in FIG. 17C , the temperature of each heating particle 21 is calculated. The closer the heating particles 21 are to the surgical tool 3, the higher their temperatures are, and the temperatures of some heating particles 21 may reach the vaporization temperature (e.g., 100° C.). In this example, the temperatures of the heating particles 21 found in FIG. 17A reach the vaporization temperature.
[0153] 16 , the solver removes the vaporized heating particles 21 and modifies the particles 20 and constraints of the biological tissue 2. As shown by the dashed-dotted lines in (D) of Fig. 17 , the vaporized heating particles 21 and adjacent thermal springs are removed. The solver modifies (updates) the particles 20 and constraints contained therein so as to obtain a model of the remaining biological tissue 2 after removing them.
[0154] 16, the solver modifies the heat transfer model in accordance with the changes to the particles 20 and constraints described above, and emits smoke particles 22 representing the vaporization of the biological tissue 2 by the surgical tool 3. As mentioned above, the smoke particles 22 are rendered using a known graphics engine such as UnrealEngine.
[0155] Referring again to FIG. 3, heat transfer from the surgical tool 3 to the living tissue 2 is simulated, and smoke generation is also simulated, for example, as described above.
[0156] 1.3 Fluid Simulation Bleeding that may occur due to cutting of the biological tissue 2 by the surgical tool 3 is also simulated. This simulation is based on fluid dynamics, more specifically, position-based fluid (PBF) dynamics. This will be explained with reference to Figures 18 to 25.
[0157] FIG. 18 is a flowchart showing an example of a process (simulation method) executed in the simulator 1.
[0158] In step S500, the CPU 61 samples the boundary particles 23 and generates fluid particles 24. This will be described with reference to FIGS.
[0159] FIG. 19 is a diagram schematically illustrating step S500. As shown in FIG. 19A, a biological tissue 2 is modeled. As shown in FIGS. 19B and 19C, boundary particles 23 are sampled from a polyhedral mesh of multiple particles 20 located on the outer surface of the biological tissue 2, more densely than the particles 20. This is to prevent fluid particles 24 from passing between the boundary particles 23. As shown in FIG. 19D, the boundary particles 23 are arranged to cover the surface of the biological tissue 2. The internal structure of the biological tissue 2 is exposed by cutting the biological tissue 2 with the surgical tool 3, and fluid particles 24 modeling blood are generated as shown in FIG. 19E. As shown in FIG. 19F, the movement of the fluid particles 24, i.e., bleeding, is simulated by solving fluid dynamics. For example, the fluid particles 24 move on the boundary particles 23 (on the outer surface of the biological tissue 2).
[0160] The presence or absence of bleeding according to the specific portion of the living tissue 2 cut by the surgical tool 3 may also be simulated. The following description will be made with reference to FIG.
[0161] 20 is a diagram showing an example of the presence or absence of bleeding. The biological tissue 2 is rendered with a texture including blood vessels 25. Faces F1 and F2 are shown as examples of facets of a mesh of polyhedral elements including particles 20. Face F1 includes blood vessels 25, while face F2 does not.
[0162] When a portion of the biological tissue 2 that includes a blood vessel 25 is cut, bleeding occurs. In this example, when the face F1 is cut, fluid particles 24 that model blood are generated. On the other hand, when a portion of the biological tissue 2 that does not include a blood vessel 25 is cut, bleeding does not occur. In this example, when the face F2 is cut, fluid particles 24 are not generated. The realism of the simulation can be improved by causing bleeding only when the blood vessel 25 is cut.
[0163] 18, as a result of the processing in step S500 described above, data relating to the fluid particles 24 and boundary particles 23, such as data indicating their positions, masses, and velocities, is obtained.
[0164] In step S600, the processing unit 6 copies (transfers) the above-mentioned data from the CPU memory 71 to the GPU memory 72. This data becomes input data to the GPU 62.
[0165] In step S700, a solver implemented on the GPU 62, more specifically, a fluid dynamics parallel solver, executes various internal processes required for the simulation. This internal process corresponds to the processing for one time step Δt in the simulation. Data indicating the results obtained by the solver's processing is stored in the GPU memory 72.
[0166] In step S800, data is copied from the GPU memory 72 to the CPU memory 71. This data can also be considered output data of the GPU 62. Examples of the output data include data relating to the fluid particles 24, such as data indicating their position, velocity, and anisotropy (described below), and are shown after step S800 in FIG.
[0167] In FIG. 18, the process of step S700 is further divided into steps S710 to S760.
[0168] In step S710, the solver finds neighboring particles, as described with reference to Figures 21 and 22.
[0169] 21 is a flowchart showing a detailed example of step S710. Fluid particles 24 interact with other particles in their vicinity, more specifically, other fluid particles 24 or boundary particles 23 located within a radius of influence h'. For each fluid particle 24, a list of neighboring boundary particles 23 and fluid particles 24 is calculated. In this example, a spatial hashing algorithm is used to efficiently find neighboring particles.
[0170] In steps S711 and S712, the solver calculates spatial hashes based on the positions and influence radii h' of the fluid particles 24 and the boundary particles 23. This will be described with reference to FIG.
[0171] FIG. 22 illustrates an example of spatial hash calculation. A space containing particles is divided into multiple rectangular grid cells. One grid cell corresponds to a key code in a hash key. The hash key is calculated based on the discretized coordinates of the particle. A particle is inserted into a hash table (associated in the hash table) according to the key of the grid cell containing the particle. Nearby particles can be efficiently searched for by checking particles associated with the same grid cell and adjacent grid cells in the hash table. In the example shown in FIG. 22, for efficiency reasons, the size of the grid cell is set to be equal to the influence radius h'. In a three-dimensional space, it is sufficient to check 27 grid cells. Note that nine of these grid cells are indicated by hatching in FIG. 22.
[0172] Returning to FIG. 21, the fluid particles 24 and their respective spatial hash maps are obtained by the processes in steps S711 and S712.
[0173] In steps S713 and S714, the solver finds fluid particles 24 that are close to each other, and also finds fluid particles 24 and boundary particles 23 that are close to each other. As previously described with reference to Figure 22, efficient discovery is possible by using the results of spatial hash calculations.
[0174] Returning to FIG. 18, the processes of steps S720 to S750 are executed as an inner loop in which one time step Δt in the simulation is divided into n substeps (sub-time step h=Δt / n). The processes of steps S720 to S750 are repeatedly executed a predetermined number of times (n times). Note that the number of iterations here may be a value different from the number of iterations of steps S220 to S260 in FIG. 7 described above.
[0175] In steps S720 and S730, the solver predicts the positions and solves the density constraints, as described with reference to Figures 23 and 24.
[0176] FIG. 23 is a flowchart showing a detailed example of step S720 and step S730.
[0177] In step S721, the solver predicts the position of the fluid particle 24. Specifically, the solver predicts (calculates) the next position of the fluid particle 24 by integrating the position, mass, velocity, gravity (an example of an external force), etc. of the fluid particle 24 over the sub-time step h. Note that the position of the fluid particle 24 before prediction is temporarily stored as a backup, for example, in the GPU memory 72, so that it can be used for subsequent velocity calculation of the fluid particle 24.
[0178] After the fluid particle 24 moves to the next position, in step S731, the solver solves the density constraints described above. Specifically, the solver calculates the current density based on the positions and masses of the fluid particles 24 or the fluid particles 24 and the boundary particles 23 that are close to each other, and the position of the boundary particles 23 predicted in the previous step S721. Furthermore, the solver calculates the displacement of the fluid particle 24 that satisfies the static density. This will be described with reference to FIG. 24 as well.
[0179] 24 is a diagram showing an example of a density constraint, which acts to enforce incompressibility on fluid particles 24.
[0180] As shown in Figure 24A, a given fluid particle 24, in this example the central fluid particle 24, interacts with other fluid particles 24 located within the radius of influence h', indicated by the dashed line. Initially, the fluid particles 24 are uniformly dispersed, assuming their rest density. As external forces (such as gravity) cause the fluid particles 24 to move and their density to change, the density constraint is resolved so that each fluid particle 24 retains the incompressibility of the fluid. For example, as shown in Figure 24B, if neighboring fluid particles 24 are scattered and their density becomes smaller than the rest density, they are pulled back to reach the rest density. Conversely, as shown in Figure 24C, if neighboring fluid particles 24 are attracted to each other and their density becomes larger than the rest density, they are pushed apart to reach the rest density.
[0181] Regarding the relationship with the boundary particle 23, as shown in Fig. 24(D), the fluid particle 24 behaves at its rest density so as to maintain a given rest distance from the boundary particle 23. For example, as shown in Fig. 24(E), when the fluid particle 24 moves away from the boundary particle 23 within the influence radius h', the fluid particle 24 is pulled back toward the boundary particle 23 so as to maintain the rest density. Conversely, as shown in Fig. 24(F), when the fluid particle 24 is pressed toward the boundary particle 23, the fluid particle 24 is moved away from the boundary particle 23 so as to maintain the rest density.
[0182] 23 , the displacement of the fluid particle 24 is obtained by the processing of step S731. In step S732, the solver moves the fluid particle 24 by the calculated displacement of the fluid particle 24. This movement modifies the position of the fluid particle 24. The modified position of the fluid particle 24 is also referred to as the modified position of the fluid particle 24.
[0183] Since the solution does not necessarily converge by performing the processes of steps S731 and S732 only once, the processes of steps S731 and S732 are repeated a fixed number of times based on the corrected position of the fluid particle 24 (position iterations).
[0184] Returning to Figure 18, in steps S740 and S750, the solver updates the velocities and solves the updated velocities. See also Figure 25 for an explanation.
[0185] FIG. 25 is a flowchart showing a detailed example of step S740 and step S750.
[0186] In step S741, the solver updates the velocity of the fluid particle 24 based on the corrected position of the fluid particle 24 and the previous position of the fluid particle 24. The corrected position of the fluid particle 24 here is the position of the fluid particle 24 after correction in the previous step 730, and corresponds to the latest position of the fluid particle 24. The previous position of the fluid particle 24 is the position of the fluid particle 24 backed up in the previous step S720.
[0187] In step S751, the solver calculates the velocity change of the fluid particle 24 based on the updated velocity of the fluid particle 24 and the current velocity constraint. The velocity constraint may be a velocity constraint due to a phenomenon, factor, etc. that cannot be modeled by the density constraint alone. As mentioned above, examples of velocity constraints include constraints due to viscosity, vorticity, adhesion, adsorption, friction at particle boundaries, cohesion, surface tension, etc. For example, various velocity constraints are solved in parallel (simultaneously), and the corresponding velocity changes (vectors) are calculated simultaneously.
[0188] In step S752, the solver adds the calculated velocity changes of the boundary particles 23. For example, the velocity changes corresponding to the various velocity constraints simultaneously calculated in the previous step S751 are added to the current velocity. This modifies the velocity of the fluid particle 24. The modified velocity of the fluid particle 24 is also referred to as the modified velocity of the fluid particle 24.
[0189] Since a solution will not necessarily converge after just one execution of steps S751 and S752, steps S751 and S752 are repeated a fixed number of times (speed iterations) based on the corrected velocity of the fluid particles 24.
[0190] 18 , in step S760, the solver calculates the anisotropy of the fluid particles 24. The anisotropy of the fluid particles 24 indicates the magnitude, size, etc., of the local distribution of the fluid particles 24. For example, the solver calculates the anisotropy of at least some of the multiple fluid particles 24 so that they have an elliptical shape. This will also be described with reference to FIG. 26 .
[0191] FIG. 26 is a flowchart showing a detailed example of step S760.
[0192] In step S751, the solver smoothes the fluid particle 24 based on the latest position of the fluid particle 24, the influence radius h′, etc. For example, a smoothing filter is used to reduce the non-uniformity of the fluid particle 24. The smoothed position of the fluid particle 24 is obtained.
[0193] In step S752, the solver performs Principal Component Analysis (PCA) on each fluid particle 24 and its neighbors to find their geometric centroids and principal directions. This process is performed in parallel for each fluid particle 24.
[0194] In the example shown in Figure 26, fluid particles 24 located inside the spherical shape have no main direction, whereas fluid particles 24 located on the surface are flattened to have an elliptical shape oriented towards the fluid free surface and have a main direction.
[0195] Note that the anisotropy of the fluid particles 24 is calculated for rendering purposes. Blood modeled with a plurality of fluid particles 24, including fluid particles 24 having an ellipsoidal shape as described above, is rendered to provide a more realistic graphical effect.
[0196] 3 again, bleeding is simulated, for example, as described above. The various simulations described so far are realized by processing by the processing unit 6, more specifically, cooperation between the CPU 61 and the GPU 62. The results of the simulation are fed back to the haptic device 8 or displayed as an image on the display 9. In particular, bleeding and smoke that may occur due to treatment such as cutting of the biological tissue 2 with the surgical tool 3 are also simulated, and the rendered blood and smoke are presented to the user U via the display 9. These realistic graphical effects are provided.
[0197] 2. Modifications The disclosed technology is not limited to the above-described embodiment. Some modifications will be described.
[0198] The simulation described so far may be applied to a surgical robot simulation. In the surgical robot simulation, the user U operates a robot arm that supports a surgical tool via the haptic device 8. Since the contact force is fed back to the user U, it is also possible to evaluate whether the robot arm is being operated with an appropriate operating force. This allows for more effective training than when there is no feedback of the contact force.
[0199] In one embodiment, the simulation may be incorporated into a master-slave system, as described with reference to FIG.
[0200] 27 is a diagram showing a modified example. The simulator system 100 includes a leader device 40 and a follower device 50.
[0201] The leader device 40 includes the haptic device 8 described above, a display 9, a processing unit 6, and a storage unit 7. The leader device 40 is configured to be able to communicate with the follower device 50. A command value for controlling the haptic device 8 is sent from the processing unit 6 to the haptic device 8.
[0202] The follower device 50 includes a processing unit 51, a camera 52, and a robotic arm 53. The processing unit 51 observes the slave environment, including the patient's condition, based on the image captured by the camera 52. The slave environment may also include non-visual information such as the patient's blood pressure and heart rate. The processing unit 51 also controls the robotic arm 53 in response to the operation of the haptic device 8 on the reader device 40. The control is performed by sending a command value from the processing unit 51 to the robotic arm 53.
[0203] The leader device 40 and the follower device 50 function as a master device and a slave device capable of bilateral control. The haptic device 8 is operated by the user U to control the robot arm 53 of the follower device 50, which supports a surgical tool, in a master-slave manner. The image from the camera 52 of the follower device 50 is transmitted from the follower device 50 to the leader device 40 and displayed on the display 9 of the leader device 40. Other information about the slave environment is also transmitted from the follower device 50 to the leader device 40 and presented. The motion (information on position, posture, force, etc.) from the haptic device 8 of the leader device 40 is transmitted from the leader device 40 to the follower device 50 and reflected in the control of the robot arm 53. The motion from the robot arm 53 is transmitted from the follower device 50 to the leader device 40 and fed back to the user via the haptic device 8 of the leader device 40.
[0204] The leader device 40 calculates the collision, deformation (position change of the particle 20, position / orientation change of the rigid body 30, etc.), contact force, etc. between the biological tissue 2 and the surgical tool 3 as described above. In the simulator system 100 shown in Fig. 16, the calculation results of the leader device 40 are reflected in the control of the robot arm 53. For example, when a motion from the haptic device 8 of the leader device 40 is transmitted to the follower device 50, the control of the robot arm 53 is limited to ensure safety.
[0205] Specifically, the processing unit 6 of the leader device 40 executes a safety determination process based on the calculation results of the simulation model. For example, the processing unit 6 determines whether or not operation of the robot arm 53 of the follower device 50 via the leader device 40 is dangerous. If it is determined to be dangerous, the processing unit 6 intervenes in the operation of the follower device 50 by the leader device 40. For example, the processing unit 6 modifies the motion transmitted from the leader device 40 to the follower device 50 so as to limit the amount of movement or the operating force of the robot arm 53. The modified motion is transmitted from the leader device 40 to the follower device 50. In addition, the determination results and control results by the processing unit 6 are notified to the leader device 40 and the follower device 50.
[0206] In one embodiment, the functions of the simulator 1 may be distributed. This will be described with reference to FIG.
[0207] 28 is a diagram showing a modified example. The simulator 1 includes an edge device 1e and a cloud device 1c. The edge device 1e and the cloud device 1c are a first information processing device and a second information processing device that cooperate with each other to configure the simulator 1. The edge device 1e and the cloud device 1c are connected via a network N and transmit and receive (share) data related to the simulation with each other.
[0208] In one embodiment, the network N may be an all-optical communication network. In this case, the connection between the cloud device 1c and the edge device 1e via the network N means that all communication between at least the endpoint routers is established using optical communication. Note that communication after the endpoint router may be electrical communication after photoelectric conversion. By using a broadband all-optical communication network, the cloud device 1c and the edge device 1e are connected with low latency. This is particularly useful for real-time simulations, etc.
[0209] The edge device 1e is placed in an edge area including a haptic device 8 and a display 9. The cloud device 1c is placed in a cloud area located on the opposite side of the network N from the edge area. The location of the cloud area may be distant from the edge area or close to the edge area (for example, in the same building or the same room).
[0210] The cloud device 1c is located in a cloud area CF. The cloud area CF is an area located on the opposite side of the network N from the edge area EF, and is, for example, an area distant from the edge area EF.
[0211] Any of the functions of the simulator 1 described so far may be distributed and allocated to the edge device 1e and the cloud device 1c. As an example, the CPU 61 and the CPU memory 71 may be allocated to the edge device 1e, and the GPU 62 and the GPU memory 72 may be allocated to the cloud device 1c. For example, the portion of the program 70 corresponding to the edge device 1e may be allocated to the edge device 1e, and the portion corresponding to the cloud device 1c may be allocated to the cloud device 1c.
[0212] According to the above-described simulator 1, for example, a high-performance information processing device located in a cloud area can be used as the cloud device 1c. This increases the possibility that the simulator 1 can be used as a haptic surgery simulator capable of large-scale calculations. For example, it can be used for training and preoperative planning for surgeries that handle biological tissues (soft organs, etc.), such as laparoscopic surgery, thoracic surgery, and neurosurgery. Training while feeling the deformation and realistic reaction forces of biological tissues allows for more accurate preoperative experiences, which is expected to improve skills and surgical outcomes.
[0213] The simulation techniques described above may be applied to simulations of endoscopic treatment using a rigid endoscope or a flexible endoscope.
[0214] 29 is a diagram showing an example of the hardware configuration of the simulator 1. The simulator 1 is realized by, for example, a computer 1000 shown in FIG.
[0215] The computer 1000 includes a CPU / GPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.
[0216] The CPU / GPU 1100 operates based on programs stored in the ROM 1300 or the HDD 1400 and controls each component. For example, the CPU / GPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs. An example of a program is the program 70 described above with reference to FIG. 2.
[0217] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU / GPU 1100 when the computer 1000 starts up, and programs that depend on the hardware of the computer 1000 .
[0218] The HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU / GPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records a program for the information processing method according to the present disclosure, which is an example of program data 1450.
[0219] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU / GPU 1100 receives data from other devices and transmits data generated by the CPU / GPU 1100 to other devices via the communication interface 1500.
[0220] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU / GPU 1100 receives data from input devices such as a keyboard or a mouse via the input / output interface 1600. The CPU / GPU 1100 also transmits data to output devices such as a display, a speaker, or a printer via the input / output interface 1600. For example, the CPU / GPU 1100 receives motion data from the haptic device 8 or transmits simulated contact forces to the haptic device 8 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined computer-readable recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, and semiconductor memories.
[0221] When the computer 1000 functions as the simulator 1 described above, the CPU / GPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the processing unit 6. The program may be stored in the HDD 1400. The CPU / GPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU / GPU 1100 may obtain the program from another device via an external network 1550.
[0222] Each of the above components may be configured using general-purpose materials or may be configured using hardware specialized for the function of each component. Such configurations may be changed as appropriate depending on the technical level at the time of implementation.
[0223] 4. Summary The technology described above can be specified, for example, as follows. One of the disclosed technologies is a simulator 1. As described with reference to FIGS. 1 to 3 and 16 to 26 , the simulator 1 includes a processing unit 6 including at least one processor (e.g., a CPU 61, a GPU 62). The processing unit 6 simulates at least one of bleeding and smoking from the biological tissue 2 based on heat generated in the biological tissue 2 by an energy device (surgical tool 3) and heat propagation within the biological tissue 2. For example, the energy device is a heating surgical tool such as an electric scalpel or an ultrasonic scalpel, and heat generation in the biological tissue 2 occurs as a result of treatment of the biological tissue 2 by the energy device. Such a simulator 1 enables simulation of surgery, including bleeding and smoking.
[0224] 3, 5, 16, 17, etc., the biological tissue 2 includes a plurality of particles 20, and the processing unit 6 may remove particles 20 (heated particles 21) that have reached a vaporization temperature due to heat generation from the plurality of particles 20 from the biological tissue 2. In this manner, for example, bleeding and smoking caused by heat transfer from the surgical tool 3 to the biological tissue 2 can be simulated.
[0225] 16 and 17 , the processing unit 6 may use the smoke particles 22 to model smoke generated from a portion of the biological tissue 2 from which particles 20 that have reached the vaporization temperature (heated particles 21 in an overheated state) have been removed, in order to simulate smoking. For example, smoking can be simulated in this manner.
[0226] 18 and 19 , in order to simulate bleeding, the processing unit 6 may use a plurality of fluid particles 24 to model blood flowing out from a portion of the biological tissue 2 from which the particles 20 (heating particles 21) that have reached the vaporization temperature have been removed. In this way, for example, bleeding can be simulated.
[0227] 18 , 23 , 24 , etc., the processing unit 6 may calculate the positions of the plurality of fluid particles 24 so as to satisfy the density constraint of the plurality of fluid particles 24. In this case, the density constraint may include a constraint that the plurality of fluid particles 24 maintain a static density. In this way, for example, the flow of blood can be simulated.
[0228] 23 and 24 , the processing unit 6 may sample the plurality of boundary particles 23 located on the surface of the biological tissue 2 more densely than the plurality of fluid particles 24 contained in the biological tissue 2, and calculate the positions of the plurality of fluid particles 24 so as to satisfy density constraints on the plurality of fluid particles 24 and the plurality of boundary particles 23. In this case, the density constraint may include a constraint that a fluid particle 24 that is close to a boundary particle 23 among the plurality of fluid particles 24 maintains a given stationary distance from the boundary particle 23. For example, in this manner, blood flowing on the surface of the biological tissue 2 can be simulated.
[0229] 18, 25, etc., the processing unit 6 may calculate the change in velocity of the plurality of fluid particles 24 so as to satisfy the velocity constraints of the plurality of fluid particles 24. The velocity constraints may include at least one of a velocity constraint due to viscosity, a velocity constraint due to vorticity, a velocity constraint due to adhesion, a velocity constraint due to adsorption, a velocity constraint due to friction, a velocity constraint due to cohesion, and a velocity constraint due to surface tension. In this manner, for example, the flow of blood can be simulated.
[0230] 18, 26, etc., the processing unit 6 may calculate the anisotropy of the plurality of fluid particles 24 so that at least some of the fluid particles 24 have an elliptical shape, and may render blood modeled by the plurality of fluid particles 24 including the fluid particles 24 having an elliptical shape, thereby providing a more realistic graphical effect.
[0231] 18 to 20 , the processing unit 6 may render the biological tissue 2 with a texture including blood vessels, and bleeding may occur when a portion of the biological tissue 2 including the blood vessels 25 (for example, the surface F1) is cut, thereby improving the realism of the simulation.
[0232] The simulation method described with reference to Figures 1 to 3 and 16 to 26 is also one of the disclosed techniques. The simulation method includes simulating at least one of bleeding and smoking from the biological tissue 2 based on heat generated in the biological tissue 2 by the energy device (surgical tool 3) and heat propagation within the biological tissue 2. Such a simulation method also makes it possible to simulate surgery that includes bleeding and smoking.
[0233] The program 70 (simulation program) described with reference to Figures 1 to 3, 16 to 26, and 29 is also one of the disclosed technologies. The program 70 causes the computer 1000 to execute a process for simulating at least one of bleeding and smoking from biological tissue based on heat generated by an energy device in the biological tissue and heat propagation within the biological tissue. Such program 70 also makes it possible to simulate surgery that includes bleeding and smoking.
[0234] The simulator system 100 described with reference to Figures 1 to 3 and 16 to 26 is also one of the disclosed technologies. The simulator system 100 includes an input device (haptic device 8) that accepts operation of an energy device (surgical tool 3), and a display device (display 9) that displays at least one of bleeding and smoking from the biological tissue 2, simulated based on heat generated in the biological tissue 2 by the energy device and heat propagation within the biological tissue 2. Such a simulator system 100 also makes it possible to simulate surgery that includes bleeding and smoking.
[0235] The effects described in this disclosure are merely examples and are not limited to the disclosed contents. Other effects may also be obtained.
[0236] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.
[0237] The present technology may also be configured as follows: (1) A simulator including a processing unit including at least one processor, wherein the processing unit simulates at least one of bleeding and smoke generation from biological tissue based on heat generation in the biological tissue caused by an energy device and heat propagation within the biological tissue. (2) The simulator described in (1), wherein the energy device is a heating surgical tool, and the heat generation in the biological tissue is caused by treatment of the biological tissue with the heating surgical tool. (3) The simulator described in (1) or (2), wherein the biological tissue includes a plurality of particles, and the processing unit removes from the biological tissue particles that have reached a vaporization temperature due to the heat generation. (4) The simulator described in (3), wherein the processing unit uses smoke particles to model smoke generated from a portion of the biological tissue from which the particles that have reached the vaporization temperature have been removed, in order to simulate the smoke generation. (5) The simulator according to (3) or (4), wherein the processing unit uses a plurality of fluid particles to model blood flowing out from a portion of the biological tissue from which particles that have reached the vaporization temperature have been removed, in order to simulate the bleeding. (6) The simulator according to (5), wherein the processing unit calculates positions of the plurality of fluid particles to satisfy density constraints of the plurality of fluid particles. (7) The simulator according to (6), wherein the density constraints include a constraint that the plurality of fluid particles maintain a stationary density. (8) The simulator according to (6) or (7), wherein the processing unit samples a plurality of boundary particles located on the surface of the biological tissue more densely than a plurality of fluid particles contained in the biological tissue, and calculates positions of the plurality of fluid particles to satisfy the density constraints of the plurality of fluid particles and the plurality of boundary particles. (9) The simulator according to (8), wherein the density constraints include a constraint that fluid particles of the plurality of fluid particles that are close to the boundary particle maintain a given stationary distance from the boundary particle. (10) The simulator according to any one of (6) to (9), wherein the processing unit calculates changes in velocity of the plurality of fluid particles so as to satisfy velocity constraints of the plurality of fluid particles.(11) The simulator according to (10), wherein the velocity constraint includes at least one of a velocity constraint due to viscosity, a velocity constraint due to vorticity, a velocity constraint due to adhesion, a velocity constraint due to absorption, a velocity constraint due to friction, a velocity constraint due to cohesion, and a velocity constraint due to surface tension. (12) The simulator according to any of (5) to (11), wherein the processing unit calculates the anisotropy of at least some of the plurality of fluid particles so that the fluid particles have an ellipsoidal shape, and renders blood modeled by the plurality of fluid particles including the fluid particles having the ellipsoidal shape. (13) The simulator according to any of (1) to (12), wherein the processing unit renders the biological tissue with a texture including blood vessels, and the bleeding occurs when a portion of the biological tissue including the blood vessels is cut. (14) A simulation method including simulating at least one of bleeding and smoking from biological tissue based on heat generated in the biological tissue by an energy device and heat propagation within the biological tissue. (15) A simulation program that causes a computer to execute a process of simulating at least one of bleeding and smoking from the biological tissue based on heat generated in the biological tissue by an energy device and heat propagation within the biological tissue. (16) A simulator system comprising: an input device that accepts operation of an energy device, and a display device that displays at least one of bleeding and smoking from the biological tissue simulated based on heat generated in the biological tissue by the energy device and heat propagation within the biological tissue.
[0238] 100 Simulator system 1 Simulator 1c Cloud device 1e Edge device 2 Biological tissue 20 Particle 21 Heating particle 22 Smoke particle 23 Boundary particle 24 Fluid particle 25 Blood vessel 3 Surgical tool (energy device) 30 Rigid body 31 Collision shape 3T Tip 40 Leader device 50 Follower device 51 Processing unit 52 Camera 53 Robot arm 6 Processing unit 61 CPU 62 GPU 7 Storage unit 70 Program 71 CPU memory 72 GPU memory 8 Haptic device 9 Display CF Cloud area EF Edge area F1 Surface F2 Surface N Network U User 1000 Computer 1050 Bus 1100 CPU / GPU 1200 RAM 1300 ROM 1400 HDD 1450 Program data 1500 Communication interface 1600 Input / output interface 1650 Input / output device
Claims
1. A simulator comprising a processing unit including at least one processor, the processing unit simulating at least one of bleeding and smoking from biological tissue based on heat generation in the biological tissue caused by an energy device and heat propagation within the biological tissue.
2. The simulator according to claim 1, wherein the energy device is a heating surgical tool, and the heat generation in the biological tissue is caused by treatment of the biological tissue with the heating surgical tool.
3. The simulator according to claim 1, wherein the biological tissue includes a plurality of particles, and the processing unit removes from the biological tissue those particles among the plurality of particles that have reached a vaporization temperature due to the heat generation.
4. The simulator of claim 3, wherein the processing unit uses smoke particles to model smoke arising from a portion of the biological tissue from which particles that have reached the vaporization temperature have been removed in order to simulate the smoke generation.
5. The simulator according to claim 3, wherein the processing unit uses a plurality of fluid particles to model blood flowing out of a portion of the biological tissue from which particles that have reached the vaporization temperature have been removed, in order to simulate the bleeding.
6. The simulator according to claim 5, wherein the processing unit calculates positions of the plurality of fluid particles so as to satisfy density constraints for the plurality of fluid particles.
7. The simulator of claim 6, wherein the density constraints include a constraint that the plurality of fluid particles maintain a rest density.
8. The simulator according to claim 6, wherein the processing unit samples a plurality of boundary particles located on a surface of the biological tissue more densely than a plurality of fluid particles contained in the biological tissue, and calculates positions of the plurality of fluid particles so as to satisfy density constraints of the plurality of fluid particles and the plurality of boundary particles.
9. The simulator of claim 8, wherein the density constraints include a constraint that fluid particles of the plurality of fluid particles proximate to the boundary particle maintain a given rest distance from the boundary particle.
10. The simulator according to claim 6, wherein the processing unit calculates changes in velocity of the plurality of fluid particles so as to satisfy velocity constraints of the plurality of fluid particles.
11. The simulator of claim 10, wherein the velocity constraints include at least one of: a velocity constraint due to viscosity, a velocity constraint due to vorticity, a velocity constraint due to adhesion, a velocity constraint due to absorption, a velocity constraint due to friction, a velocity constraint due to cohesion, and a velocity constraint due to surface tension.
12. The simulator of claim 5, wherein the processing unit calculates anisotropy of at least some of the plurality of fluid particles such that they have an ellipsoidal shape, and renders blood modeled by the plurality of fluid particles including fluid particles having the ellipsoidal shape.
13. The simulator according to claim 1, wherein the processing unit renders the biological tissue with a texture including blood vessels, and the bleeding occurs when a portion of the biological tissue including the blood vessels is cut.
14. A simulation method including simulating at least one of bleeding and smoking from biological tissue based on heat generation in the biological tissue caused by an energy device and heat propagation within the biological tissue.
15. A simulation program that causes a computer to execute a process of simulating at least one of bleeding and smoking from biological tissue based on heat generation in the biological tissue caused by an energy device and heat propagation within the biological tissue.
16. A simulator system comprising: an input device that accepts operations of an energy device; and a display device that displays at least one of bleeding and smoking from the biological tissue simulated based on heat generation in the biological tissue caused by the energy device and heat propagation within the biological tissue.
Citation Information
Patent Citations
Method of simulation for cutting three-dimensional object
JP2003141566A
Real-time soft tissue burning simulation method based on position dynamics and biological conduction equation
CN107688720A
Mixed particle blood model bleeding simulation method based on SPH
CN111047707A
Fluid simulation method, device and equipment and storage medium
CN115688533A
Method for simulating fluid
KR1020130078102A
