Information processing device, information processing method, program, and simulation system

Abstract

The present technology relates to an information processing device, an information processing method, a program, and a simulation system that enable a mesh to be subdivided while maintaining consistency of physical characteristics. An information processing device according to one aspect of the present technology performs a simulation after an increase or decrease in the number of vertices of a polyhedron constituting an object obtained by meshing a 3D model of a flexible object, the simulation being performed on the basis of the mass and momentum of the vertices before the increase or decrease. For example, when a polyhedron is divided and the number of vertices increases or decreases, the mass of the vertices after the increase or decrease is calculated so as to maintain the mass of the vertices before the increase or decrease, and the momentum of the vertices after the increase or decrease is calculated so as to maintain the momentum of the vertices before the increase or decrease. The present technology is applicable to simulations of surgical environments.

Description

Information processing device, information processing method, program, and simulation system

[0001] The present technology relates to an information processing device, an information processing method, a program, and a simulation system, and more particularly to an information processing device, an information processing method, a program, and a simulation system that enable mesh subdivision while maintaining consistency of physical characteristics.

[0002] There is a surgical simulator that uses a physics-based simulation method called Extended Position Based Dynamics (XPBD).Surgical simulators are used for surgical training, etc. XPBD makes it possible to model the interaction between soft objects such as organs and rigid objects such as surgical instruments, and to express the deformation of the soft objects in real time and present the associated tactile sensations to the user.

[0003] International Publication No. 2023 / 171413 Japanese Patent Application Laid-Open No. 2003-141566

[0004] For realistic training, more precise deformation and visual representation of soft-bodied objects are required, but achieving this comes with a trade-off: increased computational time.

[0005] The present technology has been developed in light of these circumstances, and makes it possible to subdivide meshes while maintaining consistency of physical properties.

[0006] An information processing device according to one aspect of the present technology includes a calculation unit that performs a simulation after increasing or decreasing vertices of a polyhedron that constitutes an object obtained by meshing a 3D model of a flexible object, based on the mass and momentum of the vertices before the increase or decrease.

[0007] In one aspect of the present technology, a simulation after increasing or decreasing vertices of a polyhedron constituting an object obtained by meshing a 3D model of a flexible object is performed based on the mass and momentum of the vertices before the increase or decrease.

[0008] 1 is a diagram illustrating an example of a simulation in a surgical simulator according to an embodiment of the present technology. FIG. 1 is a block diagram illustrating a configuration example of a surgical simulator. FIG. 2 is a diagram illustrating an example of a mesh of a soft body object. FIG. 3 is a diagram illustrating an example of mesh subdivision using a subdivision algorithm. FIG. 4 is a diagram illustrating an example of a division pattern of a tetrahedron. FIG. 5 is a diagram illustrating an example of division of a tetrahedron. FIG. 6 is a diagram illustrating another example of division of a tetrahedron. FIG. 7 is a flowchart illustrating division processing. FIG. 8 is a diagram illustrating a flow of division of a tetrahedron. FIG. 9 is a diagram illustrating division results of a tetrahedron. FIG. 10 is a flowchart illustrating integration processing. FIG. 11 is a diagram illustrating a flow of integration of tetrahedrons. FIG. 12 is a diagram illustrating an example of two-stage division. FIG. 13 is a diagram illustrating an example of two-stage division. FIG. 14 is a diagram illustrating a state of a tetrahedron. FIG. 15 is a diagram illustrating an example of a division pattern. FIG. 16 is a diagram illustrating the state of each vertex before and after division. FIG. 17 is a diagram illustrating the state of each vertex before and after division. FIG. 18 is a diagram illustrating an example of a cutting plane. FIG. 19 is a diagram illustrating the flow of division when only the mesh of the cutting plane is subdivided. FIG. 20 is a diagram illustrating the flow of division of a tetrahedron through which the cutting plane passes. FIG. 21 is a diagram illustrating the flow of division when cutting using an electric scalpel. FIG. 22 is a block diagram illustrating a configuration example of a computer.

[0009] Hereinafter, embodiments of the present technology will be described in the following order: 1. Overview of the present technology 2. Configuration of surgical simulator 3. Segmentation algorithm 4. Modified example

[0010] <<Outline of the Present Technology>> <Subdivision Algorithm> A surgical simulator according to an embodiment of the present technology uses a subdivision algorithm that dynamically subdivides the mesh of a soft-body object representing a flexible object such as an organ. The mesh subdivision using the subdivision algorithm is performed by subdividing only a portion of the mesh that is of interest. Dynamically subdividing only a portion of the mesh enables high-speed, high-precision simulation while suppressing an increase in the amount of calculation compared to when high-resolution meshes are prepared in advance.

[0011] The subdivided meshes are integrated and returned to their original state when attention is released. By returning the subdivided parts to their original state, it is possible to suppress an increase in the amount of calculation.

[0012] The mesh refinement and integration are performed based on the laws of conservation of mass and momentum, so that physical properties are maintained before and after processing. Because physical properties such as mass and momentum are maintained, it is possible to simulate behavior that is consistent before and after processing.

[0013] Furthermore, in a surgical simulator according to an embodiment of the present technology, a simulation is performed using Extended Position Based Dynamics (XPBD). The deformed state of a soft-body object representing a flexible body such as an organ is calculated in real time using XPBD. In addition, contact forces with surgical instruments such as scalpels and forceps are calculated and fed back to the user via a haptic device. Furthermore, the use of ray tracing technology enables highly realistic rendering. A realistic simulation is realized by presenting realistic visual information and tactile sensations.

[0014] Calculating the deformed state of a soft body object, generating haptic feedback, and rendering the soft body object based on XPBD is described, for example, in WO 2023 / 171413.

[0015] <<Configuration of Surgical Simulator>> FIG. 1 is a diagram illustrating an example of a simulation in a surgical simulator according to an embodiment of the present technology.

[0016] The surgical simulator 1 is configured to include an information processing device 11, a haptic device 12, and a monitor 13. The surgical simulator 1 is a simulation system used, for example, for surgical training. A doctor or other person who performs surgical training is a user of the surgical simulator 1. The lower left corner of FIG. 1 shows the hand of a user operating the haptic device 12.

[0017] The information processing device 11 is a simulator device that simulates a surgical environment. In Fig. 1, a virtual surgical environment is schematically shown inside the information processing device 11. In the surgical environment, a soft object O1, a rigid object O2, a robot arm R, and a camera C are prepared as virtual elements related to surgery. The behaviors and states of these virtual elements are calculated to simulate the surgical environment.

[0018] The soft-body object O1 is an object that represents a flexible object such as an organ. The soft-body object O1 is prepared as an object that is a mesh of a 3D model of the flexible object. The rigid-body object O2 is an object that represents a surgical instrument such as a scalpel or forceps. Various operations on the soft-body object O1, such as grasping, cutting, and suturing with a needle, are virtually performed using the rigid-body object O2.

[0019] The robot arm R is a device that supports the camera C. The camera C is a surgical camera that captures images of the surgical field including the soft object O1 and the rigid object O2. The imaging range of the camera C is controlled by the robot arm R. In the information processing device 11, a moving image of the imaging range of the camera C is generated as a simulation image and supplied to the monitor 13. As appropriate, a still image is generated as a simulation image and supplied to the monitor 13.

[0020] The haptic device 12 is an input device used to operate the rigid object O2. The haptic device 12 is also used to control the robot arm R and adjust the position and angle of the camera C. The haptic device 12 outputs a signal representing the content of the user's operation to the information processing device 11. The haptic device 12 also generates vibrations, impacts, etc. based on the information supplied from the information processing device 11, and presents a tactile sensation to the user.

[0021] The monitor 13 is a display device used to present various visual information such as a simulation image supplied from the information processing device 11. A surgical image including at least a part of the soft object O1 and the rigid object O2 is displayed on the monitor 13. The user operates the haptic device 12 while watching the simulation image displayed on the monitor 13, and performs surgical training.

[0022] FIG. 2 is a block diagram showing an example of the configuration of the surgery simulator 1.

[0023] The information processing device 11 is configured to include an information processing unit 21 and a storage unit 22. The information processing unit 21 is configured with processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). By executing a program stored in the storage unit 22, a calculation unit 31, an image generation unit 32, and a feedback generation unit 33 are realized in the information processing unit 21. The storage unit 22 stores various data such as the program executed by the information processing unit 21 and 3D data of each element in the surgical environment.

[0024] The calculation unit 31 performs a simulation of the surgical environment by calculation based on the XPBD. As part of the simulation of the surgical environment, for example, the movement of the rigid object O2 is calculated based on motion information supplied from the haptic device 12. The motion information is information indicating the position, posture, force, etc. of the rigid object O2 as the content of the user's operation.

[0025] Furthermore, the state of the soft-body object O1 after deformation to which an external force is applied by the rigid-body object O2 is calculated based on XPBD. For example, the state of the soft-body object O1 after deformation to which an external force is applied is calculated from the distances between the particles that make up the tetrahedron and the masses of the particles, with the condition of minimizing the potential energy density of the tetrahedron being a constraint (constraint in XPBD).

[0026] In a simulation of a surgical environment, the contact force generated by contact between a soft-body object O1 and a rigid-body object O2 is calculated based on XPBD. For example, when a particle constituting the soft-body object O1 collides with a rigid-body object O2, the contact force is calculated by determining the penetration depth and contact normal vector for each collision pair between the particle that the rigid-body object O2 collided with and the adjacent particle, and then calculating the sum of the forces acting on each particle. The simulation, including the calculation of the deformed state of the soft-body object O1 and the calculation of the contact force, is executed, for example, on a GPU.

[0027] Fig. 3 is a diagram showing an example of a mesh of a soft-bodied object O1. As shown on the left side of Fig. 3, the soft-bodied object O1 is an object that has been meshed using a plurality of polyhedrons. For example, tetrahedrons are used as the polyhedrons that make up the mesh.

[0028] Particles are placed at each vertex of the tetrahedron. The soft body object O1 is configured to include a plurality of tetrahedrons and a plurality of particles. In the example of FIG. 3, the soft body object O1 is configured to include a plurality of tetrahedrons t ０ ~t ｍ－１ The soft body object O1 is composed of m tetrahedrons. ０ ~p ｎ－１ As shown in the balloon, each particle p has information on its position p, mass m, and velocity v.

[0029] The calculation unit 31 performs a simulation of the surgical environment including the meshed soft-body object O1, etc., as described above. For example, when a rigid object O2 comes into contact with the soft-body object O1 and an external force is applied, the soft-body object O1 deforms. The deformation of the soft-body object O1 is, for example, an elastic deformation, and is performed based on the laws of physics. Furthermore, the calculation unit 31 performs mesh refinement in response to the contact of the rigid object O2 and the application of an external force.

[0030] FIG. 4 is a diagram showing an example of mesh subdivision using a subdivision algorithm.

[0031] As shown in Fig. 4, mesh subdivision is performed by focusing on tetrahedrons in contact with the rigid object O2 and tetrahedrons adjacent to those tetrahedrons, and dividing the tetrahedrons of interest. For example, when the surface of the soft-body object O1 is grasped using the rigid object O2 representing forceps, the tetrahedrons at and near the grasped position are divided, resulting in a mesh subdivision. Similarly, when a portion of the soft-body object O1 is cut using the rigid object O2 representing a scalpel, the cut surface and the tetrahedrons near it are divided, resulting in a mesh subdivision. Details of mesh subdivision using the subdivision algorithm will be described later.

[0032] Returning to the explanation of FIG. 2 , the image generation unit 32 performs image processing using 3D rendering and ray tracing based on the simulation results from the calculation unit 31, and generates a simulation image of the surgical field. The simulation image serves as information that visually represents the simulation results from the calculation unit 31. Ray tracing is a technology that models the interaction between a light source and a virtual object and generates realistic reflections and shadows. For example, photorealistic rendering generates a realistic image that is close to an actual surgery. The data of the simulation image generated by the image generation unit 32 is supplied to the monitor 13 and is used to present information to the user.

[0033] The feedback generating unit 33 generates tactile feedback information that represents the contact force calculated by the simulation performed by the computing unit 31. The tactile feedback information generated by the feedback generating unit 33 is supplied to the haptic device 12 and used as feedback to the user.

[0034] <<Subdivision Algorithm>> <Tetrahedron Division Pattern> Mesh subdivision using the subdivision algorithm is performed by dividing a portion of the tetrahedrons constituting the soft body object O1 that are of interest. The tetrahedrons are divided by adding vertices to the edges of the tetrahedrons and dividing them with planes that pass through the added vertices and the other vertices. One tetrahedron is made up of vertices (particles), edges, and faces.

[0035] Fig. 5 shows an example of a division pattern for a tetrahedron. In Fig. 5, the tetrahedron shown in the upper left corner is the tetrahedron before division. In Fig. 5, the four vertices of the tetrahedron before division are omitted. Dots on the edges indicate vertices to be added.

[0036] The dashed lines on the 10 tetrahedrons other than the one shown in the upper left corner indicate the division planes of the tetrahedrons. The numbers in the upper left corner of each tetrahedron indicate the number of vertices added to the edge. The letters next to the numbers indicate the division pattern for each number of vertices added.

[0037] Each division pattern will be explained below using numbers and letters as appropriate. For example, division pattern 2a indicates the first division pattern when dividing by adding two vertices. Division pattern 2b indicates the second division pattern when dividing by adding two vertices.

[0038] The method for dividing the tetrahedron shown in FIG. 5 is described, for example, in Reference 1. Reference 1: D. Ruprecht, H. Muller, “A scheme for edge-based adaptive tetrahedron subdivision,” Mathematical Visualization: Algorithms, Applications and Numerics. Berlin, Heidelberg: Springer Berlin Heidelberg, 1998.

[0039] FIG. 6 is a diagram showing an example of division when division pattern 1 is used.

[0040] When a vertex is added to one edge as shown on the left side of Figure 6, the original tetrahedron is divided into two tetrahedrons, as indicated by the arrows, using a plane passing through the added vertex and the diagonal vertex.

[0041] FIG. 7 is a diagram showing an example of division when division pattern 2a is used.

[0042] When vertices are added to two edges as shown on the left side of Figure 7, the original tetrahedron is divided into one tetrahedron (upper row) and one polyhedron with quadrilateral faces (lower row), as indicated by the arrows, using a plane passing through the added vertex and the diagonal vertex. When the 3D shape of the soft body object O1 is represented by tetrahedrons, each face of each subdivided polyhedron must be triangular. The quadrilateral-faced polyhedron is further divided into multiple tetrahedrons, each with a triangular face.

[0043] In this way, if a quadrilateral face appears in a polyhedron divided using the added vertices, the polyhedron having that quadrilateral face is further divided into multiple tetrahedrons. For example, when divided using division pattern 3a, three quadrilateral faces appear, and when divided using division pattern 3c, two quadrilateral faces appear. When divided using division pattern 4a, two quadrilateral faces appear, and when divided using division pattern 4b, four quadrilateral faces appear. When divided using division pattern 5, two quadrilateral faces appear.

[0044] The position where the vertex is added may be the center of the edge, or may be a position shifted from the center of the edge. Also, the soft body object O1 may be configured as a polyhedron other than a tetrahedron.

[0045] <Details of Division Process> Fig. 8 is a flowchart illustrating the division process using the subdivision algorithm. The process shown in Fig. 8 is started when it is detected that a rigid object O2 has come into contact with a soft-body object O1. Division of the tetrahedron is performed in response to the contact of the rigid object O2, which is another object, with the soft-body object O1.

[0046] In step S1, the calculation unit 31 determines tetrahedrons in contact with the rigid object O2 as division targets. The division targets are determined so as to include the tetrahedrons in contact with the rigid object O2.

[0047] In step S2, the calculation unit 31 adds vertices to all edges of the tetrahedron to be divided and divides it. In this example, the division of the tetrahedron in contact with the rigid object O2 is performed using division pattern 6. The division of the tetrahedron in contact with the rigid object O2 may also be performed using a division pattern other than division pattern 6.

[0048] In step S3, the calculation unit 31 determines a division pattern for tetrahedrons adjacent to the tetrahedron in contact with the rigid object O2. For example, the division pattern for the adjacent tetrahedrons is determined depending on the division status of the tetrahedron in contact with the rigid object O2. The calculation unit 31 then divides the tetrahedron adjacent to the tetrahedron in contact with the rigid object O2 using the determined division pattern.

[0049] In step S4, the calculation unit 31 sets the status of the tetrahedron after division (the tetrahedron generated by division) to valid and the status of the tetrahedron before division to invalid. A valid status indicates that the tetrahedron will be subject to calculation by XPBD, and an invalid status indicates that the tetrahedron will not be subject to calculation by XPBD. Tetrahedrons with a valid status are used to display the soft-body object O1 and calculate contact forces. The above process is repeated while the user is training.

[0050] Here, the following methods are available for detecting contact with the rigid object O2: - A method in which candidate tetrahedrons are extracted using an Axis Aligned Bounding Box (aka AABB), and then the presence or absence of contact with the faces (triangular faces) of the candidate tetrahedrons is determined; - A method in which calculations are made using a contact detection algorithm between the Oriented Bounding Box (aka OBB) of a tetrahedron and a Sphere-box, Capsule-box, or Box-box.

[0051] The reason why the status of the tetrahedron before division is invalidated in step S4 is to link the tetrahedron before division with the tetrahedron after division and manage them as a Linked-List. In other words, although it is not reflected in the display, information on the vertices and edges of the tetrahedron before division is retained. This makes it possible to identify the vertices and edges used in division from the model structure at the time of initialization, and to identify the state of the tetrahedron before division from the tetrahedron after division.

[0052] Fig. 9 is a diagram showing the flow of tetrahedron division. In Fig. 9, each tetrahedron constituting the soft body object O1 is shown as a triangle. Fig. 9 shows 12 of the tetrahedrons constituting the soft body object O1. When compared with the rigid body object O2, the size of the tetrahedrons constituting the soft body object O1 is actually even smaller than the size shown in Fig. 9. The same applies to other figures such as Fig. 10, which will be described later.

[0053] As shown on the left side of FIG. 9, the rigid object O2 is a tetrahedron t １０１ If it is detected that the tetrahedron t １０１ is determined as the object to be divided (step S1). １０２ and tetrahedron t １０３ But the tetrahedron t １０１ is a tetrahedron adjacent to

[0054] Tetrahedron t １０１ When is determined as the object of division, the tetrahedron t １０１ Each edge of １０１ ~p １０３ is added, and the tetrahedron t １０１ is divided (step S2). As explained with reference to Fig. 5, division pattern 6 is a division pattern that adds vertices to each edge of the tetrahedron. Vertices are also added to edges that do not appear in Fig. 9.

[0055] Also, as shown on the right side of FIG. １０１ The tetrahedron t adjacent to １０２ and tetrahedron t １０３ are divided into tetrahedrons t １０２ is the vertex p１０２ The added edge is the tetrahedron t １０１ is a tetrahedron shared with vertex p １０２ The tetrahedron t １０３ is the vertex p １０３ The added edge is the tetrahedron t １０１ is a tetrahedron shared with vertex p １０３ The division is performed using at least

[0056] FIG. 10 is a diagram showing the division results of the tetrahedron.

[0057] As shown in circles #1 to #3 in FIG. 10, the rigid object O2 is a tetrahedron t １０１ When the rigid object O2 is in contact with the tetrahedron t １０１ is divided into four tetrahedrons. １０１ Four tetrahedrons are generated from the original tetrahedron. １０１ The tetrahedron t adjacent to １０２ and tetrahedron t １０３ are each divided into two tetrahedrons. １０２ Two tetrahedrons are generated using the original tetrahedron, and the tetrahedron t １０３ Two tetrahedrons are generated using the original tetrahedron.

[0058] Typically, the rigid object O2 comes into contact with multiple tetrahedrons at the same time. When the rigid object O2 comes into contact with multiple tetrahedrons, the above-described division is performed on each tetrahedron and its adjacent tetrahedrons. This divides all tetrahedrons within a certain range, achieving high-resolution meshes.

[0059] A certain range near the contact position of the rigid object O2 is considered to be the range that the user is paying attention to. By increasing the resolution of the mesh in a certain range near the contact position of the rigid object O2, it is possible to improve the image quality of the simulation image in the range that the user is paying attention to. Furthermore, by calculating the contact force based on the high-resolution mesh, it is possible to improve the realism of the haptic feedback.

[0060] <Details of Integration Process> Fig. 11 is a flowchart illustrating the integration process using the subdivision algorithm. If the rigid object O2 is in contact with a tetrahedron at a position different from the tetrahedron to be integrated, the division process described with reference to Fig. 8 is executed in parallel with the integration process.

[0061] In step S11, the calculation unit 31 detects that the tetrahedron that was in contact with the rigid object O2 has become no longer in contact with the rigid object O2.

[0062] In step S12, the calculation unit 31 invalidates the vertices added in the division process.

[0063] In step S13, the calculation unit 31 invalidates the edge including the invalidated vertex and the face including that edge.

[0064] In step S14, the calculation unit 31 identifies the original tetrahedron that includes the invalidated edge. Here, the tetrahedron before division that was invalidated in step S4 of Fig. 8 is identified as the original tetrahedron. The calculation unit 31 also deletes all post-division elements generated by dividing the identified original tetrahedron.

[0065] In step S15, the calculation unit 31 validates the original tetrahedron from which all elements have been deleted. That is, the status of the tetrahedron that was invalidated in step S4 of Fig. 8 becomes valid. The above process is performed on the tetrahedron that has been divided in response to contact with the rigid object O2.

[0066] 12 and 13 are diagrams showing the flow of tetrahedron integration.

[0067] As shown on the left side of FIG. 12, the rigid object O2 is in contact with the tetrahedron t １０１ A case where the rigid object O2 is separated from the object O2 and this is detected will be described (step S11).

[0068] In this case, as shown in the center of FIG. 12, the tetrahedron t １０１In the example of FIG. 12, the status of the vertex added to the tetrahedron t １０４ Vertex p １１１ ~p １１３ Vertex p １１１ ~p １１３ is the tetrahedron t by the division process that is running in parallel with the integration process. １０４ In FIG. 12, vertex p １０１ ~p １０３ is the vertex p １１１ ~p １１３ The difference between the two is indicated by the hatching. １０１ ~p １０３ Indicates that the status is invalid.

[0069] Also, as shown on the right side of FIG. 12, the invalid vertex p １０１ ~p １０３ The edges including the tetrahedron t and the faces including the edges are invalidated (step S13). １０１ ~t １０３ The hatch type of each tetrahedron is changed by dividing the vertex p １０１ ~p １０３ indicates that the edge containing the edge and the face containing the edge are invalid.

[0070] As shown on the left side of FIG. 13, the original tetrahedron containing the invalidated edge, tetrahedron t １０１ ~t １０３ is identified, and the tetrahedron t １０１ ~t １０３ All elements of the tetrahedron generated using the original tetrahedron are deleted (step S14).

[0071] After the tetrahedron elements are deleted, the original tetrahedron, tetrahedron t １０１ ~t １０３ becomes valid (step S15).

[0072] As shown on the right side of FIG. 13, the rigid object O2 is a tetrahedron t １０４ If it is in contact with the tetrahedron t １０４ Together with the tetrahedron t １０４The tetrahedron t adjacent to １０３ is divided into multiple tetrahedrons by the division process.

[0073] In this way, the divided tetrahedrons are integrated and returned to their pre-division state when they are released from contact with the rigid object O2. This makes it possible to prevent the number of tetrahedrons constituting the soft-body object O1 from continuing to increase. Typically, the range that the user focuses on is the range in which the rigid object O2 is in contact. By returning the tetrahedrons to their original state starting with those that have released contact with the rigid object O2, it is possible to prevent an increase in the number of tetrahedrons, which would otherwise increase the amount of calculation required for the simulation.

[0074] <Multi-Level Division> The above describes a single-level division of a tetrahedron that is in contact with a rigid object O2, but it is also possible to further divide the tetrahedron generated by division. If a rigid object O2 is in contact with a tetrahedron generated by dividing a tetrahedron, multiple-level division (such as two-level, three-level, etc.) is performed.

[0075] 14 and 15 are diagrams showing examples of two-stage division. Division in more than two stages can be performed in the same manner.

[0076] As shown on the left side of FIG. 14, the rigid object O2 is a tetrahedron t １０１ If it is detected that the tetrahedron t １０１ Each edge of １０１ ~p １０３ is added, and the tetrahedron t １０１ and its adjacent tetrahedron t １０２ , tetrahedron t １０３ is divided (first stage division).

[0077] Also, the tetrahedron t １０１ tetrahedron t among the tetrahedrons generated by dividing １１１ When the rigid object O2 is in contact with the tetrahedron t １１１ Each edge of １１１ ~p １１３ is added, and the tetrahedron t １１１is divided along with the adjacent tetrahedrons (second division).

[0078] From the state shown on the right side of FIG. 14, the rigid object O2 is moved to the tetrahedron t １０４ When the state of contact with the rigid object O2 is changed to one where the rigid object O2 is in contact with the rigid object O2, the tetrahedrons that have been released from contact with the rigid object O2 are integrated as shown on the left side of Fig. 15. The tetrahedrons generated by the second division are integrated, and then the tetrahedrons generated by the first division are integrated, resulting in the tetrahedron t １０１ ~t １０３ is valid. Tetrahedron t １０１ ~t １０３ The state of the rigid object O2 is the tetrahedron t １０１ returns to the state it was in before contact.

[0079] Also, the tetrahedron t １０４ Each edge of １３１ ~p １３３ is added, and the tetrahedron t １０４ and its adjacent tetrahedron t １０３ is divided (first stage division).

[0080] Tetrahedron t １０４ tetrahedron t among the tetrahedrons generated by dividing １１２ When the rigid object O2 is in contact with the tetrahedron t １１２ Each edge of １４１ ~p １４３ is added, and the tetrahedron t １１２ is divided along with the adjacent tetrahedrons (second division).

[0081] By performing multi-level division in the manner described above, it is possible to prevent the number of divided tetrahedrons from continuing to increase, while further increasing the resolution of the mesh in a certain range where the rigid object O2 continues to contact.

[0082] <Physical Properties of Vertices> The number of vertices increases or decreases as a result of dividing and integrating a tetrahedron. The calculation unit 31 calculates the physical properties of each vertex so that the physical properties are maintained before and after dividing and integrating the tetrahedron. The physical properties of each vertex after dividing the tetrahedron are calculated based on the mass and momentum of each vertex before dividing, and the physical properties of each vertex after integrating the tetrahedron are calculated based on the mass and momentum of each vertex before integrating. The calculation unit 31 performs a simulation after increasing or decreasing the vertices of the tetrahedron, which is a polyhedron that makes up the soft-body object O1, based on the mass and momentum of the vertices before the increase or decrease. Here, a method for calculating mass and velocity as the physical properties of the vertices will be described.

[0083] FIG. 16 is a diagram showing the state of a tetrahedron.

[0084] The state of the tetrahedron is classified into two types: an unloaded state and a moving state, as shown in Figure 16. The moving state is a state in which the tetrahedron is deformed due to the application of an external force, etc., as a result of the simulation.

[0085] Vertex p in the unloaded state ０ ~p ３ have information on the position x and mass m, respectively. The subscript numbers of the position x and mass m correspond to the number of the vertex p that has that information. The superscript bar added to the position x indicates that it is in an unloaded state. On the other hand, the vertex p in the moving state ０ ~p ３ have information on the position x, mass m, and velocity v, respectively.

[0086] Each tetrahedron has information on its volume V. The volume V is expressed by equation (1) using information on the vertex p in the unloaded state.

[0087] The relationship between the mass m and the volume V of each vertex is expressed by equation (2) using the density ρ. If a vertex belongs to multiple tetrahedrons, the sum of the masses of the tetrahedrons is used as the mass m of the vertex.

[0088] Each tetrahedron also has stiffness information. Since the finite element method (FEM) is used in XPBD, stiffness is expressed as Young's modulus and Poisson's ratio. The calculation unit 31 manages information on each tetrahedron that makes up the soft-body object O1 and information on each vertex that makes up each tetrahedron.

[0089] Calculation method of physical properties when dividing As shown in FIG. 17, a case where a tetrahedron is divided using division pattern 6 will be described. In division pattern 6, vertex p ０ and vertex p １ , vertex p １ and vertex p ２ , vertex p ０ and vertex p ２ , vertex p ０ and vertex p ３ , vertex p ２ and vertex p ３ , vertex p １ and vertex p ３ On the edge between each vertex of ４ ~p ９ will be added.

[0090] Figure 18 shows the state of each vertex before and after division. In Figure 18, a dash (') added to each symbol indicates the state after division. The physical properties of each vertex after division are updated to satisfy the two conservation laws of mass and momentum.

[0091] The state where the law of conservation of mass is satisfied is expressed as equation (3), where the sum of the masses of the vertices before division is equal to the sum of the masses of the vertices after division. By calculating based on the volume and density in an unloaded state, the mass that satisfies the law of conservation of mass can be determined as the mass of each vertex after division.

[0092] On the other hand, the state that satisfies the law of conservation of momentum is expressed as equation (4), where the sum of the products of the masses and velocities of the vertices before division is equal to the sum of the products of the masses and velocities of the vertices after division. ｊ is calculated so as to satisfy the formula (4), where i represents the number of vertices before division (4), and j represents the number of vertices after division.

[0093] The division of a tetrahedron is performed by adding vertices to the edges. The velocity of each vertex after division is calculated for each edge. As shown in Figure 19, ０ and vertex p １ A vertex p added on the edge between ４ If we focus on vertex p ４ Velocity v' ４ is the vertex p ０ and vertex p １ The vertex p is calculated by the formula (5) based on the parameters. ０ , p １ Mass m' after division of ０ , m' １ has been determined based on the law of conservation of mass.

[0094] Vertex p before division ０ , p １ are n ０ Pieces and n １ When generalized to be used for edges, equation (5) can be expressed as equation (6).

[0095] Vertex p ４ The velocity v' of each vertex other than ｊ is also calculated in the same way.

[0096] - Calculation method of physical properties during integration Similarly, during integration, the physical properties of each vertex after integration are updated so that they satisfy the two conservation laws of mass and momentum.

[0097] The state where the law of conservation of mass is satisfied can be expressed as an equation where the sum of the masses of the vertices before integration is equal to the sum of the masses of the vertices after integration. By calculating based on the volume and density in an unloaded state, the mass that satisfies the law of conservation of mass can be determined as the mass of each vertex after integration.

[0098] On the other hand, a state that satisfies the law of conservation of momentum is expressed as an equation in which the sum of the products of the masses and velocities of the vertices before integration matches the sum of the products of the masses and velocities of the vertices after integration.

[0099] As shown in FIG. 20, by integrating the tetrahedrons, the vertex p ０ and vertex p １ Vertex p on the edge between ４When we focus on the case where we delete the state that satisfies the law of conservation of momentum, it is expressed by equation (7).

[0100] When the relationship between the speeds before and after integration is expressed by equation (8), equation (7) can be expressed as equation (9).

[0101] Based on equation (9), Δv is expressed as in equation (10).

[0102] Using Δv, vertex p ０ and vertex p １ The velocities before and after the integration are expressed by equations (11) and (12).

[0103] As with the division, the vertex p before integration ０ , p １ are n ０ Pieces and n １ When generalized as being used for edges, equations (11) and (12) can be expressed as equations (13) and (14), respectively.

[0104] In this manner, information about each tetrahedron constituting the soft-body object O1 and information indicating the physical properties of each vertex constituting each tetrahedron are calculated and managed in the calculation unit 31 when dividing or integrating tetrahedrons. By updating the physical properties of each vertex so as to satisfy the laws of conservation of mass and momentum, it becomes possible to simulate behavior that is consistent before and after the division or integration process.

[0105] <Simulation of Cutting of a Soft-Body Object> Subdivision of Cutting Surface While the subdivision of meshes when deforming a soft-body object O1 by grasping it with forceps, etc., has been described, this technology can also be applied to simulation of cutting it with a scalpel. When cutting it with a scalpel as a rigid object O2, changes occur in the geometry of the soft-body object O1.

[0106] FIG. 21 is a diagram showing an example of a cross section.

[0107] Here, we will explain the mesh subdivision when cutting a part of the soft body object O1 along the dotted line L1 shown in Figure 21. ２０１ ~t ２０４ The cutting plane is formed so as to pass through the

[0108] FIG. 22 is a diagram showing the flow of division when only the mesh of the cutting surface is subdivided.

[0109] In response to the cutting operation along the dotted line L1, the tetrahedron t ２０１ ~t ２０４ Vertex p is on the edge of the cutting surface of ２０１ ~p ２０５ In the example of FIG. 22, a tetrahedron t ２０１ vertex p on the edge of ２０１ and vertex p ２０２ is added, and the tetrahedron t ２０２ vertex p on the edge of ２０２ and vertex p ２０３ is added. Also, the tetrahedron t ２０３ vertex p on the edge of ２０３ and vertex p ２０４ is added, and the tetrahedron t ２０４ vertex p on the edge of ２０４ and vertex p ２０５ Vertex p is added. ２０２ The added edge is the tetrahedron t ２０１ and tetrahedron t ２０２ is a shared edge, and vertex p ２０３ The added edge is the tetrahedron t ２０２ and tetrahedron t ２０３ is the edge shared by vertex p ２０４ The added edge is the tetrahedron t ２０３ and tetrahedron t ２０４ is the edge shared by

[0110] As shown in the center of FIG. 22, the tetrahedron t ２０１ ~t ２０４ is divided, and then, as shown on the right side of FIG. ２０１ ~p ２０５ Vertex p' is a duplicate of ２０１ ~p' ２０５is generated. ２０１ ~p ２０５ constitute the vertices of the tetrahedron that appears on the left side of the cut surface after division, and vertex p' ２０１ ~p' ２０５ The vertices of the tetrahedron that appears on the right side of the cut plane after division are the vertices of the tetrahedron after division. ２０１ ~p ２０５ and vertex p' ２０１ ~p' ２０５ There is no edge connection between the tetrahedrons t ２０１ ~t ２０４ A cutting plane passing through is formed.

[0111] In this way, when an operation to cut a part of the soft-body object O1 is performed, the mesh is subdivided so as to divide the polyhedron that forms the cut surface. ２０１ ~p ２０５ and vertex p' ２０１ ~p' ２０５ The parameters that indicate the physical properties of the object are calculated so as to satisfy the laws of conservation of mass and momentum of the object before and after cutting.

[0112] FIG. 23 is a diagram showing the flow of division of a tetrahedron through which a cutting plane passes.

[0113] In the example of Fig. 23, not only the cutting surface but also the mesh in the vicinity of the cutting surface is subdivided. That is, in response to the operation of cutting along the dotted line L1, the tetrahedron t ２０１ ~t ２０４ is divided by division pattern 6. ２０１ ~t ２０４ The tetrahedrons adjacent to are also split in the same manner when contact with rigid object O2 is detected.

[0114] By using a refinement algorithm to continuously refine the mesh that makes up the cutting surface, it is possible to generate a highly detailed cutting surface. The mesh is refined in a way that preserves the mass and momentum of the vertices before and after cutting, so it is possible to guarantee consistency of behavior even when the geometry is changed.

[0115] Example of using an electric scalpel When tissue is cut using an electric scalpel, a part of the tissue disappears due to a rise in temperature. The surgical simulator 1 also simulates the disappearance of the tissue and reflects it in the display of the simulation video, etc.

[0116] FIG. 24 is a diagram showing the flow of division when cutting using an electric scalpel.

[0117] The state shown on the left and center of Fig. 24 is the same as the state of the soft body object O1 described with reference to Fig. 22. ２０１ ~t ２０４ After the vertex p is divided, the vertex to be deleted is determined by energy propagation, and the vertex and the associated edges and faces are deleted, as shown on the right side of Figure 24. In the example of Figure 24, the vertex p ２０１ , vertex p ２０２ At the same time, the vertex p ２０１ , vertex p ２０２ The edge has been added and the face containing that edge has been deleted.

[0118] In this way, the subdivision algorithm can also be applied to simulate tissue loss. When the geometry of the soft-body object O1 is changed due to cutting or the like, the division of the tetrahedrons may be maintained as is, or may be merged when the contact with the rigid-body object O2 is released.

[0119] <<Modifications>> Although the application of the subdivision algorithm to a simulation using XPBD has been described, the subdivision algorithm described above can also be applied to other simulations using vertex-based 3D models.

[0120] In the above, the object to be subdivided into meshes is a soft-body object obtained by meshing a 3D model of an organ or the like. However, the subdivision algorithm can also be applied to the subdivision of objects obtained by meshing a 3D model of various soft objects other than organs.

[0121] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0122] 25 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program. The information processing device 11 is configured, for example, by a computer having the configuration shown in Fig. 25. A CPU 1001 corresponds to the information processing unit 21, and a memory unit 1008 corresponds to the memory unit 22.

[0123] A CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0124] An input / output interface 1005 is also connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, etc., and an output unit 1007 including a display, a speaker, etc. are connected to the input / output interface 1005. In addition, a storage unit 1008 including a hard disk, a nonvolatile memory, etc., a communication unit 1009 including a network interface, etc., and a drive 1010 that drives removable media 1011 are also connected to the input / output interface 1005.

[0125] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0126] The program executed by the CPU 1001 is installed in the storage unit 1008 by being recorded on, for example, a removable medium 1011 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0127] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0128] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0129] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0130] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0131] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0132] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0133] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0134] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0135] (1) An information processing device including a calculation unit that performs a simulation after increasing or decreasing vertices of polyhedrons constituting an object obtained by meshing a 3D model of a flexible object, based on the mass and momentum of the vertices before the increase or decrease. (2) The information processing device described in (1), in which the calculation unit divides some of the polyhedrons and simulates physical properties of each vertex of the polyhedrons generated by the division. (3) The information processing device described in (2), in which the calculation unit divides some of the polyhedrons including the polyhedron contacted by another object in response to the object being brought into contact with the other object. (4) The information processing device described in (3), in which the calculation unit divides the polyhedron contacted by the other object and the polyhedron adjacent to the polyhedron contacted by the other object as some of the polyhedrons. (5) The information processing device described in (3) or (4), in which the calculation unit further divides the polyhedron contacted by the other object, among the polyhedrons generated by the division. (6) The information processing device according to any one of (3) to (5), wherein the calculation unit integrates the polyhedron generated by division into a state before division and simulates physical properties of each vertex of the integrated polyhedron. (7) The information processing device according to (6), wherein the calculation unit integrates the polyhedron from which the contact of the other object has been released into a state before division. (8) The information processing device according to (6) or (7), wherein the calculation unit divides the polyhedron and integrates the polyhedron at a position different from the polyhedron to be divided, in parallel. (9) The information processing device according to any one of (1) to (8), wherein the calculation unit simulates the increase or decrease of the polyhedron so that the sum of the masses of the vertices before the increase or decrease matches the sum of the masses of the vertices after the increase or decrease, and so that the sum of the momentum of the vertices before the increase or decrease matches the sum of the momentum of the vertices after the increase or decrease. (10) The information processing device according to any one of (3) to (8), wherein the operation of the object using the other object that is a rigid object includes an operation of grasping the object and an operation of cutting the object.(11) The information processing device according to (10), wherein the calculation unit divides the polyhedron constituting the cutting surface when an operation to cut the object is performed. (12) The information processing device according to (6), wherein the object represents an organ, and the other object represents a surgical instrument. (13) The information processing device according to (12), wherein the calculation unit divides the polyhedron constituting the organ in response to contact with the surgical instrument. (14) The information processing device according to (13), wherein the calculation unit integrates the polyhedrons after contact with the surgical instrument is released back to their pre-division state. (15) The information processing device according to any of (12) to (14), wherein the operation using the surgical instrument includes an operation of grasping the organ using forceps as the surgical instrument and an operation of cutting the organ using a scalpel as the surgical instrument. (16) An information processing method, wherein the information processing device performs a simulation after increasing or decreasing vertices of polyhedrons constituting an object obtained by meshing a 3D model of a flexible object, based on the mass and momentum of the vertices before the increase or decrease. (17) A program for causing a computer to execute a process of simulating after increasing or decreasing vertices of a polyhedron constituting an object obtained by meshing a 3D model of a flexible object, based on the mass and momentum of the vertices before the increase or decrease. (18) A simulation system including: an information processing device that simulates after increasing or decreasing vertices of a polyhedron constituting an object obtained by meshing a 3D model of a flexible object, based on the mass and momentum of the vertices before the increase or decrease, a display device that displays an image showing the results of the simulation, and an input device that is used by a user to operate other objects that are used to operate the object in a virtual environment.

[0136] REFERENCE SIGNS LIST 1 Surgical simulator, 11 Information processing device, 12 Haptic device, 13 Monitor, 21 Information processing unit, 22 Storage unit, 31 Calculation unit, 32 Image generation unit, 33 Feedback generation unit

Claims

1. An information processing device having a calculation unit that performs simulations after increasing or decreasing the vertices of polyhedrons that make up an object in which a 3D model of a flexible object has been meshed, based on the mass and momentum of the vertices before the increase or decrease.

2. The information processing device according to claim 1, wherein the calculation unit divides a part of the polyhedron and simulates the physical properties of each vertex of the polyhedron generated by the division.

3. The information processing device according to claim 2, wherein the calculation unit, in response to another object coming into contact with the object, divides a portion of the polyhedron including the polyhedron that is in contact with the other object.

4. The information processing device according to claim 3, wherein the calculation unit divides the polyhedron that is in contact with the other object and the polyhedron adjacent to the polyhedron that is in contact with the other object as a part of the polyhedron.

5. The information processing device according to claim 3, wherein the calculation unit further divides, of the polyhedrons generated by division, those polyhedrons that come into contact with the other object.

6. The information processing device according to claim 3, wherein the calculation unit integrates the polyhedrons generated by division into the state before division, and simulates the physical properties of each vertex of the integrated polyhedron.

7. The information processing device according to claim 6, wherein the calculation unit integrates the polyhedron from which the contact with the other object has been released back into the state before division.

8. The information processing device according to claim 6, wherein the calculation unit performs division of the polyhedron and integration of the polyhedrons at positions different from the polyhedron to be divided in parallel.

9. The information processing device according to claim 1, wherein the calculation unit performs a simulation of the polyhedron after the increase or decrease so that the sum of the masses of the vertices before the increase or decrease is equal to the sum of the masses of the vertices after the increase or decrease, and the sum of the momentum of the vertices before the increase or decrease is equal to the sum of the momentum of the vertices after the increase or decrease.

10. The information processing device according to claim 3, wherein the manipulation of the object using the other object, which is a rigid object, includes an operation of grasping the object and an operation of cutting the object.

11. The information processing device according to claim 10, wherein the calculation unit divides the polyhedron that forms the cutting plane when an operation to cut the object is performed.

12. The information processing device according to claim 6, wherein the object represents an organ, and the other object represents a surgical instrument.

13. The information processing device according to claim 12, wherein the calculation unit divides the polyhedron constituting the organ in response to contact with the surgical instrument.

14. The information processing device according to claim 13, wherein the calculation unit integrates the polyhedron from which the surgical instrument has been released back into the state before division.

15. The information processing device according to claim 12, wherein the operations using the surgical instrument include an operation of grasping the organ using forceps as the surgical instrument and an operation of cutting the organ using a scalpel as the surgical instrument.

16. An information processing method in which an information processing device performs a simulation after increasing or decreasing the vertices of a polyhedron that makes up an object in which a 3D model of a flexible object has been meshed, based on the mass and momentum of the vertices before the increase or decrease.

17. A program that causes a computer to perform a process that simulates the increase or decrease of the vertices of polyhedrons that make up a meshed 3D model of a flexible object, based on the mass and momentum of the vertices before the increase or decrease.

18. A simulation system comprising: an information processing device that performs a simulation of the increase or decrease of vertices of a polyhedron that constitutes an object in which a 3D model of a flexible object is meshed, based on the mass and momentum of the vertices before the increase or decrease; a display device that displays an image representing the results of the simulation; and an input device that is used by a user to operate other objects that are used to operate the object in a virtual environment.

Images