Device and simulation system for simulating soft tissue deformation

Abstract

The present application discloses a device and a simulation system for simulating soft tissue deformation. The present application achieves a good balance between the real-time performance and rendering authenticity of the simulation system through local deformation algorithm and mesh optimization of the surgical area, and can realize the puncture and cutting simulation of any area of ​​soft tissue, so as to have the effect of free puncture site and cutting path, and quickly simulate the soft tissue deformation process caused by virtual surgery, thereby realizing realistic visual rendering and mechanical feedback to meet the needs of virtual surgery.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a device and a simulation system for simulating soft tissue deformation. Background Art

[0002] Virtual reality (VR) technology is interactive, immersive and imaginative. The craniomaxillofacial surgery simulation system developed based on VR technology can assist clinical practice in many aspects, such as surgical plan design, postoperative face prediction, surgical teaching, operation training, etc. Compared with traditional clinical teaching and practice processes, the virtual surgery simulation system has the advantages of high efficiency, economy and convenience, and has broad development prospects in the medical field.

[0003] Most of the craniomaxillofacial surgery simulation systems that have been developed so far only have bone tissue models and surgical operation simulations. For some simulation systems that have soft tissue models, their soft tissue models do not have geometric modeling with real biomechanical characteristics. The selection and operation of surgical incisions are the most common and challenging steps in craniomaxillofacial surgery. Compared with other soft tissue organs, facial soft tissue has a more complex and subtle anatomical structure, and incision operations require more skills. In order to simulate facial incision operations more realistically and increase clinical practicality, it is very important to use a physical model of facial soft tissue with real biomechanical characteristics to simulate surgical incision operations.

[0004] The soft tissue deformation caused by surgical operation is manifested as mesh deformation, destruction and reconstruction on the physical model of facial soft tissue. If the simulation accuracy of visual rendering and force feedback rendering is to be improved, the resulting calculation scale will be large, and the time required for calculation will increase; if the simulation system is to meet the real-time requirements of surgical operation visual feedback and force feedback rendering, the calculation accuracy needs to be sacrificed to reduce the calculation scale.

[0005] Therefore, for the simulation of soft tissue deformation, researchers have been working hard to find a method that takes both authenticity and real-time into account. Summary of the invention

[0006] The purpose of the present application is to provide a device and a simulation system for simulating soft tissue deformation in the embodiments of the present application, which achieves a good balance between the real-time performance and rendering authenticity of the simulation system through local deformation algorithm and surgical area mesh optimization, and can realize puncture and cutting simulation of any area of ​​soft tissue, so as to have the effect of free puncture site and cutting path, and quickly simulate the soft tissue deformation process caused by virtual surgery, thereby realizing realistic visual rendering and mechanical feedback to meet the needs of virtual surgery.

[0007] According to the first aspect of the present application, an embodiment of the present application provides a device for simulating soft tissue deformation. The device includes: a collision detection module, which is used to determine whether a collision occurs between a virtual surgical instrument and a three-dimensional soft tissue physical model; when a collision is determined to occur, the position information and depth information of the collision node are obtained; a mesh deformation module, which is used to calculate the influence area of ​​the three-dimensional soft tissue physical model based on the position information of the collision node; establish a stiffness matrix for all triangular units in the influence area; input the collision node depth information into the stiffness matrix to calculate the displacement of the remaining nodes in the influence area; and render the mesh deformation of the influence area based on the displacement of all nodes in the influence area.

[0008] Optionally, in some embodiments, the device for simulating soft tissue deformation also includes: a soft tissue physics module, which includes a geometric modeling unit and a physical modeling unit; the geometric modeling unit is used to obtain medical digital image communication data of a virtual object, and perform threshold screening, interactive segmentation and construction based on the scanned image data to obtain a three-dimensional soft tissue model, and obtain a grid structure of the three-dimensional soft tissue model; the physical modeling unit is used to set boundary conditions for the three-dimensional soft tissue model to obtain virtual partitions; obtain the mechanical properties of different partitions; and fit the mechanical properties of different partitions to the corresponding partitions of the three-dimensional soft tissue model to obtain a three-dimensional soft tissue physical model.

[0009] Optionally, in some embodiments, the geometric modeling unit is also used to determine the surgical area and non-surgical area of ​​the three-dimensional soft tissue model according to preset goals, and to increment the mesh of the surgical area and decrement the mesh of the non-surgical area according to preset rules.

[0010] Optionally, in some embodiments, the device for simulating soft tissue deformation also includes: an instrument modeling module, which is used to obtain laser scanning image data of the virtual surgical instrument; and render a three-dimensional virtual surgical instrument model based on the laser scanning image data of the virtual surgical instrument.

[0011] Optionally, in some embodiments, the virtual surgical instrument and the three-dimensional soft tissue model are both composed of triangular faces.

[0012] Optionally, in some embodiments, the device for simulating soft tissue deformation also includes a puncture operation module and a force feedback acquisition module; the puncture operation module is used to control the virtual surgical instrument to move toward the three-dimensional soft tissue physical model; after the collision detection module determines that a collision has occurred, based on the depth information of the collision node, the feedback force of the collision node is calculated; it is determined whether the feedback force is greater than a preset maximum puncture strength; when it is determined that the feedback force is less than the maximum puncture strength, it is determined that the grid of the three-dimensional soft tissue physical model is not destroyed; when it is determined that the feedback force is greater than the maximum puncture strength, it is determined that the grid of the three-dimensional soft tissue physical model is punctured; the inclination angle of the virtual surgical instrument is determined; when it is determined that the inclination angle of the virtual surgical instrument is within the first preset angle range, it is determined that the operation of the virtual surgical instrument on the three-dimensional soft tissue physical model is a puncture operation; the force feedback acquisition module is used to obtain the feedback force of the collision node, and output it to the force feedback device based on the obtained feedback force.

[0013] Optionally, in some embodiments, the device for simulating soft tissue deformation also includes a cutting operation module; the cutting operation module is used to determine that the operation of the virtual surgical instrument on the three-dimensional soft tissue physical model is a cutting operation when the inclination angle of the virtual surgical instrument is determined to be within a second preset angle range; the mesh deformation module is also used to deconstruct the constraint relationship of the mesh of the cut part of the three-dimensional soft tissue physical model; reconstruct the cut mesh of the three-dimensional soft tissue physical model; and update the color and texture of the cut mesh.

[0014] Optionally, in some embodiments, the mesh deformation module includes an influence area calculation unit, which is used to define the range of establishing the stiffness matrix and obtain calculation parameters of the influence area, wherein the calculation parameters include a preset displacement error, an elastic modulus of the material of the three-dimensional soft tissue physical model, and a surface skin thickness of the three-dimensional soft tissue physical model; based on the calculation parameters, the influence area of ​​the three-dimensional soft tissue physical model is calculated.

[0015] Optionally, in some embodiments, the mesh deformation module includes a displacement calculation unit, which is used to construct a triangular unit stiffness matrix; based on the depth information, an input of the stiffness matrix is ​​obtained; based on the input of the stiffness matrix, the stiffness matrix is ​​solved to obtain the displacement of all nodes in the affected area.

[0016] Optionally, in some embodiments, the device for simulating soft tissue deformation also includes a process auxiliary module, which is used to simulate auxiliary processes when performing virtual surgery; wherein the auxiliary process includes performing punctuation operations to form a virtual surgical path planning.

[0017] Optionally, in some embodiments, the device for simulating soft tissue deformation also includes a visualization module, which is used to display the mesh deformation of the affected area, and to display whether the mesh of the three-dimensional soft tissue physical model is destroyed, and to display the color and texture of the mesh of the three-dimensional soft tissue physical model after cutting.

[0018] According to the second aspect of the present application, an embodiment of the present application provides a simulation system, which includes a device for simulating soft tissue deformation, a display device, a force feedback device and an auxiliary input device as described in any embodiment of the present application; wherein the display device is connected to the visualization module of the device for simulating soft tissue deformation, and is used for visual interaction with the visualization module; the force feedback device is connected to the force feedback acquisition module, and is used for tactile interaction with the force feedback acquisition module; the auxiliary input device is connected to the process auxiliary module, and is used for auxiliary input interaction with the process auxiliary module.

[0019] The embodiments of the present application provide a device and a simulation system for simulating soft tissue deformation, which achieve a good balance between the real-time performance and rendering authenticity of the simulation system through local deformation algorithms and surgical area mesh optimization. The device for simulating soft tissue deformation described in the present application can simulate the puncture and cutting of any area of ​​soft tissue, thereby having the effect of free puncture sites and cutting paths, and quickly simulates the soft tissue deformation process caused by virtual surgery to achieve realistic visual rendering and mechanical feedback to meet the needs of virtual surgery. In addition, by repeatedly using the device, not only can the cost of surgical training be reduced, but the success rate of surgery can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0021] Figure 1 This is a block diagram of a device for simulating soft tissue deformation according to an embodiment of the present application.

[0022] Figure 2 Schematic diagram of the edge distance of the triangle mesh.

[0023] Figure 3 A schematic diagram of the impact area generated by the collision node.

[0024] Figure 4 Schematic diagram of the puncture operation and cutting operation process.

[0025] Figure 5A Comparison of the effect diagram of the puncture operation and the corresponding schematic diagram of the grid change.

[0026] Figure 5BThis is a comparison of the effect of forehead skin cutting operation and the corresponding grid change diagram.

[0027] Figure 5C This is a comparison of the effect of the skin cutting operation in the masseter area and the corresponding schematic diagram of the grid change.

[0028] Figure 5D Comparison of the effect after chin skin cutting operation and the corresponding grid change diagram.

[0029] Figure 6 FIG. 1 is a schematic diagram of a simulation system according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] The terms "first" and "second" in this document are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0032] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or mutual communication; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0033] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0034] An embodiment of the present application provides a device for simulating soft tissue deformation, which includes: a collision detection module, used to determine whether a collision occurs between a virtual surgical instrument and a three-dimensional soft tissue physical model; when it is determined that a collision occurs, obtaining position information and depth information of the collision node; a mesh deformation module, used to calculate the influence area of ​​the three-dimensional soft tissue physical model based on the position information of the collision node; establish a stiffness matrix for all nodes in the influence area; input depth information into the stiffness matrix to calculate the displacement of the remaining nodes in the influence area; and render the mesh deformation of the influence area based on the displacement of all nodes in the influence area.

[0035] The device for simulating soft tissue deformation in the present application achieves a good balance between the real-time performance and rendering authenticity of the simulation system through local deformation algorithms and mesh optimization of the surgical area. Furthermore, the device for simulating soft tissue deformation described in the present application can simulate the puncture and cutting of any area of ​​the soft tissue, thereby having the effect of free puncture sites and cutting paths, and quickly simulates the soft tissue deformation process produced by virtual surgery to achieve realistic visual rendering and mechanical feedback to meet the needs of virtual surgery. In addition, by repeatedly using the device, not only can the cost of surgical training be reduced, but the success rate of surgery can also be improved.

[0036] The functions of the various modules of the device 100 for simulating soft tissue deformation will be further described below in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, the device 100 for simulating soft tissue deformation includes a soft tissue physics module 110, a collision detection module 120, a mesh deformation module 130, an instrument modeling module 140, a puncture operation module 150, a cutting operation module 160, a force feedback acquisition module 170, a process auxiliary module 180 and a visualization module 190.

[0038] Specifically, the soft tissue physics module 110 includes a geometric modeling unit 111 and a physical modeling unit 112. The geometric modeling unit 111 is used to obtain medical digital image communication data of a virtual object, and perform threshold screening, interactive segmentation and construction based on the scanned image data to obtain a three-dimensional soft tissue model, and obtain a mesh of the three-dimensional soft tissue model. Among them, the virtual object refers to a virtual human body, which is the same below. Medical digital image communication data (DICOM data) can be CT (computed tomography, electronic computer tomography) data, or MRI (magnetic resonance imaging, i.e. magnetic resonance imaging) data. It should be noted that the three-dimensional soft tissue model can be generated by CT data or MRI data. The three-dimensional bone tissue model can only be generated by CT data. . In this embodiment, the CT data can be spiral CT data, i.e. spiral computed tomography. The target object is photographed from the top of the skull to the lower neck area by setting the layer thickness to 1.25 mm to obtain relevant data and store it as DICOM format data. The three-dimensional soft tissue model refers to a facial model of the human body. Of course, in other embodiments, it can also refer to models of other parts of the human body.

[0039] The geometric modeling unit 111 can be based on the MITK (medical imaging toolkit) class library, according to the different gray value ranges of bone tissue and soft tissue in the obtained image data, perform threshold screening to preliminarily segment the bone tissue and soft tissue parts in the electronic computer tomography image. Then, interactive segmentation is performed to erase and fill the parts of the bone tissue or soft tissue in the image data that do not conform to the actual anatomical structure. Finally, the MC algorithm (marching cube, an algorithm for extracting isosurfaces in a three-dimensional discrete data field) can be used to construct a three-dimensional soft tissue model of the face (which is a surface model). In the process of constructing the three-dimensional soft tissue model of the face, a triangular face mesh can be used to generate a three-dimensional soft tissue model of the craniofacial region with the highest quality, and output a three-dimensional STL format file to prepare for subsequent physical modeling. It should be noted that in this embodiment, the three-dimensional soft tissue model is composed of multiple meshes, each of which is a triangular face. Of course, in other embodiments, each mesh can also be a quadrilateral face, not limited to this.

[0040] In some embodiments, the geometric modeling unit 111 is also used to divide the mesh of the three-dimensional soft tissue model of the face, that is, to optimize the density of the triangular facets. For example, according to the preset target, the surgical area and the non-surgical area of ​​the three-dimensional soft tissue model are determined. The preset target can be the forehead area, the masseter area, and the chin area, but is not limited thereto. For example, if the preset target is the masseter area, the mesh of the surgical area can be incrementally refined, and the mesh of the non-surgical area can be meshed reduced and optimized, thereby ensuring the real-time calculation and the accuracy of the mesh deformation described below.

[0041] The physical modeling unit 112 is used to set boundary conditions for the three-dimensional soft tissue model to obtain virtual partitions; obtain mechanical properties of different partitions; and fit the mechanical properties of different partitions to the corresponding partitions of the three-dimensional soft tissue model to obtain a three-dimensional soft tissue physical model. It should be noted that the above mechanical properties refer to real mechanical properties, that is, mechanical property information obtained during actual surgical operations.

[0042] Specifically, soft tissue includes a variety of anatomical structures such as skin, muscles, mucosa, glands, etc., with rich layers, and the biomechanical characteristics of different anatomical structures and different layers of tissues are different. If all are modeled one by one, the resulting calculation scale will be very large, and it is difficult to meet the real-time performance of the device used to simulate soft tissue deformation. Therefore, in the embodiment of the present application, the soft tissue structure is first simplified to a single layer of soft tissue, and boundary conditions are set for the single layer of soft tissue to obtain a virtual partition, that is, to simulate the real anatomical partition. Since the segmented face has a certain thickness of soft tissue, including skin, muscle, etc., the specific method of simplifying it to a single layer of soft tissue is to treat the entire area of ​​the three-dimensional soft tissue model as skin, and the parameters added thereafter are only skin parameters.

[0043] The above boundary conditions can be used to set the division rules in the algorithm through the coordinate values ​​of the world coordinate system, wherein the division rules are to make the segmented area contain all the required anatomical structures as much as possible and not contain unnecessary anatomical structures (such as bone tissue). The boundary conditions define anatomical partitions for the three-dimensional soft tissue model, and different mechanical properties can be set for different partitions, such as the fitting of the cubic polynomial of the puncture force equation, or the definition of the cutting feedback force (Y, Z axis) parameters.

[0044] In addition, the physical modeling unit 112 is also used to assign values ​​to the material properties of the three-dimensional soft tissue model and to lock the deformation state of the soft tissue.

[0045] Continue reading Figure 1As shown, the instrument modeling module 140 is used to obtain laser scanning image data of a virtual surgical instrument; based on the laser scanning image data of the virtual surgical instrument, render a three-dimensional virtual surgical instrument model, such as a virtual scalpel. As described above, the virtual surgical instrument and the three-dimensional soft tissue model are both composed of triangular facets. The mesh formed by the triangular facets can be called a triangular mesh, which is the same below.

[0046] The collision detection module 120 is used to determine whether a collision occurs between the virtual surgical instrument and the three-dimensional soft tissue physical model; when a collision is determined to occur, the position information and depth information of the collision node are obtained.

[0047] Specifically, when the virtual surgical instrument performs a puncture or cutting operation, the collision detection module 120 detects the collision node where the virtual surgical instrument collides with the three-dimensional soft tissue physical model in the frame loop to obtain the position information and depth information of the collision node, thereby obtaining the input conditions required for the soft tissue mesh deformation and force feedback. It should be noted that the frame loop refers to the real-time changes of the mesh of the three-dimensional soft tissue physical model displayed through each frame of the display device. In some embodiments, collision detection includes a rough detection stage and a detailed detection stage, wherein the detailed detection stage includes two levels: a gradual refinement layer and a precise intersection layer, and adopts a method based on an axis-aligned bounding box structure.

[0048] The mesh deformation module 130 is used to calculate the influence area of ​​the three-dimensional soft tissue physical model based on the position information of the collision node; establish a stiffness matrix for all triangular units in the influence area; input the depth information of the collision node into the stiffness matrix to calculate the displacement of the remaining nodes in the influence area; and render the mesh deformation of the influence area based on the displacement of all nodes in the influence area.

[0049] Specifically, the mesh deformation module 130 can calculate the impact area of ​​the three-dimensional soft tissue physical model based on the position information of the collision node. If a node on the triangular mesh is displaced and the displacement is a fixed amount, or the node is subjected to an external force and causes the displacement of other nodes to be less than a preset error, then the node is regarded as affecting other nodes, and the area composed of the affected nodes is defined as the impact area. At this time, the size of the impact area can be characterized by minN. It should be noted that the distance between two nodes on the triangular mesh is:

[0050] l min =min(l v1,v2 )=min(ΣE i )

[0051] In the above formula, v1,v2Represents the edge distance between two nodes, that is, the shortest edge distance among the multiple edges from node V1 to node V2. ΣE i Represents the sum of the lengths of multiple edges, where i ≥ 1.

[0052] When the triangle shape of the triangular mesh is not distorted and the side lengths are not different, the above l min It can be expressed as the sum of the lengths of the minimum number of edges between node V1 and node V2, that is,

[0053] Where minN is the minimum number of edges between node V1 and node V2.

[0054] As mentioned above, the size of the impact area can be represented by minN. Figure 2 As shown in , if minN = 2, it means that the node only affects the area composed of nodes that are directly connected to the node and connected to it by two edges. Figure 3 As shown, the influence area of ​​node V0 is DI, and minN=3 at this time.

[0055] The mesh deformation module 130 includes an influence area calculation unit, which is used to define (i.e., limit) the range of establishing the stiffness matrix and obtain calculation parameters of the influence area, wherein the calculation parameters include a preset displacement error, the elastic modulus of the material of the three-dimensional soft tissue physical model, and the surface skin thickness of the three-dimensional soft tissue physical model; based on the calculation parameters, the influence area of ​​the three-dimensional soft tissue physical model is calculated. Therefore, according to the position information of the collision node (such as point coordinates) and through the influence area calculation unit, the influence area can be obtained. It should be noted that the influence area DI can be expressed by the following formula:

[0056] DI=f di (ε,E m , d), where ε represents the preset displacement error, E m represents the elastic modulus of the material of the three-dimensional soft tissue physical model, and d represents the surface skin thickness of the three-dimensional soft tissue physical model.

[0057] It can be seen from the above formula that when the displacement or external force is larger, the affected area is larger; the larger the elastic modulus is, the larger the affected area is.

[0058] Furthermore, after obtaining the influence area, a stiffness matrix can be established for all triangular elements in the influence area, and based on the depth information of the collision node (ie, the displacement of the connection point described below), the displacement of all nodes in the influence area can be calculated.

[0059] In this embodiment, the mesh deformation module 130 includes a displacement calculation unit, which is used to construct a triangular unit stiffness matrix; based on the depth information, the input of the stiffness matrix is ​​obtained, and based on the input of the stiffness matrix, the stiffness matrix is ​​solved to obtain the displacement of all nodes in the affected area. Further, each triangular unit has a stiffness matrix, and the solution of the stiffness matrix of a triangular unit (such as the displacement of the remaining two vertices of the triangle) can be used as the input of the stiffness matrix of the adjacent triangular unit, so that the displacement of all nodes in the affected area can be obtained by calculation.

[0060] It should be noted that the stiffness matrix of the triangular element is defined as K e , which represents the deformation resistance K of a single triangular unit e =λDKDK.

[0061] The triangular mesh S is composed of n triangular elements, so the stiffness matrix K of the triangular mesh S in three-dimensional space is s The stiffness matrices of n triangular units form a 3n-order matrix. In other words, a stiffness matrix is ​​established for each triangular unit. The stiffness matrix of the entire triangular mesh is obtained by superimposing the stiffness matrices of all the triangular facets that make up the mesh. Since it is in three-dimensional space, a third-order matrix is ​​formed.

[0062] Assume that there is a set of connection points in the triangular mesh S and the triangular mesh S', and the connection points have an initial displacement (for example, displacement occurs due to external force. Specifically, the virtual surgical instrument collides with the three-dimensional soft tissue physical model due to force, causing the three meshes in the three-dimensional soft tissue physical model to be displaced). The initial displacement is converted into a geometric constraint to solve the deformation of the two triangular meshes respectively. According to the properties of the connection points, the actual forces acting on the connection points on the two triangular meshes under the initial displacement can be calculated respectively. If the calculated forces are of the same magnitude and opposite directions, the initial displacement of the above connection points is the correct coordinated deformation position, and the calculation ends. Otherwise, according to the magnitude and direction of the calculated resultant force, the position value is adjusted (that is, the position value of the connection point is adjusted) to obtain a new displacement vector, and the process is repeated for iterative solution.

[0063] Specifically, suppose there is a set of connection points in the triangle mesh S and the triangle mesh S′, where the nodes corresponding to the connection points in the triangle mesh S are {V} = {V1, V2, ..., V n}, the corresponding nodes in the triangular mesh S′ are {V}′={V′1,V′2,...,V′ n When establishing constraints (i.e., deformation constraints, the influence between nodes), the initial displacement value of each node is taken as the midpoint of the corresponding node in S and S′ as the initial displacement value, that is,

[0064] {V1,V2,...,V n The initial displacement value of} is:

[0065]

[0066] {V′1,V′2,...,V′ n The initial displacement value of} is:

[0067]

[0068] Using {δ} i and {δ}′ i Iterate the value of to determine whether the magnitude of the force at the connection point satisfies the following formula:

[0069]

[0070] Where ε is the convergence error.

[0071] If this condition is met, the current displacement value {δ} i and {δ}′ i is the actual displacement value of the node. Otherwise, calculate the iterative displacement value of the relevant node in S and S′ as follows:

[0072]

[0073] Where K s is the stiffness matrix at the relevant nodes, and calculates {F} i+1 and {F}′ i+1 , and continue the iterative process until the iterative error judgment condition is met.

[0074] The network deformation module is also used to render the mesh deformation of the affected area through the visualization module 190 after obtaining the displacement of all nodes in the affected area.

[0075] Continue reading Figure 1 and Figure 4As shown in (b), (c) and (d) in the figure, the puncture operation module 150 is used to control the virtual surgical instrument to move toward the three-dimensional soft tissue physical model; after the collision detection module 120 determines that a collision occurs, the feedback force of the collision node is calculated based on the depth information of the collision node; it is determined whether the feedback force is greater than the preset maximum puncture strength; when it is determined that the feedback force is less than the maximum puncture strength, it is determined that the grid of the three-dimensional soft tissue physical model is not destroyed; when it is determined that the feedback force is greater than the maximum puncture strength, it is determined that the grid of the three-dimensional soft tissue physical model is punctured; the inclination angle of the virtual surgical instrument is determined; when it is determined that the inclination angle of the virtual surgical instrument is within the first preset angle range, it is determined that the operation of the virtual surgical instrument on the three-dimensional soft tissue physical model is a puncture operation, wherein the first preset angle range is 90°±5°.

[0076] In this embodiment, before performing the puncture operation, the process auxiliary module 180 can be called to simulate the auxiliary process when performing virtual surgery. The auxiliary process may include performing punctuation operations to form a virtual surgery path plan, such as Figure 4 By using the process auxiliary module 180, the simulation system can be further improved to provide strong support for surgical plan design, postoperative shape prediction, and operation training.

[0077] The force feedback acquisition module is used to obtain the feedback force of the collision node, and output it to the force feedback device based on the obtained feedback force. The force feedback device can simulate the corresponding force, vibration or passive movement to transmit it to the user. This stimulation can help the user feel the objects in the virtual environment from the sense of touch, directly feel the interaction between the force and the model, thereby increasing the sense of immersion. The force feedback device can be connected to electronic devices such as computers through a network cable interface, and the open source OpenHaptics driver toolkit is installed. After debugging the parameters, the force feedback device can be used to perform proxy rendering of the calculated feedback force and display it through a display device.

[0078] like Figure 4 As shown in (e), the cutting operation module 160 is used to determine that the operation of the virtual surgical instrument on the three-dimensional soft tissue physical model is a cutting operation when the inclination angle of the virtual surgical instrument is determined to be within a second preset angle range, wherein the second preset angle range is 45°±5°.

[0079] At this time, the mesh deformation module 130 is also used to determine whether a collision occurs between the virtual surgical instrument and the three-dimensional soft tissue physical model through the collision detection module 120 in a frame loop when performing a cutting operation. When a collision is determined to occur, the position information and depth information of the collision node are obtained, and the constraint relationship of the mesh of the cut part of the three-dimensional soft tissue physical model is continuously deconstructed, the cut mesh of the three-dimensional soft tissue physical model is reconstructed, and the color and texture of the cut mesh are updated.

[0080] In this embodiment, the apparatus for simulating soft tissue deformation further includes a visualization module 190, which is used to display the mesh deformation of the affected area, and to display whether the mesh of the three-dimensional soft tissue physical model is destroyed, and to display the color and texture of the mesh of the three-dimensional soft tissue physical model after cutting. In this way, after the cutting operation is performed, the triangular mesh of the cut part in the three-dimensional soft tissue physical model is rendered with different colors and materials, so that the cutting effect is more in line with the actual situation, which is convenient for users to observe, such as Figure 5A , Figure 5B , Figure 5C and Figure 5D Furthermore, the visualization module 190 can set the color, lighting, and material properties of the three-dimensional soft tissue model through the OPENGL tool library, and update the network nodes of the soft tissue in the deformation area in real time, so that the user can have a visual interaction with the device for simulating soft tissue deformation through a display device (such as a display, a projection device, etc.), thereby improving the user experience.

[0081] In addition, through the cooperation of the process auxiliary module 180, the grid deformation module 130, the cutting operation module 160 and the visualization module 190, an arc-shaped incision can be realized, thereby achieving the effect of a free cutting path.

[0082] Based on the same inventive concept, the embodiment of the present application provides a simulation system 1000, such as Figure 6 As shown, the simulation system 1000 includes the device 100 for simulating soft tissue deformation described in any embodiment of the present application, a display device 300, a force feedback device 200, and an auxiliary input device 400. The display device 300 is connected to the visualization module 190 of the device 100 for simulating soft tissue deformation, and is used for visual interaction with the visualization module 190; the force feedback device 200 is connected to the force feedback acquisition module 170, and is used for tactile interaction with the force feedback acquisition module 170; the auxiliary input device 400 is connected to the process auxiliary module 180, and is used for auxiliary input interaction with the process auxiliary module 180.

[0083] The embodiments of the present application provide a device and a simulation system for simulating soft tissue deformation, which achieve a good balance between the real-time performance and rendering authenticity of the simulation system through local deformation algorithms and surgical area mesh optimization. The device described in the present application can simulate the puncture and cutting of any area of ​​soft tissue, thereby having the effect of free puncture sites and cutting paths, and quickly simulates the soft tissue deformation process caused by virtual surgery to achieve realistic visual rendering and mechanical feedback to meet the needs of virtual surgery. In addition, by repeatedly using the device, not only can the cost of surgical training be reduced, but the success rate of surgery can also be improved.

[0084] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0085] The above is a detailed introduction to a device and a simulation system for simulating soft tissue deformation provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the technical solution and its core idea of ​​the present application. Ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A device for simulating soft tissue deformation, characterized in that: The device for simulating soft tissue deformation comprises: A soft tissue physics module, the soft tissue physics module includes a geometric modeling unit and a physical modeling unit; the geometric modeling unit is used to obtain medical digital image communication data of a virtual object, and perform threshold screening, interactive segmentation and construction based on the medical digital image communication data to obtain a three-dimensional soft tissue model, and obtain a grid structure of the three-dimensional soft tissue model; the physical modeling unit is used to simplify the three-dimensional soft tissue model into a single layer of soft tissue, and set boundary conditions for the single layer of soft tissue to obtain virtual partitions; and fit the mechanical properties of different partitions to the corresponding partitions of the three-dimensional soft tissue model to obtain a three-dimensional soft tissue physical model; wherein the boundary condition is to set a partitioning rule in the algorithm through the coordinate value of the world coordinate system, and the partitioning rule is to try to make the segmented area contain all required anatomical structures and not contain unnecessary anatomical structures; The collision detection module is used to determine whether a collision occurs between the virtual surgical instrument and the three-dimensional soft tissue physical model; when a collision is determined to occur, the position information and depth information of the collision node are obtained; A mesh deformation module is used to calculate the influence area of ​​the three-dimensional soft tissue physical model based on the position information of the collision node; establish a stiffness matrix for all triangular units in the influence area; input the depth information of the collision node into the stiffness matrix to calculate the displacement of the remaining nodes in the influence area; and render the mesh deformation of the influence area based on the displacement of all nodes in the influence area.

2. The device for simulating soft tissue deformation according to claim 1, characterized in that: The geometric modeling unit is also used to determine the surgical area and non-surgical area of ​​the three-dimensional soft tissue model according to a preset target, and to increment the mesh of the surgical area and decrement the mesh of the non-surgical area according to a preset rule.

3. The device for simulating soft tissue deformation according to claim 1, characterized in that: The device for simulating soft tissue deformation also includes: an instrument modeling module, which is used to obtain laser scanning image data of a virtual surgical instrument; and render a three-dimensional virtual surgical instrument model based on the laser scanning image data of the virtual surgical instrument.

4. The device for simulating soft tissue deformation according to claim 3, characterized in that: The virtual surgical instrument and the three-dimensional soft tissue model are both composed of triangular facets.

5. The device for simulating soft tissue deformation according to claim 1, characterized in that: The device for simulating soft tissue deformation also includes a puncture operation module and a force feedback acquisition module; the puncture operation module is used to control the virtual surgical instrument to move toward the three-dimensional soft tissue physical model; after the collision detection module determines that a collision occurs, the feedback force of the collision node is calculated based on the depth information of the collision node; Determine whether the feedback force is greater than a preset maximum puncture strength; When it is determined that the feedback force is less than the maximum puncture strength, it is determined that the mesh of the three-dimensional soft tissue physical model is not destroyed; When it is determined that the feedback force is greater than the maximum puncture strength, it is determined that the grid of the three-dimensional soft tissue physical model is punctured; the inclination angle of the virtual surgical instrument is determined; when it is determined that the inclination angle of the virtual surgical instrument is within the first preset angle range, it is determined that the operation of the virtual surgical instrument on the three-dimensional soft tissue physical model is a puncture operation; the force feedback acquisition module is used to obtain the feedback force of the collision node, and output it to the force feedback device based on the obtained feedback force.

6. The device for simulating soft tissue deformation according to claim 5, characterized in that: The device for simulating soft tissue deformation also includes a cutting operation module; the cutting operation module is used to determine that the operation of the virtual surgical instrument on the three-dimensional soft tissue physical model is a cutting operation when the inclination angle of the virtual surgical instrument is determined to be within the second preset angle range; the mesh deformation module is also used to deconstruct the constraint relationship of the mesh of the cut part of the three-dimensional soft tissue physical model; reconstruct the cut mesh of the three-dimensional soft tissue physical model; and update the color and texture of the cut mesh.

7. The device for simulating soft tissue deformation according to claim 1, characterized in that: The mesh deformation module includes an influence area calculation unit, which is used to define the range of establishing the stiffness matrix and obtain calculation parameters of the influence area, wherein the calculation parameters include a preset displacement error, the elastic modulus of the material of the three-dimensional soft tissue physical model, and the surface skin thickness of the three-dimensional soft tissue physical model; based on the calculation parameters, the influence area of ​​the three-dimensional soft tissue physical model is calculated.

8. The device for simulating soft tissue deformation according to claim 1, characterized in that: The mesh deformation module includes a displacement calculation unit, which is used to construct a triangular unit stiffness matrix; based on the depth information, an input of the stiffness matrix is ​​obtained; Based on the input of the stiffness matrix, the stiffness matrix is ​​solved to obtain the displacements of all nodes in the affected area.

9. The device for simulating soft tissue deformation according to claim 1, characterized in that: The device for simulating soft tissue deformation also includes a process auxiliary module, which is used to simulate auxiliary processes when performing virtual surgery; wherein the auxiliary process includes performing punctuation operations to form a virtual surgery path planning.

10. The device for simulating soft tissue deformation according to claim 1, characterized in that: The device for simulating soft tissue deformation also includes a visualization module, which is used to display the mesh deformation of the affected area, display whether the mesh of the three-dimensional soft tissue physical model is destroyed, and display the color and texture of the mesh of the three-dimensional soft tissue physical model after cutting.

11. A simulation system, characterized in that: The simulation system comprises the device for simulating soft tissue deformation according to any one of claims 1 to 10, a display device, a force feedback device and an auxiliary input device; wherein the display device is connected to the visualization module of the device for simulating soft tissue deformation, and is used for visual interaction with the visualization module; The force feedback device is connected to the force feedback acquisition module and is used for performing tactile interaction with the force feedback acquisition module; the auxiliary input device is connected to the process auxiliary module and is used for performing auxiliary input interaction with the process auxiliary module.

Images