Stomach soft tissue modeling and collision detection method in virtual operation

Abstract

The invention discloses a stomach soft tissue modeling and collision detection method in a virtual operation, and belongs to the field of soft tissue modeling in the virtual operation. Important indexes of whether the virtual operation system can reach the operation training effect or not comprise the real-time performance, stability and accuracy of physical modeling and collision detection. According to the method, the biomechanical characteristics of stomach soft tissue under the action of external force are analyzed, an improved deformation model based on a reset spring is provided based on a soft tissue deformation method of a mass point-spring deformation model, the problems of super-elasticity and collapse of a traditional mass point-spring model are solved, and the authenticity of soft tissue deformation is improved. Meanwhile, for a muscle layer, modeling is carried out by using a collision detection algorithm combining an axial alignment bounding box (AABB) and a spherical bounding box (Sphere); modeling of a mucous membrane layer and other soft tissues of the stomach is realized by adopting a mixed hierarchical algorithm of an AABB (Axial Alignment Burst Box) and a Sphere (Sphere); and the real-time performance and the accuracy of collision detection are considered.

Description

Technical Field

[0001] The present invention relates to the field of soft tissue modeling in virtual surgery, and in particular to a method for modeling and collision detection of stomach soft tissue in virtual surgery. Background Art

[0002] At present, the main treatments for gastric cancer are subtotal gastrectomy and total gastrectomy. The choice of surgical methods and techniques has a crucial impact on the survival and quality of life of patients. Virtual surgical technology can accurately simulate and operate gastric surgery, which is very necessary to improve the success rate of surgery and the recovery rate of patients. Completing the core tasks such as intersection testing of virtual surgical instruments and human soft tissues, deformation calculation, and tactile feedback force calculation within the limited time of the virtual surgical system is of great significance to promoting the development of virtual surgical systems. It can not only optimize surgical plans and improve the success rate of surgery, but also promote the development of medicine and education.

[0003] Soft tissue modeling is the core of virtual surgery. The mass-spring model was first proposed in 1987 and has become one of the most widely used soft body deformation modeling methods, especially in the deformation simulation of surface models. The core idea of ​​the mass-spring model is to discretize the surface model of soft tissue into mass points with a certain mass, and then connect the geometrically connected mass points through virtual spring elements according to the geometric topological relationship of the model.

[0004] In existing studies on modeling of gastric soft tissue, a mass-spring model is constructed by using bending springs and structural springs to simulate the soft tissue deformation process. This will cause the model to be unable to return to its original state under the action of a specific spring force, resulting in the "collapse" problem of the gastric soft tissue model. In addition, when the external force acting on the gastric soft tissue model increases to a certain extent, it may also cause the "superelasticity" problem. However, the real gastric soft tissue is composed of a mucosal layer, a muscular layer, and a serosa layer, which can effectively expand and contract during the digestion process, giving the stomach a certain elasticity and toughness. Therefore, "superelasticity" and "collapse" are impossible in the real surgical environment of the stomach. In addition, soft tissue models are usually of high complexity and sophistication. The existing collision detection algorithms for virtual surgery mostly use a single high-precision collision detection algorithm, which leads to unnecessary computational overhead.

[0005] The Chinese patent application, publication number CN 110610039 A, publication date December 24, 2019, records a method for simulating soft tissue deformation of a particle spring model. The finite element method is used to improve the parameter setting of the particle spring model, including the initial parameter setting of the particle spring model, the determination of the spring parameters of the particle spring model during the soft tissue deformation process, and the parameterization of the various anisotropic characteristics of the soft tissue. The accuracy of the particle spring model is improved, and a hybrid bounding box algorithm combining the axis-aligned bounding box AABB and the directional bounding box OBB is used for collision detection. The improved collision detection algorithm improves the interactivity and real-time performance of the model. Although the method has improved the accuracy through parameter adjustment and algorithm optimization, it still does not solve the problem of reduced simulation accuracy of the particle spring model under conditions such as extreme deformation or high-speed collision. In addition, the invention uniformly adopts a hybrid bounding box algorithm for the stomach, and does not further refine the design of the collision detection algorithm to distinguish the characteristics of different soft tissues such as the muscle layer and mucosa layer of the stomach. Summary of the invention

[0006] 1. Technical problems to be solved

[0007] The technical problem to be solved by the patent of the present invention is: it is necessary to complete the physical modeling in virtual surgery and design a collision detection algorithm according to the characteristics of the soft tissue of the stomach, so as to solve the problems of "collapse" and "hardening" in the deformation recovery process of the soft tissue simulated by the mass-spring model, and the problem of reducing the amount of calculation while maintaining the accuracy of the collision detection link. The present invention provides an improved mass-spring model. On the basis of the traditional soft tissue model, the concept of a reset spring is introduced to model the soft tissue of the stomach more accurately, so that the deformation effect of the improved soft tissue model under the action of external force is more realistic, thereby more accurately simulating the mechanical behavior of biological tissue. The collision detection algorithm is designed according to the different characteristics of the gastric soft tissue base layer, mucosal layer and other soft tissue structures of the stomach, and different mixed-level encirclement collision detection algorithms are adopted to maintain the accuracy of collision detection while reducing the computational overhead.

[0008] 2. Technical solution

[0009] The purpose of the present invention is achieved through the following technical solutions.

[0010] The first aspect of the present invention provides a method for modeling and collision detection of soft tissue of the stomach in virtual surgery, the method steps are as follows:

[0011] Step S100, obtaining original images of various soft tissues of the stomach from a medical image set and preprocessing the original images, specifically, including image reading, image denoising, image resampling, adjusting the contrast and brightness of the resampled images, and image segmentation operations, to obtain preprocessed images with uniform resolution and size.

[0012] Step S200, using a threshold-based segmentation algorithm to process image data in the preprocessed image to identify and segment different soft tissue areas of the stomach and save the data. Specifically, the stomach soft tissue is divided into three categories: muscle layer, mucosal layer and other soft tissues, and the three categories of soft tissues are classified and marked, and the shape and size data of the three categories of soft tissues are saved.

[0013] Preferably, in step S200, a threshold-based segmentation algorithm is used to process the image data to extract three types of soft tissue areas of the stomach and save the relevant data.

[0014] Step S300, based on the shape and size data of the three types of stomach soft tissues segmented in step S200, combined with the different nonlinear and viscoelastic mechanical properties corresponding to the three types of soft tissues, a mass-spring model of the stomach soft tissue composed of multiple mass points and springs is constructed. The mass-spring model includes a structural spring, a bending spring, and a return spring. The elastic coefficient, damping coefficient and other parameters of the spring are set according to the biomechanical properties of the soft tissue. All parameter combinations form internal and external constraints; the constructed mass-spring model is visually rendered.

[0015] The specific process of step S300 includes the following sub-steps:

[0016] Sub-step S310, based on the shape and size data of the three types of stomach soft tissues segmented in step S200, and in combination with the different nonlinear and viscoelastic mechanical properties corresponding to the three types of soft tissues, a three-dimensional mesh model of the soft tissues is constructed using three-dimensional reconstruction technology.

[0017] Sub-step S320, using a mesh simplification algorithm to perform surface reduction processing on the three-dimensional mesh model. Preferably, the number of triangular faces on the surface of the model after surface reduction processing is in the tens of thousands.

[0018] Sub-step S330, for the triangular patch structure of the outer surface of the model, the surface vertices of the model are used as mass points to construct a mass-spring model. The mass-spring model includes structural springs and bending springs, and each mass point is assigned a reset spring. One end of the reset spring is connected to the current position of the mass point, and the other end is connected to the initial position of the mass point without deformation. The original length of the reset spring is set to 0. The elastic coefficient and damping coefficient of the spring are set according to the biomechanical properties of the soft tissue, and the combination of all parameters forms internal and external constraints.

[0019] Specifically, the detailed process of sub-step S330 is as follows:

[0020] First, a soft tissue foundation framework based on the mass-spring model is constructed.

[0021] Assume that the surface of the soft body deformation model contains n vertices, then these n vertices constitute the discrete mass points in the mass-spring model. i According to the topological relationship between the vertices in the soft body deformation model surface, the discrete mass points are connected by spring elements. If mass point i and mass point j are discrete points in the mass spring model, the spring element between mass point i and mass point j can be represented by L ij (i, j∈[0, n-1]). Taking particle i at time t as an example, we analyze its motion. According to Newton's second law of motion, we must first find the resultant force F on particle i. i , F i It can be obtained by synthesizing the internal force (elastic force and damping force of the spring connected to it) and external force (gravity and force applied by the outside world) on particle i:

[0022]

[0023] Among them, F i represents the net force acting on particle i, represents the internal force on particle i, represents the external force on particle i;

[0024] Assuming that the mass point adjacent to i is j, the resultant force of the elastic force and damping force applied by j to i is:

[0025] F ij =K s (|X ij |-l ij )+K d (V j -V i )

[0026] Among them, F ij K represents the resultant force of the elastic force and damping force applied to i by the mass point j adjacent to i. s represents the spring stiffness coefficient, X ij represents the displacement difference between particles i and j, l ij Indicates the original length of the spring, K d represents the damping coefficient of the spring, V j represents the velocity of particle j, V i represents the velocity of particle i;

[0027] Can be obtained:

[0028]

[0029] Sub-step S340, applying the Delaunay algorithm to tetrahedron the model after the face reduction process to obtain a more refined model structure. According to the result of the tetrahedron division, a filling volume structure is constructed at the vertex of each tetrahedron.

[0030] Sub-step S350, using OpenGL to perform visual rendering on the constructed mass-spring model.

[0031] Step S400, on the mass point-spring model of the stomach soft tissue constructed in step S300, according to the data of different tissues obtained in step S200 and the biological characteristics of different stomach soft tissues, different hybrid hierarchical bounding boxes are set for the soft tissues using a hybrid hierarchical bounding box collision detection algorithm, a hybrid hierarchical bounding box tree is constructed from top to bottom, and different hybrid hierarchical bounding box collision detection algorithms are adopted for different soft tissues. For the muscle layer, a collision detection algorithm combining a spherical bounding box and a directed bounding box is adopted for modeling; for the mucosal layer and other stomach soft tissues, a hybrid hierarchical algorithm of an axis-aligned bounding box and a spherical bounding box is adopted to achieve modeling of the mucosal layer and other stomach soft tissues, and real-time detection and collision response are performed.

[0032] Furthermore, the specific method of constructing a mixed hierarchical bounding tree from top to bottom is to divide the gastric soft tissue into several subsets, construct a specific bounding volume for each of the divided subsets, and then recursively perform similar operations on each subset until the current subset contains only one vertex.

[0033] Furthermore, according to the structural characteristics of different soft tissues of the stomach, different hybrid hierarchical bounding box collision detection algorithms are adopted: a hybrid hierarchical bounding box collision detection algorithm combining a spherical bounding box and a directed bounding box is used for the muscle layer; and a hybrid hierarchical bounding box collision detection algorithm combining an axis-aligned bounding box and a spherical bounding box is used for the mucosal layer and other soft tissues.

[0034] Step S500, the external force on the model is calculated based on the depth information between the surgical instrument and the soft tissue model. When a collision occurs, the local bounding box where the collision point is located is first determined, and the bounding box of the parent node is updated layer by layer until the root node is updated. Specifically, the position of the model surface vertex corresponding to each bounding box is updated to complete the deformation simulation of the soft tissue under force.

[0035] After introducing the return spring, we use the implicit Euler algorithm to solve the relationship between displacement, velocity and time as follows:

[0036]

[0037] Among them, V t represents the velocity of the particle at time t, represents the velocity of the particle at t0, represents the integral of the ratio of the resultant force on the particle and the mass of the particle during the period between t0 and t, X t represents the displacement of the particle at time t, Indicates that the particle is at t o The displacement of time, It represents the integral of the particle's velocity during the period between t0 and t.

[0038] The above equations are solved using the finite difference method to approximate their derivatives:

[0039]

[0040] in, Represents the first-order derivative of the particle velocity with respect to time, V t+1 represents the velocity of the particle at time t+1, V t represents the velocity of the particle at time t, Δt represents the time step, O(Δt 2 ) represents the remainder when the time step approaches 0, represents the first-order derivative of the particle displacement with respect to time, X t+1 represents the displacement of the particle at time t+1, X t represents the displacement of the particle at time t;

[0041] The velocity and position of the particle at the next time step can be obtained from the above integral formula:

[0042]

[0043] Among them, V t+1 represents the velocity of the particle at time t+1, V t represents the velocity of the particle at time t, Δt represents the time step, F(t) represents the resultant force on the particle at time t, m represents the mass of the particle, and X t+1 represents the displacement of the particle at time t+1, X t represents the displacement of the particle at time t;

[0044] The improved deformation differential equation of the mass-spring system is:

[0045]

[0046] in, It represents the first derivative of the particle displacement with respect to time; represents the first-order derivative of the particle's velocity with respect to time; V represents the particle's velocity; F represents the resultant force on the particle, M represents the mass matrix of the particle, and the expression of F can be written as follows:

[0047]

[0048] in, represents the internal force on particle i, represents the external force on particle i, F s Represents the elastic force of the mass spring, F d represents the damping force of the mass spring, K represents the stiffness matrix of the system, X represents the displacement of the mass, D represents the damping matrix of the system, and X represents the first-order derivative of the displacement of the mass with respect to time;

[0049] From this we can get:

[0050]

[0051] Where M represents the mass matrix of the particle, Representing the second-order derivative of the particle displacement with respect to time, we get the following equation:

[0052]

[0053] Among them, m i represents the mass of particle i, a represents the acceleration of particle i during its motion, F s Represents the elastic force of the mass spring, F d represents the damping force of the mass spring;

[0054] The above formula can be used to obtain the point differential equation of the surface point-spring model of the biological soft tissue deformation model as follows:

[0055]

[0056] in, represents the first-order derivative of the particle displacement with respect to time, V represents the velocity of the particle, represents the first-order derivative of the velocity of the particle with respect to time, and m represents the mass of the particle;

[0057] The springs between the surface mass points of the biological soft tissue simulation model and the filling body model are constructed, and the mapping relationship between the surface mass points of the biological soft tissue deformation simulation model and the filling body model is established.

[0058] A return spring is added to the traditional spring model, so that all the mass points on the surface of the biological soft tissue deformation simulation model are connected to the center of the closest filling body model through an independent spring. When the filling body model moves or rotates, the mass spring model on the surface of the biological soft tissue deformation simulation model will follow it to make corresponding movements. The movements of all filling body models in the biological soft tissue deformation simulation model satisfy the following dynamic differential equations:

[0059]

[0060] Among them, m i represents the mass of particle i, represents the second-order derivative of the displacement of particle i with respect to time, Represents the external force on particle i. According to Newton's second law, the formula is:

[0061]

[0062] in, represents the first-order derivative of the displacement of particle i with respect to time, V i represents the velocity of particle i, represents the first-order derivative of the velocity of particle i with respect to time, represents the second-order derivative of the displacement of particle i with respect to time, m i is the mass of particle i.

[0063] Step S600, after the deformation calculation is completed, the graphics of the virtual surgical scene are refreshed to display the new shape and position of the soft tissue model; and the feedback force of the model is input into the tactile device.

[0064] Step S700, verifying and optimizing the generated soft tissue model, wherein the verification operation includes model accuracy evaluation and comparison verification with actual soft tissue.

[0065] Preferably, the verification and optimization of the generated soft tissue model includes: comparing the consistency of the soft tissue physical properties predicted by the model with the actual soft tissue measurement values ​​in key parameters; performing mechanical property tests on the actual soft tissue and comparing the test results with the model prediction results to evaluate the accuracy of the model in mechanical behavior; adjusting the model parameters according to the evaluation results to improve the accuracy and stability of the model.

[0066] 3. Beneficial effects

[0067] Compared with the prior art, the advantages of the present invention are:

[0068] 1. The present invention can improve the spring-mass model by adding a reset spring, and discretize the complex soft tissue structure into a series of mass points and springs connecting these mass points. This discretization method greatly simplifies the complexity of the model, making the calculation process that originally required continuous parameterization simpler and more direct. Improve the real-time and interactivity of the model. Provide an important basis for the optimization of surgical models. The main contributions are: combining biological principles and repair mechanisms, introducing the concept of a reset spring, which can efficiently simulate some extreme soft tissue deformation conditions and improve the precision and accuracy of the model.

[0069] 2. The present invention conducts in-depth research on different soft tissues of the stomach, combines their different physiological characteristics, and adopts a hybrid hierarchical collision detection algorithm, which can reduce the number of precise detections and repeated updates, effectively reduce the amount of calculation in the soft tissue modeling process in virtual surgery, and improve the calculation efficiency of the model, thereby improving the real-time and interactivity of the model, providing an important basis for the optimization of the surgical model.

[0070] The muscular layer is composed of smooth muscle and has strong extensibility. For the muscular layer, the present invention uses a collision detection algorithm combining a spherical bounding box (Sphere) and an oriented bounding box (OBB) to model: the axis-aligned bounding box can provide fast collision detection; the oriented bounding box can follow the direction and shape changes of the muscular layer, providing more accurate collision judgment.

[0071] The mucosal layer and other soft tissues of the stomach are extremely soft and elastic, and are suitable for a relatively tight bounding box modeling method. The present invention adopts a hybrid hierarchical algorithm of axis-aligned bounding box (AABB) and spherical bounding box (Sphere) to achieve modeling of the mucosal layer and other soft tissues of the stomach. The axis-aligned bounding box (AABB) can quickly determine the approximate area of ​​the collision, while the spherical bounding box can more closely fit the soft shape of the mucosal layer and other soft tissues of the stomach, which can effectively improve the accuracy and real-time performance of the experiment.

[0072] 3. The implicit Euler integration method is used to solve the equation. It has strong stability in the solution process and can reduce the number of iterations and avoid numerical instability by increasing the time step. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 Flowchart of gastric soft tissue modeling and collision detection method in virtual surgery;

[0074] Figure 2 Schematic diagram of the top-down method for constructing a hybrid hierarchical bounding box tree;

[0075] Figure 3 This is a schematic diagram of the Sphere-AABB hybrid hierarchical bounding box of the gastric mucosa layer and other gastric soft tissues;

[0076] Figure 4 This is a schematic diagram of the Sphere-OBB hybrid hierarchical bounding box tree of the stomach muscle layer;

[0077] Figure 5 This is a schematic diagram of the process of updating the positions of the model surface vertices corresponding to each bounding box after a collision occurs. DETAILED DESCRIPTION

[0078] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] Some of the professional terms and terminology that appear in this article are explained as follows:

[0080] Mass-spring model: The Mass-spring Model (MSM) treats the vertices in the soft tissue geometry model as mass points in the deformation model. The topological relationship between the vertices is described by virtual spring elements, such as structural springs and bending springs. The Mass-spring model mainly analyzes the force on each mass point and calculates the corresponding velocity and displacement properties according to Newton's law, thereby simulating the deformation effect of soft tissue in real time.

[0081] Return-Spring: A return spring is a virtual spring between a mass point and itself, and the initial length of the return spring is 0. Specifically, when a mass point deviates from its equilibrium position, the spring between the real-time position and the initial position of the same mass point is the return spring.

[0082] AABB (Axis-aligned bounding box): The corresponding Chinese interpretation is axis-aligned bounding box, which is the smallest hexahedron that contains the object and whose sides are parallel to the coordinate axes. Therefore, only six scalars are needed to describe an AABB. AABB is relatively simple in structure and has a small storage space, but poor in compactness, especially for irregular geometric shapes, where the redundant space is large. When the object is rotated, it cannot be rotated accordingly. The processed object is rigid and convex, which is not suitable for complex virtual environments containing soft body deformations. AABB is also a relatively simple type of bounding box. However, for slender objects placed diagonally, its compactness is poor. Due to the simplicity and good compactness of AABB intersection testing, it has been widely used and can also be used for collision detection of soft objects.

[0083] OBB bounding box (Oriented bounding box): The corresponding Chinese interpretation is oriented bounding box, any smallest rectangular parallelepiped. The biggest feature of OBB is the arbitrariness of its direction, which allows it to enclose objects as tightly as possible according to the shape characteristics of the enclosed object, but it also makes its intersection test complicated. The OBB bounding box is closer to the object than the AABB bounding box and the bounding sphere, and can significantly reduce the number of bounding volumes, thereby avoiding a large number of intersection detections between bounding volumes. However, the intersection detection between OBBs is more time-consuming than the intersection detection between AABBs or bounding spheres.

[0084] Sphere bounding box: It is a spherical bounding box, which is the smallest sphere that contains the object. To determine the spherical bounding box, first calculate the mean of the x, y, and z coordinates of all the elements in the basic geometric element set that make up the object to determine the center of the bounding sphere, and then determine the radius r by the distance between the center of the sphere and the point determined by the three maximum coordinates. The collision detection of the bounding sphere mainly compares the radius between the two spheres and the distance from the center of the sphere.

[0085] Deformation modeling based on physical methods is a key part of the virtual surgery system. One of the key points of soft tissue deformation modeling is how to accurately simulate the stress-strain relationship of soft tissue. Accurate soft tissue deformation modeling methods often require more computing time, which in turn leads to a decrease in the real-time performance of the virtual surgery system. Therefore, it is necessary to improve the accuracy of the soft tissue deformation modeling method as much as possible without reducing the real-time performance of the system.

[0086] The mass-spring model has been widely used in the field of soft tissue deformation modeling due to its advantages such as simple principle, easy implementation and small amount of calculation. This method also has certain limitations. One of its disadvantages is that the stability of the mass-spring model is poor. When simulating large deformation of soft bodies, "hardening" phenomenon often occurs.

[0087] The purpose of the present invention is to solve the problems existing in the prior art, to improve the accuracy of the model as much as possible while ensuring real-time performance, and to simulate the collision behavior of different soft tissues more accurately and in real time. First, we introduced the concept of a return spring in the virtual surgery system. The return spring is a virtual spring between the mass point and itself, and its initial length is 0. When a mass point deviates from its equilibrium position, the mass point is not only affected by the structure and the bending spring, but also by the return spring. The return spring will drive the displaced mass point back to its initial position. Thereby more realistically simulating the deformation of the soft tissue of the stomach.

[0088] In addition, the present invention combines the specific characteristics of the gastric soft tissue and designs collision detection algorithms according to the different characteristics of the gastric soft tissue base layer, mucosal layer and other gastric soft tissue structures, thereby maintaining the collision detection accuracy while reducing the computational overhead.

[0089] Example

[0090] In a first aspect of the present invention, a method for modeling and collision detection of soft tissue of the stomach in virtual surgery is provided. Figure 1 As shown, the method steps are as follows:

[0091] Step S100, preprocessing of stomach soft tissue images: obtaining original images of various stomach soft tissues from a medical image set (such as CT, MRI and other scanning images) and preprocessing the original images.

[0092] First, image reading is performed. In some embodiments, medical image data can be read from the medical image collection of the specialized medical image processing library pydicom, and loaded into the program to extract the required image data, including data of different ages, genders, and disease states (such as normal, gastritis, gastric ulcer, gastric cancer, etc.).

[0093] Next, image denoising and image resampling operations are performed. Preferably, the image can be denoised using a mean filter method to improve the quality of the image. The denoised image is then resampled to normalize voxels of different sizes in the image to the same size for subsequent processing and analysis.

[0094] Furthermore, the contrast and brightness of the resampled images were adjusted to make the soft tissue structures in the images more prominent.

[0095] Finally, the image segmentation operation is performed to obtain a preprocessed image with uniform resolution and size.

[0096] Step S200, identification of the soft tissue area of ​​the stomach. Using image target detection technology, the soft tissue area of ​​the stomach is identified and segmented in the preprocessed image and the data is saved. The soft tissue of the stomach is divided into three categories: muscle layer, mucosal layer and other soft tissues, and the three categories of soft tissues are classified and marked, and the shape and size data of the three categories of soft tissues are saved. This lays the foundation for the subsequent model construction and mixed hierarchical bounding box collision detection. Preferably, a threshold-based segmentation algorithm is used to process the image data to extract the three categories of soft tissue areas of the stomach and save the relevant data.

[0097] Step S300, constructing a mass-spring model of the stomach soft tissue: based on the shape and size data of the three types of stomach soft tissues segmented in step S200, combined with the different nonlinear and viscoelastic mechanical properties corresponding to the three types of soft tissues, a mass-spring model of the stomach soft tissue consisting of multiple mass points and springs is generated. The mass-spring model includes a structural spring, a bending spring, and a return spring. The elastic coefficient, damping coefficient and other parameters of the spring are set according to the biomechanical properties of the soft tissue. All parameter combinations form internal and external constraints; the constructed mass-spring model is visually rendered.

[0098] The specific process of step S300 includes the following sub-steps:

[0099] Sub-step S310, based on the shape and size data of the three types of stomach soft tissues segmented in step S200, and in combination with the different nonlinear and viscoelastic mechanical properties corresponding to the three types of soft tissues, a three-dimensional mesh model of the soft tissues is constructed using three-dimensional reconstruction technology.

[0100] Sub-step S320, using a mesh simplification algorithm to perform face reduction processing on the three-dimensional mesh model, so that the number of triangular faces on the model surface is within a reasonable range, usually the number of triangular faces on the model surface after face reduction processing is in the tens of thousands level.

[0101] Sub-step S330, for the triangular patch structure of the outer surface of the model, the surface vertices of the model are used as mass points to construct a mass-spring model. The mass-spring model includes structural springs and bending springs, and each mass point is assigned a reset spring. One end of the reset spring is connected to the current position of the mass point, and the other end is connected to the initial position of the mass point when it is not deformed. Therefore, when the mass point is deformed by an external force, it will not only be affected by the force of the structural spring and bending spring connected to it, but also by the tension of the reset spring. The original length of the reset spring is set to 0, and they always try to pull the connected mass point back to its initial position, thereby simulating the elastic recovery behavior of soft tissue when subjected to external forces.

[0102] The elastic coefficient, damping coefficient and other parameters of the spring are set according to the biomechanical properties of the soft tissue, and all parameter combinations form internal and external constraints.

[0103] Sub-step S340, applying the Delaunay algorithm to tetrahedron the model after the face reduction process to obtain a more refined model structure. According to the result of the tetrahedron division, a filling volume structure is constructed at the vertex of each tetrahedron.

[0104] Sub-step S350, visually render the constructed mass-spring model using OpenGL, so as to take into account both the authenticity and real-time requirements in the soft tissue deformation simulation process of the organism.

[0105] Then OpenGL is used to render the constructed mass-spring model visually.

[0106] Specifically, in sub-step S330, for the triangular patch structure of the outer surface of the model, the vertices of the model surface are used as mass points to construct a mass-spring model. The detailed process of completing the construction of the mass-spring on the model surface is as follows:

[0107] First, a soft tissue foundation framework based on the mass-spring model is constructed.

[0108] Assume that the surface of the soft body deformation model contains n vertices, then these n vertices constitute the discrete mass points in the mass-spring model. i According to the topological relationship between the vertices in the soft body deformation model surface, the discrete mass points are connected by spring elements. If mass point i and mass point j are discrete points in the mass spring model, the spring element between mass point i and mass point j can be represented by L ij(i, j∈[0, n-1]). Taking particle i at time t as an example, we analyze its motion. According to Newton's second law of motion, we must first find the resultant force F on particle i. i , F i It can be obtained by synthesizing the internal force (elastic force and damping force of the spring connected to it) and external force (gravity and force applied by the outside world) on particle i:

[0109]

[0110] Among them, f i represents the net force acting on particle i, represents the internal force on particle i, represents the external force on particle i;

[0111] Assuming that the mass point adjacent to i is j, the resultant force of the elastic force and damping force applied by j to i is:

[0112] F ij =K s (|X ij |-l ij )+K d (V j -V i )

[0113] Among them, F ij K represents the resultant force of the elastic force and damping force applied to i by the mass point j adjacent to i. s is the spring stiffness coefficient, X ij represents the displacement difference between particles i and j, l ij Indicates the original length of the spring, K d represents the damping coefficient, V j represents the velocity of particle j, V i represents the velocity of particle i;

[0114] Can be obtained:

[0115]

[0116] Step S400, real-time collision detection of gastric soft tissue particle-spring model: on the gastric soft tissue particle-spring model constructed in step S300, a mixed hierarchical bounding box tree is constructed from top to bottom according to the data of different tissues obtained in step S200 and the biological characteristics of different gastric soft tissues, and different mixed hierarchical bounding box collision detection algorithms are adopted to perform real-time detection and collision response.

[0117] like Figure 2As shown, the specific method of constructing a mixed hierarchical bounding tree from top to bottom is to divide the stomach soft tissue into several subsets, construct a specific bounding volume for each of the divided subsets, and then recursively perform similar operations on each subset until the current subset contains only one vertex.

[0118] like Figure 3 , Figure 4 As shown in the figure, according to the structural characteristics of different soft tissues of the stomach, the specific method of adopting different hybrid hierarchical bounding box collision detection algorithms is as follows: for the muscle layer, the present invention uses a hybrid hierarchical bounding box collision detection algorithm combining a spherical bounding box and a directed bounding box. The spherical bounding box provides fast collision detection, while the directed bounding box can follow the direction and shape changes of the muscle layer, providing more accurate collision judgment, which can effectively improve the accuracy and real-time performance of the experiment.

[0119] The OBB bounding box can be understood as an AABB bounding box with a direction. The bounding box constructed using the OBB algorithm can adapt to the construction of the OBB bounding box for three-dimensional model detection at any rotation angle, which can be expressed as:

[0120] R={O+br1e1+br2e2+br3e3|b∈(-1,1)}

[0121] Among them, R represents the radius of the spherical bounding box, O represents the coordinates of the center point of the bounding box; b is a parameter whose value varies between -1 and 1, which is used to generate the boundary or expansion area of ​​the OBB, r1, r2, r3 represent the radii of the bounding box on the x-axis, y-axis, and z-axis in the three-dimensional space; e1, e2, e3 represent orthogonal vectors used to describe the directions of the bounding box on the x-axis, y-axis, and z-axis in the three-dimensional space.

[0122] For the mucosal layer and other soft tissues of the stomach, since they are extremely soft and elastic, they are suitable for a tighter bounding box. Therefore, in this experiment, a hybrid hierarchical bounding box collision detection algorithm of axis-aligned bounding box and spherical bounding box is used for this layer. The spherical bounding box can quickly determine the approximate area of ​​the collision, while the axis-aligned bounding box can fit the soft shape of the mucosal layer more closely, thereby improving the accuracy of collision detection.

[0123] Assuming there is an AABB bounding box, the calculation method for the S coordinate of the center of the Sphere is:

[0124]

[0125] Among them, G min (x, y, z) represents the point where the bounding box obtains the minimum value in the three-dimensional coordinate system; G max (x, y, z) represents the point where the bounding box achieves the maximum value in the three-dimensional coordinate system;

[0126] The radius R of the Sphere is:

[0127]

[0128] Among them, R represents the radius of the spherical bounding box, G max (x) represents the maximum value of the bounding box on the x-axis of the three-dimensional coordinate system, G min (x) represents the minimum value of the bounding box on the x-axis of the three-dimensional coordinate system, G max (y) represents the maximum value of the bounding box on the y-axis of the three-dimensional coordinate system, G min (y) represents the minimum value of the bounding box on the y-axis of the three-dimensional coordinate system, G max (z) represents the maximum value of the bounding box on the z-axis of the three-dimensional coordinate system, G min (z) represents the minimum value of the bounding box on the z-axis of the three-dimensional coordinate system.

[0129] The efficiency of using different hybrid hierarchical bounding box collision detection algorithms for different soft tissues in the stomach is much higher than that of using a single hybrid hierarchical bounding box collision detection algorithm for different soft tissues in traditional soft tissue modeling, thereby ensuring the real-time and accuracy of the virtual surgery simulation training system to a large extent; balancing the accuracy and real-time of collision detection in the virtual surgery system. The collision detection experiment based on the stomach shows that the hybrid hierarchical bounding box method proposed in this paper can improve the efficiency of collision detection, and the higher the complexity of the soft tissue geometric model, the more obvious the improvement effect of collision detection efficiency.

[0130] Step S500, calculation of deformation of the gastric soft tissue point-spring model: Figure 5 As shown in the figure, the external force on the model is calculated based on the depth information between the surgical instrument and the soft tissue model. When a collision occurs, the local bounding box where the collision point is located is first determined, and the bounding box of the parent node is updated layer by layer until the root node is updated. Specifically, the positions of the model surface vertices corresponding to each bounding box are updated to achieve deformation simulation of soft tissue under force.

[0131] After introducing the reset spring, we can get the relationship between displacement, velocity and time. Choosing a reasonable and efficient numerical integration method is critical to the accuracy, stability and speed of the numerical solution. The implicit Euler algorithm has the advantage of small calculation amount and can improve the real-time performance and accuracy of the model. Therefore, the implicit Euler algorithm can be used for solution:

[0132]

[0133] Among them, V t represents the velocity of the particle at time t, represents the velocity of the particle at t0, represents the integral of the ratio of the resultant force on the particle and the mass of the particle during the period between t0 and t, X t is the displacement of the particle at time t, X 0 is the displacement of the particle at time t0, It represents the integral of the velocity of the particle during the period between t0 and t.

[0134] The above equations are solved using the finite difference method to approximate their derivatives:

[0135]

[0136] in, Represents the first-order derivative of the particle velocity with respect to time, V t+1 represents the velocity of the particle at time t+1, V t represents the velocity of the particle at time t, Δt represents the time step, O(Δt 2 ) represents the remainder when the time step approaches 0, represents the first-order derivative of the particle displacement with respect to time, X t+1 represents the displacement of the particle at time t+1, X t represents the displacement of the particle at time t;

[0137] From the above integral formula, the velocity and position of the particle in the next time step can be obtained as:

[0138]

[0139] Among them, V t+1 represents the velocity of the particle at time t+1, V t represents the velocity of the particle at time t, Δt represents the time step, F(t) represents the resultant force on the particle at time t, m is the mass of the particle, and X t+1 represents the displacement of the particle at time t+1, X t represents the displacement of the particle at time t;

[0140] The improved deformation differential equation of the mass-spring system is:

[0141]

[0142] in, It represents the first derivative of the particle displacement with respect to time; represents the first-order derivative of the particle's velocity with respect to time; V represents the particle's velocity; F represents the resultant force on the particle, M represents the mass matrix of the particle, and the expression of F can be written as follows:

[0143]

[0144] in, represents the internal force on particle i, represents the external force on particle i, F s Represents the elastic force of the mass spring, F d represents the damping force of the mass spring, K represents the stiffness matrix of the system, X represents the displacement of the mass, and D represents the damping matrix of the system. It represents the first derivative of the particle displacement with respect to time;

[0145] From this we can get:

[0146]

[0147] Where M represents the mass matrix of the particle, Representing the second-order derivative of the particle displacement with respect to time, we get the following equation:

[0148]

[0149] Among them, m i represents the mass of particle i, a represents the acceleration of particle i during its motion, F s Represents the elastic force of the mass spring, F d represents the damping force of the mass spring;

[0150] The above formula can be used to obtain the point differential equation of the surface point-spring model of the biological soft tissue deformation model as follows:

[0151]

[0152] in, represents the first-order derivative of the particle displacement with respect to time, V represents the velocity of the particle, represents the first-order derivative of the velocity of the particle with respect to time, and m represents the mass of the particle;

[0153] The combination of traditional mass-spring and return spring can more accurately simulate the deformation of soft tissue in complex situations, laying the foundation for mapping constraint conditions of biological soft tissue deformation simulation model in the next step.

[0154] The springs between the surface mass points of the biological soft tissue simulation model and the filling body model are constructed, and the mapping relationship between the surface mass points of the biological soft tissue deformation simulation model and the filling body model is established.

[0155] All mass points on the surface of the biological soft tissue deformation simulation model are connected to the center of the closest filling body model through independent springs. When the filling body model moves or rotates, the mass spring model on the surface of the biological soft tissue deformation simulation model will follow it to make corresponding movements. The movements of all filling body models in the biological soft tissue deformation simulation model satisfy the following dynamic differential equations:

[0156]

[0157] Among them, m i represents the mass of particle i, represents the second-order derivative of the displacement of particle i with respect to time, is the external force on particle i, and according to Newton’s second law, we get the formula:

[0158]

[0159] in, represents the first-order derivative of the displacement of particle i with respect to time, V i represents the velocity of particle i, represents the first-order derivative of the velocity of particle i with respect to time, represents the second-order derivative of the displacement of particle i with respect to time, m i is the mass of particle i.

[0160] Step S600, visual rendering: After the deformation calculation is completed, the virtual surgical scene is graphically refreshed to display the new shape and position of the soft tissue model through visual rendering. The feedback force of the model is output to the tactile device so that the user can feel the interactive force between the surgical instrument and the soft tissue. This can be achieved using existing technology.

[0161] Step S700, model optimization and verification: verify and optimize the generated soft tissue model, and the verification operation includes model accuracy evaluation and comparison verification with actual soft tissue to ensure that it meets actual requirements.

[0162] Verification and optimization of the generated soft tissue model mainly include the following aspects: Compare the soft tissue physical properties predicted by the model (such as elastic modulus, Poisson's ratio, etc.) with the actual soft tissue measurements to ensure that the two are consistent in key parameters. Perform mechanical property tests (such as tension, compression, shear, etc.) on the actual soft tissue, and compare the test results with the model prediction results to evaluate the accuracy of the model in mechanical behavior. Then adjust the model parameters based on the evaluation results to improve the accuracy and stability of the model. This step is a prior art.

[0163] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited thereto, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all belong to the protection scope of the present invention. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate the name, and do not indicate any specific order.

Claims

1. A method for modeling and collision detection of gastric soft tissue in virtual surgery, characterized in that: The steps include: Step S100, obtaining original images of various soft tissues of the stomach from a medical image set and preprocessing the original images; specifically, the steps include image reading, image denoising, image resampling, adjusting the contrast and brightness of the resampled images, and image segmentation operations to obtain preprocessed images with uniform resolution and size; Step S200, using a threshold-based segmentation algorithm in the preprocessed image to identify and segment different soft tissue areas of the stomach and save the data. Specifically, the stomach soft tissue is divided into three categories: muscle layer, mucosal layer and other soft tissues, and the three categories of soft tissues are classified and marked, and the shape and size data of the three categories of soft tissues are saved; Step S300, based on the shape and size data of the three types of stomach soft tissues segmented in step S200, combined with the different nonlinear and viscoelastic mechanical properties corresponding to the three types of soft tissues, a mass-spring model of the stomach soft tissue consisting of multiple mass points and springs is constructed, wherein the mass-spring model includes a structural spring, a bending spring, and a return spring, and the elastic coefficient, damping coefficient and other parameters of the spring are set according to the biomechanical properties of the soft tissue, and the combination of all parameters forms internal and external constraints; Visual rendering of the constructed mass-spring model; Step S400, on the mass point-spring model of the stomach soft tissue constructed in step S300, according to the data of different tissues obtained in step S200 and the biological characteristics of different stomach soft tissues, different hybrid hierarchical bounding boxes are set for the soft tissues using a hybrid hierarchical bounding box collision detection algorithm, a hybrid hierarchical bounding box tree is constructed from top to bottom, and different hybrid hierarchical bounding box collision detection algorithms are adopted for different soft tissues to perform real-time detection and collision response; Step S500, the external force on the model is calculated based on the depth information between the surgical instrument and the soft tissue model. When a collision occurs, the local bounding box where the collision point is located is first determined, and the bounding box of the parent node is updated layer by layer until the root node is updated; specifically, the position of the model surface vertex corresponding to each bounding box is updated to complete the deformation simulation of the soft tissue under force; Step S600, after the deformation calculation is completed, the virtual surgical scene is graphically refreshed to display the new shape and position of the soft tissue model; the feedback force of the model is input into the tactile device; Step S700, verifying and optimizing the generated soft tissue model, wherein the verification operation includes model accuracy evaluation and comparison verification with actual soft tissue.

2. The method for modeling and collision detection of gastric soft tissue in virtual surgery according to claim 1, characterized in that: In step S200, a threshold-based segmentation algorithm is used to process the image data.

3. The method for modeling and collision detection of gastric soft tissue in virtual surgery according to claim 1, characterized in that: The specific process of step S300 includes the following sub-steps: Sub-step S310, based on the shape and size data of the three types of stomach soft tissues segmented in step S200, and in combination with the different nonlinear and viscoelastic mechanical properties corresponding to the three types of soft tissues, a three-dimensional mesh model of the soft tissues is constructed using a three-dimensional reconstruction technology; Sub-step S320, using a mesh simplification algorithm to perform surface reduction processing on the three-dimensional mesh model; Sub-step S330, for the triangular patch structure of the outer surface of the model, the surface vertices of the model are used as mass points to construct a mass-spring model; the mass-spring model includes a structural spring and a bending spring, and each mass point is assigned a reset spring; one end of the reset spring is connected to the current position of the mass point, and the other end is connected to the initial position of the mass point without deformation, and the original length of the reset spring is set to 0; the elastic coefficient and damping coefficient of the spring are set according to the biomechanical properties of the soft tissue, and all the parameter combinations form internal and external constraints; Sub-step S340, tetrahedroning the model after the face reduction process, and constructing a filling volume structure at the vertex of each tetrahedron according to the result of the tetrahedroning; Sub-step S350, performing visual rendering on the constructed mass-spring model.

4. The method for modeling and collision detection of gastric soft tissue in virtual surgery according to claim 3, characterized in that: After surface reduction, the number of triangles on the model surface is in the tens of thousands.

5. The method for modeling and collision detection of gastric soft tissue in virtual surgery according to claim 1, characterized in that: In step S400, the specific method of constructing the mixed hierarchical bounding tree from top to bottom is: dividing the stomach soft tissue into several subsets, constructing a specific bounding volume for each of the divided subsets, and then recursively performing similar operations on each subset until the current subset contains only one vertex.

6. The method for modeling and collision detection of gastric soft tissue in virtual surgery according to claim 1, characterized in that: The specific method of adopting different hybrid hierarchical bounding box collision detection algorithms for different soft tissues in step S400 is: using a hybrid hierarchical bounding box collision detection algorithm combining a spherical bounding box and a directed bounding box for the muscle layer, and using a hybrid hierarchical bounding box collision detection algorithm combining an axis-aligned bounding box and a spherical bounding box for the mucosa layer and other soft tissues.

7. The method for modeling and collision detection of gastric soft tissue in virtual surgery according to claim 1, characterized in that: In step S500, the implicit Euler algorithm is used to solve the relationship between displacement, velocity and time, and the method is as follows: Among them, V t represents the velocity of the particle at time t, represents the velocity of the particle at time t0, represents the integral of the ratio of the resultant force on the particle and the mass of the particle during the period between t0 and t, X t represents the displacement of the particle at time t, represents the displacement of the particle at time t0, It represents the integral of the velocity of the particle during the period between t0 and t; The above equations are solved using the finite difference method to approximate their derivatives: in, Represents the first-order derivative of the particle velocity with respect to time, V t+1 represents the velocity of the particle at time t+1, V t represents the velocity of the particle at time t, Δt represents the time step, O(Δt 2 ) represents the remainder when time approaches 0, represents the first-order derivative of the particle displacement with respect to time, X t+1 represents the displacement of the particle at time t+1, X t represents the displacement of the particle at time t; the velocity and position of the particle in the next time step are: Among them, V t+1 represents the velocity of the particle at time t+1, V t represents the velocity of the particle at time t, Δt represents the time step, F(t) represents the resultant force on the particle at time t, m represents the mass of the particle, and X t+1 represents the displacement of the particle at time t+1, X t represents the displacement of the particle at time t.

8. According to the above-mentioned method for modeling and collision detection of soft tissue of the stomach in virtual surgery, it is characterized in that: The deformation differential equation of the improved mass-spring model is: in, It represents the first derivative of the particle displacement with respect to time; represents the first-order derivative of the particle's velocity with respect to time; V represents the particle's velocity; F represents the resultant force on the particle, M represents the mass matrix of the particle, and the expression of F can be written as follows: in, represents the internal force on particle i, represents the external force on particle i, F s Represents the elastic force of the mass spring, F d represents the damping force of the mass spring, K represents the stiffness matrix of the system, X represents the displacement of the mass, and D represents the damping matrix of the system. Represents the first derivative of the particle displacement with respect to time.

9. The method for modeling and collision detection of gastric soft tissue in virtual surgery according to claim 8, characterized in that: The point differential equation of the surface mass-spring model of the biological soft tissue deformation model is as follows: in, represents the first-order derivative of the particle displacement with respect to time, V represents the velocity of the particle, represents the first-order derivative of the particle velocity with respect to time, represents the external force on particle i, F s Represents the elastic force of the mass spring, F d represents the damping force of the mass spring, and m represents the mass of the mass.

10. The method for modeling and collision detection of gastric soft tissue in virtual surgery according to claim 9, characterized in that: A return spring is added to the traditional spring model. All mass points on the surface of the biological soft tissue deformation simulation model are connected to the center of the closest filling body model through an independent spring. When the filling body model moves or rotates, the mass spring model on the surface of the biological soft tissue deformation simulation model will follow it to make corresponding movements. The movements of all filling body models in the biological soft tissue deformation simulation model satisfy the following dynamic differential equations: Among them, m i represents the mass of particle i, represents the second-order derivative of the displacement of particle i with respect to time, Represents the external force on particle i. According to Newton's second law, the formula is: in, represents the first-order derivative of the displacement of particle i with respect to time, V i represents the velocity of particle i, represents the first-order derivative of the velocity of particle i with respect to time, represents the second-order derivative of the displacement of particle i with respect to time, represents the external force on particle i, m i represents the mass of particle i.

Images