A three-dimensional model registration method, medical terminal and device
By registering and deformation operations of the intraoperative endoscopic images and preoperative three-dimensional models in the medical terminal, the problem of poor matching of the three-dimensional models during the operation is solved, and the safety and accuracy of the operation are improved.
Patent Information
- Application Number
- CN202210728705.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-24
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-06-24
AI Technical Summary
During the operation, due to changes in the patient's position and deformation caused by surgical operations, the preoperative three-dimensional model cannot accurately guide the operation, reducing the safety of the operation.
By using a processor in a medical terminal to rigidly register the intraoperative endoscopic images and preoperative three-dimensional models, the displacement information of the model key points and image key points is determined, and deformation operations are performed on the tetrahedral mesh are obtained and the surgery is obtained. 3D model of morphological matching in target.
It improves the safety of the surgery, ensures the accurate matching of the three-dimensional model with the target part during the operation, and enhances the accuracy and safety of the surgery.
Smart Images

Figure CN115187531B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image registration, and in particular to a three-dimensional model registration method, a medical terminal and a device. Background Art
[0002] An endoscope is a detection instrument that integrates traditional optics, ergonomics, precision machinery, modern electronics, mathematics, and software. It includes image sensors, optical lenses, light sources, mechanical devices, etc. It can enter the stomach through the mouth or other natural orifices and surgical incisions into the body. In modern medicine, endoscopes can be used to see lesions that cannot be shown by X-rays.
[0003] Augmented reality technology can fuse model data reconstructed based on preoperative medical images with intraoperative images, allowing doctors to intuitively see the anatomical structure below the surface of the organ, effectively solving the problem of lack of depth perception information and tactile feedback information, making surgical operations faster, more precise and safer.
[0004] However, during the operation, the patient's body position changes (gravity factors) and various surgical operations will cause the surgical site to undergo significant deformation, resulting in a significant difference between the surgical site in its current form and the surgical site photographed before the operation. Therefore, the three-dimensional model obtained by photographing the surgical site before the operation and reconstructing it cannot accurately guide the operation, resulting in lower safety of the operation. Summary of the invention
[0005] In order to solve the existing technical problems, the embodiments of the present application provide a three-dimensional model registration method, a medical terminal and a device, which can obtain a target three-dimensional model that better matches the surgical site during surgery, so that the surgery can be guided more accurately based on the obtained target three-dimensional model, thereby improving the safety of the surgery.
[0006] In a first aspect, an embodiment of the present application provides a medical terminal, including:
[0007] A transceiver unit, used for receiving an endoscopic image of a target part acquired by an endoscope during surgery;
[0008] A memory, used to store an initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before surgery;
[0009] A processor is used to rigidly register the initial three-dimensional model according to the endoscopic image to obtain a reference three-dimensional model after rigid registration; determine at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image; the at least one model key point is a part of the points on the feature line corresponding to the target part in the reference three-dimensional model, and the at least one image key point is a part of the points on the feature line corresponding to the target part in the endoscopic image; determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of each image key point; perform a deformation operation on the tetrahedral mesh corresponding to the reference three-dimensional model according to the determined displacement information to obtain the target three-dimensional model after deformation registration.
[0010] The medical terminal provided in the embodiment of the present application can, after acquiring the initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before surgery and collecting the endoscopic image of the target part during surgery, perform rigid registration on the initial three-dimensional model according to the endoscopic image to obtain a reference three-dimensional model after rigid registration, and determine at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image, wherein at least one model key point is a partial point on a feature line corresponding to the target part in the reference three-dimensional model, and at least one image key point is a partial point on a feature line corresponding to the target part in the endoscopic image, and then determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of each image key point, and finally perform a deformation operation on the tetrahedral mesh corresponding to the reference three-dimensional model according to the determined displacement information to obtain the target three-dimensional model after deformation registration. Since the tetrahedral mesh corresponding to the three-dimensional model can be deformed according to the displacement information between the model key points in the three-dimensional model after rigid registration and the image key points in the corresponding endoscopic image, a target three-dimensional model that is deformed and registered with the endoscopic image can be obtained. This can make the obtained target three-dimensional model more closely match the target part displayed in the endoscopic image, and can more correctly and reasonably reflect the display morphology of the target part during the operation. Furthermore, based on the target three-dimensional model, the operation can be guided more accurately and the safety of the operation can be improved.
[0011] In an optional embodiment, the processor is specifically configured to:
[0012] Project each model key point and each image key point into the target coordinate system of the same dimension;
[0013] According to the position information of each model key point and the position information of each image key point, the at least one model key point and the at least one image key point are grouped into a plurality of feature point groups; wherein each feature point group includes a model key point and an image key point, and the image key point corresponding to the model key point in a feature point group is the image key point in the at least one image key point that is closest to the model key point;
[0014] Generate a covariance matrix according to the difference in position information between the model key points and the image key points in each feature point group; perform eigenvalue decomposition on the covariance matrix to obtain a transformation matrix, and perform transformation processing on each model key point according to the transformation matrix;
[0015] If the average distance between each model key point after the transformation and the corresponding image key point is not greater than a preset first distance threshold, determining the image key point corresponding to each model key point according to the multiple feature point groups;
[0016] If the mean distance between each model key point after the transformation and the corresponding image key point is greater than a preset first distance threshold, the model key points in the multiple feature point groups are replaced by the model key points after the transformation, and the step of generating a covariance matrix based on the difference in position information between the model key points and the image key points in each feature point group is returned to be executed.
[0017] The above-mentioned medical terminal, after projecting each model key point and each image key point into the target coordinate system under the same dimension, can respectively determine the model key point closest to the image key point for each image key point, and take the model key point and the image key point as a corresponding point pair, determine the transformation matrix for transforming each model key point according to each corresponding point pair, and after transforming each model key point according to the transformation matrix to obtain each transformed model key point, if the mean value of the distance between each transformed model key point and the corresponding image key point is not greater than the set threshold, determine the image key point corresponding to each model key point as the image key point closest to the model key point, otherwise, replace each model key point with each transformed model key point, and repeat the above operation until the mean value of the distance between each transformed model key point and the corresponding image key point is not greater than the set threshold. Thus, for each model key point, the image key point corresponding to the model key point can be accurately determined.
[0018] In an optional embodiment, the processor is specifically configured to:
[0019] Mapping the at least one model key point to mesh vertices of the tetrahedral mesh corresponding to the reference three-dimensional model, respectively, to obtain a model mesh vertex corresponding to each model key point;
[0020] According to the displacement information between each model key point and the corresponding image key point, each model mesh vertex is moved to the position of the corresponding image key point, and the three-dimensional model obtained after moving each model mesh vertex is used as the target three-dimensional model after deformation registration.
[0021] The above-mentioned medical terminal can map at least one model key point to the mesh vertex of the tetrahedral mesh corresponding to the reference three-dimensional model, obtain the model mesh vertex corresponding to each model key point, and move each model mesh vertex to the position of the corresponding image key point according to the displacement information between each model key point and the corresponding image key point, and use the three-dimensional model obtained after the movement of each model mesh vertex as the target three-dimensional model after deformation registration. Since the corresponding model mesh vertex on the tetrahedral mesh can be moved to the position of the corresponding image key point according to the displacement information between the model key point and the corresponding image key point, a target three-dimensional model that better matches the target part displayed in the endoscopic image can be obtained.
[0022] In an optional embodiment, the displacement information includes distance information and direction information, and the processor is specifically configured to:
[0023] The following operations are performed for the model mesh vertex corresponding to any model key point: determining a first moving distance for moving the model mesh vertex this time according to the distance information between the model key point and the corresponding image key point; and moving the model mesh vertex by the first moving distance within a set time step according to the direction information between the model key point and the corresponding image key point;
[0024] If the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is not greater than a preset second distance threshold, it is determined to move each of the model mesh vertices to the positions of the corresponding image key points.
[0025] The above-mentioned medical terminal can determine the first moving distance of the model mesh vertex corresponding to any model key point according to the distance information between the model key point and the corresponding image key point, and move the model mesh vertex corresponding to the model key point by the first moving distance within the set time step according to the direction information between the model key point and the corresponding image key point. If the sum of the distances between each model mesh vertex and the corresponding image key point after the movement is not greater than the preset second distance threshold, it can be determined that each model mesh vertex is moved to the position of the corresponding image key point. Therefore, the corresponding model mesh vertex can be reasonably moved to the position of the corresponding image key point on the tetrahedral mesh according to the displacement information between the model key point and the corresponding image key point.
[0026] In an optional embodiment, the processor is further configured to:
[0027] If the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is greater than a preset second distance threshold, each of the moved model mesh vertices is used as each of the target model mesh vertices, and the following operations are performed for each of the target model mesh vertices:
[0028] Determining a second moving distance for moving the mesh vertex of the target model this time according to the distance information between the model key point and the corresponding image key point and the first moving distance;
[0029] According to the direction information between the image key point and the corresponding model key point, the target model mesh vertex is moved by the second moving distance within a set time step until the sum of the distances between each target model mesh vertex and the corresponding image key point after the move is no greater than a preset second distance threshold.
[0030] The above-mentioned medical terminal, when the model mesh vertex corresponding to the model key point is moved once on the tetrahedral mesh according to the displacement information between the model key point and the corresponding image key point, if the sum of the distances between the model mesh vertex after the movement and the corresponding image key point is greater than the preset distance threshold, the model mesh vertex after the movement can be used as the target model mesh vertex, and the target model mesh vertex is moved again on the tetrahedral mesh according to the displacement information between the model key point and the corresponding image key point, until the sum of the distances between the target model mesh vertex after the movement and the corresponding image key point is not greater than the preset distance threshold, it can be determined that the model mesh vertex corresponding to the model key point is moved to the position of the corresponding image key point. In this way, the deformation operation can be performed on the tetrahedral mesh in a physically consistent manner, so that the obtained target three-dimensional model is consistent with the characteristic edge of the target part in the endoscopic image, and can more correctly reflect the display form of the target part during the operation.
[0031] In an optional embodiment, the processor is specifically configured to:
[0032] Determining a target speed for moving the mesh vertices of the target model according to the first moving distance and a set time step;
[0033] Determine an accelerated moving distance for moving the target model mesh vertex according to the target speed and the set time step, and determine a deformed moving distance for moving the target model mesh vertex according to distance information between the model key point and the corresponding image key point and the set time step;
[0034] A second moving distance for moving the mesh vertices of the target model this time is determined according to the accelerated moving distance and the deformed moving distance.
[0035] The above-mentioned medical terminal can determine the target speed for moving the target model mesh vertex according to the first moving distance and the set time step, and then determine the accelerated moving distance for moving the target model mesh vertex according to the target speed and the set time step, and determine the deformation moving distance for moving the target model mesh vertex according to the distance information between the model key point and the corresponding image key point and the set time step, and finally determine the second moving distance for moving the target model mesh vertex this time according to the accelerated moving distance and the deformation moving distance. In this way, the second moving distance for moving the target model mesh vertex in each time step can be determined physically, and then after moving the target model mesh vertex according to the second moving distance, the target model mesh vertex can be more reasonably moved to the position of the corresponding image key point.
[0036] In an optional embodiment, the processor is specifically configured to:
[0037] Performing mesh simplification and mesh repair processing on the surface mesh of the reference three-dimensional model to obtain a repaired three-dimensional model;
[0038] The repaired three-dimensional model is meshed with tetrahedral units to obtain a tetrahedral mesh corresponding to the reference three-dimensional model.
[0039] The above-mentioned medical terminal can first perform mesh simplification and mesh repair on the surface mesh of the reference three-dimensional model to obtain a repaired three-dimensional model, and then perform mesh division of the repaired three-dimensional model into tetrahedral units to obtain a tetrahedral mesh corresponding to the reference three-dimensional model. Since the three-dimensional model obtained based on three-dimensional reconstruction cannot directly perform deformation operations, the three-dimensional model can be obtained by mesh simplification, mesh repair and tetrahedralization to obtain a tetrahedral mesh, and then deformation operations can be performed based on the tetrahedral mesh, so as to obtain a target three-dimensional model that can more correctly and reasonably reflect the display morphology of the target part during the operation.
[0040] In a second aspect, an embodiment of the present application provides a three-dimensional model registration method, comprising:
[0041] Acquire an initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before surgery, and an endoscopic image of the target part collected during surgery; perform rigid registration on the initial three-dimensional model according to the endoscopic image to obtain a reference three-dimensional model after rigid registration;
[0042] Determine at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image; the at least one model key point is a partial point on a feature line corresponding to the target part in the reference three-dimensional model, and the at least one image key point is a partial point on a feature line corresponding to the target part in the endoscopic image;
[0043] Determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of each image key point;
[0044] A deformation operation is performed on the tetrahedral mesh corresponding to the reference three-dimensional model according to the determined displacement information to obtain a target three-dimensional model after deformation and registration.
[0045] In an optional embodiment, the image key point corresponding to each model key point is determined according to the following method:
[0046] Project each model key point and each image key point into the target coordinate system of the same dimension;
[0047] According to the position information of each model key point and the position information of each image key point, the at least one model key point and the at least one image key point are grouped into a plurality of feature point groups; wherein each feature point group includes a model key point and an image key point, and the image key point corresponding to the model key point in a feature point group is the image key point in the at least one image key point that is closest to the model key point;
[0048] Generate a covariance matrix according to the difference in position information between the model key points and the image key points in each feature point group; perform eigenvalue decomposition on the covariance matrix to obtain a transformation matrix, and perform transformation processing on each model key point according to the transformation matrix;
[0049] If the average distance between each model key point after the transformation and the corresponding image key point is not greater than a preset first distance threshold, determining the image key point corresponding to each model key point according to the multiple feature point groups;
[0050] If the mean distance between each model key point after the transformation and the corresponding image key point is greater than a preset first distance threshold, the model key points in the multiple feature point groups are replaced by the model key points after the transformation, and the step of generating a covariance matrix based on the difference in position information between the model key points and the image key points in each feature point group is returned to be executed.
[0051] In an optional embodiment, performing a deformation operation on the tetrahedral mesh corresponding to the reference three-dimensional model according to the determined displacement information to obtain a target three-dimensional model after deformation registration includes:
[0052] Mapping the at least one model key point to mesh vertices of the tetrahedral mesh corresponding to the reference three-dimensional model, respectively, to obtain a model mesh vertex corresponding to each model key point;
[0053] According to the displacement information between each model key point and the corresponding image key point, each model mesh vertex is moved to the position of the corresponding image key point, and the three-dimensional model obtained after moving each model mesh vertex is used as the target three-dimensional model after deformation registration.
[0054] In an optional embodiment, the displacement information includes distance information and direction information, and the step of moving each model mesh vertex to a position of the corresponding image key point according to the displacement information between each model key point and the corresponding image key point includes:
[0055] The following operations are performed for the model mesh vertex corresponding to any model key point: determining a first moving distance for moving the model mesh vertex this time according to the distance information between the model key point and the corresponding image key point; and moving the model mesh vertex by the first moving distance within a set time step according to the direction information between the model key point and the corresponding image key point;
[0056] If the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is not greater than a preset second distance threshold, it is determined to move each of the model mesh vertices to the positions of the corresponding image key points.
[0057] In an optional embodiment, the method further includes:
[0058] If the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is greater than a preset second distance threshold, each of the moved model mesh vertices is used as each of the target model mesh vertices, and the following operations are performed for each of the target model mesh vertices:
[0059] Determining a second moving distance for moving the mesh vertex of the target model this time according to the distance information between the model key point and the corresponding image key point and the first moving distance;
[0060] According to the direction information between the image key point and the corresponding model key point, the target model mesh vertex is moved by the second moving distance within a set time step until the sum of the distances between each target model mesh vertex and the corresponding image key point after the move is no greater than a preset second distance threshold.
[0061] In an optional embodiment, determining the second moving distance for moving the target model mesh vertex this time according to the distance information between the model key point and the corresponding image key point and the first moving distance includes:
[0062] Determining a target speed for moving the mesh vertices of the target model according to the first moving distance and a set time step;
[0063] Determine an accelerated moving distance for moving the target model mesh vertex according to the target speed and the set time step, and determine a deformed moving distance for moving the target model mesh vertex according to distance information between the model key point and the corresponding image key point and the set time step;
[0064] A second moving distance for moving the mesh vertices of the target model this time is determined according to the accelerated moving distance and the deformed moving distance.
[0065] In an optional embodiment, the tetrahedral mesh is determined according to the following method:
[0066] Performing mesh simplification and mesh repair processing on the surface mesh of the reference three-dimensional model to obtain a repaired three-dimensional model;
[0067] The repaired three-dimensional model is meshed with tetrahedral units to obtain a tetrahedral mesh corresponding to the reference three-dimensional model.
[0068] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the three-dimensional model registration method of the second aspect described above is implemented.
[0069] In a fourth aspect, an embodiment of the present application further provides a three-dimensional model registration device, including:
[0070] A rigid registration unit is used to obtain an initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before surgery, and an endoscopic image of the target part collected during surgery; rigidly register the initial three-dimensional model according to the endoscopic image to obtain a reference three-dimensional model after rigid registration;
[0071] A key point determination unit, used to determine at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image; the at least one model key point is a partial point on a feature line corresponding to the target part in the reference three-dimensional model, and the at least one image key point is a partial point on a feature line corresponding to the target part in the endoscopic image;
[0072] A displacement information determination unit, used to determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of each image key point;
[0073] The deformation registration unit is used to perform deformation operations on the tetrahedral mesh corresponding to the reference three-dimensional model according to the determined displacement information to obtain a target three-dimensional model after deformation registration.
[0074] In an optional embodiment, the device further includes a corresponding point determining unit, configured to:
[0075] Project each model key point and each image key point into the target coordinate system of the same dimension;
[0076] According to the position information of each model key point and the position information of each image key point, the at least one model key point and the at least one image key point are grouped into a plurality of feature point groups; wherein each feature point group includes a model key point and an image key point, and the image key point corresponding to the model key point in a feature point group is the image key point in the at least one image key point that is closest to the model key point;
[0077] Generate a covariance matrix according to the difference in position information between the model key points and the image key points in each feature point group; perform eigenvalue decomposition on the covariance matrix to obtain a transformation matrix, and perform transformation processing on each model key point according to the transformation matrix;
[0078] If the average distance between each model key point after the transformation and the corresponding image key point is not greater than a preset first distance threshold, determining the image key point corresponding to each model key point according to the multiple feature point groups;
[0079] If the mean distance between each model key point after the transformation and the corresponding image key point is greater than a preset first distance threshold, the model key points in the multiple feature point groups are replaced by the model key points after the transformation, and the step of generating a covariance matrix based on the difference in position information between the model key points and the image key points in each feature point group is returned to be executed.
[0080] In an optional embodiment, the deformation registration unit is specifically used to:
[0081] Mapping the at least one model key point to mesh vertices of the tetrahedral mesh corresponding to the reference three-dimensional model, respectively, to obtain a model mesh vertex corresponding to each model key point;
[0082] According to the displacement information between each model key point and the corresponding image key point, each model mesh vertex is moved to the position of the corresponding image key point, and the three-dimensional model obtained after moving each model mesh vertex is used as the target three-dimensional model after deformation registration.
[0083] In an optional embodiment, the displacement information includes distance information and direction information, and the deformation registration unit is specifically used to:
[0084] The following operations are performed for the model mesh vertex corresponding to any model key point: determining a first moving distance for moving the model mesh vertex this time according to the distance information between the model key point and the corresponding image key point; and moving the model mesh vertex by the first moving distance within a set time step according to the direction information between the model key point and the corresponding image key point;
[0085] If the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is not greater than a preset second distance threshold, it is determined to move each of the model mesh vertices to the positions of the corresponding image key points.
[0086] In an optional embodiment, the deformation registration unit is further used for:
[0087] If the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is greater than a preset second distance threshold, each of the moved model mesh vertices is used as each of the target model mesh vertices, and the following operations are performed for each of the target model mesh vertices:
[0088] Determining a second moving distance for moving the mesh vertex of the target model this time according to the distance information between the model key point and the corresponding image key point and the first moving distance;
[0089] According to the direction information between the image key point and the corresponding model key point, the target model mesh vertex is moved by the second moving distance within a set time step until the sum of the distances between each target model mesh vertex and the corresponding image key point after the move is no greater than a preset second distance threshold.
[0090] In an optional embodiment, the deformation registration unit is further used for:
[0091] Determining a target speed for moving the mesh vertices of the target model according to the first moving distance and a set time step;
[0092] Determine an accelerated moving distance for moving the target model mesh vertex according to the target speed and the set time step, and determine a deformed moving distance for moving the target model mesh vertex according to distance information between the model key point and the corresponding image key point and the set time step;
[0093] A second moving distance for moving the mesh vertices of the target model this time is determined according to the accelerated moving distance and the deformed moving distance.
[0094] In an optional embodiment, the device further comprises a tetrahedron generation unit, which is used to:
[0095] Performing mesh simplification and mesh repair processing on the surface mesh of the reference three-dimensional model to obtain a repaired three-dimensional model;
[0096] The repaired three-dimensional model is meshed with tetrahedral units to obtain a tetrahedral mesh corresponding to the reference three-dimensional model.
[0097] The technical effects brought about by any one of the implementation methods of the second aspect, the third aspect or the fourth aspect can refer to the technical effects brought about by the implementation method of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0098] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0099] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;
[0100] Figure 2 A schematic diagram of the structure of a medical terminal provided in an embodiment of the present application;
[0101] Figure 3 A schematic diagram of the structure of another medical terminal provided in an embodiment of the present application;
[0102] Figure 4 A flowchart of a three-dimensional model registration method provided in an embodiment of the present application;
[0103] Figure 5 A flowchart for determining corresponding points provided in an embodiment of the present application;
[0104] Figure 6 A flowchart of tetrahedral mesh generation provided in an embodiment of the present application;
[0105] Figure 7a A flowchart of obtaining a target three-dimensional model provided in an embodiment of the present application;
[0106] Figure 7b A flow chart for determining a second moving distance provided in an embodiment of the present application;
[0107] Figure 8 A flowchart of another three-dimensional model registration method provided in an embodiment of the present application;
[0108] Figure 9a A schematic diagram of characteristic lines on a target part in an endoscopic image provided in an embodiment of the present application;
[0109] Figure 9b A schematic diagram of characteristic lines on a target part in a reference three-dimensional model provided in an embodiment of the present application;
[0110] Fig.10 A schematic diagram of a tetrahedral grid provided in an embodiment of the present application;
[0111] Fig.11 A schematic diagram of each model mesh vertex and each corresponding image key point on a tetrahedral mesh provided in an embodiment of the present application;
[0112] Fig.12 A schematic diagram of moving model mesh vertices according to displacement information provided in an embodiment of the present application;
[0113] Fig.13 A schematic diagram of the final deformation result of a tetrahedral mesh provided in an embodiment of the present application;
[0114] Fig.14 A schematic diagram of a change process of the sum of distances between each model mesh vertex and the corresponding image key point provided in an embodiment of the present application;
[0115] Fig.15 A schematic diagram of a comparison between a tetrahedral mesh after deformation and a tetrahedral mesh before deformation provided in an embodiment of the present application;
[0116] Fig.16 A schematic diagram of the structure of a three-dimensional model registration device provided in an embodiment of the present application;
[0117] Fig.17 A schematic diagram of the structure of another three-dimensional model registration device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0118] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0119] It should be noted that the terms "first", "second" etc. involved in the documents of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in a sequence other than the content illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0120] The following is a brief introduction to the design concept of the embodiment of the present application:
[0121] Laparoscopic liver resection has the advantages of small incision, less pain, fast recovery, and small scars, and has quickly become the development trend of current surgery. However, due to the lack of depth perception information and tactile feedback information, it poses a greater challenge to surgeons. Augmented reality technology can fuse the model data reconstructed based on preoperative medical images with intraoperative images, allowing doctors to intuitively see the anatomical structure below the surface of the organ, effectively solving the problem of lack of depth perception information and tactile feedback information, and making surgical operations faster, more accurate, and safer. That is, the three-dimensional model and the liver image are registered and fused for display, which can achieve the ability of "seeing through the liver". On the basis of realizing the fusion display, it is also necessary to reduce the matching error as much as possible so that the technology can better help doctors perform surgical operations.
[0122] At present, most of the augmented reality intraoperative navigation is still based on the registration scheme of affine transformation (rotation, translation and scaling): using the positioning device and the point cloud registration algorithm, the optical positioning coordinate system of the patient and the CT image coordinate system are rigidly transformed (rotated and translated) to unify them; secondly, the chessboard and positioning system are used to calibrate the endoscopic camera to obtain the rigid transformation relationship between the endoscopic camera coordinate system and the positioning coordinate system, so as to achieve the rigid registration of the preoperative three-dimensional model with the liver in the laparoscopic video. However, during the operation, the patient's body position changes (gravity factors), pneumoperitoneum in the surgical process, and operations such as falciform ligament resection during the operation will cause the organs to undergo relatively large deformation. The liver in its current form is quite different from the patient's liver when the CT was taken before the operation. Therefore, the liver, blood vessels, and lesions reconstructed by the preoperative CT for guiding the operation will lack reference, which is a great difficulty for precision surgery.
[0123] In addition to rigid registration, there are also some 3D point cloud deformation registration, such as Coherent PointDrift, which is a robust point set matching algorithm based on Gaussian mixture model. However, obtaining intraoperative 3D points requires additional equipment, such as positioning equipment, binocular endoscopes, etc. At the same time, the point set-based deformation registration algorithm does not consider parameters such as the elastic modulus between tissues. Therefore, it is impossible to transmit the displacement of the surface points of the organ to the inside of the organ in accordance with physics (tissue elasticity model, incompressibility, etc.), and thus it is impossible to correctly display the displacement of blood vessels and lesions inside the organ.
[0124] Therefore, in order to make the three-dimensional model obtained by three-dimensional reconstruction of the target part before surgery match the target part in the intraoperative endoscopic image, better reflect the display form of the target part during surgery, and thus be able to more accurately guide the surgery and improve the safety of the surgery, the embodiment of the present application provides a three-dimensional model registration method, a medical terminal and a device. After obtaining the initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before surgery, and collecting the endoscopic image of the target part during surgery, the initial three-dimensional model can be rigidly registered according to the endoscopic image to obtain a reference three-dimensional model after rigid registration, and at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image are determined, wherein at least one model key point is a partial point on the feature line corresponding to the target part in the reference three-dimensional model, and at least one image key point is a partial point on the feature line corresponding to the target part in the endoscopic image, and then according to the position information of each model key point and the position information of each image key point, the displacement information between each model key point and the corresponding image key point is determined, and finally, according to the determined displacement information, a deformation operation is performed on the tetrahedral mesh corresponding to the reference three-dimensional model to obtain the target three-dimensional model after deformation registration.
[0125] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0126] The word "exemplary" used hereinafter means "serving as an example, embodiment, or illustrative". Any embodiment described as "exemplary" is not necessarily to be interpreted as superior or better than other embodiments. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.
[0127] See also Figure 1 FIG. 1 is a schematic diagram of an optional application scenario of an embodiment of the present application, which may include an endoscope 100 , a medical device 200 , and an image scanning device 300 .
[0128] The endoscope 100 is used to collect endoscopic images of the target part during surgery, and the image scanning device 300 is used to photograph the target part before surgery to obtain medical digital image data of the target object. The endoscope 100 and the medical device 200, and the medical device 200 and the image scanning device 300 can communicate with each other in a wired or wireless manner.
[0129] Optionally, the image scanning device 300 can perform CT or MRI on the target part before surgery to obtain medical digital image data of the target object.
[0130] In one embodiment, the medical device 200 can receive medical digital image data of the target part sent by the image scanning device 300, perform three-dimensional reconstruction on the medical digital image data of the target part to obtain an initial three-dimensional model, and store the initial three-dimensional model of the target part; and receive in real time the endoscopic image of the target part sent by the endoscope 100.
[0131] In another embodiment, the medical device 200 can also receive an initial three-dimensional model obtained by preoperative photography and three-dimensional reconstruction of the target part sent by the image scanning device 300, and store the initial three-dimensional model of the target part; and receive in real time the endoscopic image of the target part sent by the endoscope 100.
[0132] Figure 2 FIG. 2 is a schematic diagram showing a structure of a medical terminal provided by an embodiment of the present application. Figure 2 As shown, the medical terminal provided in the embodiment of the present application includes a processor 203, a memory 202 and a transceiver unit 201.
[0133] Wherein, the transceiver unit 201 is used to receive the endoscopic image of the target part collected by the endoscope during the operation;
[0134] The memory 202 is used to store the initial 3D model obtained by photographing the target part and 3D reconstruction before the operation;
[0135] The processor 203 may include one or more central processing units (CPUs), or digital processing units, etc. The processor 203 is used to perform rigid registration on the initial three-dimensional model according to the endoscopic image to obtain a reference three-dimensional model after rigid registration; determine at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image; at least one model key point is a partial point on a feature line corresponding to a target part in the reference three-dimensional model, and at least one image key point is a partial point on a feature line corresponding to a target part in the endoscopic image; determine displacement information between each model key point and a corresponding image key point according to position information of each model key point and position information of each image key point; perform deformation operation on a tetrahedral mesh corresponding to the reference three-dimensional model according to each determined displacement information to obtain a target three-dimensional model after deformation registration.
[0136] The specific connection medium between the transceiver unit 201, the memory 202 and the processor 203 is not limited in the embodiment of the present application. Figure 2In the embodiment, the transceiver unit 201, the memory 202 and the processor 203 are connected via a bus 204, and the connection between other components is only for schematic illustration and is not intended to be limiting. The bus 204 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0137] Optionally, another optional application scenario of the embodiment of the present application; wherein the medical terminal may be an endoscope, such as Figure 3 As shown, the endoscope may include a processor 303 , a memory 302 and an image acquisition unit 301 .
[0138] The image acquisition unit 301 is used to acquire an endoscopic image of the target part during surgery;
[0139] It should be noted that the image acquisition unit 301 may be a camera.
[0140] The memory 302 is used to store the initial 3D model obtained by photographing the target part and 3D reconstruction before the operation;
[0141] The processor 303 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 303 is used to implement the three-dimensional model registration method provided in the embodiment of the present application.
[0142] The specific connection medium between the image acquisition unit 301, the memory 302 and the processor 303 is not limited in the embodiment of the present application. Figure 3 In the embodiment, the image acquisition unit 301, the memory 302 and the processor 303 are connected via a bus 304, and the connection between other components is only for schematic illustration and is not intended to be limiting. The bus 304 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0143] It should be noted that the endoscope in the embodiment of the present application specifically refers to a surgical endoscope, which may be one or more of a laparoscope, a thoracoscope, an arthroscope, a nasal endoscope, a cystoscope, a colonoscope, a duodenoscope, a mediastinoscope and a cardioscope.
[0144] In some embodiments, the flowchart of the three-dimensional model registration method performed by the above medical terminal can be found in Figure 4 As shown, the following steps are included:
[0145] Step S401, obtaining an initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before the operation, and an endoscopic image of the target part collected during the operation.
[0146] Before the operation, the patient's target part is photographed to obtain the corresponding medical digital image data, and the medical digital image data is reconstructed in three dimensions to obtain an initial three-dimensional model corresponding to the target part.
[0147] The medical digital image data may be obtained by performing CT scanning on the target part, or may be obtained by performing MRI scanning on the target part, which is not limited in the present application.
[0148] During the operation, real-time intraoperative images are acquired through the endoscope to obtain endoscopic images of the target area.
[0149] Step S402 , rigidly registering the initial three-dimensional model according to the endoscopic image to obtain a rigidly registered reference three-dimensional model.
[0150] After obtaining the initial three-dimensional model of the preoperative target part and the endoscopic image of the intraoperative target part, the initial three-dimensional model can be rigidly registered according to the endoscopic image to obtain a reference three-dimensional model after rigid registration.
[0151] The initial three-dimensional model is rigidly registered according to the endoscopic image, and the initial three-dimensional model can be automatically registered with the help of an optical positioning device, or manually registered with the initial three-dimensional model.
[0152] Step S403: determining at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image.
[0153] Among them, at least one model key point is a partial point on a feature line corresponding to the target part in the reference three-dimensional model, and at least one image key point is a partial point on a feature line corresponding to the target part in the endoscopic image.
[0154] Optionally, you can Figure 5 The process shown in is used to determine the image key points corresponding to each model key point. Specifically, Figure 5 As shown, the following steps may be included:
[0155] Step S501, projecting each model key point and each image key point into a target coordinate system in the same dimension.
[0156] Since the coordinate system of each model key point is a three-dimensional coordinate system, and the coordinate system of each image key point is a two-dimensional coordinate system, when determining the correspondence between each model key point and each image key point, it is necessary to project each model key point and each image key point into the target coordinate system of the same dimension.
[0157] Specifically, each model key point may be projected into a two-dimensional coordinate system, and the coordinate system of each image key point remains unchanged; or each image key point may be projected into a three-dimensional coordinate system, and the coordinate system of each model key point remains unchanged.
[0158] Step S502: According to the position information of each model key point and the position information of each image key point, at least one model key point and at least one image key point are grouped into a plurality of feature point groups.
[0159] Each feature point group includes a model key point and an image key point, and the image key point corresponding to the model key point in a feature point group is the image key point that is closest to the model key point among at least one image key point.
[0160] Step S503: Generate a covariance matrix according to the difference in position information between the model key points and the image key points in each feature point group.
[0161] Step S504, performing eigenvalue decomposition on the covariance matrix to obtain a transformation matrix, and performing transformation processing on each model key point according to the transformation matrix.
[0162] Step S505, determining whether the mean distance between each model key point after transformation and the corresponding image key point is greater than a preset first distance threshold; if not, executing step S506; if yes, executing step S507.
[0163] Step S506: determining the image key point corresponding to each model key point according to the multiple feature point groups.
[0164] Step S507: Use each model key point after transformation to replace the model key points in the plurality of feature point groups.
[0165] After replacing each model key point with each model key point after transformation processing, you can return to step S502 and repeat the steps of determining multiple feature point groups based on the position information of each model key point and the position information of each image key point, and generating a covariance matrix based on the difference in position information between the model key points and the image key points in each feature point group, until the mean distance between each model key point after transformation processing and the corresponding image key point is no greater than the preset first distance threshold.
[0166] Step S404: Determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of each image key point.
[0167] For each model key point, the distance information and direction information between the model key point and the corresponding image key point can be determined respectively according to the position information of the model key point and the position information of the corresponding image key point, and the displacement information between each model key point and the corresponding image key point can be obtained according to the distance information and direction information between each model key point and the corresponding image key point.
[0168] Step S405 , performing a deformation operation on the tetrahedral mesh corresponding to the reference three-dimensional model according to the determined displacement information, to obtain a target three-dimensional model after deformation and registration.
[0169] After obtaining the reference 3D model after rigid registration, the reference 3D model can be converted into a tetrahedral mesh. Specifically, the following can be used: Figure 6 The flowchart shown converts the reference 3D model into a tetrahedral mesh, including the following steps:
[0170] Step S601 , performing mesh simplification and mesh repair processing on the surface mesh of the reference three-dimensional model to obtain a repaired three-dimensional model.
[0171] The mesh simplification process is used to reduce the number of triangles or polygonal meshes in the surface mesh of the reference three-dimensional model, and the mesh repair process is used to repair the mesh quality of the surface mesh of the reference three-dimensional model.
[0172] Step S602 , meshing the repaired three-dimensional model with tetrahedral units to obtain a tetrahedral mesh corresponding to the reference three-dimensional model.
[0173] After the surface mesh of the reference three-dimensional model is mesh-simplified and mesh-repaired, the obtained repaired three-dimensional model is tetrahedralized, that is, the repaired three-dimensional model is meshed by tetrahedral units to generate a corresponding tetrahedral mesh.
[0174] After obtaining the tetrahedral mesh corresponding to the reference three-dimensional model, a deformation operation can be performed on the tetrahedral mesh according to the determined displacement information to obtain the target three-dimensional model after deformation registration. Figure 7a The process shown in FIG. 1 is used to obtain the target three-dimensional model, such as Figure 7a As shown, the following steps are included:
[0175] Step S701 : Map at least one model key point to mesh vertices of a tetrahedral mesh corresponding to a reference three-dimensional model, and obtain a model mesh vertex corresponding to each model key point.
[0176] Step S702 , performing the following operations for each model mesh vertex corresponding to any model key point: determining a first moving distance for moving the model mesh vertex this time according to distance information between the model key point and the corresponding image key point.
[0177] Specifically, for a model mesh vertex corresponding to any model key point, the magnitude of the external force applied to the model mesh vertex corresponding to the model key point is determined according to the distance information between the model key point and the corresponding image key point.
[0178] The magnitude of the external force is proportional to the distance between the model key point and the corresponding image key point. The greater the distance, the greater the external force, and the smaller the distance, the smaller the external force.
[0179] In the embodiment of the present application, the external force applied to the model mesh vertex can be determined by the following formula:
[0180]
[0181] in, represents external force, f basic is a constant, indicating the magnitude of the basic external force, represents the direction between the model keypoint and the corresponding image keypoint, Represents the distance between the model keypoint and the corresponding image keypoint.
[0182] After determining the magnitude of the external force applied to the model mesh vertex, the first moving distance of the model mesh vertex can be determined according to the following formula:
[0183] Δx1=Δt 2 M -1 f ext
[0184] Among them, Δx1 is the first moving distance, Δt is the set time step, M is the mass matrix, and f ext is the magnitude of the external force applied to the model mesh vertices.
[0185] Step S703: according to the direction information between the model key point and the corresponding image key point, the model mesh vertex is moved by a first moving distance within a set time step.
[0186] Step S704, determining whether the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is greater than a preset second distance threshold; if not, executing step S705; if yes, executing step S706.
[0187] Step S705, determining to move each model mesh vertex to the position of the corresponding image key point.
[0188] Step S706: Using the moved model mesh vertices as target model mesh vertices.
[0189] Step S707, performing the following operations for any target model mesh vertex: determining a second moving distance for moving the target model mesh vertex this time according to the distance information between the model key point and the corresponding image key point and the first moving distance.
[0190] Specifically, according to Figure 7b The process shown in FIG. 1 is used to determine the second moving distance for moving the target model mesh vertex this time, such as Figure 7b As shown, the following steps are included:
[0191] Step S7071, determining a target speed for moving the target model mesh vertices according to the first moving distance and the set time step.
[0192] Among them, the target speed of moving the target model mesh vertex can be determined by the following formula:
[0193] v=Δx1 / Δt
[0194] Among them, v is the target speed, Δx1 is the first moving distance, and Δt is the set time step.
[0195] Step S7072, determining the accelerated moving distance for moving the target model mesh vertices according to the target speed and the set time step.
[0196] The accelerated moving distance for moving the target model mesh vertex can be determined by the following formula:
[0197] Δx g =Δtv
[0198] Where Δx g is the acceleration moving distance, v is the target speed, and Δt is the set time step.
[0199] Step S7073, determining the deformation movement distance for moving the target model mesh vertex according to the distance information between the model key point and the corresponding image key point and the set time step.
[0200] Specifically, first, the magnitude of the external force applied to the target model mesh vertex is determined according to the distance information between the model key point and the corresponding image key point. Then, the deformation movement distance Δx of the target model mesh vertex is determined according to the magnitude of the external force applied to the target model mesh vertex and the set time step.s .
[0201] Step S7074: Determine a second moving distance for moving the target model mesh vertices this time according to the accelerated moving distance and the deformed moving distance.
[0202] The second moving distance can be determined by the following formula:
[0203] Δx2=Δx g +Δx s
[0204] Among them, Δx2 is the second moving distance, Δx g is the acceleration distance, Δx s is the deformation movement distance.
[0205] Step S708: moving the target model mesh vertex by a second moving distance within a set time step according to the direction information between the image key point and the corresponding model key point.
[0206] Step S709, determining whether the sum of the distances between each of the moved target model mesh vertices and the corresponding image key points is greater than a preset second distance threshold; if not, executing step S710; if yes, executing step S711.
[0207] Step S710, determining to move each target model mesh vertex to the position of the corresponding image key point.
[0208] Step S711: Use each moved target model mesh vertex to replace each target model mesh vertex, and use each second moving distance to replace each first moving distance.
[0209] After replacing each model key point with each target model mesh vertex after movement, and replacing each first moving distance with each second moving distance, you can return to step S707 and repeat the steps of determining the second moving distance for moving the target model mesh vertex this time according to the distance information between the model key point and the corresponding image key point, and the first moving distance, and moving the target model mesh vertex the second moving distance within a set time step according to the direction information between the image key point and the corresponding model key point, until the sum of the distances between each target model mesh vertex after movement and the corresponding image key point is no greater than the preset second distance threshold.
[0210] In other embodiments, the three-dimensional model registration method performed by the medical terminal can also be performed according to Figure 8 The process shown is implemented. Figure 8 As shown, the following steps may be included:
[0211] Step S801 : determining feature lines corresponding to the target part in the endoscopic image and the reference three-dimensional model respectively.
[0212] The endoscopic image is obtained by collecting images of the target part using an endoscope during the operation, and the reference 3D model is an initial 3D model obtained by photographing the target part and 3D reconstruction before the operation, which is obtained after rigid registration with the endoscopic image.
[0213] When determining the feature lines corresponding to the target parts in the endoscopic image and the reference three-dimensional model, manual marking can be used, or artificial intelligence algorithms can be used for automatic extraction and classification.
[0214] Optionally, when the target site is the liver, the determined characteristic lines can be the lower ridge line, posterior contour and falciform ligament of the liver, that is, the lower ridge line, posterior contour and falciform ligament of the liver can be determined in the endoscopic image, and the lower ridge line, posterior contour and falciform ligament of the liver can be determined on the reference three-dimensional model.
[0215] For example, the lower ridge and posterior contour of the liver determined in the endoscopic image can be as follows: Figure 9a As shown. Among them, Figure 9a The solid line in is the lower ridge of the liver in the endoscopic image. Figure 9a The dotted line in is the posterior contour of the liver in the endoscopic image.
[0216] The lower ridge and posterior contour of the liver determined on the three-dimensional model can be Figure 9b As shown. Among them, Figure 9b The black solid line in the figure is the lower ridge of the liver on the reference 3D model. Figure 9b The white dashed line in the figure is the posterior outline of the liver on the reference 3D model.
[0217] Step S802, determining at least one image key point located on a feature line corresponding to the target part in the endoscopic image, and at least one model key point located on a feature line corresponding to the target part in the reference three-dimensional model.
[0218] When determining at least one key point on a feature line, it can be determined by manual marking or by downsampling. The first and last points on the feature line can be connected to obtain a straight line, and the distance between the middle point on the feature line and the straight line can be determined. If the distance is less than a set threshold, the middle point is used as a key point. Otherwise, the feature line is divided into two sub-feature lines according to the middle point, and each sub-feature line is used as a feature line. The process of connecting the first and last points on the feature line to obtain a straight line, determining the distance between the middle point on the feature line and the straight line, and judging whether the distance is less than the set threshold is repeated until all points that meet the conditions on the feature line are determined and these points are used as key points.
[0219] Step S803, matching each model key point with each image key point according to the position information of each model key point and the position information of each image key point, and determining the image key point corresponding to each model key point.
[0220] Among them, the specific process of determining the image key points corresponding to each model key point can be as follows Figure 5 The process shown in is used for execution, and the embodiments of the present application are not described in detail here.
[0221] Step S804: Determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of the corresponding image key point.
[0222] The displacement information includes distance information and direction information.
[0223] Step S805 , performing mesh simplification processing, mesh repair processing, and tetrahedral unit meshing on the surface mesh of the reference three-dimensional model to obtain a corresponding tetrahedral mesh.
[0224] Step S806: Map at least one model key point to mesh vertices of the tetrahedral mesh respectively to obtain a model mesh vertex corresponding to each model key point.
[0225] For example, when the target site is the liver, Figure 9b The surface mesh of the reference 3D model in is meshed by mesh simplification, mesh repair, and tetrahedral unit meshing to obtain the following Fig.10 The tetrahedral mesh shown.
[0226] After obtaining the tetrahedral mesh corresponding to the reference three-dimensional model of the liver, at least one model key point located on the lower ridge line of the liver in the reference three-dimensional model can be mapped to the mesh vertices of the tetrahedral mesh to obtain the model mesh vertex corresponding to each model key point. Fig.10As shown, each white point on the edge of the tetrahedral mesh is the vertices of the model mesh obtained by mapping.
[0227] Step S807, according to the displacement information between each model key point and the corresponding image key point, each model mesh vertex is moved to the position of the corresponding image key point, and the three-dimensional model obtained after the movement of each model mesh vertex is used as the target three-dimensional model after deformation registration.
[0228] For each model key point, after mapping the model key point to the mesh vertex of the tetrahedral mesh and obtaining the corresponding model mesh vertex, the external force applied to the model mesh vertex can be determined based on the displacement information between the model key point and the corresponding image key point.
[0229] After determining the external force applied to the model mesh vertex corresponding to each model key point, a Position Based Dynamic (PBD) algorithm can be used, using the Neo hooken elastic model, with the determined external force applied to each model mesh vertex as input, and the Young modulus of the target part (e.g., when the current target part is the liver, the normal liver is 60000Pa) and the Possion ratio (e.g., when the target part is the liver, the Possion ratio can be 0.49). Among them, the Young modulus reflects the elasticity metric of the target part, and the Possion ratio reflects the incompressible property of the target part. The value range of the Possion ratio is 0-0.5, and the closer the Possion ratio is to 0.5, the more incompressible the object is. The number of iterations and time steps for solving the input equation are input, and the elastic deformation simulation of the tetrahedral mesh obtained in step S805 is performed until the total error between each model mesh vertex on the tetrahedral mesh and each corresponding image key point on the endoscopic image is less than a certain threshold, and the iteration stops.
[0230] Specifically, for each model mesh vertex, a step of explicit Euler iteration is performed first:
[0231] x n+1 =x n +Δtv n +Δt 2 M -1 f ext
[0232] Where x represents the model mesh vertex, v represents the velocity of the model mesh vertex, Δt is the time step, M is the mass matrix, and f ext Indicates the magnitude of external force.
[0233] That is, for each model mesh vertex, the n+1 position to which the model mesh vertex is to be moved this time needs to be determined based on the current position n of the model mesh vertex and the moving distance, and the moving distance is determined by the speed v of this movement, the time step Δt, and the external force magnitude f. ext to be sure.
[0234] Secondly, λ0=0
[0235] when When executing:
[0236] Determine Δλ using the following formula:
[0237]
[0238] Determine Δx using the following formula:
[0239]
[0240] After determining Δλ and Δx, we can update and λ k+1 =λ k +Δλ.
[0241] And k = k + 1,
[0242] Among them, λ k represents the Lagrange multiplier coefficient, Ψ is the Neo hookean strain tensor, For elastic force.
[0243] Ψ can be determined by the following formula:
[0244]
[0245] This can be determined by:
[0246]
[0247] in, F represents the deformation matrix, and J=|F| is the determinant of the deformation F.
[0248] For example, after mapping at least one model key point located on the lower ridgeline of the liver in the reference three-dimensional model to the mesh vertices of the tetrahedral mesh and obtaining the corresponding model mesh vertices, contour constraints can be set for the corresponding model mesh vertices according to the image key points on the lower ridgeline of the liver in the endoscopic image. Fig.11As shown, the white three-dimensional area represents the tetrahedral mesh, the white points on the edge of the tetrahedral mesh represent the vertices of the model mesh, the gray line represents the lower ridge line of the liver in the endoscopic image, and the black points represent the image key points on the lower ridge line of the liver in the endoscopic image.
[0249] Then, an external force is applied to each model mesh vertex, the magnitude of the external force is determined according to the distance information between the corresponding model key point and the corresponding image key point, and the direction of the external force is determined according to the direction information between the corresponding model key point and the corresponding image key point. The effect of applying an external force to each model mesh vertex so that the tetrahedral mesh is deformed can be shown as follows: Fig.12 As shown by Fig.12 It can be seen that each model mesh vertex is moving closer to the position of the corresponding image key point.
[0250] Finally, based on the contour constraints composed of the image key points on the lower ridge of the liver in the endoscopic image, the corresponding model mesh vertices are moved multiple times. When the sum of the distances between the moved model mesh vertices and the corresponding image key points is not greater than the set threshold, the movement of the model mesh vertices can be stopped. The final deformation result of the tetrahedral mesh can be shown as follows: Fig.13 As shown. Fig.13 It can be seen that each model mesh vertex on the tetrahedral mesh is basically moved to the position of the corresponding image key point.
[0251] Furthermore, in the process of iterative deformation of the tetrahedral mesh, each model mesh vertex is transformed from Fig.11 Move to the position shown in Fig.13 During the process of changing the position shown in FIG. 1 , the change process of the sum of the distances between each model mesh vertex and the corresponding image key point can be shown as follows: Fig.14 shown. Fig.14 Only the change of the sum of the distances between some corresponding points is shown, and the threshold value set in the embodiment of the present application is 0.01, then Fig.14 It can be seen that when the sum of the distances between corresponding points is less than 0.01, the tetrahedral mesh will no longer perform iterative deformation.
[0252] According to the displacement information between each model key point and the corresponding image key point, the comparison between the tetrahedron mesh obtained after deformation of each model mesh vertex and the tetrahedron mesh before deformation can be as follows: Fig.15 As shown. Among them, Fig.15 The white area in the figure is the tetrahedral mesh after deformation, and the black area is the tetrahedral mesh before deformation.
[0253] The three-dimensional model registration method provided in the embodiment of the present application can, after obtaining the reference three-dimensional model of the preoperative target part after rigid registration, match at least one model key point corresponding to the target part in the reference three-dimensional model with at least one image key point corresponding to the target part in the intraoperative endoscopic image, determine the image key point corresponding to each model key point, and based on the position information of each model key point and the position information of the corresponding image key point, determine the displacement information between each model key point and the corresponding image key point, and map each model key point to the mesh vertices of the tetrahedral mesh corresponding to the reference three-dimensional model, so that based on the obtained displacement information, the mesh vertices of the tetrahedral mesh are deformed to obtain the target three-dimensional model after deformation registration. The three-dimensional model registration method in the embodiment of the present application does not need to consider the external force applied by the outside world, but only considers the deformation result of the target part after the external force acts, and matches the contour of the three-dimensional model obtained according to the preoperative reconstruction with the target part in the endoscopic image, so that the mesh deformation of the tetrahedral mesh corresponding to the target part can be physically achieved and kept consistent with the edge of the target part in the endoscopic image, thereby being able to more correctly display the position of blood vessels and lesions in the target part, more accurately guide the precise resection of lesions during surgery, and improve the safety of the surgery.
[0254] and Figure 4 The three-dimensional model registration method shown is based on the same inventive concept, and a three-dimensional model registration device is also provided in the embodiment of the present application, and the three-dimensional model registration device can be arranged in the medical terminal. Since the device is the device corresponding to the three-dimensional model registration method of the present application, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated.
[0255] Fig.16 A schematic diagram of the structure of a three-dimensional model registration device provided in an embodiment of the present application is shown. Fig.16 As shown, the three-dimensional model registration device includes a rigid registration unit 1601, a key point determination unit 1602, a displacement information determination unit 1603 and a deformation registration unit 1604.
[0256] The rigid registration unit 1601 is used to obtain an initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before surgery, and an endoscopic image of the target part collected during surgery; rigidly register the initial three-dimensional model according to the endoscopic image to obtain a reference three-dimensional model after rigid registration;
[0257] A key point determination unit 1602 is used to determine at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image; at least one model key point is a partial point on a feature line corresponding to the target part in the reference three-dimensional model, and at least one image key point is a partial point on a feature line corresponding to the target part in the endoscopic image;
[0258] The displacement information determining unit 1603 is used to determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of each image key point;
[0259] The deformation registration unit 1604 is used to perform deformation operations on the tetrahedral mesh corresponding to the reference three-dimensional model according to the determined displacement information to obtain a target three-dimensional model after deformation registration.
[0260] In an optional embodiment, if Fig.17 As shown, the above device may further include a corresponding point determining unit 1701, which is used to:
[0261] Project each model key point and each image key point into the target coordinate system of the same dimension;
[0262] According to the position information of each model key point and the position information of each image key point, at least one model key point and at least one image key point are grouped into a plurality of feature point groups; wherein each feature point group includes a model key point and an image key point, and the image key point corresponding to the model key point in a feature point group is the image key point that is closest to the model key point in at least one image key point;
[0263] Generate a covariance matrix based on the difference in position information between the model key points and the image key points in each feature point group; perform eigenvalue decomposition on the covariance matrix to obtain a transformation matrix, and perform transformation processing on each model key point based on the transformation matrix;
[0264] If the average distance between each model key point after the transformation and the corresponding image key point is not greater than a preset first distance threshold, then determining the image key point corresponding to each model key point according to the multiple feature point groups;
[0265] If the average distance between each model key point after the transformation and the corresponding image key point is greater than a preset first distance threshold, the model key points in multiple feature point groups are replaced by each model key point after the transformation, and the step of generating a covariance matrix based on the difference in position information between the model key points and the image key points in each feature point group is returned.
[0266] In an optional embodiment, the deformation registration unit 1604 is specifically configured to:
[0267] Mapping at least one model key point to mesh vertices of a tetrahedral mesh corresponding to the reference three-dimensional model, respectively, to obtain a model mesh vertex corresponding to each model key point;
[0268] According to the displacement information between each model key point and the corresponding image key point, each model mesh vertex is moved to the position of the corresponding image key point, and the three-dimensional model obtained after moving each model mesh vertex is used as the target three-dimensional model after deformation registration.
[0269] In an optional embodiment, the displacement information includes distance information and direction information, and the deformation registration unit 1604 is specifically used to:
[0270] The following operations are performed for the model mesh vertex corresponding to any model key point: according to the distance information between the model key point and the corresponding image key point, the first moving distance of the model mesh vertex is determined; and according to the direction information between the model key point and the corresponding image key point, the model mesh vertex is moved by the first moving distance within a set time step;
[0271] If the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is not greater than a preset second distance threshold, it is determined to move each of the model mesh vertices to the positions of the corresponding image key points.
[0272] In an optional embodiment, the deformation registration unit 1604 is further configured to:
[0273] If the sum of the distances between the moved model mesh vertices and the corresponding image key points is greater than a preset second distance threshold, the moved model mesh vertices are used as target model mesh vertices, and the following operations are performed for any target model mesh vertex:
[0274] Determine a second moving distance for moving the target model mesh vertex this time according to the distance information between the model key point and the corresponding image key point and the first moving distance;
[0275] According to the direction information between the image key point and the corresponding model key point, the target model mesh vertices are moved by a second moving distance within a set time step until the sum of the distances between each target model mesh vertex and the corresponding image key point after the move is no greater than a preset second distance threshold.
[0276] In an optional embodiment, the deformation registration unit 1604 is further configured to:
[0277] Determining a target speed for moving the mesh vertices of the target model according to the first moving distance and the set time step;
[0278] Determine the acceleration moving distance of the target model mesh vertex according to the target speed and the set time step, and determine the deformation moving distance of the target model mesh vertex according to the distance information between the model key point and the corresponding image key point and the set time step;
[0279] A second moving distance for moving the mesh vertices of the target model is determined according to the accelerated moving distance and the deformed moving distance.
[0280] In an optional embodiment, if Fig.17 As shown, the above apparatus may further include a tetrahedron generating unit 1702, which is used to:
[0281] Performing mesh simplification and mesh repair on the surface mesh of the reference three-dimensional model to obtain a repaired three-dimensional model;
[0282] The repaired three-dimensional model is meshed with tetrahedral units to obtain a tetrahedral mesh corresponding to the reference three-dimensional model.
[0283] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the registration method of the three-dimensional model in the above-mentioned embodiment.
[0284] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0285] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A medical terminal, characterized in that: include: A transceiver unit, used for receiving an endoscopic image of a target part acquired by an endoscope during surgery; A memory, used to store an initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before surgery; A processor, configured to perform rigid registration on the initial three-dimensional model according to the endoscopic image to obtain a rigidly registered reference three-dimensional model; Determining at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image; The at least one model key point is a partial point on a feature line corresponding to the target part in the reference three-dimensional model, and the at least one image key point is a partial point on a feature line corresponding to the target part in the endoscopic image; Determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of each image key point; The at least one model key point is respectively mapped to the mesh vertices of the tetrahedral mesh corresponding to the reference three-dimensional model to obtain the model mesh vertex corresponding to each model key point; according to the displacement information between each model key point and the corresponding image key point, each model mesh vertex is moved to the position of the corresponding image key point, and the three-dimensional model obtained after the movement of each model mesh vertex is used as the target three-dimensional model after deformation alignment.
2. The medical terminal according to claim 1, characterized in that: The processor is specifically used for: Project each model key point and each image key point into the target coordinate system of the same dimension; According to the position information of each model key point and the position information of each image key point, the at least one model key point and the at least one image key point are grouped into a plurality of feature point groups; Each feature point group includes a model key point and an image key point, and the image key point corresponding to the model key point in a feature point group is the image key point that is closest to the model key point among the at least one image key point; Generate a covariance matrix according to the difference in position information between the model key points and the image key points in each feature point group; perform eigenvalue decomposition on the covariance matrix to obtain a transformation matrix, and perform transformation processing on each model key point according to the transformation matrix; If the average distance between each model key point after the transformation and the corresponding image key point is not greater than a preset first distance threshold, determining the image key point corresponding to each model key point according to the multiple feature point groups; If the mean distance between each model key point after the transformation and the corresponding image key point is greater than a preset first distance threshold, the model key points in the multiple feature point groups are replaced by the model key points after the transformation, and the step of generating a covariance matrix based on the difference in position information between the model key points and the image key points in each feature point group is returned to be executed.
3. The medical terminal according to claim 1, characterized in that: The displacement information includes distance information and direction information, and the processor is specifically used for: The following operations are performed for the model mesh vertex corresponding to any model key point: determining a first moving distance for moving the model mesh vertex this time according to the distance information between the model key point and the corresponding image key point; and moving the model mesh vertex by the first moving distance within a set time step according to the direction information between the model key point and the corresponding image key point; If the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is not greater than a preset second distance threshold, it is determined to move each of the model mesh vertices to the positions of the corresponding image key points.
4. The medical terminal according to claim 3, characterized in that: The processor is further configured to: If the sum of the distances between each of the moved model mesh vertices and the corresponding image key points is greater than a preset second distance threshold, each of the moved model mesh vertices is used as each of the target model mesh vertices, and the following operations are performed for each of the target model mesh vertices: Determining a second moving distance for moving the mesh vertex of the target model this time according to the distance information between the model key point and the corresponding image key point and the first moving distance; According to the direction information between the image key point and the corresponding model key point, the target model mesh vertex is moved by the second moving distance within a set time step until the sum of the distances between each target model mesh vertex and the corresponding image key point after the move is no greater than a preset second distance threshold.
5. The medical terminal according to claim 4, characterized in that: The processor is specifically used for: Determining a target speed for moving the mesh vertices of the target model according to the first moving distance and a set time step; Determine an accelerated moving distance for moving the target model mesh vertex according to the target speed and the set time step, and determine a deformed moving distance for moving the target model mesh vertex according to distance information between the model key point and the corresponding image key point and the set time step; A second moving distance for moving the mesh vertices of the target model this time is determined according to the accelerated moving distance and the deformed moving distance.
6. The medical terminal according to claim 1, characterized in that: The processor is specifically used for: Performing mesh simplification and mesh repair on the surface mesh of the reference three-dimensional model to obtain a repaired three-dimensional model; The repaired three-dimensional model is meshed with tetrahedral units to obtain a tetrahedral mesh corresponding to the reference three-dimensional model.
7. A three-dimensional model registration method, characterized in that: include: Acquire an initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before surgery, and an endoscopic image of the target part collected during surgery; Performing rigid registration on the initial three-dimensional model according to the endoscopic image to obtain a rigidly registered reference three-dimensional model; Determining at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image; The at least one model key point is a partial point on a feature line corresponding to the target part in the reference three-dimensional model, and the at least one image key point is a partial point on a feature line corresponding to the target part in the endoscopic image; Determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of each image key point; The at least one model key point is respectively mapped to the mesh vertices of the tetrahedral mesh corresponding to the reference three-dimensional model to obtain the model mesh vertex corresponding to each model key point; according to the displacement information between each model key point and the corresponding image key point, each model mesh vertex is moved to the position of the corresponding image key point, and the three-dimensional model obtained after the movement of each model mesh vertex is used as the target three-dimensional model after deformation alignment.
8. The method according to claim 7, characterized in that Determine the image keypoints corresponding to each model keypoint in the following way: Project each model key point and each image key point into the target coordinate system of the same dimension; According to the position information of each model key point and the position information of each image key point, the at least one model key point and the at least one image key point are grouped into a plurality of feature point groups; Each feature point group includes a model key point and an image key point, and the image key point corresponding to the model key point in a feature point group is the image key point that is closest to the model key point among the at least one image key point; Generate a covariance matrix according to the difference in position information between the model key points and the image key points in each feature point group; perform eigenvalue decomposition on the covariance matrix to obtain a transformation matrix, and perform transformation processing on each model key point according to the transformation matrix; If the average distance between each model key point after the transformation and the corresponding image key point is not greater than a preset first distance threshold, determining the image key point corresponding to each model key point according to the multiple feature point groups; If the mean distance between each model key point after the transformation and the corresponding image key point is greater than a preset first distance threshold, the model key points in the multiple feature point groups are replaced by the model key points after the transformation, and the step of generating a covariance matrix based on the difference in position information between the model key points and the image key points in each feature point group is returned to be executed.
9. A three-dimensional model registration device, characterized in that: include: A rigid registration unit is used to obtain an initial three-dimensional model obtained by photographing and three-dimensionally reconstructing the target part before surgery, and an endoscopic image of the target part collected during surgery; rigidly register the initial three-dimensional model according to the endoscopic image to obtain a reference three-dimensional model after rigid registration; a key point determination unit, configured to determine at least one model key point corresponding to the reference three-dimensional model and at least one image key point corresponding to the endoscopic image; The at least one model key point is a partial point on a feature line corresponding to the target part in the reference three-dimensional model, and the at least one image key point is a partial point on a feature line corresponding to the target part in the endoscopic image; A displacement information determination unit, used to determine the displacement information between each model key point and the corresponding image key point according to the position information of each model key point and the position information of each image key point; A deformation registration unit, used for mapping the at least one model key point to mesh vertices of the tetrahedral mesh corresponding to the reference three-dimensional model, to obtain a model mesh vertex corresponding to each model key point; According to the displacement information between each model key point and the corresponding image key point, each model mesh vertex is moved to the position of the corresponding image key point, and the three-dimensional model obtained after moving each model mesh vertex is used as the target three-dimensional model after deformation registration.
