3D surface rigid registration method, system and electronic device assisted by geometric information

Through the geometric information-assisted 3D surface rigid registration method, combined with the geometric Transformer model and Sinkhorn iterative algorithm, the problem of difficulty in 3D point cloud registration in surgery is solved, the registration accuracy and effect are improved, and it is suitable for space allocation in surgical navigation allow.

CN118505763BActive Publication Date: 2024-11-05ZHEJIANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410956320.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-05
Estimated Expiration
2044-07-17

AI Technical Summary

Technical Problem

The prior art has difficulty in registering 3D point clouds due to changes in the patient's position and soft tissue deformation during the operation, and the quality of the point clouds reconstructed intraoperatively is uneven, which affects the registration results.

Method used

Using geometric information-assisted 3D surface rigid registration method, the transformation matrix module is used to calculate the rigid transformation matrix for point cloud registration through feature extraction, weighting and solving, and the transformation matrix module is combined with the geometric Transformer model and Sinkhorn iterative algorithm, which can reduce the reconstruction of intraoperative endoscopy. Data quality requirements.

Benefits of technology

It improves the accuracy and effect of 3D/3D registration, reduces the impact of point cloud reconstruction quality, enhances feature expression, is suitable for spatial registration in surgical navigation, and provides a more stable registration solution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118505763B_ABST
    Figure CN118505763B_ABST
Patent Text Reader

Abstract

The present invention discloses a geometric information-assisted 3D surface rigid registration method, system and electronic device, which belongs to the field of point cloud registration. The present invention samples the first surface point cloud of the preoperative 3D model and the second surface point cloud of the intraoperative 3D model to the same dimension as the input of the 3D surface rigid registration model, and sequentially passes through the feature extraction module, feature weighting module and transformation matrix solving module in the 3D surface rigid registration model to obtain the rigid transformation matrix between the two surface point clouds, and then uses the rigid transformation matrix to register the two surface point clouds. The 3D surface rigid registration model can deeply explore the local (curvature) and global (normal direction) geometric features of the point cloud, which can not only reduce the impact of the point cloud reconstruction quality, but also enhance the feature expression, and improve the 3D / 3D registration effect. The present invention can provide a new solution for spatial registration in surgical navigation and has broad application prospects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional point cloud registration, and in particular relates to a geometric information-assisted 3D surface rigid registration method, system and electronic equipment. Background Art

[0002] The liver is the largest solid organ in the human body, with a large number of irregularly distributed arteries, veins and bile ducts. The complex anatomical structure of the liver makes it prone to accidental damage during surgical operations such as liver resection, resulting in a series of intraoperative or postoperative complications.

[0003] Currently, augmented reality technology can fuse the 3D liver surface reconstructed by laparoscopic video with the preoperative 3D liver model, saving surgeons some cognitive burden during surgery. The fusion display can provide precise surgical positioning and guidance, helping surgeons to accurately locate the lesion while avoiding damage to normal tissues and organs.

[0004] In order to ensure the safety of laparoscopic liver surgery and avoid massive bleeding, incomplete or excessive lesion treatment caused by intraoperative operational errors, high-precision 3D registration is required to correctly fuse the 3D liver reconstructed by laparoscopic video with the preoperative 3D liver model. By obtaining the real-time status of the liver surgical area, dynamic registration of preoperative and intraoperative information is achieved, in the hope that the preoperatively reconstructed 3D model can be consistent in spatial position with the intraoperative positioning information and in anatomical structure with the images obtained during the operation.

[0005] Therefore, 3D-3D registration has the ability to produce either rigid or non-rigid alignment results. Rigid registration allows global alignment of the preoperative 3D CT model with the intraoperative 3D surface using manually or automatically detected landmarks, regardless of any deformation of the tissue. Non-rigid registration is often used for deformation alignment after an initial rigid alignment. The clinical standard registration error for laparoscopic liver MIS is estimated to be 5 mm, while the accuracy of rigid semi-automatic registration has been clinically validated to be approximately 13.9 mm.

[0006] However, in actual operation, the existing automatic 3D-3D registration process still has the following problems:

[0007] 1. During the operation, due to the obvious changes in the patient's body position and interference from muscle movement in the body, coupled with the large-scale soft tissue deformation caused by the surgeon's operation of instruments to squeeze or cut the liver, the relative position relationship between the instrument and the image is offset, making it difficult to align the intraoperative state with the preoperative state.

[0008] 2. There is a common problem between 3D liver point clouds reconstructed by CT and intraoperatively: compared with the 3D liver point cloud reconstructed by CT, the 3D point cloud acquired intraoperatively has a high density and low reconstruction quality, resulting in the alignment of sparse-dense point clouds being disturbed by more noise points, affecting the registration results.

[0009] 3. The quality of the point cloud of the liver reconstructed during surgery varies, and the surface conditions of the 3D liver models reconstructed by different tools are different. At present, the preprocessing of deep learning training data involves the step of downsampling the point cloud, which not only fails to eliminate interference points, but also may reduce the probability of high-quality corresponding point pairs between the 3D point clouds reconstructed during surgery and before surgery.

[0010] As a key technology in the field of surgical navigation, the high-accuracy 3D / 3D registration technology can register the real-time image data of the endoscope with the three-dimensional simulated point cloud data reproduced by CT data, realize multimodal data fusion display during the operation, achieve augmented reality display effects, provide reference for doctors' surgical operations, and provide guarantee for surgical safety.

[0011] Most learning-based 3D registration methods are coordinate-only and ignore a lot of geometric information in point clouds, which makes the learned features less geometrically identifiable and easily leads to incorrect correspondences. In fact, the geometric structure information in point clouds can provide important clues for point cloud registration tasks. EGST proposes an enhanced geometric structure transformer to learn enhanced contextual features of geometric structures in point clouds and model the structural consistency between point clouds. It explores three geometric descriptors of point cloud coordinate calculations - surface normals, distances between points, and angles between vectors to extract reliable correspondences.

[0012] The use of Transformer shows good promise in fusing contextual information in point cloud features, but the potential of incorporating geometric information is still being explored. GeoTransformer and UTOPIC combine the geometric transformer to embed local geometric features based on global feature learning to achieve robust overlapping point matching. GeoTransformer combines the geometric transformer to embed local geometric features based on global feature learning to achieve robust overlapping point matching. GeoTransformer encodes pairwise distances and triplet angles of point clouds, making it robust in low overlap situations and invariant to rigid transformations, but its lack of key surface normal information hinders the representation of the external structure of the point cloud. Since GeoTransformer and other geometric transformer-based methods are specifically designed for point cloud registration problems in life scenarios, they are relatively lacking in direct geometric expressions of liver point cloud surfaces in clinical practice, which makes them unable to cope well with the point cloud registration problem of the liver. Of course, in addition to the liver, other internal organs of the human body also have the same point cloud registration problem.

[0013] Therefore, it is of great significance and value to conduct in-depth research on the high-precision 3D / 3D registration problem of the organ 3D point cloud reconstructed by intraoperative stereo endoscope and the 3D model reconstructed by preoperative CT. Therefore, it is of great significance and value to conduct in-depth research on the 3D / 3D registration method that can be realized only by knowing the intraoperative and intraoperative organ 3D point cloud pairs. Summary of the invention

[0014] The purpose of the present invention is to solve the problem that the 3D point cloud of an organ during surgery is difficult to accurately align, and to provide a 3D surface rigid alignment method, system and electronic equipment assisted by geometric information.

[0015] In order to achieve the above-mentioned invention object, the specific technical scheme adopted by the present invention is as follows:

[0016] In a first aspect, the present invention provides a geometric information-assisted 3D surface rigid registration method for realizing registration between an intraoperative 3D point cloud and a preoperative 3D point cloud for a target organ, comprising:

[0017] S1, reading a preoperative 3D model reconstructed based on a preoperative CT image of the target organ, and saving it as a first surface point cloud;

[0018] S2, reading the intraoperative 3D model reconstructed based on the intraoperative visual image of the target organ and saving it as a second surface point cloud;

[0019] S3, sampling the first surface point cloud and the second surface point cloud to the same dimension as the input of the 3D surface rigid registration model, and sequentially passing through the feature extraction module, feature weighting module and transformation matrix solving module in the 3D surface rigid registration model to obtain a rigid transformation matrix between the first surface point cloud and the second surface point cloud;

[0020] The feature extraction module is used to extract a first general feature and a first geometric feature from the first surface point cloud, and to extract a second general feature and a second geometric feature from the second surface point cloud;

[0021] The feature weighting module includes a representative point selector and a feature matching module; the representative point selector performs clustering and dimensionality reduction based on the normal direction of the point cloud, so as to obtain representative point masks corresponding to the first surface point cloud and the second surface point cloud respectively; the feature matching module obtains a first enhanced feature after geometric embedding by passing the first general feature and the first geometric feature through a geometric Transformer model, obtains a second enhanced feature after geometric embedding by passing the second general feature and the second geometric feature through a geometric Transformer model, and then outputs a point correspondence matrix by an exponential function after passing the first enhanced feature and the second enhanced feature through an affinity layer and a Sinkhorn iterative algorithm; finally, the first enhanced feature and the second enhanced feature are weighted with the point correspondence matrix and the representative point mask respectively to obtain a new first general feature and a new second general feature;

[0022] The transformation matrix solving module calls another feature matching module of the same structure, takes the new first general feature and the first geometric feature, the new second general feature and the second geometric feature as input, calculates a new point correspondence matrix, and finally obtains a rigid transformation matrix by singular value decomposition based on the new point correspondence matrix with the goal of minimizing the registration error;

[0023] S4. Use the rigid transformation matrix to align the first surface point cloud and the second surface point cloud.

[0024] As a preferred embodiment of the first aspect, the intraoperative visual image is an intraoperative endoscopic image of the patient obtained through an endoscope.

[0025] As a preferred embodiment of the first aspect, the feature extraction module includes a general feature extraction module and a geometric feature extraction module;

[0026] The universal feature extraction module is formed by cascading an AdaptConv network and a multi-layer perceptron, and is used to perform point-by-point feature learning on each input surface point cloud to obtain universal features;

[0027] The geometric feature extraction module is used to extract point pair distance features, triplet angle features, triplet perimeter difference features, Gaussian curvature distance features and average curvature distance features from each input surface point cloud, and concatenate the five features to obtain geometric features; the point pair distance feature is composed of the Euclidean distances of all point pairs in the point cloud; the triplet angle feature is composed of the triplet angles of all point pairs in the point cloud; the triplet perimeter difference feature is composed of the triplet perimeter differences of all point pairs in the point cloud; the Gaussian curvature feature is composed of the Gaussian curvature differences of all point pairs in the point cloud; and the average curvature feature is composed of the average curvature differences of all points in the point cloud.

[0028] As a preferred embodiment of the above-mentioned first aspect, in the representative point selector, each input surface point cloud is first divided into voxels to obtain a voxel set; then, for each voxel in the voxel set, all points contained in each voxel are K-means clustered based on the normal vectors of each point in the point cloud, and a local surface is formed by the points contained in each cluster; finally, for each local surface in each voxel, the point closest to the center of mass in the local surface is taken as the representative point of the local surface, and a representative point mask corresponding to each input surface point cloud is generated according to all the representative points in the voxel set.

[0029] As a preferred embodiment of the above first aspect, the target organ is human liver.

[0030] In a second aspect, the present invention provides a surgery-assisted virtual-real fusion system, comprising:

[0031] A preoperative 3D model construction module is used to reconstruct a preoperative 3D model based on the preoperative CT image of the target organ;

[0032] An intraoperative 3D model construction module is used to reconstruct an intraoperative 3D model based on the intraoperative visual image of the target organ;

[0033] A rigid registration module, configured to register the surface point clouds of the preoperative 3D model and the intraoperative 3D model according to the geometric information-assisted 3D surface rigid registration method described in any one of the first aspects above;

[0034] The image display module is used to fuse the virtual and real images of the registered pre-operative 3D model and the intra-operative 3D model to generate a virtual and real fused image for display.

[0035] As a preferred embodiment of the second aspect, in the preoperative 3D model construction module, the preoperative 3D model is reconstructed based on the preoperative CT image by using MITK software.

[0036] As a preferred embodiment of the second aspect, in the intraoperative 3D model construction module, the intraoperative 3D model is reconstructed based on the intraoperative visual image by a StaSiS-Net model.

[0037] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the geometric information-assisted 3D surface rigid registration method as described in any one of the schemes of the first aspect above is implemented.

[0038] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0039] The memory is used to store computer programs;

[0040] The processor is used to implement the geometric information-assisted 3D surface rigid registration method as described in any one of the solutions of the first aspect above when executing the computer program.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1) The present invention provides a low-overlap 3D surface rigid registration model assisted by geometric information, which can deeply mine the local (curvature) and global (normal direction) geometric features of the point cloud, which can not only reduce the impact on the point cloud reconstruction quality, but also enhance the feature expression and improve the 3D / 3D registration effect.

[0043] 2) The present invention can not only solve the problem of unstable effects of the current surgical navigation registration-related algorithms, but also reduce the requirements of the registration method on the quality of intraoperative endoscopic reconstruction data, achieving higher registration accuracy and better registration effects. The present invention can provide a new solution for spatial registration technology in surgical navigation, using a robust registration scheme to achieve better registration effects, and has a broader application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of the steps of the geometric information-assisted 3D surface rigid registration method of the present invention;

[0045] Figure 2 This is a schematic diagram of the network structure of the 3D surface rigid registration model;

[0046] Figure 3 Schematic diagram of the principle of the geometric Transformer model that combines curvature information;

[0047] Figure 4 This is a schematic diagram of the principle of the representative point electoral device. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0049] In the description of the present invention, it is to be understood that when an element is considered to be "connected" to another element, it may be directly connected to the other element or indirectly connected, that is, there are intermediate elements. On the contrary, when an element is said to be "directly" connected to another element, there are no intermediate elements.

[0050] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0051] like Figure 1 As shown, in a preferred embodiment of the present invention, a geometric information-assisted 3D surface rigid registration method is provided, which is used to realize the registration between the intraoperative 3D point cloud and the preoperative 3D point cloud for the target organ, and includes the following registration steps:

[0052] S1. Read the preoperative 3D model reconstructed based on the preoperative CT image of the target organ and save it as the first surface point cloud.

[0053] It should be noted that the preoperative 3D model in this step needs to be reconstructed in advance for reading by the present invention, and the reading of the preoperative 3D model can be online reading or offline reading, which is not limited. The target organ of the present invention can be any organ in the body that requires three-dimensional point cloud registration. In the embodiment of the present invention, the human liver is taken as an example for explanation.

[0054] The method of reconstructing the preoperative 3D model from the preoperative CT image of the present invention can be implemented by any existing technology. In an embodiment of the present invention, it can be implemented by using software that can realize 3D reconstruction of CT images, such as MITK. The preoperative 3D model can be saved in the form of a point cloud file (ply) to achieve subsequent registration.

[0055] S2. Read the intraoperative 3D model reconstructed based on the intraoperative visual image of the target organ and save it as a second surface point cloud.

[0056] It should be noted that in this step, the intraoperative 3D model needs to be reconstructed in advance for reading by the present invention, and reading the intraoperative 3D model can be online reading or offline reading, which is not limited.

[0057] In an embodiment of the present invention, an intraoperative endoscopic image of the patient obtained through an endoscope can be used as an intraoperative visual image to construct an intraoperative 3D model. The 3D model of the patient's liver is obtained during the operation. During the laparotomy, an experienced doctor can use CBCT, structured light scanning and other facilities to obtain it. In minimally invasive surgery, the open source project StaSiS-Net can be used to automatically reconstruct the 3D point cloud of the organ surface from the binocular image taken by the stereo endoscope, and multiple doctors can confirm whether it conforms to the actual surface morphology of the liver. However, it should be noted that the operation of the image acquisition device, the acquisition of images and the reconstruction of the model during minimally invasive surgery do not belong to the process that needs to be performed in the above-mentioned S2 step of the present invention. In addition, for surgeries on organs such as bronchi, the bronchoscope used is generally a monocular endoscope, so the specific intraoperative image type needs to be selected according to actual conditions.

[0058] S3. After sampling the first surface point cloud and the second surface point cloud to the same dimension, they are used as the input of the 3D surface rigid registration model, and the rigid transformation matrix between the first surface point cloud and the second surface point cloud is obtained by sequentially passing through the feature extraction module, feature weighting module and transformation matrix solving module in the 3D surface rigid registration model.

[0059] It should be noted that the above-mentioned downsampling of the first surface point cloud and the second surface point cloud can be determined according to the dimensionality requirement of the 3D surface rigid registration model for the input point cloud. The number of points in the two point clouds needs to be the same in order to facilitate registration.

[0060] The 3D surface rigid registration model in the present invention is the core of realizing the registration of two model surface point clouds, which includes three modules: feature extraction module, feature weighting module and transformation matrix solution module. The model structure is as follows: Figure 2 As shown below. Figure 2 The specific implementation process of each of the three modules is described in detail. For the convenience of description, the first surface point cloud and the second surface point cloud are respectively denoted as and .

[0061] The first module in the 3D surface rigid registration model of the present invention is a feature extraction module, which is used to extract the first surface point cloud from the first surface point cloud. Extract the first common feature and the first geometric feature , and also from the second surface point cloud Extract the second general feature and the second geometric feature .

[0062] The feature extraction module has the same feature extraction process for the first surface point cloud and the second surface point cloud, and includes a general feature extraction module and a geometric feature extraction module, which are used to extract general features and geometric features respectively. The general feature extraction module is formed by cascading an AdaptConv network and a multi-layer perceptron (MLP), and is used to perform point-by-point feature learning on each input surface point cloud to obtain general features. Both the AdaptConv network and the MLP are network models in the prior art and will not be described in detail. In the embodiment of the present invention, the input point cloud is processed by AdaptConv and MLP. and By learning point-by-point features separately, we can get two common features and . The geometric feature extraction module is used to extract point pair distance features, triplet angle features, triplet perimeter difference features, Gaussian curvature distance features and mean curvature distance features from each input surface point cloud, and concatenate the five features to obtain geometric features. The point pair distance feature is composed of the Euclidean distances of all point pairs in the point cloud; the triplet angle feature is composed of the triplet angles of all point pairs in the point cloud; the triplet perimeter difference feature is composed of the triplet perimeter differences of all point pairs in the point cloud; the Gaussian curvature feature is composed of the Gaussian curvature differences of all point pairs in the point cloud; and the mean curvature feature is composed of the mean curvature differences of all points in the point cloud.

[0063] Combine the following Figure 3 As shown in a) in the figure, the specific calculation methods of the above-mentioned point pair distance, triple angle, triple perimeter difference, Gaussian curvature and mean curvature are respectively shown. For example, for any two points in the point cloud, , their geometric structure embedding is calculated by the point pair Euclidean distance, triple angle, triple perimeter difference, Gaussian curvature difference and mean curvature difference as follows:

[0064] 1) Point-to-point Euclidean distance : The point-to-point distance is the spatial distance between two points in Euclidean space, and the calculation formula is ;

[0065] 2) Triple angle : First search the distance The two nearest neighbors Combined into a triple; then by the point and Point Construct a vector for the starting point , by point and Point Construct a vector for the starting point , the vector and vector The vector sum is defined as , and define the point and Point Construct a vector for the starting point , and then calculate the angle of the triplet as ;

[0066] 3) Triple circumference difference : Search distance The two nearest neighbors Combined into a triple, search distance The two nearest neighbors Combined into another triple, the perimeter difference between the two triples is calculated as ,in , , They are arrive spatial distance, arrive spatial distance, arrive The spatial distance , , They are arrive spatial distance, arrive spatial distance, arrive The spatial distance between two points is the same as the Euclidean distance of the above point pair. The calculation formula is the same as ;

[0067] 4) Gaussian curvature difference and the mean curvature difference : Point Cloud The surface normal vector is , based on the surface normal vector, the point can be calculated The principal curvatures of the point cloud surface (including the maximum curvature and minimum curvature ), based on the principal curvature, Gaussian curvature and mean curvature can be further calculated. The calculation expressions are and Similarly, we can also calculate the point Gaussian curvature and mean curvature Then click The difference in their respective Gaussian curvatures is the Gaussian curvature difference , the difference of their respective mean curvatures is the mean curvature difference .

[0068] Based on the above calculation formulas, the point cloud can be obtained Any point pair The point-to-point Euclidean distance between , triple angle , triple perimeter difference , Gaussian curvature difference and the mean curvature difference , and if the point cloud If there are a total of N points in the , then N×N point pairs can be formed, and then the eigenvalues ​​of the N×N point pairs are combined into a feature matrix to obtain the point cloud Corresponding point pair distance features , triple angle feature , triple perimeter difference feature , Gaussian curvature distance feature and mean curvature distance features , these five features are all N×N feature matrices, satisfying , the five features can be spliced ​​to get the point cloud The corresponding first geometric feature Similarly, for point clouds , the second geometric feature can also be obtained by calculating five features and concatenating them , the method is the same and will not be repeated here.

[0069] The second module in the 3D surface rigid registration model of the present invention is a feature weighting module, which is divided into two branches, corresponding to the representative point selector and the feature matching module. Among them, the representative point selector performs clustering and dimensionality reduction based on the normal direction of the point cloud, so as to obtain the representative point mask corresponding to the first surface point cloud and the second surface point cloud respectively; the feature matching module obtains the first enhanced feature after geometric embedding by passing the first general feature and the first geometric feature through the geometric Transformer model, and obtains the second enhanced feature after geometric embedding by passing the second general feature and the second geometric feature through the geometric Transformer model, and then outputs the point correspondence matrix by the exponential function after the first enhanced feature and the second enhanced feature pass through the affinity layer and the Sinkhorn iterative algorithm; finally, the representative point selector fuses the outputs of the two branches, and weights the first enhanced feature and the second enhanced feature with the point correspondence matrix and the representative point mask respectively to obtain a new first general feature and a new second general feature.

[0070] The feature matching module in the present invention takes two pairs of paired general features and geometric features as inputs. The first pair of the first general features and the first geometric features are subjected to a geometric Transformer model to obtain enhanced features after geometric embedding. The second pair of the second general features and the second geometric features are subjected to a geometric Transformer model to obtain enhanced features after geometric embedding. The two enhanced features are then calculated separately through an affinity layer and a Sinkhorn iterative algorithm, and the calculation results are processed by an exponential function to output a point correspondence matrix.

[0071] The geometric Transformer model used in the present invention belongs to the prior art, also known as the Geometric Transformer. It calls geometric structure embedding (GA) on the basis of the traditional Transformer to realize the enhanced learning of general features. Transformer has been proposed and widely used to realize global context information enhanced learning. It is generally composed of a self-attention module and a cross-attention module for learning features. The geometric Transformer model embeds a new combination of geometric structure embedding in the self-attention module, which not only encodes high-level point features, but also explicitly captures the geometric structure within the point cloud and the geometric consistency between point clouds. The attention module that introduces the geometric structure embedding can be better used to learn the features between each point cloud and the global correlation of the geometric space. The following is combined with Figure 3 As shown in b), the point cloud As an example, the calculation of The calculation is the same as that of the geometric Transformer model:

[0072] For point clouds The first common feature , is the feature dimension of a single point, the feature matrix output by the geometric Transformer model is the projection of all input features The weighted sum of:

[0073]

[0074] The weight coefficient is the attention score The row-by-row softmax calculation on The calculation is as follows:

[0075]

[0076] Where: The first geometric feature The eigenvalues ​​in are embedded as the aforementioned geometric structure; is the projection matrix of query, key, and value, is the projection matrix of the geometric structure embedding.

[0077] The above geometric Transformer model is for point clouds The output feature matrix This is the first enhanced feature after geometric embedding , while the same geometric Transformer model for point clouds The output feature matrix This is the second enhanced feature after geometric embedding However, it should be noted that the above calculation process belongs to the existing technology of the geometric Transformer model, and is only explained here for easier understanding.

[0078] At the same time, the affinity layer (Affinity Layer) used in the present invention also belongs to the prior art, and is used to calculate the affinity matrix of the first enhanced feature and the second enhanced feature in the feature space. This affinity matrix reflects the similarity or relationship between the two features and is the basis for subsequent optimization and analysis. Subsequently, the normalized affinity matrix is ​​sent as input to the Sinkhorn iterative algorithm to approximate a double random matrix. The Sinkhorn algorithm is an iterative algorithm for solving the regularized optimal transmission problem, which belongs to the prior art and will not be elaborated. The final double random matrix has the characteristic that the sum of each row and column does not exceed 1, and can be used for further data analysis and processing. After the double random matrix is ​​processed by the exponential function, the final point correspondence matrix is ​​obtained. The point corresponds to the matrix , can be combined with another branch for point cloud and Output representative point mask and , and the first enhanced feature after geometric embedding and the second enhanced feature after geometric embedding , two new common features are obtained through weighted fusion. Specifically, the point correspondence matrix With the first enhanced feature Perform matrix multiplication, and then add the result to the representative point mask Perform matrix dot product operation to obtain the new first universal feature ; Similarly, first correspond the point to the matrix With the second enhancement feature Perform matrix multiplication, and then add the result to the representative point mask Perform matrix dot product operation to obtain the new second universal feature .

[0079] In the present invention, the intraoperative-preoperative 3D point cloud data is a dense-sparse 3D point cloud data, and the ordinary sampling method will affect the number of points and the relationship between these points. That is to say, for the same object, using different devices to shoot from the same viewpoint, the point cloud reconstructed during surgery shows more points than the point cloud reconstructed by preoperative CT, which will affect the local geometric expression (normal, curvature) of all points in the point cloud. Because point cloud registration can actually be determined by the correspondence of a coarser subset of points. The point-by-point correspondence of the original point cloud is redundant, and sometimes the correspondence that is too clustered is useless. The representative point selector (RPSelector) used in the present invention can select representatives from the point cloud to replace the point set of the local surface of the point cloud through clustering and dimensionality reduction, thereby realizing the registration process between dense-sparse point clouds and ignoring the influence of point cloud density. The representative point selector is not a traditional sampling process. Its core is to achieve point cloud matching and select the points that are most likely to match each other to represent the local surface.

[0080] The representative point selector used in the present invention takes two surface point clouds as input, namely, point clouds and , whose output is a point cloud and The corresponding representative point mask and . The process of generating a representative point mask for each input surface point cloud inside the representative point selector is the same, specifically: each input surface point cloud is divided by voxels (Volume), and the voxel size can be optimized as a hyperparameter, and a voxel set is obtained by voxelizing the point cloud; then for each voxel in the voxel set, all points contained in each voxel are K-means clustered based on the normal vectors of each point in the point cloud, and a local surface is formed by the points contained in each cluster; finally, for each local surface in each voxel, the point closest to the center of mass in the local surface is taken as the representative point of the local surface, and a representative point mask corresponding to each input surface point cloud is generated based on all representative points in the voxel set. The above implementation process of the representative point selector is divided into three basic steps: voxelization, clustering, and representative point extraction. The schematic diagrams of the three basic steps are shown as follows. Figure 4 As shown in (a), (b) and (c).

[0081] It should also be noted that according to the requirements of the clustering algorithm, if the number of points in a voxel is less than the number of clusters, clustering cannot be performed. Therefore, considering that the number of points in some voxels may be less than the number of clusters in the actual voxel division process, it is best to count the number of points in the voxel before clustering. If such a special case exists, clustering is not required, and all points in the voxel can be directly selected as a whole category for representative point selection. Therefore, in the embodiment of the present invention, the point cloud is used as the representative point of the voxel. For example, point cloud The corresponding normal vector set , For point The calculation steps of the normal vector of the representative point selector RPSelector are as follows:

[0082] 1) Voxelization: Convert point cloud Partition V voxels v into a voxel set V p , maintaining the topological details of the surface. The voxelization process needs to be careful because the amount of calculation depends on the number of adjacent points, and the two point clouds and The density of is different, so voxelization with the same voxel size aims to generate equivalent local regions in the two point clouds. Therefore, the voxel size parameter is crucial and should be chosen optimally to maintain the basic characteristics of the two point clouds.

[0083] 2) Clustering based on normal vectors: The number of clusters k of k-means clustering is preset. The specific k value can be optimized and selected as a hyperparameter. In this embodiment, k=6. Statistical voxel set V p The number of points K contained in each voxel v. If the number of points K contained in the voxel v is K≤k, there is no need for subsequent clustering, and these K points are directly regarded as a local surface s. If the number of points K contained in the voxel v is K>k, all points are k-means clustered based on the normal vector n of each point in the voxel v to obtain k groups of points. Each group of points after clustering shares a similar normal direction, and each group of points is regarded as a local surface s.

[0084] 3) Election of representative points: For each local surface s in the voxel v, a point is selected from the points contained in the local surface s to represent the surface. The present invention uses the point closest to the centroid of the local surface as the representative point. p Each local surface s will generate a representative point, and the position number of the representative point in the point cloud will be recorded to form a position number set , where num is the point cloud The final number of representative points satisfies num≤V*k.

[0085] 4) Get the point mask: initialize an element to 0 As the representative point mask, according to step 3), obtain each representative point in the original point cloud The set of serial numbers in , each representative point is The corresponding serial number position is assigned a value of 1, and the non-representative point is assigned a value of 0, thus forming a binary record point cloud. A one-dimensional mask representing the point location in .

[0086] The above is for point cloud The representative point mask generation process, point cloud The representative point mask generation process is the same, by obtaining the representative point cloud number set , which can generate representative point masks , I will not go into details here.

[0087] The third module in the 3D surface rigid registration model of the present invention is a transformation matrix solving module, in which another feature matching module with the same structure is called, and a new first general feature and a first geometric feature, a new second general feature and a second geometric feature are used as input to calculate a new point correspondence matrix. Finally, based on the new point correspondence matrix, the rigid transformation matrix is ​​solved by singular value decomposition with the goal of minimizing the registration error.

[0088] It should be noted that the feature matching module used in the transformation matrix solving module has the same network structure as the feature matching module in the feature weighting module, and the inputs of the two are also the same. The only difference is that the first general feature paired with the first geometric feature and the second general feature paired with the second geometric feature are replaced by the new first general feature and the new second general feature, respectively. The feature matching module takes two pairs of paired new general features and the original geometric features as input, and obtains two enhanced features after geometric embedding through the geometric Transformer model respectively. The two enhanced features are then calculated through the affinity layer and the Sinkhorn iterative algorithm respectively. The calculation results are processed by the exponential function to output the new point correspondence matrix. Based on the new point correspondence matrix and combined with singular value decomposition (SVD), the rigid transformation matrix required to register the first surface point cloud with the second surface point cloud can be obtained. The solution goal of SVD decomposition is to minimize the registration error between the first surface point cloud and the second surface point cloud. This solution goal requires the introduction of a new point correspondence matrix Calculate the rotation matrix R and the translation vector t as the parameters to be solved, and solve the final rigid transformation matrix {R, t}. The form of the objective function is as follows:

[0089]

[0090] Where: is the number of representative points in the first surface point cloud or the second surface point cloud, and the number of representative points in both is the same; is the new point correspondence matrix The element value of the i-th row and j-th column in ; and The first surface point cloud and the second surface point cloud The representative point in , R is the rotation matrix, and t is the translation vector.

[0091] S4. Finally, the first surface point cloud and the second surface point cloud are registered using the obtained rigid transformation matrix to obtain the registered preoperative 3D model and intraoperative 3D model. These two models can be used for virtual-reality fusion display to assist the smooth progress of the operation.

[0092] In order to better demonstrate the technical effect that can be achieved by the geometric information-assisted 3D surface rigid registration method shown in S1 to S4 above, the liver is taken as the target organ of the embodiment to demonstrate the registration results between the intraoperative 3D point cloud and the preoperative 3D point cloud.

[0093] Example

[0094] The basic process of the geometric information-assisted 3D surface rigid registration method in this embodiment is shown in S1 to S4 above. For the convenience of description, the 3D surface rigid registration model of the present invention is called GIA-Net, and the parameters in the model are set as follows: the voxel size is set to 0.025 when the representative point is selected, the voxel size growth rate is set to 1.5, the number of clusters k in each voxel is set to 6; the number of repetitions Nt of the geometric transformer is set to 3. The data sets used in this embodiment include the self-built N1 data set and the open source data set DePoLL, and their respective data conditions are as follows:

[0095] A. Self-built N1 dataset

[0096] The self-built N1 dataset includes 9 pairs of preoperative and intraoperative point cloud data of pig liver surfaces in different deformation states. Among them, 6 groups used intraoperative structured light scanning to obtain 3D liver surface point cloud data, while the other 3 groups came from stereoscopic laparoscopic images. The laparoscopic dataset constructed by this embodiment used 4 Parmesan pigs weighing between 30-40 kg. During minimally invasive surgery, 8 titanium clips with a length of 1 cm were evenly placed on the edge of each pig liver. The dataset includes 4 sets of preoperative CT models and corresponding liver surface point cloud data, capturing various deformation states encountered during surgery. The experimental pigs controlled low-frequency breathing through a ventilator during data collection. First, minimally invasive opening was performed, and standard carbon dioxide was filled into the abdominal cavity with a peritoneal insulator to keep the abdominal wall bulging, and different perspectives were obtained by moving the laparoscope to different positions in the abdominal cavity. The laparoscopic surface data at each perspective was recorded in the video using a 3D camera system. The binocular stereo matching method of OpenCV was used to reconstruct the 3D surface of the liver and obtain liver surface point cloud data. Subsequently, open surgery was performed, and a structured light scanner was used to obtain liver surface point cloud data in an open environment. The preoperative liver surface point cloud data were sampled from enhanced CT data, and the slice thickness was set to 6 mm.

[0097] B. Open Source Dataset DePoLL

[0098] The DePoLL dataset includes 13 pairs of preoperative and intraoperative point cloud data of the pig liver surface in different deformation states. The dataset comes from a male white pig weighing 50 kg. Fifteen metal clips with a length of 1 cm and 45 metal balls with a diameter of 2 cm were evenly placed on the liver surface through laparotomy. After infusing standard carbon dioxide pressure (12 mmHg), 13 different laparoscopic configurations were obtained by moving the four liver lobes to different positions. The surface data under laparoscopic configurations were recorded in the video using a 3D camera system. The intraoperative liver surface data were obtained using Photoscanpro software and the Structure from Motion (SfM) method to reconstruct the 3D surface of the laparoscopic video under each configuration. The preoperative liver surface point cloud data were sampled from the CT data in each configuration. The experimental pig remained without breathing during data collection. The preoperative and intraoperative liver surface point clouds were registered and aligned.

[0099] In this embodiment, about 1024 points were randomly sampled from the preoperative and intraoperative point cloud data, and the position information of 15 metal clips on the liver surface was used to calculate the target registration error (TRE).

[0100] A random rotation angle in the range of [0°, 45°] and a random translation in the range of [-0.5m, 0.5m] are added to the point cloud reconstructed from the laparoscopic stereoscopic view (called the target point cloud) to calculate the transformation matrix error. The added random rotation and translation matrix is ​​called GT.

[0101] In this embodiment, the main evaluation indicators used are the target registration error (TRE) of the registration point error and the average isotropic rotation and translation error of the registration transformation matrix. The target registration error (TRE) refers to the distance between corresponding points other than the reference point after registration. The isotropic rotation and translation errors represent the errors Error(R) and Error(t) between the predicted rotation matrix, translation vector and the true transformation matrix. In addition, the root mean square error (RMSE) and mean absolute error (MAE) of the rotation matrix and translation vector are calculated as reference indicators.

[0102] For comparison, this example compares multiple state-of-the-art point matching methods (SOTA): ROPNet, LiverMatch, RPMNet, and geometry-based methods: GeoTransformer, UTOPIC, and RGM. LiverMatch is a method specifically designed for liver point cloud registration. All methods were fine-tuned with the liver training dataset and the same training settings for fair comparison. The corresponding experimental results are shown below.

[0103] 1. Comparison with SOTA methods

[0104] 1) Test results on the DePoLL dataset

[0105] In this embodiment, the DePoLL dataset is used for testing. The dataset consists of a three-dimensional point cloud captured by stereo vision at 13 different angles and a three-dimensional CT model established before surgery.

[0106] Table 1 shows the comparison results on the DePoLL dataset, including various indicators of transformation matrix error: RMSE, MAE, Error(R) and Error(t). The best performance is marked in bold and the suboptimal is underlined.

[0107]

[0108] Table 2 shows the comparison results on the DePoLL dataset, including the TRE indicator of the transformation matrix error. The best performance is marked in bold and the suboptimal is underlined.

[0109]

[0110] As can be seen from Table 1, in Error(R), GIA-Net is still the smallest compared with other deep learning methods. GAI-Net not only recovers the rotation matrix best, but also has a high prediction accuracy for the translation vector.

[0111] Table 2 shows the TRE metrics of 8 algorithms on 13 liver samples. The best results are marked in bold. From the variance of each column, it can be seen that GIA-Net has stable registration prediction results.

[0112] 2) Test results on the self-built N1 dataset

[0113] In this embodiment, a self-built N1 dataset is also used for testing. The intraoperative 3D point cloud data in this dataset includes data reconstructed by stereo laparoscopic imaging and data obtained by structured light scanning. Since the self-built N1 dataset increases the diversity of test data, it can be used to verify the generalization performance of the model.

[0114] Table 3 shows the comparison results on the self-built N1 dataset, including various indicators of the transformation matrix error: RMSE, MAE, Error(R) and Error(t). The best performance is marked in bold and the suboptimal is underlined.

[0115]

[0116] As shown in Table 3, the proposed GIA-Net achieves the lowest Error(t) compared to other state-of-the-art deep learning methods. However, compared to its performance on DePoLL, GIA-Net performs poorly in registering samples from the self-built N1 test set. This is largely due to the significant time difference between collecting porcine CT scans and stereo laparoscopic or structured light scanning data, which leads to a lot of deformation and scale changes between point clouds. Further research on deformation registration is needed to achieve good registration results. Due to the deformation between the source and target point clouds, the TRE of the self-built N1 test dataset was not calculated.

[0117] 2. Ablation experiment

[0118] This embodiment also conducts an ablation study on the DePoLL dataset to better demonstrate two key improvements in the GIA-Net model of the present invention, namely, the introduction of curvature features (i.e., Gaussian curvature distance features and average curvature distance features) and the introduction of the representative point selector RPSelector in the geometric Transformer model. In order to evaluate the impact of these two improvements on the model performance, three ablation experiments were set up, in which the two components C and R represent the two improvements in the geometric Transformer model, namely, the introduction of curvature features and the introduction of the representative point selector RPSelector, which can respectively mine the local (curvature) and global (normal direction) geometric features of the point cloud, and G represents the basic model components in the GIA-Net model after removing these two improvements, so G+C+R represents the GIA-Net model of the present invention.

[0119] Table 4 shows the results of the ablation experiment on the DePoLL dataset, including various indicators of the transformation matrix error: RMSE, MAE, Error(R) and Error(t). This is to evaluate the impact of different model components on the registration effect. The best performance is marked in bold, and the suboptimal is underlined.

[0120]

[0121] As can be seen from Table 4, the addition of RPSelector can minimize the impact of interference points on the registration effect. Removing any component of G, C, and R will reduce the overall performance, which shows that all components are beneficial for accurate registration results. However, as shown in the penultimate row of Table 4, when G (universal geometric features: point pair distance, triple perimeter difference, and triple angle) in the basic code is removed, the registration effect decreases. Through analysis, it can be seen that since the positions with large curvature changes in the liver surface point cloud are usually edges, the position information obtained by the curvature encoding is relatively sparse and cannot support the formation of global position encoding alone. Therefore, the model cannot mark points at different positions well, which will reduce the registration effect.

[0122] Table 5 Ablation experiment results on the DePoLL dataset, including the TRE indicator of the transformation matrix error. To evaluate the impact of different model components on the registration effect, G, C and R represent the model with general geometry structure, curvature structure and RPSelector respectively. The best performance is marked in bold and the suboptimal is underlined.

[0123]

[0124] As can be seen from the results in Table 5, the GIA-Net model of the present invention achieves the best results in the TRE index. In the point cloud registration evaluation, although Error(R) (rotation error) and Error(t) (translation error) respectively measure the rotation and translation accuracy in the registration process, TRE (Target Registration Error) is usually more important. Error(R) represents the error between the registered rotation matrix and the true value, and Error(t) represents the error between the registered translation vector and the true value. These two indicators reflect the rotation and translation accuracy in the registration process, respectively, and are suitable for evaluating the detail accuracy of each step in the registration process. However, TRE, as a comprehensive indicator, directly reflects the overall accuracy of the registered point cloud. It measures the distance error between the target point and the reference point, covering all errors of rotation and translation. Therefore, TRE provides a more comprehensive accuracy evaluation. In most application scenarios, especially those fields that require high overall accuracy, such as medical imaging, 3D scanning, and robot navigation, TRE can better reflect the final effect and practical application value of registration, so the GIA-Net model of the present invention has broad application value in these fields.

[0125] In the present invention, the geometric information-assisted 3D surface rigid registration method based on the GIA-Net model can be used to implement a surgery-assisted virtual-real fusion system. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a surgery-assisted virtual-real fusion system, which includes:

[0126] A preoperative 3D model construction module is used to reconstruct a preoperative 3D model based on the preoperative CT image of the target organ;

[0127] An intraoperative 3D model construction module is used to reconstruct an intraoperative 3D model based on the intraoperative visual image of the target organ;

[0128] A rigid registration module, used to register the surface point clouds of the preoperative 3D model and the intraoperative 3D model according to the geometric information-assisted 3D surface rigid registration method described in S1 to S4 above;

[0129] The image display module is used to fuse the virtual and real images of the registered pre-operative 3D model and the intra-operative 3D model to generate a virtual and real fused image for display.

[0130] In the above-mentioned preoperative 3D model construction module, the preoperative 3D model is reconstructed based on the preoperative CT image by using the MITK software. In the above-mentioned intraoperative 3D model construction module, the intraoperative 3D model is reconstructed based on the intraoperative visual image by using the StaSiS-Net model.

[0131] Since the principle of solving the problem by the rigid registration module in the surgical-assisted virtual-reality fusion system in the embodiment of the present invention is similar to the 3D surface rigid registration method assisted by geometric information in the above-mentioned embodiment of the present invention, the specific implementation forms of the modules of the system in this embodiment that are not fully explained can also be referred to the specific implementation forms of the above-mentioned method, and the repeated parts will not be repeated.

[0132] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the geometric information-assisted 3D surface rigid registration method provided in the above embodiment, which includes a memory and a processor;

[0133] The memory is used to store computer programs;

[0134] The processor is used to implement the geometric information-assisted 3D surface rigid registration method as described above when executing the computer program.

[0135] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0136] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the geometric information-assisted 3D surface rigid registration method provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement the geometric information-assisted 3D surface rigid registration method as described above.

[0137] Specifically, in the computer-readable storage medium of the above two embodiments, the stored computer program is executed by the processor to perform the above steps S1 to S4.

[0138] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0139] It can be understood that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0140] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0141] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A geometric information-assisted 3D surface rigid registration method for realizing the registration between the intraoperative 3D point cloud and the preoperative 3D point cloud for the target organ, characterized in that: include: S1, reading a preoperative 3D model reconstructed based on a preoperative CT image of the target organ, and saving it as a first surface point cloud; S2, reading the intraoperative 3D model reconstructed based on the intraoperative visual image of the target organ and saving it as a second surface point cloud; S3, sampling the first surface point cloud and the second surface point cloud to the same dimension as the input of the 3D surface rigid registration model, and sequentially passing through the feature extraction module, feature weighting module and transformation matrix solving module in the 3D surface rigid registration model to obtain a rigid transformation matrix between the first surface point cloud and the second surface point cloud; The feature extraction module is used to extract a first general feature and a first geometric feature from the first surface point cloud, and to extract a second general feature and a second geometric feature from the second surface point cloud; The feature weighting module includes a representative point selector and a feature matching module; The representative point selector performs clustering and dimensionality reduction based on the normal direction of the point cloud, so as to obtain representative point masks corresponding to the first surface point cloud and the second surface point cloud respectively; the feature matching module obtains the first enhanced feature after geometric embedding by passing the first general feature and the first geometric feature through the geometric Transformer model, obtains the second enhanced feature after geometric embedding by passing the second general feature and the second geometric feature through the geometric Transformer model, and then outputs the point correspondence matrix by the exponential function after the first enhanced feature and the second enhanced feature pass through the affinity layer and the Sinkhorn iterative algorithm; finally, the first enhanced feature and the second enhanced feature are weighted with the point correspondence matrix and the representative point mask respectively to obtain a new first general feature and a new second general feature; The transformation matrix solving module calls another feature matching module of the same structure, takes the new first general feature and the first geometric feature, the new second general feature and the second geometric feature as input, calculates a new point correspondence matrix, and finally obtains a rigid transformation matrix by singular value decomposition based on the new point correspondence matrix with the goal of minimizing the registration error; S4, registering the first surface point cloud and the second surface point cloud using the rigid transformation matrix; The feature extraction module includes a general feature extraction module and a geometric feature extraction module; The universal feature extraction module is formed by cascading an AdaptConv network and a multi-layer perceptron, and is used to perform point-by-point feature learning on each input surface point cloud to obtain universal features; The geometric feature extraction module is used to extract point pair distance features, triple angle features, triple perimeter difference features, Gaussian curvature distance features and average curvature distance features from each input surface point cloud, and concatenate the five features to obtain geometric features; the point pair distance feature is composed of the Euclidean distances of all point pairs in the point cloud; the triple angle feature is composed of the triple angles of all point pairs in the point cloud; the triple perimeter difference feature is composed of the triple perimeter differences of all point pairs in the point cloud; the Gaussian curvature distance feature is composed of the Gaussian curvature differences of all point pairs in the point cloud; the average curvature feature is composed of the average curvature differences of all points in the point cloud; First surface point cloud For any pair of points , the calculation formulas for point-to-point Euclidean distance, triple angle, triple perimeter difference, Gaussian curvature difference, and mean curvature difference are as follows: 1) Point-to-point Euclidean distance The calculation process is: ; 2) Triple angle The calculation process is: first search the distance The two nearest neighbors Combined into a triple; then by the point and Point Construct a vector for the starting point , by point and Point Construct a vector for the starting point , the vector and vector The vector sum is defined as , and define the point and Point Construct a vector for the starting point , and then calculate the angle of the triplet as ; 3) The calculation process of the perimeter difference of the triple is: search distance The two nearest neighbors Combined into a triple, search distance The two nearest neighbors Combined into another triple, the perimeter difference between the two triples is calculated as ,in , , They are arrive spatial distance, arrive spatial distance, arrive The spatial distance , , They are arrive spatial distance, arrive spatial distance, arrive spatial distance; 4) Gaussian curvature difference and the mean curvature difference The calculation process is as follows: point cloud The surface normal vector is , based on the surface normal vector Calculate the principal curvature of a point on the point cloud surface, which contains the maximum curvature and minimum curvature , based on the principal curvature, the Gaussian curvature and mean curvature are further calculated. The calculation expressions are: and ; In the same way, calculate the point Gaussian curvature and mean curvature ; Point to The difference in their respective Gaussian curvatures is the Gaussian curvature difference , the difference of their respective mean curvatures is the mean curvature difference ; The first surface point cloud and the second surface point cloud The five features are calculated in the same way.

2. The geometric information-assisted 3D surface rigid registration method according to claim 1, characterized in that: The intraoperative visual image is an intraoperative endoscopic image of the patient obtained through an endoscope.

3. The geometric information-assisted 3D surface rigid registration method according to claim 1, characterized in that: In the representative point selector, each input surface point cloud is first divided into voxels to obtain a voxel set; then, for each voxel in the voxel set, all points contained in each voxel are K-means clustered based on the normal vectors of each point in the point cloud, and a local surface is formed by the points contained in each cluster; finally, for each local surface in each voxel, the point closest to the center of mass in the local surface is taken as the representative point of the local surface, and a representative point mask corresponding to each input surface point cloud is generated according to all representative points in the voxel set.

4. The geometric information-assisted 3D surface rigid registration method according to claim 1, characterized in that: The target organ is human liver.

5. A surgery-assisted virtual-real fusion system, characterized in that: include: A preoperative 3D model construction module is used to reconstruct a preoperative 3D model based on the preoperative CT image of the target organ; An intraoperative 3D model construction module is used to reconstruct an intraoperative 3D model based on the intraoperative visual image of the target organ; A rigid registration module, used for registering the surface point clouds of the preoperative 3D model and the intraoperative 3D model according to the geometric information-assisted 3D surface rigid registration method according to any one of claims 1 to 4; The image display module is used to fuse the virtual and real images of the registered pre-operative 3D model and the intra-operative 3D model to generate a virtual and real fused image for display.

6. The surgery-assisted virtual-real fusion system according to claim 5, characterized in that: In the preoperative 3D model construction module, the preoperative 3D model is reconstructed based on the preoperative CT image by using MITK software.

7. The surgery-assisted virtual-real fusion system according to claim 5, characterized in that: In the intraoperative 3D model construction module, the intraoperative 3D model is reconstructed based on the intraoperative visual image through the StaSiS-Net model.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the geometric information-assisted 3D surface rigid registration method as described in any one of claims 1 to 4 is implemented.

9. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the geometric information-assisted 3D surface rigid registration method as described in any one of claims 1 to 4 when executing the computer program.