Real-time registration method from preoperative model to intraoperative point cloud based on deep learning

Abstract

The present invention provides a real-time registration method from a preoperative model to an intraoperative point cloud based on deep learning, and relates to the field of 3D point cloud registration. In the present invention, a pre-reconstructed preoperative model is obtained; a real-time intraoperative point cloud is obtained based on an in vivo flexible rapid three-dimensional reconstruction method of a digital elevation model; preoperative vertex features that characterize the anisotropic properties of the preoperative model are extracted, and the preoperative vertex features are spliced ​​with the corresponding intraoperative point cloud features, and used as the input of a graph convolutional neural network, and the deformed preoperative model is inferred and output; using a pair of detection feature points with a corresponding relationship as a supervisory signal, a loss function is constructed to train the graph convolutional neural network until convergence, and finally a real-time registered mesh three-dimensional model is output. Based on a deep learning network framework including feature extraction, feature point detection, and deformation prediction, real-time registration from a non-rigid preoperative model to an intraoperative point cloud is achieved to meet the requirements of a real minimally invasive laparoscopic surgery scenario.

Description

Technical Field

[0001] The present invention relates to the field of 3D point cloud registration, and in particular to a real-time registration method from a preoperative model to an intraoperative point cloud based on deep learning. Background Art

[0002] 3D point cloud registration is one of the key research issues in computer vision and has important application value in many fields. For example, 3D point cloud registration in laparoscopic surgical navigation plays a very critical role in assisting medical diagnosis and surgical planning.

[0003] Patent CN115527003A discloses a preoperative and intraoperative liver point cloud data registration system, method, terminal and storage medium. The method includes: extracting local mixed features from preoperative point cloud data and intraoperative point cloud data respectively; fusing the local mixed features to obtain global features of the preoperative point cloud and global features of the intraoperative point cloud; fusing the local mixed features, global features and global features of the preoperative point cloud to obtain fused features of the preoperative point cloud; similarly obtaining fused features of the intraoperative point cloud; fusing the fused features of the preoperative point cloud and the fused features of the intraoperative point cloud to obtain respective overlapping area masks and decoding features; obtaining the spatial transformation matrix of the preoperative point cloud and the intraoperative point cloud; applying the spatial transformation matrix to the preoperative point cloud data to obtain the registration result of the preoperative point cloud data and the intraoperative point cloud data.

[0004] However, the feature extraction and fusion process of the original technical solution is too complicated, and the overall performance of the algorithm is based on the performance of feature extraction and fusion. At the same time, the original technology did not achieve non-rigid preoperative and intraoperative model alignment, but only achieved rigid transformation of preoperative and intraoperative space without considering the flexible deformation of tissues in the in vivo environment. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a real-time registration method from a preoperative model to an intraoperative point cloud based on deep learning, which solves the technical problem that the prior art only realizes the rigid transformation of the preoperative and intraoperative space but does not take into account the flexible deformation of tissues in the in vivo environment.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A real-time registration method from a preoperative model to an intraoperative point cloud based on deep learning, comprising:

[0010] Obtaining a preoperative model reconstructed in advance using preoperative CT data of the patient;

[0011] An in vivo flexible and fast 3D reconstruction method based on digital elevation model is used to obtain the intraoperative point cloud corresponding to the current color image frame during laparoscopic surgery;

[0012] Extracting preoperative vertex features characterizing the anisotropic properties of the preoperative model, and splicing the preoperative vertex features with corresponding intraoperative point cloud features;

[0013] Each pre-operative vertex feature after splicing and updating is used as the input of the graph convolutional neural network, and the deformed pre-operative model is inferred and output;

[0014] Using the detection feature point pairs with corresponding relationships between the preoperative model and the intraoperative point cloud as supervision signals, a loss function is constructed to train the graph convolutional neural network until convergence, and the deformed preoperative model output by the final reasoning is used as a mesh three-dimensional model for real-time registration.

[0015] Preferably, the in vivo flexible rapid 3D reconstruction method based on a digital elevation model, for obtaining an intraoperative point cloud corresponding to a current color image frame during laparoscopic surgery, comprises:

[0016] Acquire a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence;

[0017] Using the pinhole camera model, each frame of color image and its depth map are restored into a one-to-one corresponding color point cloud;

[0018] Using the improved digital elevation model, constructing a local three-dimensional model corresponding to each of the color point clouds;

[0019] Gets a deformation field consisting of a series of deformation nodes, including:

[0020] Reconstruct the current local three-dimensional model, and use the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; minimize the total energy as the objective function, and optimize the deformation vector of each deformation node of the current deformation field;

[0021] Using the current deformation field to warp the last fused three-dimensional model, to obtain a warped three-dimensional model;

[0022] The warped three-dimensional model and the current local three-dimensional model are fused to obtain the current fused three-dimensional model as the intraoperative point cloud corresponding to the current color image frame.

[0023] Preferably, the improved digital elevation model consists of two parts: initial coordinates and a grid set; the method of using the improved digital elevation model to construct a local three-dimensional model corresponding to each of the color point clouds includes:

[0024] Calculating the XOY plane size of each of the color point clouds, dividing the XOY plane into grids according to a uniform grid size, so that each grid of the grid set contains a number of points of the color point cloud, and each point carries its position information and color information;

[0025] The information entropy index function is used to calculate the information entropy index function value of each grid. If it is less than the first preset threshold, it is considered that the corresponding grid only retains one average point, which is called a single-point grid; otherwise, multiple average points are retained, which is called a multi-point grid.

[0026] For each of the single-point grids, the position information of the single average point is calculated based on the position information of each point in the grid and the initial coordinates of the improved digital elevation model; and the color information of the single average point is averaged based on the color information of each point in the grid;

[0027] For each of the multi-point grids, the ISODATA clustering algorithm is used to automatically generate the position information of the multiple average points retained; and based on the color information of each group of aggregated points, the color information of the multiple average points retained is obtained by averaging respectively;

[0028] A local three-dimensional model corresponding to each of the color point clouds is obtained.

[0029] Preferably, reconstructing the current local three-dimensional model and sampling the three-dimensional position of each deformation node of the current deformation field by using the importance of each grid of the reconstructed model includes:

[0030] Referring to the processing method of single-point grids, the original multi-point grids of the current local 3D model are converted into single-point grids one by one, and the reconstructed model is obtained by combining the original single-point grids;

[0031] Using the VIP index function, the importance index value of each grid of the reconstructed model is calculated. If the importance index value is greater than a second preset threshold, the grid is considered to be of high importance and is called an important grid.

[0032] Prioritizing sampling of important grids of the reconstructed model, and using position information of a single average point in each of the important grids as the three-dimensional position of a corresponding deformation node of the current deformation field;

[0033] Then, the non-important grids are traversed, and the Euclidean distance between a single average point in the current non-important grid and the sampled grid is calculated. If the distance is greater than a third preset threshold, the position of the single average point in the non-important grid is used as the three-dimensional position of the corresponding deformation node of the current deformation field;

[0034] Preferably, the objective function is expressed as:

[0035]

[0036] in, Represents the current deformation field Minimize the total energy under t Indicates the frame label; λ represents the weight of the motion consistency regularization term; γ represents the loss of TPS energy term;

[0037] E data Represents the data energy term of the deformation vector supervised by optical flow information; s i and t i Respectively represent the 3D coordinates of a pair of matching points on the point cloud generated by the previous color image frame and the current image frame, N m is the logarithm of three-dimensional matching points; Indicates the three-dimensional matching point s i The predicted deformation vector at is obtained by weighted average of the deformation vectors of the deformation nodes in the neighborhood. The calculation method is expressed as , represents the neighborhood, w ik represents the corresponding weight, , They represent the first k The three-dimensional position and deformation vector of each deformation node;

[0038] E MCT The regularization term representing the consistency of motion is used to ensure that the motion of adjacent deformation nodes maintains a similar trend; N w Indicates the number of deformation nodes; Represents the current deformation field i The deformation vector of the deformation node, , Respectively represent its adjacent deformation nodes j The three-dimensional position and deformation vector of w ij represents the corresponding weight;

[0039] E TPS The TPS energy term represents the change in the surface supervision elevation interpolated by TPS; N a Represents the last fused 3D model S t-1 The number of all average points in ; Represents taking out the three-dimensional vector z Component operations; express S t-1 No. i The elevation value of the average point; Indicates the latest elevation value of the point predicted by the TPS algorithm;

[0040] Preferably, the optimization and solution of the deformation vector of each deformation node of the current deformation field includes:

[0041] use Represents the deformation vectors carried by all deformation nodes, and obtains the following matrix forms respectively:

[0042]

[0043] Among them, the superscript T represents transpose, represents the set of real numbers;

[0044] Represents the first sparse matrix, the first element F ij = ;

[0045] , represents the three-dimensional matching point s i A collection of;

[0046] , represents the three-dimensional matching point t i A collection of;

[0047] represents the identity matrix;

[0048] Represents the second sparse matrix, the second element Y ij = ;

[0049] Represents the third sparse matrix, the third element M ij = , express S t-1 No. i The three-dimensional position of the average point;

[0050] , represents the three-dimensional position A collection of;

[0051] , represents the three-dimensional position predicted by the TPS algorithm A collection of; , and Respectively represent the extraction of three-dimensional vectors x and y Operation of weight;

[0052] At this point, the gradient of the objective function is obtained:

[0053]

[0054] The CUBLAS library is used to optimize and accelerate the solution of the objective function, and finally the deformation vector of each deformation node of the current deformation field is obtained.

[0055] Preferably, use Represents the warping process, including:

[0056] The warped 3D model The initial coordinates of O w Set as current local 3D model The initial coordinates of ;

[0057] For the previous fused 3D model S t-1 All single-point grids, update the row index of the grid i , column index j And the elevation value ; The update formula is as follows:

[0058]

[0059] in,( i w , j w , e w ) represents the index and elevation value of the average point in the updated single-point grid; v t Indicates the change in the grid index of the average point; Indicates p The predicted deformation vector at a single average point within a single-point grid; O t-1 express S t-1 The initial coordinates of O t-1 , B Indicates a uniform grid size;

[0060] For the previous fused 3D model S t-1 All multi-point grids, apply the current deformation field to all average points in the grid; the update formula is as follows:

[0061]

[0062] in, Indicates i The average point coordinates in the deformed multi-point grid, It is i The average point coordinates in the undeformed multi-point grid, It is in i The predicted deformation vectors at the deformation points in the multi-point grid;

[0063] Finally, the warped 3D model is obtained. .

[0064] Preferably, use Represents the fusion process, including:

[0065] Calculate the current local 3D model 3D model after warping The fusion weight of each pair of coincident grids is as follows:

[0066]

[0067] in, express Upper Grid The fusion weight of Indicates the number of average points in the grid; represents the VIP indicator function;

[0068] Different fusion strategies are adopted according to different situations:

[0069] (1) If and ,in express On the grid The fusion weights of overlapping meshes, represents the average number of points in the grid, To express the hyperparameters of the difference in geometry and color information between the two grids, the fusion operation is performed only on the elevation values:

[0070]

[0071] in, e local express The elevation value of the average point that needs to be merged;

[0072] (2) If and , no fusion operation is performed, only The average point information carried by the grid;

[0073] (3) Otherwise, use the ISODATA clustering algorithm to generate a new average point;

[0074] After the fusion is completed, the latest fused DEM model will be obtained S t .

[0075] Preferably, any pre-operative vertex feature after splicing and updating is expressed as:

[0076]

[0077] in, Represents a three-dimensional vertex υ i The coordinates of E i , Represent three-dimensional vertices υ i Young's modulus and Poisson's rate are used to characterize important information for anisotropic deformation reasoning; Represents a three-dimensional vertex υ i The corresponding normal vector is, 、 、 Respectively represent the normal vector in x , y and z Components on the axis; is the corresponding intraoperative point cloud feature; i Represents the index of a three-dimensional vertex, N m Represents the number of three-dimensional vertices.

[0078] Preferably, the graph convolutional neural network includes a fully connected layer, a graph residual convolution block, a group normalization layer and a ReLU activation layer; the concatenated and updated preoperative vertex features are used as inputs of the graph convolutional neural network, and the deformed preoperative model is inferred and outputted, including:

[0079] The concatenated updated features of each pre-operative vertex are used as the input of the first fully connected layer to achieve dimensionality reduction;

[0080] The vertex features after dimension reduction are used as inputs of four consecutive graph residual convolution blocks to achieve aggregation and update of vertex features; each of the graph residual convolution blocks sequentially includes a first group of normalization layers, a first ReLU activation layer, a first fully connected layer, a second group of normalization layers, a second ReLU activation layer, a graph convolution layer, a third group of normalization layers, a third ReLU activation layer and a second fully connected layer;

[0081] The output of the last graph residual convolution block is passed through the group normalization layer, the ReLU activation layer and the last fully connected layer to infer and output the deformed pre-operative model.

[0082] Preferably, the loss function is composed of the following two weighted parts:

[0083] (1) Losses used to ensure compliance with geometric deformation intuition:

[0084] Since the preoperative model before and after deformation maintains the same three-dimensional vertex index relationship, the detection feature points of the preoperative model after deformation are obtained based on the detection feature points of the preoperative model before deformation; the detection feature points of the preoperative model after deformation are recorded as , The three-dimensional coordinates of the detected feature point are represented by the nearest detected feature point corresponding to the intraoperative point cloud. , Represents the three-dimensional coordinates of the nearest detected feature point, then the loss function of this part is expressed as:

[0085]

[0086] in, represents the loss used to ensure compliance with the intuitive geometric deformation, N p Indicates the number of detected feature points; represents the L2 norm;

[0087] (2) Smooth deformation loss: A smooth deformation regularization term is used to prevent the network from generating noisy and low-quality meshes, which is expressed as follows:

[0088]

[0089] in, represents the smooth deformation loss; υ i The preoperative model before deformation M pre The three-dimensional vertices of The preoperative model after deformation M est The three-dimensional vertices of N mrepresents the number of three-dimensional vertices; L(*) represents the Laplace-Beltrami operator; represents the L1 norm.

[0090] A real-time registration system from preoperative model to intraoperative point cloud based on deep learning, including:

[0091] An acquisition module, used for acquiring a preoperative model reconstructed in advance from preoperative CT data of the patient;

[0092] The reconstruction module is used for the flexible and fast 3D reconstruction method in vivo based on the digital elevation model to obtain the intraoperative point cloud corresponding to the current color image frame in laparoscopic surgery;

[0093] An extraction module, used for extracting preoperative vertex features characterizing the anisotropic properties of the preoperative model, and splicing the preoperative vertex features with corresponding intraoperative point cloud features;

[0094] The inference module is used to use each pre-operative vertex feature after splicing and updating as the input of the graph convolutional neural network, and infer and output the deformed pre-operative model;

[0095] The registration module is used to use the detection feature point pairs with corresponding relationships between the preoperative model and the intraoperative point cloud as supervision signals, construct a loss function to train the graph convolutional neural network until convergence, and use the deformed preoperative model output by the final reasoning as a mesh three-dimensional model for real-time registration.

[0096] A computer-readable storage medium stores a computer program, which can implement the real-time registration method of a preoperative model to an intraoperative point cloud as described above when a processor executes the computer program.

[0097] An electronic device, comprising:

[0098] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for performing real-time registration of a preoperative model to an intraoperative point cloud as described above.

[0099] (III) Beneficial effects

[0100] The present invention provides a real-time registration method from a preoperative model to an intraoperative point cloud based on deep learning. Compared with the prior art, it has the following beneficial effects:

[0101] In the present invention, a pre-reconstructed preoperative model is obtained; a real-time intraoperative point cloud is obtained based on a flexible and rapid three-dimensional reconstruction method in vivo of a digital elevation model; preoperative vertex features that characterize the anisotropic properties of the preoperative model are extracted, and the preoperative vertex features are spliced ​​with the intraoperative point cloud features corresponding to the intraoperative point cloud, and used as the input of a graph convolutional neural network to infer and output the deformed preoperative model; a pair of detected feature points with corresponding relationships is used as a supervisory signal, and a loss function is constructed to train the graph convolutional neural network until convergence, and finally a real-time registered mesh three-dimensional model is output. Based on a deep learning network framework including feature extraction, feature point detection, and deformation prediction, real-time registration of a non-rigid preoperative model to an intraoperative point cloud is achieved to meet the requirements of a real minimally invasive laparoscopic surgery scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0103] Figure 1 A deep learning network framework provided for an embodiment of the present invention;

[0104] Figure 2 A block diagram of a real-time registration method from a preoperative model to an intraoperative point cloud based on deep learning provided by an embodiment of the present invention;

[0105] Figure 3 A block diagram of a method for flexible and rapid three-dimensional reconstruction of an in vivo body based on a digital elevation model provided by an embodiment of the present invention;

[0106] Figure 4 A complete technical roadmap for a method for flexible and rapid three-dimensional reconstruction of the body based on a digital elevation model provided by an embodiment of the present invention;

[0107] Figure 5 A grid-based Schematic diagram of a centrosymmetric grid set;

[0108] Figure 6 A schematic diagram of the structure of a graph convolutional neural network provided by an embodiment of the present invention;

[0109] Figure 7 A schematic diagram of the structure of a graph residual convolution block provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0110] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0111] The embodiments of the present application provide a real-time registration method from a preoperative model to an intraoperative point cloud based on deep learning, thereby solving the technical problem that the prior art only realizes the rigid transformation of the preoperative and intraoperative space but does not take into account the flexible deformation of tissues in the in vivo environment.

[0112] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0113] The embodiment of the present invention intends to achieve non-rigid registration from the preoperative model to the intraoperative point cloud based on deep learning, which is more in line with the requirements of minimally invasive laparoscopic surgery for real-time visual feedback. At the same time, through the analysis of the anatomical structure of abdominal organs, it can be seen that the organs and their internal vascular structures and tumors have different biomechanical properties, and the objective laws of interaction between organs need to be considered in the preoperative mesh modeling process.

[0114] Based on this, the embodiment of the present invention proposes a deep learning network framework that takes into account the anisotropic characteristics of the preoperative model and can achieve real-time non-rigid preoperative model to intraoperative point cloud registration, such as Figure 1 The network framework can predict the real-time displacement field of different tissue structures in the laparoscopic video, taking into account the biomechanical properties of different tissue structures inside the organ, and realize non-rigid registration fusion before and during surgery.

[0115] It should be noted that the term "anisotropic properties" refers to the different stiffness and strength of tissues in different directions to meet the needs of the metabolic system under various complex motion and load conditions. The embodiment of the present invention defines parameters such as Young's modulus and Poisson's rate of organs in different directions to characterize the anisotropic properties of the preoperative model.

[0116] In particular, the embodiments of the present invention also take into account that in traditional laparoscopic surgery, doctors can often only observe the patient's internal body scene through two-dimensional images, with poor three-dimensional perception and limited observation field of view. Therefore, the embodiments of the present invention hope to incrementally and quickly reconstruct the internal body three-dimensional model through the color image frame sequence of the internal body tissue taken by the laparoscope to obtain the intraoperative point cloud, thereby solving the technical difficulties of the lack of reconstruction of the entire scene and the slow reconstruction speed in the internal body three-dimensional reconstruction, thereby enhancing the doctor's intraoperative three-dimensional perception ability, expanding the surgical field of view, and enhancing the safety of the operation.

[0117] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0118] Embodiment 1:

[0119] like Figure 2 As shown, an embodiment of the present invention provides a real-time registration method from a preoperative model to an intraoperative point cloud based on deep learning, comprising:

[0120] S1. Obtain a preoperative model reconstructed from preoperative CT data of the patient;

[0121] S2, a flexible and fast 3D reconstruction method in vivo based on digital elevation model, to obtain the intraoperative point cloud corresponding to the current color image frame in laparoscopic surgery;

[0122] S3, extracting preoperative vertex features characterizing the anisotropic properties of the preoperative model, and splicing the preoperative vertex features with corresponding intraoperative point cloud features;

[0123] S4, taking each pre-operative vertex feature after splicing and updating as the input of the graph convolutional neural network, and inferring and outputting the deformed pre-operative model;

[0124] S5. Using the detection feature point pairs with corresponding relationships between the preoperative model and the intraoperative point cloud as supervision signals, constructing a loss function to train the graph convolutional neural network until convergence, and using the deformed preoperative model output by the final reasoning as a mesh three-dimensional model for real-time registration.

[0125] In an embodiment of the present invention, based on a deep learning network framework including feature extraction, feature point detection and deformation prediction, real-time registration of a non-rigid preoperative model to an intraoperative point cloud is achieved to meet the requirements of a real minimally invasive laparoscopic surgery scenario.

[0126] Next, we will introduce the steps of the above scheme in detail:

[0127] In step S1 , a preoperative model reconstructed in advance using preoperative patient CT data is acquired.

[0128] In this step, the preoperative (grid) model is reconstructed based on a series of 2D image data from a series of CT scans. ,in N m It represents the number of three-dimensional vertices in the preoperative model, and the number 3 represents the dimension information of the three-dimensional vertex coordinates.

[0129] In step S2, the intraoperative point cloud corresponding to the current color image frame in laparoscopic surgery is obtained based on the in vivo flexible rapid 3D reconstruction method of the digital elevation model, such as Figure 3 As shown, the specific steps are as follows:

[0130] S21, obtaining a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence;

[0131] S22, using a pinhole camera model, restoring each frame of the color image and its depth map into a one-to-one corresponding color point cloud;

[0132] S23, using the improved digital elevation model, constructing a local three-dimensional model corresponding to each of the color point clouds;

[0133] S24. Obtain a deformation field composed of a series of deformation nodes, including:

[0134] Reconstruct the current local three-dimensional model, and use the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; minimize the total energy as the objective function, and optimize the deformation vector of each deformation node of the current deformation field;

[0135] S25, using the current deformation field to warp the last fused three-dimensional model to obtain a warped three-dimensional model;

[0136] S26, fusing the warped three-dimensional model and the current local three-dimensional model to obtain the current fused three-dimensional model and use it as the intraoperative point cloud corresponding to the current color image frame.

[0137] Through the above-mentioned three-dimensional reconstruction method, the embodiment of the present invention can generate a three-dimensional model of the three-dimensional scene in the body in real time, helping doctors to observe small tissue structures during surgery; and can splice time-series color image frames to form a relatively complete three-dimensional scene, thereby expanding the surgical field of view.

[0138] like Figure 4 As shown, Figure 4 A complete technical roadmap of the method for flexible and rapid 3D reconstruction of the body based on a digital elevation model provided by an embodiment of the present invention is given. Figure 4 The various steps of the above three-dimensional reconstruction method are introduced in detail:

[0139] In step S21, a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence are obtained.

[0140] In this step, a color image frame sequence of laparoscopic surgery is obtained. And its corresponding depth map sequence ,in t=0,1,2,… Represents a time series. Exemplarily, the depth map here can be obtained using a depth estimation algorithm such as MIOL.

[0141] In step S22, each frame of color image and its depth map are restored into a one-to-one corresponding color point cloud using a pinhole camera model.

[0142] This step can be simply described as:

[0143]

[0144] in, x is the restored point cloud 3D coordinate, d uv Depth map At coordinates ( u , v ), K is the camera intrinsic parameter matrix, c for x The color of the dots, C uv Represents a color map ( u , v ) coordinates, traversing the color image frame All ( u , v )coordinate.

[0145] Finally, the obtained x Collection and c The collection together constitutes the colored point cloud.

[0146] In step S23, a local three-dimensional model corresponding to each of the color point clouds is constructed using the improved digital elevation model.

[0147] This step converts the restored color point cloud into a digital elevation model (DEM), that is, the improved digital elevation model DEM is used to represent the local three-dimensional model corresponding to each color point cloud. S t .

[0148] It should be noted that an improved digital elevation model consists of two parts: 1. Initial coordinates O = (x0, y0, 0). 2. Grid set ,in i and j Respectively represent the row and column index of the grid. The grid has a uniform grid size B , each grid has a single or multiple average points, each of which carries its position information and color information .

[0149] Specifically, this step includes:

[0150] S231, distribute the colored point cloud in the grid divided by the XOY plane

[0151] When calculating the XOY plane size of each color point cloud, according to the uniform grid size B The XOY plane is divided into grids, so that each grid of the grid set contains a number of points of the color point cloud, and each point carries its position information and color information.

[0152] S232. Determine the average number of points that should be retained in each grid

[0153] Next, we use the information entropy index function to calculate the information entropy index function value of each grid:

[0154]

[0155] in, E ij express The information entropy index function value is N Indicates the number of points contained in the grid, α represents the weight, Indicates the operation of extracting the z component of a three-dimensional vector;

[0156] Indicates the operation of converting RGB color to grayscale value;

[0157] p k represents the kth point in the colored point cloud contained in the grid, c k It represents the RGB value of the point.

[0158] If the value of this indicator function is less than the first preset threshold, it is considered that the corresponding grid retains only one average point, which is called a single-point grid; otherwise, multiple average points are retained, which is called a multi-point grid.

[0159] S233, calculate the position and color of the average point

[0160] (1) For each of the single-point grids

[0161] According to the position information of each point in the grid, combined with the initial coordinates of the improved digital elevation model, the position information of the retained single average point is calculated; the calculation formula is as follows:

[0162]

[0163] And according to the color information of each point in the grid, the color information of a single average point is obtained by averaging; the calculation formula is as follows:

[0164]

[0165] (2) For each of the multi-point grids

[0166] The ISODATA clustering algorithm is used to automatically generate the location information of multiple retained average points.

[0167] And based on the color information of each group of aggregated points, the color information of multiple retained average points is obtained by averaging.

[0168] At this point, the local three-dimensional model corresponding to each of the color point clouds is obtained.

[0169] In step S24, a deformation field consisting of a series of deformation nodes is obtained, including:

[0170] It should be noted that the embodiment of the present invention assumes that the deformation field here is composed of a series of deformation nodes, each of which has its three-dimensional position , and both carry a deformation vector .

[0171] Accordingly, this step includes:

[0172] S241, reconstructing the current local three-dimensional model, and using the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field, specifically including:

[0173] It should be noted that, in order to simplify the calculation, a reconstructed DEM model with only a single-point grid is reconstructed here to meet the needs of deformation field sampling, that is, referring to the processing method of the single-point grid, the original multi-point grids of the current local 3D model are converted into single-point grids one by one, and the reconstructed model is obtained by combining the original single-point grids;

[0174] The VIP index function is used to calculate the importance index value of each grid of the reconstructed model. If it is greater than the second preset threshold, the grid is considered to be of high importance and is called an important grid. The calculation formula is as follows:

[0175]

[0176] in, Sym Represented by a grid is a set of 4 pairs of centrally symmetric grids, such as Figure 5 As shown, Sym = The grids with letter subscripts correspond to the grids indicated by the corresponding letters in the representation diagrams. and Indicates the elevation value of the average point in the grid x and grid y. Is a grid The elevation value of . and Represents the RGB color value of the average point of grid x and grid y, βRepresents a weight value.

[0177] Since the deformation nodes are sampled from the average points in the grid, the important grids of the reconstructed model are sampled first, and the position information of the single average point in each of the important grids is used as the three-dimensional position of the corresponding deformation node of the current deformation field; then the non-important grids are traversed, and the Euclidean distance between the single average point in the current non-important grid and the sampled grid is calculated. If it is greater than the third preset threshold, the position of the single average point in the non-important grid is used as the three-dimensional position of the corresponding deformation node of the current deformation field.

[0178] S242, taking minimization of total energy as the objective function, optimizing and solving the deformation vector of each deformation node of the current deformation field.

[0179] After constructing the DEM and deformation field of the two frames, the following objective function is introduced to optimize the calculation of the deformation vector in the deformation field so that the two DEM frames can be registered and fused:

[0180]

[0181] in, Represents the current deformation field Minimize the total energy under t Indicates the frame label; λ represents the weight of the motion consistency regularization term; γ represents the loss of TPS energy term;

[0182] E data Represents the data energy term of the deformation vector supervised by optical flow information; s i and t i Respectively represent the 3D coordinates of a pair of matching points on the point cloud generated by the previous color image frame and the current image frame, N m is the logarithm of three-dimensional matching points; Indicates the three-dimensional matching point s i The predicted deformation vector at is obtained by weighted average of the deformation vectors of the deformation nodes in the neighborhood. The calculation method is expressed as , represents the neighborhood, w ik represents the corresponding weight, , They represent the first k The three-dimensional position and deformation vector of each deformation node;

[0183] E MCTThe regularization term representing the consistency of motion is used to ensure that the motion of adjacent deformation nodes maintains a similar trend; N w Indicates the number of deformation nodes; Represents the current deformation field i The deformation vector of the deformation node, , Respectively represent its adjacent deformation nodes j The three-dimensional position and deformation vector of w ij Indicates the corresponding weight;

[0184] E TPS The TPS energy term represents the change in the surface supervision elevation interpolated by TPS; N a Represents the last fused 3D model S t-1 The number of all average points in ; Represents taking out the three-dimensional vector z Component operations; express S t -1 No. i The elevation value of the average point; Indicates the latest elevation value of the point predicted by the TPS algorithm.

[0185] Then simplify the objective function into matrix form to facilitate accelerated calculation:

[0186] In order to speed up this optimization process, we use Represents the deformation vectors carried by all deformation nodes, and obtains the following matrix forms respectively:

[0187]

[0188] Among them, the superscript T represents transpose, represents the set of real numbers;

[0189] Represents the first sparse matrix, the first element F ij = ;

[0190] , represents the three-dimensional matching point s i A collection of;

[0191] , represents the three-dimensional matching point ti A collection of;

[0192] represents the identity matrix;

[0193] Represents the second sparse matrix, the second element Y ij = ;

[0194] Represents the third sparse matrix, the third element M ij = , express S t-1 No. i The three-dimensional position of the average point;

[0195] , represents the three-dimensional position A collection of;

[0196] , represents the three-dimensional position predicted by the TPS algorithm A collection of; , and Respectively represent the extraction of three-dimensional vectors x and y Quantity operation.

[0197] At this point, the gradient of the objective function is obtained:

[0198]

[0199] The CUBLAS library is used to optimize and accelerate the solution of the objective function, and finally the deformation vector of each deformation node of the current deformation field is obtained.

[0200] In step S25, the last fused three-dimensional model is warped using the current deformation field to obtain a warped three-dimensional model.

[0201] This step uses the deformation field information to update the average point information (i.e., warp) and The warping process is represented by:

[0202] The warped 3D model The initial coordinates of O w Set as current local 3D model The initial coordinates of ;

[0203] For the previous fused 3D model S t-1 All single-point grids, update the row index of the grid i , column index j And the elevation value ; The update formula is as follows:

[0204]

[0205] in,( i w , j w , e w ) represents the index and elevation value of the average point in the updated single-point grid; v t Indicates the change in the grid index of the average point; Indicates p The predicted deformation vector at a single average point within a single-point grid; O t-1 express S t-1 The initial coordinates of O t-1 , B Indicates a uniform grid size;

[0206] For the previous fused 3D model S t-1 All multi-point grids, apply the current deformation field to all average points in the grid; the update formula is as follows:

[0207]

[0208] in, Indicates i The average point coordinates in the deformed multi-point grid, It is i The average point coordinates in the undeformed multi-point grid, It is in i The predicted deformation vectors at the deformation points in the multi-point grid;

[0209] Finally, the warped 3D model is obtained. .

[0210] In step S26, the warped three-dimensional model and the current local three-dimensional model are fused to obtain a current fused three-dimensional model.

[0211] The warped 3D model in this step is weightedly fused with the current local 3D model, and the Represents the fusion process, including:

[0212] Calculate the current local 3D model 3D model after warping The fusion weight of each pair of coincident grids is as follows:

[0213]

[0214] in, express Upper Grid The fusion weight of Indicates the number of average points in the grid; represents the VIP indicator function;

[0215] Different fusion strategies are adopted according to different situations:

[0216] (1) If and ,in express On the grid The fusion weights of overlapping meshes, represents the average number of points in the grid, To express the hyperparameters of the difference in geometry and color information between the two grids, the fusion operation is performed only on the elevation values:

[0217]

[0218] Among them, elocal means The elevation value of the average point that needs to be merged;

[0219] (2) If and , no fusion operation is performed, only The average point information carried by the grid;

[0220] (3) Otherwise, use the ISODATA clustering algorithm to generate a new average point;

[0221] After the fusion is completed, the latest fused DEM model will be obtained S t .

[0222] Finally, the obtained S t As the intraoperative point cloud corresponding to the current color image frame ,in N y It represents the number of three-dimensional vertices of the intraoperative point cloud, and the number 6 represents the six-dimensional information composed of the three-dimensional coordinates and RGB colors of the intraoperative point cloud.

[0223] Thus, the embodiment of the present invention has designed a set of real-time tissue deformation estimation algorithms, which can construct a complete three-dimensional model by splicing and fusing point cloud information at different times in a short time, and then obtain the intraoperative point cloud. In addition, the constructed three-dimensional model expression based on the digital elevation model greatly reduces the amount of model calculation.

[0224] In step S3, the preoperative vertex features characterizing the anisotropic properties of the preoperative model are extracted, and the preoperative vertex features are spliced ​​with the corresponding intraoperative point cloud features, as follows:

[0225] (1) Use feature extraction methods such as PointNet++ and FPFH to obtain high-dimensional features of intraoperative point clouds and obtain a dimension of Intraoperative point cloud features ,in is a feature dimension hyperparameter that can be adjusted.

[0226] (2) Extracting preoperative vertex features that characterize the anisotropic properties of the preoperative model, including coordinates of three-dimensional vertices and their Young's modulus, Poisson's rate, normal vector, etc.

[0227] (3) The preoperative vertex features are concatenated with the corresponding intraoperative point cloud features. Any concatenated and updated preoperative vertex features are expressed as:

[0228]

[0229] in, Represents a three-dimensional vertex υ i The coordinates of E i , Represent three-dimensional vertices υ i Young's modulus and Poisson's rate are used to characterize important information for anisotropic deformation reasoning; Represents a three-dimensional vertex υ i The corresponding normal vector is, 、 、 Respectively represent the normal vector in x , y and z Components on the axis; is the corresponding intraoperative point cloud feature; i Represents the index of a three-dimensional vertex, N m Represents the number of three-dimensional vertices.

[0230] In step S4, each preoperative vertex feature after splicing and updating is used as the input of the graph convolutional neural network, and the deformed preoperative model is inferred and output.

[0231] In this step, the graph convolution operation uses etc., among which represents the adjacency matrix after row normalization, which can be calculated based on the three-dimensional vertices of the preoperative model and the connection relationship between the vertices; It is a weight matrix that is continuously updated as training progresses. The graph convolution operation performs a full connection operation on each 3D vertex, followed by neighborhood averaging to obtain a smoothly deformed 3D mesh model.

[0232] like Figure 6 As shown, the graph convolutional neural network includes a fully connected layer, a graph residual convolution block, a group normalization layer and a ReLU activation layer. Figure 6 The symbols on the connecting lines between the upper layers indicate the changes in dimensional information.

[0233] Accordingly, the spliced ​​and updated pre-operative vertex features are used as the input of the graph convolutional neural network, and the deformed pre-operative model is inferred and output, including:

[0234] First, concatenate the updated preoperative vertex features As the input of the first fully connected layer, its dimension is reduced to 1 / 2 of the original input through the fully connected layer.

[0235] Secondly, the vertex features after dimensionality reduction are used as the input of four consecutive graph residual convolution blocks to achieve the aggregation and update of vertex features. The mathematical expression is:

[0236]

[0237] in, X (l) It is l The node feature matrix of the layer, for The degree matrix of . The diagonal elements of yes No. i The sum of the rows, W (l) It is l Layer weight matrix, σ(*) represents the activation function, which can be ReLU, LeakyReLU, etc.

[0238] Please note that Figure 7As shown, each of the graph residual convolution blocks sequentially includes a first group of normalized layers, a first ReLU activation layer, a first fully connected layer, a second group of normalized layers, a second ReLU activation layer, a graph convolution layer, a third group of normalized layers, a third ReLU activation layer, and a second fully connected layer. Figure 7 The characters under the box represent the feature dimension information after passing through this layer.

[0239] by N m × k Taking the input of feature information of dimension as an example, the implementation details are described as follows: the input layer represents the initial form of the input feature, and then group normalization normalizes the input data to adjust the distribution of the data to speed up the convergence speed and stabilize the training process. This step does not change the feature dimension. The ReLU activation layer applies the ReLU activation function to perform nonlinear transformation on the data to increase the expression ability of the model. The fully connected layer applies full connection to each node to transform the feature dimension from N m × k Reduce to N m ×0.5 k , and then apply group normalization and ReLU activation function again. Next, the graph convolution layer performs graph convolution operation to fuse the feature information of each node and its neighborhood, enhance the local smoothness of feature representation, and the output feature dimension is still N m ×0.5 k Finally, after group normalization, ReLU activation layer and full connection, feature information is output.

[0240] Finally, the output of the last graph residual convolution block is passed through the group normalization layer, the ReLU activation layer and the last fully connected layer to infer and output the deformed pre-operative model.

[0241] In step S5, the detection feature point pairs with corresponding relationships between the preoperative model and the intraoperative point cloud are used as supervisory signals, and a loss function is constructed to train the graph convolutional neural network until convergence, and the deformed preoperative model output by the final reasoning is used as a mesh three-dimensional model for real-time registration.

[0242] In this step, feature point detection is first performed as follows:

[0243] Detect and extract the preoperative model Point cloud Specific characteristic tissue structures, such as the falciform ligament and ridges in the liver, have a corresponding relationship between the characteristic detection points extracted from different tissue structures before and during surgery.

[0244] The three-dimensional feature line contour information extracted in this part will be used as the supervision information of the network, respectively. and Indicates that N p and N q They represent the number of feature points in the preoperative 3D model and the intraoperative model respectively, and the number 3 represents the dimensional information of the 3D feature point coordinates.

[0245] Next, network reasoning is performed. It should be noted that the network reasoning time of the embodiment of the present invention is much lower than that of the traditional finite element analysis-based method.

[0246] Specifically, the loss function is set to consist of the following two weighted parts:

[0247] (1) Loss used to ensure compliance with intuitive geometric deformation: The network output is constrained by the distance between the detected feature points of the deformed preoperative model and the detected feature points of the intraoperative point cloud, so that it conforms to the deformation law of the key tissue structure.

[0248] Since the preoperative model before and after deformation maintains the same three-dimensional vertex index relationship, the detection feature points of the preoperative model after deformation are obtained based on the detection feature points of the preoperative model before deformation; the detection feature points of the preoperative model after deformation are recorded as , The three-dimensional coordinates of the detected feature point are represented by the nearest detected feature point corresponding to the intraoperative point cloud. , Represents the three-dimensional coordinates of the nearest detected feature point, then the loss function of this part is expressed as:

[0249]

[0250] in, represents the loss used to ensure compliance with the intuitive geometric deformation, N p Indicates the number of detected feature points; represents the L2 norm;

[0251] (2) Smooth deformation loss: A smooth deformation regularization term is used to prevent the network from generating noisy and low-quality meshes, which is expressed as follows:

[0252]

[0253] in, represents the smooth deformation loss; υ i The preoperative model before deformation M pre The three-dimensional vertices of The preoperative model after deformation Mest The three-dimensional vertices of N m represents the number of three-dimensional vertices; L(*) represents the Laplace-Beltrami operator; represents the L1 norm.

[0254] So far, the embodiment of the present invention has completed the entire process of the real-time registration method from the preoperative model to the intraoperative point cloud based on deep learning, and can realize the real-time superposition of the deformed preoperative model and its internal vascular structure, tumor body, etc. into the surgical scene, achieving the effect of augmented reality and assisting doctors in visual perception in complex and restricted laparoscopic scenes.

[0255] Embodiment 2:

[0256] The embodiment of the present invention provides a real-time registration system from a preoperative model to an intraoperative point cloud based on deep learning, comprising:

[0257] An acquisition module, used for acquiring a preoperative model reconstructed in advance from preoperative CT data of the patient;

[0258] The reconstruction module is used for the flexible and fast 3D reconstruction method in vivo based on the digital elevation model to obtain the intraoperative point cloud corresponding to the current color image frame in laparoscopic surgery;

[0259] An extraction module, used for extracting preoperative vertex features characterizing the anisotropic properties of the preoperative model, and splicing the preoperative vertex features with corresponding intraoperative point cloud features;

[0260] The inference module is used to use each pre-operative vertex feature after splicing and updating as the input of the graph convolutional neural network, and infer and output the deformed pre-operative model;

[0261] The registration module is used to use the detection feature point pairs with corresponding relationships between the preoperative model and the intraoperative point cloud as supervision signals, construct a loss function to train the graph convolutional neural network until convergence, and use the deformed preoperative model output by the final reasoning as a mesh three-dimensional model for real-time registration.

[0262] Embodiment 3:

[0263] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When a processor executes the computer program, the real-time registration method from the preoperative model to the intraoperative point cloud as described in Example 1 can be implemented.

[0264] Embodiment 4:

[0265] An embodiment of the present invention provides an electronic device, including:

[0266] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for performing real-time registration of a preoperative model to an intraoperative point cloud as described above.

[0267] It can be understood that the real-time registration system, storage medium and electronic device of the preoperative model to the intraoperative point cloud based on deep learning provided in the embodiments of the present invention correspond to the real-time registration method of the preoperative model to the intraoperative point cloud based on deep learning provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the real-time registration method, which will not be repeated here.

[0268] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0269] 1. In an embodiment of the present invention, based on a deep learning network framework including feature extraction, feature point detection and deformation prediction, real-time registration of a non-rigid preoperative model to an intraoperative point cloud is achieved to meet the requirements of a real minimally invasive laparoscopic surgery scenario.

[0270] 2. Different from traditional methods and solutions, the network architecture designed in the embodiment of the present invention can fully learn the anisotropic characteristics of the preoperative model, conform to the objective laws of deformation of human tissues and organs, and can realize flexible deformation registration of three-dimensional bodies.

[0271] 3. The embodiments of the present invention can realize real-time superposition of the deformed preoperative model and its internal vascular structure, tumor body, etc. into the surgical scene, thereby achieving an augmented reality effect and assisting doctors in visual perception in complex and restricted laparoscopic scenes.

[0272] 4. The embodiment of the present invention can generate a three-dimensional point cloud model of the in-vivo three-dimensional scene in real time, helping doctors to observe small tissue structures during surgery; and can splice time-series color image frames to form a relatively complete three-dimensional scene, expanding the surgical field of view.

[0273] 5. The embodiment of the present invention designs a set of real-time tissue deformation estimation algorithms, which can construct a complete three-dimensional model by splicing and fusing point cloud information at different times in a short time, and then obtain intraoperative point cloud. In addition, the constructed three-dimensional model expression based on the digital elevation model greatly reduces the amount of model calculation.

[0274] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0275] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time registration method from preoperative model to intraoperative point cloud based on deep learning, characterized in that: include: Obtaining a preoperative model reconstructed in advance using preoperative CT data of the patient; An in vivo flexible and fast 3D reconstruction method based on digital elevation model is used to obtain the intraoperative point cloud corresponding to the current color image frame during laparoscopic surgery; Extracting preoperative vertex features characterizing the anisotropic properties of the preoperative model, and splicing the preoperative vertex features with corresponding intraoperative point cloud features; Each pre-operative vertex feature after splicing and updating is used as the input of the graph convolutional neural network, and the deformed pre-operative model is inferred and output; Using the detection feature point pairs with corresponding relationships between the preoperative model and the intraoperative point cloud as supervision signals, constructing a loss function to train the graph convolutional neural network until convergence, and using the deformed preoperative model output by final reasoning as a mesh three-dimensional model for real-time registration; The extracting of preoperative vertex features characterizing the anisotropic properties of the preoperative model and splicing the preoperative vertex features with corresponding intraoperative point cloud features include: Use feature extraction method to obtain high-dimensional features of intraoperative point cloud, and obtain the dimension Intraoperative point cloud features ,in is the feature dimension hyperparameter, N m Indicates the number of three-dimensional vertices; the feature extraction method includes PointNet++ or FPFH; Extracting preoperative vertex features characterizing the anisotropic properties of the preoperative model, including coordinates of three-dimensional vertices and their Young's modulus, Poisson's rate, and normal vectors; The preoperative vertex features are spliced ​​with the corresponding intraoperative point cloud features. Any preoperative vertex feature after splicing and updating is expressed as: in, Represents a three-dimensional vertex υ i The coordinates of E i , Represent three-dimensional vertices υ i Young's modulus and Poisson's rate are used to characterize important information for anisotropic deformation reasoning; Represents a three-dimensional vertex υ i The corresponding normal vector is, 、 、 Respectively represent the normal vector in x , y and z Components on the axis; is the corresponding intraoperative point cloud feature; i Represents the index of a 3D vertex.

2. The real-time registration method according to claim 1, characterized in that: The in vivo flexible rapid 3D reconstruction method based on the digital elevation model obtains the intraoperative point cloud corresponding to the current color image frame in the laparoscopic surgery, including: Acquire a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence; Using the pinhole camera model, each frame of color image and its depth map are restored into a one-to-one corresponding color point cloud; Using the improved digital elevation model, constructing a local three-dimensional model corresponding to each of the color point clouds; Gets a deformation field consisting of a series of deformation nodes, including: Reconstruct the current local three-dimensional model, and use the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; minimize the total energy as the objective function, and optimize the deformation vector of each deformation node of the current deformation field; Using the current deformation field to warp the last fused three-dimensional model, to obtain a warped three-dimensional model; The warped three-dimensional model and the current local three-dimensional model are fused to obtain the current fused three-dimensional model as the intraoperative point cloud corresponding to the current color image frame.

3. The real-time registration method according to claim 2, characterized in that: The improved digital elevation model consists of two parts: initial coordinates and a grid set; the improved digital elevation model is used to construct a local three-dimensional model corresponding to each color point cloud, including: Calculating the XOY plane size of each of the color point clouds, dividing the XOY plane into grids according to a uniform grid size, so that each grid of the grid set contains a number of points of the color point cloud, and each point carries its position information and color information; The information entropy index function is used to calculate the information entropy index function value of each grid. If it is less than the first preset threshold, it is considered that the corresponding grid only retains one average point, which is called a single-point grid; otherwise, multiple average points are retained, which is called a multi-point grid. For each of the single-point grids, the position information of the single average point is calculated based on the position information of each point in the grid and the initial coordinates of the improved digital elevation model; and the color information of the single average point is averaged based on the color information of each point in the grid; For each of the multi-point grids, the ISODATA clustering algorithm is used to automatically generate the position information of the multiple average points retained; and based on the color information of each group of aggregated points, the color information of the multiple average points retained is obtained by averaging respectively; Obtaining a local three-dimensional model corresponding to each of the color point clouds; and / or The reconstructing the current local three-dimensional model and sampling the three-dimensional position of each deformation node of the current deformation field by using the importance of each grid of the reconstructed model includes: Referring to the processing method of single-point grids, the original multi-point grids of the current local 3D model are converted into single-point grids one by one, and the reconstructed model is obtained by combining the original single-point grids; The VIP index function is used to calculate the importance index value of each grid of the reconstructed model. If it is greater than the second preset threshold, the grid is considered to be of high importance and is called an important grid. The calculation formula is as follows: in, represents the VIP indicator function; Indicates i Line j A grid of columns; Sym Represented by a grid It is a set of 4 pairs of grids that are centrally symmetric; and Represents the elevation value of the average point in grid x and grid y, Representation Grid The elevation value of Indicates the operation of converting RGB color value to grayscale value; and Represents the RGB color value of the average point in grid x and grid y; Representation Grid RGB color value; β Represents the weight value; Prioritizing sampling of important grids of the reconstructed model, and using position information of a single average point in each of the important grids as the three-dimensional position of a corresponding deformation node of the current deformation field; Then, the non-important grids are traversed, and the Euclidean distance between a single average point in the current non-important grid and the sampled grid is calculated. If the distance is greater than a third preset threshold, the position of the single average point in the non-important grid is used as the three-dimensional position of the corresponding deformation node of the current deformation field; and / or The objective function is expressed as: in, Represents the current deformation field Minimize the total energy under t Indicates the frame label; λ represents the weight of the motion consistency regularization term; γ represents the loss of TPS energy term; E data Represents the data energy term of the deformation vector supervised by optical flow information; s i and t i Respectively represent the 3D coordinates of a pair of matching points on the point cloud generated by the previous color image frame and the current image frame, N m is the logarithm of three-dimensional matching points; Indicates the three-dimensional matching point s i The predicted deformation vector at is obtained by weighted average of the deformation vectors of the deformation nodes in the neighborhood. The calculation method is expressed as , represents the neighborhood, w ik represents the corresponding weight, , They represent the first k The three-dimensional position and deformation vector of each deformation node; E MCT The regularization term representing the consistency of motion is used to ensure that the motion of adjacent deformation nodes maintains a similar trend; N w Indicates the number of deformation nodes; Represents the current deformation field i The deformation vector of the deformation node, , Respectively represent its adjacent deformation nodes j The three-dimensional position and deformation vector of w ij represents the corresponding weight; E TPS The TPS energy term represents the change in the surface supervision elevation interpolated by TPS; N a Represents the last fused 3D model S t-1 The number of all average points in ; Represents taking out the three-dimensional vector z Component operations; express S t-1 No. i The elevation value of the average point; Indicates the latest elevation value of the point predicted by the TPS algorithm; and / or The optimization step of solving the deformation vector of each deformation node of the current deformation field includes: use Represents the deformation vectors carried by all deformation nodes, and obtains the following matrix forms respectively: Among them, the superscript T represents transpose, represents the set of real numbers; Represents the first sparse matrix, the first element F ij = ; , represents the three-dimensional matching point s i A collection of; , represents the three-dimensional matching point t i A collection of; represents the identity matrix; Represents the second sparse matrix, the second element Y ij = ; Represents the third sparse matrix, the third element M ij = , express S t-1 No. i The three-dimensional position of the average point; , represents the three-dimensional position A collection of; , represents the three-dimensional position predicted by the TPS algorithm A collection of; , and Respectively represent the extraction of three-dimensional vectors x and y Operation of weight; At this point, the gradient of the objective function is obtained: The CUBLAS library is used to optimize and accelerate the solution of the objective function, and finally the deformation vector of each deformation node of the current deformation field is obtained.

4. The real-time registration method according to claim 3, characterized in that: use Represents the warping process, including: The warped 3D model The initial coordinates of O w Set as current local 3D model The initial coordinates of ; For the previous fused 3D model S t-1 All single-point grids, update the row index of the grid i , column index j And the elevation value ; The update formula is as follows: in,( i w , j w , e w ) represents the index and elevation value of the average point in the updated single-point grid; v t Indicates the change in the grid index of the average point; Indicates p The predicted deformation vector at a single average point within a single-point grid; O t-1 express S t-1 The initial coordinates of O t-1 , B Indicates a uniform grid size; For the previous fused 3D model S t-1 All multi-point grids, apply the current deformation field to all average points in the grid; the update formula is as follows: in, Indicates i The average point coordinates in the deformed multi-point grid, It is i The average point coordinates in the undeformed multi-point grid, It is in i The predicted deformation vectors at the deformation points in the multi-point grid; Finally, the warped 3D model is obtained. ; and / or use Represents the fusion process, including: Calculate the current local 3D model 3D model after warping The fusion weight of each pair of coincident grids is as follows: in, express Upper Grid The fusion weight of Indicates the number of average points in the grid; Different fusion strategies are adopted according to different situations: (1) If and ,in express On the grid The fusion weights of overlapping meshes, represents the average number of points in the grid, To express the hyperparameters of the difference in geometry and color information between the two grids, the fusion operation is performed only on the elevation values: in, e local express The elevation value of the average point that needs to be merged; (2) If and , no fusion operation is performed, only The average point information carried by the grid; (3) Otherwise, use the ISODATA clustering algorithm to generate a new average point; After the fusion is completed, the latest fused DEM model will be obtained S t .

5. The real-time registration method according to claim 1, characterized in that: The graph convolutional neural network includes a fully connected layer, a graph residual convolution block, a group normalization layer and a ReLU activation layer; the concatenated and updated preoperative vertex features are used as inputs of the graph convolutional neural network, and the deformed preoperative model is inferred and outputted, including: The concatenated updated features of each pre-operative vertex are used as the input of the first fully connected layer to achieve dimensionality reduction; The vertex features after dimension reduction are used as inputs of four consecutive graph residual convolution blocks to achieve aggregation and update of vertex features; each of the graph residual convolution blocks sequentially includes a first group of normalization layers, a first ReLU activation layer, a first fully connected layer, a second group of normalization layers, a second ReLU activation layer, a graph convolution layer, a third group of normalization layers, a third ReLU activation layer and a second fully connected layer; The output of the last graph residual convolution block is passed through the group normalization layer, the ReLU activation layer and the last fully connected layer to infer and output the deformed pre-operative model.

6. The real-time registration method according to claim 1, characterized in that: The loss function is composed of the following two weighted parts: (1) Losses used to ensure compliance with geometric deformation intuition: Since the preoperative model before and after deformation maintains the same three-dimensional vertex index relationship, the detection feature points of the preoperative model after deformation are obtained based on the detection feature points of the preoperative model before deformation; the detection feature points of the preoperative model after deformation are recorded as , The three-dimensional coordinates of the detected feature point are represented by , Represents the three-dimensional coordinates of the nearest detected feature point, then the loss function of this part is expressed as: in, represents the loss used to ensure compliance with the intuitive geometric deformation, N p Indicates the number of detected feature points; represents the L2 norm; (2) Smooth deformation loss: A smooth deformation regularization term is used to prevent the network from generating noisy and low-quality meshes, which is expressed as follows: in, represents the smooth deformation loss; υ i The preoperative model before deformation M pre The three-dimensional vertices of The preoperative model after deformation M est The three-dimensional vertices of N m represents the number of three-dimensional vertices; L(*) represents the Laplace-Beltrami operator; represents the L1 norm.

7. A real-time registration system from preoperative model to intraoperative point cloud based on deep learning, characterized in that: The method for performing the real-time registration as claimed in claim 1 comprises: An acquisition module, used for acquiring a preoperative model reconstructed in advance from preoperative CT data of the patient; The reconstruction module is used for the flexible and fast 3D reconstruction method in vivo based on the digital elevation model to obtain the intraoperative point cloud corresponding to the current color image frame in laparoscopic surgery; An extraction module, used for extracting preoperative vertex features characterizing the anisotropic properties of the preoperative model, and splicing the preoperative vertex features with corresponding intraoperative point cloud features; The inference module is used to use each pre-operative vertex feature after splicing and updating as the input of the graph convolutional neural network, and infer and output the deformed pre-operative model; The registration module is used to use the detection feature point pairs with corresponding relationships between the preoperative model and the intraoperative point cloud as supervision signals, construct a loss function to train the graph convolutional neural network until convergence, and use the deformed preoperative model output by the final reasoning as a mesh three-dimensional model for real-time registration.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when a processor executes the computer program, a real-time registration method from a preoperative model to an intraoperative point cloud as described in any one of claims 1 to 6 can be implemented.

9. An electronic device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the real-time registration method of the preoperative model to the intraoperative point cloud as described in any one of claims 1 to 6.

Images