In vivo flexible rapid 3D reconstruction method and system based on digital elevation model

Abstract

The present invention provides a method and system for flexible and rapid three-dimensional reconstruction in vivo based on a digital elevation model, and relates to the field of flexible and rapid three-dimensional reconstruction in vivo. The present invention first obtains a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence; secondly, uses a pinhole camera model to restore the color image frame and its depth map into a color point cloud; again, uses an improved digital elevation model to construct a local three-dimensional model; obtains a deformation field composed of a series of deformation nodes; then uses the current deformation field to warp the last fused three-dimensional model to obtain a warped three-dimensional model; finally, fuses the warped three-dimensional model and the current local three-dimensional model to obtain the current fused three-dimensional model. A three-dimensional model of a three-dimensional scene in vivo can be generated in real time to help doctors observe small tissue structures during surgery; and a color image frame of a time sequence can be spliced ​​to form a relatively complete three-dimensional scene, expanding the scope of the surgical field of view.

Description

Technical Field

[0001] The invention relates to the field of flexible rapid three-dimensional reconstruction of an in vivo body, and in particular to a flexible rapid three-dimensional reconstruction method and system of an in vivo body based on a digital elevation model. Background Art

[0002] Rapid three-dimensional reconstruction of flexible tissues in the body can provide doctors with the most realistic three-dimensional tissue information during surgery, enhance doctors' three-dimensional perception during surgery, and help doctors perform precise surgical operations.

[0003] In the related art, there are two original technical solutions:

[0004] 1. In-vivo SLAM algorithm, by tracking the camera pose and dense mapping, realizes the generation of in-vivo 3D model. Its main principle is to extract and match the motion between consecutive frames based on image feature points, build the in-vivo environment map based on this motion information, and finally obtain the final in-vivo 3D scene through loop detection and optimization. However, SLAM-type methods have slow reconstruction speed and low reconstruction accuracy, which makes it difficult to apply such methods to intraoperative scenarios.

[0005] Second, binocular depth estimation is performed through binocular laparoscopes to obtain a depth map for each frame, and then a point cloud is generated based on the depth map and the camera's internal parameters to achieve the construction of a three-dimensional model. However, although the depth estimation based on binocular laparoscopes meets the real-time requirements, the reconstruction range is limited and depends on the device. This type of method can only reconstruct local scenes in the body. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the deficiencies in the prior art, the present invention provides a method and system for flexible and rapid three-dimensional reconstruction of the body based on a digital elevation model, which solves the technical problem that the three-dimensional scene in the body cannot be generated in real time.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0010] A flexible and rapid three-dimensional reconstruction method for an in vivo body based on a digital elevation model, comprising:

[0011] Acquire a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence;

[0012] Using the pinhole camera model, each frame of color image and its depth map are restored into a one-to-one corresponding color point cloud;

[0013] Using the improved digital elevation model, constructing a local three-dimensional model corresponding to each of the color point clouds;

[0014] Gets a deformation field consisting of a series of deformation nodes; including:

[0015] Reconstruct the current local three-dimensional model, and use the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; minimize the total energy as the objective function, and optimize the deformation vector of each deformation node of the current deformation field;

[0016] Using the current deformation field to warp the last fused three-dimensional model, to obtain a warped three-dimensional model;

[0017] The warped three-dimensional model and the current local three-dimensional model are fused to obtain a current fused three-dimensional model.

[0018] Preferably, the improved digital elevation model consists of two parts: initial coordinates and a grid set; the method of using the improved digital elevation model to construct a local three-dimensional model corresponding to each of the color point clouds comprises:

[0019] Calculating the XOY plane size of each of the color point clouds, dividing the XOY plane into grids according to a uniform grid size, so that each grid of the grid set contains a number of points of the color point cloud, and each point carries its position information and color information;

[0020] The information entropy index function is used to calculate the information entropy index function value of each grid. If it is less than the first preset threshold, it is considered that the corresponding grid only retains one average point, which is called a single-point grid; otherwise, multiple average points are retained, which is called a multi-point grid.

[0021] For each of the single-point grids, the position information of the single average point is calculated based on the position information of each point in the grid and the initial coordinates of the improved digital elevation model; and the color information of the single average point is averaged based on the color information of each point in the grid;

[0022] For each of the multi-point grids, the ISODATA clustering algorithm is used to automatically generate the position information of the multiple average points retained; and based on the color information of each group of aggregated points, the color information of the multiple average points retained is obtained by averaging respectively;

[0023] A local three-dimensional model corresponding to each of the color point clouds is obtained.

[0024] Preferably, the reconstructing the current local three-dimensional model and sampling the three-dimensional position of each deformation node of the current deformation field by using the importance of each grid of the reconstructed model comprises:

[0025] Referring to the processing method of single-point grids, the original multi-point grids of the current local 3D model are converted into single-point grids one by one, and the reconstructed model is obtained by combining the original single-point grids;

[0026] Using the VIP index function, the importance index value of each grid of the reconstructed model is calculated. If the importance index value is greater than a second preset threshold, the grid is considered to be of high importance and is called an important grid.

[0027] Prioritizing sampling of important grids of the reconstructed model, and using position information of a single average point in each of the important grids as the three-dimensional position of a corresponding deformation node of the current deformation field;

[0028] Then, the non-important grids are traversed to calculate the Euclidean distance between a single average point in the current non-important grid and the sampled grid. If the distance is greater than the third preset threshold, the position of the single average point in the non-important grid is used as the three-dimensional position of the corresponding deformation node of the current deformation field.

[0029] Preferably, the objective function is expressed as:

[0030]

[0031] in, Represents the current deformation field Minimize the total energy under t Indicates the frame label; λ represents the weight of the motion consistency regularization term; γ represents the loss of TPS energy term;

[0032] E data Represents the data energy term of the deformation vector supervised by optical flow information; s i and t i Respectively represent the 3D coordinates of a pair of matching points on the point cloud generated by the previous color image frame and the current image frame, N m is the logarithm of three-dimensional matching points; Indicates the three-dimensional matching point s i The predicted deformation vector at is obtained by weighted average of the deformation vectors of the deformation nodes in the neighborhood. The calculation method is expressed as , represents the neighborhood, w ik represents the corresponding weight, , They represent the first k The three-dimensional position and deformation vector of each deformation node;

[0033] E MCT The regularization term representing the consistency of motion is used to ensure that the motion of adjacent deformation nodes maintains a similar trend; N w Indicates the number of deformation nodes; Represents the current deformation field i The deformation vector of the deformation node, , Respectively represent its adjacent deformation nodes j The three-dimensional position and deformation vector of w ij Indicates the corresponding weight;

[0034] E TPS The TPS energy term represents the change in the surface supervision elevation interpolated by TPS; N a Represents the last fused 3D model S t-1 The number of all average points in ; Represents taking out the three-dimensional vector z Component operations; express S t -1 No. i The elevation value of the average point; Indicates the latest elevation value of the point predicted by the TPS algorithm.

[0035] Preferably, the step of optimizing and solving the deformation vector of each deformation node of the current deformation field comprises:

[0036] use Represents the deformation vectors carried by all deformation nodes, and obtains the following matrix forms respectively:

[0037]

[0038] Among them, the superscript T represents transpose, represents the set of real numbers;

[0039] Represents the first sparse matrix, the first element F ij = ;

[0040] , represents the three-dimensional matching point s i A collection of;

[0041] , represents the three-dimensional matching point t i A collection of;

[0042] represents the identity matrix;

[0043] Represents the second sparse matrix, the second element Y ij = ;

[0044] Represents the third sparse matrix, the third element M ij = , express S t-1 No. i The three-dimensional position of the average point;

[0045] , represents the three-dimensional position A collection of;

[0046] , represents the three-dimensional position predicted by the TPS algorithm A collection of; , and Respectively represent the extraction of three-dimensional vectors x and y Operation of weight;

[0047] At this point, the gradient of the objective function is obtained:

[0048]

[0049] The CUBLAS library is used to optimize and accelerate the solution of the objective function, and finally the deformation vector of each deformation node of the current deformation field is obtained.

[0050] Preferably, use Represents the warping process; specifically includes:

[0051] The warped 3D model The initial coordinates of O w Set as current local 3D model The initial coordinates of ;

[0052] For the previous fused 3D model S t-1 All single-point grids, update the row index of the gridi , column index j And the elevation value ; The update formula is as follows:

[0053]

[0054] in,( i w , j w , e w ) represents the index and elevation value of the average point in the updated single-point network; v t Indicates the change in the grid index of the average point; Indicates p The predicted deformation vector at a single average point within a single-point network; O t-1 express S t-1 The initial coordinates of O t-1 , B Indicates a uniform grid size;

[0055] For the previous fused 3D model S t-1 All multi-point grids, apply the current deformation field to all average points in the grid; the update formula is as follows:

[0056]

[0057] in, Indicates i The average point coordinates in the deformed multi-point grid, It is i The average point coordinates in the undeformed multi-point grid, It is in i The predicted deformation vectors at the deformation points in the multi-point grid;

[0058] Finally, the warped 3D model is obtained. .

[0059] Preferably, use Represents the fusion process; specifically includes:

[0060] Calculate the current local 3D model 3D model after warping The fusion weight of each pair of coincident grids is as follows:

[0061]

[0062] in, express Upper Grid The fusion weight of Indicates the number of average points in the grid; represents the VIP indicator function;

[0063] Different fusion strategies are adopted according to different situations:

[0064] (1) If and ,in express On the grid The fusion weights of overlapping meshes, represents the average number of points in the grid, ℇ To express the hyperparameters of the difference in geometry and color information between the two grids, the fusion operation is performed only on the elevation values:

[0065]

[0066] in, e local express The elevation value of the average point that needs to be merged;

[0067] (2) If and , no fusion operation is performed, only The average point information carried by the grid;

[0068] (3) Otherwise, use the ISODATA clustering algorithm to generate a new average point;

[0069] After the fusion is completed, the latest fused DEM model will be obtained S t .

[0070] An in vivo flexible rapid three-dimensional reconstruction system based on a digital elevation model, comprising:

[0071] An acquisition module, used for acquiring a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence;

[0072] A restoration module is used to restore each frame of color image and its depth map into a one-to-one corresponding color point cloud using a pinhole camera model;

[0073] A construction module, used to construct a local three-dimensional model corresponding to each of the color point clouds using an improved digital elevation model;

[0074] The acquisition module is used to obtain the deformation field composed of a series of deformation nodes; including:

[0075] Reconstruct the current local three-dimensional model, and use the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; minimize the total energy as the objective function, and optimize the deformation vector of each deformation node of the current deformation field;

[0076] A warping module, used to warp the last fused three-dimensional model using the current deformation field to obtain a warped three-dimensional model;

[0077] The fusion module is used to fuse the warped three-dimensional model with the current local three-dimensional model to obtain the current fused three-dimensional model.

[0078] A computer-readable storage medium stores a computer program, which can implement the in-vivo flexible rapid three-dimensional reconstruction method based on a digital elevation model as described above when a processor executes the computer program.

[0079] An electronic device, comprising:

[0080] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the in vivo flexible rapid three-dimensional reconstruction method based on a digital elevation model as described above.

[0081] (III) Beneficial effects

[0082] The present invention provides a method and system for flexible and rapid three-dimensional reconstruction of the body based on a digital elevation model. Compared with the prior art, it has the following beneficial effects:

[0083] In the present invention, firstly, a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence are obtained; secondly, each frame of color image and its depth map are restored into a one-to-one corresponding color point cloud using a pinhole camera model; again, a local three-dimensional model corresponding to each color point cloud is constructed using an improved digital elevation model; a deformation field composed of a series of deformation nodes is obtained; then, the previous fused three-dimensional model is warped using the current deformation field to obtain a warped three-dimensional model; finally, the warped three-dimensional model and the current local three-dimensional model are fused to obtain the current fused three-dimensional model. The present invention can generate a three-dimensional model of a three-dimensional scene in the body in real time, helping doctors observe small tissue structures during surgery; and can splice time-series color image frames to form a relatively complete three-dimensional scene and expand the scope of surgical field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0085] Figure 1 A block diagram of a method for flexible and rapid three-dimensional reconstruction of an in vivo body based on a digital elevation model provided by an embodiment of the present invention;

[0086] Figure 2 A complete technical roadmap for a method for flexible and rapid three-dimensional reconstruction of the body based on a digital elevation model provided by an embodiment of the present invention;

[0087] Figure 3 A grid-based Schematic diagram of a centrosymmetric grid collection. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] The embodiments of the present application solve the technical problem of being unable to generate in-vivo three-dimensional scenes in real time by providing a method and system for flexible and rapid three-dimensional reconstruction of the body based on a digital elevation model.

[0090] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0091] In traditional laparoscopic surgery, doctors can only observe the patient's internal body scene through two-dimensional images, with poor three-dimensional perception and limited observation field of view. Therefore, the embodiment of the present invention hopes to incrementally and quickly reconstruct the internal body three-dimensional model through the color image frame sequence of the internal body tissue taken by laparoscope, thereby solving the technical difficulties of the lack of reconstruction of the entire scene and the slow reconstruction speed of the internal body three-dimensional reconstruction, so as to enhance the doctor's intraoperative three-dimensional perception ability, expand the surgical field of view, and enhance the safety of the operation.

[0092] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0093] Embodiment 1:

[0094] like Figure 1 As shown, an embodiment of the present invention provides a method for flexible and rapid three-dimensional reconstruction of an in vivo body based on a digital elevation model, comprising:

[0095] S1, obtaining a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence;

[0096] S2, using the pinhole camera model, restore each frame of color image and its depth map into a one-to-one corresponding color point cloud;

[0097] S3, using the improved digital elevation model to construct a local three-dimensional model corresponding to each of the color point clouds;

[0098] S4. Obtaining a deformation field consisting of a series of deformation nodes; including:

[0099] Reconstruct the current local three-dimensional model, and use the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; minimize the total energy as the objective function, and optimize the deformation vector of each deformation node of the current deformation field;

[0100] S5, using the current deformation field to warp the last fused three-dimensional model to obtain a warped three-dimensional model;

[0101] S6. Fusing the warped three-dimensional model and the current local three-dimensional model to obtain a current fused three-dimensional model.

[0102] The embodiments of the present invention can generate a three-dimensional model of a three-dimensional scene in the body in real time, helping doctors to observe fine tissue structures during surgery; and can splice time-sequential color image frames to form a relatively complete three-dimensional scene, thereby expanding the surgical field of view.

[0103] like Figure 2 As shown, Figure 2 A complete technical roadmap of the in vivo flexible rapid three-dimensional reconstruction method based on the digital elevation model provided by the embodiment of the present invention is given.

[0104] Next, we will combine Figure 2 The following are the steps of the above solution:

[0105] In step S1, a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence are obtained.

[0106] In this step, a color image frame sequence of laparoscopic surgery is obtained. And its corresponding depth map sequence ,in t=0,1,2,… Represents a time series. Exemplarily, the depth map here can be obtained using a depth estimation algorithm such as MIOL.

[0107] In step S2, each frame of color image and its depth map are restored into a one-to-one corresponding color point cloud using a pinhole camera model.

[0108] This step can be simply described as:

[0109]

[0110] in, x is the restored point cloud 3D coordinate, d uv Depth map At coordinates ( u , v ), K is the camera intrinsic parameter matrix, c for x The color of the dots, C uv Represents a color map ( u , v ) coordinates, traversing the color image frame All ( u , v )coordinate.

[0111] Finally, the obtained x Collection and c The collection together constitutes the colored point cloud.

[0112] In step S3, a local three-dimensional model corresponding to each of the color point clouds is constructed using the improved digital elevation model.

[0113] This step converts the restored color point cloud into a digital elevation model (DEM), that is, the improved digital elevation model DEM is used to represent the local three-dimensional model corresponding to each color point cloud. S t .

[0114] It should be noted that an improved digital elevation model consists of two parts: 1. Initial coordinates O = (x0, y0, 0). 2. Grid set ,in i and j Respectively represent the row and column index of the grid. The grid has a uniform grid size B , each grid has a single or multiple average points, each of which carries its position information and color information .

[0115] Specifically, this step includes:

[0116] S31. Distribute the colored point cloud in the grid divided by the XOY plane

[0117] When calculating the XOY plane size of each color point cloud, according to the uniform grid size B The XOY plane is divided into grids, so that each grid of the grid set contains a number of points of the color point cloud, and each point carries its position information and color information.

[0118] S32. Determine the average number of points that should be retained in each grid

[0119] Next, we use the information entropy index function to calculate the information entropy index function value of each grid:

[0120]

[0121] in, E ij express The information entropy index function value is N Indicates the number of points contained in the grid, α represents the weight, Indicates the operation of extracting the z component of a three-dimensional vector;

[0122] Indicates the operation of converting RGB color to grayscale value;

[0123] p k represents the kth point in the colored point cloud contained in the grid, c k It represents the RGB value of the point.

[0124] If the value of this indicator function is less than the first preset threshold, it is considered that the corresponding grid retains only one average point, which is called a single-point grid; otherwise, multiple average points are retained, which is called a multi-point grid.

[0125] S33. Calculate the position and color of the average point

[0126] (1) For each of the single-point grids

[0127] According to the position information of each point in the grid, combined with the initial coordinates of the improved digital elevation model, the position information of the retained single average point is calculated; the calculation formula is as follows:

[0128]

[0129] And according to the color information of each point in the grid, the color information of a single average point is obtained by averaging; the calculation formula is as follows:

[0130]

[0131] (2) For each of the multi-point grids

[0132] The ISODATA clustering algorithm is used to automatically generate the location information of multiple retained average points.

[0133] And based on the color information of each group of aggregated points, the color information of multiple retained average points is obtained by averaging.

[0134] At this point, the local three-dimensional model corresponding to each of the color point clouds is obtained.

[0135] In step S4, a deformation field consisting of a series of deformation nodes is obtained; including:

[0136] It should be noted that the embodiment of the present invention assumes that the deformation field here is composed of a series of deformation nodes, each of which has its three-dimensional position , and both carry a deformation vector .

[0137] Accordingly, this step includes:

[0138] S41, reconstructing the current local three-dimensional model, and using the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; specifically including:

[0139] It should be noted that, in order to simplify the calculation, a reconstructed DEM model with only a single-point grid is reconstructed here to meet the needs of deformation field sampling, that is, referring to the processing method of the single-point grid, the original multi-point grids of the current local 3D model are converted into single-point grids one by one, and the reconstructed model is obtained by combining the original single-point grids;

[0140] The VIP index function is used to calculate the importance index value of each grid of the reconstructed model. If it is greater than the second preset threshold, the grid is considered to be of high importance and is called an important grid. The calculation formula is as follows:

[0141]

[0142] in, Sym Represented by a grid is a set of 4 pairs of centrally symmetric grids, such as Figure 3 As shown, Sym = The grids with letter subscripts correspond to the grids indicated by the corresponding letters in the representation diagrams. and Indicates the elevation value of the average point in the grid x and grid y. Is a grid The elevation value of . and Represents the RGB color value of the average point of grid x and grid y, β Represents a weight value.

[0143] Since the deformation nodes are sampled from the average points in the grid, the important grids of the reconstructed model are sampled first, and the position information of the single average point in each of the important grids is used as the three-dimensional position of the corresponding deformation node of the current deformation field; then the non-important grids are traversed, and the Euclidean distance between the single average point in the current non-important grid and the sampled grid is calculated. If it is greater than the third preset threshold, the position of the single average point in the non-important grid is used as the three-dimensional position of the corresponding deformation node of the current deformation field.

[0144] S42, taking minimization of total energy as the objective function, optimizing and solving the deformation vector of each deformation node of the current deformation field.

[0145] After constructing the DEM and deformation field of the two frames, the following objective function is introduced to optimize the calculation of the deformation vector in the deformation field so that the two DEM frames can be registered and fused:

[0146]

[0147] in, Represents the current deformation field Minimize the total energy under t Indicates the frame label; λ represents the weight of the motion consistency regularization term; γ represents the loss of TPS energy term;

[0148] E data Represents the data energy term of the deformation vector supervised by optical flow information; s i and t i Respectively represent the 3D coordinates of a pair of matching points on the point cloud generated by the previous color image frame and the current image frame, N m is the logarithm of three-dimensional matching points; Indicates the three-dimensional matching point s i The predicted deformation vector at is obtained by weighted average of the deformation vectors of the deformation nodes in the neighborhood. The calculation method is expressed as , represents the neighborhood, w ik represents the corresponding weight, , They represent the first k The three-dimensional position and deformation vector of each deformation node;

[0149] E MCT The regularization term representing the consistency of motion is used to ensure that the motion of adjacent deformation nodes maintains a similar trend; N w Indicates the number of deformation nodes; Represents the current deformation field i The deformation vector of the deformation node, , Respectively represent its adjacent deformation nodes j The three-dimensional position and deformation vector of w ij Indicates the corresponding weight;

[0150] E TPS The TPS energy term represents the change in the surface supervision elevation interpolated by TPS; N a Represents the last fused 3D model S t-1 The number of all average points in ; Represents taking out the three-dimensional vector z Component operations; express S t -1 No. i The elevation value of the average point; Indicates the latest elevation value of the point predicted by the TPS algorithm.

[0151] Then simplify the objective function into matrix form to facilitate accelerated calculation:

[0152] In order to speed up this optimization process, we use Represents the deformation vectors carried by all deformation nodes, and obtains the following matrix forms respectively:

[0153]

[0154] Among them, the superscript T represents transpose, represents the set of real numbers;

[0155] Represents the first sparse matrix, the first element F ij = ;

[0156] , represents the three-dimensional matching point s i A collection of;

[0157] , represents the three-dimensional matching point t i A collection of;

[0158] represents the identity matrix;

[0159] Represents the second sparse matrix, the second element Y ij = ;

[0160] Represents the third sparse matrix, the third element M ij = , express S t-1 No. i The three-dimensional position of the average point;

[0161] , represents the three-dimensional position A collection of;

[0162] , represents the three-dimensional position predicted by the TPS algorithm A collection of; , and Respectively represent the extraction of three-dimensional vectors x and y Quantity operation.

[0163] At this point, the gradient of the objective function is obtained:

[0164]

[0165] The CUBLAS library is used to optimize and accelerate the solution of the objective function, and finally the deformation vector of each deformation node of the current deformation field is obtained.

[0166] In step S5, the last fused three-dimensional model is warped using the current deformation field to obtain a warped three-dimensional model.

[0167] This step uses the deformation field information to update the average point information (i.e., warp) and The warping process is represented by:

[0168] The warped 3D model The initial coordinates of O w Set as current local 3D model The initial coordinates of ;

[0169] For the previous fused 3D model S t-1 All single-point grids, update the row index of the grid i , column index j And the elevation value ; The update formula is as follows:

[0170]

[0171] in,( i w , j w , e w ) represents the index and elevation value of the average point in the updated single-point network; v t Indicates the change in the grid index of the average point; Indicates p The predicted deformation vector at a single average point within a single-point network; O t-1 express S t-1 The initial coordinates of O t-1 , B Indicates a uniform grid size;

[0172] For the previous fused 3D model S t-1 All multi-point grids, apply the current deformation field to all average points in the grid; the update formula is as follows:

[0173]

[0174] in, Indicates i The average point coordinates in the deformed multi-point grid, It is i The average point coordinates in the undeformed multi-point grid, It is in i The predicted deformation vectors at the deformation points in the multi-point grid;

[0175] Finally, the warped 3D model is obtained. .

[0176] In step S6, the warped three-dimensional model and the current local three-dimensional model are fused to obtain a current fused three-dimensional model.

[0177] The warped 3D model in this step is weightedly fused with the current local 3D model, and the Represents the fusion process; specifically includes:

[0178] Calculate the current local 3D model 3D model after warping The fusion weight of each pair of coincident grids is as follows:

[0179]

[0180] in, express Upper Grid The fusion weight of Indicates the number of average points in the grid; represents the VIP indicator function;

[0181] Different fusion strategies are adopted according to different situations:

[0182] (1) If and ,in express On the grid The fusion weights of overlapping meshes, represents the average number of points in the grid, ℇ To express the hyperparameters of the difference in geometry and color information between the two grids, the fusion operation is performed only on the elevation values:

[0183]

[0184] Among them, elocal means The elevation value of the average point that needs to be merged;

[0185] (2) If and , no fusion operation is performed, only The average point information carried by the grid;

[0186] (3) Otherwise, use the ISODATA clustering algorithm to generate a new average point;

[0187] After the fusion is completed, the latest fused DEM model will be obtained S t .

[0188] Thus, the embodiment of the present invention has designed a set of real-time tissue deformation estimation algorithms, which can construct a complete three-dimensional model by splicing and fusing point cloud information at different times in a short time. In addition, the constructed three-dimensional model expression based on the digital elevation model greatly reduces the amount of model calculation.

[0189] Embodiment 2:

[0190] The embodiment of the present invention provides a flexible and rapid three-dimensional reconstruction system for an in vivo body based on a digital elevation model, comprising:

[0191] An acquisition module, used for acquiring a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence;

[0192] A restoration module is used to restore each frame of color image and its depth map into a one-to-one corresponding color point cloud using a pinhole camera model;

[0193] A construction module, used to construct a local three-dimensional model corresponding to each of the color point clouds using an improved digital elevation model;

[0194] The acquisition module is used to obtain the deformation field composed of a series of deformation nodes; including:

[0195] Reconstruct the current local three-dimensional model, and use the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; minimize the total energy as the objective function, and optimize the deformation vector of each deformation node of the current deformation field;

[0196] A warping module, used to warp the last fused three-dimensional model using the current deformation field to obtain a warped three-dimensional model;

[0197] The fusion module is used to fuse the warped three-dimensional model with the current local three-dimensional model to obtain the current fused three-dimensional model.

[0198] Embodiment 3:

[0199] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When a processor executes the computer program, the in vivo flexible rapid three-dimensional reconstruction method based on a digital elevation model as described in Example 1 can be implemented.

[0200] Embodiment 4:

[0201] An embodiment of the present invention provides an electronic device, including:

[0202] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the in vivo flexible rapid three-dimensional reconstruction method based on a digital elevation model as described in Example 1.

[0203] It can be understood that the in vivo flexible rapid 3D reconstruction system based on digital elevation model, storage medium and electronic device provided in the embodiments of the present invention correspond to the in vivo flexible rapid 3D reconstruction method based on digital elevation model provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the in vivo flexible rapid 3D reconstruction method and will not be repeated here.

[0204] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0205] 1. The embodiments of the present invention can generate a three-dimensional model of a three-dimensional scene in the body in real time, helping doctors to observe fine tissue structures during surgery; and can splice time-sequential color image frames to form a relatively complete three-dimensional scene, thereby expanding the surgical field of view.

[0206] 2. The embodiment of the present invention designs a set of real-time tissue deformation estimation algorithms, which can splice and fuse point cloud information at different times to build a complete three-dimensional model in a short time. In addition, the constructed three-dimensional model expression based on the digital elevation model greatly reduces the amount of model calculation.

[0207] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0208] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A flexible and rapid three-dimensional reconstruction method for an in vivo body based on a digital elevation model, characterized in that: include: Acquire a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence; Using the pinhole camera model, each frame of color image and its depth map are restored into a one-to-one corresponding color point cloud; Using the improved digital elevation model, constructing a local three-dimensional model corresponding to each of the color point clouds; Gets a deformation field consisting of a series of deformation nodes; including: Reconstruct the current local three-dimensional model, and use the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; minimize the total energy as the objective function, and optimize the deformation vector of each deformation node of the current deformation field; Using the current deformation field to warp the last fused three-dimensional model, to obtain a warped three-dimensional model; Fusing the warped three-dimensional model and the current local three-dimensional model to obtain a current fused three-dimensional model; The improved digital elevation model consists of two parts: initial coordinates and a grid set; the improved digital elevation model is used to construct a local three-dimensional model corresponding to each color point cloud; including: Calculating the XOY plane size of each of the color point clouds, dividing the XOY plane into grids according to a uniform grid size, so that each grid of the grid set contains a number of points of the color point cloud, and each point carries its position information and color information; The information entropy index function is used to calculate the information entropy index function value of each grid. If it is less than the first preset threshold, it is considered that the corresponding grid only retains one average point, which is called a single-point grid; otherwise, multiple average points are retained, which is called a multi-point grid. For each of the single-point grids, the position information of the single average point is calculated based on the position information of each point in the grid and the initial coordinates of the improved digital elevation model; and the color information of the single average point is averaged based on the color information of each point in the grid; For each of the multi-point grids, the ISODATA clustering algorithm is used to automatically generate the position information of the multiple average points retained; and based on the color information of each group of aggregated points, the color information of the multiple average points retained is obtained by averaging respectively; Obtaining a local three-dimensional model corresponding to each of the color point clouds; The reconstructing the current local three-dimensional model and using the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field comprises: Referring to the processing method of single-point grids, the original multi-point grids of the current local 3D model are converted into single-point grids one by one, and the reconstructed model is obtained by combining the original single-point grids; Using the VIP index function, the importance index value of each grid of the reconstructed model is calculated. If the importance index value is greater than a second preset threshold, the grid is considered to be of high importance and is called an important grid. Prioritizing sampling of important grids of the reconstructed model, and using position information of a single average point in each of the important grids as the three-dimensional position of a corresponding deformation node of the current deformation field; Then, the non-important grids are traversed to calculate the Euclidean distance between a single average point in the current non-important grid and the sampled grid. If the distance is greater than the third preset threshold, the position of the single average point in the non-important grid is used as the three-dimensional position of the corresponding deformation node of the current deformation field.

2. The in vivo flexible rapid three-dimensional reconstruction method according to claim 1, characterized in that: The objective function is expressed as: in, Represents the current deformation field Minimize the total energy under t Indicates the frame label; λ represents the weight of the motion consistency regularization term; γ represents the loss of TPS energy term; E data Represents the data energy term of the deformation vector supervised by optical flow information; s i and t i Respectively represent the 3D coordinates of a pair of matching points on the point cloud generated by the previous color image frame and the current image frame, N m is the logarithm of three-dimensional matching points; Indicates the three-dimensional matching point s i The predicted deformation vector at is obtained by weighted average of the deformation vectors of the deformation nodes in the neighborhood. The calculation method is expressed as , represents the neighborhood, w ik represents the corresponding weight, , They represent the first k The three-dimensional position and deformation vector of each deformation node; E MCT The regularization term representing the consistency of motion is used to ensure that the motion of adjacent deformation nodes maintains a similar trend; N w Indicates the number of deformation nodes; Represents the current deformation field i The deformation vector of the deformation node, , Respectively represent its adjacent deformation nodes j The three-dimensional position and deformation vector of w ij Indicates the corresponding weight; E TPS The TPS energy term represents the change in the surface supervision elevation interpolated by TPS; N a Represents the last fused 3D model S t-1 The number of all average points in ; Represents taking out the three-dimensional vector z Component operations; express S t-1 No. i The elevation value of the average point; Indicates the latest elevation value of the point predicted by the TPS algorithm.

3. The in vivo flexible rapid three-dimensional reconstruction method according to claim 2, characterized in that: The optimization process of solving the deformation vector of each deformation node of the current deformation field comprises: use Represents the deformation vectors carried by all deformation nodes, and obtains the following matrix forms respectively: Among them, the superscript T represents transpose, represents the set of real numbers; Represents the first sparse matrix, the first element F ij = ; , represents the three-dimensional matching point s i A collection of; , represents the three-dimensional matching point t i A collection of; represents the identity matrix; Represents the second sparse matrix, the second element Y ij = ; Represents the third sparse matrix, the third element M ij = , express S t-1 No. i The three-dimensional position of the average point; , represents the three-dimensional position A collection of; , represents the three-dimensional position predicted by the TPS algorithm A collection of; , and Respectively represent the extraction of three-dimensional vectors x and y Operation of weight; At this point, the gradient of the objective function is obtained: The CUBLAS library is used to optimize and accelerate the solution of the objective function, and finally the deformation vector of each deformation node of the current deformation field is obtained.

4. The in vivo flexible rapid three-dimensional reconstruction method according to claim 3, characterized in that: use Indicates the warping process; Specifically include: The warped 3D model The initial coordinates of O w Set as current local 3D model The initial coordinates of ; For the previous fused 3D model S t-1 All single-point grids, update the row index of the grid i , column index j And the elevation value ; The update formula is as follows: in,( i w , j w , e w ) represents the index and elevation value of the average point in the updated single-point network; v t Indicates the change in the grid index of the average point; Indicates p The predicted deformation vector at a single average point within a single-point network; O t-1 express S t-1 The initial coordinates of O t-1 , B Indicates a uniform grid size; For the previous fused 3D model S t-1 All multi-point grids, apply the current deformation field to all average points in the grid; the update formula is as follows: in, Indicates i The average point coordinates in the deformed multi-point grid, It is i The average point coordinates in the undeformed multi-point grid, It is in i The predicted deformation vectors at the deformation points in the multi-point grid; Finally, the warped 3D model is obtained. .

5. The in vivo flexible rapid three-dimensional reconstruction method according to claim 4, characterized in that: use Represents the fusion process; specifically includes: Calculate the current local 3D model 3D model after warping The fusion weight of each pair of coincident grids is as follows: in, express Upper Grid The fusion weight of Indicates the number of average points in the grid; represents the VIP indicator function; Different fusion strategies are adopted according to different situations: (1) If and ,in express On the grid The fusion weights of overlapping meshes, represents the average number of points in the grid, ℇ To express the hyperparameters of the difference in geometry and color information between the two grids, the fusion operation is performed only on the elevation values: in, e local express The elevation value of the average point that needs to be merged; (2) If and , no fusion operation is performed, only The average point information carried by the grid; (3) Otherwise, use the ISODATA clustering algorithm to generate a new average point; After the fusion is completed, the latest fused DEM model will be obtained S t .

6. A flexible and rapid in vivo 3D reconstruction system based on a digital elevation model, used to execute the flexible and rapid in vivo 3D reconstruction method according to claim 1, characterized in that: include: An acquisition module, used for acquiring a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence; A restoration module is used to restore each frame of color image and its depth map into a one-to-one corresponding color point cloud using a pinhole camera model; A construction module, used to construct a local three-dimensional model corresponding to each of the color point clouds using an improved digital elevation model; The acquisition module is used to obtain the deformation field composed of a series of deformation nodes; including: Reconstruct the current local three-dimensional model, and use the importance of each grid of the reconstructed model to sample the three-dimensional position of each deformation node of the current deformation field; minimize the total energy as the objective function, and optimize the deformation vector of each deformation node of the current deformation field; A warping module, used to warp the last fused three-dimensional model using the current deformation field to obtain a warped three-dimensional model; The fusion module is used to fuse the warped three-dimensional model with the current local three-dimensional model to obtain the current fused three-dimensional model.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when a processor executes the computer program, the in vivo flexible rapid three-dimensional reconstruction method based on a digital elevation model as described in any one of claims 1 to 5 can be implemented.

8. An electronic device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the in vivo flexible rapid three-dimensional reconstruction method based on a digital elevation model as described in any one of claims 1 to 5.

Images