Three-dimensional Mesh Model Registration and Fusion System for Laparoscopic Surgery Navigation

Abstract

The present invention provides a three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation, which relates to the technical field of laparoscopic surgery navigation. The mesh data introduced by the three-dimensional mesh model registration and fusion system provided by the present invention takes into account the connection relationship between the vertices of the model compared with the traditional point cloud, improving the accuracy of subsequent multi-modal fusion; and can present the internal information of the tissue to the doctor, assisting the doctor to make clinical decisions, reducing the surgical risk and improving the surgical efficiency at the same time. Among them, the system generates training data through manual annotation and interpolation based on real data, trains a multi-modal registration and fusion network in a supervised manner, and finally further improves the registration accuracy through unsupervised fine-tuning. In addition, the introduced online self-supervised learning depth estimation method based on a binocular endoscope, the binocular depth estimation network used by which has the ability of fast overlearning, can continuously adapt to new scenarios by using self-supervised information, and thus ensure the accuracy of the intraoperative tissue mesh model.

Description

Technical Field

[0001] The present invention relates to the technical field of laparoscopic surgery navigation, and particularly relates to a three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation. Background Art

[0002] Compared with traditional open surgery, minimally invasive surgery (such as endoscopic surgery) has the advantages of small incisions, less bleeding, and fast recovery speed, and is gradually widely adopted. However, due to the limitation of the endoscopic field of view, the endoscope can only see a relatively narrow surgical area each time, and the apparent features of the internal tissues are extremely similar. Most of the key anatomical structures and some diseased parts are buried inside the tissues and cannot be judged by apparent recognition and analysis. It is necessary to combine preoperative inspection image information such as CT and MRI for judgment. If the preoperative and intraoperative tissue mesh models of the tissue are unified in the same coordinate system and presented to the physician, it can facilitate the estimation of the lesion location and internal structure, and reduce the operation time and risk.

[0003] Currently, most of the surgical navigation systems based on multi-modal fusion focus on surgeries with less non-rigid deformation of tissues such as orthopedics and dentistry, and their applications in technologies with more non-rigid deformation of tissues such as the abdominal cavity are still limited. For example, Chinese Patent CN201910253168.6 discloses an augmented reality surgical navigation method, system and electronic device based on video superposition. The method includes: obtaining a preoperative three-dimensional anatomical model of a target tissue organ; constructing point cloud data on the surface of the target tissue organ according to the laparoscopic video information collected by an intraoperative stereoscopic laparoscope; registering the preoperative three-dimensional model and the point cloud data, and obtaining the target transformation relationship between the preoperative three-dimensional anatomical model and the point cloud data corresponding to the current laparoscopic view according to the registration result; and superimposing and displaying the preoperative three-dimensional anatomical model on the laparoscopic view through augmented reality technology according to the target transformation relationship.

[0004] However, when representing the internal tissue model in the above solution, only the point cloud data type is used, which limits the accuracy of multi-modal fusion. Summary of the Invention

[0005] (1) Technical Problems to be Solved

[0006] In view of the deficiencies of the prior art, the present invention provides a three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation, which solves the technical problem that the accuracy of multi-modal fusion needs to be improved.

[0007] (2) Technical Solutions

[0008] To achieve the above object, the present invention is realized through the following technical solutions:

[0009] A three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation includes:

[0010] The first modeling module is used to obtain a preoperative tissue mesh model with tissue semantic information;

[0011] The second modeling module is used to obtain an intraoperative tissue mesh model according to the depth value of the specified binocular endoscope image frame;

[0012] The feature extraction module is used to respectively obtain corresponding multi-level features according to the preoperative tissue mesh model and the intraoperative tissue mesh model;

[0013] The overlap prediction module is used to obtain the overlapping area of the preoperative tissue mesh model and the intraoperative tissue mesh model according to the multi-level features, and obtain the pose transformation relationship of the vertices of the preoperative tissue mesh model within the overlapping area;

[0014] The global fusion module is used to obtain the coordinates of all vertices after registration of the preoperative tissue mesh model according to the coordinates and pose transformation relationship of the vertices of the preoperative tissue mesh model within the overlapping area, and the coordinates of the vertices within the non-overlapping area;

[0015] The information display module is used to display the internal tissue information of the preoperative tissue mesh model in the intraoperative tissue mesh model according to the coordinates of all vertices after registration of the preoperative tissue mesh model.

[0016] Preferably, the feature extraction module uses Chebyshev spectral graph convolution to extract the multi-level features of the preoperative tissue mesh model and the intraoperative tissue mesh model:

[0017]

[0018]

[0019] Among them, the preoperative tissue mesh model M is defined as pre =(V pre , E pre ), where V pre represents the spatial coordinates of the vertices of the preoperative tissue mesh model, and E pre represents the edges between the vertices of the preoperative tissue mesh model; the intraoperative tissue mesh model M in =(V in , E in ), where V in represents the spatial coordinates of the vertices of the preoperative tissue mesh model, and E in represents the edges between the vertices of the intraoperative tissue mesh model;

[0020] and respectively represent the downsampling scale features of the n+1th layer and the nth layer of the preoperative tissue model, and initialize as Vpre ; and represent the features of the (n + 1)-th layer and the n-th layer of the intraoperative tissue model respectively, and initialize as V in ;

[0021] are the b-th order Chebyshev polynomials calculated from their respective vertices and their B-ring neighborhoods respectively, are the scaled Laplacian matrices calculated from the edges E in , E pre respectively, are the learning parameters of the neural network.

[0022] Preferably, the overlap prediction module is specifically configured to:

[0023] Obtain the overlapping region of the preoperative tissue mesh model and the intraoperative tissue mesh model by using an attention mechanism, including:

[0024]

[0025]

[0026] wherein, O pre represents the mask of the overlapping region of the preoperative tissue mesh model M pre ; O in represents the mask of the overlapping region of the intraoperative tissue mesh model M in ; cross and self represent self-attention and cross-attention operations respectively; and represent the m-th level downsampling scale features of the vertices of the preoperative tissue mesh model and the intraoperative tissue mesh model respectively;

[0027] According to the masks O pre and O in , obtain the vertices and their features that are respectively within the overlapping region, and use a multi-layer perceptron MLP to calculate the corresponding points of the vertices pre in the preoperative tissue mesh model M :

[0028]

[0029] wherein, is the vertex in the intraoperative tissue mesh model M in , corresponding to the vertex pre in the preoperative tissue mesh model M represents the cosine similarity calculation, Indicates performing a position encoding operation on the vertices of the intraoperative tissue mesh model within the overlapping region;

[0030] Use the k-nearest neighbor search (KNN) to establish the local neighborhood of the vertex and use the singular value decomposition (SVD) to solve for the rotation matrix. The formula is as follows:

[0031]

[0032] where, represents the rotation matrix of vertex ; represents using the KNN algorithm to construct a local neighborhood belonging to vertex ; is the neighborhood point of vertex of the preoperative tissue mesh model, is the vertex of the intraoperative tissue mesh model corresponding to the neighborhood point ;

[0033] Use the rotation matrix to change the point cloud coordinates to obtain Use an MLP to predict the displacement vector of vertex . The formula is as follows:

[0034]

[0035] where, represents the displacement vector of the vertex of the preoperative tissue mesh model in the overlapping region, and together with the rotation matrix constitutes the pose transformation relationship.

[0036] Preferably, the global fusion module is specifically configured to:

[0037] Use an MLP to regress the rotation matrix and displacement vector of all vertices of the preoperative tissue mesh model:

[0038]

[0039] where R pre , t pre respectively represent the rotation matrix and displacement vector of all vertices of the preoperative tissue mesh model; represents the weight calculated based on the distance between the vertex in the overlapping region and all vertices v pre of the preoperative tissue mesh model;

[0040]

[0041] where, represents the coordinates of all vertices of the preoperative tissue mesh model after registration.

[0042] Preferably, in the training stage of the three-dimensional mesh model registration and fusion system, a training set is generated based on real data:

[0043] According to the feature point pairs between the specified binocular endoscope image frames and the preoperative tissue mesh model, the preoperative tissue mesh model and the intraoperative tissue mesh model are registered using a non-rigid algorithm based on the feature points. For any feature point:

[0044]

[0045] where Non_rigid_ICP represents the non-rigid registration algorithm ICP, represents the a-th feature point for non-rigid registration of the preoperative tissue mesh model, corresponds to the feature point of the intraoperative tissue mesh model, T G is the overall transformation matrix of the preoperative tissue mesh model, T l,a is the local deformation transformation matrix belonging to the feature point v pre,a ;

[0046] The local deformation transformation matrix T of all vertices in the preoperative tissue mesh model is obtained by quaternion interpolation l , and the registered coordinate label of the vertex v in the preoperative tissue mesh model is obtained through the transformation relationship pre ;

[0047] Preferably, in the training stage of the three-dimensional mesh model registration and fusion system, the following supervised loss function is constructed:

[0048]

[0049] where Loss s represents the supervised loss function in the training stage;

[0050] β s and γ s represent the supervised loss term coefficients respectively;

[0051] N1 represents the number of vertices of the preoperative tissue mesh model M pre ;

[0052] represents the l2 true value loss based on the manually annotated data set, represents the coordinates of all vertices after registration of the preoperative tissue mesh model;

[0053] I c +II c +III cDenotes the Cauchy-Green invariant, which is used to constrain the degree of deformation of the internal tissue, I c Constrains the length of the arc distance between two points on the surface to remain unchanged, II c Constrains the surface area of the tissue to remain unchanged, III c Constrains the volume of the tissue to remain unchanged.

[0054] Preferably, the three-dimensional mesh model registration and fusion system further includes:

[0055] An accuracy fine-tuning module, which is used to introduce an unsupervised loss to fine-tune the network, and assist the global fusion module to obtain all vertex coordinates after the preoperative tissue mesh model is registered.

[0056] Preferably, during the application of the unsupervised loss fine-tuning network, the following unsupervised loss function is constructed:

[0057]

[0058] Among them, Loss u Denotes the unsupervised loss function;

[0059] β u ,γ u Respectively denote the coefficients of the unsupervised loss terms, and Are both the vertex coordinates after the preoperative tissue mesh model is registered during unsupervised training, Denotes the vertex in the intraoperative tissue mesh model that is the closest to the vertex of the preoperative tissue mesh model after registration of, Denotes the vertex and 's Euclidean distance, Denotes the vertex in the registered preoperative tissue mesh model that is the closest to the vertex v of the intraoperative tissue mesh model in,b of, Denotes the vertex v in,b and the vertex 's Euclidean distance;

[0060] N1 represents the number of vertices of the preoperative tissue mesh model M pre , N2 represents the number of vertices of the intraoperative tissue mesh model M in ;

[0061] Denotes the Cauchy-Green invariant during unsupervised training, Constrains the length of the arc distance between two points on the surface to remain unchanged, Constrains the surface area of the tissue to remain unchanged, Constrains the volume of the tissue to remain unchanged.

[0062] Preferably, the second modeling module uses an online self-supervised learning depth estimation method based on a binocular endoscope to obtain the depth value of the specified binocular endoscope image frame; the binocular depth estimation network used in the online self-supervised learning depth estimation method has the ability of fast overlearning and can continuously adapt to new scenarios using self-supervised information;

[0063] In the real-time reconstruction mode, the second modeling module is specifically used for overfitting continuous video frames of images to obtain the depth value of the specified binocular endoscope image frame, including:

[0064] An extraction unit, configured to obtain binocular endoscope images and extract multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network;

[0065] A fusion unit, configured to fuse the multi-scale features using the decoder network of the current binocular depth estimation network to obtain the disparity of each pixel point in the current frame image;

[0066] A conversion unit, configured to convert the disparity into depth according to the internal and external parameters of the camera and output it as the result of the current frame image;

[0067] A first estimation unit, configured to update the parameters of the current binocular depth estimation network using self-supervised loss without introducing external ground truth for depth estimation of the next frame image.

[0068] Preferably, in the precise measurement mode, the second modeling module is specifically used for overfitting key video frames of images, including:

[0069] A second estimation unit, without introducing external ground truth, according to the binocular depth estimation network obtained in the real-time reconstruction mode for the previous frame image of the specified binocular endoscope image frame, updates the parameters of the aforementioned binocular depth estimation network using the self-supervised loss corresponding to the specified binocular endoscope image frame until convergence, and uses the converged binocular depth estimation network for precise depth estimation of the specified binocular endoscope image frame to obtain the depth value of the specified binocular endoscope image frame.

[0070] (III) Beneficial effects

[0071] The present invention provides a three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation. Compared with the prior art, it has the following beneficial effects:

[0072] In the present invention, a preoperative tissue mesh model and an intraoperative tissue mesh model are obtained. The mesh data takes into account the connection relationship between the vertices of the model compared with the traditional point cloud, improving the accuracy of subsequent multi-modal fusion. According to the preoperative and intraoperative tissue mesh models, corresponding multi-level features are obtained respectively. According to the multi-level features, the overlapping area of the preoperative and intraoperative tissue mesh models is obtained, and the pose transformation relationship of the vertices of the preoperative tissue mesh model within the overlapping area is obtained. According to the coordinates and pose transformation relationship of the vertices of the preoperative tissue mesh model within the overlapping area and the coordinates of the vertices within the non-overlapping area, all the vertex coordinates of the preoperative tissue mesh model after registration are obtained. And the internal information of the tissue is presented to the doctor to assist the doctor in making clinical decisions, reducing the surgical risk and improving the surgical efficiency at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0074] Figure 1 It is a framework diagram of a three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation provided by an embodiment of the present invention;

[0075] Figure 2 It is another framework diagram of a three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation provided by an embodiment of the present invention;

[0076] Figure 3 It is a schematic diagram of the technical framework of an online self-supervised learning depth estimation method based on a binocular endoscope provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0077] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention. [[ID=2】4]]

[0078] By providing a three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation in the embodiments of the present application, the technical problem that the accuracy of multi-modal fusion needs to be improved is solved.

[0079] The overall idea of the technical solutions in the embodiments of the present application to solve the above technical problems is as follows:

[0080] Embodiments of the present invention are mainly applied to, but not limited to, surgical endoscope scenarios such as laparoscopic surgery scenarios. In the field of view of endoscopic surgery, doctors can only see the surface of tissues, and information such as the positions of blood vessels and lesion areas inside the tissues depends on the doctor's experience and judgment. The pre-operative reconstruction model of CT / MRI contains information about blood vessels and lesion areas inside the tissues. The non-rigid registration and fusion algorithm can register the pre-operative tissue mesh model into the intra-operative tissue mesh model, and present the internal tissue information to doctors with the help of conventional display technologies, assisting doctors in making clinical decisions, reducing surgical risks and improving surgical efficiency at the same time.

[0081] Specifically, in the embodiments of the present invention, a pre-operative tissue mesh model and an intra-operative tissue mesh model are obtained. The mesh data takes into account the connection relationship between the vertices of the model compared with traditional point clouds, improving the accuracy of subsequent multi-modal fusion. According to the pre-operative and intra-operative tissue mesh models, corresponding multi-level features are respectively obtained. According to the multi-level features, the overlapping area of the pre-operative and intra-operative tissue mesh models is obtained, and the pose transformation relationship of the vertices of the pre-operative tissue mesh model in the overlapping area is obtained. According to the coordinates and pose transformation relationship of the vertices of the pre-operative tissue mesh model in the overlapping area, and the coordinates of the vertices in the non-overlapping area; according to the coordinates of all vertices of the pre-operative tissue mesh model after registration, the internal tissue information of the pre-operative tissue mesh model is displayed in the intra-operative tissue mesh model.

[0082] In addition, the intra-operative tissue mesh model can be obtained according to the depth value of the specified binocular endoscope image frame. Specifically, an online self-supervised learning depth estimation method based on a binocular endoscope can be used to obtain the depth value of the specified binocular endoscope image frame; the binocular depth estimation network used in the online self-supervised learning depth estimation method has the ability of fast overlearning and can continuously adapt to new scenarios using self-supervised information. And the online self-supervised learning depth estimation method also provides two modes, namely a real-time reconstruction mode and a precise measurement mode, for determining the depth value of the specified binocular endoscope image frame.

[0083] The depth estimation with dual-mode switching can not only provide a real-time point cloud of the intra-operative anatomical structure to assist doctors in intuitively understanding the intra-operative three-dimensional structure, but also achieve high-precision reconstruction of the specified binocular endoscope image frame specified by doctors based on single-frame overfitting, providing a basis for subsequent processing, so as to balance speed and accuracy in applications.

[0084] To better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the accompanying drawings of the specification and specific embodiments.

[0085] Embodiment:

[0086] As Figure 1 shown, embodiments of the present invention provide a three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation, including:

[0087] The first modeling module is used to obtain a preoperative tissue mesh model with tissue semantic information;

[0088] The second modeling module is used to obtain an intraoperative tissue mesh model according to the depth value of a specified binocular endoscope image frame;

[0089] The feature extraction module is used to respectively obtain corresponding multi-level features according to the preoperative tissue mesh model and the intraoperative tissue mesh model;

[0090] The overlap prediction module is used to obtain the overlapping area between the preoperative tissue mesh model and the intraoperative tissue mesh model according to the multi-level features, and obtain the pose transformation relationship of the vertices of the preoperative tissue mesh model within the overlapping area;

[0091] The global fusion module is used to obtain the coordinates of all vertices after registration of the preoperative tissue mesh model according to the coordinates and pose transformation relationship of the vertices of the preoperative tissue mesh model within the overlapping area, and the coordinates of the vertices within the non-overlapping area;

[0092] The information display module is used to display the internal tissue information of the preoperative tissue mesh model in the intraoperative tissue mesh model according to the coordinates of all vertices after registration of the preoperative tissue mesh model.

[0093] The mesh data introduced by the above three-dimensional mesh model registration and fusion system considers the connection relationship between the vertices of the model compared with the traditional point cloud, improving the accuracy of subsequent multi-modal fusion; and can present the internal tissue information to the doctor, assisting the doctor to make clinical decisions, reducing the surgical risk while improving the surgical efficiency.

[0094] Next, each component module of the above technical solution will be introduced in detail:

[0095] For the first modeling module, it is used to obtain a preoperative tissue mesh model with tissue semantic information.

[0096] Exemplarily, this module uses software such as 3D Slicer to reconstruct CT / MRI tissues to obtain a three-dimensional mesh model, and then uses deep learning algorithms such as DeepLab or manual segmentation to divide tissues such as blood vessels and the liver, and finally forms a preoperative tissue mesh model M pre =(V pre , E pre ), where V pre represents the vertex spatial coordinates of the model, and E pre represents the edges between vertices.

[0097] For the second modeling module, it is used to obtain an intraoperative tissue mesh model according to the depth value of a specified binocular endoscope image frame.

[0098] Exemplarily, this module uses an online self-supervised learning depth estimation based on a binocular endoscope (see the following text for details) to estimate the depth value D of a pixel point; and calculates the spatial coordinates of the pixel point in the camera coordinate system through a pinhole camera model, and its formula is

[0099]

[0100]

[0101] z = D

[0102] where D is the depth estimation value of the pixel point; x, y, and z respectively represent the x coordinate, y coordinate, and z coordinate in the camera coordinate system;

[0103] c x ,c y ,f x ,f y are the corresponding parameters in the left or right endoscope of the binocular endoscope and the camera intrinsic matrix , and convert the picture into a point cloud V in ={v in,a |=1,2,…N1}, where v in,a represents the spatial coordinates of the a-th pixel point;

[0104] Finally, use Delaunay triangulation to generate the adjacent edges E in of the point cloud V in , and finally form the intraoperative tissue mesh model M in =(V in ,E in ).

[0105] For the feature extraction module, it is used to respectively obtain corresponding multi-level features according to the preoperative tissue mesh model and the intraoperative tissue mesh model.

[0106] Specifically, the feature extraction module uses Chebyshev spectral graph convolution to extract the multi-level features of the preoperative tissue mesh model and the intraoperative tissue mesh model:

[0107]

[0108]

[0109] where the preoperative tissue mesh model M pre =(V pre ,E pre ), V pre represents the spatial coordinates of the vertices of the preoperative tissue mesh model, and E preRepresents the edges between the vertices of the preoperative tissue mesh model; the intraoperative tissue mesh model M in =(V in , E in ), where V in represents the spatial coordinates of the vertices of the preoperative tissue mesh model, and E in represents the edges between the vertices of the intraoperative tissue mesh model;

[0110] and respectively represent the downsampling scale features of the (n + 1)-th layer and the n-th layer of the preoperative tissue model, and initialize as V pre ; and respectively represent the features of the (n + 1)-th layer and the n-th layer of the intraoperative tissue model, and initialize as V in ;

[0111] are respectively the b-th order Chebyshev polynomials calculated from their respective vertices and their B-ring neighborhoods, are respectively the scaled Laplacian matrices calculated from the edges E in , E pre , are the learning parameters of the neural network.

[0112] For the overlap prediction module, it is used to obtain the overlapping region of the preoperative tissue mesh model and the intraoperative tissue mesh model according to the multi-level features, and obtain the pose transformation relationship of the vertices of the preoperative tissue mesh model within the overlapping region; wherein the pose transformation relationship includes a rotation matrix and a displacement vector.

[0113] Specifically, the overlap prediction module is used for:

[0114] Adopt an attention mechanism to obtain the overlapping region of the preoperative tissue mesh model and the intraoperative tissue mesh model, including:

[0115]

[0116]

[0117] where O pre represents the mask of the overlapping region of the preoperative tissue mesh model M pre ; O in represents the mask of the overlapping region of the intraoperative tissue mesh model M in ; cross and self respectively represent self-attention and cross-attention operations; and Represent the m - level down - sampling scale features of the vertices of the preoperative tissue mesh model and the intraoperative tissue mesh model respectively;

[0118] According to the mask O pre and O in , obtain the vertices within their respective overlapping regions and their features And use a multi - layer perceptron MLP to calculate the corresponding points of the vertices pre in the preoperative tissue mesh model M :

[0119]

[0120] Wherein, is the vertex in the intraoperative tissue mesh model M in , corresponding to the vertex pre in the preoperative tissue mesh model M represents the cosine similarity calculation, represents the position encoding operation on the vertices of the intraoperative tissue mesh model within the overlapping region;

[0121] Adopt the k - nearest neighbor search KNN to establish the local neighborhood of the vertex , and use the singular value decomposition SVD to solve the rotation matrix. The formula is as follows:

[0122]

[0123] Wherein, represents the rotation matrix of the vertex ; represents using the KNN algorithm to construct a local neighborhood belonging to the vertex ; is the neighborhood point of the vertex in the preoperative tissue mesh model, is the vertex of the intraoperative tissue mesh model corresponding to the neighborhood point ;

[0124] Adopt the rotation matrix to change the point cloud coordinates to obtain Adopt MLP to predict the displacement vector of the vertex . The formula is as follows:

[0125]

[0126] Wherein, represents the displacement vector of the vertices of the preoperative tissue mesh model within the overlapping region.

[0127] For the global fusion module, it is used to obtain the coordinates of all vertices of the pre-operative tissue mesh model after registration according to the coordinates and pose transformation relationships of the vertices in the overlapping region of the pre-operative tissue mesh model and the coordinates of the vertices in the non-overlapping region;

[0128] Specifically, the global fusion module is used for:

[0129] Using MLP to regress the rotation matrix and displacement vector of all vertices of the pre-operative tissue mesh model:

[0130]

[0131] where, R pre , t pre respectively represent the rotation matrix and displacement vector of all vertices of the pre-operative tissue mesh model; represents the weight calculated according to the distance between the vertices in the overlapping region and all vertices v pre of the pre-operative tissue mesh model (where all vertices include the vertices in the overlapping region and the vertices in the non-overlapping region);

[0132]

[0133] where, represents the coordinates of all vertices of the pre-operative tissue mesh model after registration.

[0134] Accordingly, it can be clarified that the multi-modal fusion network based on grid data proposed in the embodiments of the present invention predicts the overlapping region and its displacement field through the overlapping prediction module, and combines the Cauchy-Green invariant to constrain the non-rigid deformation of the pre-operative tissue mesh model, making the multi-modal fusion model more reasonable and reducing the errors of multi-modal fusion.

[0135] For the information display module, it is used to display the internal tissue information of the pre-operative tissue mesh model in the intra-operative tissue mesh model according to the coordinates of all vertices of the pre-operative tissue mesh model after registration.

[0136] Exemplarily, in this module, a VR glasses can be used to display the two registered three-dimensional models in a unified coordinate system, or the registered pre-operative tissue mesh model can be superimposed on the endoscopic image according to the basic principle of camera imaging. These two optional display methods can both present the internal tissue information to the doctor, assist the doctor in making clinical decisions, reduce the surgical risk and improve the surgical efficiency at the same time.

[0137] It should be noted that, compared with the virtual registration dataset for constructing biomechanical models in the prior art, in the embodiments of the present invention, in view of the characteristics of the in-vivo flexible dynamic environment, a dataset is constructed using real endoscopic images and medical test data, and the network registered with this dataset has higher accuracy.

[0138] Specifically, in the training stage of the three-dimensional grid model registration and fusion system, a training set is generated based on real data, including:

[0139] Based on the feature point pairs between the specified binocular endoscopic image frames and the preoperative tissue grid model, a non-rigid algorithm is used to register the preoperative tissue grid model and the intraoperative tissue grid model based on the feature points. For any feature point, there is:

[0140]

[0141] Among them, Non_rigid_ICP represents the non-rigid registration algorithm ICP, represents the a-th feature point for non-rigid registration of the preoperative tissue grid model, corresponds to the feature point of the intraoperative tissue grid model, T G is the overall transformation matrix of the preoperative tissue grid model, T l,a is the local deformation transformation matrix belonging to the feature point v pre,a ;

[0142] The local deformation transformation matrix T l of all vertices in the preoperative tissue grid model is obtained by quaternion interpolation, and the registered coordinate label pre of the vertex v

[0143] in the preoperative tissue grid model is obtained through the transformation relationship.

[0144]

[0145] Among them, Loss s represents the supervised loss function in the training stage;

[0146] β s and γ s respectively represent the supervised loss term coefficients;

[0147] N1 represents the number of vertices of the preoperative tissue grid model M pre ;

[0148] represents the l2 true value loss based on the manually labeled dataset, Represents all vertex coordinates after preoperative tissue mesh model registration;

[0149] I c +II c +III c Represents the Cauchy-Green invariant, which is used to constrain the degree of in vivo tissue deformation. I c Constrains the arc distance length between two points on the surface to be invariant. II c Constrains the tissue surface area to be invariant. III c Constrains the tissue volume to be invariant.

[0150] In particular, when registering the specified binocular endoscope image frames in the embodiments of the present invention, due to the differences in endoscope lighting and patient individuality, there are also differences between the reconstructed intraoperative tissue mesh model and the data set. These differences may lead to a decrease in registration accuracy. Using an unsupervised loss fine-tuning network can improve the registration accuracy.

[0151] Accordingly, as Figure 2 shown, the three-dimensional mesh model registration and fusion system provided by the embodiments of the present invention further includes:

[0152] An accuracy fine-tuning module, which is used to introduce an unsupervised loss fine-tuning network to assist the global fusion module in obtaining all vertex coordinates after the preoperative tissue mesh model registration.

[0153] Then, during the application of the unsupervised loss fine-tuning network, the following unsupervised loss function needs to be constructed:

[0154]

[0155] Among them, Loss u represents the unsupervised loss function;

[0156] β u , γ u respectively represent the coefficients of the unsupervised loss terms. and are both vertex coordinates after the preoperative tissue mesh model registration during unsupervised training. represents the vertex in the intraoperative tissue mesh model that is the closest to the vertex of the preoperative tissue mesh model after registration. represents the vertex and 's Euclidean distance. represents the vertex in the preoperative tissue mesh model after registration that is the closest to the vertex v in,b of the intraoperative tissue mesh model. represents the vertex v in,b and the vertex 's Euclidean distance;

[0157] Let \(N1\) denote the number of vertices of the preoperative tissue mesh model \(M\) pre and \(N2\) denote the number of vertices of the intraoperative tissue mesh model \(M\) in ;

[0158] denote the Cauchy - Green invariant during unsupervised training, constrain the arc - length distance between two points on the surface to be invariant, constrain the surface area of the tissue to be invariant, constrain the volume of the tissue to be invariant.

[0159] In the embodiments of the present invention, an unsupervised fine - tuning mechanism with bidirectional nearest neighbor as the loss function is constructed to achieve the precise fusion of the preoperative tissue mesh model and the intraoperative tissue mesh model under the specified binocular endoscopic image frames.

[0160] In addition, in addition to the factors mentioned above that may affect the fusion accuracy, how the second modeling module obtains the depth value of the specified binocular endoscopic image frame is also one of the key factors, because this directly affects the accuracy of the intraoperative tissue mesh model.

[0161] As mentioned above, the second modeling module uses an online self - supervised learning depth estimation method based on binocular endoscopy to obtain the depth value of the specified binocular endoscopic image frame; the binocular depth estimation network used in the online self - supervised learning depth estimation method has the ability of fast over - learning and can continuously adapt to new scenarios using self - supervised information;

[0162] In the real - time reconstruction mode, the second modeling module is specifically used to over - fit the continuous video frames of the image to obtain the depth value of the specified binocular endoscopic image frame, including:

[0163] An extraction unit, used to obtain binocular endoscopic images and extract multi - scale features of the current frame image using the encoder network of the current binocular depth estimation network;

[0164] A fusion unit, used to fuse the multi - scale features using the decoder network of the current binocular depth estimation network to obtain the disparity of each pixel point in the current frame image;

[0165] A conversion unit, used to convert the disparity into depth according to the internal and external parameters of the camera and output it as the result of the current frame image;

[0166] A first estimation unit, used to update the parameters of the current binocular depth estimation network using self - supervised loss without introducing external ground truth for the depth estimation of the next frame image.

[0167] This depth estimation scheme utilizes the similarity of consecutive frames to extend the overfitting idea on a pair of binocular images to overfitting in the time series. By continuously updating the model parameters through online learning, high-precision tissue depth can be obtained in various binocular endoscopic surgical environments.

[0168] In the pre-training stage of the binocular depth estimation network, the traditional training mode is abandoned, and the idea of meta-learning is adopted. The network is allowed to learn a certain image to predict the depth of another image, thereby calculating the loss for network update. This can effectively promote the generalization of the network to new scenarios and improve the robustness to low-texture complex illumination, while significantly reducing the time required for subsequent overfitting.

[0169] As Figure 3 shown in part b, the initial model parameters corresponding to the binocular depth estimation network are obtained through training in the meta-learning manner, specifically including:

[0170] S100. Randomly select an even number of pairs of binocular images {e1, e2,..., e 2K}, and evenly divide them into a support set and a query set and Randomly pair the images in to form K tasks

[0171] S200. Inner-loop training: Calculate the loss based on the support set images in for one parameter update;

[0172]

[0173] where, represents the network parameters after inner-loop update; represents taking the derivative, α is the learning rate of the inner loop, is the support set image of the k-th task, is the loss calculated according to the initial model parameters φ m ; f represents the binocular depth estimation network;

[0174] S300. Outer-loop training: According to the query set images in , use the updated model to calculate the meta-learning loss and directly update the initial model parameters φ m to φ m+1 ;

[0175]

[0176] where, β is the learning rate of the outer loop; is the query set image of the k-th task, is the meta-learning loss.

[0177] The following is a detailed description of each unit included in the second modeling module:

[0178] For the extraction unit, as Figure 3 shown in part a of , it obtains binocular endoscope images and uses the encoder network of the current binocular depth estimation network to extract multi-scale features of the current frame image.

[0179] Exemplarily, in this unit, the encoder of the binocular depth estimation network adopts a ResNet18 network, which is used to extract feature maps of 5 scales for the current frame image (left eye and right eye) respectively.

[0180] For the fusion unit, as Figure 3 shown in part a of , it uses the decoder network of the current binocular depth estimation network to fuse the multi-scale features and obtain the disparity of each pixel point in the current frame image; specifically including:

[0181] Using the decoder network to pass the coarse-scale feature map through a convolutional block and upsampling, splicing it with the fine-scale feature map, and then passing through a convolutional block again for feature fusion, where the convolutional block is constructed by combining a reflection padding layer, a convolutional layer, and a non-linear activation unit ELU;

[0182] Calculating the disparity directly according to the output with the highest network resolution:

[0183] d = k · (sigmoid(conv(Y)) - TH)

[0184] where d represents the estimated disparity value of the pixel point; k is the preset maximum disparity range, Y is the output with the highest resolution; TH represents a parameter related to the type of binocular endoscope, taking 0.5 when the endoscope image has negative disparity and taking 0 when the endoscope images are all positive disparities; conv is the convolutional layer; sigmoid performs range normalization.

[0185] For the conversion unit, it converts the disparity into depth according to the internal and external parameters of the camera and outputs it as the result of the current frame image

[0186] Converting the disparity into depth in this unit means:

[0187]

[0188] where c x1 、 are the corresponding parameters in the left and right binocular endoscopes in the binocular endoscope and the camera internal parameter matrix respectively; if f x takes the corresponding internal parameter of the left camera When d takes the disparity estimation value of the left-eye pixel point, D is the depth estimation value of the left-eye pixel point at this time; if f x takes the corresponding internal parameters of the right-eye camera then d takes the disparity estimation value of the right-eye pixel point, and D is the depth estimation value of the right-eye pixel point at this time; b is the baseline length, that is, the external parameters of the binocular camera.

[0189] For the first estimation module, as shown in part b of Figure 3 , without introducing external ground truth, it uses self-supervised loss to update the parameters of the current binocular depth estimation network for depth estimation of the next frame of image.

[0190] It is not difficult to understand that the "external ground truth" mentioned in the embodiments of the present invention is the label (or called "supervision information"), which is a well-known expression in this field.

[0191] In this unit, as shown in part b of Figure 3 , the self-supervised loss is expressed as:

[0192]

[0193] Among them, L self represents the self-supervised loss; α1, α2, α3, α4 are all hyperparameters, l corresponds to the left image, and r corresponds to the right image.

[0194] Because the binocular observes the same scene, the values of the corresponding pixel points on the left and right depth maps should be equal after being transformed into the same coordinate system. Introducing and

[0195] (1) represents the geometric consistency loss of the left image:

[0196]

[0197] Among them, P1 represents the set of first valid pixel points (i.e., right-eye valid pixel points); represents the left-eye depth obtained by transforming the valid pixel point p from the right-eye depth map through the camera pose transformation, and D l [[ID= forty-six]]′(p) represents the left-eye depth sampled on the left-eye depth map using the predicted right-eye disparity Dis R .

[0198] (2) represents the geometric consistency loss of the right image:

[0199]

[0200] Among them, P2 represents the set of second valid pixel points (i.e., left-eye valid pixel points); Indicates that the right-eye depth D obtained by the effective pixel point p from the left-eye depth map after camera pose transformation r ′(p) represents the right-eye depth obtained by sampling the effective pixel point p using the predicted left-image disparity Dis L , on the right-eye depth map.

[0201] It is achieved by adding geometric consistency constraints to the training loss, ensuring the network's generality for hardware and enabling autonomous adaptation to unconventional binocular images such as surgical endoscopes.

[0202] Assuming constant brightness and spatial smoothness in endoscopic surgery, reprojection between the left and right-eye images can be used to achieve the reconstruction of the other eye. At the same time, the structural similarity loss is increased, normalizing and comparing the brightness, contrast, and structure of the two images, introducing and

[0203] (3) Represents the photometric loss of the left image:

[0204]

[0205] where I L (p) represents the left image, I L ′(p) represents the left-eye endoscopic reconstruction image generated using the right image and the predicted left-image disparity Dis L (p), λ i and λ s are balance parameters, and SSIM LL′ (p) represents the image structural similarity between I L (p) and I L ′(p);

[0206] (4) Represents the photometric loss of the right image:

[0207]

[0208] where I R (p) represents the right image, and I′ R (p) represents the right-eye endoscopic reconstruction image generated using the left image and the predicted right-image disparity Dis R (p), and SSIM RR′ (p) represents the image structural similarity between I R (p) and I′ R (p).

[0209] In regions of low-texture and single-color tissues, smooth priors are used to assist in inference and the depth is regularized, introducing and

[0210] (5) Indicates the smoothness loss of the left image:

[0211]

[0212] where, represents the normalized left-eye depth map, and represent the first-order derivatives along the horizontal and vertical directions of the image;

[0213] (6) Indicates the smoothness loss of the right image:

[0214]

[0215] where, represents the normalized right-eye depth map, and represent the first-order derivatives along the horizontal and vertical directions of the image.

[0216] Specifically, the processes for obtaining the first set of valid pixel points P1 and the second set of valid pixel points P2 are as follows:

[0217] Define the left-eye visual difference predicted by the current binocular depth estimation network as and the right-eye visual difference as Then the formulation of the left-eye and right-eye cross-validation masks is as follows:

[0218]

[0219]

[0220] where, are respectively used to determine whether the pixel at the (i, j) position in the left-eye and right-eye images is within the stereo matching range; i ranges from all integers between [1, W]; j ranges from all integers between [1, H]; W represents the image width, and H represents the image height;

[0221] Let c take L or R. When it indicates that the pixel at the (i, j) position is within the stereo matching range under the current calculation method, otherwise it is not within the stereo matching range;

[0222] Use the pinhole camera model, binocular pose transformation, and predicted depth for projection to obtain the valid region mask based on 3D points Take 0 or 1. When it indicates that the pixel at the (i, j) position is within the stereo matching range under the current calculation method, otherwise it is not within the stereo matching range;

[0223] Obtain the final valid region mask

[0224]

[0225] If pixel point p satisfies When c takes R, the first set of valid pixel points P1 is obtained. When c takes L, the second set of valid pixel points P2 is obtained.

[0226] In the corrected stereo image, for the extra area caused by the viewing angle shift, no matching pixels can be found. However, the embodiments of the present invention take into account that the low texture and uneven illumination of the in-vivo tissue will make the local features less obvious, and the pixels in these invalid areas often find similar pixels in the adjacent areas. Therefore, as mentioned above, the embodiments of the present invention propose a cross-validation binocular valid area recognition algorithm, which excludes the misleading of the self-supervised loss of the invalid area pixels to the network learning and improves the accuracy of depth estimation.

[0227] In addition, to avoid insufficient robustness in depth estimation for pure texture or low-light scenes,

[0228] (7) Represents the sparse optical flow loss:

[0229]

[0230] Among them, Dis L (p) represents the predicted left-eye disparity map, OF L (p) represents the left-eye sparse disparity map, Dis R (p) represents the predicted right-eye disparity map, OF R (p) represents the right-eye sparse disparity map; P3 represents the third set of valid pixels in the left-eye sparse disparity map OF L (p); P4 represents the fourth set of valid pixels in the right-eye sparse disparity map OF R (p); γ1 and γ2 are balance parameters, both non-negative numbers and not both zero at the same time.

[0231] Specifically, the obtaining processes of the third set of valid pixel points P3 and the fourth set of valid pixel points P4 are as follows:

[0232] Using the LK (Lucas-Kanade) optical flow solution algorithm, calculate the sparse optical flow (Δx, Δy) every n pixels in the row and column directions, where Δx represents the horizontal offset of the pixel point, and Δy represents the vertical offset of the pixel point;

[0233] When solving the optical flow from the left image to the right image, only when And Δx > thred1, the disparity at this pixel position is retained as Δx, where KT and thred1 are corresponding preset thresholds. For those not meeting the above conditions or for which the disparity at the sparse optical flow position is not calculated, it is set to 0 to obtain the final sparse disparity map OF L (p), OF L Pixels with OF(p)≠0 form the third set of valid pixel points P3;

[0234] When solving the optical flow from the right image to the left image, only when And Δx < thred2, the disparity at this pixel position is retained as Δx, where thred2 is the corresponding preset threshold. For those not meeting the above conditions or for which the disparity at the sparse optical flow position is not calculated, it is set to 0 to obtain the final sparse disparity map OF R (p), OF R Pixels with OF(p)≠0 form the fourth set of valid pixel points P4.

[0235] As mentioned above, in the embodiments of the present invention, the traditional Lucas-Kanade optical flow is introduced to deduce the sparse disparity between binocular images, giving the network a reasonable learning direction, improving the fast learning ability and reducing the probability of falling into local optimum.

[0236] It should be particularly emphasized that, in addition to the real-time reconstruction mode, the online self-supervised learning depth estimation method adopted by the second modeling module in the embodiments of the present invention also sets a precise measurement mode. As Figure 3 shown in part b, in the precise measurement mode, the second modeling module is specifically used to overfit the key image video frames, including:

[0237] The second estimation unit, without introducing external ground truth, according to the binocular depth estimation network obtained in the real-time reconstruction mode from the previous frame of the specified binocular endoscope image frame, updates the parameters of the aforementioned binocular depth estimation network using the self-supervised loss corresponding to the specified binocular endoscope image frame until convergence, and uses the converged binocular depth estimation network for precise depth estimation of the specified binocular endoscope image frame to obtain the depth value of the specified binocular endoscope image frame.

[0238] It should be noted that the technical details such as the depth estimation network, self-supervised loss function, calculation of the valid region mask, and meta-learning pre-training method in the precise measurement mode are all consistent with the technical details extended in the real-time reconstruction mode, and will not be elaborated here.

[0239] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0240] 1. The mesh data introduced by the 3D mesh model registration and fusion system provided by the embodiments of the present invention takes into account the connection relationship between the vertices of the model compared with the traditional point cloud, improving the accuracy of subsequent multi-modal fusion; and can present the internal information of the tissue to the doctor, assisting the doctor in making clinical decisions, reducing the surgical risk while improving the surgical efficiency.

[0241] 2. The embodiments of the present invention generate training data through artificial annotation and interpolation based on real data, train a multi-modal registration and fusion network in a supervised manner, and finally further improve the registration accuracy through unsupervised fine-tuning.

[0242] 3. The embodiments of the present invention introduce an online self-supervised learning depth estimation method based on a binocular endoscope, and its beneficial effects at least include:

[0243] 3.1 The switched depth estimation can not only provide the real-time point cloud of the intraoperative anatomical structure to assist the doctor in intuitively understanding the intraoperative three-dimensional structure, but also realize the high-precision reconstruction of the key frames manually selected by the doctor based on single-frame overfitting, providing a basis for subsequent measurements, so as to balance speed and accuracy in the application.

[0244] 3.2 Utilizing the similarity of consecutive frames, the overfitting idea on a pair of binocular images is extended to overfitting on a time series, and the model parameters are continuously updated through online learning, enabling high-precision tissue depth to be obtained in various binocular endoscope surgical environments.

[0245] 3.3 In the pre-training stage of the network model, the traditional training mode is abandoned, and the idea of meta-learning is adopted. The network is allowed to learn a certain image to predict the depth of another image, thereby calculating the loss for network update, which can effectively promote the generalization of the network to new scenarios and enhance the robustness to low-texture complex illumination, while significantly reducing the time required for subsequent overfitting.

[0246] 3.4 It is realized by adding geometric consistency constraints to the training loss, ensuring the universality of the network for hardware and realizing autonomous adaptation to unconventional binocular images such as surgical endoscopes.

[0247] 3.5 Regarding the depth estimation of each frame of binocular image as an independent task, high-precision models suitable for the current frame are obtained through real-time overfitting; and high-precision depth estimation results can be obtained through online learning to quickly learn new scenarios.

[0248] 3.6 Based on the cross-validation binocular effective region recognition algorithm, the misleading of the self-supervised loss of invalid region pixels to network learning is excluded, improving the accuracy of depth estimation.

[0249] 3.7. Introduce the traditional Lucas-Kanade optical flow to derive the sparse disparity between binocular images, give the network a reasonable learning direction, improve the fast learning ability and reduce the probability of falling into local optima.

[0250] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0251] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional mesh model registration and fusion system for laparoscopic surgery navigation, characterized in that Including: A first modeling module, configured to obtain a preoperative tissue mesh model with tissue semantic information; A second modeling module, configured to obtain an intraoperative tissue mesh model according to the depth value of a specified binocular endoscope image frame; A feature extraction module, configured to respectively obtain corresponding multi-level features according to the preoperative tissue mesh model and the intraoperative tissue mesh model; An overlap prediction module, configured to obtain an overlapping area between the preoperative tissue mesh model and the intraoperative tissue mesh model according to the multi-level features, and obtain a pose transformation relationship of vertices of the preoperative tissue mesh model within the overlapping area; A global fusion module, configured to obtain coordinates of all vertices of the preoperative tissue mesh model after registration according to coordinates and pose transformation relationships of vertices of the preoperative tissue mesh model within the overlapping area, and coordinates of vertices within a non-overlapping area; An information display module, configured to display internal tissue information of the preoperative tissue mesh model in the intraoperative tissue mesh model according to coordinates of all vertices of the preoperative tissue mesh model after registration; 2. The three-dimensional mesh model registration and fusion system according to claim 1, wherein The feature extraction module extracts multi-level features of the preoperative tissue mesh model and the intraoperative tissue mesh model by using Chebyshev spectral graph convolution: Among them, the preoperative tissue mesh model M is defined pre =(V pre , E pre ), where V pre represents the spatial coordinates of the vertices of the preoperative tissue mesh model, and E pre represents the edges between the vertices of the preoperative tissue mesh model; the intraoperative tissue mesh model M in =(V in , E in ), where V in represents the spatial coordinates of the vertices of the preoperative tissue mesh model, and E in represents the edges between the vertices of the intraoperative tissue mesh model; and respectively represent the downsampled scale features of the (n + 1)-th layer and the n-th layer of the preoperative tissue model, and initialize as V pre ; and respectively represent the features of the (n + 1)-th layer and the n-th layer of the intraoperative tissue model, and initialize as V in ; The b-th Chebyshev polynomials calculated respectively from their respective vertices and their B-ring neighborhoods, are respectively the scaled Laplacian matrices calculated from the edges E in , E pre , and are the learning parameters of the neural network.

3. The three-dimensional grid model registration and fusion system according to claim 2, wherein The overlap prediction module is specifically configured to: Obtain the overlapping area between the preoperative tissue mesh model and the intraoperative tissue mesh model by using an attention mechanism, including: Among them, O pre represents the mask of the overlapping region of the preoperative tissue mesh model M pre ; O in represents the mask of the overlapping region of the intraoperative tissue mesh model M in ; cross and self respectively represent self-attention and cross-attention operations; and respectively represent the m-level downsampled scale features of the vertices of the preoperative tissue mesh model and the intraoperative tissue mesh model; According to mask O pre and O in , obtain the vertices and their features that are respectively within the overlapping region and use the multi-layer perceptron MLP to calculate the corresponding points of the vertices in the preoperative tissue mesh model M pre : ​ Among them, is the vertex in the intraoperative tissue mesh model M in corresponding to the vertex in the preoperative tissue mesh model M pre among the vertices represents the cosine similarity calculation, represents the position encoding operation on the vertices of the intraoperative tissue mesh model within the overlapping region; Build vertices using the K-nearest neighbor search (KNN) and establish the local neighborhood of . Solve the rotation matrix using the singular value decomposition (SVD). The formula is as follows: and solve the rotation matrix using the singular value decomposition (SVD). The formula is as follows: Among them, represents the rotation matrix of the vertex ; represents constructing a local neighborhood belonging to the vertex using the KNN algorithm; is the neighborhood point of the vertex of the preoperative tissue mesh model, is the vertex of the intraoperative tissue mesh model corresponding to the neighborhood point ; Using a rotation matrix to obtain the changed point cloud coordinates Using MLP to predict the displacement vector of the vertex The formula is as follows: Among them, represent the displacement vectors of the vertices of the preoperative tissue mesh model in the overlapping area, and form the pose transformation relationship with the rotation matrix ​ 4. The three-dimensional grid model registration and fusion system according to claim 3, wherein The global fusion module is specifically configured to: Use MLP to regress rotation matrices and displacement vectors of all vertices of the preoperative tissue mesh model; Among them, R pre , t pre respectively represent the rotation matrix and displacement vector of all vertices of the preoperative tissue mesh model; represents the vertices located in the overlapping area and the weight calculated according to the distance between all vertices v pre of the preoperative tissue mesh model; Among them, represents all vertex coordinates after the registration of the preoperative tissue mesh model.

5. The three-dimensional mesh model registration and fusion system according to claim 1, wherein In a training stage of the three-dimensional mesh model registration and fusion system, generate a training set based on real data: Based on feature point pairs between a specified binocular endoscope image frame and a preoperative tissue mesh model, register the preoperative tissue mesh model and the intraoperative tissue mesh model by using a non-rigid algorithm based on feature points. For any feature point: Among them, Non_rigid_ICP represents the non-rigid registration algorithm ICP, represents the a-th feature point for non-rigid registration of the preoperative tissue mesh model, corresponding to the feature point of the intraoperative tissue mesh model, T G is the overall transfer matrix of the preoperative tissue mesh model, T l，a is the local deformation transfer matrix belonging to the feature point v pre，a ; The local deformation transfer matrix T of all vertices in the preoperative tissue mesh model is obtained by four - element interpolation l , and the vertex v in the preoperative tissue mesh model is obtained through the transformation relationship pre Coordinate label after registration 6. The three-dimensional mesh model registration and fusion system according to claim 5, characterized in that In a training stage of the three-dimensional mesh model registration and fusion system, construct the following supervised loss function: Among them, Loss s represents the supervised loss function in the training stage; β s and γ s represent the coefficients of the supervised loss term, respectively. $N_1$ represents the number of vertices of the preoperative tissue mesh model $M$ pre ; Denotes the l2 true value loss based on the manually annotated dataset, Denotes all vertex coordinates after preoperative tissue mesh model registration; I c + II c + III c represent the Cauchy-Green invariants, which are used to constrain the degree of deformation of the tissue in the body, I c constrain the length of the arc distance between two points on the surface to remain unchanged, II c constrain the surface area of the tissue to remain unchanged, III c constrain the volume of the tissue to remain unchanged.

7. The three-dimensional grid model registration and fusion system according to claim 1, characterized in that Further including: An accuracy fine-tuning module, configured to introduce an unsupervised loss to fine-tune the network, and assist the global fusion module to obtain coordinates of all vertices of the preoperative tissue mesh model after registration.

8. The three-dimensional mesh model registration and fusion system according to claim 7, wherein When the unsupervised loss fine-tuning network is applied, construct the following unsupervised loss function: Among them, Loss u represents an unsupervised loss function; β u ,γ u respectively represent the coefficients of the unsupervised loss term, and are both the vertex coordinates after the registration of the preoperative tissue mesh model during unsupervised training, represents the closest point to the vertex of the preoperative tissue mesh model after registration in the intraoperative tissue mesh model, represents the vertex and the Euclidean distance between them, represents the closest point to the vertex v in，b of the intraoperative tissue mesh model in the preoperative tissue mesh model after registration, represents the vertex v in，b and the vertex the Euclidean distance between them; Let \(N_1\) denote the number of vertices of the preoperative tissue mesh model \(M\). pre Let \(N_2\) denote the number of vertices of the intraoperative tissue mesh model \(M\). in ; Denote the Cauchy-Green invariant during unsupervised training, Constrain the arc distance length between two points on the surface to be invariant, Constrain the surface area of the tissue to be invariant, Constrain the volume of the tissue to be invariant.

9. The three-dimensional mesh model registration and fusion system according to any one of claims 1 to 8, characterized in that The second modeling module uses an online self-supervised learning depth estimation method based on a binocular endoscope to obtain the depth value of the specified binocular endoscope image frame; the binocular depth estimation network used by the online self-supervised learning depth estimation method has the ability of fast overlearning and can continuously adapt to new scenarios by using self-supervised information; In a real-time reconstruction mode, the second modeling module is specifically configured to overfit continuous image video frames to obtain the depth value of a specified binocular endoscope image frame, including: An extraction unit, configured to obtain binocular endoscope images, and extract multi-scale features of the current frame image by using an encoder network of the current binocular depth estimation network; A fusion unit, configured to fuse the multi-scale features by using a decoder network of the current binocular depth estimation network to obtain the disparity of each pixel point in the current frame image; A conversion unit, configured to convert disparity into depth according to the internal and external parameters of the camera and output the result as the current frame image; A first estimation unit, configured to update the parameters of the current binocular depth estimation network using a self-supervised loss without introducing external ground truth for depth estimation of the next frame image.

10. The 3D mesh model registration and fusion system according to claim 9, wherein In the precise measurement mode, the second modeling module is specifically configured to overfit the key image video frames, including: A second estimation unit, without introducing external ground truth, according to the binocular depth estimation network obtained from the previous frame image of the specified binocular endoscope image frame in the real-time reconstruction mode, updates the parameters of the aforementioned binocular depth estimation network using the self-supervised loss corresponding to the specified binocular endoscope image frame until convergence, and uses the converged binocular depth estimation network for precise depth estimation of the specified binocular endoscope image frame to obtain the depth value of the specified binocular endoscope image frame.

Images