Method and system for model and organ registration to assist navigation

Abstract

The invention relates to electrical digital data processing, and discloses a method and a system for model and organ registration to assist navigation so as to improve accuracy and real-time performance. The method comprises the following steps: after performing feature matching on color images on the left side and the right side along an epipolar direction by adopting a Transform architecture, only reserving parallax candidates of which the matching probabilities among pixels meet a preset confidence threshold, constructing sparse probability parallax cost volumes, performing regularization smoothing processing on each parallax cost volume, then converting each parallax cost volume into normalized probability distribution, and finally performing feature matching on the color images on the left side and the right side along the epipolar direction. Calculating the expectation of the probability distribution as a parallax estimation value, and converting the parallax estimation value of each pixel point into depth information to obtain an initial depth map; performing self-supervised optimization on binocular depth estimation based on NeRF; and performing three-dimensional matching according to the two-dimensional information observed in the color image on the left side or the right side in combination with the depth information of each pixel and the shape and the position of the virtual organ model, so that the virtual organ model is aligned with the current form of the real organ.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular, to a method and system for model-organ registration to assist navigation. Background Art

[0002] Currently, mixed reality (MR), deep learning, organ biomechanical modeling, and stereoscopic vision technology are increasingly widely used in medical surgical navigation. Mixed reality technology can fuse digital models with real images, providing surgeons with intuitive three-dimensional anatomical structure visualization, which helps improve surgical accuracy and reduce the cognitive load of surgeons. For example, in anterior cruciate ligament reconstruction surgery, introducing a three-dimensional model reconstructed from preoperative CT (Computed Tomography) and visualizing it through MR significantly improves the accuracy of femoral tunnel positioning (the positioning error in the MR group is about 0.22±0.16 mm, which is significantly lower than 2.57±0.30 mm of the traditional method). MR also supports human-computer interfaces such as voice control and gesture interaction, enabling information to be obtained and operated without contact during surgery. Deep learning has performed well in medical image processing in recent years, capable of automatically extracting features and identifying different organ information from intraoperative images such as laparoscopy. However, deep learning models can only extract and analyze the features of existing intraoperative images and cannot effectively incorporate information such as preoperative patient examinations. Organ biomechanical modeling simulates the deformation behavior of three-dimensional organs after being stretched or compressed through physical models, but cannot be adapted to intraoperative two-dimensional images such as laparoscopy in real time. Binocular laparoscopic stereoscopic imaging can obtain the depth information of the surgical field scene in real time and obtain a three-dimensional reconstruction of the organ surface, but can only give doctors a three-dimensional stereoscopic visual sense and cannot provide additional information to the doctor. However, there are still many limitations and deficiencies in the prior art for integrating the above technologies for intraoperative real-time navigation.

[0003] Limitations of Stereo Laparoscopic Technology: Traditional binocular stereo matching algorithms rely on manually designed feature matching and are vulnerable to interference from complex factors in the surgical environment. In a minimally invasive surgery (MIS) environment, there are situations such as smoke, liquid (blood), instrument occlusion, and lighting changes, all of which pose challenges to stereo matching. Uneven lighting and reflections can cause changes in image contrast, and traditional algorithms often struggle to find robust corresponding points in weakly textured areas or areas with high repetitive textures. Therefore, the stereo reconstruction accuracy is not ideal in these scenarios, and the depth map often exhibits noise and holes. In particular, for the surface of organs in laparoscopic surgery, many areas lack significant features, making it difficult for traditional methods to match, resulting in insufficient depth reconstruction accuracy. In addition, the movement of laparoscopic cameras and organs introduces dynamic changes in the parallax range, and the fixed parallax search range preset by conventional stereo algorithms is difficult to handle rapid / large deformation scenarios. On the other hand, to meet the real-time requirements of intraoperative applications, the algorithm needs to complete calculations within a limited time, which often forces a compromise between accuracy and speed. Algorithms implemented on traditional CPUs often cannot meet the real-time requirements, and although GPU parallel acceleration speeds up the processing, more optimized algorithms are still needed to further reduce the latency.

[0004] Problems of Organ Deformation and Registration: Organs during surgery (such as the liver, kidney, etc.) continuously deform due to breathing, traction, etc. This organ deformation weakens the accuracy of navigation guided by preoperative images because the preoperative images do not match the intraoperative organ morphology. Traditional surgical navigation mostly relies on using preoperative images for intraoperative reference, but the displacement and deformation of organs limit this method to almost rigid or near-rigid anatomical structures (such as bones). To solve this problem, it is necessary to combine biomechanical models with intraoperative acquired data to update the preoperative model in real time to match the intraoperative situation. Biomechanical models such as the Finite Element Method (FEM) are considered promising methods to simulate organ deformation based on mechanical properties and perform elastic deformation correction on preoperative images. However, fine FEM modeling is computationally complex and has too high a computational cost when directly applied to the intraoperative situation. In addition, sufficient intraoperative data needs to be obtained to constrain and drive the model (for example, obtaining the position changes of organ surface point clouds or marker points during surgery). Current laparoscopic stereo reconstruction provides a means to obtain surface point clouds, but how to use sparse / semi-dense point clouds to drive high-dimensional non-rigid models remains a difficult problem. If relying solely on manual registration, it is not only time-consuming but may also result in misalignment between virtual and real. Existing research has pointed out that when manually adjusting the position of the holographic model to align with the real organ in MR navigation, there is a risk of model misalignment and the accuracy is difficult to guarantee. Therefore, more intelligent algorithms are needed to autonomously complete the registration between the model and the organ that is difficult for humans to accurately complete.

[0005] Existing deficiencies of mixed reality surgical navigation: Although MR has shown great potential in surgical navigation, there are still limitations in registration accuracy in current applications: Many MR navigation systems rely on optical or manual initialization for registration, and error accumulation may occur in the position correspondence between holographic images and real anatomy. Especially when organs are dynamically changing, if the initial registration is inaccurate, obvious deviations may occur during the operation. For example, experiments have found that the stability of spatial registration using some MR glasses is poor, and the average positioning error between different users can reach about 6 mm, far from meeting the requirements of delicate surgeries. Therefore, improving MR registration accuracy is an urgent problem to be solved. Summary of the Invention

[0006] The purpose of the present invention is to disclose a method and system for model-organ registration to assist navigation, so as to improve accuracy and real-time performance.

[0007] To achieve the above object, the method for model-organ registration to assist navigation disclosed by the present invention includes: Step S1: Convert the disparity of the color images on the left and right sides of a binocular laparoscope into depth information to obtain an initial depth map; The color images on the left and right sides first pass through a multi-scale CNN (Convolutional Neural Networks) to extract feature pyramids, and the image features at different scales are fused to obtain a multi-channel feature representation of the left and right eyes. A Transformer architecture based on epipolar constraint is used to execute self-attention and cross-attention mechanisms. The self-attention mechanism calculates pixel-level correlation weights along the epipolar direction in the feature sequence of the same view. After the cross-attention mechanism performs feature matching along the epipolar direction between the color images on the left and right sides, only the disparity candidates whose pixel matching probabilities meet the preset confidence threshold are retained to construct a sparse probability disparity cost volume. Then, each disparity cost volume is subjected to regularization smoothing and converted into a normalized probability distribution, and then the expectation of this probability distribution is calculated as the disparity estimate value; Step S2: Perform self-supervised optimization on the binocular depth estimation based on a trained NeRF (Neural Radiance Fields) model. The training process of the NeRF model specifically includes: Step S21: Input the initial depth map as a depth prior and the corresponding left or right unilateral color image into the NeRF model together. The NeRF model synthesizes a new image from a preset perspective through volume rendering technology, and compares the synthesized new image with the input left or right unilateral color image to calculate the reprojection error; Step S22: Process pixel depth and color rendering in the same rendering manner. Replace the color of each sampling point with the corresponding depth position, then divide by the total weight to obtain the depth rendering. Then map the calculated depth rendering values to the corresponding pixels to obtain the depth rendering values of each pixel. Then calculate the depth consistency loss between the depth rendering values inferred by the rendering and the disparity estimation value. Step S23: Adopt a strategy of jointly and alternately training the NeRF model and the Transformer architecture to gradually optimize the depth estimation result. Iteratively run to fix the NeRF model and update the Transformer architecture to reduce the pixel color reprojection error and the depth consistency error, and fix the Transformer architecture and update the NeRF model to improve the fitting accuracy of the laparoscopic view geometric consistency, ultimately minimizing the weighted sum of the reprojection error and the depth consistency loss. Step S3: Perform three-dimensional matching based on the two-dimensional information observed in the left or right unilateral color image, combined with the depth information of each pixel and the shape and position of the virtual organ model, to align the virtual organ model with the current morphology of the real organ.

[0008] Preferably, in step S1, during the process of calculating the sparse probability disparity cost volume that meets the confidence threshold according to the matching probability between pixels, introduce a sparse attention strategy to limit the disparity search within the range of matching probabilities greater than the confidence threshold.

[0009] Preferably, step S3 specifically includes: Step S31: Combine the physical simulation ability of finite element analysis with the machine learning ability of the graph neural network to achieve precise registration of organ non-rigid deformation. Step S32: Use the visual SLAM (Simultaneous Localization and Mapping) algorithm to estimate the pose of the laparoscopic camera in real time. Step S33: After obtaining the organ deformation and the camera pose, superimpose the virtual organ model onto the surgical field image through mixed reality technology.

[0010] Preferably, step S31 specifically includes: Step S311: The finite element model calculates the deformation distribution of each part of the organ under the action of external forces based on the biomechanical characteristics and boundaries of the organ, providing physically realistic deformation results. Step S312: Discretely represent the organ as a graph structure, and input the mesh node displacements and topological connections calculated by the finite element into the GNN (Graph Neural Network) model. Step S313: The GNN model uses the grid nodes of the organ as graph nodes and the grid topology as the connections of the graph. By learning the correlations and deformation patterns between the nodes, it predicts the real-time deformation of the organ. The correlations include the mechanical correlations between the nodes learned using a message passing mechanism. The mechanical correlations include the mapping relationship from the boundary formed by the displacements or forces received on the organ surface to the displacement field of the internal nodes after displacement.

[0011] Preferably, step S32 specifically includes: Step S321: The SLAM algorithm obtains the direction and position trajectory of the camera in the surgical environment by minimizing the reprojection error of the feature points extracted and matched frame by frame in the color image of the left or right unilateral side, and simultaneously constructs a dense point cloud of the surgical area to constrain the camera positioning by the anatomical structure. Step S322: According to the camera positioning information, the virtual organ model is converted to the camera coordinate system for alignment rendering. Step S323: By minimizing the alignment error between the virtual image displayed by the MR device and the real picture, the posture and position of the virtual organ model are adjusted in real time, so as to ensure that the virtual organ model is spatially aligned with the actual endoscopic color image of the left or right unilateral side.

[0012] Preferably, the method of the present invention further includes: using a Kalman filter to fuse the camera pose solved by visual SLAM with the high-frequency acceleration, angular velocity measured by the IMU (Inertial Measurement Unit), and the positioning result of the EM (Electromagnetic Tracking System). The IMU and EM are different sensors deployed on the laparoscopic lens.

[0013] Preferably, step S33 specifically includes: Step S331: Obtain the internal and external parameters calibrated by the laparoscopic camera. The internal parameters include the focal length, principal point, and distortion coefficient, and the external parameters include the pose in the surgical coordinate system. Set the parameters of the virtual camera according to the calibration results of the internal and external parameters to make the viewing angle of the rendered virtual model consistent with the real laparoscopic camera. Step S332: Transform the virtual organ model to the surgical coordinate system, and adjust the size and position of the virtual organ model according to the actual anatomical ratio to ensure that the virtual organ model is spatially aligned with the real organ. Step S333: During the rendering process, perform lens lighting matching, correct the distortion of the virtual organ image according to the lens characteristics of the laparoscope, and simulate the lighting effect of the surgical light source on the virtual organ model. And use the dense point cloud reconstructed by SLAM or key anatomical features to dynamically adjust the position and scale of the virtual organ model during the operation.

[0014] Preferably, the process of dynamically adjusting the position and scale of the virtual organ model during the operation specifically includes: Step S3331: Extract a series of three-dimensional key contour vertices evenly distributed along the organ edge or the fitted contour curve from the anatomical surface of the reconstructed virtual model , For the contour vertex index, obtain the pose of the virtual organ model in the mixed reality coordinate system , where is the rotation matrix of the model, is the three-dimensional translation matrix of the model, and each three-dimensional contour point , calculate the projection of each contour point on the image plane , where is the homogeneous depth of the projected point, is the two-dimensional coordinate of the contour vertex on the image plane; the superscript is the transpose symbol; Step S3332: Using the organ edge contour in the actual picture as a reference, obtain the set of real contour pixel points For each projection point , find the nearest neighbor point of each projection point in the real contour set , calculate the alignment error of the nearest neighbor point, and use minimizing the sum of squared deviations of all contour points as the optimization goal to find the optimal rigid body fine-tuning transformation.

[0015] To achieve the above object, the present invention also discloses a system for model and organ registration to assist navigation, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above method is implemented.

[0016] The present invention has the following beneficial effects: 1. Introducing Transformer into stereo matching realizes a new mechanism for global context-aware epipolar matching. Traditional stereo matching is usually limited by local correlation, while the Transformer architecture based on the present invention can establish pixel correspondence relationships globally, significantly improving the matching accuracy; at the same time, after feature matching along the epipolar direction of the left and right color images, only the disparity candidates whose matching probability between pixels meets the preset confidence threshold are retained, a sparse probability disparity cost volume is constructed, and then each disparity cost volume is subjected to regularization smoothing processing and converted into a normalized probability distribution, and then the expectation of the probability distribution is calculated as the disparity estimate value, significantly improving the real-time performance of disparity calculation.

[0017] 2. Introduce NeRF as a deep supervision signal, transform depth optimization into a differentiable new view synthesis problem, and optimize the depth result from the perspective of global photometric consistency. Under the global supervision of NeRF, the deficiencies of traditional stereo matching in low-texture or high-gloss regions are effectively compensated, making the depth estimation in difficult regions equally accurate. At the same time, the joint optimization of depth consistency constraints and the NeRF-Transformer stereo network ensures that the depth map not only satisfies the photo consistency of reprojection but also maintains geometric accuracy; the innovative strategy of fusing neural rendering and stereo vision greatly improves the details and reliability of depth estimation, achieving higher accuracy.

[0018] 3. Perform three-dimensional matching by combining the two-dimensional information observed in the left or right unilateral color image with the depth information of each pixel and the shape and position of the virtual organ model, ensuring the accuracy of aligning the virtual organ model with the current morphology of the real organ.

[0019] The present invention will be further described in detail below with reference to the accompanying drawings. Description of the Drawings

[0020] The accompanying drawings that form a part of this application are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is a schematic flowchart of the method for model-organ registration to assist navigation disclosed in Embodiment 1 of the present invention. Detailed Embodiments

[0021] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings. However, the present invention can be implemented in many different ways defined and covered by the claims.

[0022] Embodiment 1 This embodiment discloses a method for intraoperative real-time flexible organ registration based on the fusion of binocular laparoscopy reconstruction and mixed reality. The system integrates modules such as binocular stereo vision, Neural Radiance Field (NeRF), finite element mechanics modeling, SLAM tracking, and mixed reality (MR) rendering, meeting the requirements of surgical real-time while ensuring registration accuracy. The overall process is as follows: First, three-dimensional reconstruction is performed using the color images on the left and right sides of the binocular laparoscope; Second, NeRF is introduced to perform self-supervised optimization of depth estimation, improving the reconstruction accuracy and reducing the dependence on a large amount of calibration data; Then, organ deformation prediction combining a finite element model (FEM) and a graph neural network (GNN) is constructed to achieve the physical deformation of the intraoperative organ model; Next, an improved visual SLAM algorithm is used to obtain the pose of the laparoscope camera, and combined with the pre-registered preoperative model, accurate alignment between the virtual model and the endoscopic image is achieved; Finally, optimized rendering superimposes the corrected virtual organ model on the intraoperative image in real time, providing realistic augmented reality navigation support. The technical details of each functional module are elaborated below, along with the corresponding mathematical models and performance data.

[0023] I. Optimization of the binocular Transformer stereo reconstruction module.

[0024] This module uses an improved Transformer network to complete high-precision stereo matching and depth estimation of the color images on the left and right sides of the laparoscope.

[0025] A1. Improvement of the embedding layer: The left and right laparoscope images (or: the color images on the left and right sides, where the side can also be called the eye, and will not be elaborated later) first pass through a convolutional neural network with shared weights (the left and right laparoscope images of the binocular laparoscope are respectively input into two parallel network branches using exactly the same set of convolutional kernel parameters) to extract multi-scale feature pyramids.

[0026] Preferably, this embodiment can also encode the position of each pixel and the estimated disparity range based on the internal parameters of the two cameras into the features to fuse the internal parameters of the cameras and epipolar constraint information. It can be expressed in functional form as: for each pixel coordinate of the left image, its corresponding normalized coordinate and the assumed disparity prior are mapped together into a position vector and embedded into the feature space through a linear transformation. This embedding preserves the pixel geometric position and disparity prior, helping the Transformer to make full use of the spatial relationship during subsequent processing, improving the accuracy and robustness of matching. Compared with traditional CNN features without explicit geometric encoding, the Transformer combined with sequence position embedding can capture the global dependencies of the image. This makes it more robust in finding corresponding points in weakly textured or repetitive texture regions.

[0027] Among them, the estimated parallax range is the preset depth range that each pixel may have. By default, the depth of each pixel is selected within the estimated parallax range. The acquisition method can be specifically as follows: First, obtain the parameters of the endoscopic camera from the endoscopic manufacturer, mainly the baseline length b and the focal length f; according to the typical working distance between the camera and the human tissue, set the depth range (by default, [Zmin, Zmax] = [2 cm, 10 cm]). Then, use the formula d = bf / Z to map the depth upper and lower limits to the parallax upper and lower limits, where d is the calculation formula for the parallax upper and lower limits, that is, the estimated parallax range is [dmin, dmax] = [bf / Zmax, bf / Zmin]. Finally, discretize [dmin, dmax] into {dmin, dmin + Δd, dmin + 2×Δd,..., dmax}; where, Δd can be adjusted according to the computing power of the computer and the expected accuracy of stereo matching. By default, it can be (dmax - dmin) / 50, that is, it can be discretized into 50 + 1 = 51 estimated parallax range intervals; and it is assumed that the parallax d is a specific value within the discretized estimated parallax range.

[0028] A2. Optimization of the attention mechanism: In the Transformer matching module, in this embodiment, alternating self-attention and cross-attention layers are adopted to process the left and right feature sequences. The multi-head self-attention calculates long-range associations along the epipolar direction within each view (including the left and right color images) to capture the image context; subsequently, the cross-attention calculates the matching correlation using the query of one view and the key and value of the other view to achieve feature alignment across views. This alternating attention mechanism breaks through the limitation of the fixed-length cost window and allows searching for matches along the entire epipolar line (that is: enabling each pixel to "look far" across the entire horizontal line, being able to associate with more macroscopic textures and structures, so that there is both local accuracy and global consistency during matching). Through context awareness, it can help the model find more reliable matching points in weak-texture or repetitive-texture regions by combining more obvious features in the distance; thus having stronger adaptability to large parallax and occlusion regions. In addition, to reduce the global attention calculation overhead, this embodiment introduces a sparse attention strategy: only calculating the attention weights at possible matching positions, which significantly reduces the computational complexity. For example, restricting the parallax search to a candidate set with high matching confidence (that is, the range where the matching probability is greater than the preset confidence threshold) rather than the full range, so that the matching complexity can be reduced from the original to approximately (where, is the number of pixels, is the parallax search range, is the number of candidates per pixel); thereby ensuring that the algorithm can run in real time while maintaining accuracy.

[0029] A3. Cost Volume Construction and Regularization: Based on the matching results output by the Transformer, in this embodiment, a dense cost volume is explicitly constructed to fuse the matching evidence. Different from the traditional method of calculating the matching cost by traversing all disparities pixel by pixel, in this embodiment, the matching probabilities output by the attention layer are directly used to form the cost volume, and only the disparity positions with high confidence are retained, thereby reducing redundant calculations. Let represent the pixel at disparity , then the cost volume value can be calculated only at the where is relatively large, and other disparities are ignored. The initial cost volume is locally regularized and smoothed by 3D convolution to filter out noise and gross errors. During the processing, based on the local smoothing characteristics of the convolutional layers in the residual module and its feed-forward network of the Transformer architecture and the self-attention aggregation mechanism of the Transformer, long-range correlation modeling can be performed on the cost volume in the depth direction, thereby improving the accuracy and stability of disparity estimation and further optimizing the global consistency result of the cost distribution.

[0030] A4. Disparity Regression and Depth Output: After obtaining the regularized cost volume, a disparity map with sub-pixel accuracy can be obtained through Softmax regression. Specifically, for each pixel , the normalized matching distribution is calculated; then the disparity estimate is taken as the expectation , which can be interpolated to the sub-pixel level. Then, according to the internal parameters calibrated by the camera, the disparity is converted into a dense depth , for example, for a parallel binocular camera, there is a relationship . The output depth map is geometrically aligned with the real scene at the sub-pixel level, providing accurate three-dimensional information for subsequent steps. Experimental calibration shows that compared with the baseline network without using the Transformer, this module reduces the average depth error by about 20%, and shows excellent generalization performance on both open datasets and clinical data. This benefit stems from the improved Transformer in this embodiment having stronger global matching capabilities: compared with traditional algorithms assuming a fixed disparity range, the Transformer stereo matching is not limited by the preset range, can adaptively match larger depth differences and identify occluded regions, thus significantly improving the reconstruction integrity and accuracy.

[0031] II. Depth Supervision and Optimization Strategy Based on NeRF.

[0032] To further improve the reconstruction robustness and reduce the dependence on real calibration data, this embodiment integrates the Neural Radiance Field (NeRF) technology to perform self-supervised optimization on binocular depth estimation. NeRF can learn an implicit 3D scene representation from a small number of RGB images with different viewpoints and render realistic images from any new viewpoint. Taking advantage of this feature, this embodiment combines the depth predicted by the binocular network with the NeRF rendering process during the training phase, introducing a cross-view reprojection consistency constraint. The specific implementation is as follows: First, the initial depth map output by the binocular Transformer stereo reconstruction module and the color image on the left or right side are input into the NeRF model together (if the left view is used as the reference and the right view is used as the matching map, a depth map of the left view will be generated eventually; if the right view is used as the reference and the left view is used as the matching map, the algorithm translation vector parameters need to be adjusted accordingly). NeRF represents the scene as a radiance field function , where p is the spatial coordinate, is the viewing direction, and the output color and volume density are obtained. For each ray emitted by the camera, a point or local region on this viewing direction is sampled according to the predicted depth. By discretizing the volume rendering integral , the volume rendering equation of NeRF is obtained to calculate the synthesized color of this ray from another viewpoint: ; where, represents the sampling point index along the ray, is the distance between adjacent sampling points, represents the transmittance in front of the th point. Referring to the above formula, by simply replacing the color of each sampling point with its depth position and then dividing by the total weight, the depth rendering is obtained. By simply mapping the depth rendering of this ray to the pixel (i, j), that is, , and then normalizing it, the depth rendering of this pixel is obtained.

[0033] ; Then, the rendered color is compared with the real color of the corresponding pixel in the actual other viewpoint to construct a reprojection loss: . Where, represents a set of sampled ray collections (such as all pixels or randomly sampled pixels). Meanwhile, this embodiment introduces a depth consistency loss to constrain the depth Consistent with the depth deduced by NeRF through volume rendering, that is, add . The comprehensive optimization goal is to minimize . ( is the trade-off coefficient). The stereo matching network and NeRF parameters are updated simultaneously through backpropagation, enabling the two to converge synergistically. In this way, NeRF provides a self-supervised signal source for the stereo network, using the information in the image itself to optimize depth estimation. Compared with supervised training that only relies on limited real depth data, this strategy can make full use of unlabeled laparoscopic videos for training, alleviate the problem of data scarcity, and approximate the geometric solution of physical reality.

[0034] Furthermore, this embodiment adopts an iterative fusion optimization strategy: alternately optimizing the binocular Transformer stereo reconstruction network and the NeRF model. Initially, the binocular Transformer network refines the predicted depth interval by increasing the number of disparity candidates D' to obtain an accurate depth result as the training benchmark ground truth, and trains NeRF to fit this depth; then uses the new view images synthesized by NeRF to supervise the binocular Transformer stereo reconstruction network to continue training for fine matching with a smaller number of disparity candidates D; after updating the depth, train NeRF again, and cycle in this way to gradually improve the depth accuracy. Through the iterative fusion optimization strategy, the binocular Transformer stereo reconstruction network + NeRF joint optimization process of this embodiment makes the depth map of the final binocular endoscopic image mainly contributed by stereo matching in the textured area, with rich high-frequency details; while in the textureless or reflective area, it is compensated by NeRF to make the edge contour more accurate; at the same time, the binocular Transformer stereo reconstruction network can also accurately output accurate depth with a smaller number of disparity candidates D, compressing the algorithm computational complexity from O(N·D') to O(N·D), reducing the intraoperative real-time flexible registration computing power pressure and reducing the calculation time. Experimental results show that: after introducing NeRF self-supervision, the average depth error of the reconstruction is reduced from about 3.0 mm before introduction to about 2.0 mm, with a reduction amplitude of about 33%; the depth quality improvement is more significant on the light-changing or low-texture surface, for example, the depth error at the edge of small blood vessels is reduced by about 30%. At the same time, due to the addition of NeRF constraints, the network's demand for the amount of training data is reduced, and stable performance can still be maintained with only limited samples. This fusion strategy effectively improves the reconstruction accuracy and generalization ability.

[0035] ​In this embodiment, NeRF's volume rendering essentially represents pixel color as an integral along a camera ray—that is, color is the expected value of the weighted cumulative volume density along the line of sight. Pixel color is equivalent to the weighted average of the colors at each point along the ray, where the weight is proportional to the transmittance-density product at that point. The calculation of pixel depth follows the same structure as color rendering: if the "color" of each sampling point is replaced by the depth at that point, the rendering result is the expected depth for that ray. Therefore, the weight of a sampling point's contribution to the final color also reflects the probability distribution of the interaction between the ray and the object at that location. NeRF's color rendering and depth rendering are mathematically unified: both can be viewed as weighted integrals of attributes along the ray: one attribute is color, the other is depth. This provides a rigorous mathematical basis for calculating depth using the same weights as color, namely, depth is the expected position of the density distribution along the ray.

[0036] 3. Deformation registration module integrating finite element calculation and GNN.

[0037] In response to the non-rigid deformation of organs during surgery, this embodiment establishes a physics-driven nonlinear finite element (FEM) model and combines it with a graph neural network (GNN) to achieve high-speed deformation prediction, so as to align the pre-operative organ model with the intraoperative state in real time.

[0038] C1. Nonlinear finite element modeling of organs: The three-dimensional model of the organ before surgery is used as the initial reference configuration, and its coordinates are defined as , the coordinates after deformation are , then the displacement field is The deformation is expressed using Lagrangian description, and the deformation gradient tensor is defined as: , where I is the second-order unit tensor that retains the original configuration baseline. hour, ,when hour, When considering large deformation, the Green–Lagrange strain tensor is introduced: ; Its expansion includes the first-order and second-order terms of displacement, which can accurately describe the deformation under large rotation and large strain. Assuming that the organ is an isotropic nonlinear elastic body, a suitable strain energy function can be selected To characterize its elastic mechanical properties. For example, under the St.Venant-Kirchhoff model, the strain energy in the form of a quadratic polynomial is: ;in, Characterizes the resistance of a controlled material to volume changes, Characterizes the shear modulus, both are Lamé constants (both related to Young's modulus and Poisson's ratio related); tr() is the trace operator, representing the sum of all main diagonal elements; The volume energy term corresponds to the energy contribution stored in the volume change (expansion or compression) of the human organ; is the shear energy term, corresponding to the energy stored in the process of shape change (shearing, torsion, anisotropic stretching) of the human organ. According to this strain energy, the second-order Piola-Kirchhoff stress can be obtained: . Further through It is mapped to the first-order Piola-Kirchhoff stress , which is used to calculate the internal nodal forces of the finite element. Considering that most organs are approximately incompressible, in this embodiment, a volume penalty term is added to the material model or the form of decomposed strain energy is adopted to satisfy the approximate volume constraint, so as to more realistically simulate the deformation behavior of the organ under working conditions such as being squeezed. The above non-linear finite element modeling provides a physical prior for registration, making the deformation calculation conform to the mechanical laws. The deformation solution of the organ can be transformed into an energy function minimization problem: find the displacement field such that the total energy of the system reaches the minimum, that is, satisfies the equilibrium equation . The total energy includes elastic deformation energy, external force work, and registration constraint terms, etc.: ; Among them, is the volume force density (such as gravity), is the surface force, and the last term is the registration error term, represents the coordinates of the real-time reconstructed surface points (or feature points), is the position of the corresponding point on the preoperative model under the action of the displacement , is the weight parameter, represents the total number of corresponding point pairs used for model-organ registration, represents the three-dimensional volume integration domain of the organ model. By minimizing this objective function, the displacement field that best matches the reconstructed surface after simulation deformation can be solved. Since this optimization is non-linear, this embodiment can use Newton–Raphson iteration to solve and use implicit time integration to maintain the numerical stability of the solution. To meet the real-time requirement, this embodiment performs GPU parallel acceleration on the solution process and at the same time introduces an adaptive solution strategy: adaptively refine the finite element mesh according to the stress / strain distribution of the current deformation field, increase the mesh density in the high-strain region, and keep a coarser mesh in the gently deformed region to reduce the global calculation amount while ensuring local accuracy. This adaptive strategy effectively reduces the degrees of freedom of each iterative solution, thereby improving the calculation efficiency.

[0039] C2. Deformation Prediction Assisted by Graph Neural Networks: Despite the above optimizations, the real-time solution of non-linear FEM still has a relatively high computational complexity. Therefore, in this embodiment, a GNN is innovatively introduced as a surrogate model for FEM. Specifically, a large amount of offline-generated finite element simulation data (covering various organ anatomical boundaries and external forces) is used, and the difference between the current organ surface and the initial model surface is compared in combination with a dense point cloud. According to this difference, the surface displacement field is extracted and used as a constraint condition for GNN prediction. Next, a GNN model is trained to directly learn the mapping relationship from the displaced (organ anatomical) boundaries (such as the displacement or force received by the organ surface) to the internal node displacement field. This GNN uses the organ mesh nodes as graph nodes and the mesh topology as the connection of the graph, and adopts a message passing mechanism to learn the mechanical correlation between nodes. After training, when given new (organ anatomical) boundaries, the GNN model can instantaneously predict the approximate deformation result of the organ, greatly accelerating the calculation. Existing research shows that a well-trained GNN model only needs about 47 seconds to simulate the 80-second organ compression deformation, while the same process requires 15 hours for high-precision FEM simulation. However, the average error of the predicted node positions of the two can still be controlled within 1 mm. Undoubtedly, GNN can accelerate organ mechanical simulation by hundreds of times while maintaining accuracy.

[0040] In this embodiment, the deformation field predicted by GNN is used as the initial solution for FEM iteration, that is, the "GNN + FEM two-stage calculation" mode is adopted: first, GNN quickly gives an estimated displacement field, and then the FEM simulation corrects the details. The combination of the two not only ensures real-time performance but also retains physical accuracy, greatly reducing the number of Newton iteration steps and accelerating convergence. At the same time, due to the self-learning ability of GNN, the system can also gradually fine-tune the model parameters according to the actual deformation data in each operation, achieving the effect of "becoming more and more accurate with use".

[0041] C3. Flexible Organ Registration and Accuracy Evaluation: Through the above FEM physical modeling and GNN-accelerated prediction, this embodiment can deform and align the preoperative organ model to the organ surface reconstructed by the endoscope in real time during the operation. The registration error mainly comes from the inaccurate deformation prediction and the noise of the reconstructed surface.

[0042] Preferably, in this embodiment, the surface distance error is used as the registration accuracy evaluation index, that is, the average distance error between the surface of the registered virtual model and the point cloud of the real reconstructed surface is calculated. The experimental results show that the method of this embodiment can control this error at a low level under various working conditions. For example, for the simulated liver pressing deformation scenario, if only rigid registration is used, the deviation between the model surface and the real surface exceeds 10 mm, while the method of this embodiment reduces the average surface distance error to within 2 mm through FEM deformation compensation. Even in the case where there are few feature matching points in the local area, the introduced elastic regularization term ensures the smoothness and reasonableness of the deformation field, and no local excessive distortion will occur.

[0043] This embodiment also compares an elastic registration algorithm with only geometric constraints and the physical mechanics fusion method of this embodiment. When there is no external force deformation, their accuracies are comparable; but when there is external force pressing, the error of the pure geometric method increases to more than 4 mm, while the method of this embodiment still controls the error within 2.5 mm under the same conditions due to the introduction of FEM physical constraints. It shows that this embodiment is superior to the existing technical methods in terms of registration accuracy and stability. To meet the real-time requirement, the depth estimation network is deployed on an Nvidia RTX 4090 GPU, and the computational workload of the disparity estimation step is controlled within 1.5 ms by using CUDA parallel computing; the SLAM module eliminates low-information feature points to speed up the matching; the finite element solution is limited to 50 iterations per step and uses the result of the previous frame for warm start. Through the above methods, and thanks to the good initial solution provided by the GNN and GPU acceleration, the method of this embodiment can complete a complete deformation solution update within 100 ms, achieving a registration refresh rate close to 10 Hz; for smaller-range deformation updates, the calculation frame rate can be increased to more than 20 Hz, thus basically meeting the real-time requirements of laparoscopic surgery. In summary, the intraoperative flexible organ registration based on the combination of deep learning and biomechanical models ensures the high-precision alignment of the virtual model and the real organ in a complex surgical environment, providing a reliable anatomical positioning reference for doctors.

[0044] IV. SLAM Camera Tracking and Virtual-Reality Fusion Optimization.

[0045] This module aims to accurately locate the pose of the laparoscopic camera and stably fuse the virtual content into the real surgical field. Optionally, this embodiment is improved based on the ORB-SLAM2 algorithm (a classic algorithm in the existing SLAM field) to meet the special requirements of laparoscopic surgery.

[0046] D1. Visual SLAM Tracking Optimization: To address possible situations such as fast movement, going out of the field of view, and occlusion in the surgical scenario, this embodiment adds various constraints to the front - end and back - end of SLAM to improve robustness. For example, during the feature tracking stage, a motion model prediction based on the rigidity of local organs is added to reduce the loss of feature matching during fast movement; in the back - end optimization, the reconstructed dense surface point cloud is integrated as a geometric constraint and participates in the camera pose optimization together with sparse feature points, so as to maintain the non - divergence of camera positioning in texture - poor areas. The camera pose is obtained by minimizing the reprojection error: , where is the 3D point in the map, is the 2D feature observed in the current frame, is the projection function. After introducing the dense point cloud constraint, the optimization objective also includes , represents the points reconstructed in real - time, is the pre - stored organ surface model, and K is the set of all outermost points of S, which makes the camera positioning subject to anatomical structure constraints. These improvements significantly enhance the robustness of SLAM in difficult situations. Experiments show that in in - vivo and in - vitro tests, the improved SLAM in this embodiment can still maintain continuous tracking and is not prone to losing positioning when encountering fast camera movement, temporarily going out of the field of view, and partial occlusion. Compared with the original ORB - SLAM2, the tracking loss rate of this embodiment is reduced by about 50%, and the mean square error of camera pose estimation is reduced by about 20%. Since only a small number of optimization constraints are added, the algorithm processing frame rate still remains at the real - time level (above 30Hz) and can keep up with the frame rate output of laparoscopic videos.

[0047] D2. Hybrid Tracking and Multi - sensor Fusion: To further improve the robustness of tracking, the system in this embodiment supports an optical + inertial / electromagnetic hybrid tracking mode. A micro - IMU and an electromagnetic positioning (EM) sensor are installed near the laparoscopic lens, and the camera pose solved by visual SLAM is fused with the high - frequency acceleration / angular velocity measured by the IMU and the EM positioning result using Kalman filtering. The fusion process combines the advantages of vision and external sensing: vision provides high - precision relative motion estimation, the IMU provides high - frequency continuous motion compensation, and the EM provides a reference to counteract magnetic field occlusion drift, so that the finally obtained 6 - DoF camera trajectory is both smooth and accurate. Animal experiment verification shows that after enabling hybrid tracking, even if the laparoscopic lens is blocked by blood for up to 2 seconds, the system can still rely on the IMU / EM to maintain the stability of registration, and there is no obvious jump in the virtual superposition position after vision recovery. This proves that multi - sensor fusion significantly enhances the robustness of the system in extreme situations. Compared with pure vision tracking, the pose error of the fusion scheme is reduced by nearly 80% during short - term vision failure (the position drift during occlusion is significantly reduced), providing higher positioning reliability for clinical operations.

[0048] V. Mixed Reality Rendering and Visual Calibration.

[0049] After obtaining the high-precision organ deformation and camera pose, in this embodiment, the virtual model is correctly superimposed on the surgical field image through mixed reality technology. To ensure the accuracy and realism of the virtual-real fusion, this embodiment has been optimized in two aspects: online calibration alignment and realistic rendering.

[0050] E1. Online Calibration and Alignment Refinement: Initially, the preoperative virtual model is aligned with the laparoscopic coordinate system through offline calibration. However, due to equipment errors and human movements during the operation, there may be slight deviations when directly superimposing. For this reason, the system binds the preoperative virtual model to the MR Hand Tracking module, and the operator can perform preliminary manual registration with the laparoscopic field of view through gesture dragging, rotation, and scaling. Next, this embodiment adopts an online calibration optimization strategy: using the dense point cloud reconstructed by SLAM or key anatomical features to automatically and dynamically fine-tune the position and scale of the virtual model during the operation. For example, first, a series of three-dimensional key contour vertices evenly distributed along the organ edge or the fitted contour curve are extracted from the anatomical surface of the reconstructed virtual model , obtain the pose of the virtual organ model in the mixed reality coordinate system , where R is the rotation matrix of the model is the translation matrix of the three dimensions of the model, and for each three-dimensional contour point , calculate its projection on the image plane , where is the homogeneous depth of the projected point; the superscript is the transpose symbol. Taking the organ edge contour in the endoscopic image as a reference, obtain the set of real contour pixel points , for each projected point , find its nearest neighbor point in the real contour set , calculate the alignment error of this point , take the sum of the squares of the deviations of all contour points as the optimization target , by adjusting the rotation matrix and translation matrix parameters in the model pose , search for an optimal model pose in the rigid body transformation space that can minimize , so as to obtain the optimal fine-tuning transformation , to find the optimal rigid body fine-tuning transformation, apply this to the virtual model, and the projected contour of the virtual organ can be exactly coincident with the image contour of the real organ. For any vertex in the virtual model, first expand it to homogeneous coordinates ; then right-multiply with the updated transformation matrix ; then multiply on the right with the updated transformation matrix , discard the last dimension to obtain the calibrated three-dimensional coordinates . Through the above operations, the superimposition error can be reduced to the sub-pixel level. Experiments show that after online calibration, the average deviation between the virtual model anatomical structure and the endoscopic image feature points can be controlled within 0.5 pixels, reducing the projection error by about 40% compared to before calibration. This precise calibration ensures the exact coincidence of the virtual enhanced information and the real organ anatomy, providing reliable navigation guidance for the surgeon.

[0051] E2. Real-time rendering and photorealism enhancement: In terms of graphics rendering, this embodiment is optimized for medical scenarios to improve the realism of mixed reality. The physically based rendering (PBR) shading model is adopted, and the physical material properties of the organ (such as diffuse reflection coefficient , specular highlight coefficient , subsurface scattering coefficient, etc.) are used for lighting calculation to more realistically represent the texture of the organ. Ambient occlusion and global illumination are considered in the rendering to make the virtual organ consistent with the surrounding environmental lighting conditions. Specifically, for each rendered pixel, the diffuse component and the specular component are calculated simultaneously: , where describes the cosine of the angle between the incident light and the surface normal, describes the specular highlight attenuation of specular reflection ( is the reflection direction, is the viewing direction, is the specular exponent). By adjusting parameters such as to match the gloss and reflection characteristics of the laparoscopic image, and using the estimated ambient light intensity to simulate the subsurface scattering effect of the organ, the rendered virtual organ can be integrated with the surrounding real organs in terms of color and brightness. The rendering result is superimposed on the laparoscopic real-time video frame by frame and finally presented on the MR device or monitor for the surgeon to view and interact. Thanks to the acceleration of the GPU, the rendering pipeline of this system takes no more than 10 ms per frame, the overall volume rendering frame rate can reach 60 Hz, and the corresponding visual delay is less than 20 ms. In addition, combined with the above tracking and calibration optimization, the total delay of the system from capturing the image to presenting the virtual overlay is controlled at about 50 ms, which is much lower than the threshold of the delay perceptible to the human eye. This means that the virtual model can be synchronously and stably superimposed as the endoscope moves, without obvious lag or jitter, ensuring the accuracy and immersion of the MR overlay during the operation.

[0052] In summary, this embodiment constructs a complete intraoperative real-time organ registration method, which organically integrates binocular laparoscopic three-dimensional reconstruction, NeRF self-supervised optimization, FEM physical modeling, SLAM tracking, and mixed reality rendering. With the synergistic effect of multiple key technologies, the system has reached a high level in terms of depth estimation accuracy, deformation registration error, and real-time performance. For example, compared with the baseline method without these improvements, the reconstruction accuracy of this embodiment is increased by about 60%, the registration error is reduced by more than half, and the processing frame rate is increased by more than 2 times. Verified by in vitro animal experiments, the system can achieve stable and reliable virtual-real alignment in a complex and changeable laparoscopic surgical environment, with an average registration error of only 2-3 mm, meeting the clinical accuracy requirements; at the same time, it has good real-time performance, and can reach a processing speed of 20-30 Hz in a typical hardware environment, and the mixed reality rendering delay can be controlled within dozens of milliseconds, without interfering with the surgical process. Theoretical analysis and experimental results both show that the method of this embodiment has significant advantages in the field of organ navigation registration, can provide doctors with intuitive and accurate three-dimensional anatomical positioning and decision-making support, and improve the safety and precision of surgery.

[0053] In summary, the method for model and organ registration for auxiliary navigation disclosed in this embodiment, as Figure 1 shown, its core content mainly includes: Step S1: Convert the disparity of the color images on the left and right sides of the binocular laparoscope into depth information to obtain an initial depth map.

[0054] The color images on the left and right sides first pass through a multi-scale CNN (Convolutional Neural Networks) to extract a feature pyramid, and fuse the image features at different scales to obtain a multi-channel feature representation of the left and right eyes. A Transformer architecture based on epipolar constraint is used to execute the self-attention and cross-attention mechanisms. The self-attention mechanism calculates the pixel-level correlation weights along the epipolar direction in the feature sequence of the same view. After the cross-attention mechanism performs feature matching along the epipolar direction on the color images on the left and right sides, only the disparity candidates with the pixel matching probability meeting the preset confidence threshold are retained to construct a sparse probability disparity cost volume. Then, each disparity cost volume is subjected to regularization smoothing and converted into a normalized probability distribution, and then the expectation of this probability distribution is calculated as the disparity estimate value.

[0055] Step S2: Perform self-supervised optimization on the binocular depth estimation based on the trained NeRF model. The training process of the NeRF model specifically includes: Step S21: Use the initial depth map as a depth prior and input it together with the color image corresponding to the left or right unilateral into the NeRF model. The NeRF model synthesizes a new image from a preset perspective through volume rendering technology, and compares the synthesized new image with the input left or right unilateral color image to calculate the reprojection error.

[0056] Step S22: Process pixel depth and color rendering in the same rendering manner. Replace the color of each sampling point with the corresponding depth position, then divide by the total weight to obtain the depth rendering, and then map the calculated values of each depth rendering to the corresponding pixels to obtain the depth rendering values of each pixel. Then calculate the depth consistency loss between the depth rendering value inferred by the rendering and the disparity estimate value.

[0057] Step S23: Adopt a strategy of joint and alternating training of the NeRF model and the Transformer architecture to gradually optimize the depth estimation result. Iteratively run to fix the NeRF model to update the Transformer architecture to reduce the pixel color reprojection error and depth consistency error, and fix the Transformer architecture to update the NeRF model to improve the fitting accuracy of the laparoscopic view geometric consistency, ultimately minimizing the weighted sum of the reprojection error and the depth consistency loss.

[0058] Step S3: Perform three-dimensional matching based on the two-dimensional information observed in the left or right unilateral color image in combination with the depth information of each pixel and the shape and position of the virtual organ model to align the virtual organ model with the current morphology of the real organ.

[0059] Preferably, in step S1, during the process of calculating the sparse probability disparity cost volume that meets the confidence threshold according to the matching probability between pixels, a sparse attention strategy is introduced to limit the disparity search within the range of matching probabilities greater than the confidence threshold.

[0060] Preferably, in step S3, it specifically includes: Step S31: Combine the physical simulation ability of finite element analysis with the machine learning ability of graph neural networks to achieve precise registration of organ non-rigid deformation.

[0061] Step S32: Use the visual SLAM algorithm to estimate the pose of the laparoscopic camera in real time.

[0062] Step S33: After obtaining the organ deformation and camera pose, superimpose the virtual organ model onto the surgical field image through mixed reality technology.

[0063] Preferably, step S31 specifically includes: Step S311: Based on the biomechanical properties of the organ and its boundaries (usually the anatomically distinguishable boundaries visible to the naked eye), the finite element model calculates the deformation distribution of each part of the organ under the action of external forces, providing physically realistic deformation results; Step S312: The organ is discretely represented as a graph structure, and the mesh node displacements and topological connections obtained from the finite element calculation are input into the GNN model; Step S313: The GNN model uses the organ mesh nodes as graph nodes and the mesh topology as the connections of the graph. By learning the correlations and deformation patterns between nodes, it predicts the real-time deformation of the organ. The correlations include the mechanical correlations between nodes learned using the message passing mechanism, and the mechanical correlations include the mapping relationship from the boundary formed by the displacements or forces received on the organ surface to the internal node displacement field.

[0064] Preferably, step S32 specifically includes: Step S321: The SLAM algorithm obtains the direction and position trajectory of the camera in the surgical environment by minimizing the reprojection error of the feature points extracted and matched frame by frame in the color image of the left or right unilateral, and simultaneously constructs a dense point cloud of the surgical area to constrain the camera positioning by the anatomical structure.

[0065] Step S322: The virtual organ model is converted to the camera coordinate system for alignment and rendering according to the camera positioning information. Among them, the virtual organ model is usually obtained by preoperative CT and MRI reconstruction.

[0066] Step S323: By minimizing the alignment error between the virtual image displayed by the MR device and the real picture, the attitude and position of the virtual organ model are adjusted in real time, so as to ensure the spatial alignment of the virtual organ model and the actual endoscopic color image of the left or right unilateral. The specific implementation method of this step can be: calibrate the MR display coordinate system and the surgical coordinate system in advance by the optical positioning method to initially align the virtual model projection with the real image; then use the iterative closest point (ICP) algorithm to minimize the alignment error between the virtual organ model and the laparoscopic color image with pixel depth information during the superposition and alignment process of the MR device, and update the rigid body transformation of the virtual model for each frame, so as to ensure the spatial alignment of the virtual organ model and the actual endoscopic color image of the left or right unilateral.

[0067] Preferably, the method of the present invention further includes: fusing the camera pose solved by the visual SLAM with the high-frequency acceleration, angular velocity measured by the IMU and the EM positioning result using the Kalman filter, where the IMU and EM are different sensors deployed on the laparoscopic lens.

[0068] Preferably, step S33 specifically includes: Step S331: Obtain the internal and external parameters of laparoscopic camera calibration. The internal parameters include focal length, principal point, and distortion coefficients, and the external parameters include the pose in the surgical coordinate system. Set the parameters of the virtual camera according to the calibration results of the internal and external parameters to make the viewing angle of the rendered virtual model consistent with that of the real laparoscopic camera.

[0069] Step S332: Transform the virtual organ model into the surgical coordinate system, and adjust the size and position of the virtual organ model according to the actual anatomical ratio to ensure that the virtual organ model is spatially aligned with the real organ.

[0070] Step S333: During the rendering process, perform lens lighting matching, correct the distortion of the virtual organ image according to the lens characteristics of the laparoscope, and simulate the lighting effect of the surgical light source on the virtual organ model. And use the dense point cloud or key anatomical features reconstructed by SLAM to dynamically adjust the position and scale of the virtual organ model during the operation.

[0071] It should be noted that: in the prior art, there are some overlapping keywords with the above steps. For example, in the method for establishing a parallax prediction model and depth estimation method of endoscopic images disclosed in the domestic patent application document CN113435573A, there is content related to the association between epipolar lines and parallax depth. In the method for generating new perspective images with large parallax based on NERF disclosed in the patent application document CN118037915A, new perspective images with large parallax are generated based on the predicted depth information. However, for such prior arts, there are many essential differences in the causal relationships between the overlapping keywords and the scenario of this embodiment. Specifically: The epipolar lines mentioned in the patent application document CN118037915A are only used to screen the data set to ensure that the constructed data set is a data set corrected by epipolar lines, which is essentially different from the present invention that performs feature matching of left and right images based on epipolar lines, and then screens parallax candidates according to the matching results to construct the intermediate variable of sparse probability parallax cost volume. Although both are used for parallax calculation, the present invention uses probability volume and threshold screening to reduce the search space, while the patent application document CN118037915A directly calculates the convolutional cost within the full parallax range without using a similar discretized parallax cost volume structure, and the technical implementation methods and optimization focuses are different.

[0072] Similarly, there is an essential difference in motivation between the patent application document CN113435573A, which generates new perspective images with large disparities based on predicted depth information, and this embodiment, which optimizes the initial depth information in a self-supervised manner based on synthesized new images (in other words, although both mention volume rendering, the volume rendering in the present invention is used for view synthesis to assist depth optimization, while the volume rendering in CN118037915A is used to directly generate new perspective images with large disparities, and their functions and contexts are different, so the application scenarios and technical effects are not equivalent); in addition, the attention mechanism in the patent application document CN113435573A is used for stereo matching under multi-perspective information fusion, which is different from the matching strategy of the self-attention in the same view feature sequence and cross-attention for matching features between the left and right views along the epipolar direction in the present invention, so the technical effects are also inconsistent.

[0073] In other words, in this embodiment, the self-attention and cross-attention along the epipolar direction are used to filter out low-confidence matches and convert the matching probability into the expected disparity; then, the NeRF model is used to perform self-supervised optimization on the depth map; finally, three-dimensional matching is performed by combining the shape and position information of the virtual organ model; the core purpose it focuses on is to achieve high-precision real-time navigation registration. Among them, the Transformer matching architecture combined with confidence screening can effectively filter out incorrect matches and improve the accuracy of the initial depth map; the self-supervised optimization based on NeRF further makes the depth estimation more consistent with the real view geometry, enhancing the accuracy and stability of the navigation result. Although some existing technologies are meaningful for the implementation of each discrete technical point of the present invention, the whole of them cannot provide relevant technical inspiration for those skilled in the art on how to convert and apply them to the technology of providing navigation registration for models and organs in clinical applications such as surgery.

[0074] Embodiment 2 This embodiment discloses a system for model-organ registration to assist navigation, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method of the above embodiment is implemented.

[0075] In summary, the methods and systems respectively disclosed in the embodiments of the present invention have at least the following beneficial effects: 1. Introducing Transformer into stereo matching realizes a new epipolar matching mechanism with global context awareness. Traditional stereo matching is usually limited by local correlation, while the Transformer architecture based on the present invention can establish pixel correspondence relationships globally, significantly improving the matching accuracy. At the same time, after feature matching is performed along the epipolar direction on the left and right color images, only the disparity candidates whose matching probabilities between pixels meet the preset confidence threshold are retained to construct a sparse probability disparity cost volume. Then, after performing regularization smoothing on each disparity cost volume, it is converted into a normalized probability distribution, and then the expectation of this probability distribution is calculated as the disparity estimate value, significantly improving the real-time performance of disparity calculation.

[0076] 2. Introducing NeRF as a depth supervision signal converts depth optimization into a new differentiable view synthesis problem, and optimizes the depth result from the perspective of global photometric consistency. Under the global supervision of NeRF, the deficiencies of traditional stereo matching in low-texture or high-gloss regions are effectively compensated, making the depth estimation in difficult regions equally accurate. At the same time, the joint optimization of depth consistency constraints and the NeRF-Transformer stereo network ensures that the depth map not only satisfies the photo consistency of reprojection but also maintains geometric accuracy. The innovative strategy of fusing neural rendering and stereo vision greatly improves the details and reliability of depth estimation, achieving higher accuracy.

[0077] 3. Three-dimensional matching is performed according to the two-dimensional information observed in the left or right single-sided color image in combination with the depth information of each pixel and the shape and position of the virtual organ model, ensuring the accuracy of aligning the virtual organ model with the current morphology of the real organ.

[0078] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for model and organ registration to assist navigation, characterized in that Including: Step S1: Convert the disparity of the color images on the left and right sides of the binocular laparoscope into depth information to obtain an initial depth map. Specifically, it includes: The color images on the left and right sides first pass through a multi-scale CNN to extract a feature pyramid, and the image features at different scales are fused to obtain a multi-channel feature representation of the left and right eyes. A Transformer architecture based on epipolar constraint is used to execute self-attention and cross-attention mechanisms. The self-attention mechanism calculates pixel-level correlation weights along the epipolar direction in the feature sequence of the same view. After the cross-attention mechanism performs feature matching along the epipolar direction between the color images on the left and right sides, only the disparity candidates whose pixel matching probabilities meet the preset confidence threshold are retained to construct a sparse probability disparity cost volume. Then, each disparity cost volume is subjected to regularization smoothing and converted into a normalized probability distribution, and then the expectation of this probability distribution is calculated as the disparity estimate value; Step S3: Based on the trained NeRF model, perform self-supervised optimization on binocular depth estimation. The training process of the NeRF model specifically includes: Step S21: Input the initial depth map as a depth prior and the corresponding color image of the left or right single side into the NeRF model. The NeRF model synthesizes a new image from a preset perspective through volume rendering technology, and compares the synthesized new image with the input color image of the left or right single side to calculate the reprojection error; Step S22: Process pixel depth and color rendering in the same rendering manner. Replace the color of each sampling point with the corresponding depth position, and then divide by the total weight to obtain depth rendering. Then, map the calculated values of each depth rendering to the corresponding pixels to obtain the depth rendering values of each pixel. Then calculate the depth consistency loss between the depth rendering values inferred by the rendering and the disparity estimate value; Step S23: Adopt a strategy of jointly and alternately training the NeRF model and the Transformer architecture to gradually optimize the depth estimation result. Iteratively run to fix the NeRF model to update the Transformer architecture to reduce the pixel color reprojection error and depth consistency error, and fix the Transformer architecture to update the NeRF model to improve the fitting accuracy of the geometric consistency of the laparoscope view, and finally minimize the weighted sum of the reprojection error and the depth consistency loss; Step S3: Perform three-dimensional matching based on the two-dimensional information observed in the color image of the left or right single side, combined with the depth information of each pixel and the shape and position of the virtual organ model, so that the virtual organ model is aligned with the current morphology of the real organ.

2. The method for model and organ registration to assist navigation according to claim 1, characterized in that, In step S1, during the process of calculating the sparse probability disparity cost volume that meets the confidence threshold according to the matching probability between pixels, a sparse attention strategy is introduced to limit the disparity search within the range of matching probabilities greater than the confidence threshold.

3. The method for model and organ registration to assist navigation according to claim 1, wherein In step S3, it specifically includes: Step S31: Combine the physical simulation ability of finite element analysis with the machine learning ability of graph neural networks to achieve precise registration of organ non-rigid deformation; Step S32: Use the visual SLAM algorithm to estimate the pose of the laparoscope camera in real time; Step S33: After obtaining the organ deformation and the camera pose, the virtual organ model is superimposed on the surgical field image through mixed reality technology.

4. The method for model and organ registration to assist navigation according to claim 3, characterized in that, The specific steps of step S31 include: Step S311: The finite element model calculates the deformation distribution of each part of the organ under the action of external forces based on the biomechanical characteristics and boundaries of the organ, and provides physically realistic deformation results. Step S312: The organ is discretely represented as a graph structure, and the grid node displacements and topological connections obtained from the finite element calculation are input into the GNN model. Step S313: The GNN model uses the organ grid nodes as graph nodes and the grid topology as the connections of the graph, and predicts the real-time deformation of the organ by learning the correlations and deformation patterns between the nodes. The correlations include the mechanical correlations between the nodes learned by using the message passing mechanism, and the mechanical correlations include the mapping relationship from the boundary formed by the displacements or forces received on the organ surface to the internal node displacement field.

5. The method for model and organ registration to assist navigation according to claim 3, characterized in that The specific steps of step S32 include: Step S321: The SLAM algorithm obtains the direction and position trajectory of the camera in the surgical environment by minimizing the reprojection error of the feature points extracted and matched frame by frame in the left or right unilateral color image, and simultaneously constructs a dense point cloud of the surgical area to constrain the camera positioning by the anatomical structure. Step S322: The virtual organ model is converted to the camera coordinate system for alignment rendering according to the camera positioning information. Step S323: By minimizing the alignment error between the virtual image displayed by the MR device and the real picture, the attitude and position of the virtual organ model are adjusted in real time, so as to ensure that the virtual organ model is spatially aligned with the actual endoscopic left or right unilateral color image.

6. The method for model and organ registration to assist navigation according to claim 5, characterized in that, It also includes: Using the Kalman filter to fuse the camera pose solved by the visual SLAM with the high-frequency acceleration, angular velocity measured by the IMU and the EM positioning result, where the IMU and EM are different sensors deployed on the laparoscopic lens.

7. The method for model and organ registration to assist navigation according to claim 6, characterized in that, The specific steps of step S33 include: Step S331: Obtain the internal parameters and external parameters of the laparoscopic camera calibration. The internal parameters include the focal length, principal point, and distortion coefficient, and the external parameters include the pose in the surgical coordinate system. Set the parameters of the virtual camera according to the calibration results of the internal parameters and external parameters, so that the perspective of the rendered virtual model is consistent with the real laparoscopic camera. Step S332: Transform the virtual organ model to the surgical coordinate system, and adjust the size and position of the virtual organ model according to the actual anatomical ratio to ensure that the virtual organ model is spatially aligned with the real organ. Step S333: During the rendering process, perform lens lighting matching, correct the distortion of the virtual organ image according to the lens characteristics of the laparoscope, and simulate the lighting effect of the surgical light source on the virtual organ model. And use the dense point cloud reconstructed by the SLAM or the key anatomical features to dynamically adjust the position and scale of the virtual organ model during the operation.

8. The method for model and organ registration to assist navigation according to claim 7, wherein The process of dynamically adjusting the position and scale of the virtual organ model during the operation specifically includes: Step S3331: Extract a series of three-dimensional key contour vertices evenly distributed along the organ edge or the fitted contour curve from the anatomical surface of the reconstructed virtual model , For the contour vertex index, obtain the pose of the virtual organ model in the mixed reality coordinate system , where is the rotation matrix of the model, is the three-dimensional translation matrix of the model. For each three-dimensional contour point , calculate the projection of each contour point on the image plane , where is the homogeneous depth of the projected point, is the two-dimensional coordinate of the contour vertex on the image plane; the superscript is the transpose symbol; Step S3332: Using the organ edge contour in the actual image as a reference, obtain the set of real contour pixel points For each projection point , find the nearest neighbor point of each projection point in the real contour set . Calculate the alignment error of this nearest neighbor point, take the minimization of the sum of squared deviations of all contour points as the optimization goal, and find the optimal rigid fine-tuning transformation 9. A system for model and organ registration to assist navigation, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 8 above.

Images