A method for three-dimensional reconstruction and navigation of endoscopic cavity
By reconstructing three-dimensional Gaussian units and evaluating navigation reliability fields, the adaptation problem of endoscopy navigation in the absence of external truth values was solved, achieving accurate navigation and improved stability, thus meeting the clinical needs of endoscopy navigation.
Patent Information
- Application Number
- CN202611174506.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-04
- Publication Date
- 2026-11-17
AI Technical Summary
Existing endoscopy navigation technology is difficult to adapt to real patient cavities without external ground truth data, has poor interpretability of navigation errors, and is prone to reduced navigation accuracy during repositioning operations without reliable gating constraints.
A three-dimensional Gaussian cell reconstruction method is adopted. The transformation matrix is calculated through the image-depth joint loss function to construct a navigation reliability field and evaluate the navigation accuracy. The parameters of the three-dimensional Gaussian cell are iteratively optimized, and the longitudinal arc length and radial offset error are calculated to achieve endoscope navigation without external truth values.
It improves the clinical adaptability and operational stability of endoscopy navigation, enables precise navigation in the absence of external truth values, reduces the decrease in navigation accuracy caused by frequent repositioning, and makes errors easy to interpret clinically.
Smart Images

Figure CN123118900A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of medical robotics and computer vision technology, and in particular to a method for three-dimensional reconstruction and navigation of endoscopic cavity. Background Technology
[0002] Minimally invasive endoscopic surgery of the digestive tract is a core method for the examination and minimally invasive treatment of gastrointestinal lesions. When performing diagnostic and therapeutic surgeries using natural orifice endoscopes such as gastroscopes and colonoscopes, physicians need to obtain real-time three-dimensional structures of the cavities and accurately position the endoscope tip within the lumen to support lesion marking, planning of the insertion and withdrawal path, and intraoperative navigation assistance. The positioning accuracy and operational stability of the endoscopic navigation system directly affect the efficiency and safety of the surgery. Clinically, there is a high demand for endoscopic three-dimensional reconstruction navigation technology that can be deployed in real time and adapted to the actual surgical environment.
[0003] Currently, endoscopy navigation technology generally uses preoperative CT / MRI image registration or simulation datasets with real trajectory annotations for drift correction, and calculates the absolute trajectory error in three-dimensional Euclidean space.
[0004] However, existing technologies are highly dependent on external truth data. Without external truth data, it is difficult to implement them directly in the cavities of real patients. When calculating the absolute trajectory error in three-dimensional Euclidean space, it is impossible to distinguish between the longitudinal deviation caused by the endoscope moving along the center line of the cavity and the radial deviation caused by the endoscope deviating from the lumen wall, resulting in poor clinical interpretability. In addition, the navigation error will increase when the pose is forcibly corrected without reliable gating constraints. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a three-dimensional reconstruction and navigation method for endoscopes, which solves the problems of insufficient adaptability to clinical scenarios without true values, poor clinical interpretability of navigation errors, and easy reduction of navigation accuracy during repositioning operations without reliable gating constraints.
[0006] The technical means employed in this invention are as follows: In a first aspect, the present invention provides a method for three-dimensional reconstruction and navigation of endoscopic cavity, comprising: Obtain the definition parameters of the 3D Gaussian element; Based on the defined parameters, render the current frame's color image and depth image to generate a rendered image and rendering depth. Based on the rendered image and the rendered depth, an image-depth joint loss function is constructed, and based on the image-depth joint loss function, the transformation matrix from the world coordinate system to the camera coordinate system is calculated; Calculate the center coordinates of the endoscope camera in the current frame based on the transformation matrix, and determine whether the current frame meets the keyframe triggering condition based on the center coordinates. If the keyframe triggering condition is met, perform the addition and deletion of three-dimensional Gaussian units, and iteratively optimize the definition parameters of the three-dimensional Gaussian units until the first iteration termination condition is met. Then, output a three-dimensional Gaussian map based on the iteratively optimized three-dimensional Gaussian units. Construct a navigation reliability field and determine whether the value of the navigation reliability field is lower than a preset threshold; If the value of the navigation reliability field is not lower than the preset threshold, a cavity centerline is constructed, the estimated coordinates of the current camera center are projected onto the cavity centerline, and the normalized arc length and radial offset are calculated. A longitudinal arc length error is constructed based on the normalized arc length, and a radial offset error is constructed based on the radial offset. The cavity centerline target point registration error is calculated based on the longitudinal arc length error and the radial offset error. Navigation accuracy is evaluated based on the cavity centerline target point registration error. If the navigation reliability field is lower than the preset threshold, the transformation matrix is iteratively updated until the second iteration termination condition is met, and the endoscopic camera pose is updated based on the iteratively updated transformation matrix.
[0007] Furthermore, the definition parameters of the three-dimensional Gaussian unit include at least one of the following: the center position coordinates of the three-dimensional Gaussian unit, logarithmic opacity, logarithmic scale, rotation quaternion, and color feature.
[0008] Furthermore, the construction of the navigation reliability field includes: Calculate the map support score based on the center coordinates and neighborhood radius of the endoscope camera; Calculate trajectory curvature risk score based on endoscopic camera center trajectory sequence; Motion consistency score is calculated based on the center coordinates of the endoscope camera and the amount of trajectory drift within the window. A depth consistency score is constructed based on the rendering depth. A navigation reliability field is constructed based on the map support score, the trajectory curvature risk score, the motion consistency score, and the depth consistency score.
[0009] Furthermore, the preset threshold is 0.35.
[0010] Further, if the navigation reliability field is lower than the preset threshold, iteratively updating the transformation matrix until the second iteration termination condition is met, and updating the endoscopic camera pose based on the iteratively updated transformation matrix, includes: An effective depth pixel mask is generated based on the effective depth region within the image and the endoscope contour region. The mask region is determined based on the effective depth pixel mask, and the average alignment loss function within the mask region is constructed. With minimizing the average alignment loss function as the optimization objective, the transformation matrix is iteratively optimized until the second iteration termination condition is met; Obtain the optimal average alignment loss after iteration, and calculate the decrease in the optimal average alignment loss compared to the original average alignment loss before iteration; Determine whether the decrease exceeds a preset percentage threshold; If the decrease exceeds a preset proportional threshold, the endoscopic camera pose is updated based on the iteratively optimized transformation matrix. If the decrease does not exceed a preset percentage threshold, the original endoscopic camera pose is retained.
[0011] Furthermore, the formula for calculating the average alignment loss function is as follows:
[0012] in, Let M represent the average alignment loss function for frame t; M represents the set of effective pixel masks. Indicates the total number of valid pixels within the mask; This represents the rendering depth of pixel coordinate u in frame t; This represents the depth image of the current frame.
[0013] Furthermore, the image-depth joint loss function The calculation formula is:
[0014] in, Represents the image loss weight coefficients; Indicates the rendered image; Indicates the current frame's color image; Indicates the depth loss weighting coefficient; Indicates rendering depth; Represents the depth image of the current frame; This represents the effective depth pixel mask for frame t. Represents the L1 norm; This represents the Hadama pixel-wise multiplication operator.
[0015] Secondly, the present invention also provides a three-dimensional reconstruction and navigation device for endoscopic cavity, comprising: The Gaussian parameter acquisition module is used to obtain the definition parameters of a three-dimensional Gaussian element; The rendering generation module is used to render the current frame's color image and depth image based on the defined parameters, and generate a rendered image and rendering depth. The transformation matrix solving module is used to construct an image-depth joint loss function based on the rendered image and the rendered depth, and to calculate the transformation matrix from the world coordinate system to the camera coordinate system based on the image-depth joint loss function; The keyframe determination module is used to calculate the center coordinates of the endoscope camera in the current frame based on the transformation matrix, and determine whether the current frame meets the keyframe triggering conditions based on the center coordinates. The map optimization module is used to perform 3D Gaussian cell addition and deletion operations if the keyframe triggering condition is met, and iteratively optimize the definition parameters of the 3D Gaussian cells until the first iteration termination condition is met, and output a 3D Gaussian map based on the iteratively optimized 3D Gaussian cells. A reliability field construction module is used to construct a navigation reliability field and determine whether the value of the navigation reliability field is lower than a preset threshold. The navigation accuracy assessment module is used to construct a cavity centerline if the value of the navigation reliability field is not lower than the preset threshold, project the estimated coordinates of the current camera center onto the cavity centerline, calculate the normalized arc length and radial offset, construct a longitudinal arc length error based on the normalized arc length, construct a radial offset error based on the radial offset, calculate the cavity centerline target point registration error based on the longitudinal arc length error and the radial offset error, and perform navigation accuracy assessment based on the cavity centerline target point registration error. The pose correction module is used to iteratively update the transformation matrix if the navigation reliability field is lower than the preset threshold, until the second iteration termination condition is met, and update the pose of the endoscope camera based on the iteratively updated transformation matrix.
[0016] Thirdly, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the endoscopic three-dimensional reconstruction and navigation method for cavity endoscopic arteries as described in any one of the first aspects.
[0017] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the three-dimensional reconstruction and navigation method for cavity endoscopy as described in any one of the first aspects.
[0018] Compared with the prior art, the present invention has the following advantages: This invention utilizes 3D Gaussian units to reconstruct the cavity scene, solves the transformation matrix from the world coordinate system to the camera coordinate system through an image-depth joint loss function, and optimizes the 3D Gaussian unit parameters through keyframe iteration to output an accurate 3D Gaussian map. By setting a judgment threshold through a navigation reliability field, repositioning is performed to update the pose when the value is lower than a preset threshold. At the same time, navigation accuracy is evaluated by calculating the longitudinal arc length error and radial offset error through the lumen centerline. It can adapt to the intraoperative clinical scenario without external ground truth. The two-dimensional error is easy to interpret clinically, and the reliability threshold can avoid the decrease in navigation accuracy caused by frequent repositioning, effectively improving the clinical adaptability and operational stability of endoscopic navigation. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating a three-dimensional reconstruction and navigation method for cavity endoscopes according to an embodiment of the present invention; Figure 2 This is a schematic diagram of four-source fusion of NRF navigation reliability field in an embodiment of the present invention; Figure 3 This is a schematic diagram of LC-TRE error decomposition of the lumen centerline in an embodiment of the present invention; Figure 4 This is a timing diagram of depth-gated NDAR relocation without truth value in an embodiment of the present invention. Detailed Implementation
[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0023] The embodiments of the present invention will now be described with reference to the accompanying drawings.
[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating a three-dimensional reconstruction and navigation method for cavity endoscopes according to an embodiment of the present invention.
[0025] This invention employs an endoscope equipped with an RGB-D sensor to acquire intracavitary color and depth images at a fixed frame rate, and uses an NVIDIA Thor edge computing device to run the 3DGS-SLAM algorithm in real time to complete synchronous localization of the endoscope and dense three-dimensional mapping of the cavity.
[0026] This invention provides a method for three-dimensional reconstruction and navigation of endoscopes, comprising the following steps: Step 101: Obtain the definition parameters of the three-dimensional Gaussian element to provide a data foundation for subsequent steps.
[0027] In some embodiments, the definition parameters of the three-dimensional Gaussian element include the center position coordinates of the three-dimensional Gaussian element. Logarithmic opacity Logarithmic scale Rotation Quaternions and color characteristics At least one of them.
[0028] Step 102: Render the current frame's color image and depth image based on the defined parameters to generate a rendered image and a rendered depth image. Rendering operations on the current frame's color image and depth image are performed using the defined parameters of a 3D Gaussian unit to generate a rendered image and a rendered depth image suitable for the observed image features, providing the image data foundation for solving the image-depth joint tracking loss.
[0029] Specifically, logarithmic opacity Opacity is obtained by mapping using an activation function. :
[0030] in, This represents the Sigmoid activation function; Represents the natural constant.
[0031] Logarithmic scale The length of the three-dimensional semi-axis is obtained by taking the exponent. Rotate quaternions Convert to rotation matrix The covariance matrix is obtained as follows:
[0032] in, This represents the operation of generating diagonal matrices.
[0033] Pixel coordinates The corresponding two-dimensional Gaussian projection response is The cumulative transmittance is The formula for calculating intracavitary color image rendering is:
[0034] in, This represents the pixel value of the rendered color image at pixel coordinate u; This represents the color feature of the j-th three-dimensional Gaussian unit; This represents the mixing weight of the j-th 3D Gaussian unit at pixel coordinate u; This represents the cumulative transmittance of the j-th three-dimensional Gaussian element; This represents the opacity of the k-th 3D Gaussian unit.
[0035] The formula for calculating intracavity depth image rendering is:
[0036] in, This represents the pixel value of the rendered depth image at pixel coordinate u; Indicates the first The depth value from the center of a 3D Gaussian unit along the camera's line of sight to the plane.
[0037] Step 103: Construct an image-depth joint loss function based on the rendered image and rendering depth, and calculate the transformation matrix from the world coordinate system to the camera coordinate system based on the image-depth joint loss function. Constrain the rendered image and rendering depth through the image-depth joint loss function, and calculate the pixel deviation between the color observation image and the depth observation image respectively, so as to solve the transformation matrix from the world coordinate system to the camera coordinate system that adapts to the cavity scene of the current frame, and obtain the transformation matrix that matches the real endoscope pose.
[0038] Specifically, the SLAM system performs tracking operations frame by frame. Under the constraint of a fixed or locally updated Gaussian map, it calculates the transformation matrix from the world coordinate system to the camera coordinate system by minimizing the photometric-depth joint loss function. In some embodiments, the image-depth joint loss function The calculation formula is:
[0039] in, Represents the image loss weight coefficients; Indicates the rendered image; Indicates the current frame's color image; Indicates the depth loss weighting coefficient; Indicates rendering depth; Represents the depth image of the current frame; This represents the effective depth pixel mask for frame t. Represents the L1 norm; This represents the Hadama pixel-wise multiplication operator.
[0040] Step 104: Calculate the center coordinates of the intraocular camera in the current frame based on the transformation matrix, and determine whether the current frame meets the keyframe triggering conditions based on the center coordinates.
[0041] Keyframe triggering conditions can include two types of judgment rules: one is the displacement judgment rule, which calculates the spatial Euclidean distance between the center coordinates of the endoscope camera in the current frame and the center coordinates of the endoscope camera in the previous stored keyframe. When the distance is greater than a preset displacement threshold, the current frame is determined to meet the keyframe triggering condition. The other is the rotation judgment rule, which solves the camera rotation angle between the two frames based on the transformation matrix from the world coordinate system to the camera coordinate system corresponding to the current frame and the previous stored keyframe. When the rotation angle is greater than a preset angle threshold, the current frame is determined to meet the keyframe triggering condition. The keyframe update operation can be triggered if either of the two judgment rules is met.
[0042] Center coordinates of the endoscope camera The calculation formula is:
[0043] in, This represents the transpose of the rotation matrix; This represents the translation vector.
[0044] Step 105: If the keyframe triggering condition is met, perform the addition and deletion of three-dimensional Gaussian cells, and iteratively optimize the definition parameters of the three-dimensional Gaussian cells until the first iteration termination condition is met. Output a three-dimensional Gaussian map based on the iteratively optimized three-dimensional Gaussian cells.
[0045] If the current frame meets the keyframe triggering condition, first, add and delete 3D Gaussian units. Add 3D Gaussian units to the currently observed blank areas of the cavity, and remove 3D Gaussian units with low rendering contribution and high spatial redundancy. Then, iteratively optimize the definition parameters corresponding to all remaining 3D Gaussian units. The iteration process continues until the first iteration termination condition is met. After the iteration process is completed, a complete 3D Gaussian map is output based on the combination of all 3D Gaussian units with optimized parameters. This can reduce the rendering deviation between the 3D Gaussian map and the real cavity shape, and improve the ability of the 3D Gaussian map to restore cavity details.
[0046] The first iteration termination condition can be reaching a preset number of iterations or reaching a preset convergence condition. The number of parameter optimization iterations completed is counted, and the first iteration termination condition is satisfied when the cumulative number of iterations reaches a preset iteration threshold; alternatively, the image-depth joint loss difference between two adjacent iterations is calculated, and the first iteration termination condition is satisfied when the loss difference is less than a preset loss convergence threshold.
[0047] Step 106: Construct a navigation reliability field and determine whether the value of the navigation reliability field is lower than a preset threshold; in some embodiments, the preset threshold is 0.35. Specifically, extract the 3D Gaussian map, camera pose, and observation depth map corresponding to the current frame, and calculate the map support score, trajectory curvature risk score, motion consistency score, and depth consistency score respectively. Perform a weighted logarithmic fusion operation on the four scores to construct the navigation reliability field; compare the calculated value of the navigation reliability field with the preset threshold to determine whether the value of the navigation reliability field is lower than the preset threshold; comprehensively measure the stability of the endoscope navigation state in the current frame through the navigation reliability field and compare it with the preset threshold to divide the navigation reliable interval and the navigation failure interval, thereby reducing the interference of unnecessary repositioning correction operations on the endoscope camera pose.
[0048] In some embodiments, constructing a navigation reliability field includes the following steps: Step 1061: Calculate the map support score based on the center coordinates and neighborhood radius of the endoscope camera.
[0049] Specifically, using the center coordinates of the endoscopic camera Using r as the center and r as the neighborhood radius, calculate the map support score for a weighted 3D Gaussian map within the range. :
[0050] in, Represents the set of effective three-dimensional Gaussian elements in the neighborhood; This represents the natural index operator. The normalized map support score characterizes the sufficiency of support for the geometric and textural features of the current endoscopic camera map.
[0051] Step 1062: Calculate the trajectory curvature risk score based on the endoscopic camera center trajectory sequence.
[0052] Specifically, based on the center trajectory sequence of the endoscopic camera Calculate discrete curvature Based on discrete curvature Calculate the trajectory curvature risk score The calculation formula is:
[0053] in, Indicates the displacement from the previous frame; Indicates the displacement in the next frame; Represents the inverse cosine operation function; Indicates the median of the curvature sequence; Represents a minimal constant. Dot product. The angle between the two ends is measured; the higher the trajectory curvature risk score, the more likely the endoscope is in the cavity bending area, and the higher the risk of positioning failure.
[0054] Step 1063: Calculate the motion consistency score based on the center coordinates of the endoscope camera and the amount of trajectory drift within the window.
[0055] Specifically, set the length of the sliding window. The current image frame number t, the total displacement of the endoscope camera in the comparison sliding window, the cumulative displacement of each frame, and the trajectory drift within the window. To calculate the motion consistency score.
[0056] The formula for calculating the motion consistency score is:
[0057] in, , , , Represents the t-th frame, the (k+1)-th frame, the k-th frame, and the... The coordinates of the center position of the endoscopic camera in the frame; This represents the median of all drift values in the sliding window sequence. This represents a very small constant.
[0058] Step 1064: Construct a depth consistency score based on the rendering depth.
[0059] Specifically, within the effective depth pixel mask area, the depth consistency score is calculated. :
[0060] in, This represents the root mean square error between the rendered depth and the measured depth of frame t. This represents the median of the RMSE of all frame depths within the sliding window.
[0061] Step 1065: Construct a navigation reliability field based on map support score, trajectory curvature risk score, motion consistency score, and depth consistency score.
[0062] Specifically, the map support score, trajectory curvature risk score, motion consistency score, and depth consistency score are fused using a log-geometric weighted method to obtain the navigation reliability field. :
[0063] in, The weights of map support score, trajectory curvature risk score, motion consistency score, and depth consistency score are integrated to meet the following requirements. ; The scores represent map support score, trajectory curvature risk score, motion consistency score, and depth consistency score, with k indicating the iteration over these four variables.
[0064] Please see Figure 2 The navigation reliability field is obtained by logarithmic geometric fusion of four signals: map support, trajectory curvature risk, motion consistency, and depth rendering consistency. The fusion weights of the four indicators can be adjusted independently to adapt to different cavity morphologies such as the colon and stomach. Each indicator is calculated solely based on the map, pose, and depth observation data output by SLAM itself, without the need for external ground truth assistance, and outputs a reliability value in the range of 0 to 1 in real time, providing a basis for subsequent relocalization triggering.
[0065] Step 107: If the value of the navigation reliability field is not lower than the preset threshold, construct the cavity centerline, project the estimated coordinates of the current camera center onto the cavity centerline, and calculate the normalized arc length and radial offset. Based on the normalized arc length, construct the longitudinal arc length error, and based on the radial offset, construct the radial offset error. Based on the longitudinal arc length error and the radial offset error, calculate the cavity centerline target point registration error, and based on the cavity centerline target point registration error, evaluate the navigation accuracy.
[0066] Specifically, for the camera center trajectory sequence Fit a third-order B-spline curve as the lumen centerline To characterize the overall topology of natural cavities, the centerline of the lumen The calculation formula is:
[0067] in, Represents p-order B-spline basis functions; These are the control vertices for the B-spline.
[0068] Discrete sampling of the lumen centerline to solve for the arc length function Estimate the coordinates of the current camera center Projected onto the centerline of the lumen, The function matches the nearest centerline sampling point and outputs the index of that sampling point. Solve for the normalized arc length coordinates Radial offset distance The calculation formula is:
[0069] in, The sampling parameter is the arc length of the center line; This represents the coordinates of the centerline of the cavity obtained by substituting the sampling parameter u into the solution. Indicates index idx t The corresponding centerline arc length sampling parameters.
[0070] During the evaluation phase, the center coordinate trajectory and reference trajectory of the endoscopic camera are simultaneously projected onto the lumen centerline, and two types of navigation errors are identified: longitudinal arc length error along the lumen advancement direction. Vertical pipe wall radial offset error Constructing the target point registration error of the lumen centerline This is to comprehensively quantify navigation deviation. The formula for calculating the target point registration error along the lumen centerline is:
[0071] Define navigation error percentage It is used for overall navigation accuracy assessment, and the calculation formula is:
[0072] in, It indicates the total arc length of the cavity through which the endoscope travels.
[0073] Registration error of target point on lumen centerline The index decomposes the traditional single Euclidean error into longitudinal and radial components, which is more in line with the interpretation logic of clinicians compared to the absolute trajectory error (ATE) index. During the operation, the lumen centerline is fitted online through real-time SLAM trajectory. The reference trajectory can be a preoperative image model or a historical stable trajectory, and no frame-by-frame correction is required throughout the entire process.
[0074] Please see Figure 3 Based on camera center trajectory sequence By fitting a 3D B-spline lumen centerline, projecting the estimated coordinates of the camera center onto the centerline, and then decomposing the longitudinal arc length error and radial wall offset error, the LC-TRE target registration error is generated. Traditional 3D Euclidean error cannot distinguish between endoscope advance / retreat direction deviation and wall offset deviation. The error decomposition method of this invention can intuitively reflect the two different types of endoscope offset problems during surgery, making it easier for physicians to quickly determine the source of navigation deviation.
[0075] Step 108: If the navigation reliability field is lower than a preset threshold, iteratively update the transformation matrix until the second iteration termination condition is met, and update the endoscope camera pose based on the iteratively updated transformation matrix. When the navigation reliability field... Below the preset threshold If the current navigation is deemed unreliable, the system outputs a navigation warning signal and executes an adaptive relocation process. By comparing the navigation reliability field value with a preset threshold to determine the navigation status, and by initiating an adaptive relocation process to iteratively update the transformation matrix from the world coordinate system to the camera coordinate system, the system can correct for significant deviations in the endoscopic camera pose and prevent trajectory drift during endoscopy navigation.
[0076] In some embodiments, if the navigation reliability field is lower than a preset threshold, the transformation matrix is iteratively updated until the second iteration termination condition is met. The endoscope camera pose is then updated based on the iteratively updated transformation matrix, including the following steps: Step 1081: Generate an effective depth pixel mask based on the effective depth region and the endoscope contour region within the image.
[0077] Specifically, in length of Within the historical sliding window interval, the navigation reliability field value corresponding to each frame of image within the interval is traversed, and the image frame with the largest navigation reliability field value is selected as the local navigation reference anchor point. For the t-th frame image that triggers the adaptive relocalization process, the camera pose output by the conventional tracking stage of the synchronous localization and map building process is used as the reference, and iterative optimization calculation is performed to solve for the transformation matrix from the initial world coordinate system to the camera coordinate system before the optimization calculation. With the initial translation vector The two sets of parameters are used as the original input parameters for the depth alignment optimization operation.
[0078] During the depth alignment optimization process, the entire 3D Gaussian map parameter set is frozen, and iterative correction is performed only on the translation vector corresponding to the current frame's endoscope camera. Before the calculation, an effective depth pixel mask is pre-constructed, and only pixels corresponding to the effective depth region and endoscope contour region within the image participate in the calculation process. The formula for constructing the effective depth pixel mask is:
[0079] in, Used to remove null pixels with no effective depth. Indicates the inner diameter profile discriminant function, mask A value of 1 indicates that the pixel depth data is reliable and included in the loss calculation, while a value of 0 directly masks it, eliminating the interference of reflection, occlusion, and invalid depth on the optimization. This indicates a preset threshold.
[0080] Step 1082: Determine the mask region based on the effective depth pixel mask, and construct the average alignment loss function within the mask region.
[0081] Using the pixel region defined by the effective depth pixel mask as the calculation range, an average alignment loss function is constructed. This loss function is used to quantize the rendering depth D. render With observation depth D obs The overall numerical deviation between the two values; the higher the output value of the loss function, the lower the degree of matching between the rendering depth and the observation depth corresponding to the current endoscopic camera translation vector. The calculation process uses gradient descent to continuously correct the current frame's endoscopic camera translation vector. Endoscopic positioning drift is corrected by reducing the output value of the loss function.
[0082] In some embodiments, the average alignment loss function is calculated using the following formula:
[0083] in, Let M represent the average alignment loss function for frame t; M represents the set of effective pixel masks. Indicates the total number of valid pixels within the mask; This represents the rendering depth of pixel coordinate u in frame t; This represents the depth image of the current frame.
[0084] Step 1083: With minimizing the average alignment loss function as the optimization objective, iteratively optimize the transformation matrix until the second iteration termination condition is met; obtain the optimal average alignment loss after iteration, and calculate the decrease in the optimal average alignment loss compared to the original average alignment loss before iteration; determine whether the decrease exceeds a preset ratio threshold; if the decrease exceeds the preset ratio threshold, update the endoscope camera pose based on the iteratively optimized transformation matrix; if the decrease does not exceed the preset ratio threshold, retain the original endoscope camera pose.
[0085] The original loss value before the average alignment loss function iteration is compared with the optimal average alignment loss value after the iteration converges. The endoscope camera translation correction obtained in this solution is updated only when the decrease of the optimal average alignment loss relative to the original average alignment loss is greater than the preset improvement ratio threshold. If the decrease of loss does not reach the preset improvement ratio threshold, the initial endoscope camera pose before optimization is maintained without adjustment to alleviate the distortion of the endoscope motion trajectory caused by image noise.
[0086] The second iteration termination condition can be reaching a preset number of rounds or the loss difference reaching a preset convergence threshold.
[0087] If the depth improvement gating condition is met, calculate the translation difference before and after optimization, and introduce the L2 norm constraint to limit the maximum spatial offset distance of a single correction. The constraint formula is as follows:
[0088] in, This indicates the difference in translation correction for the endoscopic camera; Indicates the maximum spatial offset distance for a single correction; This represents the optimized translation vector. This represents the translation vector before optimization. Amplitude constraints are used to suppress large jumps in pose during a single operation, ensuring trajectory continuity.
[0089] Starting from the trigger frame, the corrected camera pose is recursively updated over a finite interval of up to H frames. The recursive update calculation formula is as follows:
[0090] in, Represents the transformation matrix from the world coordinate system to the camera coordinate system after correction in the j-th frame; This represents the original relative pose transformation matrix between adjacent frames; H represents the transformation matrix from the world coordinate system to the camera coordinate system after correction in the (j-1)th frame; H represents the maximum number of frames in the pose recursive update interval.
[0091] Image frames outside the propagation range maintain their original pose, while a repositioning cooling frame number is set. The interval between two repositioning triggers must not be less than the cooling frame number to avoid frequent iterations that consume edge computing resources.
[0092] Please see Figure 4The complete timing logic for depth-gated NDAR-based ground truth relocation is as follows: When the navigation reliability field (NRF) is below a threshold, relocation is triggered. First, a high-reliability anchor point within the window is selected, and pose optimization is applied to the current frame. The Gaussian map is frozen, and depth alignment iteration is performed. The depth improvement gating determines whether to adopt the correction amount. If the correction condition is met, the trajectory is updated for a limited number of frames, and then a cooling-off period begins. Relocation is not triggered again in the short term. If the gating condition is not met, the original pose is directly retained without any correction to avoid deterioration of navigation accuracy.
[0093] The entire algorithm can be deployed on the NVIDIA Thor edge computing platform, using C3VD dataset RGB-D endoscopic images as input (image resolution 675×540, tracking resolution 360×450, map optimization resolution 540×675). Parameters such as NRF fusion weights, depth improvement threshold, relocalization propagation frames, and cooling interval can all be independently adjusted to adapt to different surgical scenarios. In ten C3VD endoscopic scenario tests, the mean navigation error of this solution was 1.59±0.72%, with all scenarios having a navigation error of less than 5%. Dense reconstruction PSNR reached 21.60±2.74dB, and the absolute trajectory error was only 0.36±0.22mm. The depth gating mechanism completely avoids the navigation error spike caused by unconstrained relocalization, and the navigation stability is significantly better than traditional 3D GSLAM baseline solutions.
[0094] This invention also provides a three-dimensional reconstruction and navigation device for endoscopes, comprising: a Gaussian parameter acquisition module for acquiring the definition parameters of three-dimensional Gaussian units; a rendering generation module for rendering a color image and a depth image of the current frame based on the definition parameters, generating a rendered image and a rendered depth; a transformation matrix solving module for constructing an image-depth joint loss function based on the rendered image and the rendered depth, and calculating a transformation matrix from the world coordinate system to the camera coordinate system based on the image-depth joint loss function; a keyframe determination module for calculating the center coordinates of the endoscope camera in the current frame based on the transformation matrix, and determining whether the current frame meets the keyframe triggering condition based on the center coordinates; and a map optimization module for performing three-dimensional Gaussian unit addition and deletion operations if the keyframe triggering condition is met, and iteratively optimizing the definition parameters of the three-dimensional Gaussian units until the first iteration termination condition is met. The system outputs a 3D Gaussian map based on iteratively optimized 3D Gaussian units; a reliability field construction module is used to construct a navigation reliability field and determine whether the value of the navigation reliability field is lower than a preset threshold; a navigation accuracy evaluation module is used to construct the lumen centerline if the value of the navigation reliability field is not lower than the preset threshold, project the estimated coordinates of the current camera center onto the lumen centerline, and calculate the normalized arc length and radial offset, construct the longitudinal arc length error based on the normalized arc length, construct the radial offset error based on the radial offset, calculate the lumen centerline target point registration error based on the longitudinal arc length error and the radial offset error, and evaluate the navigation accuracy based on the lumen centerline target point registration error; a pose correction module is used to iteratively update the transformation matrix if the navigation reliability field is lower than the preset threshold until the second iteration termination condition is met, and update the endoscope camera pose based on the iteratively updated transformation matrix.
[0095] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for three-dimensional reconstruction and navigation of endoscopic cavity.
[0096] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for three-dimensional reconstruction and navigation of a cavity endoscope.
[0097] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.
[0098] This invention utilizes 3D Gaussian units to reconstruct the cavity scene, solves the transformation matrix from the world coordinate system to the camera coordinate system through an image-depth joint loss function, and optimizes the 3D Gaussian unit parameters through keyframe iteration to output an accurate 3D Gaussian map. By setting a judgment threshold through a navigation reliability field, repositioning is performed to update the pose when the value is lower than a preset threshold. At the same time, navigation accuracy is evaluated by calculating the longitudinal arc length error and radial offset error through the lumen centerline. It can adapt to the intraoperative clinical scenario without external ground truth. The two-dimensional error is easy to interpret clinically, and the reliability threshold can avoid the decrease in navigation accuracy caused by frequent repositioning, effectively improving the clinical adaptability and operational stability of endoscopic navigation.
[0099] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for three-dimensional reconstruction and navigation of endoscopic cavity, characterized in that, include: Obtain the definition parameters of the 3D Gaussian element; Based on the defined parameters, render the current frame's color image and depth image to generate a rendered image and rendering depth. Based on the rendered image and the rendered depth, an image-depth joint loss function is constructed, and based on the image-depth joint loss function, the transformation matrix from the world coordinate system to the camera coordinate system is calculated; Calculate the center coordinates of the endoscope camera in the current frame based on the transformation matrix, and determine whether the current frame meets the keyframe triggering condition based on the center coordinates. If the keyframe triggering condition is met, perform the addition and deletion of three-dimensional Gaussian units, and iteratively optimize the definition parameters of the three-dimensional Gaussian units until the first iteration termination condition is met. Then, output a three-dimensional Gaussian map based on the iteratively optimized three-dimensional Gaussian units. Construct a navigation reliability field and determine whether the value of the navigation reliability field is lower than a preset threshold; If the value of the navigation reliability field is not lower than the preset threshold, a cavity centerline is constructed, the estimated coordinates of the current camera center are projected onto the cavity centerline, and the normalized arc length and radial offset are calculated. A longitudinal arc length error is constructed based on the normalized arc length, and a radial offset error is constructed based on the radial offset. The cavity centerline target point registration error is calculated based on the longitudinal arc length error and the radial offset error. Navigation accuracy is evaluated based on the cavity centerline target point registration error. If the navigation reliability field is lower than the preset threshold, the transformation matrix is iteratively updated until the second iteration termination condition is met, and the endoscopic camera pose is updated based on the iteratively updated transformation matrix.
2. The method for three-dimensional reconstruction and navigation of endoscopic cavity according to claim 1, characterized in that, The definition parameters of the three-dimensional Gaussian element include at least one of the following: the center position coordinates of the three-dimensional Gaussian element, logarithmic opacity, logarithmic scale, rotation quaternion, and color feature.
3. The method for three-dimensional reconstruction and navigation of endoscopic cavity according to claim 1, characterized in that, The construction of the navigation reliability field includes: Calculate the map support score based on the center coordinates and neighborhood radius of the endoscope camera; Calculate trajectory curvature risk score based on endoscopic camera center trajectory sequence; Motion consistency score is calculated based on the center coordinates of the endoscope camera and the amount of trajectory drift within the window. A depth consistency score is constructed based on the rendering depth. A navigation reliability field is constructed based on the map support score, the trajectory curvature risk score, the motion consistency score, and the depth consistency score.
4. The method for three-dimensional reconstruction and navigation of endoscopic cavity according to claim 1, characterized in that, The preset threshold is 0.
35.
5. The method for three-dimensional reconstruction and navigation of endoscopic cavity according to claim 1, characterized in that, If the navigation reliability field is lower than the preset threshold, the transformation matrix is iteratively updated until the second iteration termination condition is met. The endoscope camera pose is then updated based on the iteratively updated transformation matrix, including: An effective depth pixel mask is generated based on the effective depth region within the image and the endoscope contour region. The mask region is determined based on the effective depth pixel mask, and the average alignment loss function within the mask region is constructed. With minimizing the average alignment loss function as the optimization objective, the transformation matrix is iteratively optimized until the second iteration termination condition is met; Obtain the optimal average alignment loss after iteration, and calculate the decrease in the optimal average alignment loss compared to the original average alignment loss before iteration; Determine whether the decrease exceeds a preset percentage threshold; If the decrease exceeds a preset proportional threshold, the endoscopic camera pose is updated based on the iteratively optimized transformation matrix. If the decrease does not exceed a preset percentage threshold, the original endoscopic camera pose is retained.
6. The method for three-dimensional reconstruction and navigation of endoscopic cavity according to claim 1, characterized in that, The formula for calculating the average alignment loss function is as follows: in, Let M represent the average alignment loss function for frame t; M represents the set of effective pixel masks. Indicates the total number of valid pixels within the mask; This represents the rendering depth of pixel coordinate u in frame t; This represents the depth image of the current frame.
7. The method for three-dimensional reconstruction and navigation of endoscopic cavity according to claim 1, characterized in that, The image-depth joint loss function The calculation formula is: in, Represents the image loss weight coefficients; Indicates the rendered image; Indicates the current frame's color image; Indicates the depth loss weighting coefficient; Indicates rendering depth; Represents the depth image of the current frame; This represents the effective depth pixel mask for frame t. Represents the L1 norm; This represents the Hadama pixel-wise multiplication operator.
8. A three-dimensional reconstruction and navigation device for cavity endoscopes, characterized in that, include: The Gaussian parameter acquisition module is used to obtain the definition parameters of a three-dimensional Gaussian element; The rendering generation module is used to render the current frame's color image and depth image based on the defined parameters, and generate a rendered image and rendering depth. The transformation matrix solving module is used to construct an image-depth joint loss function based on the rendered image and the rendered depth, and to calculate the transformation matrix from the world coordinate system to the camera coordinate system based on the image-depth joint loss function; The keyframe determination module is used to calculate the center coordinates of the endoscope camera in the current frame based on the transformation matrix, and determine whether the current frame meets the keyframe triggering conditions based on the center coordinates. The map optimization module is used to perform 3D Gaussian cell addition and deletion operations if the keyframe triggering condition is met, and iteratively optimize the definition parameters of the 3D Gaussian cells until the first iteration termination condition is met, and output a 3D Gaussian map based on the iteratively optimized 3D Gaussian cells. A reliability field construction module is used to construct a navigation reliability field and determine whether the value of the navigation reliability field is lower than a preset threshold. The navigation accuracy assessment module is used to construct a cavity centerline if the value of the navigation reliability field is not lower than the preset threshold, project the estimated coordinates of the current camera center onto the cavity centerline, calculate the normalized arc length and radial offset, construct a longitudinal arc length error based on the normalized arc length, construct a radial offset error based on the radial offset, calculate the cavity centerline target point registration error based on the longitudinal arc length error and the radial offset error, and perform navigation accuracy assessment based on the cavity centerline target point registration error. The pose correction module is used to iteratively update the transformation matrix if the navigation reliability field is lower than the preset threshold, until the second iteration termination condition is met, and update the pose of the endoscope camera based on the iteratively updated transformation matrix.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a method for three-dimensional reconstruction and navigation of cavity endoscopes according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a three-dimensional reconstruction and navigation method for cavity endoscopy according to any one of claims 1 to 7.
