A pure visual endoscope video real-time generalizable three-dimensional reconstruction method and system

Abstract

This invention discloses a real-time generalizable 3D reconstruction method and system for endoscopic videos based on pure vision. The method first acquires image sequences from endoscopic videos, constructs adjacent frame pairs with overlapping viewpoints, and preprocesses these adjacent frame pairs. Next, it constructs an endoscopic 3D reconstruction model based on a base model, including a semantic feature extraction module, a feature aggregation module, and a 3D attribute prediction head. Then, it introduces additional errors, constructs preprocessed adjacent frame pairs as training data, and performs self-supervised training on the endoscopic 3D reconstruction model. Finally, it performs frame-by-frame inference on real-time acquired endoscopic videos of arbitrary length, generating a 3D reconstructed view in real time. This invention effectively solves the scale collapse problem, suppresses texture artifacts, significantly improves the geometric fidelity of depth maps and point clouds, and can effectively overcome the domain differences from macroscopic natural scenes to microscopic endoscopic scenes, achieving zero-shot generalization.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and medical robotics, and more specifically, to a method and system for real-time generalizable 3D reconstruction of endoscopic video using pure vision. Background Technology

[0002] High-fidelity 3D reconstruction of endoscopic videos is a prerequisite for precise surgical navigation, augmented reality assistance, and automated operation. By recovering the 3D geometry of tissues from 2D endoscopic videos, doctors can perceive depth information and avoid high-risk areas.

[0003] Currently, 3D reconstruction methods for endoscopic scenes are mainly divided into traditional geometric methods and deep learning-based methods. Traditional geometric methods, such as Simultaneous Localization and Mapping (SLAM) and Structure of Motion (SfM), primarily rely on matching and triangulation of image feature points to recover sparse point clouds. However, the internal tissues under endoscopy are usually smooth, lack texture, and are subject to reflective interference, making feature extraction difficult. The reconstruction results are often sparse and incomplete, making it difficult to describe the fine surface structure of soft tissues. Deep learning-based methods, such as Neural Radiation Field (NeRF) and 3D Gaussian Sputtering (3DGS), have achieved breakthroughs in reconstruction density and realism, but they essentially rely on a "scene-by-scene optimization" paradigm. That is, for each new video or each new patient, time-consuming model training or parameter optimization is required again. The inference speed cannot meet the real-time requirements of surgical robots, and there is a lack of cross-scene generalization ability.

[0004] Recently, basic models for monocular depth estimation or 3D reconstruction, pre-trained on large-scale data, have shown some zero-shot generalization potential. However, most existing basic models are trained on natural scenes (macroscale, Lambertian illumination), and when directly applied to endoscopic scenes (microscale, non-Lambertian wetted surfaces), they suffer from a serious "domain drift" problem. Specifically, this manifests as follows:

[0005] First, the motion priors learned by existing basic models pre-trained on natural scenes mainly correspond to significant inter-frame displacements in macroscopic scenes. However, continuous frames of endoscopic videos typically have only minimal baselines or sub-millimeter-level motions, causing the camera poses obtained by direct regression to tend to be unit transformations, which in turn leads to scale drift, pose instability, and trajectory error accumulation.

[0006] Secondly, endoscopic scenes involve coaxial illumination, non-Lambertian reflections from moist tissue surfaces, and specular highlights, which disrupt the assumption of brightness consistency between images. Existing photometric self-supervised methods often mistakenly treat appearance changes as geometric errors and backpropagate them, thus contaminating the learning process of depth and pose branches.

[0007] Third, endoscopic images contain a wide range of high-frequency details such as vascular textures and mucosal textures. Traditional edge-aware smoothing constraints based on RGB gradients can easily misidentify texture edges as real geometric boundaries, resulting in texture replication artifacts that affect the geometric realism of depth maps and point clouds.

[0008] Therefore, there is an urgent need for an endoscopic 3D reconstruction method that does not rely on 3D ground truth labels, does not require scene-by-scene retraining, can overcome differences from macroscopic to microscopic domains, and can achieve real-time high-fidelity inference. Summary of the Invention

[0009] The purpose of this invention is to overcome the problems of high computational time, weak cross-scene generalization ability, and heavy reliance on dense depth ground truth in existing endoscopic 3D reconstruction technologies, and to provide a purely vision-based real-time generalizable 3D reconstruction method and system for endoscopic videos. This method can recover high-fidelity depth maps and camera poses in real time from endoscopic video sequences of arbitrary length based on a single feedforward inference, and can effectively overcome the domain differences between macroscopic natural scenes and microscopic endoscopic scenes, achieving zero-sample generalization.

[0010] To achieve the above objectives, this invention proposes a real-time, generalizable 3D reconstruction method for endoscopic videos based solely on visual perception, comprising:

[0011] S1: Acquire an image sequence from an endoscopic surgery video, and construct the image sequence into adjacent frame pairs with overlapping viewpoints; preprocess the adjacent frame pairs to extract tensor data for network input;

[0012] S2: Construct an endoscope 3D reconstruction model based on a base model adaptation. The 3D reconstruction model includes a semantic feature extraction module, a feature aggregation module, and a 3D attribute prediction head.

[0013] The feature aggregation module consists of cascaded alternating blocks with identical structures, with each layer sequentially performing intra-frame self-attention and global self-attention calculations. The feature aggregation module receives semantic features from the semantic feature extraction module and outputs aggregated spatiotemporal features. .

[0014] The 3D attribute prediction head includes a depth prediction head and a camera prediction head; the depth prediction head generates a dense depth map based on aggregated spatiotemporal features, and obtains the target frame respectively. and source frame Corresponding target frame depth map and source frame depth map The camera prediction head outputs an initial pose estimate based on aggregated spatiotemporal features.

[0015] Furthermore, a residual pose adaptation module is set after the camera prediction head. The residual is predicted and corrected by a dual-stream fusion method of semantic flow and geometric flow. The initial pose is then calibrated a second time to output the final camera pose.

[0016] To address the issue of near-identical pose degradation caused by minimal inter-frame motion in continuous endoscopic videos, a residual pose adaptation module is introduced after the initial pose. This module jointly encodes the semantic features of the intermediate layer with the geometric state of the initial pose, and then uses a residual fusion network to predict the rotation and translation residuals, thereby performing conditional secondary calibration of the initial pose. The key to this approach is utilizing semantic context to assist pose correction under minute motions, which distinguishes it from existing direct pose regression methods, feature matching methods, and feedforward methods that simply decouple depth and pose.

[0017] S3: To minimize the additional error introduced by camera parallax, preprocessed adjacent frame pairs are constructed as training data to perform self-supervised training on the endoscope 3D reconstruction model.

[0018] During the training phase, a parallel dual-branch self-supervised training framework is constructed, including a geometry branch for depth, pose, and geometry reprojection, and an appearance decoupling branch for appearance perturbation modeling.

[0019] Among them, the geometry branch performs cross-view rigid reprojection based on the dense depth map output by the 3D reconstruction model and the final camera pose to generate a synthetic target view, while the appearance decoupling branch performs appearance residual modeling on reflections, highlights, local brightness drift and non-rigid lighting changes in the target frame through the appearance flow decoupling network.

[0020] Furthermore, by minimizing the total loss function, which includes appearance calibration photometric loss, semantic guidance smoothing loss, and appearance flow regularization loss, the network parameters of the endoscopic 3D reconstruction model and the adaptation parameters of the residual pose adaptation module are jointly optimized.

[0021] Unlike existing self-supervised endoscopic reconstruction methods that typically construct only a single geometric branch and directly constrain the photometric consistency between the reprojected synthetic view and the original target frame, this invention constructs a parallel dual-branch self-supervised optimization mechanism during the training phase. The geometric branch is responsible for learning rigid structural information consistent across viewpoints, while the appearance decoupling branch is responsible for modeling appearance residuals caused by specular reflections in endoscopic imaging, highlights on moist tissue surfaces, coaxial illumination variations, and local non-rigid appearance perturbations. Furthermore, the appearance decoupling branch predicts appearance residuals based on the shared geometric representation after stopping gradient processing, ensuring that the gradient of the appearance branch does not backpropagate to the shared geometric backbone. This limits appearance errors to absorption within the auxiliary branch, without contaminating the geometric learning process of the depth and pose branches.

[0022] S4: For endoscopic videos of arbitrary length, perform frame-by-frame reasoning using the trained endoscopic 3D reconstruction model to generate a 3D reconstructed view.

[0023] Optionally, in step S2, the residual pose adaptation module calibrates the initial pose through a two-stream fusion mechanism. Its calculation logic includes: aggregating the spatiotemporal features F and the initial pose respectively. The latent variables are mapped to semantic and geometric latent variables. The semantic and geometric latent variables are concatenated and input into a residual fusion network to predict the pose correction. The initial pose is then updated based on the pose correction to obtain the final camera pose. .

[0024] Optionally, in step S3, an appearance flow decoupling network is set up, adopting an encoder-decoder structure, and based on the input adjacent frame pairs, outputs an appearance flow field with the same resolution as the target frame image. The appearance decoupling branch employs a gradient isolation method, confining the appearance compensation gradient generated by the photometric loss during appearance calibration within the appearance decoupling branch and preventing it from propagating back to the geometry branch, thus achieving gradient decoupling between the appearance and geometry branches. This utilizes the appearance flow field. For the target frame image The images are added together for visual calibration to obtain the calibrated target image. .

[0025] Furthermore, in the geometric branch, the final camera pose is utilized. Source frame depth map And the camera intrinsic parameter matrix K, which will be the source frame image Reprojecting the image onto the target frame's viewpoint yields the composite target view. Simultaneously, utilizing the aforementioned external flow field For the target frame image Perform appearance calibration to obtain the calibrated target image. .

[0026] In another aspect, the present invention also provides a real-time generalizable 3D reconstruction system for pure vision endoscopic video, used to implement the aforementioned 3D reconstruction method, comprising the following modules:

[0027] The adjacent frame pair construction module is used to acquire image sequences of endoscopic video, construct adjacent frame pairs with overlapping viewpoints, and preprocess the adjacent frame pairs.

[0028] The endoscopic 3D reconstruction module is used to build an endoscopic 3D reconstruction model based on a base model, including a semantic feature extraction module, a feature aggregation module, and a 3D attribute prediction head.

[0029] The self-supervised training module is used to introduce additional errors. It uses preprocessed adjacent frame pairs as training data to perform self-supervised training on the endoscopic 3D reconstruction model.

[0030] The 3D reconstruction view output module is used to generate a 3D reconstruction view from endoscopic videos of arbitrary length by performing frame-by-frame inference through a trained endoscopic 3D reconstruction model.

[0031] The beneficial effects of this invention are as follows:

[0032] Real-time generalization capability: This invention constructs a reconstruction model that includes a residual pose adaptation module and uses a dual-stream fusion mechanism to combine the general priors of the basic model with the microscopic features of the endoscopic scene, effectively solving the cross-domain gap problem and achieving zero-sample real-time reconstruction of unseen surgical scenes (inference speed > 40 FPS).

[0033] High-precision reconstruction with resistance to interference: In response to interference from strong reflections and rich vascular textures under endoscopy, this invention explicitly decouples illumination changes and geometric structures in self-supervised training through appearance flow decoupling network and semantically guided smoothing loss, and suppresses texture artifacts, significantly improving the pose geometry fidelity of depth.

[0034] Low-cost clinical deployment: This invention adopts a pure visual self-supervised learning framework, which does not require expensive CT / MRI scans to obtain three-dimensional ground truth labels. Training and deployment can be completed with only monocular or binocular endoscopic video, thus lowering the threshold for clinical application. Attached Figure Description

[0035] Figure 1 A flowchart illustrating the steps of the method according to the present invention is shown;

[0036] Figure 2 A detailed network architecture diagram of the 3D reconstruction model in a specific implementation case is shown;

[0037] Figure 3 The results of scene reconstruction on the SCARED long sequence dataset are shown. Detailed Implementation

[0038] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0039] Example 1: A Real-Time Generalizable 3D Reconstruction Method for Endoscopic Video Based on Pure Vision

[0040] This embodiment provides a real-time, generalizable 3D reconstruction method for endoscopic videos based on pure vision. For example... Figure 1As shown, this method is designed for endoscopic video sequences and employs a feedforward 3D reconstruction framework. The overall framework includes steps such as data acquisition and preprocessing, semantic feature extraction, spatiotemporal feature aggregation, depth and point cloud prediction, initial camera pose estimation and residual correction, self-supervised training optimization, and real-time inference output. In particular, a residual pose adaptation module is introduced in the pose estimation stage, and appearance decoupling branch and semantic-guided smoothing constraint are introduced in the training stage, thereby achieving joint reconstruction of depth map, camera pose, and dense point cloud in endoscopic scenes.

[0041] The preferred implementation environment for this embodiment is an NVIDIA L20 graphics processing unit.

[0042] like Figure 2 As shown, the specific steps of this embodiment are as follows.

[0043] S1: Data Acquisition and Tensor Preprocessing

[0044] 1. Video Sequence Acquisition: Real-time surgical video streams are acquired from the endoscope of a minimally invasive surgical robot (such as the da Vinci Surgical System). The video resolution is 1280×1024, and the frame rate is 30fps.

[0045] 2. Keyframe Extraction and Pairing: To construct an effective self-supervised signal, it is necessary to build adjacent frame pairs with overlapping viewpoints. In this embodiment, images are extracted in chronological order to construct training pairs. ,in The target frame at the current moment, For adjacent source frames (t+1 or t-1 frames).

[0046] 3. Image preprocessing:

[0047] The input image was adjusted to a fixed resolution of 518×518 to balance memory usage and reconstruction details.

[0048] Normalize pixel values ​​to The intervals were defined and standardized using the mean and variance of the ImageNet dataset; then the images were converted to tensor format. Where B represents the batch size, S represents the sequence length, C represents the number of channels, H represents the image height, and W represents the image width.

[0049] S2: Construct a 3D reconstruction model based on base model adaptation

[0050] This embodiment adopts a feedforward architecture consisting of a semantic feature extraction module, a feature aggregation module, and a three-dimensional attribute prediction head.

[0051] 1. Semantic Feature Extraction Module

[0052] The semantic feature extraction module employs a pre-trained visual encoder, DINOv2, which is a self-supervised visual representation model based on a visual Transformer, preferably using the ViT-L / 14 architecture. This module includes an image patch embedding layer and a multi-layer Transformer module, used to divide the input image into fixed-size image patches and extract patch-level semantic features. Preferably, the input image is divided into 14×14 image patches and mapped to high-dimensional embedding vectors. To preserve the general visual representations learned by the pre-trained model on natural images, the parameters of this module are preferably frozen during the training phase.

[0053] 2. Feature aggregation module

[0054] The feature aggregation module consists of 24 cascaded alternating blocks with identical structures. Each layer sequentially performs intra-frame self-attention and global self-attention calculations. Intra-frame self-attention establishes local associations between single frames to finely capture tissue surface texture and local geometry, while global self-attention aggregates all views across frames to establish long-range correspondences between multi-frame images. The two are stacked alternately to achieve geometric consistency modeling across views while maintaining spatial resolution. The feature aggregation module receives semantic features from the semantic feature extraction module and outputs aggregated spatiotemporal features. .

[0055] 3. 3D attribute prediction head and residual pose adaptation module

[0056] This embodiment includes a depth prediction head and a camera prediction head.

[0057] (1) Depth prediction head

[0058] The deep prediction head employs a DPT (a dense Transformer decoder structure) architecture to aggregate spatiotemporal features. Decode the target frames to obtain them. and source frame Corresponding target frame depth map and source frame depth map The depth map output is constrained to a preset depth range using the Sigmoid function. Inside.

[0059] (2) Camera prediction head and residual pose adaptation module

[0060] The camera prediction head first bases its prediction on aggregated spatiotemporal features. Output the initial relative pose from the source frame to the target frame. Since the pre-trained base model is mainly derived from natural scenes, direct prediction... Near-identical degradation or scale drift may occur under conditions of minute endoscopic motion.

[0061] Therefore, a residual pose adaptation module is further set up after the camera prediction head. This module receives the output features of the intermediate layer of the aggregation network. and initial pose And construct semantic flow and geometric flow respectively:

[0062]

[0063]

[0064] in, This represents the intermediate semantic features output from the 16th layer of the feature aggregation module; To represent the semantic projection layer, it is preferable to use The two-layer, multi-layer perceptron structure is used to... Mapping to semantic latent variables ; To represent the geometric mapping layer, preferably using A two-layer geometric mapping structure is used to map the flattened initial pose. Mapping to geometric latent variables ; The representation layer normalization operation is used to achieve scale alignment between the semantic flow and the geometric flow.

[0065] Subsequently, semantic latent variables With geometric latent variables Perform splicing and input into the residual fusion network. The pose correction residual is obtained as follows:

[0066]

[0067] in, The transformation matrix is ​​3x4 and can be decomposed into rotation corrections. Translation correction , Indicates feature splicing; The residual fusion network is represented by a multilayer perceptron structure containing linear residual blocks, with skip connections inside to enhance the ability to model small motion perturbations and the stability of gradient propagation.

[0068] Finally, the initial pose is corrected according to the following formula to obtain the final camera pose. :

[0069]

[0070]

[0071] in, This represents an exponential mapping used to map rotational residuals in axis-angle form to rotation matrices. Represents the final rotation matrix; Indicates the final translation vector; This indicates the final camera pose after correction.

[0072] S3: Appearance Decoupling Self-Supervised Training Process

[0073] Due to the lack of realistic 3D supervised labels, this embodiment constructs a parallel dual-branch self-supervised training framework during the training phase. The parallel dual branches include a geometry branch and an appearance decoupling branch. The geometry branch is responsible for learning the consistency of depth maps, camera pose, and rigid reprojection; the appearance decoupling branch is responsible for modeling appearance perturbations such as reflections, specular highlights, and local brightness drift. This dual-branch structure allows geometry learning and appearance compensation to cooperate while remaining decoupled from each other.

[0074] 1. Appearance decoupling branch

[0075] During the training phase, an appearance decoupling branch is set up in parallel. This appearance decoupling branch includes an appearance flow decoupling network, which employs a U-Net structure with skip connections, outputting an appearance flow field at the same resolution as the target frame image. This is used to characterize appearance perturbations caused by specular reflections, highlights, local brightness drift, and non-rigid lighting changes. A stopping gradient operation is used to prevent the gradients of the appearance decoupled branches from backpropagating to the shared geometric backbone, thereby avoiding appearance perturbations from contaminating depth and pose learning.

[0076] 2. Geometric branching and view composition

[0077] The geometric branch is based on the source frame depth map obtained in step S2. Final camera pose and camera intrinsic parameter matrix , source frame image Rigid reprojection onto the target frame view yields the synthetic target view. At the same time, utilizing the external flow field For the target frame image Perform appearance calibration to obtain the calibrated target image. :

[0078]

[0079] in, Indicates pixel position; Indicates the target frame image in pixels Pixel value at; Indicates the appearance flow field at the pixel The residual value at the location; This indicates the target image after appearance calibration at the pixel level. The pixel value at that location.

[0080] 3. Photometric loss during appearance calibration

[0081] To constrain the consistency between the rigid reprojected image and the appearance-calibrated target image, an appearance calibration photometric loss is defined. for:

[0082]

[0083] in, Indicates the effective projected pixel area. Indicates the number of valid pixels. Indicates the composite target view in pixels Pixel value at; It is a hybrid loss function of structural similarity SSIM and L1 distance.

[0084] Unlike existing methods that directly apply photometric constraints to the reprojected synthetic view and the unprocessed original target frame, this invention first uses an appearance decoupling branch to compensate for appearance perturbations in the target frame, and then applies consistency constraints to the synthetic view and the calibrated target image. Meanwhile, since the input of the appearance flow decoupling network preferably comes from the shared representation after the gradient stops, the appearance branch is only responsible for absorbing appearance errors and will not reverse-write the depth and pose features in the geometric backbone.

[0085] 4. Semantic-guided smoothing loss

[0086] To suppress texture replication artifacts caused by vascular and mucosal textures, a semantically guided smoothing loss is constructed based on deep semantic features. .

[0087] First, based on the source frame depth map Constructing an inverse depth map Then, normalization is performed to obtain the normalized inverse depth map. Its expression is:

[0088]

[0089] in, To prevent tiny constants with a denominator of zero, Represents inverse depth map The mean value within the effective pixel area.

[0090] Furthermore, the semantic feature map output by the feature aggregation module is extracted. And construct semantic smoothing weights based on the gradients of the semantic feature map in the horizontal and vertical directions:

[0091]

[0092]

[0093] in, Represents semantic feature maps at the pixel level The eigenvector at that location; and These represent the gradient operators at the pixel along the horizontal and vertical directions, respectively. This is the semantic gradient sensitivity coefficient, used to adjust the influence of semantic boundaries on the strength of smoothness constraints.

[0094] Based on this, the semantically guided smoothing loss is defined as follows:

[0095]

[0096] in, and These represent the normalized inverse depth map at pixels. The horizontal and vertical gradient magnitudes at the location; Represents inverse depth map In pixels The value at that location, and These represent the semantic features at the pixel level. The smoothing weights generated at that point.

[0097] 5. Surface flow regularization loss

[0098] To limit the numerical amplitude and spatial oscillations of the external flow field, an external flow regularization loss is defined. for:

[0099]

[0100] in, Indicates the appearance flow field at the pixel The residual value at the location; Indicates the appearance flow field at the pixel Spatial gradient at a given location; The regularization balance coefficient is used to balance the magnitude constraint of the appearance residual and its spatial smoothing constraint. The appearance flow regularization loss is used to limit the numerical amplitude and spatial oscillation of the appearance flow field, preventing the appearance decoupling branch from erroneously absorbing the geometric displacement caused by camera motion into the appearance branch.

[0101] Because vascular textures, mucosal textures, and local highlights in endoscopic images often produce high-frequency edges unrelated to real 3D geometry, traditional smoothing methods easily misjudge these texture edges as real anatomical boundaries, resulting in texture replication artifacts. This invention employs semantically guided smoothing loss, which focuses more on anatomical semantics and structural continuity, preserving depth abrupt changes at real anatomical boundaries and enhancing depth smoothing in surface texture regions.

[0102] 6. Total Loss Function and Optimization Process

[0103] The total loss function is defined as:

[0104]

[0105] in, To calibrate photometric loss for appearance, used to constrain cross-view image consistency after rigid reprojection; Semantic-guided smoothing loss is used to constrain the spatial smoothness of the depth map in unstructured boundary regions; This is the regularization loss for the external flow, used to prevent non-physical overfitting of the external flow field; and These are hyperparameters used to balance the contributions of the smoothing loss term and the appearance regularization term to the total loss, respectively.

[0106] During the optimization process, the AdamW optimizer is preferred, with an initial learning rate set to [value missing]. The learning rate is decayed using a cosine annealing strategy. During the training phase, only the parameters of the intermediate layer of the feature aggregation module, the 3D attribute prediction head, the residual pose adaptation module, and the appearance flow decoupling network are updated, while the parameters of the pre-trained visual encoder and the shallow and deep layers of the aggregation network are kept frozen.

[0107] S4: Real-time generation of 3D reconstruction views

[0108] For real-time acquired endoscopic video streams, any number of image frames are input into the trained endoscopic 3D reconstruction model. The dense depth map of the corresponding frame is output through the depth prediction head, and the camera pose of each frame is output through the camera prediction head and the residual pose adaptation module.

[0109] Subsequently, based on the camera pose, the dense depth map of each frame is back-projected into a local 3D point cloud in the camera coordinate system, and each local point cloud is transformed to a unified world coordinate system.

[0110] At the same time, the RGB color information of the original image is mapped to the corresponding 3D vertices, and the point cloud color attributes are assigned to reconstruct surface texture details;

[0111] Finally, incremental fusion and rendering projection are performed on the time-accumulated colored point cloud to output a 3D reconstructed view from the current perspective in real time.

[0112] In another aspect, the present invention also provides a real-time generalizable 3D reconstruction system for pure vision-based endoscopic video, used to implement the aforementioned 3D reconstruction method, comprising an adjacent frame pair construction module, an endoscopic 3D reconstruction module, a self-supervised training module, and a 3D reconstruction view output module:

[0113] The adjacent frame pair construction module is used to acquire image sequences of endoscopic videos, construct adjacent frame pairs with overlapping perspectives, and preprocess the adjacent frame pairs.

[0114] The endoscopic 3D reconstruction module is used to construct an endoscopic 3D reconstruction model based on a base model, including a semantic feature extraction module, a feature aggregation module, and a 3D attribute prediction head.

[0115] The self-supervised training module is used to introduce additional errors by using preprocessed adjacent frame pairs as training data to perform self-supervised training on the endoscopic 3D reconstruction model.

[0116] The 3D reconstruction view output module is used to generate a 3D reconstruction view by performing frame-by-frame reasoning on endoscopic videos of arbitrary length using a trained endoscopic 3D reconstruction model.

[0117] Example:

[0118] To verify the effectiveness, generalization ability, and real-time performance of the method of the present invention, dataset testing, quantitative evaluation, and result analysis were conducted on this embodiment.

[0119] (I) Introduction to the dataset:

[0120] This embodiment primarily focuses on training and testing on the SCARED dataset, and performs cross-dataset generalization validation on the Hamlyn and C3VD datasets. The SCARED dataset is an endoscopic dataset collected in robot-assisted minimally invasive surgery scenarios, containing real surgical interference factors such as reflections, highlights, complex tissue textures, and perspective changes. The Hamlyn and C3VD datasets are used to test the cross-domain generalization ability of this invention in scenarios without an endoscope.

[0121] Training, validation, and testing were performed on the SCARED dataset according to the publicly available partitioning method; the training set contained 15351 frames, the validation set contained 1705 frames, and the test set contained 551 frames; pose evaluation used 833 frames of independent trajectory sequences.

[0122] (II) Experimental setup and evaluation indicators:

[0123] This embodiment employs a self-supervised training method, which does not rely on 3D ground truth point clouds as training supervision signals. During the training phase, adjacent frame pairs are used as input, and the AdamW optimizer is employed for optimization.

[0124] Depth estimation uses the following evaluation metrics: absolute relative error (AbsRel), squared relative error (SqRel), root mean square error (RMSE), and threshold accuracy. Among these, the smaller AbsRel, SqRel, and RMSE are, the better. The bigger the better.

[0125] The following evaluation metrics are used for pose estimation: absolute trajectory error (ATE), relative pose translation error (RPE-t), and relative pose rotation error (RPE-r). The smaller the above metrics, the more accurate the pose estimation.

[0126] Real-time performance is evaluated using inference frame rate (FPS), with a higher FPS indicating better real-time performance.

[0127] (III) Experimental Results:

[0128] Table 1 shows a comparison of the depth estimation results of the method of this invention and existing methods on the SCARED dataset:

[0129] Table 1

[0130]

[0131] Table 2 presents the depth estimation results of the method of the present invention on the Hamlyn and C3VD datasets:

[0132] Table 2

[0133]

[0134] Table 3 shows the pose estimation results of the method of the present invention on the SCARED dataset:

[0135] Table 3

[0136]

[0137] Table 4 presents a comparison of the inference speed between the method of this invention and existing methods:

[0138] Table 4

[0139]

[0140] (iv) Results Analysis and Explanation:

[0141] On the SCARED dataset, the scene reconstruction results are as follows: Figure 3As shown, the depth estimation error of the method of the present invention is lower than that of the existing best methods, indicating that the method of the present invention effectively alleviates the problems of pose degradation and scale drift under the condition of micro-motion of endoscope, and improves the accuracy of endoscopic three-dimensional reconstruction in surgical scenarios.

[0142] Results on the Hamlyn and C3VD datasets demonstrate that the method of this invention maintains good depth estimation performance in unseen data domains, indicating that the method of this invention has significantly improved cross-scene generalization ability.

[0143] The pose estimation results show that the method of the present invention has good stability in terms of relative pose translation error and rotation error, indicating that the residual pose adaptation module can improve the pose correction capability under small parallax conditions.

[0144] The inference speed results show that the method of the present invention can achieve high frame rate feedforward inference without scene-by-scene optimization, and is therefore suitable for real-time application scenarios such as intraoperative navigation.

[0145] Summary of the Implementation: This implementation constructs a pure vision-based feedforward 3D reconstruction method suitable for endoscopic video scenarios through the above steps. This method performs secondary correction of the initial pose through a residual pose adaptation module, reduces the interference of reflection and brightness changes on geometric learning through an appearance-decoupled dual-branch self-supervised training mechanism, and suppresses texture replication artifacts through semantically guided smoothing loss. Experimental results show that the method of this invention has good depth estimation performance, pose estimation performance, and real-time inference capability on the SCARED, Hamlyn, and C3VD datasets, and can provide effective 3D perception support for surgical navigation and robot-assisted operation.

Claims

1. A purely vision-based endoscopic video real-time generalizable 3D reconstruction method, characterized in that, Includes the following steps: S1: Obtain the image sequence of the endoscopic video, construct it into adjacent frame pairs with overlapping viewpoints, and preprocess the adjacent frame pairs; S2: Construct an endoscope 3D reconstruction model based on the basic model adaptation, including a semantic feature extraction module, a feature aggregation module, and a 3D attribute prediction head; S3: The introduced additional error is to use the preprocessed adjacent frame pairs as training data to perform self-supervised training on the endoscopic 3D reconstruction model. S4: For endoscopic videos of arbitrary length, perform frame-by-frame reasoning using the trained endoscopic 3D reconstruction model to generate a 3D reconstructed view.

2. The method for real-time generalizable 3D reconstruction of endoscopic video using pure vision as described in claim 1, characterized in that, The adjacent frame pair includes a target frame at a current time and an adjacent source frame .

3. The pure vision endoscopic video real-time generalizable 3D reconstruction method of claim 2, wherein, In the endoscopic 3D reconstruction model, the semantic feature extraction module uses a pre-trained visual encoder to learn general visual representations and outputs semantic features. During the training phase, the parameters of this module are kept frozen. The feature aggregation module fuses intra-frame texture information and inter-frame geometric dependency through an intra-frame self-attention mechanism and a global self-attention mechanism, receives semantic features output by the semantic feature extraction module, and outputs aggregated spatiotemporal features .

4. The method for real-time generalizable 3D reconstruction of endoscopic video using pure vision as described in claim 3, characterized in that, The 3D attribute prediction head includes a depth prediction head and a camera prediction head; The depth prediction head generates a dense depth map based on aggregated spatiotemporal features, and obtains the target frame respectively. and source frame Corresponding target frame depth map and source frame depth map ; The camera prediction head outputs an initial pose estimate based on aggregated spatiotemporal features; A residual pose adaptation module is set after the camera prediction head. The residual is predicted and corrected by a two-stream fusion method of semantic flow and geometric flow. The initial pose is then calibrated twice to output the final camera pose.

5. The method for real-time generalizable 3D reconstruction of endoscopic video using pure vision as described in claim 4, characterized in that, The residual pose adaptation module calibrates the initial pose through a two-stream fusion mechanism. Its calculation logic includes: aggregating spatiotemporal features respectively. and initial pose Mapped to semantic latent variables and geometric latent variables; Semantic and geometric latent variables are concatenated and input into a residual fusion network to predict pose corrections. The pose corrections are then decomposed to obtain separate values ​​representing rotation corrections. Translation correction ; The initial pose is updated based on the pose correction, and the updated rotation matrix is: The updated translation vector is ,in, Represents the exponential mapping function, and The final camera pose is obtained by using the initial rotation matrix and translation vector, respectively. .

6. The method for real-time generalizable 3D reconstruction of endoscopic video using pure vision as described in claim 5, characterized in that, In step S3, a parallel dual-branch self-supervised training framework is constructed during the training phase, including a geometric branch for depth, pose, and geometric reprojection, and an appearance decoupling branch for appearance perturbation modeling. The geometry branch performs cross-view rigid reprojection based on the dense depth map output by the 3D reconstruction model and the final camera pose to generate a synthetic target view; the appearance decoupling branch performs appearance residual modeling on reflections, highlights, local brightness drift and non-rigid lighting changes in the target frame through the appearance flow decoupling network. By minimizing the total loss function, which includes appearance calibration photometric loss, semantic guidance smoothing loss, and appearance flow regularization loss, the network parameters of the endoscopic 3D reconstruction model and the adaptation parameters of the residual pose adaptation module are jointly optimized.

7. The method for real-time generalizable 3D reconstruction of endoscopic video using pure vision as described in claim 6, characterized in that, The calculation process for the appearance calibration photometric loss includes: setting up an appearance flow decoupling network, adopting an encoder-decoder structure, and outputting an appearance flow field with the same resolution as the target frame image based on the input adjacent frame pairs. The appearance decoupling branch employs a gradient isolation method, confining the appearance compensation gradient generated by the photometric loss during appearance calibration within the appearance decoupling branch and preventing it from propagating back to the geometry branch, thus achieving gradient decoupling between the appearance and geometry branches; utilizing the appearance flow field For the target frame image The images are added together for visual calibration to obtain the calibrated target image. ; In the geometry branch, the final camera pose is used. Source frame depth map And the camera intrinsic parameter matrix K, which will be the source frame image Reprojecting the image onto the target frame's viewpoint yields the composite target view. ; Based on the composite target view at the pixel pixel value at Target frame in pixels pixel value at and the calibrated target image In pixels The pixel value at that location is calculated. Firstly based on and The mixed loss of structural similarity (SSIM) and L1 distance is calculated, and then the mean is obtained to obtain the appearance calibration photometric loss. The construction process of the semantically guided smoothing loss includes: firstly, based on the source frame depth map... Constructing an inverse depth map Then, normalization is performed to obtain the normalized inverse depth map. ; Based on semantic features Semantic smoothing weights are constructed using the gradients in the horizontal and vertical directions, respectively; the horizontal semantic smoothing weight is... , represented as In the vertical direction, the vertical semantic smoothing weight is... , represented as ,in Represents the deep semantic feature map at the pixel level. The eigenvector at that location; and These represent the pixels along the horizontal and vertical directions, respectively. Gradient operator at the location; This is the semantic gradient sensitivity coefficient. Describing the L1 norm, and These represent the semantic features at the pixel level. The smoothing weights generated at the location; and These represent the normalized inverse depth map at pixels. The horizontal and vertical gradient magnitudes at the location, Represents inverse depth map In pixels The value at the location; the normalized inverse depth map at the pixel The horizontal and vertical gradient magnitudes at point [0, 1] are respectively related to the values ​​at pixel [0, 1] Smoothing weights generated at the point and We perform a weighted summation to obtain the smoothness; then we take the average of the smoothness of all pixels to obtain the semantically guided smoothing loss. The appearance flow regularization loss is defined as: the appearance flow field in pixels The absolute value of the residual and the appearance flow field at the pixel The absolute values ​​of the spatial gradients at each point are weighted and summed to obtain the appearance flow regularization degree. Then, the appearance flow regularization degree of all pixels is averaged to obtain the appearance flow regularization loss. The total loss function is obtained by weighted summation of appearance calibration photometric loss, semantic guided smoothing loss, and appearance flow regularization loss.

8. The method for real-time generalizable 3D reconstruction of endoscopic video using pure vision as described in claim 7, characterized in that, The specific implementation process of step S4 is as follows: For real-time acquired endoscopic video streams, any number of image frames are input into the trained endoscopic 3D reconstruction model. The dense depth map of the corresponding frame is output through the depth prediction head, and the camera pose of each frame is output through the camera prediction head and the residual pose adaptation module. Subsequently, based on the camera pose, the dense depth map of each frame is back-projected into a local 3D point cloud in the camera coordinate system, and each local point cloud is transformed to a unified world coordinate system. At the same time, the RGB color information of the original image is mapped to the corresponding 3D vertices, and the point cloud color attributes are assigned to reconstruct surface texture details; Finally, incremental fusion and rendering projection are performed on the time-accumulated colored point cloud to output a 3D reconstructed view from the current perspective in real time.

9. A purely visual endoscopic video real-time generalizable 3D reconstruction system, used to implement the 3D reconstruction method according to any one of claims 1 to 8, characterized in that, Includes the following modules: The adjacent frame pair construction module is used to acquire the image sequence of the endoscopic video, construct it into adjacent frame pairs with overlapping viewpoints, and preprocess the adjacent frame pairs; The endoscopic 3D reconstruction module is used to build an endoscopic 3D reconstruction model based on the base model, including a semantic feature extraction module, a feature aggregation module, and a 3D attribute prediction head. The self-supervised training module is used to introduce additional errors. It uses preprocessed adjacent frame pairs as training data to perform self-supervised training on the endoscopic 3D reconstruction model. The 3D reconstruction view output module is used to generate a 3D reconstruction view from endoscopic videos of arbitrary length by performing frame-by-frame inference through a trained endoscopic 3D reconstruction model.

Images