A monocular endoscope self-supervised generalization depth estimation method based on a DAE model
By using a self-supervised depth estimation method based on the DAE model, the problem of insufficient generalization ability of monocular depth estimation in endoscopic surgery is solved, and the high generalization ability of endoscopes in complex environments is achieved, supporting high-fidelity 3D reconstruction and precise surgical navigation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2026-02-06
- Publication Date
- 2026-07-24
AI Technical Summary
Existing monocular depth estimation methods have insufficient generalization ability in minimally invasive endoscopic surgery, making it difficult to achieve high-fidelity 3D reconstruction and precise surgical navigation in diverse endoscopic environments.
A self-supervised depth estimation method based on the DAE model is adopted, including a two-layer expert hybrid model, learnable gradient coordination and semantic distribution calibration. The model parameters are optimized by minimizing the total loss function. Combined with view synthesis, dynamic routing and customized depth guidance, the endoscope achieves high generalization ability in complex environments.
It achieves high generalization capability of endoscopes in complex environments, improves the reliability and accuracy of monocular depth estimation, and supports high-fidelity 3D reconstruction and precise surgical navigation in diverse endoscopic environments.
Smart Images
Figure CN122453891A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of depth estimation technology, and in particular to a self-supervised generalized depth estimation method for monocular endoscopes based on the DAE model. Background Technology
[0002] Monocular depth estimation plays a crucial role in minimally invasive endoscopic surgery, enabling high-fidelity 3D reconstruction, augmented reality experiences, and precise surgical navigation. However, due to concerns about safety, privacy, and surgical expertise, collecting large-scale, reliable real-world depth data for supervised model training remains extremely challenging. Given these practical limitations, various self-supervised methods have achieved significant breakthroughs by mitigating inter-frame brightness differences, utilizing structure-of-motion (SfM) to obtain sparse supervisory cues, and designing efficient network architectures. Nevertheless, these methods are still primarily limited to depth estimation in specific domains, inevitably restricting their generalization ability across diverse endoscopic environments.
[0003] Therefore, providing a self-supervised generalized depth estimation method for monocular endoscopes based on the DAE model to address the difficulties in existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0004] In view of this, the present invention provides a self-supervised generalized depth estimation method for monocular endoscopes based on the DAE model, which can achieve monocular depth estimation with high generalization ability in complex environments.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: A self-supervised generalized depth estimation method for monocular endoscopes based on a DAE model includes the following steps: Acquire endoscopic scene data and preprocess the scene data to obtain an initial dataset; A DAE model is constructed based on the principle of self-supervised deep estimation, including a two-layer expert hybrid model, learnable gradient coordination, and semantic distribution calibration. The initial dataset is input into the DAE model, and the model parameters are optimized by minimizing the total loss function to obtain the trained DAE model. The data to be detected is input into the trained DAE model to detect the depth of the target frame image.
[0006] Optionally, the self-supervised depth estimation principle includes: defining the learning process as novel view synthesis, minimizing the photometric error between the synthesized frame and the target frame, and including two networks: a depth network predicts depth from a single image, and a pose network estimates a 6-DOF pose transformation, setting the target frame f. t and source frame f s The view composition process is defined as follows: , in, This represents the pixel coordinate mapping from the target view t to the source view s, where Pt is the pixel coordinate in view t, and K is the camera's intrinsic parameter. It is a 6-DOFpose transformation matrix. From the target depth map, this transformation yields the deformed frame. : , in, This indicates a twist operation.
[0007] Optionally, a two-layer expert hybrid model includes two layers: the internal layer and the external layer. The MoE within the model achieves dynamic routing through a deep augmentation model, selecting LoRA experts of different levels and adapter experts of different kernel sizes based on input features; The external MoE provides customized deep guidance, with each expert following a static routing strategy determined by domain metrics.
[0008] Optionally, the MoE within the model includes a MoE-LoRA layer and a MoE-Adapter layer to obtain predictions; Each MoE-LoRA layer consists of multiple low-rank adaptation experts. Experts are dynamically selected and merged through a gating mechanism, and the output of the current layer is represented as follows: , , ; in, For the output of the MoE-LoRA layer, This represents the query, key, and value weights of the attention module in the pre-trained VFM, which remain unchanged during training. and This represents a trainable low-rank matrix. This represents the trainable weights of the selector; This is the temperature coefficient that controls the smoothness of the weights; it is set to 1. The MoE-Adapter layer consists of multiple adapter experts, and its output is determined based on the weights of the multilayer perceptron (MLP) in the pre-trained VFM.
[0009] Optionally, the external MoE introduces a domain-specific teacher hybrid expert model to provide direct, in-depth guidance; a static routing strategy determined by the discrete domain indicator D is employed to establish an input f. tThe system establishes a one-to-one mapping relationship between the expert and its corresponding expert EGz(·), and sets a scale decoupling loss function to reduce expert guidance error.
[0010] Optional, learnable gradient coordination includes generating a domain-specific coordination factor that adaptively aligns the gradient magnitudes of the deep network and the pose network to a matching range.
[0011] Optionally, semantic distribution calibration aligns model features with the high-level semantics of the pre-trained encoder, including: We extract features from the DAE encoder and the semantically guided encoder, set a feature space mapping function and minimize it to reduce feature transformation error, and normalize the features of the DAE encoder and the semantically guided encoder by combining a gated recurrent unit projector, and derive the semantic similarity measure in the semantic distribution latent space.
[0012] As can be seen from the above technical solutions, compared with the prior art, the present invention provides a self-supervised generalization depth estimation method for monocular endoscopy based on the DAE model, which has the following beneficial effects: 1) The present invention proposes a unified self-supervised depth estimation framework, which can achieve reliable generalization ability in various surgical operations using heterogeneous endoscopic data; 2) The present invention proposes a two-layer model enhancement (MoE) adaptation scheme to enable the Vision Foundation model to adapt to different endoscopic environments; 3) The present invention designs a learnable gradient coordination mechanism to stabilize joint depth-pose optimization, and combines it with a semantic distribution calibration module to enforce semantic consistency of depth prediction. Attached Figure Description
[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0014] Figure 1 This is a flowchart of a self-supervised generalized depth estimation method for monocular endoscopes based on the DAE model disclosed in this invention. Detailed Implementation
[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0016] Reference Figure 1 As shown, this invention discloses a self-supervised generalized depth estimation method for monocular endoscopes based on a DAE model, comprising the following steps: Acquire endoscopic scene data and preprocess the scene data to obtain an initial dataset; A DAE model is constructed based on the principle of self-supervised deep estimation, including a two-layer expert hybrid model, learnable gradient coordination, and semantic distribution calibration. The initial dataset is input into the DAE model, and the model parameters are optimized by minimizing the total loss function to obtain the trained DAE model. The data to be detected is input into the trained DAE model to detect the depth of the target frame image.
[0017] Furthermore, the self-supervised depth estimation principle includes: defining the learning process as novel view synthesis, minimizing the photometric error between the synthesized frame and the target frame, and including two networks: a depth network predicts depth from a single image, and a pose network estimates a 6-DOF pose transformation, setting the target frame f. t and source frame f s The view composition process is defined as follows: , in, P represents the pixel coordinate mapping from the target view t to the source view s. t These are the pixel coordinates in view t, and K is an intrinsic camera parameter. It is a 6-DOFpose transformation matrix. From the target depth map, this transformation yields the deformed frame. : , in, This indicates a twist operation. with f t The photometric differences between [15,54] were used as self-supervision. , in Indicates luminosity loss, Set to 0.85.
[0018] Furthermore, the two-layer expert hybrid model includes two levels: the internal level and the external level. The MoE within the model achieves dynamic routing through a deep augmentation model, selecting LoRA experts of different levels and adapter experts of different kernel sizes based on input features; The external MoE provides customized deep guidance, with each expert following a static routing strategy determined by domain metrics.
[0019] Furthermore, the MoE within the model includes the MoE-LoRA layer and the MoE-Adapter layer; The MoE-LoRA layer includes: integrating LoRA into an expert hybrid architecture, where multiple LoRA modules with different ranks r act as experts. A lightweight selector SL(·) queries based on the input. Based on its characteristics, it dynamically routes the data to the optimal expert. Each MoE-LoRA layer consists of multiple low-rank adapting experts. Experts are dynamically selected and merged through a gating mechanism. The output of the current layer is represented as follows: , , ; in, For the output of the MoE-LoRA layer, This represents the query, key, and value weights of the attention module in the pre-trained VFM, which remain unchanged during training. and This represents a trainable low-rank matrix. This represents the trainable weights of the selector; This is the temperature coefficient that controls the smoothness of the weights; it is set to 1. The MoE-Adapter layer consists of multiple adapter experts, and its output is determined based on the weights of the multilayer perceptron (MLP) in the pre-trained VFM.
[0020] Furthermore, since the visual feature model based on Visual Transformer (ViT) inherently lacks inductive bias, a convolutional adapter is introduced to capture local visual patterns and structural details, forming the MoE-Adapter layer. Similar to the routing strategy in the MoE-LoRA layer, a selector SA(·) adaptively routes the input query to the adapter expert. Each MoE-Adapter layer is assumed to contain M experts, and the output of this layer is: , in These are the frozen weights of the multilayer perceptron (MLP) in the pre-trained visual manifold model (VFM). Each adapter expert is implemented as a convolutional block. To reduce computational overhead, the input is first reduced in channel dimension by 1×1 convolution, then processed by expert-specific convolution (including the corresponding kernel size), and finally restored to the original channel dimension by 1×1 convolution.
[0021] Furthermore, the external MoE introduces a domain-specific teacher hybrid expert model to provide direct, in-depth guidance; a static routing strategy determined by the discrete domain indicator D is employed to establish an input f tCorresponding guidance experts The one-to-one mapping relationship between (·) is expressed as follows: ,in This represents a specific metric value. Each domain expert model employs a self-supervised approach and is pre-trained on corresponding domain data. However, given the inherent scale ambiguity of monocular systems, to prevent deep augmentation expert models from becoming biased towards the expert scale, a scale decoupling loss function is set to ensure that the model learns meaningful depth guidance. This loss function consists of two components: scale-invariant loss and ranking loss.
[0022] Furthermore, given the depth prediction and its corresponding guidance, the scale-invariant loss [] is expressed as: , , , Here, ∈ avoids the division by zero operation.
[0023] The ranking loss is used to enforce depth order consistency among sampled pairs by eliminating scale bias while utilizing expert-guided ordinal cues. Given a set of Q pairs... Depth (d) i，1 d i，2 ) and guidance depth ( , The loss is defined as: , , , in, The set tolerance threshold is 0.03. To reduce expert guidance error, the consistency between the predicted depth and the guidance depth is calculated, and the top 10% of pixels with the largest error are removed to avoid misleading the training signal.
[0024] Furthermore, learnable gradient coordination includes generating a domain-specific coordination factor that adaptively aligns the gradient magnitudes of the deep network and the pose network to a matching range.
[0025] Specifically, given domain metrics associated with the input data. ∈D is first mapped to a learnable domain embedding through an embedding layer, and then this embedding is input into an MLP layer to generate a domain-specific coordination factor. This factor is applied to the output of the pose network in the following manner: , And light loss The gradient with respect to M becomes: ,θ, When displayed This indicates the coordination factor By directly adjusting the gradient magnitude of the pose network, the gradient magnitudes of the deep network and the pose network are adaptively aligned to a matching range, thereby coordinating their optimization contributions.
[0026] Furthermore, semantic distribution calibration aligns model features with the high-level semantics of the pre-trained encoder, including: We extract features from the DAE encoder and the semantically guided encoder, set a feature space mapping function and minimize it to reduce feature transformation error, and normalize the features of the DAE encoder and the semantically guided encoder by combining a gated recurrent unit projector, and derive the semantic similarity measure in the semantic distribution latent space.
[0027] Specifically, DINOv3 is used as the semantic guided encoder. For a given f t The semantic prior features are The features extracted by the DAE encoder are F t The feature space mapping function is expressed as: , in These are learnable parameters of T(·). To instantiate T(·), a gated cyclic unit projector is used, and its hidden state is initialized to tanh-activated. Original As input, and with the iteration count set to 2, the projected features are then... and F t Normalization along the feature dimension yields the feature distribution. and Using KL divergence to measure the difference between them, the semantic distribution correction loss is defined as: = .
[0028] Furthermore, the total loss function is expressed as follows: , in λ represents the edge-aware smoothing loss, and λ1, λ2, λ3 and λ4 are the weights of the corresponding loss functions, specifically set to 0.1, 0.01, 0.01 and 0.001.
[0029] In one specific embodiment, the framework proposed in this application is implemented in PyTorch and trained on a single NVIDIA RTX A5000 GPU using the AdamW optimizer. The learning rate is initialized to 1×10⁻⁴ and reduced by 0.1x after 10 epochs. Data augmentation strategies are employed, and the model is trained for 20 epochs with a batch size of 8. In the MoE-LoRA and MoE-Adapter layers, the number of experts is set to 4, and the Top-1 expert is selected as the optimal solution for each input. The predefined ranks of MoE-LoRA are 4, 8, 16, and 32, while the kernel sizes of MoE-Adapter are 3, 5, 7, and 9. The adjustable layers in VFMDepthAnythingv2 are [2, 4, 5, 7, 8, 10, 11], and the remaining layers are fixed. The results are evaluated using standard evaluation metrics.
[0030] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A self-supervised generalized depth estimation method for monocular endoscopes based on a DAE model, characterized in that, Includes the following steps: Acquire endoscopic scene data and preprocess the scene data to obtain an initial dataset; A DAE model is constructed based on the principle of self-supervised deep estimation, including a two-layer expert hybrid model, learnable gradient coordination, and semantic distribution calibration. The initial dataset is input into the DAE model, and the model parameters are optimized by minimizing the total loss function to obtain the trained DAE model. The data to be detected is input into the trained DAE model to detect the depth of the target frame image.
2. The self-supervised generalization depth estimation method for monocular endoscope based on the DAE model according to claim 1, characterized in that, The principle of self-supervised depth estimation includes: defining the learning process as novel view synthesis, minimizing the photometric error between the synthesized frame and the target frame, and including two networks: a depth network predicts depth from a single image, and a pose network estimates a 6-DOF pose transformation, setting the target frame f. t and source frame f s The view composition process is defined as follows: ， in, P represents the pixel coordinate mapping from the target view t to the source view s. t These are the pixel coordinates in view t, and K is an intrinsic camera parameter. It is a 6-DOFpose transformation matrix. From the target depth map, this transformation yields the deformed frame. : ， in, This indicates a twist operation.
3. The self-supervised generalization depth estimation method for monocular endoscope based on the DAE model according to claim 1, characterized in that, The two-layer expert hybrid model includes two layers: the internal layer and the external layer. The MoE within the model achieves dynamic routing through a deep augmentation model, selecting LoRA experts of different levels and adapter experts of different kernel sizes based on input features; The external MoE provides customized deep guidance, with each expert following a static routing strategy determined by domain metrics.
4. The self-supervised generalization depth estimation method for monocular endoscope based on the DAE model according to claim 3, characterized in that, The MoE within the model includes the MoE-LoRA layer and the MoE-Adapter layer to obtain predictions; Each MoE-LoRA layer consists of multiple low-rank adaptation experts. Experts are dynamically selected and merged through a gating mechanism, and the output of the current layer is represented as follows: ， ， ； in, For the output of the MoE-LoRA layer, This represents the query, key, and value weights of the attention module in the pre-trained VFM, which remain unchanged during training. and This represents a trainable low-rank matrix. This represents the trainable weights of the selector; This is the temperature coefficient that controls the smoothness of the weights; it is set to 1. The MoE-Adapter layer consists of multiple adapter experts, and its output is determined based on the weights of the multilayer perceptron (MLP) in the pre-trained VFM.
5. The self-supervised generalization depth estimation method for monocular endoscope based on the DAE model according to claim 4, characterized in that, The external MoE introduces a domain-specific teacher hybrid expert model to provide direct, in-depth guidance; it employs a static routing strategy determined by the discrete domain indicator D to establish an input f. t The system establishes a one-to-one mapping relationship between the expert and its corresponding expert EGz(·), and sets a scale decoupling loss function to reduce expert guidance error.
6. The self-supervised generalization depth estimation method for monocular endoscope based on the DAE model according to claim 2, characterized in that, Learnable gradient coordination includes generating domain-specific coordination factors that adaptively align the gradient magnitudes of deep networks and pose networks to a matching range.
7. The self-supervised generalization depth estimation method for monocular endoscope based on the DAE model according to claim 1, characterized in that, Semantic distribution calibration aligns model features with the high-level semantics of the pre-trained encoder, including: We extract features from the DAE encoder and the semantically guided encoder, set a feature space mapping function and minimize it to reduce feature transformation error, and normalize the features of the DAE encoder and the semantically guided encoder by combining a gated recurrent unit projector, and derive the semantic similarity measure in the semantic distribution latent space.