Endoscope monocular image depth estimation method and system

Through the DA-ICGA model, combined with multiple modules and loss functions, the artifact problem of overexposed areas in endoscopic images is solved, and the accuracy and robustness of depth estimation are improved, especially showing significant improvement in overexposed areas.

CN120655691AActive Publication Date: 2025-09-16JIANGNAN UNIV

Patent Information

Application Number
CN202510709164.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-29
Publication Date
2025-09-16
Estimated Expiration
2045-05-29

AI Technical Summary

Technical Problem

Existing deep learning-based methods cannot effectively solve the artifact problem caused by overexposed areas in endoscopic images, resulting in low accuracy and insufficient robustness of monocular depth estimation.

Method used

The DA-ICGA model is used, combined with the PoseNet module, DAM-LoRA module, dual-branch geometric perception module and image-level contrast learning module. By constructing multiple loss functions and self-supervised learning, the ability to model deep features under complex lighting conditions is enhanced, artifacts are reduced and robustness is improved.

Benefits of technology

The accuracy and robustness of depth estimation in endoscope monocular images are significantly improved, especially showing good depth estimation effect in overexposed areas, and enhancing the adaptability of the model in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120655691A_ABST
    Figure CN120655691A_ABST
Patent Text Reader

Abstract

The invention relates to an endoscope monocular image depth estimation method and system, and the method comprises the steps: obtaining the front and rear frame images of an endoscope monocular; an endoscope monocular is trained by using a DA-I CGA model based on a DARES, the DA-I CGA model comprises a PoseNet module and a DAM-LoRA module, and a double-branch geometric perception module and an image-level contrast learning module are creatively introduced; and performing depth estimation on the to-be-detected endoscope monocular image through the trained DA-I CGA model. According to the method, artifacts generated in an overexposure region can be effectively solved, and the monocular depth estimation precision is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of endoscope monocular image depth estimation, and in particular to an endoscope monocular image depth estimation method and system. Background Art

[0002] Robotic-assisted endoscopic surgery (RAS) has significantly improved the accuracy and flexibility of minimally invasive surgery. Its precise navigation and tissue reconstruction are highly dependent on depth perception. Monocular image depth estimation technology, as a key technology for recovering depth information from a single endoscopic image, plays a vital role in the surgical process. Traditional monocular image depth estimation methods mainly rely on illumination consistency constraints and geometric consistency constraints. However, in endoscopic scenes, due to non-Lambertian surfaces and dynamic illumination changes, illumination consistency constraints easily fail in areas lacking rich textures, making it difficult for the model to learn effective depth information. In addition, methods relying on geometric constraints also face two challenges in this scene. First, the lack of fixed scale information in the monocular perspective makes the depth estimation results prone to scale ambiguity. Second, the presence of overexposed areas in endoscopic images makes it difficult to stabilize geometric constraints based on feature matching, which will introduce depth ambiguity and artifacts. These factors ultimately make it difficult for traditional methods to achieve good results. Deep learning-based methods have been widely explored for depth estimation in monocular endoscopic images. One of the most representative approaches is self-supervised depth estimation based on optical flow, exemplified by AF-SfMLearner. These methods leverage motion cues to infer the scene's depth structure by learning optical flow or appearance flow information between adjacent frames. Compared to traditional illumination consistency-based approaches, optical flow-based deep learning methods not only focus on pixel motion but also consider brightness variations. However, due to the highly dynamic tissue surfaces and drastic illumination variations in endoscopic scenes, these methods are often susceptible to motion inconsistencies, leading to artifacts in depth estimation. Furthermore, optical flow calculation relies on pixel matching, which makes it difficult to stably match feature points in overexposed areas, further reducing the accuracy and robustness of depth estimation. In recent years, the development of the depth-anything model (DAM), a foundational model for depth estimation in monocular images, has enabled high-accuracy depth recovery in natural scenes. These methods rely on large-scale datasets for training and offer advantages such as data-driven representation learning, strong generalization, and the absence of handcrafted features. These methods mitigate artifacts associated with reliance on illumination consistency. Researchers have introduced deep object models (DAMs) into the endoscopy field, such as Surgical-DINO, which has successfully expanded the sources and coverage of endoscopic data and provided more diverse scene information for model training. However, DAMs are primarily based on data-driven deep learning and lack geometric consistency constraints, resulting in significant depth estimation errors in endoscopic environments. This is particularly true in overexposed areas, where specular reflections can destroy depth information, making it difficult for the model to accurately recover 3D structures and introducing severe artifacts.In summary, although the overexposure problem remains a common challenge for the above two methods, since DAM is pre-trained on a large-scale and diverse depth dataset, it is easier to learn the depth bias of the exposed area than the optical flow method and the results are more robust.

[0003] DARES is one of the first studies to adapt the baseline DAM V2 (DepthAnything Model V2) model to the endoscopy domain through self-supervised learning, without requiring extensive labeled data. It successfully improves depth estimation accuracy and achieves state-of-the-art performance. DARES's innovation lies in leveraging the pre-trained DAMV2 model and introducing a context-aware mechanism to enhance modeling of depth features under complex lighting conditions and reduce depth artifacts caused by local illumination variations. Furthermore, DARES incorporates low-rank vector adaptation (LoRA) to optimize the allocation of model parameters, reducing computational complexity and alleviating the extremely time-consuming nature of the baseline DAM model. It also mitigates catastrophic forgetting, improves the stability and generalization of self-supervised learning, and partially alleviates artifacts. However, effective artifact suppression remains difficult in overexposed areas, demonstrating that even powerful pre-trained models and optimized self-supervised learning strategies cannot fully address artifacts caused by illumination environments unique to endoscopic images, such as specular reflections, non-Lambertian reflectance, and dynamic illumination variations. An analysis of DARES's network structure revealed that, despite its context-aware mechanism, DARES primarily focuses on photometric consistency in local regions and lacks global illumination adaptability. Furthermore, because DARES primarily relies on data-driven depth estimation and fails to explicitly model geometric information, the lack of geometric consistency constraints makes it difficult to correctly infer depth relationships within the exposed region and results in depth artifacts at tissue boundaries. These issues collectively limit the accuracy and robustness of DARES for monocular endoscopic depth estimation, particularly in clinical environments with complex lighting conditions.

[0004] In summary, although there has been a lot of research on monocular image depth estimation, deep learning-based methods (such as the DARES-based method) still cannot effectively solve the artifact problem caused by overexposed areas, which restricts the accuracy and robustness of DARES in endoscope monocular depth estimation. Summary of the Invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem that the deep learning-based methods in the existing technology are still unable to effectively solve the artifacts generated by overexposed areas, which in turn leads to low accuracy of monocular depth estimation.

[0006] To solve the above technical problems, the present invention provides an endoscope monocular image depth estimation method, comprising:

[0007] Acquire single front and back frame images of the endoscope;

[0008] The DA-ICGA model is trained on the endoscope monocular front and back frame images. The DA-ICGA model is based on DARES and includes a PoseNet module, a DAM-LoRA module, a dual-branch geometric perception module, and an image-level contrast learning module.

[0009] The DAM-LoRA module is used to process the front and back frame images to generate four first depth maps with different resolutions;

[0010] The PoseNet module is used to process the front and back frame images and the four first depth maps of different resolutions generated by the DAM-LoRA module to obtain synthesized adjacent frame images;

[0011] Constructing a first loss function based on the endoscope single front and back frame images and the synthesized adjacent frame images;

[0012] The image-level contrast learning module is used to generate a second depth map based on the previous and next frame images;

[0013] Constructing a second loss function based on the second depth map and four first depth maps of different resolutions;

[0014] The dual-branch geometric perception module is used to extract geometric prior and texture features of the previous and next frame images, and generate a third depth map having the geometric prior and texture features;

[0015] Constructing a third loss function based on the third depth map and four first depth maps of different resolutions;

[0016] If the first, second, and third loss functions all reach their respective preset values, the model converges and training ends; otherwise, the model continues to be trained until the model converges;

[0017] The depth of the monocular image of the endoscope to be inspected is estimated using the trained DA-ICGA model.

[0018] In one embodiment of the present invention, the method for the PoseNet module to process the preceding and following frame images and the four first depth maps of different resolutions generated by the DAM-LoRA module to obtain synthesized adjacent frame images includes:

[0019] The PoseNet module is used to input the previous and next frame images into the ResNet network to obtain camera pose information, use the camera pose information as a change matrix, and perform an inverse transformation operation on the four first depth maps of different resolutions output by the DAM-LoRA module through the change matrix to generate synthetic adjacent frame images.

[0020] In one embodiment of the present invention, the method for the DAM-LoRA module to process the preceding and following frame images to generate four first depth maps of different resolutions includes:

[0021] The DAM-LoRA module includes a ViT encoder and a DPT decoder. The ViT encoder is used to extract visual features of the front and back frame images. The DPT decoder is used to perform depth regression on the visual features of the front and back frame images extracted by the ViT encoder to generate four first depth maps of different resolutions, wherein:

[0022] The ViT encoder includes 12 LoRA Transformer modules, each of which is connected to an MLP layer and a normalization layer, and dynamically adjusts the parameter allocation of each LoRA Transformer module using the Fine-tuning strategy through Vector Low-Rank Adaptation Vector-LoRA;

[0023] The DPT decoder includes a Neck module and a Head module. The Neck module processes and integrates the features output by the ViT encoder; the Head module is used to perform depth regression on the features processed and integrated by the Neck module to generate four first depth maps of different resolutions.

[0024] In one embodiment of the present invention, the method for the image-level contrast learning module to generate a second depth map based on previous and next frame images includes:

[0025] The image-level contrast learning module performs binary Gaussian masking on each pixel in the previous and next frame images to generate image-level contrast samples, wherein the binary Gaussian mask formula is:

[0026]

[0027] Among them, M o(i,j) is the mask value of each pixel in the source image; τ is the preset threshold; M o is the occlusion mask; (i, j) is the pixel in the source image; (x0, y0) is the mean coordinate of the Gaussian distribution; σ is the standard deviation of the Gaussian distribution;

[0028] The image-level contrast samples are input into a depth estimation network to generate a second depth map.

[0029] In one embodiment of the present invention, the method for extracting geometric priors and texture features of the preceding and following frame images by the dual-branch geometric perception module and generating a third depth map having the geometric priors and texture features includes:

[0030] The dual-branch geometric perception module includes a first branch and a second branch;

[0031] The first branch is used to extract the geometric prior of the image, and the second branch is used to extract the texture features of the image through the FPN network;

[0032] The first branch includes a first convolutional layer, two first ResBlock layers, two second ResBlock layers, two third ResBlock layers, a first De-ResBlock layer, a second De-ResBlock layer, a third De-ResBlock layer and a second convolutional layer connected in sequence, and the texture features of the image are fused with the outputs of the first De-ResBlock layer, the second De-ResBlock layer and the third De-ResBlock layer respectively.

[0033] In one embodiment of the present invention, the first ResBlock layer, the second ResBlock layer, and the third ResBlock layer each include two 3x3 convolutions, and the output of the first 3x3 convolution and the output of the second 3x3 convolution are added, and then connected to a ReLU layer after the addition;

[0034] The first De-ResBlock layer, the second De-ResBlock layer, and the third De-ResBlock layer each include three 5*5 convolutions connected in sequence.

[0035] In one embodiment of the present invention, the first loss function is constructed based on the single front and back frame images of the endoscope and the synthesized adjacent frame images, and the formula is:

[0036] L ms-reproj =α·(1-MS-SSIM(I target ,I estimate ))+β·|I target -I estimate |

[0037]

[0038] Among them, MS_SSIM(x,y) is the multi-scale structural similarity index; l M (x,y),c j (x,y) and s j (x,y) represents brightness, contrast and structural similarity respectively; α M , βj and γ j is the weight used to adjust the relative importance of different components; j is the scale index in the MS_SSIM(x,y) calculation, which is an integer from 1 to M; M is the final scale in the MS_SSIM(x,y) calculation; α is the first weight parameter; β is the second weight parameter; I target I is the original single front and back frame image of the endoscope; estimate It is a composite image of adjacent frames.

[0039] In one embodiment of the present invention, the second loss function is constructed based on the second depth map and four first depth maps with different resolutions, and the formula is:

[0040]

[0041] Among them, D R and D IC They represent the four first depth maps output by the DAM-LoRA module and the second depth map output by the image-level contrast learning module; M c Confidence mask generated by the DAM-LoRA module in depth estimation.

[0042] In one embodiment of the present invention, the third loss function is constructed based on the third depth map and four first depth maps with different resolutions, and the formula is:

[0043]

[0044] Wherein, γ represents the intersection of depth intervals determined by the first depth map and the third depth map; is the depth distribution after filtering, which represents the third depth map generated by the dual-branch geometric perception module; N GT (z) is the distribution of the true depth, representing the four first depth maps output by the DAM-LoRA module; m is the discrete interval index after dividing the scene depth space; z is the pixel point in the image; M′ represents the division of the depth space of each scene into several discrete intervals.

[0045] To solve the above technical problems, the present invention provides an endoscope monocular image depth estimation system, comprising:

[0046] Acquisition module: used to obtain single front and back frame images of the endoscope;

[0047] Training module: used to train the DA-ICGA model for the endoscope monocular using the front and back frame images of the endoscope monocular. The DA-ICGA model is based on DARES and includes the PoseNet module, the DAM-LoRA module, the dual-branch geometric perception module and the image-level contrast learning module.

[0048] The DAM-LoRA module is used to process the front and back frame images to generate four depth maps with different resolutions;

[0049] The PoseNet module is used to process the front and back frame images and the four first depth maps of different resolutions generated by the DAM-LoRA module to obtain synthesized adjacent frame images;

[0050] Constructing a first loss function based on the endoscope single front and back frame images and the synthesized adjacent frame images;

[0051] The image-level contrast learning module is used to generate image-level contrast samples based on the previous and next frame images, and use the image-level contrast samples as the second depth map;

[0052] Constructing a second loss function based on the second depth map and four first depth maps of different resolutions;

[0053] The dual-branch geometric perception module is used to extract geometric prior and texture features of the previous and next frame images, and generate a third depth map having the geometric prior and texture features;

[0054] Constructing a third loss function based on the third depth map and four first depth maps of different resolutions;

[0055] If the first, second, and third loss functions all reach their respective preset range values, the model converges and training ends; otherwise, the model continues to be trained until the model converges;

[0056] Depth estimation module: used to perform depth estimation on the monocular image of the endoscope to be inspected using the trained DA-ICGA model.

[0057] The above technical solution of the present invention has the following advantages over the prior art:

[0058] The endoscope monocular image depth estimation method constructed by the present invention can solve the common artifact problem in overexposed areas, thereby improving the accuracy and robustness of depth estimation;

[0059] The present invention introduces a geometric perception module and an image-level contrast learning module into the DARES-based DA-ICGA model, enabling the DA-ICGA model to effectively correct texture loss and blurring caused by overexposure, thereby enhancing the depth estimation accuracy in high-contrast areas.

[0060] Experimental results verify that the method of the present invention can significantly improve the accuracy of depth estimation, especially in overexposed areas, where the depth estimation model exhibits good robustness and high reconstruction quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0062] Figure 1 is a flow chart of the method of the present invention;

[0063] Figure 2 is an architectural diagram of the DA-ICGA model in an embodiment of the present invention;

[0064] Figure 3 is a schematic diagram of comparison between conventional samples and image-level comparison samples in an embodiment of the present invention;

[0065] Figure 4 2 is a schematic diagram of a dual-branch geometric perception module according to an embodiment of the present invention;

[0066] Figure 5 2. It is a schematic diagram for visual comparison of depth estimation results of the exposure area of ​​the present invention and the existing method;

[0067] Figure 6 This is a visual comparison diagram of the generalization ability of the present invention and the existing method. DETAILED DESCRIPTION

[0068] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0069] Example 1

[0070] Reference Figure 1 The present invention relates to a method for estimating depth of an endoscope monocular image, comprising:

[0071] Acquire a single front and back frame image of the endoscope, wherein the front and back frame images include a current frame image and adjacent frame images;

[0072] The DA-ICGA model is trained on the endoscope monocular front and back frame images. The DA-ICGA model is based on DARES and includes a PoseNet module, a DAM-LoRA module, a dual-branch geometric perception module, and an image-level contrast learning module.

[0073] The DAM-LoRA module is used to process the front and back frame images to generate four first depth maps with different resolutions;

[0074] The PoseNet module is used to process the front and back frame images and the four first depth maps of different resolutions generated by the DAM-LoRA module to obtain synthesized adjacent frame images;

[0075] Constructing a first loss function based on the endoscope single front and back frame images and the synthesized adjacent frame images;

[0076] The image-level contrast learning module is used to generate a second depth map based on the previous and next frame images;

[0077] Constructing a second loss function based on the second depth map and four first depth maps of different resolutions;

[0078] The dual-branch geometric perception module is used to extract geometric prior and texture features of the previous and next frame images, and generate a third depth map having the geometric prior and texture features;

[0079] Constructing a third loss function based on the third depth map and four first depth maps of different resolutions;

[0080] If the first, second, and third loss functions all reach their respective preset values, the model converges and training ends; otherwise, the model continues to be trained until the model converges;

[0081] The depth of the monocular image of the endoscope to be inspected is estimated using the trained DA-ICGA model.

[0082] It should be noted that after the DA-ICGA model training is completed, when performing depth estimation on the actual endoscope monocular image (RGB image) to be detected, the DAM-LoRA module can directly generate a depth map and use it as the final result.

[0083] The following is a detailed introduction to this embodiment:

[0084] In this embodiment, a DA-ICGA (Depth Anything with Image-level Contrastive and Geometry-aware) model is proposed to combat overexposure artifacts in order to solve the problem of artifacts in overexposed areas in endoscopic images. The DA-ICGA model is based on DARES. The innovation of DARES is that in order to solve the problem of unreasonable parameter allocation and catastrophic forgetting when the basic model DAM is directly applied in the endoscopic scene, a new adaptive technology, Vector Low-Rank Adaptation (Vector-LoRA), is used to effectively optimize the model parameter allocation. However, although DARES is one of the first methods to introduce DAM as a basic model into the field of endoscopy, it has opened up a new direction for this field, but it still does not solve the problem of artifacts in overexposed areas in endoscopic images. To this end, this embodiment uses DARES as the basic framework and introduces an image-level contrast learning module and a dual-branch geometric perception module. The architecture of the DA-ICGA model in this embodiment is as follows: Figure 2As shown in the figure, it includes a PoseNet module, a DAM-LoRA module, a dual-branch geometric perception module, and an image-level contrast learning module. During model training, the previous and next frame images must be input into the four modules.

[0085] The infrastructure of the DA-ICGA model is mainly composed of the PoseNet module for pose estimation of DARES and the DAM-LoRA module for depth estimation, which uses multi-scale SSIM reprojection loss for self-supervised training. In terms of pose estimation, the PoseNet module is responsible for calculating the camera's ego-motion, that is, the camera pose change (camera pose) between adjacent frames. The PoseNet module takes the current frame and adjacent frames as input, infers their relative pose, and uses this information for subsequent calculation of the first loss function. In terms of depth estimation, the DAM-LoRA module, as the core component of the DA-ICGA model, is based on the Transformer structure and includes a ViT encoder for extracting image visual features (visual features include edges, textures, color distribution; as well as organ structure, tissue morphology, and spatial relationships between image regions) and a DPT decoder for dense prediction of feature-to-depth map conversion. To optimize model adaptability, this embodiment introduces the Vector-LoRA low-rank adaptive technology in the DAM-LoRA module to dynamically adjust the parameter allocation of the 12 multi-head self-attention blocks (LoRATransformer module) of the ViT encoder. Unlike traditional LoRA, Vector-LoRA improves the model's adaptability across different feature levels by allocating parameters across different layers through variable rank. In the Transformer architecture, each LoRA Transformer module includes Q (query), K (key), V (value), and Softmax normalization. This aims to establish correlations between input features through a self-attention mechanism, thereby improving deep feature extraction. Furthermore, the DAM-LoRA module employs a fine-tuning strategy to adapt to endoscopic surgery scenarios. Parameters in certain layers are frozen, preventing updates. This approach effectively preserves common features while reducing computational overhead and mitigating the risk of overfitting. The first loss function (the multi-scale SSIM reprojection loss) plays a core optimization role in the DA-ICGA model. This loss function optimizes depth and pose estimation by comparing the differences between the original image and the synthesized image. Unlike the traditional L1 loss, the multi-scale SSIM reprojection loss additionally considers image brightness, contrast, and structural information, making it more adaptable to the complex textures and illumination variations found in endoscopic images, thereby improving the model's robustness in real-world surgical environments.

[0086] The DA-ICGA model also includes an image-level contrast learning module and a dual-branch geometric perception module. Specifically, although the DARES model has made pioneering progress in solving the problem of unreasonable parameter allocation of monocular depth estimation in endoscopic scenes, the depth estimation effect is still not ideal when facing the artifact problem caused by overexposed areas. In order to solve this problem, this embodiment makes specific improvements on the basic architecture of DARES, namely, adding an image-level contrast learning module and a dual-branch geometric perception module. The introduction of the image-level contrast learning module is mainly to make up for the insufficient utilization of global context information by DARES to solve the problem of inaccurate depth estimation in overexposed areas. The previous model mainly relied on the photometric consistency assumption, but lacked sufficient texture information in the overexposed area, which made it difficult for the model to obtain effective supervision signals, thereby affecting the accuracy of depth estimation. The image-level contrast learning module uses a Gaussian mask to generate image-level contrast samples. Although the Gaussian mask will cause the loss of local texture information, the global context information still exists, which can capture the depth changes in the overexposed area and provide important depth estimation clues for the model. By calculating the depth estimation difference between the regular sample and the contrast sample, L ICC (Image-level Contrastive Consistency Loss) loss (i.e., the second loss function) encourages the model to enhance the utilization of global contextual information during deep learning, thereby performing more stable depth inference in overexposed areas. The dual-branch geometric perception module is designed to enhance DARES's ability to learn geometric information, thereby reducing artifacts in overexposed areas and improving the structural consistency of depth estimation. The dual-branch geometric perception module introduces two branches: one branch extracts image texture features, and the other extracts the geometric prior of the image. The image texture features and the geometric prior of the image are added element by element to further enrich the expression of geometric information. This module can better capture the details of geometric perception in the depth estimation process, thereby significantly improving the performance of the model in complex environments.

[0087] The following is a detailed introduction to the DA-ICGA model:

[0088] (1) Basic network structure of the DA-ICGA model (PoseNet module and DAM-LoRA module)

[0089] DARES, the infrastructure of the DA-ICGA model, includes a PoseNet module and a DAM-LoRA module. The PoseNet module is primarily used for pose estimation. During training, it works in conjunction with the DAM-LoRA module to provide critical pose information for depth estimation and assist in generating synthetic images. During the evaluation phase, it, along with the depth estimation results, provides fundamental data for applications such as 3D reconstruction, ensuring the model accurately perceives the spatial position and motion of images. The DAM-LoRA module includes a ViT encoder and a DPT decoder. The ViT encoder includes 12 LoRA Transformer modules, each of which is connected to an MLP layer and a normalization layer, and dynamically adjusts the parameter allocation of each LoRA Transformer module using the fine-tuning strategy through vector low-rank adaptation Vector-LoRA; the DPT decoder includes a Neck module and a Head module, and the Neck module processes and integrates the features output by the ViT encoder; the Head module is used to perform deep regression on the features processed and integrated by the Neck module to generate 4 first depth maps of different resolutions, that is, the DPT decoder completes the depth regression task through a series of convolutional and upsampling layers of the Neck module and the Head module, providing basic data support for subsequent depth estimation. This embodiment adds a Vector-LoRA (Vector Low-Rank Adaptation, Vector-LoRA) layer to the 12 LoRA Transformer modules of the ViT encoder, and assigns different ranks to different layers according to the hierarchical characteristics of the neural network. This design enables the network to better adapt to the feature learning requirements of different levels during the learning process. The early layers can use more parameters to learn general features, and the later layers are assigned appropriate parameters for more targeted task detail learning, which effectively improves the model's adaptability and learning efficiency for specific tasks and avoids the drawbacks of uniform parameter distribution in the traditional LoRA method.

[0090] (1) PoseNet module

[0091] like Figure 2 As shown in the figure, the input part of the PoseNet module (i.e., the pose estimation network) contains the current frame image Current (I t ) and adjacent frame images Adjacent(I t+1Specifically, the PoseNet module is used to process the front and back frame images and the four first depth maps of different resolutions generated by the DAM-LoRA module to obtain synthesized adjacent frame images. The specific method includes: the PoseNet module is used to input the front and back frame images into the ResNet network to obtain camera pose information (CameraPose), and the camera pose information is used as a change matrix. The four first depth maps output by the DAM-LoRA module are inversely transformed (Inverse Warping) by the change matrix to generate synthesized adjacent frame images I t ' +1 (Adjacent Syntheticframe).

[0092] (2) DAM-LoRA module

[0093] like Figure 2 As shown, the input part of the DAM-LoRA module contains the current frame image Current (I t ) and adjacent frame images Adjacent(I t+1 ), the two work together to provide the model with image information at adjacent moments, and improve the accuracy of depth estimation tasks by means of image association in time series. The DAM-LoRA module consists of a ViT encoder and a DPT decoder, where

[0094] The DPT decoder includes a Neck module and a Head module, both of which include several convolution and upsampling layers. The Neck module processes and integrates the features output by the ViT encoder; the Head module is used to perform deep regression on the features processed and integrated by the Neck module, calculate and predict four depth maps of different resolutions, and provide depth information of different levels of fineness to meet the requirements of different application scenarios for the fineness of depth information.

[0095] It should be noted that since the ViT encoder (including 12 LoRA Transformer modules) and the DPT decoder (including the Neck module and the Head module) in the DAM-LoRA module are existing modules in the DARES model, that is, they belong to the prior art, this embodiment will not go into details about their structural features.

[0096] The first loss function

[0097] In the field of self-supervised monocular depth estimation, most works usually use a combination of the structural similarity index SSIM and L1 loss to construct the reprojection error. The first loss function is used to calculate the similarity between images in terms of brightness, contrast, and structure. However, in scenarios such as robot-assisted surgery, there are special cases such as highly complex tissue textures, changing lighting conditions, and motion blur. Therefore, the DARES model proposes a multi-scale SSIM (Multiscale-SSIM, MS_SSIM) reprojection loss combined with a Tikhonov regularization term. The Multiscale-SSIM reprojection loss is used to measure the similarity between the original image and the synthesized image, and the Tikhonov regularization term is used to prevent the model from overfitting. During the training process, the model adjusts its own parameters by minimizing this loss function. The two terms in the first loss function measure the difference between the estimated image and the target image from different perspectives. MS_SSIM performs multi-scale processing on the image, and by iteratively applying filters and downsampling operations, each downsampling factor is 2, the calculation results at different scales are weighted and combined, as shown in Equation (1):

[0098]

[0099] Among them, MS_SSIM(x,y) is the multi-scale structural similarity index; l M (x,y),c j (x,y) and s j (x,y) represents brightness, contrast and structural similarity respectively; α M , β j and γ j is the weight used to adjust the relative importance of different components; j is the scale index in the MS_SSIM(x,y) calculation, which is an integer from 1 to M; M is the final scale in the MS_SSIM(x,y) calculation. On this basis, based on the multi-scale SSIM reprojection loss (i.e., the first loss function), as shown in formula (2):

[0100] L ms-reproj =α·(1-MS-SSIM(I target ,I estimate ))+β·|I target -I estimate | (2)

[0101] Among them, α is the first weight parameter in the multi-scale SSIM reprojection loss function, and its value is 0.9 after parameter adjustment; β is the second weight parameter in the multi-scale SSIM reprojection loss function, and its value is 0.1 after parameter adjustment; I target I is the original single front and back frame image of the endoscope; estimate is the synthesized adjacent frame image; 1-MS-SSIM(Itarget ,I estimate ) item prompts the model to improve the similarity of images in structure and content, |I target -I estimate The | term further constrains the difference in pixel values, so that the depth map generated by the model can more accurately reflect the true depth structure of the scene, reduce the depth estimation error, and improve the accuracy and robustness of depth estimation.

[0102] (2) Improved network structure of the DA-ICGA model (image-level contrastive learning module and dual-branch geometric perception module)

[0103] (1) Image-level contrastive learning module

[0104] In endoscope monocular depth estimation, overexposed areas often cause depth estimation artifacts. The main reason is that these highlight areas lack effective supervision signals, resulting in the failure of local photometric consistency constraints. Existing studies have attempted to alleviate this problem through an adaptive exposure compensation mechanism, usually by dynamically adjusting the image brightness to enhance the robustness of the model in highlight areas. To this end, this embodiment introduces an image-level contrast learning module based on a Gaussian mask, which selects a Gaussian distribution mask according to the characteristics of the exposed area of ​​the endoscopic image. By applying a local occlusion transformation to the image, the model focuses more on global contextual features when learning depth information, rather than simply relying on local photometric consistency. This module can effectively alleviate the artifact problem in overexposed areas and improve the accuracy and robustness of depth estimation.

[0105] Image-level comparison samples are obtained by comparing the regular samples I i (i=1,...,N) is generated by performing a transformation operation. Specifically, the occlusion transformation is performed by applying a binary mask of a Gaussian distribution to the source image (binary masking is performed on both the front and back frame images), thereby generating image-level contrast samples. Although these samples are similar to the original regular samples in overall structure, they differ in details due to the introduction of occlusion. For a source image I of size H×W×C target In this embodiment, a binary pixel mask M of size H×W is constructed based on Gaussian distribution. o, each element in the mask is generated according to the probability value of the Gaussian distribution. For example, the center of the image is used as the mean (in practice, the high-brightness area of ​​the image is used as the benchmark, and a random pixel that is relatively off-center can be selected), and the appropriate standard deviation is adjusted according to the characteristics of the scene (for example, a value of 1) to determine the probability of each pixel being blocked in the mask. During the model training process, conventional samples provide complete endoscopic scene information, while image-level comparison samples construct masks and process the source images by setting the mean and standard deviation of the Gaussian distribution, thereby simulating the failure of local photometric consistency in different regions. Inputting samples processed with a Gaussian distribution mask into the depth estimation network can prompt the model to learn how to use more contextual information for depth estimation, rather than relying solely on the photometric consistency of local areas, thereby enhancing the performance of the model in overexposed areas. For a pixel (i, j) in the image, its value M in the mask o(i,j) The calculation is shown in formula (3):

[0106]

[0107] Where τ is a threshold used to convert continuous Gaussian distribution values ​​into binary masks, M o It is represented as an occlusion mask; (i, j) is a pixel in the source image; (x0, y0) is the mean coordinate of the Gaussian distribution, usually the center of the image; σ is the standard deviation of the Gaussian distribution.

[0108] This embodiment uses M o Perform Gaussian distribution mask processing on the image to be processed, such as Figure 3 As shown, Figure 3 (a) in the figure is a regular sample. Figure 3 (b) is the image-level contrast sample generated after Gaussian distribution masking. This embodiment performs simple processing on all source images in this manner to generate image-level contrast samples. The generated image-level contrast samples are then input into the depth estimation network (the depth estimation network in this embodiment is the DAM-LoRA module) to obtain a second depth map.

[0109] The second loss function

[0110] In this embodiment, the front and back frame images are processed by Gaussian distribution mask and then passed through the depth estimation network to obtain a second depth map. The second depth map is then combined with the four first depth maps output by the DAM-LoRA module to construct a second loss function.

[0111] In the monocular image depth estimation task, the confidence mask can help the model identify those pixel depth values ​​that are more trustworthy. During the depth estimation process, the model generates a probability volume P v, which measures the probability of each pixel under different depth hypotheses. The probability map P is obtained by summing the probabilities of the four closest depth hypotheses in the depth estimation. m , set a threshold γ and compare the probability map with the threshold. m When >γ, the pixels at the corresponding positions are in the confidence mask M c A value of 1 in the confidence mask indicates that the model has high confidence in the depth estimate for that pixel; conversely, a value of 1 indicates low confidence. During model training, especially in contrastive learning, confidence masks can help the model focus more on reliable areas. When calculating the loss function and performing subsequent processing, only pixels with a value of 1 in the confidence mask are considered. This reduces the impact of inaccurate depth estimates on the overall result and improves depth estimation accuracy.

[0112] To address the photometric consistency failure problem in overexposed areas and to coordinate the consistency between the DA-ICGA infrastructure and the image-level contrast learning module, this embodiment introduces a second loss function, namely the image-level contrast consistency loss L ICC (Image-level Contrastive Consistency Loss), as shown in formula (4):

[0113]

[0114] Among them, D R and D IC They represent the four first depth maps output by the DAM-LoRA module and the second depth map output by the image-level contrast learning module; M c The confidence mask generated by the DAM-LoRA module in depth estimation is used to filter out reliable pixel areas.

[0115] The role of image-level contrast consistency loss is to encourage the model to pay more attention to contextual information, thereby achieving more complete depth estimation in overexposed areas. By minimizing the image-level contrast consistency loss, the model can learn to utilize more contextual information instead of relying solely on the photometric consistency of small areas, thereby enhancing the model's robustness and generalization ability in complex and overexposed areas.

[0116] (2) Dual-branch geometric perception module

[0117] In endoscopic images, complex tissue structures and non-uniform illumination make monocular depth estimation challenging, and the lack of geometric information often leads to unstable depth estimation of the model in high-contrast areas. The existing DARES model lacks effective use of geometric information, especially in overexposed areas, where it is difficult for the model to capture geometric structure clues and easily produces artifacts. To this end, this embodiment designs a dual-branch geometric perception module to enhance the model's perception of scene geometric information. The dual-branch geometric perception module includes two branches, one branch is used to extract texture features of the image, and the other branch extracts geometric prior information of the image and fuses them. Through the synergistic effect of these two branches, the model can comprehensively utilize texture and geometric information in the depth estimation process, thereby improving the depth estimation accuracy of overexposed areas. In addition, the dual-branch geometric perception module can optimize the fusion of global and local features, enabling the model to better understand the tissue structure, reduce the texture loss problem caused by overexposure, and thereby improve the robustness of depth estimation in high-contrast areas. It should be noted that both the previous and next frame images are input into the dual-branch geometric perception module. The input is the current frame image (adjacent frame image) and the depth image corresponding to the current frame image (adjacent frame image). The depth image is a depth map (Coarse Depth) with the same size as the previous frame image (adjacent frame image) selected from the four first depth maps output by the DAM-LoRA module. The detailed architecture of the dual-branch geometric perception module is as follows: Figure 4 shown.

[0118] Figure 4 The first row of branches is used to extract the geometric prior of the image, and the second row of branches is used to extract the texture features of the image through the FPN network and fuse them with the geometric prior.

[0119] Specifically, the method for extracting geometric priors and texture features of the front and back frame images by the dual-branch geometric perception module and generating a third depth map having the geometric priors and texture features includes:

[0120] The dual-branch geometric perception module includes a first branch and a second branch;

[0121] The first branch is used to extract the geometric prior of the image, and the second branch is used to extract the texture features of the image through the FPN network;

[0122] The first branch includes a first convolutional layer (5x5 convolution, and the convolution is followed by a BN layer and a ReLU layer), two first ResBlock layers, two second ResBlock layers, two third ResBlock layers, a first De-ResBlock layer, a second De-ResBlock layer, a third De-ResBlock layer and a second convolutional layer (3x3 convolution, and the convolution is followed by a BN layer and a ReLU layer), and the texture features of the image output by the FPN network are fused (element-wise addition) with the output of the first De-ResBlock layer and the output of the second second ResBlock layer and then input into the second De-ResBlock layer; the texture features of the image are fused (element-wise addition) with the output of the second De-ResBlock layer and the output of the second first ResBlock layer and then input into the third De-ResBlock layer; the texture features of the image are fused (element-wise addition) with the output of the third De-ResBlock layer and the output of the first convolutional layer and then input into the second convolutional layer.

[0123] Preferably, the output of the second ResBlock layer is also passed through the SEnet module (Squeeze-and-Excitation module, which is an attention mechanism module for convolutional neural networks, mainly used to model the interdependence between feature channels) and then performs an inner product operation with the De-ResBlock layer. The main function of introducing the SEnet attention mechanism in this embodiment is to enhance the network's sensitivity and representation ability to geometric priors by dynamically adjusting feature weights.

[0124] The first, second, and third ResBlock layers support geometric feature fusion and downsampling. Each layer includes two 3x3 convolutions (the 3x3 convolution parameters of the three ResBlock layers are different), and the output of the first 3x3 convolution is added to the output of the second 3x3 convolution, and then connected to a ReLU layer. In this embodiment, the convolution parameters (such as step size, padding parameter, and activation function after convolution) in the first, second, and third ResBlock layers can be adjusted.

[0125] The first, second, and third De-ResBlock layers each include three sequentially connected 5*5 convolutions (the 5*5 convolution parameters of the three De-ResBlock layers are different). In this embodiment, the convolution parameters (such as step size, padding parameter, and activation function after convolution) in the first, second, and third De-ResBlock layers can be adjusted.

[0126] The third loss function

[0127] This embodiment constructs a third loss function based on the third depth map and four first depth maps of different resolutions generated by the DAM-LoRA module. Specifically, monocular depth estimation requires the model to have a good perception of the overall depth structure of the scene. The depth distribution similarity loss can guide the model to pay attention to the depth distribution of the entire scene by calculating the sample distribution similarity between the filtered depth estimate and the true depth, which helps to enhance the model's understanding and estimation of depth in a monocular situation. Therefore, this embodiment constructs a third loss function, namely the depth distribution similarity loss Loss dds (DepthDistribution Similarity Loss), calculated as shown in formula (7):

[0128]

[0129] Where γ represents the intersection of the valid pixel set and the depth interval determined by the image-level contrast learning module (i.e., the intersection of the depth intervals determined by the first depth map and the third depth map). These depth intervals are used to screen out pixels that are meaningful for calculating the similarity of depth distribution. is the depth distribution after filtering, which represents the third depth map generated by the dual-branch geometric perception module; N GT (z) is the distribution of the true depth, representing the four first depth maps output by the DAM-LoRA module; m is the discrete depth interval index after dividing the scene depth space; z is the pixel point in the image; M′ represents the depth space of each scene divided into 48 discrete depth intervals. It should be noted that the depth interval is defined as: the distribution range of the depth value in space. In short, the depth interval is to group the distance range of objects in the image to help the computer more accurately judge "where is near and where is far", while eliminating interference to make depth estimation more reliable. The third loss function calculates the sample distribution similarity between the filtered depth estimate and the true depth using the Kullback-Leibler Divergence. and N GT The difference between (z) reflects the similarity between the depth distribution estimated by the model and the true depth distribution.

[0130] Experimental analysis

[0131] This example mainly demonstrates dataset selection, evaluation criteria selection, ablation experiments, and comparative experimental analysis. The experimental training for this example was performed on a single 4090 server with an AMD EPYC 9754 CPU (128 cores, 256 threads, 2.25-3.1 GHz), 32 GB of RAM, and Ubuntu 18.04, using PyTorch 1.10. The inference process was performed on a single 4090 host with an Intel i9-13900K CPU (24 cores, 32 threads, 3 GHz), 64 GB of RAM, and 24 cores, 32 threads, 3 GHz. The system was Ubuntu 20.04, using PyTorch 1.10.

[0132] Same as the DARES model. The depth estimation part of this embodiment uses DAM V2 as the base model, and the encoder part is a ViT encoder based on DINOv2, which contains 12 multi-head self-attention blocks (i.e., LoRA Transformer modules). Vector-LoRA is introduced in each attention block of the ViT encoder, and the rank vector is r = [14,14,12,12,10,10,8,8,8,8,8,8]. The PoseNet module is used for pose estimation in self-supervised training, and inputs adjacent frames to calculate self-motion. The image-level contrast learning module uses the same network as the conventional branch for feature extraction operations, extracts features of different scales from the input image, and provides rich feature information for subsequent cost volume construction and depth estimation. The dual-branch geometric perception module fuses the low-level features extracted by the first branch with the high-level features extracted by the second branch using the FPN network to further enhance the feature expression ability. The entire process was completed within 12 hours. For all sub-networks, the input resolution was 320×256 pixel images, and the entire training plan was carried out in an end-to-end manner.

[0133] This example uses the Adam optimizer. Adam is an optimization algorithm widely used in deep learning with fast convergence characteristics. The optimizer sets the parameter β1 to smooth the gradient of each iteration, making the gradient change more stable. The optimizer adaptively adjusts the learning rate of each parameter by setting the parameter β2, helping the model to more effectively adjust the step size in different gradient change areas, thereby improving training speed and convergence. At the same time, this example retains the initial learning rate of DARES, set to 0.0001, and decays every 10 steps, for a total of 50 epochs.

[0134] Dataset selection

[0135] This experiment mainly uses an open source endoscopy dataset for model training. The dataset contains real images of different parts of the digestive tract and corresponding high-precision depth data. To ensure the reliability of the experiment, this embodiment divides the SCARED dataset into 15351 frames for training, 1705 frames for verification, and 551 frames for testing. Comparative tests with other monocular depth estimation models were performed on the SERV-CT dataset, and in order to further verify the generalization performance of the model in cross-domain scenarios, this embodiment introduces the natural scene depth estimation field datasets Make3D and DIODE. These two datasets cover complex outdoor terrain and mixed indoor and outdoor environments, respectively, and can effectively test the performance of the model in complex situations such as strong light reflection and overexposure. An introduction to each dataset is as follows:

[0136] The SCARED dataset, collected from abdominal dissections of fresh pig carcasses, contains 35 endoscopic videos, providing ground truth data for depth maps, point clouds, and egomotion. In this dataset, the maximum range of the depth maps is scaled to 150 mm, covering the depth range of most regions of the digestive tract.

[0137] The SERV-CT dataset, extracted from pig torsos, contains 16 pairs of stereo images with ground-truth depth and disparity data. In this dataset, the depth map is capped at 180 mm to ensure coverage of the entire valid depth range.

[0138] The Make3D dataset consists of RGB-D images of real outdoor scenes, covering a variety of complex environments, including urban and natural landscapes. Its ground truth depth is collected using high-precision LiDAR and combined with multi-view geometry optimization to generate a dense depth map with high spatial consistency and detail preservation. The dataset contains hundreds of image-depth pairs, covering scenes with varying lighting conditions, scale variations, and complex geometric structures. It is suitable for verifying the model's generalization performance in real open environments for texture-missing areas and dynamically exposed areas.

[0139] DIODE-Dataset. DIODE-Dataset is the first monocular depth estimation benchmark dataset covering diverse indoor and outdoor scenes. Its ground truth depth is acquired using a high-resolution 3D laser scanner, covering complex building structures, dynamic lighting, and multi-scale object layouts. The dataset provides strictly aligned RGB images and precise depth information, encompassing extreme scene variations from narrow indoor corridors to open outdoor plazas, and covering a variety of weather conditions, including sunny and rainy days. This dataset effectively tests the adaptability of models in cross-domain environments.

[0140] Selection of evaluation criteria

[0141] Similar to the work of DAERS, this example uses standard evaluation metrics: Absolute Relative Error (Abs-Rel), Square Relative Error (Sq-Rel), Root Mean Squared Error (RMSE), Root Mean Squared Error log (RMSE-log), and accuracy (a1, a2, a3). In Table 1, d and d * Denote the predicted depth value and the corresponding true value respectively, and D denotes a set of predicted depth values.

[0142] Table 1 Evaluation indicators of depth estimation

[0143]

[0144] The absolute relative error (Abs-Rel) evaluates the depth consistency of overexposed areas of the digestive tract by calculating the ratio of the absolute difference between the predicted value and the true value. Its ability to eliminate scale differences enables it to objectively reflect the model's ability to evenly suppress artifacts in the near-field mucosal surface and far-field cavity folds of the digestive tract. When the Abs-Rel value decreases in the overexposed area, it indicates that the model has successfully suppressed specular reflection interference and improved prediction accuracy. In addition, Abs-Rel provides a unified standard to ensure that the model maintains stable accuracy for objects at different distances. The squared relative error (Sq-Rel) specifically captures depth mutation anomalies at the boundaries of overexposed areas by amplifying the square of large errors. The reduction in the Sq-Rel value in the highlight area directly reflects the reduction in errors caused by artifacts, and is particularly suitable for evaluating depth prediction failures caused by brightness fluctuations in endoscopes. By optimizing Sq-Rel, this embodiment can more effectively capture and adjust these large errors, ensure the accuracy of the model in key areas, and thereby optimize the model's performance and improve the accuracy of predictions for larger depth deviations.

[0145] The root mean square error (RMSE) measures the overall error of the model by calculating the square root of the sum of the squared errors between the predicted depth and the true depth. It is particularly sensitive to large errors, comprehensively reflecting the model's accuracy and revealing large deviations. RMSE is suitable for evaluating the overall performance of monocular image depth estimation models, especially in complex scenes. By comprehensively considering the squared effect of the errors, it can accurately assess the model's performance across different depth ranges. Optimizing RMSE helps improve prediction accuracy, ensure consistent model performance across depth ranges, and reduce the negative impact of large errors on the overall results. The logarithmic root mean square error (RMSE-log) is similar to RMSE, but takes the logarithm of the predicted and true values ​​before calculating the error. This approach focuses more on proportional errors and is suitable for evaluating the relative errors of distant objects in depth estimation. A smaller value indicates a more accurate predicted depth proportional relationship.

[0146] The accuracy index indicates the ratio of the predicted depth value to the true value within a certain threshold (such as 1.25, 1.25 2 , 1.25 3 ), by calculating the accuracy under different thresholds, the accuracy of the predicted depth value can be reflected. The higher the accuracy, the closer the predicted depth value is to the true value, and the better the performance of the model.

[0147] Ablation experiments

[0148] In order to verify the improvement effect of the image-level contrast learning module (IC) and the dual-branch geometric perception module (GA) on the artifact phenomenon in endoscopic depth estimation, this embodiment verifies the effectiveness of the proposed method through four sets of progressive ablation experiments ID1-ID4. ID1 is the baseline model DARES. The addition of the image-level contrast branch in ID2 is used to judge whether the image-level contrast can enhance the context perception ability. The geometric perception module in ID3 is used to prove whether the geometric constraint can effectively suppress the distortion of the exposure area. ID4 represents the simultaneous activation of the two modules. On the basis of the baseline model DARES, the image-level contrast module and the geometric perception module are gradually introduced. The experimental results are shown in Table 2. The table shows the evaluation indicators under different module settings, which provides quantitative support for experimental analysis. In Table 2, GA represents the dual-branch geometric perception module and IC represents the image-level contrast learning module.

[0149] Table 2 Ablation experiment

[0150]

[0151] Experimental results show that compared to the original DARES model, the image-level contrastive learning module alone improves the RMSE index through cross-sample contrastive learning, especially in the depth continuity of the edges of the digestive tract structure. By maximizing the contrast loss, the similarity of the depth distribution between normal samples and perturbed samples can be increased, thereby forcing the network to focus on the overall regularity of the digestive tract structure rather than the changes in local pixel brightness, thereby effectively improving the accuracy of depth estimation. On the other hand, the introduction of the dual-branch geometric perception module alone significantly reduces the presence of artifacts. This is due to the geometric consistency constraint that explicitly aligns the projection errors of the previous and next frames, forcing the network to learn a depth gradient distribution that conforms to the physical imaging laws. The dual-branch geometric perception module effectively suppresses the depth mutation problem caused by local texture loss by introducing geometric cues, further improving the stability of the model. When the two modules are enabled at the same time, the experimental results show that most of the evaluation indicators are further improved, indicating that there is a synergistic enhancement effect between the dual-branch geometric perception branch and the image-level contrast learning module. However, this embodiment also notes that the RMSE indicator in ID4 is lower than that of ID3 using only the GA module. This is because the Gaussian mask simulates the exposure area, and its shape and size may not completely match the actual scene. The additional contrast learning brings redundant information or additional uncertainty, causing the model to incorrectly learn part of the depth information, thereby slightly increasing the RMSE. In addition, adjusting the weights α and β of the multi-scale structural similarity reprojection loss can further optimize the performance of the model. When α = 0.9 and β = 0.1, the model reaches the optimal balance. At this time, Abs-Rel and both reach the lowest value, and the training convergence speed is accelerated. From the visualization results, it can be seen that if β is too large, the details will be too smooth, while if it is too small, local artifacts cannot be effectively suppressed. Therefore, in the weight adjustment process, choosing appropriate α and β values ​​is crucial to the performance of the model.

[0152] Comparative experiment

[0153] Representative models in the field of monocular image depth estimation were selected for comparison and evaluation of the model in this embodiment. These included several classic monocular image depth estimation models for outdoor scenes: DeFeat-Net, Fine-tuned DAM V2, and Monodepth2, as well as models AF-SfMLearner, M-DASS, and DARES, which have performed well in endoscope monocular depth estimation. DeFeat-Net proposed an innovative self-supervised representation learning model for the monocular image depth estimation task. It leveraged multi-task learning to improve depth estimation accuracy and combined unsupervised learning to enhance the model's performance in common scenarios, with a particularly significant advantage in cross-dataset generalization. Fine-tuned DAM V2, an optimized version of the original DAM model, represents the latest advancements in the field of monocular image depth estimation. It also places particular emphasis on cross-dataset generalization, which is crucial for monocular depth estimation tasks. These improvements not only improve the model's performance but also demonstrate greater adaptability and stability when processing data from diverse sources. Monodepth2 has proposed many influential improvements in the field of self-supervised monocular image depth estimation, especially effectively performing depth and camera self-motion estimation through photometric consistency loss and smoothness loss. It has become an important benchmark in the field of self-supervised learning, especially when evaluating the optimization of other self-supervised learning methods in depth estimation performance.

[0154] Unlike outdoor scenes, depth estimation of monocular images in endoscopes usually faces more challenging environments, such as uneven lighting, motion blur, and limited viewing angles. AF-SfMLearner proposed the use of appearance flow to assist depth estimation and self-motion estimation. Appearance flow can help alleviate severe brightness fluctuations between frames and extract more accurate depth. This innovation enables AF-SfMLearner to achieve the best results in the field of endoscope monocular image depth estimation. M-DASS proposes a pseudo-label-based data enhancement generation method and a network feature clustering-based method to address the problem of gastrointestinal surface occlusion and the difficulty in semantic understanding of gastrointestinal weak texture images under illumination changes. This improves the ability of the gastrointestinal surface to better achieve depth estimation under occlusion and reduces the problem of weak texture information loss caused by illumination changes. DARES, as one of the first models to introduce the DAM basic model into endoscope scenes, has achieved the best results in current endoscope monocular depth estimation.

[0155] Table 3 Evaluation index values ​​of model experiments on the SCARED dataset

[0156]

[0157] Table 3 shows the comparative results of various models tested using the SCARED dataset on various evaluation indicators. The results show that the model of this embodiment significantly surpasses other models in all evaluation indicators. Abs-Rel can measure the error of the model in different depth areas, especially in environments with large brightness fluctuations caused by illumination changes and reflections. Previous models often have unstable depth estimation due to artifacts. The model of this embodiment shows higher stability when dealing with overexposed areas, effectively suppresses artifacts caused by strong reflections and illumination changes, and improves the accuracy of depth prediction. Sq-Rel is highly sensitive to depth errors, especially under the influence of illumination changes and non-Lambertian reflections, and is easily interfered by artifacts, resulting in depth prediction distortion. Artifacts not only affect the prediction accuracy, but may also interfere with the training process. The model of this embodiment performs well in removing artifacts and handling high-noise environments, and can perform depth estimation more accurately, ensuring the stability and reliability of the prediction results.

[0158] Table 4 shows the experimental results of evaluating various models using the SERV-CT dataset. From the perspective of various evaluation indicators, the method of this embodiment is superior to the DARES model in many aspects. The Abs-Rel of the model of this embodiment is 0.081, which is significantly lower than DARES's 0.090, indicating that the model of this embodiment has improved the accuracy of depth estimation. The RMSE value also decreased from 10.836 of DARES to 9.949, indicating that the model of this embodiment is better than DARES in global depth prediction accuracy and performs better in maintaining the integrity of image structure. Overall, the method of this embodiment surpasses the current most advanced endoscope monocular depth estimation model DARES in all performance indicators, demonstrating its effectiveness and robustness in depth estimation tasks.

[0159] Table 4 Comparison of model evaluation indicators on the SERV-CT dataset

[0160]

[0161] This experiment aims to explore and compare the self-supervised models for depth estimation from monocular images commonly used in the field of endoscopy. Therefore, the ground truth of the depth map is not listed. The comparison experiment focuses on the cutting-edge technology DARES model. Figure 5 As shown in the figure, the qualitative comparison results clearly show the performance differences of each model in dealing with challenges such as overexposed areas. Compared with models such as AF-SfMLearner, M-DASS and DARES, the DA-ICGA model proposed in this example ( Figure 5Ours) has significant advantages in reducing artifacts and ensuring more accurate depth estimation results, which is crucial for improving spatial perception and safety during surgery. In addition, the superior performance of the proposed model was verified through analysis of quantitative indicators such as mean square error and absolute relative error (Abs Rel). Experimental results further show that the model of this embodiment demonstrates excellent accuracy and robustness in depth estimation tasks, especially in complex endoscopic imaging environments, and can still stably output high-quality depth results. Through detailed experimental design and analysis, these results not only demonstrate the effectiveness of the new model, but also provide new perspectives for further development in this field.

[0162] In order to further verify the generalization ability of the model in cross-domain scenarios, this embodiment directly applies the model trained based on the SCARED dataset to the Make3D dataset and DIODE-Dataset dataset in natural scenes for verification without any fine-tuning. At the same time, the models AF-SfMLearner, M-DASS, which have good effects in the field of endoscope monocular depth estimation, and the most advanced model DARES are selected for visual comparison. The Make3D dataset covers complex outdoor terrain, while the DIODE-Dataset includes mixed indoor and outdoor environments. The experiment specifically selected images with complex conditions such as strong light reflection and overexposure to test the performance of the model in real environments. The experimental results are as follows Figure 5 As shown, although texture missing areas and dynamic exposure areas often seriously interfere with depth estimation in these challenging scenes, the model of this embodiment still demonstrates excellent generalization ability and can maintain high spatial consistency and detail retention capabilities.

[0163] Specifically, in Figure 6 In the corridor scene of Make3D in row 1, models such as AF-SfMLearner, M-DASS, and DARES have depth jumps or artifacts on the shadow-covered ground and blurred walls due to the lack of geometric constraints. Figure 6 Ours) optimizes the global structural fusion of the ground and the wall by imposing geometric consistency constraints, while suppressing artifacts in the exposed area and significantly improving the ability to retain details. The model in this embodiment shows strong environmental adaptability under complex lighting conditions. Figure 6 Taking the vegetation area of ​​Make3D in the second row as an example, the other models lead to depth flattening due to overfitting of local features. However, the model of this embodiment maintains spatial consistency in fine-grained areas such as leaf gaps through dual module collaboration, and reduces the estimation deviation caused by local information loss in the overexposed area of ​​the manhole cover. Figure 6In the indoor study scene of the DIODE-Dataset in row 3, the other models have disordered depth estimation of the bookshelf area due to the lack of geometric constraints. The DA-ICGA model can effectively restore the geometric coherence of the edge structure and retain the depth details of the ornaments in the overexposed bookshelf area. Figure 6 In the brightly lit area of ​​the building in row 4, the DARES model fails to capture global contextual information, resulting in depth estimation artifacts. However, the model of this embodiment maximizes the depth similarity between regular samples and comparison samples, forcing the network to utilize a wider range of contextual information, thereby achieving continuous and complete depth reconstruction at the junction of the sky and the building. In summary, the model of this embodiment can effectively reduce the impact of artifacts under complex lighting conditions and provide stable depth estimation results. This further demonstrates the robustness of the model in cross-domain applications, its ability to adapt to various environmental changes, and its strong environmental adaptability and practical application potential.

[0164] In summary, this embodiment can solve the common artifact problem in overexposed areas, thereby improving the accuracy and robustness of depth estimation. By introducing a geometric perception module and an image-level contrast learning module, the network can effectively correct texture loss and blurring caused by overexposure, enhancing the accuracy of depth estimation in high-contrast areas. Experimental results verify that the method proposed in this embodiment can significantly improve the accuracy of depth estimation, especially in overexposed areas, where the depth estimation model exhibits good robustness and high reconstruction quality.

[0165] Example 2

[0166] This embodiment provides an endoscope monocular image depth estimation system, comprising:

[0167] Acquisition module: used to obtain single front and back frame images of the endoscope;

[0168] Training module: used to train the DA-ICGA model for the endoscope monocular using the front and back frame images of the endoscope monocular. The DA-ICGA model is based on DARES and includes the PoseNet module, the DAM-LoRA module, the dual-branch geometric perception module and the image-level contrast learning module.

[0169] The DAM-LoRA module is used to process the front and back frame images to generate four depth maps with different resolutions;

[0170] The PoseNet module is used to process the front and back frame images and the four first depth maps of different resolutions generated by the DAM-LoRA module to obtain synthesized adjacent frame images;

[0171] Constructing a first loss function based on the endoscope single front and back frame images and the synthesized adjacent frame images;

[0172] The image-level contrast learning module is used to generate image-level contrast samples based on the previous and next frame images, and use the image-level contrast samples as the second depth map;

[0173] Constructing a second loss function based on the second depth map and four first depth maps of different resolutions;

[0174] The dual-branch geometric perception module is used to extract geometric prior and texture features of the previous and next frame images, and generate a third depth map having the geometric prior and texture features;

[0175] Constructing a third loss function based on the third depth map and four first depth maps of different resolutions;

[0176] If the first, second, and third loss functions all reach their respective preset values, the model converges and training ends; otherwise, the model continues to be trained until the model converges;

[0177] Depth estimation module: used to perform depth estimation on the monocular image of the endoscope to be inspected using the trained DA-ICGA model.

[0178] Example 3

[0179] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the endoscope monocular image depth estimation method described in the first embodiment are implemented.

[0180] Example 4

[0181] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the endoscope monocular image depth estimation method described in the first embodiment are implemented.

[0182] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.

[0183] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0184] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0186] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0187] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for estimating depth of an endoscope monocular image, characterized by: include: Acquire single front and back frame images of the endoscope; The DA-ICGA model is trained on the endoscope monocular front and back frame images. The DA-ICGA model is based on DARES and includes a PoseNet module, a DAM-LoRA module, a dual-branch geometric perception module, and an image-level contrast learning module. The DAM-LoRA module is used to process the front and back frame images to generate four first depth maps with different resolutions; The PoseNet module is used to process the front and back frame images and the four first depth maps of different resolutions generated by the DAM-LoRA module to obtain synthesized adjacent frame images; Constructing a first loss function based on the endoscope single front and back frame images and the synthesized adjacent frame images; The image-level contrast learning module is used to generate a second depth map based on the previous and next frame images; Constructing a second loss function based on the second depth map and four first depth maps of different resolutions; The dual-branch geometric perception module is used to extract geometric prior and texture features of the previous and next frame images, and generate a third depth map having the geometric prior and texture features; Constructing a third loss function based on the third depth map and four first depth maps of different resolutions; If the first, second, and third loss functions all reach their respective preset values, the model converges and training ends; otherwise, the model continues to be trained until the model converges; The depth of the monocular image of the endoscope to be inspected is estimated using the trained DA-ICGA model.

2. The method for estimating depth of an endoscope monocular image according to claim 1, wherein: The method for the PoseNet module to process the preceding and following frame images and the four first depth maps of different resolutions generated by the DAM-LoRA module to obtain synthesized adjacent frame images includes: The PoseNet module is used to input the previous and next frame images into the ResNet network to obtain camera pose information, use the camera pose information as a change matrix, and perform an inverse transformation operation on the four first depth maps of different resolutions output by the DAM-LoRA module through the change matrix to generate synthetic adjacent frame images.

3. The method for estimating depth of an endoscope monocular image according to claim 1, wherein: The method for the DAM-LoRA module to process the preceding and following frame images to generate four first depth maps of different resolutions includes: The DAM-LoRA module includes a ViT encoder and a DPT decoder. The ViT encoder is used to extract visual features of the previous and next frame images, and the DPT decoder is used to perform depth regression on the visual features of the previous and next frame images extracted by the ViT encoder to generate four first depth maps with different resolutions.

4. The method for estimating depth of an endoscope monocular image according to claim 1, wherein: The method for the image-level contrast learning module to generate a second depth map according to the preceding and following frame images includes: The image-level contrast learning module performs binary Gaussian masking on each pixel in the previous and next frame images to generate image-level contrast samples, wherein the binary Gaussian mask formula is: Among them, M o(i,j) is the mask value of each pixel in the source image; τ is the preset threshold; M o is the occlusion mask; (i, j) is the pixel in the source image; (x0, y0) is the mean coordinate of the Gaussian distribution; σ is the standard deviation of the Gaussian distribution; The image-level contrast samples are input into a depth estimation network to generate a second depth map.

5. The method for estimating depth of an endoscope monocular image according to claim 1, wherein: The method of extracting geometric prior and texture features of the preceding and following frame images by the dual-branch geometric perception module and generating a third depth map having the geometric prior and texture features includes: The dual-branch geometric perception module includes a first branch and a second branch; The first branch is used to extract the geometric prior of the image, and the second branch is used to extract the texture features of the image through the FPN network; The first branch includes a first convolutional layer, two first ResBlock layers, two second ResBlock layers, two third ResBlock layers, a first De-ResBlock layer, a second De-ResBlock layer, a third De-ResBlock layer and a second convolutional layer connected in sequence, and the texture features of the image output by the FPN network are fused with the output of the first De-ResBlock layer and the output of the second second ResBlock layer and then input into the second De-ResBlock layer; the texture features of the image are fused with the output of the second De-ResBlock layer and the output of the second first ResBlock layer and then input into the third De-ResBlock layer; the texture features of the image are fused with the output of the third De-ResBlock layer and the output of the first convolutional layer and then input into the second convolutional layer.

6. The method for estimating depth of an endoscope monocular image according to claim 5, wherein: The first ResBlock layer, the second ResBlock layer, and the third ResBlock layer each include two 3x3 convolutions, and the output of the first 3x3 convolution and the output of the second 3x3 convolution are added, and then connected to a ReLU layer after the addition; The first De-ResBlock layer, the second De-ResBlock layer, and the third De-ResBlock layer each include three 5*5 convolutions connected in sequence.

7. The method for estimating depth of an endoscope monocular image according to claim 1, wherein: The first loss function is constructed based on the single front and back frame images of the endoscope and the synthesized adjacent frame images, and the formula is: L ms-reproj =α·(1-MS-SSIM(I target ,I estimate ))+β·|I target -I estimate | Among them, MS_SSIM(x,y) is the multi-scale structural similarity index; l M (x,y),c j (x,y) and s j (x,y) represents brightness, contrast and structural similarity respectively; α M , β j and γ j is the weight used to adjust the relative importance of different components; j is the scale index in the MS_SSIM(x,y) calculation, which is an integer from 1 to M; M is the final scale in the MS_SSIM(x,y) calculation; α is the first weight parameter; β is the second weight parameter; I target I is the original single front and back frame image of the endoscope; estimate are synthesized adjacent frame images.

8. The method for estimating depth of an endoscope monocular image according to claim 1, wherein: The second loss function is constructed based on the second depth map and four first depth maps with different resolutions, and the formula is: Among them, D R and D IC They represent the four first depth maps output by the DAM-LoRA module and the second depth map output by the image-level contrast learning module; M c Confidence mask generated by the DAM-LoRA module in depth estimation.

9. The method for estimating depth of an endoscope monocular image according to claim 1, wherein: The third loss function is constructed based on the third depth map and the four first depth maps with different resolutions. The formula is: Wherein, γ represents the intersection of depth intervals determined by the first depth map and the third depth map; is the depth distribution after filtering, which represents the third depth map generated by the dual-branch geometric perception module; N GT (z) is the distribution of the true depth, representing the four first depth maps output by the DAM-LoRA module; m is the discrete depth interval index after dividing the scene depth space; z is the pixel point in the image; M′ represents the division of the depth space of each scene into several discrete depth intervals.

10. An endoscope monocular image depth estimation system, characterized by: include: Acquisition module: used to obtain single front and back frame images of the endoscope; Training module: used to train the DA-ICGA model for the endoscope monocular using the front and back frame images of the endoscope monocular. The DA-ICGA model is based on DARES and includes the PoseNet module, the DAM-LoRA module, the dual-branch geometric perception module and the image-level contrast learning module. The DAM-LoRA module is used to process the front and back frame images to generate four depth maps with different resolutions; The PoseNet module is used to process the front and back frame images and the four first depth maps of different resolutions generated by the DAM-LoRA module to obtain synthesized adjacent frame images; Constructing a first loss function based on the endoscope single front and back frame images and the synthesized adjacent frame images; The image-level contrast learning module is used to generate image-level contrast samples based on the previous and next frame images, and use the image-level contrast samples as the second depth map; Constructing a second loss function based on the second depth map and four first depth maps of different resolutions; The dual-branch geometric perception module is used to extract geometric prior and texture features of the previous and next frame images, and generate a third depth map having the geometric prior and texture features; Constructing a third loss function based on the third depth map and four first depth maps of different resolutions; If the first, second, and third loss functions all reach their respective preset values, the model converges and training ends; otherwise, the model continues to be trained until the model converges; Depth estimation module: used to perform depth estimation on the monocular image of the endoscope to be inspected using the trained DA-ICGA model.

Citation Information

Patent Citations

  • Monocular endoscope video depth and pose estimation method

    CN116740170A

  • Monocular depth estimation network establishment method based on dynamic view selection and application thereof

    CN119313716A

  • Methods, systems, and computer readable media for colonoscopic blind spot detection

    WO2024186746A1

Cited By

  • Endoscope image perception method and system based on self-supervised learning

    CN121330277A

  • Endoscope image self-supervision depth estimation method based on physical light supplement model

    CN121353366A