A self-supervised depth estimation method for surgical scenarios

Abstract

This invention belongs to the field of computer vision and medical image processing technology, and specifically relates to a self-supervised depth estimation method for surgical scenes. It includes the following steps: S1: Constructing a dual-head self-supervised depth estimation network; S2: Generating the current frame's depth prediction result and a global deformation scale factor; S3: Constructing a pseudo-true depth based on the rigidity assumption; S4: Performing elastic correction based on the deformation scale factor; S5: Constructing a composite loss function; S6: Performing iterative training; S7: Performing inference deployment. This invention achieves explicit perception and active correction of non-rigid deformations of soft tissue, effectively solving the problem of lack of supervision caused by "rigid constraints" in traditional methods. This significantly improves the geometric accuracy and physical consistency of depth estimation in complex dynamic scenes, while also significantly improving the model's depth estimation accuracy and robustness in complex dynamic surgical environments.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and medical image processing technology, specifically a self-supervised depth estimation method for surgical scenes. Background Technology

[0002] Monocular depth estimation has made significant progress in rigid scenarios such as autonomous driving. The core objective of this task is to recover the dense 3D depth of a scene from a monocular video sequence, requiring the predicted depth map to maintain spatial continuity and geometric consistency over time. While monocular depth estimation technology has achieved significant progress in outdoor rigid scenarios, it faces entirely new technical bottlenecks when transferred to complex constrained environments such as minimally invasive surgery. In complex constrained environments like minimally invasive surgery, traditional algorithms face severe technical limitations due to tissue deformation caused by soft tissue creep and dynamic disturbances. Current closest existing technologies mostly employ training paradigms based on the assumption of rigid scenarios, such as distorting and aligning the depth maps of adjacent frames using optical flow to generate pseudo-ground values. However, this has a fundamental flaw: it assumes that 2D pixel alignment is equivalent to 3D geometric consistency, thus explicitly ignoring the depth motion of the camera along the optical axis (Z-axis), which easily produces erroneous solutions containing artifacts in non-rigid scenarios. Furthermore, existing solutions such as "automatic masking" or "uncertainty weighting" strategies are essentially "avoiding" errors rather than correcting them, and lack the ability to explicitly correct non-rigid deformations using learnable physical quantities. Summary of the Invention

[0003] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0004] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution: a self-supervised depth estimation method for surgical scenes, comprising the following steps:

[0005] S1: A pre-trained Vision Transformer is used as a shared encoder to extract high-dimensional semantic features from the input monocular endoscope image. A depth prediction branch and a deformation perception branch are then connected in parallel after the encoder.

[0006] S2: For the target frame The depth branch outputs the dense depth map predicted for the current frame. Deformation branch outputs global scale factor field The exponential activation function is used at the end of the deformation branch;

[0007] S3: Utilize adjacent source frames The camera intrinsic parameter matrix K and the rigid pose or rigid optical flow estimation results are used to project the source frame depth information onto the target frame coordinate system, resulting in a pseudo-ground value depth map that only considers rigid motion. ;

[0008] S4: Perform elastic correction based on deformation scale factor: using the global scale factor field For pseudo-true depth maps Perform pixel-by-pixel calibration to form a corrected self-supervised target. For rigid background areas, Adaptive convergence to 1; for tissue deformation regions, Deviation from 1 is used to absorb non-rigid errors;

[0009] S5: Construct a composite loss function: combine deformation-aware temporal consistency loss, identity regularization loss and smoothing constraint loss to perform end-to-end optimization of the dual-head network and maintain stable rigid geometric constraints in the background region.

[0010] S6: Based on a large number of unlabeled surgical endoscopy video sequences, forward inference, pseudo-truth value correction, loss calculation and backpropagation are repeatedly performed until the network converges or reaches the preset number of training rounds.

[0011] S7: After training, only the depth prediction branch is retained for inference, and a dense depth map is directly output for a single frame of surgical image.

[0012] As a preferred embodiment of the self-supervised depth estimation method for surgical scenes described in this invention, in step S2, the dense depth map predicted in the current frame... The resolution is the same as the input image.

[0013] As a preferred embodiment of the self-supervised depth estimation method for surgical scenes described in this invention, in step S4, the elastic correction relationship is expressed as:

[0014]

[0015] Is Time, location The local deformation scaling factor at a given location is always greater than 0. In a rigid background region, network learning enables... , making Maintaining rigid geometric constraints, the network adaptively adjusts in areas of instrument movement or tissue deformation. The value absorbs geometric deviations.

[0016] As a preferred embodiment of the self-supervised depth estimation method for surgical scenes described in this invention, in step S5, the total loss function is expressed as:

[0017]

[0018] in, For the total loss function, For deformation sensing temporal consistency loss, For identity regularization loss, To smooth the constraint loss, and These are the weighting coefficients for the corresponding loss terms.

[0019] As a preferred embodiment of the self-supervised depth estimation method for surgical scenes described in this invention, the deformation-aware temporal consistency loss is expressed as:

[0020]

[0021] in It is a structural similarity index. It is the balance coefficient.

[0022] As a preferred embodiment of the self-supervised depth estimation method for surgical scenes described in this invention, the identity regularization loss is expressed as:

[0023]

[0024] It is the total number of pixels in the image, and this loss term forces To prevent the value from approaching 1 in most pixel locations, thus preventing... Factor overfitting is only permitted in regions where geometric conflicts are significant. Deviation 1.

[0025] As a preferred embodiment of the self-supervised depth estimation method for surgical scenes described in this invention, the smoothing constraint loss is expressed as:

[0026]

[0027] , These represent the spatial gradient operators in the horizontal and vertical directions, respectively. It is output from the network. Take the natural logarithm.

[0028] Compared with existing technologies, the advantages of this invention are as follows: By performing self-supervised training on a large number of surgical endoscopic images, the resulting model internalizes a physical understanding of the dynamic characteristics of the surgical scene, enabling it to adaptively identify and distinguish between rigid backgrounds and non-rigid interference regions. During the inference phase, this invention not only reconstructs high-fidelity 3D anatomical structures but also effectively suppresses geometric artifacts caused by soft tissue peristalsis through its learned elastic correction mechanism, thereby achieving robust, pixel-level dense 3D depth estimation in complex minimally invasive surgical environments. This invention achieves explicit perception and active correction of non-rigid deformation of soft tissue, effectively solving the problem of lack of supervision caused by "rigid constraints" in traditional methods. This significantly improves the geometric accuracy and physical consistency of depth estimation in complex dynamic scenes, while also significantly enhancing the model's depth estimation accuracy and robustness in complex dynamic surgical environments. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0030] Figure 1 This is a flowchart of a self-supervised depth estimation method for surgical scenarios according to the present invention. Detailed Implementation

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0034] This invention provides a method for utilizing deformation scale factors A self-supervised depth estimation method with a correction mechanism is proposed. By constructing a dual-head neural network architecture including a deformation-aware branch, a pixel-by-pixel local deformation scale factor is explicitly learned, thereby establishing an elastic correction model to perform pixel-by-pixel weighted calibration on pseudo-true values ​​generated based on rigidity assumptions. This mechanism can adaptively absorb geometric errors caused by object motion, preserving effective supervision signals in dynamic regions while blocking the propagation of erroneous gradients to the depth branch, effectively solving the problems of lack of supervision and geometric artifacts caused by "rigid constraints" in traditional methods. Ultimately, it achieves highly robust, physically consistent, and pixel-level dense 3D depth estimation results in complex minimally invasive surgical environments, ensuring no additional computational overhead during the inference phase.

[0035] Specifically, a self-supervised depth estimation method for surgical scenes based on local deformation scale factor correction is proposed. Its key technologies include network construction, rigid pseudo-truth value generation, deformation scale correction, composite loss optimization, and inference deployment. Specifically, it includes:

[0036] S1. Construct a dual-head self-supervised depth estimation network. A pre-trained Vision Transformer is used as a shared encoder to extract high-dimensional semantic features from the input monocular endoscope image. A depth prediction branch and a deformation perception branch are connected in parallel after the encoder, with the depth branch outputting the predicted dense depth map for the current frame. The deformation-sensing branch is used to output the global scale factor field. .

[0037] S2 generates the current frame depth prediction result and local deformation scale factor. For the target frame... The depth branch outputs the dense depth map predicted for the current frame. Its resolution is consistent with the input image, and the deformation branch outputs a global scale factor field. ,in Indicates in Time, location The local deformation scale factor is always greater than 0. To ensure that the scale factor is always positive, an exponential activation function is used at the end of the deformation branch.

[0038] S3, constructing a pseudo-truth depth based on the rigidity assumption. Utilizing adjacent source frames. The camera intrinsic parameter matrix K and the rigid pose or rigid optical flow estimation results are used to project the source frame depth information onto the target frame coordinate system, resulting in a pseudo-ground value depth map that only considers rigid motion. .

[0039] S4 performs elastic correction based on the deformation scale factor. This utilizes a global scale factor field. For pseudo-true depth maps Perform pixel-by-pixel calibration to form a corrected self-supervised target. For rigid background areas, Adaptive convergence to 1; for tissue deformation regions, The deviation from 1 is used to absorb non-rigid errors; the elastic correction relationship can be expressed as:

[0040]

[0041] In rigid background regions, network learning enables... , making Maintaining rigid geometric constraints, the network adaptively adjusts within the deformation region. The value absorbs geometric deviations.

[0042] S5. Construct a composite loss function. Combine deformation-aware temporal consistency loss, identity regularization loss, and smoothing constraint loss to perform end-to-end optimization on the dual-head network, ensuring that the network initiates corrections only in regions with significant geometric conflicts, while maintaining stable rigid geometric constraints in background regions.

[0043] The total loss function can be expressed as:

[0044]

[0045] in, For the total loss function, For deformation sensing temporal consistency loss, For identity regularization loss, To smooth the constraint loss, and These are the weighting coefficients for the corresponding loss terms.

[0046] The deformation sensing temporal consistency loss can be expressed as:

[0047]

[0048] in It is the Structural Similarity Index Measure. It is a balance coefficient, unlike the direct calculation method used in traditional methods. and The difference in depth is used to calculate the predicted depth in this invention. Compared with the corrected pseudo-true value The loss function, which considers structural similarity and photometric differences between the components, allows the network to adjust its loss function in dynamic regions. This minimizes reprojection errors, thereby protecting the integrity of the depth map structure.

[0049] The identity regularization loss can be expressed as:

[0050]

[0051] It is the total number of pixels in the image, and this loss term forces To prevent the value from approaching 1 in most pixel locations, thus preventing... Factor overfitting is only permitted in regions where geometric conflict is significant (i.e., the rigidity error is much greater than the regularization penalty). Deviation 1. This enables automatic discovery and sparsity correction of dynamic regions.

[0052] The smoothing constraint loss can be expressed as:

[0053]

[0054] , These represent the spatial gradient operators in the horizontal and vertical directions, respectively. It is output from the network. Take the natural logarithm. This loss function does not introduce edge weights from the image gradient and is designed to... The field is subjected to a strong isotropic smoothing constraint. This mechanism can effectively suppress... Branches generate high-frequency pixel noise, forcing the network to generate a scale factor field with gradual spatial variation, thereby simulating real tissue deformation and motion field distribution.

[0055] S6, perform iterative training. Based on a large number of unlabeled surgical endoscopy video sequences, repeat forward inference, false truth correction, loss calculation and backpropagation until the network converges or reaches the preset number of training rounds.

[0056] S7 executes inference deployment. After training, only the depth prediction branch is retained for inference, and a dense depth map is directly output for a single frame of surgical image, thus ensuring inference speed and deployment efficiency.

[0057] The effectiveness of this invention compared to traditional monocular endoscopy depth estimation methods is verified on common datasets Hamlyn and SCARED. This experiment employs a spatiotemporal joint evaluation system, using temporal consistency error (…). The dynamic stability of a video sequence is measured by combining absolute relative error (AbsRel), squared relative error (SqRel), root mean square error (RMSE), and logarithmic root mean square error (RMSE). ) and threshold accuracy ( The invention employs standard metrics such as [list of metrics] to comprehensively quantify the geometric accuracy and robustness of depth estimation. It is implemented using the PyTorch 2.2.0 interface, and the main hardware component is an NVIDIA RTX A6000 graphics card with 48GB of video memory.

[0058] The experimental results are compared with those of currently popular algorithms such as TRMDSV, DepthAnythingv1, DepthAnythingv2, Zoedepth, AF-SFM, and EndoDAC. The best results are highlighted in bold.

[0059] Table 1: Performance Comparison of Depth Estimation on the Hamlyn Dataset

[0060]

[0061] The method proposed in this invention achieves optimal geometric accuracy and strongest spatiotemporal stability in all evaluations on the Hamlyn dataset, outperforming existing mainstream depth estimation methods such as TRMDSV, EndoDAC, and DepthAnything. This result fully demonstrates the technical superiority of the self-supervised depth estimation framework based on deformation scale factor correction proposed in this invention: relying on the decoupled design of the dual-head neural network, the model internalizes the physical understanding of the dynamic characteristics of the surgical scene, enabling it to adaptively identify and distinguish between rigid backgrounds and non-rigid interference regions; simultaneously, by leveraging the learned elastic correction mechanism, it utilizes... The factor actively absorbs geometric errors caused by soft tissue peristalsis, significantly reducing AbsRel (0.0485) and RMSE (8.3157) while ensuring the physical consistency of the 3D reconstruction results. It not only achieves robust depth estimation in complex real surgical sequences but also effectively avoids the lack of supervision and geometric artifacts caused by the "rigid assumptions" of existing methods.

[0062] Table 2: Performance Comparison of Depth Estimation on the SCARED Dataset

[0063]

[0064] The method proposed in this invention achieves state-of-the-art performance across all six key metrics of the SCARED dataset, outperforming existing mainstream depth estimation methods. This result further confirms the technical superiority of the self-supervised depth estimation framework based on deformation scale factor correction proposed in this invention: relying on the construction logic of deformation-aware consistency loss and identity regularization loss, it significantly improves the geometric accuracy of depth prediction, further reducing AbsRel to 0.0339; simultaneously, by leveraging… The synergistic design of factor field smoothing constraints and multimodal features ensures that the lowest time consistency error across the entire surface can be maintained even in regions of severe deformation. (0.11462) and the highest threshold accuracy (0.9894). This demonstrates the model's extremely high robustness in complex dynamic environments.

[0065] Example:

[0066] The known conditions (i.e., input variables) of this invention are: a surgical endoscopic video sequence. Let the current time be... The input image (i.e., the target frame) is Adjacent times ( or The reference image (i.e., the source frame) is The camera intrinsic parameter matrix is The training set contains a large number of unlabeled surgical endoscopic image sequences.

[0067] Undetermined variable: The density depth map of the surgical image at the current moment. .

[0068] Explanation of known conditions: This invention employs a self-supervised learning framework, with the input being a monocular endoscope video stream. During the training phase, network parameters are optimized using spatiotemporal constraints between adjacent frames; during the inference phase, only a single frame image is required as input. It can output a depth map .

[0069] Please see Figure 1 The specific steps are as follows:

[0070] Step 1: Construct and initialize a dual-head depth estimation network. Build a neural network model that includes a shared encoder and two parallel decoders (Depth Head and Gamma Head).

[0071] (1) Feature extraction: A ViT-based Depth Anything architecture is used as a shared encoder to extract features from the input image. High-dimensional semantic features .

[0072] (2) Deep prediction branch: The DPT decoding strategy is adopted to extract features Upsampling and regression are performed to generate a predicted depth map for the current frame. .

[0073] (3) Deformation-aware branch: A lightweight convolutional decoder is used for feature processing. The process is performed, and at the end, an exponential activation function is used to output the global deformation scale factor field. .

[0074] Step 2: Network Forward Propagation and Initial Prediction. The current frame image from the training set... Input network.

[0075] (1) The depth branch outputs the dense depth map predicted in the current frame. Its resolution is consistent with the input image.

[0076] (2) Deformation sensing branch outputs global scale factor field ,in Represents pixels The deformation scaling factor at the location is always greater than 0.

[0077] Step 3: Generate pseudo-true values ​​based on the rigidity assumption. Using rigid optical flow or pose estimation networks from traditional self-supervised methods, calculate the source frame... To target frame The rigid transformation relationship is used. The depth information or image information of the source frame is projected onto the current view according to the rigid transformation to generate a pseudo-true depth map based on the rigidity assumption. Here This represents an uncorrected depth projection that only considers rigid motion, and it contains errors in non-rigid regions.

[0078] Step 4: Elastic correction based on deformation scaling factor. Utilizing the global scaling factor output in Step 2. The pseudo-true depth map obtained in step three Perform pixel-by-pixel calibration to obtain the corrected self-supervised target. The calculation formula is as follows:

[0079]

[0080] In rigid background regions, network learning enables... , making Maintaining rigid geometric constraints. In regions of organizational deformation, the network adaptively adjusts. The value absorbs geometric deviations.

[0081] Step 5: Construct a composite loss function and calculate the total loss. To optimize network parameters, a composite loss function is constructed that includes deformation-aware temporal consistency, identity regularization, and smoothness constraints. :

[0082]

[0083] The specific calculation steps are as follows:

[0084] (1) Calculate the deformation sensing time consistency loss :

[0085]

[0086] in The current frame (time) The dense depth map predicted by the network. Pseudo-true depth maps generated based on the rigidity assumption The core innovation of this invention is the local deformation scale factor field. It is the Structural Similarity Index Measure. It is a balance coefficient, unlike the direct calculation method used in traditional methods. and The difference in depth is used to calculate the predicted depth in this invention. Compared with the corrected pseudo-true value The loss function, which considers structural similarity and photometric differences between the components, allows the network to adjust its loss function in dynamic regions. This minimizes reprojection errors, thereby protecting the integrity of the depth map structure.

[0087] (2) Calculate the identity regularization loss :

[0088]

[0089] It is the total number of pixels in the image. Is Time, location The local deformation scaling factor at the location, this loss term forces To prevent the value from approaching 1 in most pixel locations, thus preventing... Factor overfitting is only permitted in regions where geometric conflict is significant (i.e., the rigidity error is much greater than the regularization penalty). Deviation 1. This enables automatic discovery and sparsity correction of dynamic regions.

[0090] (3) Calculate edge-aware smoothing loss :

[0091]

[0092] , These represent the spatial gradient operators in the horizontal and vertical directions, respectively. It is output from the network. Take the natural logarithm. This loss function does not introduce edge weights from the image gradient and is designed to... The field is subjected to a strong isotropic smoothing constraint. This mechanism can effectively suppress... Branches generate high-frequency pixel noise, forcing the network to generate a scale factor field with gradual spatial variation, thereby simulating real tissue deformation and motion field distribution.

[0093] Step Six: Backpropagation and Parameter Update.

[0094] The total loss calculated in step five The encoder and decoder parameters in the dual-head depth estimation network are iteratively updated using the gradient descent algorithm. Steps two through six are repeated until the network converges or reaches the preset number of training rounds.

[0095] Step 7: Inference Phase. After training is complete, depth estimation is performed on new surgical images:

[0096] (1) Input a single frame surgical image To the trained backbone network.

[0097] (2) Enable only the depth prediction branch (Depth Head) and directly output the dense depth map. .

[0098] (3) The deformation-sensing branch (Gamma Head) does not participate in inference calculations to ensure inference speed. The output... This is the final 3D reconstruction depth result.

[0099] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A self-supervised depth estimation method for surgical scenes, characterized in that, Includes the following steps: S1: A pre-trained Vision Transformer is used as a shared encoder to extract high-dimensional semantic features from the input monocular endoscope image. A depth prediction branch and a deformation perception branch are then connected in parallel after the encoder. S2: For the target frame The depth branch outputs the dense depth map predicted for the current frame. Deformation branch outputs global scale factor field The exponential activation function is used at the end of the deformation branch; S3: Utilize adjacent source frames The camera intrinsic parameter matrix K and the rigid pose or rigid optical flow estimation results are used to project the source frame depth information onto the target frame coordinate system, resulting in a pseudo-ground value depth map that only considers rigid motion. ; S4: Perform elastic correction based on deformation scale factor: using the global scale factor field For pseudo-true depth maps Perform pixel-by-pixel calibration to form a corrected self-supervised target. ; For rigid background regions Adaptive convergence to 1; for tissue deformation regions, Deviation from 1 is used to absorb non-rigid errors; S5: Construct a composite loss function: combine deformation-aware temporal consistency loss, identity regularization loss and smoothing constraint loss to perform end-to-end optimization of the dual-head network and maintain stable rigid geometric constraints in the background region. The identity regularization loss is expressed as: Is Time, location The local deformation scale factor at the location is always greater than 0. It is the total number of pixels in the image, and this loss term forces To prevent the value from approaching 1 in most pixel locations, thus preventing... Factor overfitting is only permitted in regions where geometric conflicts are significant. Deviation 1; S6: Based on a large number of unlabeled surgical endoscopy video sequences, forward inference, pseudo-truth value correction, loss calculation and backpropagation are repeatedly performed until the network converges or reaches the preset number of training rounds. S7: After training, only the depth prediction branch is retained for inference, and a dense depth map is directly output for a single frame of surgical image.

2. The self-supervised depth estimation method for surgical scenes according to claim 1, characterized in that, In S2, the dense depth map predicted in the current frame The resolution is the same as the input image.

3. The self-supervised depth estimation method for surgical scenes according to claim 1, characterized in that, In S4, the elastic correction relationship is expressed as follows: In rigid background regions, network learning enables... , making Maintaining rigid geometric constraints, the network adaptively adjusts in areas of instrument movement or tissue deformation. The value absorbs geometric deviations.

4. The self-supervised depth estimation method for surgical scenes according to claim 1, characterized in that, In S5, the total loss function is expressed as: in, For the total loss function, For deformation sensing temporal consistency loss, For identity regularization loss, To smooth the constraint loss, and These are the weighting coefficients for the corresponding loss terms.

5. The self-supervised depth estimation method for surgical scenes according to claim 4, characterized in that, The deformation sensing temporal consistency loss is expressed as: in It is a structural similarity index. It is the balance coefficient.

6. The self-supervised depth estimation method for surgical scenes according to claim 4, characterized in that, The smoothing constraint loss is expressed as: , These represent the spatial gradient operators in the horizontal and vertical directions, respectively. It is output from the network. Take the natural logarithm.

Images