Online self-supervised learning depth estimation method based on binocular endoscope
By employing an online self-supervised learning depth estimation method based on binocular endoscopy, and utilizing self-supervised information and meta-learning techniques, the generalization and accuracy issues of depth estimation methods under different surgical scenarios are addressed. This method achieves high-precision depth estimation in complex lighting and low-texture scenes, balancing adaptability and speed.
Patent Information
- Application Number
- CN202310419384.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-07-04
- Filing Date
- 2023-04-14
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-04-14
Smart Images

Figure CN116503457B_ABST
Abstract
Description
[0001] This invention claims domestic priority for application number CN202210779462.2, filed on July 4, 2022, entitled "An Online Self-Supervised Learning Depth Estimation Method Based on Binocular Endoscopy". Technical Field
[0002] This invention relates to the field of depth estimation technology, and more specifically to an online self-supervised learning depth estimation method, system, storage medium, and electronic device based on binocular endoscopy. Background Technology
[0003] Minimally invasive surgery offers numerous advantages, including less bleeding and lower infection rates. However, traditional dual-image systems (such as laparoscopic images) can only provide qualitative spatial perception, which is detrimental to surgeons' understanding of complex anatomical structures and the creation of safe working spaces for advanced minimally invasive surgical robots. Therefore, accurately reconstructing the three-dimensional structure of the surgical scene based on binocular endoscopic images is of great significance for improving surgical efficiency and safety and promoting the development of intelligent surgical robots.
[0004] In their paper "Progressive Fusion for Unsupervised Binocular Depth Estimation Using Cycled Networks" (published on October 1, 2020, in IEEE Transactions on Pattern Analysis and Machine Intelligence), Pilzer et al. proposed a novel self-supervised deep learning method for predicting depth maps, introducing a Progressive Fusion Network (PFN) specifically designed for binocular depth estimation. This network is based on a multi-scale thinning strategy that combines information from two stereo views. Furthermore, it is proposed to stack the network twice to form a loop. This looping approach can be interpreted as a form of data augmentation, as the network learns from both training set images and synthetic images during training. This architecture is jointly trained with adversarial learning to achieve the capability of binocular depth estimation.
[0005] However, the aforementioned technical solutions have weak generalization ability across different surgical scenarios. Furthermore, most existing algorithms are based on autonomous driving scenarios, and their depth estimation robustness is poor when applied to surgical scenarios with complex lighting changes and simple tissue textures. Therefore, it is necessary to provide an online self-supervised learning depth estimation scheme based on binocular endoscopy that offers better generalization and accuracy. Summary of the Invention
[0006] (a) Technical problems to be solved
[0007] To address the shortcomings of existing technologies, this invention provides an online self-supervised learning depth estimation method, system, storage medium, and electronic device based on binocular endoscopy, which solves the technical problem that the generalization and accuracy of existing depth estimation schemes need to be improved.
[0008] (II) Technical Solution
[0009] To achieve the above objectives, the present invention provides the following technical solution:
[0010] An online self-supervised learning depth estimation method based on binocular endoscopy is proposed. The binocular depth estimation network used has the ability to quickly overlearn and continuously adapt to new scenes using self-supervised information. In real-time reconstruction mode, this method is used to overfit continuous video frames, including:
[0011] S1. Acquire binocular endoscope images and extract multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network.
[0012] S2. Using the decoder network of the current binocular depth estimation network, multi-scale features are fused to obtain the disparity of each pixel in the current frame image;
[0013] S3. Based on the camera's intrinsic and extrinsic parameters, convert the parallax into depth and output it as the result of the current frame image;
[0014] S4. Without introducing external ground truth, update the parameters of the current stereo depth estimation network using self-supervised loss for depth estimation of the next frame image.
[0015] Preferably, in precise measurement mode, the method is used to overfit key image video frames, including:
[0016] Without introducing external ground truth, the parameters of the current stereo depth estimation network are updated using the self-supervised loss corresponding to the selected keyframe image until convergence, and the converged stereo depth estimation network is used for accurate depth estimation of the keyframe image.
[0017] Preferably, the initial model parameters corresponding to the stereo depth estimation network are obtained through meta-learning training, specifically including:
[0018] S100, Randomly select an even number of pairs of stereo images {e1,e2,…,e 2K} and equally divided into support sets and query set and Images are randomly paired to form K tasks
[0019] S200, Inner Circulation Training: Based on The loss is calculated from the support set image to perform a parameter update;
[0020]
[0021] in, This represents the network parameters after the inner loop update; Let α represent the derivative, where α is the learning rate of the inner loop. For the support set image of the k-th task, It is based on the initial parameters φ of the model m The calculated loss; f represents the stereo depth estimation network;
[0022] S300, External Circulation Training: Based on The query set image is used to calculate the meta-learning loss using the updated model, and the initial parameters φ of the model are directly updated. m For φ m+1 ;
[0023]
[0024] Where β is the learning rate of the outer loop; This is the query set image for the k-th task. This is the learning loss of the meta-learning.
[0025] Preferably, the initial model parameters corresponding to the stereo depth estimation network are obtained through meta-learning training, specifically including:
[0026] S100, Randomly select an even number of pairs of stereo images {e1,e2,…,e 2K} and equally divided into support sets and query set and Images are randomly paired to form K tasks
[0027] S200, Inner Circulation Training: Based on The loss is calculated from the support set image to perform a parameter update;
[0028]
[0029] in, This represents the network parameters after the inner loop update; Let α represent the derivative, where α is the learning rate of the inner loop. For the support set image of the k-th task, It is based on the initial parameters φ of the model m The calculated loss; f represents the stereo depth estimation network;
[0030] S300, External Circulation Training: Based on The query set image is used to calculate the meta-learning loss using the updated model, and the initial parameters φ of the model are directly updated. m For φ m+1 ;
[0031]
[0032] Where β is the learning rate of the outer loop; This is the query set image for the k-th task. This is the learning loss of the meta-learning.
[0033] Preferably, in S1:
[0034] The encoder network uses a ResNet18 network to extract feature maps at five scales from the current frame image.
[0035] Preferably, S2 specifically includes:
[0036] The decoder network described above processes the coarse-scale feature map through convolutional blocks and upsampling, concatenates it with the fine-scale feature map, and then performs feature fusion through another convolutional block. The convolutional block is constructed by combining a reflection fill layer, a convolutional layer, and a nonlinear activation unit (ELU).
[0037] Calculate the disparity directly based on the output with the highest network resolution:
[0038] d = k·(sigmoid(conv(Y))-TH)
[0039] Where d represents the disparity estimate of a pixel; k is the preset maximum disparity range; Y is the highest resolution output; TH represents a parameter related to the type of binocular endoscope, taking the value of 0 or 0.5; conv is a convolutional layer; and sigmoid performs range normalization.
[0040] Preferably, in S3, converting parallax into depth means:
[0041]
[0042] Among them, c x1 , These are the intrinsic parameter matrices of the left and right eye endoscopes and cameras in a binocular endoscope. The corresponding parameter in; if f x Take the corresponding internal parameters of the left eye camera When f is the left-eye pixel, then d takes the disparity estimate of the left-eye pixel, and D is the depth estimate of the left-eye pixel; if f x Take the corresponding internal parameters of the right eye camera Then d is the disparity estimate of the right eye pixel, and D is the depth estimate of the right eye pixel; b is the baseline length, i.e. the extrinsic parameter of the binocular camera.
[0043] Preferably, the self-supervised loss in S4 is expressed as:
[0044]
[0045] Among them, L self This represents the self-supervised loss; α1, α2, α3, and α4 are all hyperparameters, l corresponds to the left figure, and r corresponds to the right figure;
[0046] (1) The geometric consistency loss is represented by the left figure:
[0047]
[0048] Wherein, P1 represents the first set of valid pixels; The effective pixel p represents the left-eye depth obtained from the right-eye depth map after camera pose transformation, and D represents the left-eye depth. l ′(p) represents the effective pixel point p using the predicted right-side disparity Dis. R The left eye depth is obtained by sampling on the left eye depth map;
[0049] (2) The geometric consistency loss is shown in the right figure:
[0050]
[0051] Wherein, P2 represents the second set of valid pixels; The effective pixel p represents the right-eye depth obtained from the left-eye depth map after camera pose transformation, and D represents the right-eye depth. r ′(p) represents the effective pixel point p using the predicted left image disparity Dis. L The right eye depth is obtained by sampling on the right eye depth map;
[0052] (3) The left image shows the luminous loss:
[0053]
[0054] Among them, I L (p) represents the left figure, I L ′(p) represents the parallax between the right image and the predicted left image Dis. L (p) Reconstructed image from the left eye endoscope, λ i and λ s To balance the parameters, SSIM LL′ (p) represents I L (p) and IL Image structural similarity of ′(p);
[0055] (4) The image on the right shows the luminous loss:
[0056]
[0057] Among them, I R (p) represents the right figure, I′ R (p) indicates the use of the disparity between the left image and the predicted right image. R (p) Generated reconstructed image from the right eye endoscope, SSIM RR′ (p) represents I R (p) and I′ R Image structural similarity (p);
[0058] (5) The smoothing loss is shown in the left graph:
[0059]
[0060] in, This represents the normalized left eye depth map. and This represents the first derivative along the horizontal and vertical directions of the image;
[0061] (6) The smoothing loss is shown in the right figure:
[0062]
[0063] in, This represents the normalized depth map of the right eye. and This represents the first derivative along the horizontal and vertical directions of the image;
[0064] (7) Indicates the loss of sparse optical flow:
[0065]
[0066] Among them, Dis L (p) represents the predicted left eye disparity map, OF L (p) represents the sparse disparity map of the left eye. R (p) represents the predicted right eye disparity map, OF R (p) represents the right eye sparse disparity map; P3 represents the left eye sparse disparity map OF. L The third set of valid pixels in (p); P4 represents the right eye sparse disparity map OF. RThe fourth set of valid pixels in (p); γ1 and γ2 are balancing parameters, both of which are non-negative and not both of which are 0 at the same time.
[0067] Preferably, the process of obtaining the first set of valid pixels P1 and the second set of valid pixels P2 is as follows:
[0068] Define the left eye disparity predicted by the current binocular depth estimation network as: Right eye parallax is The formulaic expression for the cross-validation mask for the left and right eyes is as follows:
[0069]
[0070]
[0071] in, These are used to determine whether the pixel at position (i,j) in the left and right eye images is within the stereo matching range; i takes the value of any integer between [1,W]; j takes the value of any integer between [1,H]; W represents the image width, and H represents the image height;
[0072] Let c take the value L or R, when If the value is within the range of stereo matching under the current calculation method, then the pixel at position (i,j) is within the range of stereo matching; otherwise, it is not within the range of stereo matching.
[0073] Projection is performed using a camera model, binocular pose transformation, and predicted depth to obtain an effective region mask based on 3D points. Take 0 or 1, when If the value is within the range of stereo matching under the current calculation method, then the pixel at position (i,j) is within the range of stereo matching; otherwise, it is not within the range of stereo matching.
[0074] Obtain the final valid region mask
[0075]
[0076] If pixel p satisfies When c is R, the first set of valid pixels P1 is obtained; when c is L, the second set of valid pixels P2 is obtained.
[0077] Preferably, the process for obtaining the third set of valid pixels P3 and the fourth set of valid pixels P4 is as follows:
[0078] Using the LK optical flow solution algorithm, sparse optical flow (Δx, Δy) is calculated every n pixels in the row and column directions, where Δx represents the horizontal offset of the pixel and Δy represents the vertical offset of the pixel.
[0079] When solving the optical flow from the left image to the right image, only when and Δx>thred1, the disparity at this pixel position is retained as Δx, where KT and thred1 are corresponding preset thresholds. The disparity at positions that do not meet the above conditions or where sparse optical flow is not calculated is set to 0, so as to obtain the final sparse disparity map OF L (p), OF L The pixel points where (p)≠0 constitute the third valid pixel point set P3;
[0080] When solving the optical flow from the right image to the left image, only when and Δx<thred2, the disparity at this pixel position is retained as Δx, where thred2 is a corresponding preset threshold. The disparity at positions that do not meet the above conditions or where sparse optical flow is not calculated is set to 0, so as to obtain the final sparse disparity map OF (p), OF R The pixel points where (p)≠0 constitute the fourth valid pixel point set P4.
[0081] An online self-supervised learning depth estimation system based on a binocular endoscope, the used binocular depth estimation network has the capability of fast overfitting, and can continuously adapt to new scenes by using self-supervised information; in the real-time reconstruction mode, the system is configured to overfit consecutive image video frames, comprising:
[0082] An extraction module, configured to acquire binocular endoscope images, and use the encoder network of the current binocular depth estimation network to extract multi-scale features of the current frame image;
[0083] A fusion module, configured to use the decoder network of the current binocular depth estimation network to fuse the multi-scale features to obtain the disparity of each pixel point in the current frame image;
[0084] A conversion module, configured to convert the disparity into depth according to the internal and external parameters of the camera and output the depth as the result of the current frame image;
[0085] A first estimation module, configured to update the parameters of the current binocular depth estimation network by using the self-supervised loss without introducing external ground truth, for depth estimation of the next frame image.
[0086] Preferably, in the precision measurement mode, the system is configured to overfit key image video frames, comprising:
[0087] A second estimation module, configured to update the parameters of the current binocular depth estimation network by using the self-supervised loss corresponding to the selected key frame image until convergence without introducing external ground truth, and use the converged binocular depth estimation network for precise depth estimation of the key frame image.
[0088] A storage medium storing a computer program for online self-supervised learning depth estimation based on binocular endoscopy, wherein the computer program causes a computer to execute the online self-supervised learning depth estimation method as described above.
[0089] An electronic device, comprising:
[0090] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the online self-supervised learning depth estimation method as described in any one of claims 1 to 10.
[0091] (III) Beneficial Effects
[0092] This invention provides an online self-supervised learning depth estimation method, system, storage medium, and electronic device based on binocular endoscopy. Compared with existing technologies, it has the following advantages:
[0093] In this invention, the binocular depth estimation network used has the ability to quickly overlearn and continuously adapt to new scenes using self-supervised information. The method includes: acquiring binocular endoscopic images; extracting multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network; fusing the multi-scale features using the decoder network of the current binocular depth estimation network to obtain the disparity of each pixel in the current frame image; converting the disparity into depth based on camera intrinsic and extrinsic parameters and outputting it as the result of the current frame image; updating the parameters of the current binocular depth estimation network using self-supervised loss without introducing external ground truth, for depth estimation of the next frame image. By utilizing the similarity of consecutive frames, the overfitting idea on a pair of binocular images is extended to overfitting on time series. Through online learning and continuous updating of model parameters, high-precision tissue depth can be obtained in various binocular endoscopic surgical environments. Attached Figure Description
[0094] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0095] Figure 1 A schematic diagram of the technical framework of an online self-supervised learning depth estimation method based on binocular endoscope provided in this embodiment of the invention;
[0096] Figure 2This is a flowchart illustrating an online self-supervised learning depth estimation method based on binocular endoscope, provided in an embodiment of the present invention. Detailed Implementation
[0097] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0098] This application provides an online self-supervised learning depth estimation method, system, storage medium, and electronic device based on binocular endoscopy. It solves the technical problem that the generalization and accuracy of existing depth estimation schemes need to be improved, and achieves depth estimation of complex in vivo scenes without the need to introduce external ground truth information during the training and testing phases.
[0099] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:
[0100] The embodiments of the present invention are mainly applied to, but not limited to, surgical endoscopy scenarios such as laparoscopic surgery scenarios. They aim to use continuous frame learning to approximate overfitting, and are therefore preferred for scenarios where there is little change between image frames, such as surgical endoscopy scenarios.
[0101] In this embodiment of the invention, the binocular depth estimation network used has the ability to quickly overlearn and can continuously adapt to new scenes using self-supervised information. For example... Figure 1 As shown in section b, this network framework provides a depth estimation scheme with dual-mode switching, specifically:
[0102] In real-time reconstruction mode, the provided online self-supervised learning depth estimation method is used to overfit continuous video frames. It leverages the similarity of consecutive frames to extend the overfitting idea from stereo images to video sequences, forming the real-time model parameters for the first online learning mode, in the following form:
[0103]
[0104] Where, β ol1 The learning rate is for the first mode; To express differentiation; This indicates that the model corresponding to the previous frame image is used. From the current frame image I t The calculated self-supervised loss applies to the (real-time) model parameters φ. t-1 renew.
[0105] In the precise measurement mode, the provided online self-supervised learning depth estimation method is used to prevent overfitting to key video frames selected by the physician. The formula is:
[0106]
[0107] Where, β ol2 The learning rate in the second mode; This indicates that the latest model is being used. From key video frame image I k The calculated self-supervised loss applies to the (precise) model parameters φ. k Update; this process is repeated until convergence.
[0108] The aforementioned dual-mode switching depth estimation can provide real-time point clouds of intraoperative anatomical structures to help doctors intuitively understand the intraoperative three-dimensional structure. It can also achieve high-precision reconstruction of key frames manually selected by doctors based on single-frame overfitting, providing a basis for subsequent measurements, thus balancing speed and accuracy in application.
[0109] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.
[0110] Example:
[0111] In a first aspect, embodiments of the present invention provide an online self-supervised learning depth estimation method based on binocular endoscope, characterized in that the binocular depth estimation network used has the ability to quickly overlearn and can continuously adapt to new scenarios using self-supervised information.
[0112] like Figure 2 As shown, in real-time reconstruction mode, this method is used to prevent overfitting of continuous video frames, including:
[0113] S1. Acquire binocular endoscope images and extract multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network.
[0114] S2. Using the decoder network of the current binocular depth estimation network, multi-scale features are fused to obtain the disparity of each pixel in the current frame image;
[0115] S3. Based on the camera's intrinsic and extrinsic parameters, convert the parallax into depth and output it as the result of the current frame image;
[0116] S4. Without introducing external ground truth, update the parameters of the current stereo depth estimation network using self-supervised loss for depth estimation of the next frame image.
[0117] This invention utilizes the similarity of consecutive frames to extend the overfitting concept from a pair of binocular images to overfitting over time series. By continuously updating the model parameters through online learning, it can obtain high-precision tissue depth in various binocular endoscopic surgical environments.
[0118] The following will describe in detail the steps of the above scheme in real-time reconstruction mode:
[0119] First, it should be noted that the binocular depth estimation network used in the embodiments of the present invention has the ability to quickly overlearn and can continuously adapt to new scenarios by utilizing self-supervised information.
[0120] Specifically, the pre-training stage of the network model abandons the traditional training mode and adopts the idea of meta-learning, which allows the network to learn the depth of one image to predict the depth of another image, thereby calculating the loss and updating the network. This can effectively promote the network's generalization to new scenes and improve its robustness to low-texture complex lighting, while significantly reducing the time required for subsequent overfitting.
[0121] like Figure 1 As shown in section b, the initial model parameters corresponding to the stereo depth estimation network are obtained through meta-learning training, specifically including:
[0122] S100, Randomly select an even number of pairs of stereo images {e1,e2,…,e 2K} and equally divided into support sets and query set and Images are randomly paired to form K tasks
[0123] S200, Inner Circulation Training: Based on The loss is calculated from the support set image to perform a parameter update;
[0124]
[0125] in, This represents the network parameters after the inner loop update; Let α represent the derivative, where α is the learning rate of the inner loop. For the support set image of the k-th task, It is based on the initial parameters φ of the model m The calculated loss; f represents the stereo depth estimation network;
[0126] S300, External Circulation Training: Based on The query set image is used to calculate the meta-learning loss using the updated model, and the initial parameters φ of the model are directly updated. m For φ m+1 ;
[0127]
[0128] Where β is the learning rate of the outer loop; This is the query set image for the k-th task. This is the learning loss of the meta-learning.
[0129] In step S1, as Figure 1 As shown in section a, binocular endoscopic images are acquired, and the encoder network of the current binocular depth estimation network is used to extract multi-scale features of the current frame image.
[0130] For example, in this step, the encoder of the binocular depth estimation network uses a ResNet18 network to extract feature maps at five scales for the current frame image (left and right eyes) respectively.
[0131] In step S2, as Figure 1 As shown in section a, the decoder network of the current binocular depth estimation network is used to fuse multi-scale features and obtain the disparity of each pixel in the current frame image; specifically, it includes:
[0132] The decoder network described above processes the coarse-scale feature map through convolutional blocks and upsampling, concatenates it with the fine-scale feature map, and then performs feature fusion through convolutional blocks again. The convolutional blocks are constructed by combining reflection padding, convolutional layers, and nonlinear activation units (ELUs).
[0133] Calculate the disparity directly based on the output with the highest network resolution:
[0134] d = k·(sigmoid(conv(Y))-TH)
[0135] Where d represents the disparity estimate of a pixel; k is the preset maximum disparity range; Y is the highest resolution output; TH represents a parameter related to the type of binocular endoscope, which is 0.5 when the endoscope image has negative disparity and 0 when all endoscope images have positive disparity; conv is a convolutional layer; and sigmoid performs range normalization.
[0136] In step S3, the parallax is converted into depth based on the camera's intrinsic and extrinsic parameters and output as the result of the current frame image.
[0137] In this step, converting parallax to depth means:
[0138]
[0139] Among them, c x1 , These are the intrinsic parameter matrices of the left and right eye endoscopes and cameras in a binocular endoscope. The corresponding parameter in; if f x Take the corresponding internal parameters of the left eye camera When f is the left-eye pixel, then d takes the disparity estimate of the left-eye pixel, and D is the depth estimate of the left-eye pixel; if f x Take the corresponding internal parameters of the right eye camera Then d is the disparity estimate of the right eye pixel, and D is the depth estimate of the right eye pixel; b is the baseline length, i.e. the extrinsic parameter of the binocular camera.
[0140] In step S4, as Figure 1 As shown in section b, without introducing external ground truth, the parameters of the current stereo depth estimation network are updated using self-supervised loss for depth estimation of the next frame image.
[0141] It is easy to understand that the "external truth value" mentioned in the embodiments of the present invention is the label (or "supervision information"), which is a well-known expression in the art.
[0142] In this step, such as Figure 1 As shown in part b, the self-supervised loss is expressed as:
[0143]
[0144] Among them, L self The value represents the self-supervised loss; α1, α2, α3, and α4 are all hyperparameters, l corresponds to the left figure, and r corresponds to the right figure.
[0145] Since both eyes are observing the same scene, the values of corresponding pixels on the left and right depth maps should be equal when transformed to the same coordinate system. Therefore, we introduce... and
[0146] (1) The geometric consistency loss is represented by the left figure:
[0147]
[0148] Wherein, P1 represents the first set of valid pixels (i.e., the valid pixels of the right eye); The effective pixel p represents the left-eye depth obtained from the right-eye depth map after camera pose transformation, and D represents the left-eye depth. l ′(p) represents the effective pixel point p using the predicted right-side disparity Dis. R The left eye depth is obtained by sampling on the left eye depth map.
[0149] (2) The geometric consistency loss is shown in the right figure:
[0150]
[0151] Wherein, P2 represents the second set of valid pixels (i.e., the valid pixels of the left eye); The effective pixel p represents the right-eye depth obtained from the left-eye depth map after camera pose transformation, and D represents the right-eye depth. r ′(p) represents the effective pixel point p using the predicted left image disparity Dis. L The right eye depth is obtained by sampling on the right eye depth map.
[0152] By incorporating geometric consistency constraints into the training loss, the network's general applicability to hardware is ensured, enabling it to autonomously adapt to unconventional binocular images such as surgical endoscopes.
[0153] Assuming constant brightness and spatial smoothness during endoscopic surgery, reprojection between left and right eye images can achieve reconstruction of another objective. However, this introduces structural similarity loss. The brightness, contrast, and structure of the two images are normalized and compared, and then... and
[0154] (3) The left image shows the luminous loss:
[0155]
[0156] Among them, I L (p) represents the left figure, I L ′(p) represents the parallax between the right image and the predicted left image Dis. L (p) Reconstructed image from the left eye endoscope, λ i and λ s To balance the parameters, SSIM LL′ (p) represents I L (p) and I L Image structural similarity of ′(p);
[0157] (4) The image on the right shows the luminous loss:
[0158]
[0159] Among them, I R (p) represents the right figure, I′ R (p) indicates the use of the disparity between the left image and the predicted right image. R (p) Generated reconstructed image from the right eye endoscope, SSIM RR′ (p) represents I R (p) and I′ R Image structural similarity (p).
[0160] In low-texture and monochromatic organizational regions, smoothing priors are used to aid inference, and depth regularization is applied, introducing... and
[0161] (5) The smoothing loss is shown in the left graph:
[0162]
[0163] in, This represents the normalized left eye depth map. and This represents the first derivative along the horizontal and vertical directions of the image;
[0164] (6) The smoothing loss is shown in the right figure:
[0165]
[0166] in, This represents the normalized depth map of the right eye. and This represents the first derivative along the horizontal and vertical directions of the image.
[0167] Specifically, the process of obtaining the first set of valid pixels P1 and the second set of valid pixels P2 is as follows:
[0168] Define the left eye disparity predicted by the current binocular depth estimation network as: Right eye parallax is The formulaic expression for the cross-validation mask for the left and right eyes is as follows:
[0169]
[0170]
[0171] in, These are used to determine whether the pixel at position (i,j) in the left and right eye images is within the stereo matching range; i takes the value of any integer between [1,W]; j takes the value of any integer between [1,H]; W represents the image width, and H represents the image height;
[0172] Let c take the value L or R, when If the value is within the range of stereo matching under the current calculation method, then the pixel at position (i,j) is within the range of stereo matching; otherwise, it is not within the range of stereo matching.
[0173] Projection is performed using a camera model, binocular pose transformation, and predicted depth to obtain an effective region mask based on 3D points. Take 0 or 1, when If the value is within the range of stereo matching under the current calculation method, then the pixel at position (i,j) is within the range of stereo matching; otherwise, it is not within the range of stereo matching.
[0174] Obtain the final valid region mask
[0175]
[0176] If pixel p satisfies When c is R, the first set of valid pixels P1 is obtained; when c is L, the second set of valid pixels P2 is obtained.
[0177] In the corrected stereo image, additional regions caused by viewpoint shift cannot find matching pixels. However, this embodiment of the invention considers that low texture and uneven illumination of in vivo tissues can make local features less obvious, and pixels in these invalid regions often find similar pixels in neighboring regions. Therefore, as mentioned above, this embodiment of the invention proposes a cross-validation-based binocular effective region recognition algorithm, which eliminates the misleading effect of self-supervised loss of invalid region pixels on network learning and improves the accuracy of depth estimation.
[0178] In addition, to avoid insufficient robustness of depth estimation in pure texture or low-light scenes, a method is also introduced.
[0179] (7) Indicates the loss of sparse optical flow:
[0180]
[0181] Among them, Dis L (p) represents the predicted left eye disparity map, OF L (p) represents the sparse disparity map of the left eye. R (p) represents the predicted right eye disparity map, OF R (p) represents the right eye sparse disparity map; P3 represents the left eye sparse disparity map OF. L The third set of valid pixels in (p); P4 represents the right eye sparse disparity map OF. R The fourth set of valid pixels in (p); γ1 and γ2 are balancing parameters, both of which are non-negative and not both of which are 0 at the same time.
[0182] Specifically, the process of obtaining the third set of valid pixels P3 and the fourth set of valid pixels P4 is as follows:
[0183] Using the LK (Lucas-Kanade) optical flow solving algorithm, sparse optical flows (Δx, Δy) are calculated every n pixels in the row and column directions, where Δx represents the horizontal offset of the pixel point, and Δy represents the vertical offset of the pixel point;
[0184] When solving the optical flow from the left image to the right image, only when and Δx>thred1, the disparity at this pixel position is retained as Δx, where KT and thred1 are corresponding preset thresholds, and the disparity at positions that do not meet the above conditions or where sparse optical flow is not calculated is set to 0, so as to obtain the final sparse disparity map OF L (p), OF L pixel points with (p)≠0 constitute a third valid pixel point set P3;
[0185] When solving the optical flow from the right image to the left image, only when and Δx<thred2, the disparity at this pixel position is retained as Δx, where thred2 is a corresponding preset threshold, and the disparity at positions that do not meet the above conditions or where sparse optical flow is not calculated is set to 0, so as to obtain the final sparse disparity map OF R (p), OF R pixel points with (p)≠0 constitute a fourth valid pixel point set P4.
[0186] As mentioned above, the embodiments of the present invention introduce the traditional Lucas-Kanade optical flow to derive the sparse disparity between binocular images, which provides a reasonable learning direction for the network, improves the fast learning ability and reduces the probability of falling into local optimum.
[0187] It is particularly emphasized that, in addition to the real-time reconstruction mode, the online self-supervised learning depth estimation method provided by the embodiments of the present invention is further provided with a precision measurement mode. As shown in part b of Figure 1 , in the precision measurement mode, this method is used to overfit key image video frames, including:
[0188] Under the condition that no external ground truth is introduced, the parameters of the current binocular depth estimation network are updated using the self-supervised loss corresponding to the selected key frame image until convergence, and the converged binocular depth estimation network is used for precise depth estimation of the key frame image.
[0189] It is worth noting that the technical details such as the depth estimation network, the self-supervised loss function, the valid region mask calculation, and the meta-learning pre-training method in the precision measurement mode are all consistent with the extended technical details in the real-time reconstruction mode. At this time, the binocular depth estimation network that performs parameter update for the first time in the precision measurement mode is exactly the binocular depth estimation network obtained after the previous frame image of the selected key frame image has executed the aforementioned steps S1 to S4 (real-time reconstruction mode).
[0190] Secondly, embodiments of the present invention provide an online self-supervised learning depth estimation system based on binocular endoscopes. The binocular depth estimation network used has the ability to quickly overlearn and can continuously adapt to new scenarios by utilizing self-supervised information.
[0191] In real-time reconstruction mode, the system is used to overfit continuous video frames, including:
[0192] The extraction module is used to acquire binocular endoscopic images and extract multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network.
[0193] The fusion module is used to fuse multi-scale features using the decoder network of the current binocular depth estimation network to obtain the disparity of each pixel in the current frame image.
[0194] The conversion module is used to convert parallax into depth based on camera intrinsic and extrinsic parameters and output it as the result of the current frame image.
[0195] The first estimation module is used to update the parameters of the current stereo depth estimation network using self-supervised loss without introducing external ground truth, for depth estimation of the next frame image.
[0196] In precise measurement mode, the system is used to overfit key video frames and also includes:
[0197] The second estimation module is used to update the parameters of the current stereo depth estimation network until convergence by using the self-supervised loss corresponding to the selected keyframe image without introducing external ground truth, and to use the converged stereo depth estimation network for accurate depth estimation of the keyframe image.
[0198] Thirdly, embodiments of the present invention provide a storage medium storing a computer program for online self-supervised learning depth estimation based on binocular endoscopy, wherein the computer program causes a computer to execute the online self-supervised learning depth estimation method as described above.
[0199] Fourthly, embodiments of the present invention provide an electronic device, comprising:
[0200] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the online self-supervised learning depth estimation method as described above.
[0201] It is understood that the online self-supervised learning depth estimation system, storage medium and electronic device based on binocular endoscope provided in the embodiments of the present invention correspond to the online self-supervised learning depth estimation method based on binocular endoscope provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the online self-supervised learning depth estimation method, and will not be repeated here.
[0202] In summary, compared with existing technologies, it has the following beneficial effects:
[0203] 1. The dual-mode switching depth estimation can provide real-time point clouds of intraoperative anatomical structures to help doctors intuitively understand the intraoperative three-dimensional structure. It can also achieve high-precision reconstruction of key frames manually selected by doctors based on single-frame overfitting, providing a basis for subsequent measurements, thus balancing speed and accuracy in application.
[0204] 2. By utilizing the similarity of consecutive frames, the overfitting concept on a pair of binocular images is extended to overfitting on time series. Through online learning, the model parameters are continuously updated, enabling high-precision tissue depth measurements to be obtained in various binocular endoscopic surgical environments.
[0205] 3. The pre-training stage of the network model abandons the traditional training mode and adopts the idea of meta-learning, which allows the network to learn the depth of one image to predict the depth of another image, thereby calculating the loss and updating the network. This can effectively promote the network's generalization to new scenes and improve its robustness to low-texture complex lighting, while significantly reducing the time required for subsequent overfitting.
[0206] 4. By incorporating geometric consistency constraints into the training loss, the network's general applicability to hardware is ensured, enabling it to autonomously adapt to unconventional binocular images such as surgical endoscopes.
[0207] 5. Depth estimation of each frame of stereo image is treated as an independent task, and high-precision models suitable for the current frame are obtained through real-time overfitting; and high-precision depth estimation results can be obtained by quickly learning new scenes through online learning.
[0208] 6. Based on the cross-validation binocular effective region recognition algorithm, the self-supervised loss of invalid region pixels is eliminated, which misleads the network learning and improves the accuracy of depth estimation.
[0209] 7. Introduce the sparse parallax between binocular images by using traditional Lucas-Kanade optical flow, give the network a reasonable learning direction, improve its fast learning ability and reduce the probability of getting stuck in local optima.
[0210] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0211] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An online self-supervised learning depth estimation method based on binocular endoscopy, characterized in that, The binocular depth estimation network used has the ability to learn quickly and can continuously adapt to new scenarios by utilizing self-supervised information; In real-time reconstruction mode, this method is used to prevent overfitting of continuous video frames, including: S1. Acquire binocular endoscope images and extract multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network. S2. Using the decoder network of the current binocular depth estimation network, multi-scale features are fused to obtain the disparity of each pixel in the current frame image; S3. Based on the camera's intrinsic and extrinsic parameters, convert the parallax into depth and output it as the result of the current frame image; S4. Without introducing external ground truth, update the parameters of the current stereo depth estimation network using self-supervised loss for depth estimation of the next frame image. In precise measurement mode, this method is used to overfit key image video frames, including: Without introducing external ground truth, the parameters of the current stereo depth estimation network are updated using the self-supervised loss corresponding to the selected keyframe image until convergence, and the converged stereo depth estimation network is used for accurate depth estimation of the keyframe image. In both the real-time reconstruction mode and the precise measurement mode, the initial model parameters corresponding to the binocular depth estimation network are obtained through meta-learning; and the self-supervised loss includes at least sparse optical flow loss, which is used to derive the sparse disparity between binocular images based on Lucas-Kanade optical flow, so as to give the binocular depth estimation network a reasonable learning direction.
2. The online self-supervised learning depth estimation method as described in claim 1, characterized in that, The initial model parameters for the stereo depth estimation network are obtained through training using a meta-learning approach, specifically including: S100, Randomly select an even number of pairs of stereo images And equally divided into support sets and query set , and Images are randomly paired to form K tasks ; S200, Inner Circulation Training: Based on The loss is calculated from the support set image to perform a parameter update; in, This represents the network parameters after the inner loop update; To express differentiation, The learning rate for the inner loop. For the first Support set images for each task It is based on the initial parameters of the model Calculated loss; This represents a binocular depth estimation network; S300, External Loop Training: Based on The query set image is used to calculate the meta-learning loss using the updated model, and the initial parameters of the model are directly updated. for ; in, The learning rate for the outer loop; It is the first Image of the query set for each task. This is the learning loss of the meta-learning.
3. The online self-supervised learning depth estimation method as described in claim 1 or 2, characterized in that, In S1: The encoder network uses a ResNet18 network to extract feature maps at five scales from the current frame image.
4. The online self-supervised learning depth estimation method as described in claim 1 or 2, characterized in that, S2 specifically includes: The decoder network described above processes the coarse-scale feature map through convolutional blocks and upsampling, concatenates it with the fine-scale feature map, and then performs feature fusion through another convolutional block. The convolutional block is constructed by combining a reflection fill layer, a convolutional layer, and a nonlinear activation unit (ELU). Calculate the disparity directly based on the output with the highest network resolution: in, This represents the disparity estimate of a pixel. For the preset maximum parallax range, It is the highest resolution output; A parameter related to the type of binocular endoscope, taking the value 0 or 0.5; It is a convolutional layer; Perform range normalization.
5. The online self-supervised learning depth estimation method as described in claim 1 or 2, characterized in that, In S3, converting parallax into depth means: in, , These are the intrinsic parameter matrices of the left and right eye endoscopes and cameras in a binocular endoscope. The corresponding parameters; if Take the corresponding internal parameters of the left eye camera At that time, Take the disparity estimate of the left-eye pixel, where D is the depth estimate of the left-eye pixel; if Take the corresponding internal parameters of the right eye camera ,but Take the disparity estimate of the right eye pixel, where D is the depth estimate of the right eye pixel. The baseline length is the extrinsic parameter of the binocular camera.
6. The online self-supervised learning depth estimation method as described in claim 1 or 2, characterized in that, The self-supervised loss in S4 is expressed as: in, Indicates self-monitoring loss; All of these are hyperparameters. Corresponding to the left figure, Corresponding to the image on the right; (1) The geometric consistency loss is represented by the left figure: in, Represents the first set of valid pixels; Indicates valid pixels The left-eye depth is obtained by transforming the right-eye depth map with camera pose. Indicates valid pixels Using the predicted right-side parallax The left eye depth is obtained by sampling on the left eye depth map; (2) The geometric consistency loss is shown in the right figure: in, Represents the set of the second valid pixels; Indicates valid pixels The right eye depth is obtained by transforming the left eye depth map with camera pose. Indicates valid pixels Using the predicted left image disparity The right eye depth is obtained by sampling on the right eye depth map; (3) The left image shows the luminous loss: in, The left figure is shown. This indicates the use of the parallax between the right image and the predicted left image. The generated reconstructed image from the left eye endoscope. and For balancing parameters, express and Image structural similarity; (4) The image on the right shows the luminous loss: in, The image on the right is shown. This indicates the use of the parallax between the left image and the predicted right image. The generated right eye endoscope reconstructed image, express and Image structural similarity; (5) The smoothing loss is shown in the left graph: in, This represents the normalized left eye depth map. and This represents the first derivative along the horizontal and vertical directions of the image; (6) The smoothing loss is shown in the right figure: in, This represents the normalized depth map of the right eye. and This represents the first derivative along the horizontal and vertical directions of the image; (7) Indicates the loss of sparse optical flow: in, This represents the predicted left-eye disparity map. This represents the sparse parallax map of the left eye. The right eye disparity map represents the prediction. This represents a sparse parallax map of the right eye. Represents the sparse parallax map of the left eye The third set of valid pixels; Represents the sparse parallax map of the right eye The fourth set of valid pixels; and These are balancing parameters, all of which are non-negative and not all of them are 0 at the same time.
7. The online self-supervised learning depth estimation method as described in claim 6, characterized in that, First set of valid pixels Second effective pixel set The acquisition process is as follows: Define the left eye disparity predicted by the current binocular depth estimation network as: The right eye parallax is The formulaic expression for the cross-validation mask of the left and right eyes is as follows: in, , Used to determine the left and right eye images respectively Whether the pixels at the location are within the stereo matching range; The range of values is All integers between; The range of values is All integers in between; Indicates the image width. Indicates the image height; make Pick or ,when When, it indicates the current calculation method. The pixel at the location is within the stereo matching range; otherwise, it is not within the stereo matching range. Projection is performed using a camera model, binocular pose transformation, and predicted depth to obtain an effective region mask based on 3D points. ; Take 0 or 1, when When, it indicates the current calculation method. The pixel at the location is within the stereo matching range; otherwise, it is not within the stereo matching range. Obtain the final valid region mask : If pixel satisfy ,when Pick Then obtain the first set of valid pixels. ,when Pick Then obtain the second set of valid pixels. .
8. The online self-supervised learning depth estimation method as described in claim 6, characterized in that, Third effective pixel set and / or the fourth set of valid pixels The acquisition process is as follows: use LK Optical flow calculation algorithm, every [number] times in the row and column directions Calculate sparse optical flow per pixel ,in This represents the horizontal offset of a pixel. This represents the vertical offset of a pixel. When solving for the optical flow from the left figure to the right figure, only when and Preserve the disparity at that pixel location as ,in and For each preset threshold, the disparity at locations where the above conditions are not met or where sparse optical flow is not calculated is set to 0, in order to obtain the final sparse disparity map. , The pixels constitute the third set of valid pixels. ; When solving for the optical flow from the right figure to the left figure, only when and Preserve the disparity at that pixel location as ,in For each preset threshold, the disparity at locations where the above conditions are not met or where sparse optical flow is not calculated is set to 0, in order to obtain the final sparse disparity map. , The pixels constitute the fourth set of valid pixels. .
9. An online self-supervised learning depth estimation system based on binocular endoscopy, characterized in that, The binocular depth estimation network used has the ability to learn quickly and can continuously adapt to new scenarios by utilizing self-supervised information; In real-time reconstruction mode, the system is used to overfit continuous video frames, including: The extraction module is used to acquire binocular endoscopic images and extract multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network. The fusion module is used to fuse multi-scale features using the decoder network of the current binocular depth estimation network to obtain the disparity of each pixel in the current frame image. The conversion module is used to convert parallax into depth based on camera intrinsic and extrinsic parameters and output it as the result of the current frame image. The first estimation module is used to update the parameters of the current stereo depth estimation network using self-supervised loss without introducing external ground truth, for depth estimation of the next frame image. In precise measurement mode, the system is used to overfit key video frames, including: The second estimation module is used to update the parameters of the current stereo depth estimation network until convergence by using the self-supervised loss corresponding to the selected keyframe image without introducing external truth values, and to use the converged stereo depth estimation network for accurate depth estimation of the keyframe image. In both the real-time reconstruction mode and the precise measurement mode, the initial model parameters corresponding to the binocular depth estimation network are obtained through meta-learning. Furthermore, the self-supervised loss includes at least sparse optical flow loss, which is used to derive the sparse disparity between binocular images based on Lucas-Kanade optical flow, so as to give the binocular depth estimation network a reasonable learning direction.
10. A storage medium, characterized in that, It stores a computer program for online self-supervised learning depth estimation based on binocular endoscopy, wherein the computer program causes the computer to execute the online self-supervised learning depth estimation method as described in any one of claims 1 to 8.
11. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the online self-supervised learning depth estimation method as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Three-dimensional reconstruction method and device
CN108961383A
Pyramid binocular depth estimation model with a self-lifting capability
CN109801323A
Online three-dimensional reconstruction method of binocular endoscope soft tissue image
CN111080778A
Foresight scene depth estimation method based on self-supervised learning
CN113313732A
Face anti-counterfeiting method based on adaptive meta-learning
CN113343771A
