Methods, systems, and storage media for depth estimation and 3D reconstruction of bronchial pipes.

By employing unsupervised learning-based depth estimation methods and improved bronchoscopy techniques, and utilizing vision-transformer and dense cascade networks for depth estimation and 3D reconstruction, the problem of inaccurate depth estimation in bronchoscopy navigation is solved, thereby improving detection accuracy and visualization of lesion sites.

CN117689706BActive Publication Date: 2026-11-10TONGJI UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202311592784.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-27
Publication Date
2026-11-10
Estimated Expiration
2043-11-27

AI Technical Summary

Technical Problem

Current bronchoscopy techniques rely on monocular vision systems for navigation, which lacks accuracy in depth estimation. This leads to doctors relying on experience, increases patient discomfort, and makes it difficult to achieve efficient detection of lesions.

Method used

An unsupervised learning-based depth estimation method is adopted. The depth estimation model is trained through a bronchoscopy detection video stream, and 3D reconstruction is performed by combining SIFT feature stitching. Feature extraction and aggregation are performed using a vision-transformer and a dense cascaded network. A spatial attention module and an improved photometric reconstruction loss function are also introduced.

Benefits of technology

It improves the accuracy of depth estimation and the precision of three-dimensional reconstruction of lesion sites, reduces patient suffering, reduces reliance on doctors' experience, and enables more precise detection and observation of the bronchial tract.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117689706B_ABST
    Figure CN117689706B_ABST
Patent Text Reader

Abstract

This invention discloses a method, system, and storage medium for depth estimation and 3D reconstruction of the bronchial tract, belonging to the field of intelligent medical technology. This invention obtains an unlabeled bronchial tract dataset by extracting data from a bronchoscopy video stream, calibrates the bronchoscopic intrinsic parameters, and then uses an unsupervised learning method to estimate the depth. The network part introduces a transformer instead of the traditional U-Net structure, and the depth estimation result is obtained through embedding, feature extraction, encoding, and decoding. Furthermore, a panoramic image of the area of ​​interest can be obtained through SIFT feature stitching and then reconstructed. This invention uses an unsupervised method instead of supervised methods, solving the problem of difficult acquisition of internal human body data, and the improved network structure greatly enhances the accuracy of the depth estimation results, providing a purely visual direction for future bronchoscopy navigation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent medical technology, and more specifically, to a method, system, and storage medium for depth estimation and three-dimensional reconstruction of bronchial ducts. Background Technology

[0002] Bronchoscopy is commonly used to observe lung lesions. In conjunction with a monocular vision system, it is inserted through the nasal cavity into the patient's bronchi for in vivo examination. It can detect and observe lesions and assist in surgeries such as polyp removal, playing a significant role in bronchial-related surgeries, disease detection, and postoperative monitoring. Today, flexible bronchoscopes are typically equipped with electronic vision systems, making it increasingly convenient for doctors to examine lesions and more patient-friendly. However, current bronchoscopy technology still has some drawbacks: it relies heavily on a simple monocular vision system for navigation, is highly dependent on the doctor's experience, and is not conducive to controlling the depth of the lens and the detection and observation of lesions, potentially increasing patient discomfort.

[0003] Therefore, navigation technology in monocular bronchoscopy is particularly important during bronchoscopic procedures, and a key component of the monocular visual navigation system is depth estimation. With monocular depth estimation in bronchoscopy, doctors can more accurately grasp the depth, and based on this depth, three-dimensional reconstruction of the lesion site can be achieved, which is more conducive to doctors' detection and observation of the bronchial tract, while also reducing patient discomfort.

[0004] Currently, most human body depth estimation is based on supervised learning models, which require a large amount of labeled human data. However, due to ethical and other reasons, it is often difficult to obtain sufficient labeled data to train the model. Therefore, unsupervised learning is needed to guide the training of the network through various other constraints to achieve the same or even better training results. Summary of the Invention

[0005] To address the aforementioned deficiencies in existing technologies, this invention provides a method, system, and storage medium for depth estimation and 3D reconstruction of bronchial ducts. The method employs unsupervised learning to estimate the depth using an optimized depth estimation model, and then performs 3D reconstruction after obtaining a panoramic view of the lesion (or point of interest) through SIFT feature stitching.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for estimating the depth of the bronchial duct, characterized in that it obtains a depth map of the bronchial duct from a bronchoscope inspection video stream based on an unsupervised learning depth estimation model, thereby completing the depth estimation; specifically including the following steps:

[0007] Step S1: Create a dataset based on the bronchoscopy video stream;

[0008] Step S2: Calibrate the camera internal parameters of the bronchoscope;

[0009] Step S3: Using the intrinsic parameters and dataset as input, train the depth estimation model. The depth estimation model includes two networks: a depth network and a pose network. The input of the depth network is the view of the target, and the output is the corresponding depth image processed by the convolutional neural network. The input of the pose network is the view of the target and the view of the previous frame, and the output is the relative pose transformation of the camera between these two frames. Based on the outputs calculated by the two networks, reconstruct the images of the previous and next frames, project them onto the source image, calculate the corresponding reprojection error, update the model parameters through backpropagation of the loss function, and optimize the trained network model.

[0010] Step S4: Obtain the RGB panoramic image that needs to be locally reconstructed by stitching together SIFT features;

[0011] Step S5: Input the stitched image into the trained depth estimation model to obtain the corresponding depth map.

[0012] Further, step S1 includes the following sub-steps:

[0013] Step S11: Collect the video stream from the bronchoscope at the hospital and extract it into a set of images by skipping frames;

[0014] Step S12: After cropping, further filter the images to remove those that are too dark or have too strong lighting, otherwise spots will appear in the depth map;

[0015] Step S13: Crop and resize the filtered dataset, keeping all images at the same size.

[0016] Furthermore, in step S2, the camera intrinsic parameters are calibrated using the Zhang Zhengyou camera calibration method. After detecting more than three images and more than four feature points in each image, the camera intrinsic parameters are calibrated, and then the calibrated intrinsic parameters are normalized.

[0017] Furthermore, the deep network uses a vision-transformer for feature extraction, and a dense cascaded network is used downstream for feature aggregation. In the upper vision-transformer, features are extracted while the feature map is downsampled, and finally four feature vectors of different sizes are output to the dense cascaded network. In the dense cascaded network, features of different sizes are aggregated using bilinear interpolation and connection operations.

[0018] Furthermore, a spatial attention module is embedded in the pose network.

[0019] Furthermore, the loss function L is the photometric reconstruction loss improved by minimizing photometric error. p Add smoothness loss L s :

[0020] L = L p +L s ,

[0021]

[0022] Photometric error function

[0023]

[0024]

[0025] in, Represents source image I t Relative pose to its adjacent frames; L P The goal is to find the photometric reprojection error L. P Minimum depth map D t α = 0.85, proj represents the depth map in adjacent frames. The projected 2D coordinates, <> are the sampling operators, and K is the intrinsic parameter of the camera; It is the inverse depth of the average normalized value. (i = x or y) represents the gradient of the image along the i-th direction.

[0026] Secondly, the present invention provides a three-dimensional reconstruction method for bronchial ducts, characterized in that a depth map is obtained by using the aforementioned depth estimation method, and then a single image is reconstructed in RGB-D by combining the original image and its depth map.

[0027] Thirdly, the present invention provides a depth estimation system for a bronchial pipe, characterized in that it includes the following components for implementing the depth estimation method described above:

[0028] The dataset module is used to create datasets based on bronchoscopy detection video streams;

[0029] Camera intrinsic parameter calibration module, used to calibrate camera intrinsic parameters;

[0030] The image preprocessing module is used to preprocess and stitch images to obtain the original image.

[0031] A depth estimation model is used to obtain the depth map corresponding to the original image.

[0032] Fourthly, the present invention provides a three-dimensional reconstruction system for bronchial ducts, characterized in that it is used to implement the three-dimensional reconstruction method described above, including the depth estimation system described above and a three-dimensional reconstruction module for performing single-image RGB-D reconstruction by combining the original image and its depth map.

[0033] Finally, the present invention provides a computer-readable storage medium having computer instructions stored thereon, characterized in that, when the computer instructions are executed by a processor, the depth estimation method for bronchial ducts as described above or the three-dimensional reconstruction method for bronchial ducts as described above is implemented.

[0034] Compared with the prior art, the present invention has the following technical effects:

[0035] (1) This invention uses an unsupervised method instead of a supervised method, which solves the problem of difficulty in obtaining data inside the human body;

[0036] (2) In this invention, the depth estimation model uses a vision-transformer for feature extraction, and a dense cascaded network is used downstream for feature aggregation. In the dense cascaded network, bilinear interpolation and connection operations are used to aggregate features of different sizes. A spatial attention module is added to the pose network. These improvements to the network structure greatly improve the training effect of the model and the accuracy of the depth estimation results.

[0037] (3) The loss function of the depth estimation model of the present invention is an improved photometric reconstruction loss plus a smoothness loss, which effectively solves the problem of depth discontinuity caused by pixel blocking. Attached Figure Description

[0038] Figure 1 This is a flowchart of a method for estimating the depth of a branch gas pipeline in one embodiment of the present invention;

[0039] Figure 2 This is a structural diagram of a deep network in one embodiment of the present invention;

[0040] Figure 3 This describes the processing procedure of a deep network in one embodiment of the present invention;

[0041] Figure 4 This is an example of the effect of estimating the depth of a branch gas duct in one embodiment of the present invention;

[0042] Figure 5 This is a flowchart of a three-dimensional reconstruction method for a bronchial gas pipeline according to an embodiment of the present invention;

[0043] Figure 6 This is an example of the effect diagram of three-dimensional reconstruction of a bronchial gas pipeline in one embodiment of the present invention;

[0044] Figure 7This is a structural diagram of a depth estimation system in one embodiment of the present invention;

[0045] Figure 8 This is a structural diagram of a three-dimensional reconstruction system according to an embodiment of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but these are not intended to limit the scope of the invention.

[0047] In the following detailed description, numerous specific details are set forth to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that well-known algorithms and models are not shown in detail to avoid obscuring the spirit of the invention. The detailed description includes technical terms such as epoch: meaning the training process that goes through all samples in the training dataset once (and only once); batch: meaning a set of samples input into the model at one time; loss: loss, which those skilled in the art will understand in their conventional meaning.

[0048] Furthermore, the execution order of actions, steps, etc. in the apparatus and methods shown in the claims, specification, and drawings can be implemented in any order, unless a specific order is explicitly specified, and as long as the output of the preceding processing is not used in the subsequent processing.

[0049] Example 1: Depth estimation of the gas duct

[0050] See Figure 1 This embodiment provides a method for estimating the depth of the bronchial duct. Based on an unsupervised learning depth estimation model, it obtains the depth map of the bronchial duct from the bronchoscope inspection video stream, thereby completing the depth estimation. Specifically, it includes the following steps:

[0051] Step S1: Create a dataset based on the bronchoscopy video stream. More specifically, step S1 includes the following sub-steps:

[0052] Step S11: Collect the video stream from the bronchoscope at the hospital and extract it into an image set by skipping frames. Frame-by-frame extraction is necessary because while capturing images frame-by-frame yields a large amount of data, the camera movement in the video is too small, which is detrimental to training results. In practice, one frame can be extracted every seven frames.

[0053] Step S12: After cropping, further filter the images. Because the lighting conditions of the endoscope are different, it is necessary to remove images that are too dark or have too strong lighting, otherwise spots will appear in the depth map.

[0054] Step S13: Cropped and resized the filtered dataset, keeping all images the same size. For example, cropped the images, removed the black borders around the endoscope images, and adjusted the sizes to be uniform.

[0055] Step S2: Calibrate the camera intrinsics of the bronchoscope. Preferably, the camera calibration uses the Zhang Zhengyou camera calibration method. First, print a self-made black and white checkerboard pattern. The preferred checkerboard size is 8×11, with each checkerboard size being 15mm×15mm. Then, use the bronchoscope's camera to capture a video stream of the checkerboard pattern from multiple angles. Ensure the checkerboard pattern is completely within the lens during the process and maintain smooth lens movement to avoid blurring. Use OpenCV to read the video stream and extract images, selecting a sufficient number of checkerboard pattern images from each. Place the selected images into a toolkit for matching; the more corner points detected, the better the result. Detect a sufficient number of corner points in each image. Import the MATLAB camera calibration toolkit, detect more than three images, each with more than four feature points, and then calibrate. Export the camera intrinsics. Note that normalized intrinsics are used in this invention; therefore, the exported intrinsics must be normalized before application.

[0056] Step S3: Using the intrinsic parameters and dataset as input, train the depth estimation model; based on the principle of the Structure From Motion (SFM) model, simultaneously train two networks in the depth estimation model within the convolutional neural network: a depth network and a pose network. The input to the depth network is the view of the target, and the output is the corresponding depth image processed by the convolutional neural network.

[0057] More specifically, participate Figure 2 The entire deep network is divided into four stages. First, it goes through patch embedding, which primarily divides the input RGB image into non-overlapping (2×2) patches. These patches are then flattened by a flat layer, followed by a linear layer that projects a vector into a C-dimensional feature space. This vector is then input into the next hierarchical transformer to output the feature map. To generate feature maps of different sizes, a downsampling layer is added between every two hierarchical transformers. This downsampling layer halves the image's height and width, quadruples the number of channels, and then a linear layer projects the 4C dimension to a 2C dimension before inputting it into the next hierarchical transformer to output the feature map. By concatenating four similar structures, four feature maps of different sizes (1 / 2, 1 / 4, 1 / 8, 1 / 16) are obtained and input into the next layer.

[0058] The Transformer was first applied to the field of natural language processing, while the vision-transformer is a Transformer module suitable for image processing tasks. As an example, this embodiment uses the swin-transformer module, an improved version of the vision-transformer, which performs better for image processing tasks. Each swin-transformer module outputs a set of feature maps to the dense cascaded classifier. Through downsampling, four batches of feature maps of different sizes are output and fed into the dense cascaded network (DVMnet) for decoding. More specifically, four hierarchical image features of four different resolutions (1 / 2, 1 / 4, 1 / 8, 1 / 16) from the encoder are passed to each stage of the decoding network. Each feature is first projected onto the same C-dimensional space, and features from different stages are then concatenated element-wise. The feature map of each stage is upsampled to the same size by bilinear interpolation from the feature map of the upper layer, and then concatenated with the feature map resulting from the addition of the upper layer, before being input into the depth decoder.

[0059] See Figure 3 In this embodiment, the deep network uses a vision-transformer for feature extraction. During processing, the image is first segmented into small patches, then flattened and projected for embedding. These patches are then fed into a multi-head self-attention module and a Multilayer Perception (MLP) layer for feature extraction. Simultaneously, a dense cascaded network is used downstream for feature aggregation. In the upper-layer vision-transformer, features are extracted while the feature map is downsampled, ultimately outputting four different-sized feature vectors to the dense cascaded network. In the dense cascaded network, bilinear interpolation and connection operations are used to aggregate the features of different sizes.

[0060] The pose network takes the target view and the view of the previous frame as input, and outputs the relative pose transformation of the camera between these two frames. Based on the outputs calculated by the two networks, the two frames are reconstructed, projected onto the source image, and the corresponding reprojection error is calculated. The model parameters are then updated through backpropagation using a loss function to optimize the trained network model. More specifically, the pose network also consists of an encoder and a decoder. Because a single color image cannot capture the 3D information of a scene, only two consecutive frames of video captured by a monocular camera can obtain the changes in the camera's angle and position relative to the scene. The pose network takes the previous frame image and the current frame image as input, which are a pair of color images, so the pose network receives six channels as input. The overall process of the pose network is similar to that of a deep network, as shown in the figure below. The entire network is divided into two steps: encoding and decoding. The left side shows the encoding module, and the right side shows the decoding module. The network ultimately outputs an axis-angle transformation matrix and a translation transformation matrix. A spatial attention module is also introduced into the pose network, allowing the model to pay more attention to detailed texture features such as spatial texture features in the bronchial tubes during training, which helps improve the training effect.

[0061] The process of training the depth estimation model by taking the intrinsic parameters and dataset as input is as follows:

[0062] Step S31: Image preprocessing was performed, including color enhancement and symmetry reversal, followed by model loading and initialization. The deep network used Hierarchical Transformer for encoding and DCMnet for decoding, while the pose network used the classic ResNet18 for encoding and pose_decoder for decoding. The pose network incorporated a spatial attention mechanism before the encoder, and initialization used pre-trained weights from ImageNet as initial weights, along with hyperparameters such as the learning rate and optimizer settings.

[0063] More specifically, the color enhancement module:

[0064] (1) The brightness was randomly adjusted to the original value of (0.8, 1.2).

[0065] (2) The contrast was randomly adjusted to the original (0.8, 1.2).

[0066] (3) The saturation was randomly adjusted to the original values ​​of (0.8, 1.2).

[0067] (4) The hue is randomly adjusted to the original value of (-1, 1).

[0068] Step S32: Loading data. The dataset class is loaded using a modified KITTIOdomdataset, which mainly modifies the data file loading path and image format. After the dataset is loaded, the corresponding dataloader is generated.

[0069] Step S33: Start training for each epoch. At the beginning of each epoch, first retrieve a batch of data from the dataloader. During the data retrieval process, take three consecutive frames as a group each time to facilitate error calculation. At the same time, resize each image to four different sizes and perform color enhancement. Finally, the input data to be put into the model is in the following format (color, frame_id, scale), where color represents the image tensor, frame_id represents the frame, 0 represents the original image, -1 and 1 represent the previous and next frames respectively, and scale ranges from 0 to 4 to represent four different sizes.

[0070] Step S34: Forward propagation. The input is fed into the process_batch function in the model to calculate the loss value. In process_batch, the input is first fed into the depth network and pose network for encoding and decoding to generate the corresponding depth map and the relative pose between adjacent frames. Then, reprojection is performed through generate_image_pred to reproject the two frames onto the original image. Then, the reprojection error is calculated through compute_reprojection_loss. Finally, the total loss value is calculated through the compute_loss function, which includes the smoothness loss. Also, the losses for each scale cannot be directly added together.

[0071] Step S35: Backpropagation. After calculating the loss value, propagation back through the optimizer updates the parameters. This process is repeated for 40 epochs. The loss curve is observed; a smooth decrease in the loss curve indicates model convergence. Considering that when calculating the reprojection error of multiple source images, commonly used self-supervised depth estimation methods generally average the reprojection errors calculated from multiple images, this can lead to pixel blocking issues, meaning that some pixels in the target image are not visible in some actual source images. If the network predicts the correct depth for such a pixel, the corresponding color in an occluded source image is likely to match the incorrect target, resulting in high photometric loss. These high photometric losses mainly stem from two reasons: one is pixels located too far from the edge (far enough away) to enter the field of view, and the other is pixels blocked during shooting. Masking pixels that do not enter the field of view can reduce the error caused by the first reason; however, it cannot address the issue of blocked pixel values, where averaged reprojection may lead to blurred depth discontinuities. To solve this problem, in this embodiment, the loss function L is the photometric reconstruction loss Lminimum photometric error improved. p Add smoothness loss L s :

[0072] L = L p +L s ,

[0073]

[0074] Photometric error function

[0075]

[0076]

[0077] in, Represents source image I t Relative pose to its adjacent frames; L P The goal is to find the photometric reprojection error L. P Minimum depth map D t α = 0.85, proj represents the depth map in adjacent frames. The projected 2D coordinates, <> are the sampling operators, and K is the intrinsic parameter of the camera; It is the inverse depth of the average normalization, which can prevent depth shrinkage; (i = x or y) represents the gradient of the image along the i-th direction.

[0078] The model's hyperparameters are: epoch = 40, batch_size = 4, learning rate = 1e-5, scheduler_step_size = 15. Because there is no depth ground truth, quantitative analysis of the results is not possible; analysis can only be performed on the details of the depth map.

[0079] Step S4: Obtain the RGB panoramic image that needs to be locally reconstructed by stitching together SIFT features.

[0080] Step S5: Input the stitched image into the trained depth estimation model to obtain the corresponding depth map.

[0081] Figure 4 The figure shows an example of the depth estimation of the bronchial pipe in this embodiment. As can be seen from the figure, the depth estimation model has a high accuracy in estimating the depth.

[0082] Example 2: Three-dimensional reconstruction of bronchial ducts

[0083] See Figure 5 This embodiment provides a three-dimensional reconstruction method for bronchial ducts. A depth map is obtained using the depth estimation method in Embodiment 1, and then a single-image RGB-D reconstruction is performed by combining the original image and its depth map. Specifically, the method includes the following steps:

[0084] Step S6: Determine the areas requiring local reconstruction, such as lesion points, and perform image stitching;

[0085] Step S7: Input the stitched RGB panoramic image into the depth estimation model described in Example 1 to perform depth estimation and obtain a depth map;

[0086] Step S8: Read the RGB panoramic image and the corresponding depth image for RGB-D reconstruction; more specifically, obtain the RGB-D image by calling the open3d create_from_color_and_depth function, and then generate a point cloud from the RGB-D reconstruction for single-image reconstruction.

[0087] Figure 6 The figure shows an example of the three-dimensional reconstruction of the bronchial duct in this embodiment. As can be seen from the figure, the three-dimensional reconstruction method has high visualization accuracy.

[0088] Example 3: Depth Estimation System for Branch Pipes

[0089] See Figure 7 This embodiment provides a depth estimation system for bronchial pipes, including:

[0090] Dataset module 100 is used to create a dataset based on the bronchoscopy detection video stream;

[0091] Camera intrinsic parameter calibration module 101 is used to calibrate camera intrinsic parameters;

[0092] Image preprocessing module 102 is used to preprocess and stitch images to obtain the original image;

[0093] Depth estimation model 103 is used to obtain the depth map corresponding to the original image.

[0094] The depth estimation system for the branch pipe can be used to implement the depth estimation method as described in Example 1.

[0095] Example 4: Three-dimensional reconstruction system for bronchial ducts

[0096] This embodiment provides a three-dimensional reconstruction system for bronchial ducts, used to implement the three-dimensional reconstruction method described in Embodiment 2, including the depth estimation system in Embodiment 3 and a three-dimensional reconstruction module for performing single-image RGB-D reconstruction by combining the original image and its depth map. See also Figure 8 More specifically, the three-dimensional reconstruction module includes a single-image point cloud RGB-D reconstruction module 200.

[0097] Example 5

[0098] This embodiment provides a computer program product and its storage medium. When the computer program is executed by a processor, it implements the depth estimation method or three-dimensional reconstruction method for bronchial ducts as described in Embodiment 1 or Embodiment 2. If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] In summary, this invention provides a method, system, and storage medium for bronchial depth estimation and three-dimensional reconstruction, belonging to the field of intelligent medical technology. This invention utilizes bronchoscopy for detection... videoThis invention extracts data from the streaming media to obtain a dataset of unlabeled bronchial tubules, calibrates the bronchoscope's intrinsic parameters, and then uses an unsupervised learning method to estimate depth. The network part introduces a transformer instead of the traditional U-Net structure, obtaining depth estimation results through embedding, feature extraction, encoding, and decoding. Furthermore, a panoramic image of the area of ​​interest can be obtained by SIFT feature stitching and then reconstructed. This invention uses an unsupervised method instead of supervised one, solving the problem of difficult acquisition of internal human body data. The improved network structure significantly enhances the accuracy of depth estimation results, providing a purely visual direction for future bronchoscopic navigation.

[0100] Those skilled in the art should understand that variations can be implemented by combining existing technology with the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention and will not be elaborated here.

[0101] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and the devices and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention's technical solutions still fall within the protection scope of the present invention.

[0102] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Claims

1. A method for estimating the depth of a branch pipe, characterized in that, A depth estimation model based on unsupervised learning is used to obtain the depth map of the bronchial tract from the bronchoscopy inspection video stream, thereby completing the depth estimation; specifically, it includes the following steps: Step S1: Create a dataset based on the bronchoscopy video stream; Step S2: Calibrate the camera internal parameters of the bronchoscope; Step S3: Using the intrinsic parameters and dataset as input, train a depth estimation model. The depth estimation model includes two networks: a deep network and a pose network. The input of the deep network is the view of the target, and the output is the corresponding depth image processed by a convolutional neural network. The input of the pose network is the view of the target and the view of the previous frame, and the output is the relative pose transformation of the camera between these two frames. Based on the outputs calculated by the two networks, reconstruct the images of the previous and next frames, project them onto the source image, calculate the corresponding reprojection error, and update the model parameters through backpropagation using a loss function to optimize the training network model. The deep network uses a vision-transformer for feature extraction, and its downstream uses a dense cascaded network for feature aggregation. In the upper-layer vision-transformer, features are extracted while the feature map is downsampled, and finally four feature vectors of different sizes are output to the dense cascaded network. In the dense cascaded network, features of different sizes are aggregated using bilinear interpolation and connection operations. A spatial attention module is embedded in the pose network. Step S4: Obtain the RGB panoramic image that needs to be locally reconstructed by stitching together SIFT features; Step S5: Input the stitched image into the trained depth estimation model to obtain the corresponding depth map.

2. The method for estimating the depth of a branch pipe according to claim 1, characterized in that, Step S1 includes the following sub-steps: Step S11: Collect the video stream from the bronchoscope at the hospital and extract it into a set of images by skipping frames; Step S12: After cropping, further filter the images to remove those that are too dark or have too strong lighting, otherwise spots will appear in the depth map; Step S13: Crop and resize the filtered dataset, keeping all images at the same size.

3. The method for estimating the depth of a branch pipe according to claim 1 or 2, characterized in that, In step S2, the camera intrinsic parameters are calibrated using the Zhang Zhengyou camera calibration method. After detecting more than three images and more than four feature points in each image, the camera intrinsic parameters are calibrated, and then the calibrated intrinsic parameters are normalized.

4. The method for estimating the depth of a branch pipe according to claim 1, characterized in that, The loss function L is the photometric reconstruction loss improved by minimum photometric error. Add smoothness loss : , , Photometric error function pe , , , in, Represents the source image Relative pose to its adjacent frames; It is a photometric reprojection error; the goal is to find the photometric reprojection error. Minimum depth map ; , Indicates the depth map in adjacent frames. The projected 2D coordinates, It is the sampling operator, and K is the camera's intrinsic parameter; It is the inverse depth of the average normalized value. Let be the gradient of the image along the i-th direction, where i = x or y.

5. A method for three-dimensional reconstruction of a bronchial duct, characterized in that, A depth map is obtained using the depth estimation method described in any one of claims 1 to 4, and then the original image and its depth map are combined to perform RGB-D reconstruction of a single image.

6. A depth estimation system for a branch gas duct, characterized in that, To implement the depth estimation method as described in any one of claims 1 to 4, the method comprises: The dataset module is used to create datasets based on bronchoscopy detection video streams; Camera intrinsic parameter calibration module, used to calibrate camera intrinsic parameters; The image preprocessing module is used to preprocess and stitch images to obtain the original image. A depth estimation model is used to obtain the depth map corresponding to the original image.

7. A three-dimensional reconstruction system for bronchial ducts, characterized in that, The method for implementing the three-dimensional reconstruction method as described in claim 5 includes the depth estimation system as described in claim 6 and a three-dimensional reconstruction module for performing single-image RGB-D reconstruction by combining the original image and its depth map.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the depth estimation method for the bronchial duct as described in any one of claims 1 to 4 or the three-dimensional reconstruction method for the bronchial duct as described in claim 5 is implemented.

Citation Information

Patent Citations

  • Monocular depth estimation system and method for intelligent pump cavity endoscope image

    CN115423856A