Systems and methods for endoscopic video enhancement, quantification, and surgical guidance

Abstract

An endoscope system includes: an endoscope imager configured to capture an image frame of a target site within a living body; and a processor configured to apply a spatial transformation to the preliminary group of image frames, the spatial transformation converting the image frames to cylindrical coordinates; calculating a mapping image from the spatially transformed image frames, each pixel position in the mapping image being defined by a vector of a fixed dimension; aligning the current image frame with the mapped image and applying a spatial transformation to the current image frame; fusing the current image frame after spatial transformation to the mapping image to generate a fused image; and applying an inverse spatial transformation to the fused image to generate an enhanced current image frame having a greater spatial resolution than the current image frame. The system also includes a display that displays the enhanced current image frame.

Description

[0001] Related Applications

[0002] This disclosure claims priority to U.S. Provisional Patent Application Serial No. 62 / 904,408, filed on September 23, 2019; the disclosure of which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to systems and methods for endoscopic video enhancement, quantification, and surgical guidance. Background Art

[0004] Endoscopic imagers may be used during a variety of medical interventions. The view of the patient's anatomy provided by the imager is limited by resolution and field of view. This limited view of the anatomy may increase the time of the intervention and may not provide the surgeon with all the information necessary to perform the intervention. Summary of the invention

[0005] The present disclosure relates to an endoscope system, including an endoscope imager configured to capture image frames of a target site in a living body, and a processor, the processor being configured to apply a spatial transformation to a preliminary image frame group, the spatial transformation converting the image frame into cylindrical coordinates; calculating a mapping image based on the spatially transformed image frame, each pixel position in the mapping image being defined by a vector of fixed dimension; aligning a current image frame with the mapping image, and applying a spatial transformation to the current image frame; fusing the spatially transformed current image frame with the mapping image to generate a fused image; and applying an inverse spatial transformation to the fused image to generate an enhanced current image frame having a greater spatial resolution than the current image frame. The system also includes a display for displaying the enhanced current image frame.

[0006] In one embodiment, the spatial transformation is generated based on the optical geometry of the endoscopic imager.

[0007] In one embodiment, the mapping image has a resolution that is an integer multiple greater than the resolution of the endoscopic imager.

[0008] In one embodiment, the current image frame is aligned with the mapped image based on a cross-correlation, wherein a similarity between the current image frame and the mapped image is measured.

[0009] In one embodiment, the processor is further configured to expand the field of view of the enhanced current image frame compared to the current image frame based on the mapped image region around the spatially transformed current image frame.

[0010] In one embodiment, the processor is further configured to add the spatially transformed current image frame to the mapped image.

[0011] In one embodiment, when a given pixel location in the mapped image is full, the oldest samples are deleted when new samples are added.

[0012] The present disclosure also relates to an endoscopic system including an endoscopic imager configured to capture image frames of a target site in a living body, and a processor. The processor is configured to: apply a spatial transformation to a preliminary set of image frames, the spatial transformation converting the image frames into cylindrical coordinates; calculate a mapping image from the spatially transformed image frames, each pixel position in the mapping image being defined by a vector of fixed dimension; calculate a scale space representation of the mapping image; capture a further image including a plurality of independent regions; when a predetermined amount of image data for each independent region has been acquired, develop a nonlinear spatial transformation including an independent spatial transformation for each independent region; and derive a structure from motion (SFM) depth map from the nonlinear spatial transformation.

[0013] In one embodiment, the SFM depth map is also based on tracking information of the endoscopic imager, which includes the changing pose of the endoscopic imager between captured images.

[0014] In one embodiment, the processor is further configured to: identify and segment endoscopy-related objects and objects of interest in the preliminary set of image frames; and exclude the identified endoscopy-related objects when the spatial transformation is applied to the preliminary set of images.

[0015] In one embodiment, the processor is further configured to estimate the size of the object of interest based on the depth information and the angular range in the current image frame.

[0016] In one embodiment, the endoscopy system further comprises a display configured to display a current image frame with the annotated object of interest.

[0017] In one embodiment, the object of interest is a kidney stone.

[0018] In one embodiment, the endoscopic system further comprises an electromagnetic (EM) tracker attached to the endoscopic imager, the electromagnetic (EM) tracker being configured to provide tracking data including a six-degree-of-freedom position of the endoscopic imager. The processor is further configured to segment a previously acquired 3D image volume of the target site. The tracking data is combined with the segmented image volume and the SFM depth map to provide a position estimate of the endoscopic imager.

[0019] In one embodiment, the processor is further configured to deform the segmented image volume when the tracking data is shown to breach a surface of the segmented image volume.

[0020] In addition, the present invention relates to a method, which includes applying a spatial transformation to a preliminary image frame group of a target site in a living body, which spatial transformation converts the image frames into cylindrical coordinates; calculating a mapping image based on the spatially transformed image frames, each pixel position in the mapping image is defined by a vector of fixed dimension; aligning a current image frame with the mapping image, and applying a spatial transformation to the current image frame; fusing the spatially transformed current image frame with the mapping image to generate a fused image; and applying an inverse spatial transformation to the fused image to generate an enhanced current image frame having a greater spatial resolution than the current image frame.

[0021] In one embodiment, the spatial transformation is generated based on the optical geometry of the endoscopic imager.

[0022] In one embodiment, the mapping image has a resolution that is an integer multiple greater than the resolution of the endoscopic imager.

[0023] In one embodiment, the current image frame is aligned with the mapped image based on a cross-correlation, wherein a similarity between the current image frame and the mapped image is measured.

[0024] In one embodiment, the method further comprises: expanding the field of view of the enhanced current image frame compared to the current image frame based on the mapped image region around the spatially transformed current image frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A system for performing a urological surgical procedure according to various exemplary embodiments of the present disclosure is shown.

[0026] Figure 2 A method for improving the detail and / or field of view of an endoscopic video according to a first exemplary embodiment is shown.

[0027] Figure 3 Diagram showing the spatial transformation from image coordinates to cylindrical coordinates.

[0028] Figure 4 A method for improving endoscopic video according to a second exemplary embodiment is shown.

[0029] Figure 5 A method for annotating endoscopic videos with object size information is shown.

[0030] Figure 6 An exemplary endoscopic video display showing a kidney stone being bracketed is shown.

[0031] Figure 7 A method for tracking the position of an endoscope relative to a continuously deforming anatomical structure according to a first exemplary embodiment is shown.

[0032] Figure 8A method for motion tracking in endoscopic video of a tubular structure according to a second exemplary embodiment is shown.

[0033] Fig. 9 An exemplary endoscopy video display is shown with tracking annotations shown.

[0034] Fig.10 A method for motion tracking in endoscopic video of a tubular structure according to a third exemplary embodiment is shown.

[0035] Fig.11 A method for analyzing a cystoscopic video feed and providing annotations to inform a laser therapy procedure is shown.

[0036] Fig.12 An exemplary endoscopic video display is shown showing prostate laser annotation. DETAILED DESCRIPTION

[0037] The present disclosure may be further understood with reference to the following description and accompanying drawings, wherein like reference numerals are used to designate similar elements. Exemplary embodiments describe improvements to endoscopic displays for endoscopic procedures. These improvements include, for example, enhancement of endoscopic views and quantitative feedback of video object characteristics, particularly for urological procedures. Some common urological procedures include kidney stone treatment (e.g., lithotripsy), BPH (benign prostatic hyperplasia) surgery (e.g., GreenLight TM Laser surgery), prostate ablation, bladder tumor ablation, uterine fibroid treatment, diagnosis, etc. Many of these procedures can be described as "detect and treat".

[0038] A typical procedure has an imaging medium (such as LithoVue TM or any other endoscopic imager), mechanisms for delivering fluids (for clearing the field of view and / or dilating the lumen), and therapeutic mechanisms (e.g., laser, RF energy). Exemplary embodiments may improve physician decision making through quantitative feedback on video object characteristics, including physical size estimation, possible correlation with stone composition, distinguishing various types of tissue, etc. It may also be possible to provide information about, for example, sliding speed during laser surgery, bubble size and density, Randall plaque determination during kidney examination, water vapor therapy (e.g., Rezum TM ) or surgical guidance of capsule depth during BPH surgery to improve the cognitive load and surgical efficiency.

[0039] In an exemplary embodiment, super-resolution techniques are implemented to create a higher resolution mapping image from a series of lower resolution endoscopic images (e.g., ureteroscopic or cystoscopic images), and a current image is fused with the mapping image so that the combined image has the resolution of the mapping image. The exemplary techniques are particularly suitable for use in urological surgeries, however, certain embodiments may improve endoscopic viewing of other generally tubular patient anatomical structures (e.g., veins, esophagus, etc.) or non-tubular patient anatomical structures (e.g., stomach), as long as the surrounding tissue is continuous and arranged so that unique locations on the tissue can be mapped based on longitudinal and radial coordinates as described below. "Super-resolution" can generally be defined as an improved resolution image generated by fusing lower resolution images.

[0040] In this specification, the term "super-resolution" generally refers to the creation of an image space for mapping anatomical structures to improve video display during endoscopic interventions. Related embodiments describe the derivation of a structure from motion (SfM) depth map. Other exemplary techniques may improve the display in other ways. For example, a suitably designed deep convolutional neural network (CNN) may be applied directly to the pixel data to suppress noise in the image and / or highlight differences in tissue perfusion. The described techniques may be used alone or in combination, as described in detail below.

[0041] Figure 1 A system 100 for performing a urological surgical procedure is shown in accordance with various exemplary embodiments of the present disclosure. The system 100 includes an imager 104 for acquiring image frames of a patient's anatomy during a urological surgical procedure, and a fluid mechanism 106 for providing fluid (e.g., saline) to the anatomy to remove blood and debris that may impair the field of view of the imager 104. The endoscope 102 of this embodiment includes an optional electromagnetic (EM) tracker 108 to inform determination of the position of the endoscope 102 relative to the continuously deforming patient anatomy, as described in more detail below.

[0042] The system 100 may also include a treatment device 110 selected according to the nature of the urological surgical procedure. The treatment device 110 may be run through the endoscope 102 or may be located outside the endoscope 102. For example, the treatment device 110 may be, for example, a laser or shock wave generator for crushing kidney stones, or an electrosurgical resectoscope for removing prostate tissue. When the urological surgical procedure is used for diagnostic purposes (i.e., for examining anatomical structures rather than for treating disease), the treatment device may not be used. The exemplary embodiments are described with respect to urological system imaging, however, the exemplary embodiments are not limited thereto. Certain embodiments may be applicable (e.g., esophageal imaging) where a fluid mechanism is not used.

[0043] The system 100 includes a computer 112 that processes image frames provided by the imager 104 and provides the processed images to a display 114. The computer 112 and display 114 may be located at an integrated station such as an endoscope pendant. Other features for performing urological surgical procedures may be implemented at the endoscope pendant, including, for example, actuators that control the flow rate of fluid delivered by the fluid mechanism 106. The exemplary embodiments describe an algorithmic process for altering and enhancing displayed images, typically on a continuous basis or in any other desired manner.

[0044] Figure 2 A method 200 for improving the detail and / or field of view of an endoscopic video display according to a first exemplary embodiment is shown. The method 200 may be considered a super-resolution technique particularly suitable for close-range endoscopic imaging of tubular structures (e.g., the urethra). At 205, a spatial transformation is generated to transform the image from image coordinates to cylindrical coordinates, such as Figure 3 The spatial transformation is generated based on the optical geometry of the imager 104 . Figure 3 The innermost circle in may or may not be mapped, and it is expected that at smaller radii, the number of data points available in the video display may be too small to be used. As will be understood by those skilled in the art, the number of black circles on the left should correspond to the number of black horizontal lines on the right.

[0045] At 210, a set of transforms are generated for creating a map image in cylindrical space at a resolution that is an integer multiple of the resolution of the imager 104. For example, the map image may have a resolution, i.e., the number of pixels used to construct the map image, that is three times the resolution of the imager 104. Each pixel position in the map is represented by a vector of fixed dimension. For example, a vector for a given position may have, e.g., eight elements representing eight samples accumulated from that position over a plurality of image frames.

[0046] At 215, the set of preliminary images captured by the imager 104 are correlated to align in a cylindrical coordinate system. At 220, a spatial transformation is applied to the plurality of image samples to convert the images to cylindrical coordinates, and a mapped image is calculated from the samples. The mapped image has each image location defined in terms of a vector where outlier rejection is applied. For example, the outlier rejection may be a median filter.

[0047] At 225, a current image is captured by the imager, and the current image is correlated with the mapped image to optimize the alignment of the images in cylindrical space when applying the spatial transformation. For example, the correlation can be a cross-correlation between the images to measure the similarity between them, and the images are aligned to maximize the similarity. At 230, a spatial transformation is applied to the images based on the correlation. At 235, the current transformed image and the mapped image are combined (e.g., fused) in a cylindrical coordinate system. If the mapped image has sufficient data in an area surrounding the field of view of the current image, the field of view of the combined image can be enlarged.

[0048] At 240, an inverse spatial transform is applied to the combined image to generate an image with enhanced spatial resolution in the endoscopic coordinate system. At 245, the enhanced resolution image is displayed on the display 114 to guide the endoscopic procedure with improved detail and / or improved field of view (e.g., compared to the image originally captured at the endoscopic resolution).

[0049] A cylindrical transform of the image frame is added to the mapped image by adding pixel values ​​to corresponding mapped vectors at 250. The vectors may be arranged in order based on the time at which the samples were added, such that when a vector is full, the oldest samples are deleted and new pixel values ​​are added to empty spots.

[0050] Although the image processing steps discussed above are performed on a continuous basis as new image frames are captured by the imager 104, other schedules for image processing may be employed as will be appreciated by those skilled in the art. Thus, each new image frame is visually enhanced due to improved resolution based on the fusion of the new frame with the mapped image.

[0051] Figure 4 A method 300 for improving endoscopic video according to a second exemplary embodiment is shown. Method 300 is similar to method 200, but method 300 describes a non-rigid registration process to better compensate for optical distortion, tissue motion, and point of view (POV) changes. Non-rigid registration generally refers to applying multiple local transformations to an image to better correlate with a reference image (in this case, a mapped image). In this embodiment, different regions of the image can be transformed independently to better capture changes in position between regions.

[0052] In 305, a mapped image is calculated according to steps 205-220 of method 200. In 310, a scale space representation of the mapped image is calculated. Scale space representation generally refers to representing image data as a set of images that are gradually smoothed at different scales. For example, large scale structures in the mapped image are emphasized, while small scale structures are suppressed.

[0053] At 315, initially at a low spatial frequency, a current image is captured and the current image is correlated with the mapping image. At 320, the current image is fused with the mapping image and the current image is enhanced in accordance with steps 230-250 of method 200. As new images are acquired, the image processing steps are repeated. However, as image data continues to be collected to populate the mapping image, at 325, multiple independent regional transforms are developed by optimizing progressively smaller regions at progressively higher spatial frequency scales. Relative anatomical motion can be estimated.

[0054] At 330, a structure-from-motion (SFM) depth map is derived based on a nonlinear transformation. SFM generally refers to a mapping of three-dimensional structure from a series of images captured by a moving POV, where changes in the position and orientation (pose) of the imager are tracked. The depth map is continuously populated as new images are collected.

[0055] Elements of the aforementioned image processing methods 200, 300 may be used to further enhance the display in the following program-specific manner.

[0056] Kidney stone treatments, such as laser lithotripsy, may involve breaking the stone into many different sized pieces, some of which require further reduction, while some of which may be small enough to be removed or passed naturally. Tracking stone particles in ureteroscopic videos, inferring their size, and providing appropriate annotations to the physician can increase the speed, confidence, and accuracy of these interventions. Object sizing is described below with respect to kidney stones, however, other image features such as, for example, lesions, growths, and other tissue features can be sized and annotated in the endoscope display.

[0057] Figure 5 A method 400 for annotating an endoscopic video with object size information is shown. In 405, according to steps 205-215 of method 200, a spatial transformation and image map are generated, and a preliminary set of images are captured and correlated.

[0058] At 410, endoscope-related objects are identified and segmented in the preliminary set of images. Endoscope-related objects may include, for example, laser fibers used to perform laser lithotripsy. Endoscope-related object segmentation is performed using predefined feature maps and constrained registration geometry.

[0059] In 415, objects of interest are identified and segmented in the preliminary image set. Objects of interest may include, for example, kidney stones. As will be known to those skilled in the art, object segmentation is performed using image features and blob finding algorithms.

[0060] At 420 , a spatial transformation is applied to the preliminary set of images and a super-resolution scene map is calculated that excludes the identified endoscope-related objects.

[0061] At 425, a probabilistic depth map is derived according to steps 310-330 of method 300. The depth map is based on independent region transformations and implicit camera pose transformations from super-resolution map alignment. Steps 310-330 include continuous acquisition of images and correlation / addition to the scene map to continuously improve the resolution of the scene map.

[0062] At 430 , the size of the previously identified object of interest is estimated based on the depth information and the angular range in the currently captured image.

[0063] At 435, an object of interest, such as a kidney stone, is annotated on a display of the currently captured image. In one embodiment, the size of the object may be presented directly on the display. In another embodiment, the object may be annotated with brackets to show the boundaries of the object. The brackets may be color-coded to show the size classification of the object by comparing the size estimate of the object to a predefined treatment threshold. For example, when the size estimate indicates that the object needs to be further reduced, the bracket may be colored red, when the size estimate indicates that the object is small enough to be retrieved but too large to be naturally discharged, the bracket may be colored yellow, and when the size estimate indicates that the object can be naturally passed, the bracket may be colored green. In other words, a kidney stone may be annotated to indicate the minimum size of a duct through which the kidney stone can fit. Figure 6 An exemplary endoscopic video display showing a kidney stone being bracketed is shown.

[0064] In an alternative embodiment, the inter-frame correlation or super-resolution mapping described above is used to build a statistical description of color and albedo from multiple samples at each segmented pixel. Machine learning with clustering algorithms and in vivo training data is used to create a combined map for predicting membership probabilities within identified size groups.

[0065] SFM techniques may be employed to inform a process for tracking the position of an endoscope during endoscopic surgery. The SFM pose estimate of the endoscope, as determined from the endoscopic video, may be used in conjunction with position information from the EM tracker 108 and a segmented 3D reconstruction of the patient's anatomy derived from previously acquired images to refine the determination of the position of the endoscope relative to the continuously deforming patient anatomy.

[0066] Figure 7A method 500 for tracking the position of an endoscope relative to a continuously deforming anatomical structure is shown according to a first exemplary embodiment. At 505, a previously acquired 3D image volume of a target region of an anatomical structure is received and segmented. The 3D volume may be generated from a previously captured CT image or an image captured from a similar imaging medium. At 510, an SFM depth map is derived according to steps 305-330 of method 300. The depth map provides motion, position estimates, and confidence levels in the position estimates relative to the anatomical structure.

[0067] At 515, the six degrees of freedom (6DOF) position of endoscope 102 is determined using EM tracker 108 disposed near the tip of endoscope 102. The 6DOF position from 515 and the image populating the SFM depth map from 510 are received substantially simultaneously and may be correlated.

[0068] At 520, the position data from the EM tracker 108, the position data from the SFM map, and the fixed volume of the anatomy are combined to estimate a minimum deformation of the anatomy sufficient to keep the endoscope 102 within the surface boundaries of the anatomy. The resulting model assumes that the endoscope 102 is free to move within the cavity until the position data indicates that the endoscope 102 has contacted the segmented surface model. When the position data is shown to break through the surface of the fixed model, the anatomy is deformed to keep the position of the endoscope 102 within the surface. For example, the anatomy is displaced perpendicular to the surface direction.

[0069] The combined model and 6DOF endoscope position inform and improve the position estimate from the SFM depth map at 525. The improved estimate prevents runaway integration of noise.

[0070] At 530, the processed image is displayed on the display 114 along with real-time annotated navigation feedback. In one embodiment, the annotations may reflect the location of the stone or particle from the previous imaging. Additionally, by incorporating magnetic resonance (MR) tracking or optical tracking, the annotations may reflect the path that has been navigated. The method is repeated as images are acquired.

[0071] In alternative embodiments, SFM techniques known in the art may be used instead of the SFM depth map derived in method 300. Compared to known techniques, the SFM depth map from method 300 has a simpler geometry with fewer nonlinearities required to represent typical transformations (such as those resulting from camera motion). Conventional SFM algorithms also require geometric corrections to the camera optics, but these corrections do not extend to assumptions about surface continuity and orientation. If these assumptions are valid, they greatly reduce the numerical complexity of the solution, which in turn improves performance and reduces errors.

[0072] BPH (Benign Prostatic Hyperplasia) is a disease that can be treated with a variety of minimally invasive transurethral endoscopic procedures. Knowing the position of the endoscope relative to anatomical landmarks allows annotation of endoscopic videos, helping the physician measure distances and track ongoing multi-step interventions.

[0073] Figure 8 A method 600 for motion tracking in endoscopic video of a tubular structure (e.g., the urethra) according to a second exemplary embodiment is shown. The method 600 can provide annotations including a current position relative to anatomical landmarks, a desired position associated with an intended intervention, and a progress indicator relative to the intervention.

[0074] In 605, the camera rotation center and direction of travel are mapped. In 610, the optical system is mapped to derive the physical angle between each image pixel and the endoscope's direction of travel from the endoscope's POV. Mapping the optical system effectively defines a scalar (as in latitude) for each pixel of the image that specifies the angular distance from the camera axis (the North Pole, in the latitude analogy, except that 0° would be at the pole, and 90° at the equator, not the other way around). The "pole" location at 0° in the image is the point that keeps pointing to the same location when the camera is exactly forward or backward along the camera axis.

[0075] At 615, the distance to each anatomical pixel is determined for each image based on the mapping steps 605-610. The distance is measured from a reference plane perpendicular to the endoscope 102 to the object imaged at the pixel.

[0076] At 620, a deformed image is calculated in cylindrical space based on the determined distance and the rotation angle of the endoscope relative to its own axis. The deformed image is calculated relative to a vector from the endoscope rotation point to a reference direction feature in the endoscope video. An arbitrary feature is selected to define the reference direction (e.g., Figure 3 0° angle is set in ), so all subsequent mappings are directly or indirectly related to the reference direction.

[0077] At 625, image features are identified. A saliency measure (such as, for example, the Hessian determinant) is computed in scale space from the deformed cylindrical image to identify image features.

[0078] At 630, feature descriptors are constructed wherein image features associated with any portion of the interventional instrument within the field of view of the imager 104 are ignored. The feature descriptors are based on feature region pixel values ​​in the cylindrical image. At 635, features are matched based on the descriptors. In one embodiment, the present algorithm prefers to match one image to another, however, in another embodiment, a global mapping is possible and has potential advantages. Unlike previous approaches, scale and rotation invariant calculations do not need to be performed. By making inferences about the geometry of the scene map, the geometry of the feature should maintain a consistent scale and orientation from one image to the next. For example, the orientation of the feature is relative to the feature's vector to the vanishing point of the extrapolated cylinder so that if a nearby image is subsequently acquired with the endoscope rotated substantially along its axis, it will not affect the orientation angle, allowing the feature to be matched more directly.

[0079] In 640, outlier matches are rejected and the relative camera pose transform is inferred between the image frames. The annotations will be represented in terms of their relationship to reference features and reference planes, and thus once the camera pose transform is known, the positions can be transformed back to image space.

[0080] Fig. 9 An exemplary endoscopic video display is shown showing a current position relative to anatomical landmarks, a desired position associated with an intervention, and a progress indicator relative to the intervention.

[0081] Fig.10 A method 700 for motion tracking in an endoscopic video of a tubular structure according to a second exemplary embodiment is shown. The method 700 is similar to the method 600. In 705, a deformed image is calculated in a cylindrical space according to steps 605-620 of the method 600.

[0082] At 710, a mask image is generated in which pixels corresponding to instrument objects are ignored. If the current image is the first image, then at 715, the mask image is saved as a mapping image. If the current image is a subsequent image, then at 720, a best alignment with the mapping image is calculated based on a suitable alignment criterion. For example, the best alignment can be calculated based on a correlation such as a Hessian determinant weighted correlation or an entropy weighted correlation. At 725, the image is overlaid with the mapping based on the best alignment.

[0083] At 730, an updated map image is calculated using a statistical combination of the map position samples. For example, the map image can be calculated using the median intensity of each color channel across all images in which each pixel is defined. At 735, the relative position of the camera is inferred from the alignment with the map.

[0084] One procedure used to treat BPH is laser prostate ablation, which shrinks or removes prostate tissue by passing a laser through it. The procedure is characterized by the scanning range of the laser (i.e., the range of angles traversed by the laser) and the scanning speed (i.e., the change in angle per unit time). The scanning range, scanning speed, and scanning angle are determined by the power density of the laser, the distance from the tissue to the laser, the type of tissue, the laser frequency, and the size of the fiber. Annotations can be incorporated into the endoscopic video feed to facilitate laser treatment procedures by increasing the physician's awareness and accuracy.

[0085] Fig.11 A method for analyzing cystoscopic video feeds and providing annotations to inform laser therapy procedures (eg, GreenLight TM The method 800 of the present invention is to provide a method for detecting a laser device. At 805, image features of the laser device are stored. At 810, an image is captured. At 815, if the imager has captured at least a portion of the laser device in the image, a direction of the laser device and a distance relative to the camera are calculated.

[0086] At 820, color filtering is used to determine the distribution of laser beam scatter. At 825, if a scatter peak is captured in the image, the laser direction is estimated based on the scatter peak. The scatter peak is the intensity of the reflected laser light, i.e., the bright spot as the laser pulse comes and goes, and is used to infer the location of the anatomical structure directly in the path of the laser beam. In combination with direct visualization of the laser fiber end, an estimate of the laser direction (as projected onto a camera sensor) is determined.

[0087] The ablation distance is estimated using a pre-calibrated map of distances from the laser device to the scattering peaks at 830. The distance calculation of step 815 may be used to refine the distance estimate, if available.

[0088] At 835, an optimal laser scan rate is calculated based on the distance estimate and the laser power setting. At 840, a first annotation is provided on the display 114 showing the optimal scan rate and illustrating the relationship between the optimal scan rate and the detected scan rate.

[0089] At 845, a second annotation showing the ablation depth is provided on the display 114. The displayed ablation depth may be related to the distance to the prostate capsule.

[0090] At 850, visual cues are provided to show the appropriate rate and the relationship between the optimal rate and the measured rate and the scan angle.

[0091] Fig.12An exemplary endoscopic video display is shown showing prostate laser annotations including power level relative to an inferred optimal range, apparent average depth of laser targeted tissue, inferred scan angle, and optimal scan angle / scan speed calculated from power and average depth.

[0092] In an alternative embodiment, a video annotation is provided comparing the apparent ablation depth at each scan angle to the projected urethra-to-capsule distance measured in a previously acquired 3D image volume. Since the laser position is at the (previous) urethral location, the distance to the capsule measured in the previous imaging provides the limiting depth for the ablation procedure. The user can choose not to continue ablation within or outside the capsule.

[0093] In yet another alternative embodiment, the angle and depth estimates are combined with laser angle and insertion depth measurements calculated from a 3D instrument tracking system.

[0094] Differentiating different types of tissue in the endoscopic view could further facilitate various endoscopic procedures. Magnification of even very slight fluctuations in the alignment image frame could allow correlation with cardiac phase measurements and highlight differences in tissue perfusion.

[0095] In one embodiment, a properly designed convolutional neural network (CNN) can allow ad hoc tuning of filtering and amplification parameters to create a more flexible implementation than previous methods. The input volume of the CNN feeding the first convolutional layer will have a video frame depth of approximately a few seconds, 3 times the red, green, and blue channels. The output layer will have the same length and width as the video frame, and a depth of 1, and represent the degree of cardiac correlation. The network will be trained on uncompressed in vivo endoscopic video, where the cost function measures the maximum temporal correlation of a pixel with any one of a series of predefined microwaves, which are scaled to match the specified cardiac interval. The resulting value and the maximum microwave can drive the amplification of the signal to derive differences in tissue perfusion and, accordingly, highlight differences in tissue composition. The use of a CNN allows derivation to be performed in an unsupervised manner, whereas previous methods involved repeated user intervention to appropriately filter the raw image frames.

[0096] CNNs can also be used to produce images with low noise and enhanced detail. Deep convolutional neural networks (CNNs) can be applied directly to pixel data from endoscopes. In this atypical CNN, the output volume has the same depth as the input volume (approximately 3 for red, green & blue channels), but has a width & height that is the same integer multiple of the input width and height (super-resolution multiplier), and a fractional stride of the convolutional layer. The error term of the trained network is calculated by performing a discrete Hartley transform on the original pixels and the output values, and minimizing the squared difference unless the highest spatial frequency component of the input data offsets its Nyquist limit corresponding to the highest spatial frequency representable in the output data transform. In an alternative embodiment, a spatial frequency transform of the input data is performed on a multi-frame average image calculated using a correlation-based registration transform to help suppress noise.

[0097] Various existing super-resolution or SFM methods can be combined with elements of the disclosed method to improve existing techniques. For example, the known / calibrable optical geometry of an endoscope, as discussed with respect to method 200, can be used to improve existing super-resolution techniques. An image map generated based on the optical geometry of the endoscope may have 1 sample per pixel in each frame, and known super-resolution techniques (such as "shift and blend") can be used to transform the endoscopic image.

[0098] Other techniques for enhancing images include applying various filters to the pixel data, such as band, median or high frequency filters, and using edge detection on objects of interest through turbid or cloudy environments. These further techniques may be used alone or in combination with the above embodiments.

Claims

1. An endoscope system, comprising: an endoscopic imager configured to capture image frames of a target site within a living body; as well as A processor, wherein the processor is configured as: applying a spatial transformation to the preliminary set of image frames, the spatial transformation converting the image frames into cylindrical coordinates; Calculating a mapping image based on the spatially transformed image frame, each pixel position in the mapping image is defined by a vector of fixed dimension; computing a scale-space representation of the mapped image; capturing a further image comprising a plurality of separate regions; developing a nonlinear spatial transform including a separate spatial transform for each of said separate regions when a predetermined amount of image data for each of said separate regions has been acquired; and A structure from motion (SFM) depth map is derived from the nonlinear spatial transform.

2. The endoscope system according to claim 1, wherein: The SFM depth map is also based on tracking information of the endoscopic imager, the tracking information including a changing pose of the endoscopic imager between captured images.

3. The endoscope system according to any one of claims 1 to 2, wherein: The processor is further configured to: identifying and segmenting endoscope-related objects and objects of interest in the preliminary set of image frames; and The identified endoscopy-related objects are excluded when the spatial transformation is applied to the preliminary set of images.

4. The endoscope system according to claim 3, wherein: The processor is further configured to: The size of the object of interest is estimated based on the depth information and the angle range in the current image frame.

5. The endoscope system according to claim 4, further comprising: A display is configured to display the current image frame of the object of interest with annotations.

6. The endoscope system according to claim 5, wherein: The object of interest is a kidney stone.

7. The endoscope system according to any one of claims 1 to 6, further comprising: an electromagnetic (EM) tracker attached to the endoscopic imager, the EM tracker configured to provide tracking data including a six-degree-of-freedom position of the endoscopic imager, The processor is further configured to segment the previously acquired 3D image volume of the target site, The tracking data is combined with the segmented image volume and the SFM depth map to provide a position estimate of the endoscopic imager.

8. The endoscope system according to claim 7, wherein: The processor is further configured to: When the tracking data is shown to breach a surface of the segmented image volume, the segmented image volume is deformed.

Images